# Hardware-assisted AddressSanitizer Design Documentation

This page is a design document for
**hardware-assisted AddressSanitizer** (or **HWASAN**)
a tool similar to {doc}`AddressSanitizer`,
but based on partial hardware assistance.

## Introduction

{doc}`AddressSanitizer`
tags every 8 bytes of the application memory with a 1 byte tag (using *shadow memory*),
uses *redzones* to find buffer-overflows and
*quarantine* to find use-after-free.
The redzones, the quarantine, and, to a lesser extent, the shadow, are the
sources of AddressSanitizer's memory overhead.
See the [AddressSanitizer paper][addresssanitizer paper] for details.

AArch64 has [Address Tagging][address tagging] (or top-byte-ignore, TBI), a hardware feature that allows
software to use the 8 most significant bits of a 64-bit pointer as
a tag. HWASAN uses [Address Tagging][address tagging]
to implement a memory safety tool, similar to {doc}`AddressSanitizer`,
but with smaller memory overhead and slightly different (mostly better)
accuracy guarantees.

Intel's [Linear Address Masking][linear address masking] (LAM) also provides address tagging for
x86_64, though it is not widely available in hardware yet. For x86_64, HWASAN
has a limited implementation using page aliasing instead.

## Algorithm

- Every heap/stack/global memory object is forcibly aligned by `TG` bytes
  (`TG` is e.g. 16 or 64). We call `TG` the **tagging granularity**.
- For every such object a random `TS`-bit tag `T` is chosen (`TS`, or tag size, is e.g. 4 or 8)
- The pointer to the object is tagged with `T`.
- The memory for the object is also tagged with `T` (using a `TG=>1` shadow memory)
- Every load and store is instrumented to read the memory tag and compare it
  with the pointer tag, exception is raised on tag mismatch.

For a more detailed discussion of this approach see <https://arxiv.org/pdf/1802.09517.pdf>

### Short granules

A short granule is a granule of size between 1 and `TG-1` bytes. The size
of a short granule is stored at the location in shadow memory where the
granule's tag is normally stored, while the granule's actual tag is stored
in the last byte of the granule. This means that in order to verify that a
pointer tag matches a memory tag, HWASAN must check for two possibilities:

- the pointer tag is equal to the memory tag in shadow memory, or
- the shadow memory tag is actually a short granule size, the value being loaded
  is in bounds of the granule and the pointer tag is equal to the last byte of
  the granule.

Pointer tags between 1 to `TG-1` are possible and are as likely as any other
tag. This means that these tags in memory have two interpretations: the full
tag interpretation (where the pointer tag is between 1 and `TG-1` and the
last byte of the granule is ordinary data) and the short tag interpretation
(where the pointer tag is stored in the granule).

When HWASAN detects an error near a memory tag between 1 and `TG-1`, it
will show both the memory tag and the last byte of the granule. Currently,
it is up to the user to disambiguate the two possibilities.

## Instrumentation

### Memory Accesses

In the majority of cases, memory accesses are prefixed with a call to
an outlined instruction sequence that verifies the tags. The code size
and performance overhead of the call is reduced by using a custom calling
convention that

- preserves most registers, and
- is specialized to the register containing the address, and the type and
  size of the memory access.

Currently, the following sequence is used:

```objdump
// int foo(int *a) { return *a; }
// clang -O2 --target=aarch64-linux-android30 -fsanitize=hwaddress -S -o - load.c
[...]
foo:
      stp     x30, x20, [sp, #-16]!
      adrp    x20, :got:__hwasan_shadow               // load shadow address from GOT into x20
      ldr     x20, [x20, :got_lo12:__hwasan_shadow]
      bl      __hwasan_check_x0_2_short_v2            // call outlined tag check
                                                      // (arguments: x0 = address, x20 = shadow base;
                                                      // "2" encodes the access type and size)
      ldr     w0, [x0]                                // inline load
      ldp     x30, x20, [sp], #16
      ret

[...]
__hwasan_check_x0_2_short_v2:
      sbfx    x16, x0, #4, #52                        // shadow offset
      ldrb    w16, [x20, x16]                         // load shadow tag
      cmp     x16, x0, lsr #56                        // extract address tag, compare with shadow tag
      b.ne    .Ltmp0                                  // jump to short tag handler on mismatch
.Ltmp1:
      ret
.Ltmp0:
      cmp     w16, #15                                // is this a short tag?
      b.hi    .Ltmp2                                  // if not, error
      and     x17, x0, #0xf                           // find the address's position in the short granule
      add     x17, x17, #3                            // adjust to the position of the last byte loaded
      cmp     w16, w17                                // check that position is in bounds
      b.ls    .Ltmp2                                  // if not, error
      orr     x16, x0, #0xf                           // compute address of last byte of granule
      ldrb    w16, [x16]                              // load tag from it
      cmp     x16, x0, lsr #56                        // compare with pointer tag
      b.eq    .Ltmp1                                  // if matches, continue
.Ltmp2:
      stp     x0, x1, [sp, #-256]!                    // save original x0, x1 on stack (they will be overwritten)
      stp     x29, x30, [sp, #232]                    // create frame record
      mov     x1, #2                                  // set x1 to a constant indicating the type of failure
      adrp    x16, :got:__hwasan_tag_mismatch_v2      // call runtime function to save remaining registers and report error
      ldr     x16, [x16, :got_lo12:__hwasan_tag_mismatch_v2] // (load address from GOT to avoid potential register clobbers in delay load handler)
      br      x16
```

### Heap

Tagging the heap memory/pointers is done by `malloc`.
This can be based on any malloc that forces all objects to be TG-aligned.
`free` tags the memory with a different tag.

### Stack

Stack frames are instrumented by aligning all non-promotable allocas
by `TG` and tagging stack memory in function prologue and epilogue.

Tags for different allocas in one function are **not** generated
independently; doing that in a function with `M` allocas would require
maintaining `M` live stack pointers, significantly increasing register
pressure. Instead we generate a single base tag value in the prologue,
and build the tag for alloca number `M` as `ReTag(BaseTag, M)`, where
ReTag can be as simple as exclusive-or with constant `M`.

Stack instrumentation is expected to be a major source of overhead,
but could be optional.

### Globals

Most globals in HWASAN instrumented code are tagged. This is accomplished
using the following mechanisms:

- The address of each global has a static tag associated with it. The first
  defined global in a translation unit has a pseudorandom tag associated
  with it, based on the hash of the file path. Subsequent global tags are
  incremental from the previously-assigned tag.
- The global's tag is added to its symbol address in the object file's symbol
  table. This causes the global's address to be tagged when its address is
  taken.
- When the address of a global is taken directly (i.e. not via the GOT), a special
  instruction sequence needs to be used to add the tag to the address,
  because the tag would otherwise take the address outside of the small code
  model (4GB on AArch64). No changes are required when the address is taken
  via the GOT because the address stored in the GOT will contain the tag.
- An associated `hwasan_globals` section is emitted for each tagged global,
  which indicates the address of the global, its size and its tag. These
  sections are concatenated by the linker into a single `hwasan_globals`
  section that is enumerated by the runtime (via an ELF note) when a binary
  is loaded and the memory is tagged accordingly.

A complete example is given below:

```objdump
// int x = 1; int *f() { return &x; }
// clang -O2 --target=aarch64-linux-android30 -fsanitize=hwaddress -S -o - global.c

[...]
f:
      adrp    x0, :pg_hi21_nc:x            // set bits 12-63 to upper bits of untagged address
      movk    x0, #:prel_g3:x+0x100000000  // set bits 48-63 to tag
      add     x0, x0, :lo12:x              // set bits 0-11 to lower bits of address
      ret

[...]
      .data
.Lx.hwasan:
      .word   1

      .globl  x
      .set x, .Lx.hwasan+0x2d00000000000000

[...]
      .section        .note.hwasan.globals,"aG",@note,hwasan.module_ctor,comdat
.Lhwasan.note:
      .word   8                            // namesz
      .word   8                            // descsz
      .word   3                            // NT_LLVM_HWASAN_GLOBALS
      .asciz  "LLVM\000\000\000"
      .word   __start_hwasan_globals-.Lhwasan.note
      .word   __stop_hwasan_globals-.Lhwasan.note

[...]
      .section        hwasan_globals,"ao",@progbits,.Lx.hwasan,unique,2
.Lx.hwasan.descriptor:
      .word   .Lx.hwasan-.Lx.hwasan.descriptor
      .word   0x2d000004                   // tag = 0x2d, size = 4
```

### Error reporting

Errors are generated by the `HLT` instruction and are handled by a signal handler.

### Attribute

HWASAN uses its own LLVM IR Attribute `sanitize_hwaddress` and a matching
C function attribute. An alternative would be to re-use ASAN's attribute
`sanitize_address`. The reasons to use a separate attribute are:

- Users may need to disable ASAN but not HWASAN, or vise versa,
  because the tools have different trade-offs and compatibility issues.
- LLVM (ideally) does not use flags to decide which pass is being used,
  ASAN or HWASAN are being applied, based on the function attributes.

This does mean that users of HWASAN may need to add the new attribute
to the code that already uses the old attribute.

## Comparison with AddressSanitizer

HWASAN:
: - Is less portable than {doc}`AddressSanitizer`
    as it relies on hardware [Address Tagging][address tagging] (AArch64).
    Address Tagging can be emulated with compiler instrumentation,
    but it will require the instrumentation to remove the tags before
    any load or store, which is infeasible in any realistic environment
    that contains non-instrumented code.
  - May have compatibility problems if the target code uses higher
    pointer bits for other purposes.
  - May require changes in the OS kernels (e.g. Linux seems to dislike
    tagged pointers passed from address space:
    <https://www.kernel.org/doc/Documentation/arm64/tagged-pointers.txt>).
  - **Does not require redzones to detect buffer overflows**,
    but the buffer overflow detection is probabilistic, with roughly
    `1/(2**TS)` chance of missing a bug (6.25% or 0.39% with 4 and 8-bit TS
    respectively).
  - **Does not require quarantine to detect heap-use-after-free,
    or stack-use-after-return**.
    The detection is similarly probabilistic.

The memory overhead of HWASAN is expected to be much smaller
than that of AddressSanitizer:
`1/TG` extra memory for the shadow
and some overhead due to `TG`-aligning all objects.

## Security Considerations

HWASAN is a bug detection tool and its runtime is not meant to be
linked against production executables. While it may be useful for testing,
HWASAN's runtime was not developed with security-sensitive
constraints in mind and may compromise the security of the resulting executable.

## Supported architectures

HWASAN relies on [Address Tagging][address tagging] which is only available on AArch64.
For other 64-bit architectures it is possible to remove the address tags
before every load and store by compiler instrumentation, but this variant
will have limited deployability since not all of the code is
typically instrumented.

On x86_64, HWASAN utilizes page aliasing to place tags in userspace address
bits. Currently only heap tagging is supported. The page aliases rely on
shared memory, which will cause heap memory to be shared between processes if
the application calls `fork()`. Therefore x86_64 is really only safe for
applications that do not fork.

HWASAN does not currently support 32-bit architectures since they do not
support [Address Tagging][address tagging] and the address space is too constrained to easily
implement page aliasing.

## Related Work

- [SPARC ADI][sparc adi] and [Arm MTE][arm mte] implement a similar tool mostly in hardware.
- [Effective and Efficient Memory Protection Using Dynamic Tainting][effective and efficient memory protection using dynamic tainting] discusses
  similar approaches ("lock & key").
- [Watchdog][watchdog] discussed a heavier, but still somewhat similar
  "lock & key" approach.
- *TODO: add more "related work" links. Suggestions are welcome.*

[address tagging]: http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.den0024a/ch12s05s01.html
[addresssanitizer paper]: https://www.usenix.org/system/files/conference/atc12/atc12-final39.pdf
[arm mte]: https://developer.arm.com/documentation/108035/0100/Introduction-to-the-Memory-Tagging-Extension
[effective and efficient memory protection using dynamic tainting]: https://www.cc.gatech.edu/~orso/papers/clause.doudalis.orso.prvulovic.pdf
[linear address masking]: https://software.intel.com/content/www/us/en/develop/download/intel-architecture-instruction-set-extensions-programming-reference.html
[sparc adi]: https://lazytyped.blogspot.com/2017/09/getting-started-with-adi.html
[watchdog]: https://www.cis.upenn.edu/acg/papers/isca12_watchdog.pdf
