(amdgpu-async-operations)=

# AMDGPU Asynchronous Operations


## Introduction

Asynchronous operations are operations whose completion is not tracked
internally by the compiler. A thread that initiates one or more async operations can use
*asyncmarks* to track their completion.

- Most {ref}`DMA operations <amdgpu-dma-operations>` are asynchronous.

## Asyncmarks

An *asyncmark* created by a thread can be used to track async operations
initiated by that thread.

### Stages

A *stage* names a kind of async operation. Each async operation *belongs to* the
one stage determined by the instruction that initiates it.

The stages are:

| Bit | Stage | Async operations |
|---|---|---|
| 0 | `TENSOR` | tensor loads and stores |
| 1 | `GLOBAL_LOAD_ASYNC_TO_LDS` | global loads async to LDS |
| 2 | `GLOBAL_LOAD_ASYNC_TO_LDS_MCAST` | multicast (cluster) global loads async to LDS |
| 3 | `GLOBAL_STORE_ASYNC_FROM_LDS` | async global stores from LDS |
| 5 | `BUFFER_GLOBAL_LOAD` | buffer loads to LDS and pre-gfx1250 global loads to LDS |

Bits 4 and 6 through 10 are reserved for future async operations, and no
operation belongs to them yet.

Which async operations a given subtarget actually has is described in
{ref}`AMDGPU DMA Operations <amdgpu-dma-operations>`. A stage exists on every
subtarget that supports asyncmarks, whether or not that subtarget has any
operation belonging to it.

### Stage Masks

`asyncmark` intrinsics take a *stage mask*: an 11-bit value in which a set bit
names a stage.

The zero mask, which sets no bit, is given the special meaning "every stage".
This ensures if new stages are added that programs will continue to wait on all
stages, if that was their intention.

Bits not specified in the table above are reserved for future use. It is not an
error to set them, but it could mean you have more conservative waits than
necessary when the future stages are added.

Setting a bit that is neither listed nor reserved is an error, and is rejected
by the IR verifier.

[Informational note --- Since the stages are independent, any mix of masks is
valid. A program is easiest to reason about by considering the sequence of each
stage separately.]

### Current Sequence

The abstract machine maintains a separate sequence of asyncmarks *for each
stage* during the execution of a function body, which excludes any asyncmarks
produced by calls to other functions encountered in the currently executing
function. The state of the sequence for a stage `S` at each program point in
the function is called the *current sequence of* `S`.

The sequences of distinct stages are independent: appending to one does not
affect the length or contents of any other, even though multiple sequences may
be appended to with a single call to asyncmark (by naming multiple stages).

### `@llvm.amdgcn.asyncmark(i32 %K)`

Appends an asyncmark to the current sequence of every stage named in the mask
`K`. The sequences of the stages that `K` does not name are unaffected. `K` must
be a constant stage mask.

### `@llvm.amdgcn.wait.asyncmark(i16 %N, i32 %K)`

For every stage named by the mask `K`, ensures that the length of the current
sequence of that stage is at most `N` by removing asyncmarks from the start of
that sequence if it is longer than `N`. The sequences of the stages that `K`
does not name are unaffected. `K` must be a constant stage mask.

### Completion of Asyncmarks

An asyncmark `M`, produced by an `asyncmark` operation `X`, is *completed-at* 
a `wait.asyncmark()` operation `Y` with mask `B` in the same function body if:

- `X` is *program-ordered* before `Y`, and
- `B` names the stage of the sequence that `M` belongs to, and
- `M` is not in the current sequence at any operation `Z` that
  immediately follows `Y` in *program-order*.

## Completion of Async Operations

An async operation executes outside the thread that initiated it, i.e., it is
not related in *program-order* with any other operations from that thread. But
the thread can use an asyncmark to ensure that the async operation is
*completed-at* some later operation.

An async operation `A` *initiated-by* an instruction `I` is *completed-at* some
`wait.asyncmark()` operation `Y` with stage mask `B` if there exists an
`asyncmark()` operation `X` with mask `C`:

- `I` is *program-ordered* before `X`, and
- `I` belongs to a stage in `C`, and
- `X` is *completed-at* `Y`.

### happens-before

When an instruction `I` initiates an async operation `A`, `I` *happens-before*
`A`.

If `A` is *completed-at* a `wait.asyncmark()` operation `Y`, then `A`
*happens-before* `Y`.

## Examples

Most examples below use the zero mask, so every asyncmark tracks every async
operation and there is only one sequence to reason about. The
{ref}`interleaved stages <amdgpu-async-interleaved-stages>` example shows what
narrower masks add.

### Uneven blocks of async operations

```c++
void foo(global int *g, local int *l) {
  // first block
  async_load_to_lds(l, g);
  async_load_to_lds(l, g);
  async_load_to_lds(l, g);
  asyncmark(0);

  // second block; longer
  async_load_to_lds(l, g);
  async_load_to_lds(l, g);
  async_load_to_lds(l, g);
  async_load_to_lds(l, g);
  async_load_to_lds(l, g);
  asyncmark(0);

  // third block; shorter
  async_load_to_lds(l, g);
  async_load_to_lds(l, g);
  asyncmark(0);

  // Wait for first block
  wait.asyncmark(2, 0);
}
```

### Software pipeline

```c++
void foo(global int *g, local int *l) {
  // first block
  asyncmark(0);

  // second block
  asyncmark(0);

  // third block
  asyncmark(0);

  for (;;) {
    wait.asyncmark(2, 0);
    // use data

    // next block
    asyncmark(0);
  }

  // flush one block
  wait.asyncmark(2, 0);

  // flush one more block
  wait.asyncmark(1, 0);

  // flush last block
  wait.asyncmark(0, 0);
}
```

### Ordinary function call

```c++
extern void bar(); // may or may not initiate async operations

void foo(global int *g, local int *l) {
    // first block
    asyncmark(0);

    // second block
    asyncmark(0);

    // function call
    bar();

    // third block
    asyncmark(0);

    // wait for the second block
    wait.asyncmark(1, 0);

    // wait for the third block, including bar()
    wait.asyncmark(0, 0);
}
```

(amdgpu-async-interleaved-stages)=
### Interleaved stages

Two kinds of async operation are in flight at once. Each is marked with a mask
naming only its own stage, so each can be waited for without waiting for the
other.

```c++
void foo(global int *g, local int *l, tensor_desc t) {
  // Start a long tensor load and mark it in the TENSOR stage alone.
  tensor_load_to_lds(l, t);
  asyncmark(TENSOR);

  // Start a short LDS load and mark it in its own stage alone.
  async_load_to_lds(l, g);
  asyncmark(GLOBAL_LOAD_ASYNC_TO_LDS);

  // Wait for the LDS load only. The tensor load is still in flight: this wait
  // does not name TENSOR, so it neither counts nor removes that asyncmark.
  wait.asyncmark(0, GLOBAL_LOAD_ASYNC_TO_LDS);

  // perform synchronization / use the data loaded by async_load_to_lds

  // Now wait for the tensor load.
  wait.asyncmark(0, TENSOR);
}
```

Had both marks used the zero mask, the first wait would have drained the tensor
load as well, and the overlap would have been lost.

The same applies to two stages tracked by one hardware counter. Here both stages
use the async counter, but the sequences are still separate:

```c++
void foo(global int *g, local int *l) {
  async_load_to_lds(l, g);
  asyncmark(GLOBAL_LOAD_ASYNC_TO_LDS);    // X

  async_store_from_lds(g, l);
  asyncmark(GLOBAL_STORE_ASYNC_FROM_LDS); // Y

  // Completes X. Y is in a different sequence and is not completed here, even
  // though both stages are tracked by the same counter.
  wait.asyncmark(0, GLOBAL_LOAD_ASYNC_TO_LDS);
}
```

A mask need not name just one stage. An asyncmark whose mask names several is
appended to each of their sequences, and a wait whose mask names several trims
each of them:

```c++
void foo(global int *g, local int *l, tensor_desc t) {
  tensor_load_to_lds(l, t);
  async_load_to_lds(l, g);

  // One asyncmark, appended to both sequences.
  asyncmark(TENSOR | GLOBAL_LOAD_ASYNC_TO_LDS);

  // Removes the mark in GLOBAL_LOAD_ASYNC_TO_LDS. The mark in TENSOR is a
  // separate asyncmark and stays where it is.
  wait.asyncmark(0, GLOBAL_LOAD_ASYNC_TO_LDS);

  // Removes the mark in TENSOR.
  wait.asyncmark(0, TENSOR);
}
```

## Implementation notes

[This section is informational.]

### Function Calls

In general, at a function call, if the caller uses sufficient waits to track
its own async operations, the actions performed by the callee cannot affect
correctness. But inlining such a call may result in redundant waits.

```c++
void foo() {
  ...
  asyncmark(0);         // X
  ...                   // no wait.asyncmark()
}

void bar() {
  asyncmark(0);         // B
  asyncmark(0);         // C
  foo();
  wait.asyncmark(1, 0); // D
}
```

Before inlining, it is unspecified whether `X` is *completed-at* `D`, while
`C` is **not** *completed-at* `D`. The programmer can only rely on `B`
being *completed-at* `D`.

```c++
void bar() {
  asyncmark(0);         // B
  asyncmark(0);         // C
  ...
  asyncmark(0);         // X
  ...                   // no wait.asyncmark()
  wait.asyncmark(1, 0); // D
}
```

After inlining, `C` is also *completed-at* `D` and `X` is **not**
*completed-at* `D`.

Conversely, a `wait.asyncmark` call inside a callee cannot be used to track
asyncmarks from the caller, since this `wait.asyncmark` can only
observe the current sequence of the callee.

```c++
void foo() {
  ...                   // no asyncmark()
  wait.asyncmark(0, 0); // Y
  ...
}

void bar() {
  asyncmark(0);         // B
  asyncmark(0);         // C
  foo();
  wait.asyncmark(1, 0); // D
}
```

In the above example, it is unspecified whether `B` and `C` in `bar()` are
*completed-at* `Y`, because they are not included in the sequence that can be
examined at `Y`.

```c++
void bar() {
  asyncmark(0);         // B
  asyncmark(0);         // C
  ...                   // no asyncmark()
  wait.asyncmark(0, 0); // Y
  ...
  wait.asyncmark(1, 0); // D
}
```

After inlining, both `B` and `C` are *completed-at* `Y`.

### Optimization

The implementation may eliminate/modify asyncmark/wait intrinsics in the
following cases. These are just examples and not meant to be an exhaustive list.

1. An `asyncmark` operation which remains in a current sequence along every
   path that reaches the function exit.

   ```c++
   void foo() {
     ...
     asyncmark(0);         // X
     ...                   // no wait.asyncmark()
   }
   ```

   Here, the zero mask names every stage, and each independent mark remains, so
   the entire mark `X` can be eliminated. If only some sequences qualified, then
   the `asyncmark` operation could be modified to remove just those stages from
   its mask.

2. A `wait.asyncmark` which sees an empty sequence of asyncmarks along every
   path that reaches it.

   ```c++
   void foo() {
     ...                   // no asyncmark()
     wait.asyncmark(0, 0); // Y
     ...
   }
   ```

   Here, as with `X` above, every sequence qualifies, so `Y` can be eliminated.
   If only some sequences qualified, the `wait.asyncmark` could be modified to
   only wait for the relevant stages.

3. An `asyncmark` operation that has mask bits that don't refer to any
   instruction.

    ```c++
   void foo() {
     ...
     global_load_async_to_lds
     asyncmark(0)
     ...
   }
   ```

   Here, the program can be refined to -

   ```c++
   void foo() {
     ...
     global_load_async_to_lds
     asyncmark(ASYNC_LOAD_LDS)
     ...
   }
   ```

   Since only that particular stage is actually relevant to ensuring the
   completion of async operations.
