(amdgpu-dma-operations)= # AMDGPU DMA Operations ```{contents} :local: ``` ## Introduction DMA (or "Direct Memory Access") operations transfer data between different kinds of memory directly without occupying registers in the invoking wave. They are usually {ref}`asynchronous` asynchronous, and require the user to explicitly track completion using {ref}`asyncmarks`. All DMA operations support the same cache modifiers as ordinary load/store operations from registers. They cannot be performed atomically. ### GFX9 DMA Each GFX9 DMA instruction has a synchronous counterpart (e.g., ``@llvm.amdgcn.load.to.lds`` for ``@llvm.amdgcn.load.async.to.lds``). The synchronous variants perform the same operation, but the compiler automatically ensures completion before their side-effects are used. GFX9 DMA instructions implement volatile (via ``aux/cpol`` bit 31) and nontemporal (via metadata) as if they were loads from the global address space. **Flat/Global Addressing** ```llvm void @llvm.amdgcn.load[.async].to.lds.pN( ptr addrspace(N) %src, ; base pointer to load from (per-lane) ptr addrspace(3) %lds_base, ; LDS base pointer (wave-uniform) i32 immarg %size, ; data byte size: 1/2/4 (12/16 for gfx950) i32 immarg %offset, ; offset applied to both src and LDS address i32 immarg %cpol) ; cache policy ``` Loads data from global memory to LDS. The data size can be 1, 2, or 4 bytes (gfx950 also allows 12 or 16 bytes). The LDS address is implicitly offset by ``4 * lane_id`` bytes for sizes up to 4 bytes, and by ``16 * lane_id`` bytes for larger sizes. The ``%lds_base`` pointer must be wave-uniform. The source pointer is overloaded on address space. Supported address spaces are flat (0), global (1), and buffer fat pointer (7). ``@llvm.amdgcn.load[.async].to.lds.p7`` (buffer pointer) is lowered to ``@llvm.amdgcn.raw.ptr.buffer.load[.async].lds`` before instruction selection. ```llvm void @llvm.amdgcn.global.load[.async].lds( ptr addrspace(1) %src, ; global base pointer to load from (per-lane) ptr addrspace(3) %lds_base, ; LDS base pointer (wave-uniform) i32 immarg %size, ; data byte size: 1/2/4 (12/16 for gfx950) i32 immarg %offset, ; offset applied to both global and LDS address i32 immarg %cpol) ; cache policy ``` This is identical to ``@llvm.amdgcn.load[.async].to.lds.p1``. **Buffer Addressing** ```llvm void @llvm.amdgcn.{raw|struct}[.ptr].buffer.load[.async].lds( %rsrc, ; buffer resource descriptor (SGPR): ; <4 x i32> or ptr addrspace(8) ptr addrspace(3) %lds_base, ; LDS base pointer (wave-uniform) i32 immarg %size, ; data byte size: 1/2/4 (12/16 for gfx950) [i32 %vindex,] ; VGPR buffer index (struct variants only) i32 %voffset, ; VGPR offset (included in bounds checking) i32 %soffset, ; SGPR/imm offset (excluded from bounds checking) i32 immarg %offset, ; imm offset (included in bounds checking) i32 immarg %cpol) ; cache policy ``` Loads data from a buffer resource to LDS. The ``%lds_base`` pointer must be wave-uniform. The intrinsics differ in two orthogonal ways: - **raw** vs **struct**: The ``struct`` variants add a ``%vindex`` argument for indexed buffer addressing. - **ptr** vs non-ptr: The ``ptr`` variants use ``ptr addrspace(8)`` for the buffer resource descriptor; the non-ptr variants use ``<4 x i32>``. ### GFX1250 GFX1250 LDS DMA instructions implement nontemporal (via metadata) as if they were loads from the global address space. Tensor DMA instructions do not support volatile or nontemporal. **Global Addressing** ```llvm void @llvm.amdgcn.{global|cluster}.load.async.to.lds.b( ptr addrspace(1) %src, ; global base pointer to load from (per-lane) ptr addrspace(3) %lds_base, ; LDS base pointer (per-lane) i32 immarg %offset, ; offset applied to both global and LDS address i32 immarg %cpol, ; cache policy [i32 %m0]) ; workgroup broadcast mask, cluster variants only (in M0) ``` The bit-size encoded in the name can be 8, 32, 64 or 128. Loads data from global memory to LDS. The ``%offset`` is applied to both the global and LDS addresses. The ``cluster`` variants add a ``%m0`` argument for workgroup broadcast. The broadcast mask selects which workgroups within a cluster participate in the load. ```llvm void @llvm.amdgcn.global.store.async.from.lds.b( ptr addrspace(1) %dst, ; global base pointer to store to (per-lane) ptr addrspace(3) %lds_base, ; LDS base pointer to load from (per-lane) i32 immarg %offset, ; offset applied to both global and LDS address i32 immarg %cpol) ; cache policy ``` Stores data from LDS to global memory. **Tensor Addressing** ```llvm void @llvm.amdgcn.tensor.{load.to|store.from}.lds( <4 x i32> %desc0, ; D# group 0 <8 x i32> %desc1, ; D# group 1 <4 x i32> %desc2, ; D# group 2 (zero-init for D# up to 2D) <4 x i32> %desc3, ; D# group 3 (zero-init for D# up to 2D) <8 x i32> %desc4, ; D# group 4 (reserved, use zeroinitializer) i32 immarg %cpol) ; cache policy ``` Loads or stores data between global memory and LDS using a tensor descriptor (D#). The descriptor is split across multiple groups. GFX1250 supports up to 4 descriptor groups; ``%desc4`` is reserved for future targets and must be zero-initialized. Despite the absence of ``.async`` in their names, these intrinsics are asynchronous. All arguments must be wave-uniform.