feat(tensilelite): Add TDMLoadWaveSync (#9748)
JIRA ID: AIHPBLAS-4146
## Motivation
<!-- Explain the purpose of this PR and the goals it aims to achieve.
-->
Insert a workgroup barrier between two groups of `tensor_load_to_lds` —
an urgent, waited-for group and a deferrable prefetch-ahead group. This
PR automates that barrier insertion in StinkyTofu.
## Technical Details
<!-- Explain the changes along with any relevant GitHub links. -->
Adds `TDMLoadWaveSyncPass` (gfx1250), gated by a new off-by-default
TensileLite solution param `TDMLoadWaveSync`. It runs after tensorcnt
insertion and a kernel-scope CFGBuilder, and inserts a barrier
(`s_barrier_signal -1` / `s_barrier_wait -1`) between the two TDM
groups.
- **Trigger / classification.** Each `s_wait_tensorcnt` is a trigger;
its drained token set S (read from `MemTokenData`, now attached to
emitted waits by `StinkyWaitCntInsertionPass`) splits preceding loads
into urgent (token ∈ S) vs deferrable (token ∉ S). Wait-group identity
== memtoken.
- **Backward monotone dataflow.** A per-trigger worklist scans backward,
carrying a `ScanState = {frontier of open deferrable groups, workingS}`.
`workingS` is the still-urgent subset of S and shrinks by a wait's
drained tokens when the scan crosses it (an already-drained token is no
longer urgent on that path); the flow ends when `workingS` empties or an
urgent load is reached. State is unioned at CFG joins and iterated to a
fixpoint — O(blocks × waits × lattice-height), no path enumeration. This
covers prologue→loop, steady-state in-loop, diamond, and nested
diamond-in-loop control flow.
- **No-redundant / tagging.** A pre-existing barrier already at a split
suppresses insertion; such barriers are annotated in the emitted asm (a
`TDM wait-group barrier` comment with the wait's drained set) so they
are distinguishable from freshly-inserted and plain barriers.
- **Supporting change.** `StinkyWaitCntInsertionPass` tags each emitted
`s_wait_tensorcnt` with the exact drained-token union
(`WaitDataflow::drainedTensorTokens`).
- **Naming.** `TDMLoadWaveSync` is added to the kernel-name parameter
set, so two solutions differing only by it do not dedup to one code
object.
- **Debug aid.** `DumpMemTokenIRStructurePass` (registered in
`stinkytofu-opt` as `--DumpMemTokenIRStructurePass`) dumps a reduced IR
view — branch structure + memtoken-candidate instructions + comments —
for inspecting wait-group flow.
TensileLite wiring: `ValidParameters`/`GlobalParameters` (default
`[False]`), `Solution.py` (reject unless `ScheduleIterAlg=4`; disable
when TDM is off or `NumWaves<=1`), `KernelWriter.py` (thread into the
StinkyTofu module options), and a benchmark section in
`tdm_split_gfx1250.yaml`.
## Test Plan
<!-- Explain any relevant testing done to verify this PR. -->
- `tdm_load_wave_sync.stir`: 22 FileCheck cases covering
flat/idempotent, cross-BB prologue→loop, steady-state in-loop, diamonds,
nested diamond-in-loop, per-arm token-exact barriers, multi-token
drained sets with partial-drain arms, and the no-memtoken /
intervening-drain negatives.
- `tdm_load_wave_sync_single_wave.stir`: NumWaves==1 no-op (pass inserts
nothing; matches the `Solution.py` guard that disables `TDMLoadWaveSync`
when `NumWaves<=1`).
- Producer-side memtoken checks in
`waitcnt_insertion_tensor_anchor_ds_read_num_waves_1.stir`.
- TensileLite: `tdm_split_gfx1250.yaml` exercises `TDMLoadWaveSync:
[False, True]`
(MXF8SS Block32, ScheduleIterAlg=4, TDMSplit, NumWaves>1) via the tox
kernel-gen tests.
## Test Result
<!-- Briefly summarize test outcomes. -->
Tests passed
## Submission Checklist
- [X] Look over the contributing guidelines at
https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. P
Peter committed
aadfb39aed4e86653f64f3fe08c343c0b61ebeae
Parent: 827eb7f
Committed by GitHub <noreply@github.com>
on 8/14/2026, 3:53:18 PM