AMD GPU Wedge: Root Cause Investigation
A silent compute stall on the RX 7900 XTX that every Windows-side health check reports as normal.
My Radeon RX 7900 XTX, the primary card on Mothership, would occasionally wedge under sustained AI compute. The GPU stays busy, no forward progress happens, and every Windows-side check comes back clean. No TDR events and no WHEA-Logger entries; the card quietly stops computing.
The root cause is a cross-context generation collision that poisons a compute context, the same class as the MES queue-arbitration bugs reported upstream. MES is AMD's on-GPU firmware scheduler. The same pattern appears in community reports across Linux and Windows on RDNA 3 and has not been fixed upstream.
I shipped an orchestrator guard that serializes GPU calls to eliminate the collision, and wedge frequency dropped to effectively zero in daily work. The repro recipe, the probe data, and the mitigation are filed as an upstream bug report on the ROCm tracker. The consolidation fix belongs to whoever owns the MES arbitration code upstream.
The problem the OS couldn't see
The wedge signature is subtle, and it doesn't match anything Event Viewer knows how to catch:
| Power draw | Stabilizes at 220–250 W (healthy saturated state: about 370–390 W) |
| Clocks | Pinned above 2900 MHz; the card reports itself as running fine |
| Memory load | 0–10% (healthy: about 33%) |
| Flip boundary | Approximately 14 TFLOP/s: stable above it, wedge below it during collisions |
| TDR / WHEA events | None. The OS sees a busy card that produces nothing. |
| Correlation | Churn (context cycles since boot), not uptime in hours |
The usual first question ("did TDR fire?") gets you nothing. Windows codes 4101, 141, and 117 are all absent. The hangcheck never triggers because from the OS side there is no hang to detect. The GPU is alive, it's just not computing.
The investigation
I ran a churn harness: load a ComfyUI context, render, kill the process, repeat in a scripted loop. One variable per run, so a positive result actually meant something.
Metrics I collected across runs:
- Renders-to-first-wedge
- Wedge density (wedges per 100 cycles)
- Degradation-floor recovery after kill-and-reload
The instrument that ended the investigation was a wedge discriminator probe. When a wedge appeared, I spawned a fresh Python process and ran a matmul above 14 TFLOP/s against the card. If the fresh process ran fine, the fault was process-local. If the fresh process also crawled, the fault was global at the scheduler level.
The fresh process ran fine: 76.6 and 76.3 TFLOP/s on clean spawns measured immediately after a wedge, both in line with the healthy baseline, while the poisoned original context was still crawling on the same card. Both readings came from the same card at the same instant, so occupancy cannot explain the difference. That is what makes this probe valid where a burst probe taken on a free card is not, a distinction that cost me a wrong verdict once and is now a rule on the practices page. The result rules out silicon damage and puts the fault in the scheduler's queue state, held by the poisoned context, not baked into the hardware.
See the engineering detail: hypothesis tree and discriminator methodology
Five hypotheses, ruled out one per run:
- H1: Cumulative VRAM fragmentation. Ruled out because throughput came back clean after a context kill.
- H2: Kernel age or driver runtime degradation over time. Ruled out because a fresh process spawned at wedge time measured 76.6 TFLOP/s on the same card while the poisoned context was still crawling; a runtime that had aged would have slowed both.
- H3: Thermal or power-delivery threshold. Ruled out because clocks stayed pinned high in the wedge state and thermals stayed nominal.
- H4: Churn on its own (total context count). Ruled out because 125 collision-free cycles complete cleanly, while 2 collision cycles produce 2 wedges in about 90 seconds.
- H5: Cross-context collision during a big-latent render poisons a compute context (MES class). Confirmed.
The discriminator probe was what actually settled it. Without a way to compare a fresh process against the poisoned one at wedge time, "this specific process is broken" and "the GPU scheduler is poisoned" look identical from the OS. The difference between those two decides whether the fix is killing a process or cold-booting the machine.
The root cause
Cross-context generation collision during big-latent renders poisons a compute context at the MES class.
MES is AMD's on-GPU firmware scheduler that arbitrates hardware queues across concurrent contexts. When two heavy generation contexts hit the card at once on RDNA 3, the affected queue degrades and the poisoned context collapses to a small fraction of its healthy throughput; the discriminator probe read 6.4 TFLOP/s against a 76 baseline. The GPU keeps reporting as busy, so the OS never flags anything: the card really is busy, just not making forward progress.
Killing the poisoned context recovers throughput within seconds, which means this is queue-arbitration degradation, not silicon damage. The fix at my layer is to stop creating the collision in the first place.
See the engineering detail: MES firmware, upstream reports, and cross-platform pattern
This isn't an isolated lab problem. The same MES queue-arbitration degradation appears across multiple open ROCm issue-tracker reports spanning Linux and Windows on RDNA3-family silicon:
- Reports documenting "100% busy, 0 compute" under concurrent GPU processes, with workarounds requiring concurrency limits.
- MES ring-deadlock cases where the hangcheck never fires, leading to full machine freezes on Linux.
- "Failed to remove hardware queue from MES" errors triggered by concurrent browser and PyTorch usage on 7900 XTX Linux setups, requiring cold power-off recovery.
The underlying mechanism is queue-count arbitration, and concurrency limiting is the mitigation the community keeps landing on. My repro added a deterministic minimal case (2 contexts trigger 2 wedges in about 90 seconds), the context-held proof from the discriminator probe, and a full Windows-side write-up on RDNA 3. All of that is in the upstream report linked above.
No shipped fix exists in available nightlies or changelogs as of this writing. ROCm-on-Windows nightlies regress regularly, so forward pinning isn't obviously fruitful without re-validation.
The mitigation
The orchestrator guard I shipped is straightforward. It defers image-generation calls during active renders so GPU generation stays serialized to one context at a time. Wedge frequency dropped to effectively zero for daily workloads.
Alongside the guard, I pin the ROCm runtime version against nightly regressions. The churn harness is reusable to re-validate any driver update in the future.
See the engineering detail: ROCm-on-Windows quirks and recovery ladder
ROCm on Windows has a few compounding gotchas that make the wedge harder to catch and clean up:
- No clean self-terminate. ROCm processes on Windows do not sync at exit, so orphan contexts and file handles (including adapter files) persist after kill signals. Zombie GPU state remains even after the process disappears from Task Manager.
- Cold power-off as the worst case. The upstream reports include a global variant where the scheduler is poisoned past context-level recovery and only a cold boot resets it. Every wedge in this lab so far has recovered at the context level.
- Nightly regression cycle. ROCm Windows builds regress often enough that version pinning is required to keep a working setup working. The orchestrator supports an optional runtime pin flag so A/B testing a new nightly stays clean.
Recovery ladder as used in practice:
- Kill the current context and check whether throughput recovers. This has handled every case so far.
- If fresh spawns also measure slow, the fault is global and a cold boot is needed. That branch has not fired in this lab.
- Every event lands in the rescue daemon's log with the outcome and the recovery method, so the record survives.
What's still open
The consolidation fix, whatever it ends up looking like in the MES queue-arbitration code, belongs to whoever owns that upstream. My guard holds the lab together in the meantime, and daily work isn't blocked on it.
Everything is instrumented. The rescue daemon's log records every wedge event, outcome, and recovery-ladder run for provenance. If AMD ships a fix and I want to A/B-test whether it actually helps, the harness is already sitting there ready.