32B QLoRA on 24 GB AMD: Native ROCm, Windows
Flux.2 [dev] trained as a quantized LoRA adapter on a single Radeon RX 7900 XTX under Windows ROCm, a hardware, OS, and framework combination with no published QLoRA recipe I could find at this model size.
The short version
This page documents training infrastructure: how Flux.2 [dev] (32B), a diffusion model from Black Forest Labs, runs as a QLoRA workload on a single consumer GPU that should not hold it. The stack is AMD RDNA3 (gfx1100) with native ROCm on Windows. That combination lacks the CUDA tooling, flash-attention kernels, and FP8 hardware most published QLoRA recipes assume. It runs on Mothership, the lab's primary GPU host. The stack is what I own: the 7900 XTX is the lab's largest card and the box runs Windows, so the recipe had to work there or not at all. The recipe came together in July and August 2026. None of the nine hurdles is vendor-specific in kind: they are constrained-memory staging, quantization plumbing, and allocator behavior, and they transfer to any card smaller than the model it is asked to train.
| Model | Flux.2 [dev] (32B) |
| GPU | AMD Radeon RX 7900 XTX, 24 GB VRAM, gfx1100 |
| OS / Runtime | Windows, native ROCm (PyTorch 2.12+rocm7.15) |
| Quantization | uint4 weight-only via optimum-quanto (no torchao, no FP8/FP4 hardware) |
| Attention | Triton path (no flash-attn ROCm kernels on this platform) |
| Largest published 24 GB recipe I know of | Flux.1 [dev], 12B (Flux.2 is about 2.7 times larger) |
| Measured throughput | 6.9 s/it at 448 px, batch size 1, on a healthy card |
Why the stack is unusual
The NVIDIA cards in my lab run CUDA on Windows without any of this ceremony. ROCm on Windows is still nightly-build territory, and the published QLoRA workflows assume Linux on NVIDIA silicon besides: flash-attention kernels, FP8 tensor cores, and tooling built around both. The RX 7900 XTX has none of those accelerators. Quantization falls back to uint4 weight-only through optimum-quanto, wrapped by an AMD ROCm fork of ai-toolkit (cupertinomiranda/ai-toolkit-amd-rocm-support). Attention runs on the Triton path.
The base Flux.2 model file is 64.4 GB in bf16, nearly three times the card's VRAM. Every load, quantize, and device-move operation must pass through a narrow sequence of memory optimizations. The setup requires specific patches to an AMD ROCm fork of ai-toolkit; it is not a vanilla installation, and reproducing it on a different card, driver version, or OS combination is non-trivial.
Run shape depends on how much the adapter has to learn: rank 16 and one to two thousand steps at the light end, rank 32 and three to four thousand at the heavy end. Sampling is disabled during training, so each checkpoint is judged afterward against the base model in ComfyUI, with the side-by-side method described on the image page. The adapters this pipeline produces are the ones the lab's image tool serves today. The adapter itself is a few hundred megabytes at rank 16.
Proof: an astronaut the base model did not know
The private adapters stay private, so the public proof is Peggy Whitson, trained from NASA's public-domain photographs. This rests on a different basis from the friend-group adapters: she is a public figure, the photographs are US government works in the public domain, and her record in the registry is marked as such, not as a consent grant. The adapter exists as a non-commercial demonstration that the recipe learns an identity the base model did not know, every render of her passes the same gate, and none is published here. I assembled the set and ran the training on July 23 and 24, 2026, before the boot-time work below and the runs that followed, so this is the recipe as it stood then. Before training, the base model rendered a stranger when asked for her by name; a control run showed it knew none of the three astronauts I tried. The dataset is 34 images across three eras of her career, each traceable to its NASA image id in a provenance manifest, so the whole run can be reproduced from public sources. The recipe is the exact one the private runs use: rank 16, 2,000 steps, the same batch size and resolution. After training, the adapter renders her recognizably in settings the photographs never contained: a garden, a business portrait, a full-length shot. The run also settled a question about run length. At the halfway checkpoint the identity had already locked in, but prompts outside the training distribution fell apart, and the second thousand steps bought the generalization. The run crashed once, at the resume after the first thousand steps, from the same context-collision class the wedge page describes; the fix was a quiet-card handshake in the factory's own resume path.
Nine hurdles, worked through
- Manual non-mmap loader for the 64.4 GB base weights
- CPU-side quantization of the transformer
- CPU-side quantization of the text encoder
- State-dict memory release between loads
- Cached uint4 base weights
- Block-swap (layer offloading) disabled
- Sampling disabled during training
- VRAM collision avoidance during text-encoder caching
- Expandable memory segments
How each one was worked through
| 1. Manual non-mmap loader for 64.4 GB base weights | |
safetensors mmap load_file crashes natively on the 64 GB file without a traceback, while files under approximately 33 GB load fine. A standalone script parses the 8-byte header length and JSON header, then per-tensor seeks to base+offset, reads nbytes, and rebuilds via torch.frombuffer(bytearray, dtype).reshape(shape). Bypasses the whole-file mmap crash. |
| 2. CPU-side quantization of the transformer | |
| Moving the full 64 GB bf16 transformer to the GPU before quantizing fails on the first large allocation the move attempts, roughly 38 GB, which cannot fit on a 24 GB card. Quantize the transformer on CPU first, then move only the resulting approximately 19 GB uint4 tensor (the weights plus their quantization scales and overhead) to the GPU. |
| 3. CPU-side quantization of the text encoder | |
The Mistral-Small-3.1-24B-Instruct text encoder is 48 GB in bf16. Same pattern: quantize on CPU to approximately 12 GB uint4, then move to GPU. Note that the fork's load_te reads its qtype from model_config.qtype instead of a separate qtype_te parameter, a call-site detail worth knowing when hooking custom quant paths in. |
| 4. State-dict memory release between loads | |
The bf16 transformer state dict pins 64 GB of host memory. Loading the Mistral text encoder (48 GB) on top of that, and of everything else the process already holds, pushes committed memory past the approximately 115 GB of RAM plus pagefile and triggers an access-violation crash (0xC0000005). del transformer_state_dict; gc.collect() immediately after load_state_dict(assign=True) reclaims that memory before the second load. |
| 5. Cached uint4 base weights | |
The initial manual load, quantize, and freeze sequence is expensive at roughly 8 minutes. A one-time preparation script produces a cached state-dict of quanto QTensors (the library's quantized tensor type), approximately 19 GB. Every training run loads this artifact via torch.load in seconds. |
| 6. Block-swap (layer offloading) disabled | |
layer_offloading: true deadlocks the HIP driver at pre-train sample and first step, requiring process termination. Spilling also incurs approximately 204 seconds per step for 2.27 GB of data over PCIe. Set layer_offloading: false. Do not attempt CPU offload to relieve VRAM pressure near the boundary. |
| 7. Sampling disabled during training | |
The same offload machinery that deadlocks block-swap also causes sampling deadlocks. uint4 previews at training time produce black frames or rainbow-band noise on certain seeds. Set disable_sampling: true. Evaluate finished LoRA outputs in ComfyUI on the same card after training completes. |
| 8. VRAM collision avoidance during text-encoder caching | |
During TE caching the 12 GB Mistral adapter is resident alongside the 19 GB base; both on GPU overflows 24 GB. The boot stack gates pipe.transformer.to(device_torch) on not low_vram, parking the base model on CPU during TE caching, then moves it back with a small resident-move block after TE unloads. |
| 9. Expandable memory segments | |
Moving uint4 tensors to the GPU fragments the HIP allocator and OOMs without expandable segments. Launch with PYTORCH_HIP_ALLOC_CONF=expandable_segments:True. This single env var contributed to reducing step time from approximately 40 seconds to approximately 11.6 seconds at 512 px resolution, before the later gains measured at 448 px. |
Boot time: 17 minutes to 49 seconds
Once training worked, every run still spent 17 minutes 7 seconds booting before the first step. I put timing waypoints through the startup path and the breakdown was plain. Dataloader initialization took 6 minutes 44 seconds, unloading the text encoder 2 minutes 33 seconds, moving the transformer to CPU 1 minute 33 seconds, and moving the text encoder to the GPU 1 minute 26 seconds. The remaining five minutes were spread across smaller steps. Four rounds of fixes followed, each measured against the previous boot.
The first two rounds skipped redundant device moves and stopped re-caching latents that were already on disk, which brought the boot to 9 minutes 30 seconds. The next round added a resume flag, cached the blank and trigger embeddings to disk, and skipped the text-encoder move on resume: 6 minutes 15 seconds. The decisive change was a reframe of mine. The quantized text encoder's cache always exists on this box, so its presence could gate the entire text-encoder load, with a stub standing in for the encoder during pipeline construction. That took the boot phase to 49 seconds, down from 17 minutes 7 seconds, and the total wall time of a resumed run (boot plus the fixed overhead after it) from 13 minutes to 4. One diagnostic round also found that the base model was already resident on the GPU by the time a later move ran, which turned that move into a no-op; fixing the "mystery" would have cost the move back, so it was left as it was and written down.
Performance and verification
The measured throughput on a healthy card is approximately 6.9 seconds per iteration at 448 px with batch size 1, down from roughly 40 seconds per step before the expandable-segments and resident-move fixes. The same job on a card in the stalled state described on the wedge page runs at about 67 seconds per iteration. VRAM residency holds steady at approximately 20 GB on cuda:0, verified through in-process torch.cuda.mem_get_info calls and Windows GPU Performance Counters. Host RAM drops by about 19 GB when the model migrates from CPU to GPU, a cheap sanity check that residency actually occurred. The host system runs 32 GB of physical RAM plus a pagefile that brings total virtual memory to roughly 115 GB, required because certain one-time bf16 loads cannot avoid holding both the transformer and text encoder in host memory briefly.
How to prove residency on ROCm/Windows
Cross-process torch.cuda.mem_get_info() probes report incorrect values on ROCm/Windows. They can return used=0.2 GB while the training process genuinely holds 20 GB of VRAM. One busy CPU core during training is normal; it drives the GPU workstream and does not indicate a fallback to CPU computation.
Three reliable signals:
- In-process
torch.cuda.mem_get_info: call this from within the training loop, not from a watchdog in another process. - Windows Performance Counters:
\GPU Process Memory(<pid>)\Dedicated Usagefor VRAM residency,\Shared Usagefor host memory spill,\GPU Engine(*engtype_compute)\Utilizationfor compute utilization. - Host RAM delta: an approximately 19 GB drop confirms model migration from CPU to GPU.
Companion: the GPU wedge investigation
This project's render and checkpoint loads on the same RX 7900 XTX also surfaced an intermittent driver-level compute stall in the on-GPU firmware scheduler. That investigation has its own page. The stall and its resolution are the reason the orchestrator's guard now serializes GPU generation to one context at a time.
Multi-factory training context
This training workload runs as a product order, the lab's unit of work, through the Mothership factory instance (the order-taking runner on that host), the same code that serves BigHonker, Softserve, and the Fellowship (the ARM cluster). One environment variable pointing at the install root makes the same file serve every host, and the scheduler places orders across all of them.
Every factory implements the same order contract; the orchestrator page has the mechanics.
The consent gate on trained adapters
The image tool in my lab exists for my friend group. A friend can ask Home AI for a picture of themselves, or of other friends who have said yes, and we get to have fun with the results. The gate is what makes that safe to offer. The training pipeline registers each finished adapter against a per-subject record, and that record, not the tool, decides whether a request may use it. Nothing renders for anyone who has not agreed, and an agent calling the render tool passes through the same gate I do.
How the gate works
The check runs per request and fails closed. Consent scope is a small hierarchy, from personal use outward, with an allowlist for named third parties at each level, and every grant, revocation, and change of scope appends to an audit trail on the record itself. Widening a scope binds on the next check, with no restart. Registration from training is idempotent: a re-run of the same training updates the record instead of duplicating it.
What is still open
Adapters are trained on demand as new runs are ordered through the factory, and they feed the image and identity work. They also fed the cross-scale transfer experiment, which has since concluded. Three things stay open. Sampling during training remains disabled, so previews come only after a run finishes. The recipe is pinned to one ROCm nightly, and re-validating a newer build is a manual job with the churn harness (the load, render, kill, repeat loop from the wedge investigation). Reproduction on a different card, driver version, or OS has not been attempted. The working recipe, config, and patches are public at github.com/drhawktopus/flux2-32b-qlora-rocm-windows. Questions or interest in reproduction details: nickcrowley97@gmail.com.