The 3070 that kept dying
A factory boost profile root-caused, a retracted verdict kept
root cause analysis fault isolation GPU reliability engineering observability inference optimization
The short version
An RTX 3070 host kept hard-hanging under sustained captioning. I first blamed the card, then retracted that: the real cause was an unstable factory overclock, and locking the core clock to 1,500 MHz fixed it for 2.6 percent less throughput. The same investigation turned up a nine-times captioning speedup by constraining the model's output.
An RTX 3070 host in my lab, nicknamed BigHonker, began hard-hanging under sustained vision-language captioning. The Windows DPC watchdog fired with code 0x133, which means a driver routine stopped returning; that is a different signature from the display driver timing out a shader, and the display driver's own timeout never fired. Eleven months of clean logs preceded the onset. Nothing in software had changed. The box's workload did: it moved from bursty chat and embedding tasks to continuous compute that holds a high clock for minutes.
The root cause was an unstable factory overclock. This board boosts to 1,965 through 1,995 MHz on sustained load. A reference RTX 3070 has a rated boost of about 1,725 MHz. Above roughly 1,700 MHz this board could not hold a continuous compute workload.
Before I found that, I concluded the card itself was failing. That verdict was wrong, and I retracted it. Evidence available at the time pointed away from hardware failure, and I under-weighted it because "unreliable GPU" is a familiar narrative. I kept the retraction on this page because the reasoning error is the part worth reusing.
I locked the core clock to 1,500 MHz. The card survived an hour-long soak and then a real 487-image captioning job with zero crashes. Inference on this workload is memory-bandwidth-bound, so the lock cost little. Cutting the clock 12 percent cost 2.6 percent of throughput: 33.7 seconds per image at 1,695 MHz against 34.6 at 1,500. On the real job the card ran at 4.75 seconds per image, 138 watts, and 47 to 53 °C, versus 250 watts and 68 °C at factory boost.
A boot-time scheduled task re-applies the clock lock and verifies it by reading back the current frequency. Along the way the same investigation produced a nine-times captioning speedup, described below.
The lab overview covers the fleet. The HAI Orchestrator (the scheduling layer placing work on this card) coordinates inference jobs across my hosts. My skills sheet lists broader competencies. This page focuses on the experiment and its validation.
Evidence I under-weighted at first
Four independent signals pointed away from a dying card. All four were visible before I issued the wrong verdict.
| Signal | What it showed | Interpretation |
|---|---|---|
| Hardware error events | Zero machine-check exceptions, zero PCIe error reports across all runs | The CPU and chipset never saw a silicon fault or link integrity loss |
| Driver timeout detection | TDR never fired | The display driver did not detect a shader hang to reset the GPU context |
| Thermal and power state at failure | 67 °C, 257 of 280 W consumed | The card died cool and inside its power budget. No thermal trip. |
| PCIe replay counter | Zero replays up to the instant of each hang | The bus link remained clean. No transmission corruption led into the failure. |
A failing consumer GPU usually shows at least one artifact. Machine-check exceptions indicate silicon errors. PCIe replay storms signal a dying link or bent slot. A TDR fire means the driver saw the compute queue stop and took action. Thermal throttling or sudden power draw spikes appear in sensor logs.
This card produced none of those artifacts. It simply stopped responding at unpredictable intervals during sustained load, then required a hard reset. The symptom resembled hardware failure, but none of the usual hardware signatures were present.
I know now what I missed: factory boards ship boost profiles well above reference spec, validated on benchmarks that last seconds. A card that passes every ten-second test can still fail under a ninety-minute workload at 1,950 MHz.
Experiment, fix, validation, and a nine-times inference speedup
The core question was whether the failure correlated with sustained clock frequency. I used the same vision-language captioning workload each time, on the same host, with the same model and prompt structure, and the only variable was the locked core clock. A lock pins the GPU at a fixed frequency continuously, whereas factory boost varies dynamically with load, power headroom, and temperature.
| Locked core clock | Result | Details |
|---|---|---|
| 1,500 MHz | Survived all runs | A 3,638-second soak completed 105 images cleanly. Two additional runs survived: 911 seconds (26 images) and 927 seconds (27 images). This was the only clock soaked for a full hour. |
| 1,695 MHz | Survived one window only | Ran 909 seconds completing 27 images. Never tested beyond a single fifteen-minute window, so survival here proves nothing about longer runs. |
| 1,750 MHz | Died quickly | Hang at 577 seconds after 9 images processed. |
| 1,800 MHz | Died very quickly | Hang at 240 seconds. Did not complete the first image. |
| Factory boost (dynamic, 1,965–1,995 MHz peak) | Died every time | Eleven out of eleven runs failed between 131 and 387 seconds. The workload killed the host before image twelve in every case. |
The dose-response is clean: higher sustained clock produces faster failure. The ceiling sits between 1,695 and 1,750 MHz. One thing in the table I did not investigate: the image counts at the two failing locked clocks do not line up with the surviving clocks' rate of about 34 seconds per image. The time to the hang is the measurement that matters here, and I have not chased the throughput difference.
A locked 1,800 MHz failed faster than the dynamic factory boost that peaks at 1,995 MHz. A lock holds frequency continuously. Factory boost varies and spends real time below its peak, trading clock headroom for thermal and power margin. The mechanism of failure is sustained clock, which is why a lower locked clock can be less survivable than a higher dynamic one.
Why 1,500 over 1,695
Time-to-failure grows as the clock drops. Surviving 909 seconds at 1,695 only proves that the time-to-failure exceeds fifteen minutes. It does not prove the card would survive an hour. Only the 1,500 MHz lock received a full hour soak and then proved itself on the actual workload.
The fix and its verification
A locked core clock at 1,500 MHz does not survive a Windows reboot. I set up a scheduled task that fires at boot, applies the clock reduction, then reads back the current frequency to confirm it stuck.
The verifier had a defect I caught during testing. Querying the card's maximum reported clock returns the silicon ceiling of 2,100 MHz regardless of what is actually running, which turned every successful lock into a false FAIL. The fix queries the current active clock instead of the maximum capable clock.
Validation on a real job
On 2026-08-02 the 493-image captioning corpus ran in an isolated folder with no other workload sharing the card. First batch: 246 of 246 images captioned. Second batch: 241 of 247 completed, with six correctly skipped because their audit-log entries already showed completion. That is 487 captions, with zero failures and no captions refused by the model.
Total GPU load across both runs was approximately thirty-nine minutes at 90 to 94 percent utilization. The host ran 175 minutes across the exercise without a reboot. At factory boost, this same workload had killed the box between images three and twelve in every run attempted.
A side finding: the audit trail serves as resume state. Copying a corpus folder with its log files changes what runs next, because the system checks completion status against those entries before scheduling work, and losing the log on a partial copy means duplicate effort.
A speedup found along the way
While testing the captioning pipeline, I noticed the vision-language model emitted a thinking preamble regardless of the "no thinking" flag set in the prompt. The preamble consumed about 90 percent of generated tokens without adding anything to the captions.
I wrapped the output in a one-field grammar schema. The schema makes the preamble tokens unemittable because they do not conform to the declared structure.
| Metric | Free prose (no schema) | Grammar-constrained output |
|---|---|---|
| Time per image | 32.3 seconds | 3.6 seconds |
| Generated tokens | 1,871 | 199 |
| Caption length (words) | About 184 | About 174 |
| Context window used | 8,192 tokens | 4,096 tokens |
Test conditions were five images, identical prompt, the same locked clock of 1,500 MHz, and the same host, with only the schema constraint changed. That is a nine-times speedup on the bench. On the real 493-image run it came out at about thirty-nine minutes of GPU time against a projected 4.4 hours before the change, and the captions are about the same length with the same detail.
A secondary effect: the larger 8,192-token context window had existed solely to accommodate the preamble tokens. With those gone, 4,096 tokens sufficed and kept the model resident in the card's 8 GB of VRAM without spilling to system memory.
I tested input resolution as well. A smaller 512-pixel image proved slower than 768 pixels (4.5 versus 3.6 seconds per image) once generation overhead was eliminated. I did not chase why the smaller input was slower; the prompt token count was 1,367 and identical across arms, so the difference is somewhere in the image path and not in the token count. I kept 768 pixels for finer detail, and prompt encode is now the floor for this workload at that size.
What is still open
I did not separate a silicon stability limit from a power delivery limit. The card was drawing 257 of 280 watts at failure, and a clock lock suppresses both. A power-limit cap at factory clocks, or an undervolt at high clock, would tell them apart, and I have not run either.
The 1,695 MHz clock was never soaked for a full hour. A longer test there might push the usable ceiling higher than 1,500 MHz. The card has run at 1,500 MHz since the fix, so I have not prioritized retesting.
The clock lock is a workaround applied in software. It does not fix the vendor's factory profile. A different board from the same reference design may exhibit the same behavior under sustained compute even if it passes every short-duration benchmark.
The RTX 3070 Ti on another host in my lab is clock-locked the same way and passes hour-long soaks at 1,500 MHz. See the AMD GPU wedge page (the companion investigation) for a different failure mechanism on the lab's other GPU.