Operating Doctrine

The rulebook I wrote for the AI agents that work in my lab, frontier and local alike: twenty-one failure classes, each anchored to a dated incident and enforced in the tooling. Nine of the incidents are on this page.

failure taxonomy post-incident review agent guardrails fail-closed design deterministic gates engineering practices

The lab predates the agents by years, and the largest projects on this site came out of the year I started using AI agents inside it. Frontier coding agents (Claude, Codex) and my own local models, working through the lab's orchestrator (Home AI), produce most of the implementation now. I designed the systems and decide what gets built next, and the agents work under my direction and verification.

Once the agents were doing real work, a small set of mistakes kept coming back. An agent would re-derive something the lab had already solved and documented, introduce a mechanism that duplicated one the lab already had, or assert a conclusion nobody had measured. I caught them because I know the systems well enough to notice when an agent is rebuilding a mechanism the lab already has. Every incident got written up and generalized, and the accumulated write-ups are the doctrine.

The rules have to be written down because an agent's context window is bounded, and whatever durable memory it has is small and governed by its vendor, not by me. What one session learns rarely survives to the next, so the lab keeps the memory the agents cannot. That memory is a retrieval-indexed library of reference documents, runbooks, project plans, and dated journals, with the doctrine as the distilled-lessons layer on top. My test for whether something is documented is simple. If a later session searching the obvious term would not land on the answer, it was never written down.

In industry terms it is a failure taxonomy with a post-incident review process behind it and guardrails in front of it, applied to AI agents working in a production-like environment. The deterministic gates covered on the orchestrator page (test suites, canary deployment, human-only promotion) catch a violation mechanically. How much the doctrine cuts first attempts I have not measured. What I can show is the gate catching a violation, and the dated incident behind each rule.

Case law

The first rows went in during late July 2026, the classes became their own layer on August 5, and the table has kept growing since, with the latest rows from late August. Every addition was paid for by a real incident. An agent did the thing, I caught it, and the row went in. Three of the nine incidents on this page:

Aug 6 · guards. A process-patrol guard was completely blind while 185 unit tests stayed green. A tab escape inside a single-quoted PowerShell string produced zero tabs, so the parser dropped all 267 rows. Both halves were correct on their own, and nothing checked that they agreed. Rule: a guard is not verified until a live decoy proves it.

Aug 15 · method. Two test classes shared a name 850 lines apart, and since Python keeps the last definition, two tests never ran for ten days. "Ran 313 tests … OK" is exactly what a healthy 315-test suite looks like too. Rule: know how many tests the suite should run before you read the result, because a test that was never collected prints the same OK as one that passed.

Aug 8–9 · guards. A rescue daemon's worker thread died while its heartbeat thread kept vouching for it. The tile stayed green and the pushes stayed fresh for about nineteen hours. Rule: liveness derives from the worker's own beat, because "process exists," "push is recent," and "status is Ready" are three separate lies.

Six more incidents

Aug 5 · state. A model snapshot shipped its weights in two naming formats, and a parameter filter was checked against the one the trainer did not load. It matched nothing, and the fine-tune started with zero trainable parameters. Rule: a filter that can match nothing must print its match count and assert non-zero.

Jul 28 · guards. A lane-busy guard raised a generic error, and the order taker archived on any exception. Each was fine alone, but together a stale lane claim made the guard refuse the next order, and five healthy orders were archived in nine seconds. Rule: three typed fates (wait, archive, or hold for a human), and only a property of the work item may consume the work item.

Aug 3 · method. A stable-hash work split gave each host a fixed band of the items. One host had already finished its band. When the leftovers were re-split, the fastest box in the fleet received zero items while holding the top placement score. Rule: rank-based allocation, never absolute bands, for anything that gets re-split.

Aug 15 · method. A test asserted coverage of a "now to now+1h" window against a fixture whose tomorrow was empty. At 23:48 the window crossed midnight, and the deploy gate went red against code that had been green at 22:43. Rule: a test that passes 23 hours a day is measuring the clock.

Jul 26–27 · method. A GPU was declared degraded on a drop from 76 to 6.4 TFLOP/s. The fast probe had run on a free card, and the slow ones ran while training and a renderer held it. The card read 76 again eight minutes later. Rule: a burst probe measures occupancy, not health, and one variable or it isn't a comparison.

Aug 15 · guards. A process guard matched script names by substring, so run.py matched sudo_run.py. On an idle card it killed seventeen of my own shells and five runs of another agent session's authorized work. Rule: a script signature must equal the basename of the first script token.

How it is enforced

The rules bind any agent that does lab work, whether a frontier coding session or a local model running through the orchestrator, and each kind has its own delivery path.

Two layers

The upper layer is twenty-one project-agnostic failure classes in four groups: state and artifacts, guards and supervision, measurement method, and working the lab (build and migration discipline). Under them sits the dated case-law table, laid out as "Before I … | Read first," one row per situation an agent is about to walk into, so the relevant class surfaces before the action instead of after it.

The classes became their own layer on August 5, 2026, after I noticed that the rows were written in the vocabulary of whichever project was active at the time and that the logic would be lost on the next one. The classes bind on every project, and the rows are the evidence underneath them.

The four class groups

Two rules from each, quoted as written.

State & artifacts

One name = one artifact. Timestamp every generation; a bare canonical name on the stalest copy is a trap.
A stated claim proves HELD; its absence proves nothing.

Guards & supervision

A dead reporter is not a dead worker — and 'could not tell' must never render as ANY definite state. A guard that cannot measure says so, loudly.
A guard is not verified until a LIVE decoy proves it.

Method

One variable or it isn't a comparison. Same-occupancy, same-instrument, same-config — and resolve a named confound by SEPARATING it before shipping any fix built on it.
Verify to the working END state. Compile ≠ correct, deployed ≠ running (does any process holding the file predate it?), shipped ≠ on disk, mirror ≠ synced unless just hashed, 'restarted' ≠ new PID.

Working the lab

Building is the LAST step. Search the library, read the component's docstrings, ask — and if you can't QUOTE the requirement, you invented it.
Docs win over memories; searchable TITLES win over buried asides; a superseded note gets cross-referenced in the same pass.

Two organizing ideas

The factory model

Every unit of lab work is a product order with four stages: the order (the spec), fulfillment (the factory runs its standard line), shipping (the declared deliverables), and cleanup (the line is cleared for the next one). Product versus byproduct is stated explicitly, because a training run ships weights and its renders are byproducts, while a benchmark ships a number and its weights are byproducts. That distinction went into the rules after eight gigabytes of staged adapters sat in a shared directory for nine days with no order saying they were disposable.

The rule I gave the agents is "A customer never walks the factory floor." They submit orders and read what the observation surfaces publish, and reading is always allowed. Killing a runner, placing files by hand, or editing a queue directly is the violation.

Tools inform; they do not block

A guard reports what it found and leaves the decision to the operator. That rule was paid for on August 6, 2026, when a new process guard's first live patrol killed a monitor that merely mentioned the model server in its script path. The fix split the guard into two tiers: a broad signal reports, and only a narrow, high-confidence signal may act.

Supervision follows the same idea, which I call "status, not time based." On August 7 a status tile's freshness budget was a flat five-minute constant, so a daemon that reported every minute could be dead for five minutes while the tile still said watching, with no state in between. Now the reporter declares its own cadence, the verdict derives from that, and "I cannot tell" is its own state.

What it says about the work

The doctrine is how I hold the agents to a standard. I own the architecture, the operations, and the verification, and nothing an agent produces ships until I have checked it. Its rules favor the lab's existing mechanism over a new one, because a parallel mechanism is a second thing that can drift. They also demand a measurement before an explanation. Every time an agent's story drifted toward "the hardware is unreliable," the fault turned out to be the agent's.

The rules I restate most often are the plain ones: finish what you flag in the same pass, and say so out loud if you defer it. Check whether the hard path is actually closed before recommending the soft one. Verify instead of assuming it worked.