Self-engineering
The AI that proposes changes to its own orchestrator, under human gates.
AI-agent solutions CI/CD and test gates Governance and auditability Regression evaluation Reliability engineering
This campaign built a supervised loop: my lab's local AI plans, edits, tests, and proposes changes to its own orchestrator code, and a human promotes every change to the live system. Deterministic gates prevent self-deployment, and the gate that guards promotion now passes 2,424 tests, with two skipped. It concluded on 2026-08-31 with a five-pin regression run, each pin testing one shipped mechanism, graded outside the system that produced the answers. It was not a clean sweep. Two pins passed outright, two I judged from evidence the run could not grade on its own, and one failed until I re-pointed a source that the recent library restructure had moved.
It started from an incident. In August 2026 the local AI went three days without a frontier assistant available, and the gap exposed four failures: it had no awareness of pending handoff work, it gave confident answers without evidence, it invented search results, and it accepted scope limits passively. I turned those four failures into mechanisms.
This campaign built the machinery. A later one, the Native Intelligence campaign, trains the collaboration judgment that runs on top of it.
The four failures that started it
The model could not retrieve pending work from the lab library. It relied on training memory when a dated journal entry contained the current state.
It stated facts it had no evidence for. There was no critique step to ground claims in retrieved sources.
When a tool returned no results, the model invented content and presented it as retrieved data.
The AI accepted limits it should have proposed to extend. It did not draft work for human review.
What shipped
The first rows answer the four failures directly; the rest is the machinery the loop needed once it began proposing its own changes. All of it wires into the Home AI orchestrator, which manages the lab's compute fleet and order system.
| Mechanism | What it does |
|---|---|
| Handoff awareness | A catch-up tool, named day-journal retrieval, and current-project routing. A handoff question reaches the library instead of model recollection. |
| Evidence-grounded review | Critique that states which model tier, host, and context answered. Advisory versus blocking findings. An author-dispute step. Critical unresolved findings refuse while preserving the draft; lower severities ship with a warning. |
| Bounded self-extension | The AI drafts and proposes work but is mechanically blocked from feeding its own future queue. Arming belongs to the human supervisor. |
| Work intake | Cluster map work, GPU training, and general whitelisted compute all enter through the existing order desks instead of being launched directly. |
| Supervision surface | Fleet status, watches, remedies, progress reporting, live-run roll-ups, and a numbered project board visible from the chat and dashboard. |
| Commission driver | A large coding objective decomposes into small units sized to fit an agent's working context, each carrying the exact code it needs, run in dependency order with checkpoints and an independent review between units. I rule on what the review rejects, and promotion to the live system remains a human act. |
| Evaluation as an order | Five pinned regressions compose into one durable campaign. The orchestrator decomposes and executes, the scheduler places each step, the factories own timing and model residency, and the client only reads and grades. |
| Deploy discipline | Every change goes canary first. The full test gate must pass. A human promotes. A failed canary rolls back automatically. Test gate: 2,424 tests passing, 2 skipped. |
The agents operate under the Operating Doctrine (rulebook), which binds behavior on evidence, ownership, and verification.
How a change reaches the live system
The deploy cycle is deterministic, and the one gate that matters is human. The self-engineering map draws the whole loop on a single page.
- Proposal. The AI drafts edits in an isolated workspace, tests run against the changes, and an independent review pass runs before the work is checkpointed.
- Canary deployment. The change deploys to a canary target first, where the full test gate and live health checks run.
- Human promotion. If the canary passes, I promote it, and only then does the change cut over to the live orchestrator.
- Auto-rollback. If the canary fails at any point it rolls back on its own, and nothing reaches the live system.
The promotion step is the only one I perform by hand, and there is no path that skips it. The transferable part is that discipline: an agent proposes, a tested gate and a human decide what ships, which is what any team bringing AI into its codebase has to get right.
What the campaign taught
Four findings that shaped later design work.
- A worker only knows what its packet carries. Every contract that crosses out of the coding workspace has to travel verbatim, including file keys and derived state such as whether a component is live.
- A green suite proves consistency with fixtures, not reality. A live-runs feature was internally green while its test data described an idle factory in the wrong shape. A real positive/negative live probe caught it.
- Execution and grading must stay separate. The orchestrator may record model, host, context, tool trail, and timing, but never grades its own answer.
- Initiative belongs on the investigation side of the gate. The agents search, verify, and draft proposals without waiting to be asked. They cannot arm work, deploy, decide what belongs to a project, or invent architecture.
Three things I expected to hold did not. An earlier evaluation harness that ran from a session still bypassed the order system by picking hosts itself, which is why the evaluation was moved into an ordinary order and the harness retired. The model's self-critique turned out not to be an independent grade, because the accidental drill's worst answer had been internally marked pass. And the model was often not the limiting component: several apparent failures were missing tools, wrong tier, absent evidence, stale documents, or invented packet contracts.
Closing proof: the five-pin evaluation
The campaign closed with a five-pin regression evaluation, each pin testing one shipped mechanism. All five pins executed and produced gradeable evidence, and grading ran outside the orchestrator that produced the answers, so no pin could pass on its own say-so.
Two pins passed outright. Two came back flagged for human judgment, which the harness does when it cannot grade an answer mechanically. I judged both to be passes. One had used the catch-up tool and the journals to answer a handoff question correctly. The other had produced a grounded draft naming real files and had not armed its own work. One pin failed, because the historical source document it named had been archived by the library restructure (Project 30), which had wrapped up days earlier.
The corpus moved, so the pin was re-pointed at a persisted source identified by content hash. The re-run then recovered the source's specific findings, confirming the model had read the moved document, not answered from memory.
The loop runs unattended only inside a bounded overnight window, and it has run there, but not for long enough to say the window can be widened. The pins read from a living document corpus, so a restructure can break one; pinning by content hash fixed the instance, not the class. And every gate on this page has had one person behind it, which says nothing about how the separation holds with a second reviewer.
The machinery is inherited by the Native Intelligence campaign, which trains earned collaboration judgment into local models running on lab hardware. The orchestrator it runs on, the canary promotion cycle, and the fleet it coordinates are documented on the Home AI Orchestrator page, and the rules these agents follow live in the Operating Doctrine.