Image generation for the assistant
Nine rounds to find a pipeline for identity-preserving portraits, and what a face-similarity score cannot decide.
image generation evaluation methodology measurement discipline consent gating pipeline design
The short version
This is the image-generation project for my local assistant: nine evaluation rounds to find a pipeline for identity-preserving portraits of consenting people, and the measurement discipline that came out of it. It demonstrates a controlled multi-round evaluation, a regression caught after a silently dropped metric hid it for several rounds, and the rule that a score is an alarm while the eye is the verdict. The through-line is evaluation methodology and measurement honesty: a reusable harness, a refusal to trust a misapplied metric, and a sensitive capability gated behind consent that fails closed.
Nine rounds
Over the first half of July 2026, nine evaluation rounds compared pipeline variants for identity-preserving generation from reference photos, from base generation through face-swap to Kontext reference-conditioning. I defined the arms; a harness ran them. Each round scored every arm two ways: a face-embedding similarity of the rendered subject against reference tiles, and my own side-by-side ranking. Every round produced a complete decision map of its arms, parameters, artifacts, and scores, and a round was not complete until every arm had run and the map was full.
The maps are archived per round, so any round can be reviewed at a glance or compared across rounds. They are not published here, because every cell of a map contains a subject's photograph.
How a round worked
The harness carried the scoring functions inside it. For each arm it generated the outputs, ran every registered metric, and fed the results into the map generator, which was parametrized per round so its structure matched that round's arms and decision steps.
Some rounds were two-stage: a fast preview stage for triage, then a full-quality stage for the final samples, both run for every arm when a round used them. Placement was per-request policy, with the 24 GB card for training and heavy renders and the smaller cards for lighter work.
The metric that stopped, and the fix
A subject-fidelity score existed in round four: a face-embedding similarity of the rendered subject against that subject's reference tiles. It lived outside the harness and depended on my remembering to run it, so it was not re-run in the next three rounds. During that gap, one arm took a scale change and a prompt change that shrank the subject in frame, and the regression rode invisibly until I rebuilt the metric in round eight. The score for that arm came back at 0.66, against the 0.91 it had held in round four.
The number was the alarm, not the verdict. A subject that shrinks in frame pulls a face-embedding score down on its own, so the score pointed at the arm and the eye then decided what had actually changed.
The fix was structural. Every metric the project ever used is now wired into the round harness and re-runs every round, and removing one takes a recorded change to the harness, so it cannot happen by forgetting. The rule I set down at the time: any instrument that can silently stop running will eventually report a clean result while the thing it watches degrades. A measurement that ran and found nothing wrong and one that never ran return the same result.
What the score can and cannot say
The similarity score is an ArcFace-style metric: it measures how close two face representations sit in embedding space. It was trained to verify identity on photographs, and I was pointing it at generated images at varying crops and poses, which is a different distribution and a different question. I never calibrated it against my own judgments, so I used it as a consistency check across rounds, never as a verdict. The eye made the final call, and the score existed to keep that comparison consistent.
That discipline, refusing to let a misapplied automated score stand as the verdict, carried into how I graded the language-model tournaments later. There the verdict could often be deterministic, the checks fixed before any model ran; the Gauntlet describes that bar.
The consent gate
Any adapter or pipeline that involves a real person runs behind a consent-gated registry. The check runs per request and fails closed, and an unrecognized subject and a subject who denied consent return the identical denial, so nothing leaks through the difference between them. The tool sits inside the orchestrator alongside the text tools, and the QLoRA page has the full detail on the gate's scope model.
What came after
The rounds concluded with a method and a stated ceiling. The method was a Kontext reference-conditioned edit pipeline. The ceiling was that reference-conditioning at inference time is weak for likeness, and I had no trustworthy automated grader for resemblance, only a score that misleads on this task. That ceiling is why the next project trained on the subject instead of inferring likeness from reference images at inference time.
The QLoRA 32B project trained a 32B image model's adapter on the 24 GB card, and those adapters live in the same consent-gated registry. A later question, whether an adapter could transfer across scales onto a 4B sibling for fast previews, reached a negative result; the cross-scale transfer experiment documents it.