The Gauntlet: over 1,000 model-by-device runs across my ARM cluster
Systematic evaluation of 13 small LLMs and VLMs across four ARM-device hardware classes, producing verified worker tiers for local model routing. The follow-on ensemble experiment, still open, is summarized at the end.
The question was straightforward: which small language and vision models actually run on constrained ARM devices, at what speed, and with what quality on real task suites. The answer required a full matrix, not selective testing. Earlier evaluations had skipped cells on the assumption that correctness is deterministic and hardware-independent. That reasoning holds for the textual output of a loaded model, but discards whether the model loads at all on a given node and how long inference takes. Both matter for routing.
The Gauntlet ran on the Fellowship (the ARM cluster) and closed out on August 31, 2026, the same day the self-engineering campaign closed. It ran in two phases and more than 1,000 model-on-device runs in all: a scored per-model matrix, and the follow-on ensemble experiment summarized at the end. It tested 13 models across four distinct ARM-device RAM pools, each carrying four task divisions: general chat, coding, tool and agent calls, and vision-language understanding. Behavior criteria (refusal quality, not-found recovery, blind-repeat resistance) were not rerun in the matrix; the behavior scores below carry over from the per-model evaluation run on the same cluster in late August 2026, a few days before the matrix; its scorecards ship with the results package. The matrix contained 688 cells. 512 ran to completion and were scored, and 436 of those passed the quality bar. The bar is per task and deterministic: a coding cell passes if the code runs correctly, a tool cell if it names the right tool with the required arguments, a chat or vision cell if the answer matches the expected one. The expected answers, the code checks, and the required tool arguments were fixed in the order payload before any model ran, so the bar could not move after a result came in. The rest are cells where the model could not fit the node, plus a smaller set of runs that failed, timed out, or left no record. Every cell's outcome is in the published data.
The verdict identifies three tiny-worker tiers plus one broadly deployable vision model for the Fellowship. Qwen2.5-1.5B serves as the reliable general and tool worker across all four hardware classes, scoring 8/8 on chat and composite tool tasks on every lane, and carrying a 6/6 behavior score forward from the earlier per-model evaluation, which the matrix did not rerun. Qwen2.5-Coder-1.5B is the coding champion, 5/5 on the matrix's implementation set and 7/8 on the earlier evaluation's broader eight-task suite. The 0.5B variants operate as fast specialists inside narrow envelopes when latency matters more than coverage. SmolVLM2-500M is the only VLM that completes every lane with consistent quality.
The verdict
General & tool worker
Qwen2.5-1.5BChat 8/8 on all four lanes. Composite tool 8/8 on every lane. Behavior criteria 6/6 on the earlier per-model evaluation, run on the same cluster in late August, a few days before the matrix, which did not rerun behavior. That evaluation's task counts differ by division (six behavior criteria, eight coding tasks, ten chat tasks), so its fractions are not one yardstick. Chat median wall time (the median across the lane's chat cells, each run once, so a ranking and not a variance measurement) is 10.4 s on the largest RAM pool, 11.7 s and 15.9 s on the smaller pools, and 10.8 s on the solo node; tool tasks run roughly twice as long per lane. This is the default route for chat and tool-calling tasks across the cluster.
Coding worker
Qwen2.5-Coder-1.5BCoding suite 7/8 on the earlier evaluation's broader eight-task set, 6/6 behavior there, and 5/5 coding plus 8/8 composite tool on every lane in the matrix. Routes here for implementation tasks where correctness matters more than raw speed.
Fast specialists
Qwen2.5-0.5B & Coder-0.5BThe general 0.5B completes the reduced short-answer chat set at 8/8 with 6.1 to 7.5 second medians, but failed both no-tool decline tasks in the earlier evaluation's tool set. Coder-0.5B hits 5/5 on the matrix's implementation-only code subset at similar speeds, though it scores only 4/8 on the full eight-task coding suite. These are cheap specialists for bounded envelopes, not general-purpose replacements.
Vision-language
SmolVLM2-500MThe only VLM that runs across all four hardware pools. Scores 5/7 on the synthetic task set on every lane with medians of 25.3 s (A), 28.6 s (B), 45.5 s (C), and 28.0 s on the solo node. Qwen2-VL-2B achieves higher quality at 6/7 synthetic and 5/5 real photographs, but only on class-A nodes at 83.3 s median with hard RAM exclusions on the smaller pools. SmolVLM2-2.2B earns no worker role despite matching the 500M's 5/7 score, because its 155.9 s median and RAM footprint do not justify the cost.
How it was tested
Thirteen models were evaluated: MiniCPM5-1B, Qwen2.5-0.5B, Qwen2.5-1.5B, Qwen2.5-Coder-0.5B, Qwen2.5-Coder-1.5B, Qwen2.5-Coder-3B, Qwen3.5-0.8B, Qwen3.5-2B, Qwen3.5-4B, gemma-4-E2B, and three vision models (Qwen2-VL-2B, SmolVLM2-500M, SmolVLM2-2.2B).
Four hardware pool lanes carried the work: A-pool (two nodes with 2.79 GB each), B-pool (nodes with 1.76 GB), C-pool (nodes with 1.82 GB and four cores). One node, Sam, ran as its own lane because an earlier benchmark had put it behind its hardware-class peers. In this matrix it did not lag: its chat median of 10.8 s sat with the fastest pool. Four divisions structured the task cells: chat, coding, tool and agent calls, and vision.
The cluster's head node, Tolkien, took the tournament as one order (the lab's unit of work: a spec the scheduler fulfills and reports on) of 604 text cells plus three smaller orders for the 84 vision cells, 688 in all. The same order validator that serves the GPU hosts processed them; the fan-out format is payload-agnostic. Results, harness code, and execution logs landed on a dedicated drive on the head node, so tournament state persists across sessions.
See how same-hardware pooling avoids duplicate work
Two identical A-pool nodes do not each run the full cell set. The order-taker splits items across available workers in a pool, so if 400 cells route to that pool, roughly 200 land on each node. Correctness is captured once per cell and does not vary by device; speed and fit are measured per lane, not per individual node.
Models that did not earn worker roles
Qwen3.5-2B keeps its class-A chat-champion status from the earlier per-model evaluation, where it scored 8/10 on the broader ten-task chat set. It routes only for quality-sensitive chat where higher latency and weaker not-found recovery are acceptable costs, not as a general default.
MiniCPM5-1B scored 28/32 on chat across the lanes, and the four misses were the same plural-form question on every lane. On tools it scored 12/32 because it routinely omits required arguments, so it does not route for tool work. Qwen3.5-0.8B was slower and less accurate than the 0.5B and 1.5B defaults on every completed lane and earns no route. Qwen2.5-Coder-3B, Qwen3.5-4B, and gemma-4-E2B all show insufficient coverage at this hardware tier, recording timeouts, incomplete task completion, or storage and RAM pressure. These models are recorded as poor default workers for the cluster.
Engineering detail: controls, constraints, and audit boundaries
Positive control on every order. Each execution batch carries a known-answer control cell. If the expected substring does not appear in the output, the entire run is marked VOID, not trusted. This follows the live-decoy rule from the practices page: prove the instrument fires on a known positive before trusting a negative result. A failed control invalidates the batch because it indicates something broke in the pipeline, not in any individual model.
Grammar-constrained JSON for tool tasks. The ARM-device builds of llama.cpp had a broken JSON-schema flag stub during this period. The tournament used raw GBNF grammars (the grammar format llama.cpp understands) via --grammar-file instead, constraining the model's output to valid tool-call structures at inference time. This made tool cells reliable across all thirteen models without requiring per-model prompt engineering.
RAM-based fit discovery. For borderline model-node combinations, whether a model fits on a node was discovered by attempting the load and recording the out-of-memory result when it failed. Nothing was silently skipped as "won't fit."
One run per cell. Each cell executed once, with a fixed seed. The matrix measures fit and wall time, not run-to-run variance, and its pass counts are single samples on the lab's own task sets: good for ranking, not for statistical separation. The identical-cohort arms of the ensemble experiment below are where variance is measured.
Every cell has an outcome. A cell that produced no scored row is classified in the published data with its reason: the model could not fit the node, the run failed or timed out, or the run left no record. Those classifications were preserved as recorded, not reconstructed afterward.
Data locality. All results, harnesses, and payload definitions live on the head node's local storage.
The next phase: the ensemble experiment
The tournament settled which models are reliable workers per hardware class. Phase 5 of the same program asks the question the cluster was built for: with the model held constant, what is the smallest team shape that reaches the quality ceiling for a given task type? It is a 1,172-arm experiment, and every arm varies exactly one thing, the shape of the team.
See the experiment design and current status
Three arm families cover the space. An N-ladder runs three ensemble strategies at every cohort size from one to ten workers. They are independent swarm (all N answer, results aggregated afterward), best-of-N (one answer selected by a scoring criterion), and majority vote at odd cohort sizes only, so ties cannot pollute the vote. Team layouts test structure on the coding and tool divisions, where verification is deterministic. The layouts: one solver alone against solver-plus-critic; one solver with one, three, or five critics; chain depth from solver through critic, refiner, and verifier; and one solver plus N parallel verifiers that run tests, parsers, and static analysis instead of a model. Heterogeneous specialists lock each device into one role for a run. Hardware-natural assignments come first (large-RAM nodes on heavy reasoning and vision, mid-capacity nodes as the writer pool, four-core nodes as fast simple workers), then surprise-combo arms swap exactly one role assignment, so the data decides instead of the hardware assumption.
Models per role are the tournament champions: Qwen2.5-1.5B for general and tool work, Coder-1.5B for coding, SmolVLM2-500M for vision. Deterministic checks are preferred over model judgment wherever a task allows it, because code checks are cheaper and repeat exactly. Deliberate identical-cohort arms (two N=3 runs in the same window) measure run-to-run variance without introducing a second variable, and the positive-control rule above rides every order. With ten arm64 nodes, cohorts summing to ten or fewer run in parallel windows on non-overlapping node subsets. That is what makes the full one-to-ten ladder practical; the earlier seven-node roster had forced a sequential three, five, seven ladder.
| Complete | 1,141 of the 1,165 runnable arms (7 of the 1,172 designed cannot run on the smaller nodes), data collected per the standard order contract |
| Remaining | 24 arms: 23 awaiting capacity on the final vision slice, 1 awaiting a placement decision |
Execution stands at 1,141 of 1,165 runnable arms. The remaining 24, and the appraisal, are paused behind other work for the moment, so there are no results here. The tournament above stands on its own: its tiers already set the model per task on the cluster, whatever the appraisal finds. What the aggregate will produce is a per-task-type table of the smallest team shape that reaches the ceiling. That becomes the input to routing: a single 1.5B worker, a three-worker vote, a solver-plus-critic pair, or a specialist team, depending on the task.
What this feeds into
The verdict sets the model per task division on the cluster, and the ensemble experiment's arms run on exactly these champions: chat and tool work on Qwen2.5-1.5B, coding on Coder-1.5B, vision on SmolVLM2-500M, with Qwen2-VL-2B where a class-A node is free and accuracy matters more than latency. The 0.5B variants handle time-sensitive narrow tasks when throughput matters more than coverage. Models that failed to earn roles are excluded from default routing, though nothing prevents a manual override for specialized cases.
This tournament extends the earlier per-model evaluation instead of replacing it. That evaluation established quality baselines on per-model terms; the Gauntlet established which models are reliable workers across specific hardware classes with measured speed and fit data. Both results inform the routing layer, with the earlier evaluation providing quality ground truth and the Gauntlet providing deployability evidence.
The Fellowship is one of several factories in the lab's multi-factory model. The GPU factories handle the 27B-class models for work that exceeds what these tiny workers can carry, and the ensemble experiment above builds on these worker tiers to test team shape. How the orchestrator accepts and places orders is on the orchestrator page; the node configuration is on the Fellowship page. For discussion of the tournament methodology or results, reach me at nickcrowley97@gmail.com.