Fellowship: Distributed ARM-Device Compute
Eleven nodes and seventy-six cores: an experimental compute cluster built from low-power ARM devices for independent-task parallelism and model-scale evaluation.
The Fellowship is a heterogeneous ARM-device compute cluster. It runs eleven nodes totaling seventy-six CPU cores: seventy-two arm64 cores across ten nodes, plus four armv7l cores on one legacy 32-bit device. The cluster handles embarrassingly parallel workloads (captioning sweeps, distributed evaluation banks, OCR passes, corpus preprocessing), the tasks where independent workers scale linearly and no cross-node communication is required.
The cluster began with a plainer question: what useful work could a handful of inexpensive, low-power ARM devices do? One of the answers turned into the experiment it is now built around. Can small models and specialized workers reach results comparable to a bigger model on a big card? The Fellowship makes that comparison ownable in-house.
The head node is an x86 desktop called Rigby running a container named Tolkien, which places work across all eleven devices. The Fellowship is one factory in the lab's multi-factory model. It runs the same order-taker pattern as the three x86 GPU hosts (Mothership, BigHonker, and Softserve): the central scheduler places product orders, each factory declares its capabilities and reports live state, and orders route to whichever factory can fulfill them. The Fellowship's slice of that work is independent CPU parallelism on ARM silicon; the GPU factories handle model inference and training. Same code base, same lifecycle, different substrate.
The cluster started with three devices (Sam, Frodo, and Gandalf) in early August 2026 and grew to eleven over the following month. The most recent additions, four nodes in early September, were only possible because the scheduling layer was built for capability-aware heterogeneity from day one. Round-robin placement would have broken the moment a 32-bit device joined an arm64-only cluster.
Adding hardware: the intake harness
The eleven current nodes span four device families with different silicon, different unlock procedures, different userlands, and one that is 32-bit instead of 64-bit. What keeps the Fellowship maintainable at that variety is the intake harness on the head node, which handles new-device onboarding as a documented, resumable procedure instead of a one-off migration.
A new device is plugged into the head node. The runner detects the device family from hardware IDs and hashes, then gates the run against a profile registry so a near-looking model cannot slip through as an alias. It prompts me for the physical unlock steps that must happen on the device itself, runs the family-specific flash sequence, and provisions the persistent agent and shared token. Finally it exercises a positive-control task to prove the device can actually run work, and registers the node with the head so it starts accepting placements.
The whole procedure is resumable. If a step fails or I need to walk away, the state on disk records what is already done, so the next run picks up where the previous one left off instead of repeating an unlock or a flash.
See the engineering detail: harness lineage and the physical-interaction problem
I did not build the harness as a general tool at first. It came out of getting individual devices online one at a time:
- Nodes four and five (Merry, Pippin) were the first pair to force the underlying scripts to be reusable. The flash sequence for their device family, the shared-token identity system, and the watchdog and positive-control provisioning all came out of that pair's build.
- Node seven (Aragorn) is deliberately not the template. It was flashed with one-off hardcoded scripts for its specific device profile and predates the generalized runner. The lesson from Aragorn was that hardcoding the physical-interaction steps could not scale as more device families joined.
- From the eighth node (Sméagol, September 2026) onward, new nodes go through the canonical intake harness. Sméagol is a legacy 32-bit armv7l device; the three following nodes are a different arm64 family. The same runner handled both without new scripts, because each device family lives in a profile registry with an explicit unlock command, flash mode, and disk layout.
The physical-interaction step is unavoidable. An Android bootloader unlock requires a Volume Up press on the device during the fastboot phase, which no software on the head node can automate. The harness pauses at that step, prompts me with the exact button press required, then resumes. Every other step of intake (detection, profile selection, artifact staging, flash, provisioning, agent install, registration) runs unattended.
The roster
Eleven nodes across four device families, in order of arrival. Sam, Frodo, and Gandalf came first in August 2026, and Merry and Pippin brought the cluster to five. Legolas arrived mid-August as node six, and Aragorn a week later as node seven. Sméagol joined in early September as the first 32-bit armv7l node, followed by Gimli, Boromir, and Faramir. Ten of my eleven nodes come from one hardware line across three of its generations, so one intake harness with a profile per generation covers all of them. Sméagol is a different line altogether and my only armv7l node, which is why I made the scheduler carry explicit architecture tags instead of assuming a single ISA.
New hardware enters the cluster under a simple rule: measure before naming. A device reports its capabilities, proves it can run a positive-control task, and earns its name afterward.
How work gets placed
Tolkien accepts two order kinds. A sweep carries a set of input files: image corpora for captioning, document batches for OCR, anything file-addressable. A map carries a set of opaque strings: prompt banks for distributed evaluation, classification inputs, text-processing sweeps. Both are embarrassingly parallel and need no inter-node communication during execution.
The worker budget per node is computed from live memory:
workers = min(cores, MemAvailable_GB / 0.15)A four-core device with 1.4 GB of RAM receives fewer concurrent workers than an eight-core device with 6 GB, and the budget adjusts as memory pressure shifts on a running node. There is no static configuration to edit when workload patterns change.
See the engineering detail
Capability tags are the source of truth for placement eligibility. Each node declares what it can do when it registers with Tolkien: supported task types, architecture family, hardware-decode availability, and runtime constraints. The scheduler reads those declarations instead of assuming a device class from a hostname or arrival order. Adding a new device is a matter of the agent reporting its tags and joining the pool, with no code change required.
Two valid target environments exist for native binaries: glibc payloads running inside a proot Debian container on the device, or bionic payloads running natively within Termux for smaller workloads. Which environment a job uses depends on the payload and the node's runtime capabilities, and the agent selects at dispatch time.
MPI was available but not adopted as the default placement strategy. HPL and MPI ran successfully across the cluster during early validation and passed residuals, but distributed-MPI efficiency landed at roughly 16% across WiFi at scale. That is expected from consumer 5 GHz networks on ARM SoCs. It was enough to prove the substrate works, and it confirmed that independent-task parallelism is the workload shape this hardware actually performs well under.
What kind of work it runs
The Fellowship handles CPU-bound batch work where the unit is "a few KB in, minutes of CPU, a few KB out." Anything that fits that shape and does not need cross-node communication is a good fit. Concrete workloads the cluster has run in the lab or in controlled experiments:
Measured: a 29-minute 54-second episode transcribed in 14 minutes 13 seconds on one node, 2.1× realtime, under a 35 W rated peak per node and with zero GPU usage. Each node takes independent files, so throughput scales with node count. A quiet subtitle factory on hardware that is otherwise idle.
Systematically tested against a 688-cell tournament matrix in August 2026: thirteen models (ten text, three vision) across four hardware classes, evaluating chat, coding, tool-use, and vision tasks. 512 cells actually ran; 436 passed the quality bar. Three credible worker tiers emerged: Qwen2.5-1.5B as the general/tool default (8/8 chat and 8/8 composite tool in the matrix, 6/6 on the earlier behavior evaluation), Qwen2.5-Coder-1.5B as the coding worker (7/8 broader suite), and the 0.5B pair as fast specialists inside narrow envelopes. For vision, SmolVLM2-500M is the deployable default across every hardware class; Qwen2-VL-2B is the quality specialist on the larger nodes. The full breakdown is on the Gauntlet (model tournament) page.
Full passes over the documentation library: RAG chunking, sanity checks, format normalization. The independent-worker model fits anything that walks a corpus and applies a per-document transform, because there is no shared state to synchronize.
Any problem shaped as "run N independent trials and aggregate the results" maps directly onto the map order kind: sampled parameter sweeps, Monte Carlo simulations, brute-force search over a discrete space. The cluster ran a Monte Carlo experiment back when it was closer to five nodes, which is how the independent-task order shape got its trial by fire.
Perceptual hashing, thumbnailing, checksums, file-inventory sweeps. Fits the "KB in, minutes of CPU, KB out" shape cleanly and parallelizes without contention.
Distributing throttled fetches across nodes to stay under per-source rate limits without stalling the whole job on one endpoint. Each node holds its own connection pool and coordinates through the head's queue.
Off the menu: heavy model inference above roughly 1.5–2 GB of RAM per process, GPU-accelerated work of any kind (the cluster has no GPUs), tightly coupled MPI-style tasks (WiFi round-trip latency kills them), and large-data-movement workloads (all eleven nodes share one radio link). Those live on the GPU fleet, not here.
Sméagol: bringing up a 32-bit node
Sméagol was a legacy 32-bit ARM device sitting on a shelf, running a community OS build with a terminal environment that provides sshd and a working POSIX layer. Standard 64-bit llama.cpp binaries do not execute on 32-bit ARM, so the runtime required a rebuild from source.
See the engineering detail
The build used Android NDK r27c targeting armeabi-v7a. The llamafile code path was compiled out because it relies on 64-bit atomics that armv7l does not support. The resulting binary runs real end-to-end model generation on the device, not just parsing or tokenization.
Network access is wired-only through a USB-RNDIS connection to the head node. The device's WiFi proved unreliable on this hardware revision, so the head node maintains a persistent USB-Ethernet bridge and the device sits at a fixed address on that segment. Reboot survival has been demonstrated: the head node re-asserts USB composition and addressing, while the device's boot layer restores sshd and the local agent automatically.
Sméagol running in the lab validates the cluster's capability model. Every other node is arm64; this one is armv7l with a different binary, a different network path, and a smaller worker budget. If the scheduler placed an arm64-native job on it, that job would fail. The scheduler does not make that mistake, because the capability tags carry the constraint instead of an assumption baked into dispatch logic.
The design principle
The question the Fellowship grew into is whether small models and specialized workers can reach results comparable to a bigger model on a big card, at a similar hardware cost. The ensemble experiment, Phase 5 of the Fellowship program, is the direct test. It varies team shape from one to ten workers with the model held constant, and compares the independent reviews to find where they disagree. The question is still open; execution status is on the Gauntlet page. The cluster is what makes it answerable in the lab: both approaches run on hardware already in place, outputs are compared directly, and iteration proceeds without external API calls or cloud billing. The infrastructure is here to measure the trade-off honestly instead of picking a side up front.
What's still active
The ensemble experiment stands at 1,141 of 1,165 runnable arms; the remaining 24, and the appraisal, are queued behind other work. Alongside it the cluster handles corpus preprocessing for library triage orders and capability-model growth as additional device families prove stable. The GPU factories handle the lab's heaviest reasoning loads (27B and 30B inference, QLoRA training), and the Fellowship handles independent-task parallelism and gives the lab a way to check whether some of those GPU runs would have been faster split across more workers.
ARM-device binaries are compiled on an arm64 build host running a Debian container via QEMU binfmt, not on the devices themselves, because the device hardware is too constrained for reliable compilation. The cluster's roster and topology live in a single source document that node reservations and address maps derive from, so adding a new device follows a documented procedure instead of custom wiring.
Code reference: github.com/drhawktopus. Questions or discussion: nickcrowley97@gmail.com.