Cameras

Motion-triggered detection distributed across the ARM cluster

event-driven pipelines edge inference benchmark design failure-mode verification cross-compilation edge device management

The short version

This is a bespoke, event-driven camera detection pipeline: two PoE cameras and an NVR trigger the ARM cluster to decode and run object detection only when something moves, and only for cat, dog, or person. It proves I can design an edge-inference pipeline around its real workload, benchmark model and hardware honestly, and verify a detector whose failure mode is silence with live decoys that caught three hidden defects. It ties edge ML to hard systems verification: capacity calibrated to the target, CPU-versus-GPU settled by measurement, and cross-compiled workers kept alive on locked-down Android devices.

How it works

This watches my house and tells me when a cat, dog, or person appears, running the detection on the ARM cluster only when something moves. Two PoE cameras feed an 8-channel NVR on a private subnet, and the NVR performs its own hardware motion detection. When motion fires, the event reaches Tolkien, the head node of the Fellowship (ARM cluster), which coordinates a decode-and-detect burst across available ARM devices. Each clip is hardware-decoded on demand and run through an ncnn object detector on CPU, restricted to three labels: cat, dog, and person. A detection above threshold is passed to the event routing layer, which decides whether to notify. Detection runs only during events.

The pipeline is bespoke, and it is shaped around one fact about the workload: a two-camera house only needs analysis when something moves, so nothing should decode continuously. The same ARM cluster that handles local LLM inference performs camera detection when triggered, and the house automation layer this feeds is on the energy page. The detection pipeline is built and works; I have the bursts paused as of September 2026 so the ARM cluster can run the model tournament on undivided hardware. The cameras and the NVR's own motion detection keep running throughout.

Detection design

CPU over GPU

Measured with ncnn's own benchmark tool, which runs warm-up loops before timing, the devices' integrated mobile GPUs came in at about 1.7 seconds per inference under Vulkan. CPU ran the same model at 173 to 312 ms across the cluster's device families. Detection runs on CPU, which is why the cluster is treated as GPU-less for compute.

Accuracy over throughput

Motion-triggered bursts tolerate latency, so the pipeline is tuned for accuracy. In published benchmarks yolov4-tiny outscores the lighter yolo-fastestv2 on mean average precision, 40.2% against 24.1%. Those absolute numbers do not transfer to three labels on 320 px crops, but the ordering is why yolov4-tiny is the accuracy target. It runs at about 144 ms per inference on the cluster's fastest class, fast enough for burst work. I have not compared the two on the pipeline's own positive samples.

Capacity calibrated to the target

Two cameras need about 10 inferences per second at burst. Five inference nodes yield about 35, roughly 3.5× headroom; one node at about 7 per second is insufficient and two at about 14 meet the minimum. If the active node count drops below capacity during a burst, the system degrades to the faster, lower-accuracy model instead of dropping events.

A separate x86 CPU node stays in the detection pool as a floor, so the house is still watched when every ARM device is asleep. It answers the same NVR events as the ARM nodes and does not decode continuously. How much of the pool that node absorbs is the main tradeoff between coverage and load on the ARM devices. The ARM devices are CPU-only for inference: no NNAPI drivers exist on any SoC family in the cluster. The DSP paths present on two devices require a vendor SDK and signed images that are not available.

The active model is a constant in code, not runtime-configured. The higher-accuracy target is not yet wired to its active caller, so a lighter detector runs by default.

How it was verified

Benchmark protocol: feed what the consumer feeds

The pipeline crops each motion region to a 320 px square and scales it before detection, so the benchmark had to do the same. Feeding full frames instead of cropped motion regions dropped animal detection from 10 of 24 positive samples to 1 of 24. That sample set is small and made of real motion clips from these cameras, which is why the 10 of 24 is reported as a protocol result and not as the detector's recall. The Gauntlet (over 1,000 model-by-device runs) later followed the same principle: measure what the consumer actually uses.

Verifying a detector whose failure mode is silence

Clean startup and healthy heartbeats tell you the process exists, not that detection works. I used live decoy testing to confirm the pipeline actually fires on real events, and the decoys found three defects that would have stayed hidden behind silence:

The detector now returns an error on failure instead of an empty prediction list, and the live-decoy rule is written into the practices page.

DECLARED ≠ PRESENT ≠ VISIBLE

Every ARM device has a hardware decoder block, and decoder selection is deterministic: a node not using its designated decoder is a defect. Bringing the cluster online taught a broader lesson about verifying capabilities, because three different claims can each assert something the system will not actually do.

A decoder can even start at a resolution it declares it cannot handle, then emit nothing. Only a delivered frame counts as evidence the pipeline works.

Keeping the workers alive on the devices, and build notes

Surviving on Android

Every node runs a boot hook so the worker survives reboots. The runtime is whitelisted from doze mode, so a cold start cannot interrupt a short detection burst, and a CPU wake lock is held because throttling was measured at up to 12× without it.

The wake-lock state cannot be read over SSH by design: the power query returns nothing to an app UID. Health checks must never interpret that empty reply as "no lock held." This applies to all work on the Fellowship cluster, not just camera detection. Two independent Android mechanisms suppress an ARM device, and both must be defeated for full performance.

Cross-compilation

NDK builds for the detector assert that the GPU backend is enabled when it was requested, preventing a silent CPU fallback from producing a working binary that answers the wrong question about capability. Vulkan header versions must match the NDK's C headers, and the platform level must be high enough for Vulkan 1.1 symbols.

The static C++ runtime linker flag is mandatory: the Android userland on these devices does not ship the shared library, so a binary that compiles and passes file inspection will die at execution without it.

A separate background-subtraction detector exists for bulk analysis of mined frames only. It is superseded for live operation, and its recall numbers do not describe the live system.