Part 0 of the series. The objective: serve one model that exceeds the memory of any single node in the fleet by running it across several machines at high aggregate throughput. This part covers why sparse MoE makes that tractable, and the constraint that shapes every later decision.
The problem I wanted to solve
Several large open-weight models no longer fit in the usable memory of a typical consumer or prosumer node at the encoding you may want to run. DeepSeek-R1 is a 671B-parameter model with about 37B active per token. Kimi K2 reaches one trillion total parameters, with 32B active. Qwen3-235B-A22B has 235B total and 22B active. The Llama 4 family illustrates why the machine and encoding must always be named: Scout fits on one H100 at Int4, while Maverick fits on one H100 host. A 512 GB Mac Studio can also fit the weight payload of a model above 600B parameters when the encoding and runtime overhead stay within that memory budget.
Fornax targets an exact checkpoint and encoding whose weights, KV cache, activations, runtime reserve, and operating headroom exceed every individual node in the selected commodity fleet. Running that target in-house usually requires a datacenter-class multi-GPU host, which can defeat the cost and ownership goals of local inference.
Meanwhile, the machines a small team already owns may have enough combined capacity. Consumer NVIDIA GPUs provide fast compute. An Apple Silicon Mac with 128 GB of unified memory provides a large capacity pool. Spare AMD or CPU nodes add more. The fleet may have enough aggregate memory and compute even though no selected node can hold the target alone.
Fornax is intended to be that coordination layer. Its target is one model placed across a heterogeneous fleet on a local network, serving a checkpoint larger than any single selected node can hold and optimizing for aggregate throughput without pretending the network is free.
Why sparse MoE suits a heterogeneous fleet
A dense model of this size would be impractical on commodity hardware, because every token would need to touch every weight, and the weights would be scattered across machines connected by a network that is orders of magnitude slower than the memory bus inside any one of them. Sparse Mixture-of-Experts changes the arithmetic in a way that fits a mixed fleet well.
The defining property of sparse MoE is that it splits capacity from compute. The model holds a large number of expert weights, but each token is routed to only a small subset of them. DeepSeek-V2-Lite, the model I lean on later in this series, has 64 routed experts plus 2 shared experts per MoE layer. Each token selects 6 of the 64 routed experts, while the 2 shared experts run for every token. The memory needed to hold all experts is large, while the active expert compute is a small fraction of the total routed capacity.
That gives three facts that a heterogeneous cluster can exploit:
- Stage size can reflect each node’s memory and measured speed. A Mac may hold a larger complete-layer stage because of its unified-memory capacity. A faster NVIDIA or AMD node may hold fewer layers while completing its stage in the same interval. In the baseline, every node still executes the attention, router, shared experts, and routed experts for its own layers. Separating expert capacity from dense compute belongs to the later remote-expert topology and requires its own measurements.
- Pipeline parallelism gives the baseline a clean seam. Dense and MoE transformers can both be divided into contiguous layer ranges. Each Fornax stage keeps the attention, KV state, and experts for its layers, so the data plane sends one activation tensor plus its framing metadata at each boundary. MoE adds a second possible seam inside a layer, at the router, but remote expert execution is deferred until measurements show that it pays.
- The baseline communication is bounded and schedulable. Keeping experts resident with their stage avoids a data-dependent cross-machine expert all-to-all. Stage-boundary traffic is determined by the activation shape, dtype, microbatch, and number of boundaries, independent of which experts the router selects. Sparse top-k routing reduces active expert compute; it does not reduce the bytes in a stage-boundary activation frame.
The network remains the constraint, and the engine is organized around managing it.
The unavoidable network crossing
Any model larger than the largest node forces each token across at least one machine boundary. The layers that produce the token do not all live in one place. That adds a pipeline latency floor and a synchronization cost. Overlap and concurrency can hide or amortize part of that cost, but cannot remove it.
Fornax targets aggregate throughput and hardware utilization at a stated concurrency, using continuous batching to keep the pipeline full, communication and compute overlap to hide transfers, keeping each stage’s experts and KV resident with it, and balanced stage sizes so no machine is the bottleneck. Single-stream latency includes every network and synchronization cost. A bursty user seeking the lowest time-to-first-token should prefer a model that fits on one machine when that is an option.
The throughput claim is conditional and testable. Insufficient concurrency leaves the system near the single-stream latency floor, so measurements must include that case.
Current implementation status
Fornax draws a clear line between MAX and itself. MAX owns per-node execution: graph construction and compilation, kernels and custom ops, device memory, backend-local KV primitives, and the supported model architectures. Fornax owns the cross-node engine: hardware inventory and measured links, the planner and placement search, global stage orchestration and request ownership, versioned activation framing and transport, scheduling and admission and cancellation and backpressure, and distributed result assembly. Part 3 is that boundary in detail.
The public repository is a pre-alpha executable specification and Engine v0
prototype. It contains a planner and cost model, the historical FNX1 framing and
lockstep mechanism, a candidate FNX2 ragged reference engine with an integrated
bounded scheduler, independent loopback workers, validators, and hardware
bring-up tools. These model-free paths make T0 and T1 claims only. No bundled
physical text generator, cross-vendor MaxStageBackend, or production Fornax
server exists yet.
The physical adapters that execute complete model stages on real GPUs and Macs are the current build target. They must satisfy the same logical contracts and add operator, stage-output, end-to-end numerical, and physical transport validation.
Each number in the series is labeled with its evidence class. Contract tests and simulated runs establish process, socket, queue, and failure behavior, and they make no hardware claim. Cost-model throughput is a prediction. A measurement exists only when the artifact names the physical model, fleet, MAX build, dtype, shape, and command. I keep those apart, because the value of a planner is lost the moment its predictions stop being falsifiable. AMD sits behind the same discipline: it joins the fleet only after its MAX and ROCm paths pass physical validation.
The public repository records the current code, commands, gate status, and dated bring-up notes.
The next part steps back from Fornax to examine the industry problem around it: the work required to run models on a new accelerator, and why a compiler platform like MAX exists.
Next: What Accelerator Support Requires. Series index.
Sources: DeepSeek-R1 model card; Kimi K2 model card; Qwen3-235B-A22B model card; Meta’s Llama 4 announcement.