Part 3 of the series. The architecture: the MoE seam Fornax cuts along, why the spine is pipeline-parallel, the candidate Stage ABI designed for tensor exchange between unlike machines, how the planner separates feasibility from performance, and the evidence model that keeps a simulated run from ever being mistaken for a hardware measurement.
The seam: where Fornax cuts the model
Part 0 argued that sparse MoE splits capacity from compute, and that this split is what makes a heterogeneous fleet viable. Fornax turns that observation into a concrete cut through the model. It first partitions the model into complete contiguous layer ranges, and each range becomes a stage. A stage keeps the attention, the KV state, and the experts for its own layers, so the only thing that crosses a stage boundary is one activation tensor payload plus protocol metadata.
Within a baseline stage, the same host executes the complete layer range. This includes attention, the KV cache, the router, the shared experts, and the routed experts. The last stage also owns the final normalization and output path. Where a stage has a local homogeneous accelerator group, MAX can distribute that work inside the stage. A Mac assigned a stage still has to qualify every required operation for that layer range. Capacity-only expert placement belongs to the future remote-expert design.
For each token, the router picks a small top-k subset of the routed experts. In DeepSeek-V2-Lite that is 6 of 64 routed experts, alongside 2 shared experts. The baseline keeps those experts resident with the stage that owns their layers. The flow through a single MoE block reads like this:
hidden states
-> attention, KV read/write
-> router / top-k
-> selected routed experts (resident with the stage)
-> weighted gather of expert outputs
-> next layer
Executing routed experts on a remote worker remains a future option. The deferred-work section records its qualification gate. The baseline keeps each stage’s experts and KV local because that communication pattern is bounded and can be measured directly.
The spine: pipeline parallelism across unlike nodes
The cross-node parallelism choice follows from the constraint in Part 0 and the collective gap in Part 2. When you spread a model across machines, you have to choose what crosses the wire between them, and on a commodity network that choice determines whether the design works at all.
| Strategy | What crosses the wire per token | Fornax stance |
|---|---|---|
| Tensor parallel across mismatched nodes | one or more latency-sensitive collectives per transformer block | keep it inside a qualified homogeneous island |
| Pipeline parallel by contiguous layer group | one activation tensor payload plus protocol metadata per stage boundary | the default cross-vendor spine |
| Resident experts with the stage | nothing extra; experts travel with their layers | the baseline |
| Remote expert execution | selected hidden-state batches | deferred until the stage pipeline is measured on hardware |
Tensor parallel implementations use one or more latency-sensitive collectives per transformer block. Depending on the layout, these can be all-reduce, all-gather, or reduce-scatter operations. Commodity 25 or 100 GbE is a poor default for that pattern between an NVIDIA box and a Mac unless measurements show otherwise. Pipeline parallelism by complete contiguous layer groups reduces the cross-node exchange to one activation frame at each stage boundary, which Fornax can pack, transfer, and schedule. The target cross-vendor spine is speed-balanced pipeline parallelism: stage sizes are chosen so the machines finish their assigned stages in roughly the same wall-clock time.
MAX’s supported homogeneous multi-GPU execution stays inside a stage. Where a model and backend expose tensor, data, or expert parallelism, a physically qualified accelerator island can use MAX’s collectives. Fornax adds pipeline parallelism between unlike stages.
A concrete boundary budget
For DeepSeek-V2-Lite, a BF16 stage-boundary row contains 2,048 values, or 4,096 bytes before framing. The payload for one boundary is:
payload_bytes = token_rows * 2,048 * 2
A 4,096-token prefill activation is 16 MiB. Its ideal serialization time is about 5.4 ms on a 25 Gbit/s link or 1.3 ms on a 100 Gbit/s link, before protocol, packing, copying, congestion, and receiver costs. A decode microbatch containing one row from each of eight requests is only 32 KiB, but link latency and stage synchronization remain exposed. These are arithmetic bounds, not measurements. Fornax profiles packing, transfer, unpacking, queueing, and exposed wait separately because wire bandwidth alone does not predict pipeline throughput.
The engine boundary
The current public Python Engine has a small str -> str interface over a
generator supplied by its caller. The caller provides the code that turns a
prompt string into a response string. This seam is executable and tested. Model
loading, worker startup, tokenization, sampling, and serving remain outside the
public Engine.
request text
-> public Engine
-> caller-supplied generator
-> response text
The target Fornax engine will own model execution through the planner, workers,
transport, and physical backends. Ignis, the agent harness from the
Ignis series, could then adapt that engine behind its own
Engine trait. An OpenAI-compatible service could wrap the same boundary.
Neither integration exists in the public Engine today.
The eventual network API needs a larger contract. A product adapter must define and test the exact compatibility surface:
| Concern | Contract required above the engine |
|---|---|
| Input | messages or prompt, model identity, tokenizer and chat-template hash, sampling parameters, stop conditions, maximum output |
| Streaming | ordered token or text deltas, finish reason, usage accounting, terminal error, cancellation result |
| Structured interaction | tool definitions and calls, response-format constraints, reasoning fields where the selected model exposes them |
| Errors | stable codes for validation, admission, timeout, backend, transport, cancellation, and capacity failures |
| Service behavior | model discovery, readiness, authentication, request limits, API version and compatibility policy |
The public Engine provides no HTTP server, tokenizer, sampler, model loader, or
worker manager. Those are separate implementation gates.
Request lifecycle and scheduling
The target distributed serving path begins by validating the request, fixing the model snapshot and tokenizer/template identity, and reserving enough stage-local KV and queue capacity for admission. Prefill then crosses the stages in order. Decode uses continuous batching to group ready request rows into microbatches while preserving each request’s token position and KV epoch. The final stage produces logits or samples a token, and the serving adapter emits an ordered stream event.
Cancellation follows the same ownership chain in reverse. Admission stops new work, every stage receives the cancellation, and the request completes only after the engine can report whether each stage’s KV epoch changed. Slow clients also need bounded output queues so backpressure can reach admission instead of allowing detokenized output to grow without limit. The current reference and simulated backends exercise the control contracts; tokenizer execution, physical MAX stage execution, and production streaming remain later validation work.
From a checkpoint to stage artifacts
A physical stage cannot load an arbitrary slice of a repository and infer the missing details. The build step must resolve one immutable checkpoint snapshot, record its source and commit, and hash the tokenizer, chat template, model configuration, and weight index. It then constructs an artifact for each planned layer range.
The first-stage artifact owns the embeddings and its assigned layers. Middle stages own their layer ranges. The final-stage artifact owns its layers, final normalization, and output head. Ownership must be explicit for tied embeddings, shared parameters, routers, shared experts, and routed experts so the build does not duplicate them by accident. Each artifact also preserves dtype, quantization scheme and scales, tensor names and layouts, shard ordering, and any backend-specific conversion version.
The stage manifest records the source lineage, layer range, file sizes, and checksums alongside the boundary tensor contract. Installation verifies those bytes before a worker loads them. A plan names exact stage artifacts instead of relying on a mutable model cache. The current pre-alpha reference engines use model-free fixtures; a production checkpoint-to-stage artifact builder remains to be implemented and qualified.
The Stage ABI: a cross-vendor tensor contract
The pieces MAX does not provide, from Part 2, all come down to getting tensors between machines that share neither a memory bus nor a vendor. Fornax uses a formal Stage ABI so an NVIDIA sender and an Apple receiver agree on every byte and reject mismatches before execution.
In the target physical runtime, every worker first loads a stage manifest. The manifest binds an immutable model snapshot, the tokenizer and template hashes, the MAX build, the plan hash, the layer range, the input and output tensor contract, the device requirements, the weight artifacts, and the stage-local KV policy. A mismatch stops stage admission.
The worker then presents a language-neutral stage executable contract with a small set of verbs: load, health, execute, cancel, drain, unload. This is the seam a physical MAX backend implements, and it is also the seam the reference and simulated backends implement, so the same orchestration code drives all of them.
The repository carries two wire generations. Historical FNX1 is the lockstep
Engine-v0 contract exercised by same-host loopback tests. It carries an absolute
deadline_ns based on a monotonic clock. That value is meaningful only when the
sender and receiver share the same host and clock domain. It cannot define a
cross-host deadline because monotonic clock epochs differ between machines.
Candidate FNX2 is the ragged-batch contract. It carries a remaining
deadline_budget_ns for each sequence. A receiver converts that budget into a
deadline in its own monotonic clock domain. The orchestrator subtracts elapsed
route time before forwarding, and a retry may tighten the stored deadline but
cannot extend it. A zero budget means the request has expired. Raw monotonic
timestamps never cross a process or machine boundary in FNX2.
Both protocols wrap metadata and tensor bytes in a versioned frame with bounded lengths, a monotonic sequence number, and checksums. The metadata binds the frame to a plan, manifest, request or batch, source and destination stage, token positions, KV epoch, tensor descriptor, and the deadline field defined by that ABI version. The receiver validates the frame before handing bytes to the backend. Acknowledgements, credits, cancellations, errors, and heartbeats use the same framing. Each ABI version accepts only its defined message kinds.
Transport stays outside compiled MAX graphs. A graph may pack or unpack a boundary tensor. The Fornax runtime owns network calls and reports transport failures, leaving the MAX execution unit independently testable.
Bounded transport and failing closed
Stages run in independent worker processes. Historical FNX1 uses persistent TCP connections with one message credit and a byte credit for the next frame. This enforces a bounded lockstep channel in the current loopback path. Candidate FNX2 adds receiver-authoritative credits across sequences, token rows, KV rows, queue bytes, and payload bytes, integrated with its ragged scheduler. Both paths retain a source buffer until acknowledgement or terminal release. Their current evidence is T0/T1 loopback execution. Separate control and tensor data planes remain a physical runtime target.
Failure handling is built to fail closed. Plans and manifests are immutable during execution, and sequence checks, hashes, shape checks, byte counts, and CRCs all reject bad input before a stage runs. A reconnect leaves retry authority with the orchestrator. Replay is allowed only from a boundary where the acknowledgements and every stage’s KV epoch are known, and even then only after the orchestrator installs a new replay epoch and confirms that no irreversible, client-visible token was emitted past that boundary. Ambiguous state produces a terminal request error. KV stays stage-local: each stage owns the pages for its layer range, a request carries the expected KV epoch, and a result reports the epoch before and after. Cancellation and recovery use those values to determine whether state changed. Moving KV pages between stages would require a separate distributed-state contract and is outside the baseline.
The planner: feasibility first, performance separately
The planner takes model dimensions, memory estimates, dtype, context, concurrency, declared backend capabilities, usable node memory, link inputs, and runtime constraints. Its search produces a stage sequence and explains rejected nodes or routes.
The current implementation enforces deterministic resource accounting and the declared KV capability. Its deployment-authority mode also checks exact target runtime, accepted build, required operations, activation dtype, weight quantization, and role against complete node capabilities. Referenced model, capability, measurement, calibration, and route records must resolve through a separate evidence registry with the expected type, status, validity window, and SHA-256.
The default planner path labels its result exploratory. Deployment mode fails
closed when required evidence or calibration is missing. The local registry
checks identity and bytes; it does not attest that a measurement was collected
correctly. Current fixtures and static hardware estimates also lack physical
stage, packing, route, queue, native-memory, and prediction-error calibration.
Their placements remain exploratory.
For a modeled route, the cost model keeps stage service, packing, wire transfer, unpacking, queueing, and exposed wait as separate terms. These estimates support design exploration. Deployment authority still requires physical profiles that name the hardware, MAX build, model snapshot, layer range, dtype, shape, context, concurrency, command, and evidence record.
What exists in the repository
The repository contains a pre-alpha executable contract layer. The historical FNX1 path provides an in-process reference backend and two independent worker processes over loopback TCP. It exercises lockstep frames, sockets, credits, bounded queues, cancellation, timeouts, and same-worker reconnect behavior at T0/T1.
Candidate FNX2 adds a model-free ragged reference engine, unequal prefill chunks, changing decode subsets, per-sequence results, deadline budgets, lease/replay rules, and two independent loopback workers. Its exact-wire golden vectors and slow oracle are useful protocol evidence. FNX2 remains a candidate until its review and physical conformance gates close.
There is no physical MaxStageBackend in this path. The current reference
engines do not load model weights or produce hardware logits, and loopback
results make no claim about a physical network. A future MaxStageBackend will
implement the same stage executable contract and pass the protocol conformance
corpus alongside operator and stage-output tests on a named hardware and MAX
build.
The evidence model
Fornax records three evidence classes separately.
| Class | What it establishes | What it does not |
|---|---|---|
| Contract tests and golden vectors | deterministic schemas, framing rules, planner regressions, failure validation | process behavior or hardware correctness |
| Reference and simulated execution | independent workers, persistent sockets, credits, queues, cancellation, timeouts, reconnects, deterministic fault handling | MAX kernel correctness, physical network speed, or product throughput |
| Physical distributed-path validation | cross-backend activation and logit parity, measured stage and transport cost, planner calibration on named hardware | frontier-capacity or production readiness unless separately validated |
Reference or simulated execution can produce real process, socket, queue, and wall-clock data. That data describes those backends, not a GPU or a physical network. Cost-model throughput is a prediction. Hardware throughput becomes a measurement only when the artifact names the physical model, fleet, build, and command. The product target is aggregate throughput at the concurrency needed to fill the pipeline, and a model that spans machines has an unavoidable network and synchronization floor, so Fornax does not promise single-stream latency parity with a model that fits on one node.
The operator surface
A distributed engine also needs a lifecycle outside request execution. The operator must be able to inventory devices and links, verify driver and MAX build compatibility, install an immutable plan, warm and health-check every stage, admit traffic only after the route is ready, drain it for an upgrade, and roll back to a previous plan without reinterpreting in-flight frames.
The minimum observability contract includes request, plan, stage, and microbatch identities; per-stage compute and queue time; packing, transfer, and unpacking time; credit exhaustion; KV allocation and epoch changes; cancellation; and the reason a node was selected or rejected. Production deployment adds authenticated node admission, endpoint authentication, encryption policy, certificate and secret lifecycle, audit logs, resource limits, and upgrade compatibility. Phase 0.5 implements and simulates only the bounded queues, identities, framing, and failure semantics. Physical deployment, production security, and fresh-operator install validation remain later gates.
Physical qualification and deferred work
The next physical gate starts with two named hosts and one immutable, small
checkpoint split into stage artifacts. It requires a physical
MaxStageBackend, exact hardware and software lineage, and candidate FNX2 over a
real link. The run must cover prefill, changing-subset decode, cancellation,
expiry under skewed host clocks, reconnect fencing, and cleanup.
Activations and final logits are compared with a single-process reference at
every boundary. The artifact records stage compute, packing, wire transfer,
unpacking, queueing, and exposed wait.
That gate closes only after repeated parity runs pass and the measured values calibrate the planner within the declared error bound. It establishes a physical two-stage route for the tested checkpoint and fleet. Larger models, longer contexts, and additional backends need their own evidence. Apple’s first role will be a complete layer stage if it passes the required operation and parity tests. Capacity-only expert service waits for the remote-expert contract. Part 4 describes the Apple model bring-up evidence available so far.
Several capabilities remain behind their own contracts and measurement gates. Remote expert execution follows the contiguous-stage qualification and needs evidence that batching and locality cover its routing, transport, and synchronization cost. Expert migration, quantized activation transport, distributed KV movement, RDMA-class transports, stage replication, and elasticity are future work with separate correctness and measurement gates. The project sequence begins with the bounded stage pipeline and adds each capability after its qualification data is available.
Previous: MAX Platform Assessment. Next: Model Bring-Up. Series index.
Implementation references: Fornax repository; historical FNX1 Stage ABI; candidate FNX2 ragged design; planner authority status; project plan v4.