Part 4 - Model Bring-Up

Part 4 of the series. Fornax depends on MoE checkpoints reaching the relevant runtime paths on the available hardware. A Transformers/PyTorch Qwen smoke across four H100s worked. On a MacBook using MAX, one Qwen MoE failed to compile, Kimi failed earlier, and DeepSeek-V2-Lite reached a short generation smoke after local Apple kernel patches. This part records the evidence and its limits.

The locally patched DeepSeek-V2-Lite result reaches the short-generation rung. Numerical, serving, and target-system qualification remain open.

Model bring-up before runtime integration

A planner can establish that a model fits a fleet. Operator execution is a separate gate, and after Part 2’s operator chart, that was the largest unresolved risk. Before assigning Apple a role in the distributed design, I ran MoE checkpoints through two bring-up paths: Transformers/PyTorch on a Linux box with four H100s, and a source-built MAX checkout on an M3 Max MacBook Pro with 128 GB of unified memory.

Each result below has a stated evidence scope. The Qwen result covers same-host Transformers execution. The Mac results cover a source-built MAX checkout. Production serving requires separate qualification; emitting a few tokens covers only a small part of that work.

NVIDIA bring-up: Qwen3-Omni-30B across four H100s

The first run used the published Qwen/Qwen3-Omni-30B-A3B-Instruct checkpoint. Its thinker text stack has 48 layers with 128 experts and 8 active per token, and its talker stack has 20 layers with 128 experts and 6 active. That is the routing shape Fornax is built for.

I ran it through Transformers from a cached snapshot across all four H100 80GB cards with BF16 weights and a device map that spread the parameters roughly evenly (about 9.0, 9.4, 9.4, and 7.6 billion parameters on cuda:0 through cuda:3). The model loaded from cache in about 23 seconds and generated 24 tokens from a 33-token prompt in about 9.6 seconds, around 2.5 tokens per second in this tiny smoke, and the output began sensibly: “MoE stands for Mixture of Experts, a technique in AI inference.” I promoted the ad-hoc script into the local-real-moe-serving-smoke command in the Fornax repository, with a validator that produces a checkable artifact.

The run took place on 2026-06-29 against snapshot 26291f793822fb6be9555850f06dfe95f2d7e695, using PyTorch 2.12.0+cu130, Transformers 5.8.0.dev0, BF16, and device_map=auto. GPU 0 was capped at 24 GiB because another process occupied part of that device; the other three GPUs were capped at 70 GiB each. The exact command is in the public README, and the dated development record records the output and validation. The generated JSON artifact remained under /tmp on the source machine and was not committed, so an outside reader can inspect the procedure and record but cannot verify the original artifact bytes.

This result covers same-host text generation through Transformers. MAX, vLLM, HTTP serving, full multimodal Qwen-Omni execution, and cross-machine transport are outside its scope. It establishes that this checkpoint generated coherent text across four H100s under the bounded configuration and that the Fornax command recorded the run. MAX compatibility remains a separate question.

Apple compile failure: Qwen3-30B-A3B

Apple’s large unified memory supplies the capacity described in Part 0, so I next ran a Qwen MoE on the Mac. I targeted Qwen/Qwen3-30B-A3B (Qwen3MoeForCausalLM, 128 experts, 8 active per token) on the M3 Max through a pinned MAX nightly.

It did not run. MAX resolved the model, started the pipeline, reached graph compilation, and then failed before emitting a single token, with this signature:

RuntimeError: Graph compilation failed
Metal Compiler failed to compile metallib
graph compiler could not elaborate the generated KGEN

This matches the lower-stack failure described in Part 2. The graph reached Mojo-to-Metal lowering, where compilation failed. The FP8 variant, Qwen3-30B-A3B-Instruct-2507-FP8, failed with the same signature after its shards were downloaded. Adding --sample-on-host or --prefer-module-v3 left the failure unchanged. max serve reached the same crash during model worker initialization as max generate.

Source inspection placed the failure while LogProbabilitiesMixin unconditionally built and loaded a log_probabilities_ragged_graph on the target device, before request-level settings could affect execution. The CPU path in that runtime did not accept BF16 or FP8, and a float32 30B MoE exceeded the available memory headroom once runtime overhead was included.

The failure shows why documented support still needs a local probe. The public MAX 26.4 release notes announced Apple Silicon support for many models in the Qwen family. This probe covered a specific MoE architecture, encoding, and pinned nightly, and it failed in the Metal compiler. That target remains ineligible for Fornax placement.

moonshotai/Kimi-VL-A3B-Instruct (KimiVLForConditionalGeneration, 64 routed experts, 2 shared, 6 active) failed earlier in the stack. Model configuration stopped with ValueError: 'yarn' is the only rope_scaling type currently supported, before graph compilation. The Qwen and Kimi probes exposed separate compiler and model-configuration gaps.

Apple bring-up after kernel patches: DeepSeek-V2-Lite-Chat

deepseek-ai/DeepSeek-V2-Lite-Chat is the model this series uses for Apple bring-up. Reaching generation required kernel and host-runtime changes. It is a 15.7B-parameter BF16 MoE with 64 routed and 2 shared experts and 6 active per token, and its attention is Multi-head Latent Attention (MLA) with a specific shape that drives the kernel work: a 128-dim non-RoPE part and a 64-dim RoPE part in the query and key, accumulating into a 128-dim value head, for a 192-dim query depth.

The Apple target lacked the required MLA path. The first run failed at decode with a compile-time constraint that states the limitation directly:

constraint failed: flareMLA_decoding currently only supports Nvidia and AMD GPUs.

Even a one-token generation enters the decode path. I worked in a vendored MAX source checkout and added each missing Apple component as its failure surfaced.

MLA prefill. The NVIDIA kernel relies on shared memory and a tiling scheme that differs from Apple’s flash-attention path, so Apple required its own prefill implementation. The first Apple version used a 16x16 simdgroup matrix path restricted to M5. Metal rejected that pipeline on the M3 Max because the instruction requires a newer GPU family. M3 needed a distinct 8x8 simdgroup matrix implementation. Part 5 describes that split.

MLA decode. Upstream gated MLA decode to NVIDIA and AMD, so I added an Apple route. The optimized split-K version compiled and then failed Metal pipeline creation on the M3 with an XPC interruption. The active bring-up path runs Apple’s existing naive multi-head attention over the full DeepSeek MLA cache row, then copies the latent and value portion into the MLA output. The focused smoke produced finite outputs. Numerical comparison with an independent reference remains open.

MoE bucketing. With attention unblocked, the MoE routing kernel failed on a Metal-unsupported vote primitive used to bucket tokens by expert. I replaced it with a one-thread serial fallback that avoids warp operations, shared memory, and atomics. Reference comparison and a parallel implementation remain open.

Rank-2 gather. Then the MoE permutation path failed in a generic gather kernel that lowered to Metal-invalid double-precision floor and conversion instructions, in kernels with names like gather_r2_w8_b256_gs_True_.... NVIDIA accepts that lowering; Apple’s GPU rejects it. I added a direct flat rank-2 axis-0 gather path that avoids the f64 instructions. Focused smokes passed for int32 and uint32 indices.

One host contract. After all the kernels existed, generation still failed because a host-side helper, mla_decode_dispatch_scalars, rejected the Metal API. The Apple decode fallback uses one partition, so the helper now returns a single-partition shape for Metal. This aligned the host-runtime contract with the new kernel route.

After these changes, two max generate runs on 2026-07-01 accepted the same 14-token prompt on the M3 Max. An output limit of 1 emitted !; a limit of 8 emitted !!!!!!!!. The eight-token run observed about 4.03 tokens per second. The one-token run observed 343.7 ms from completed initialization to its first token, after 104.2 seconds of model build, graph compilation, and initialization. The environment record is incomplete, so these timings are local observations and cannot support a cross-system comparison.

The punctuation-only output contains no useful linguistic signal. The two runs show that the source-built path completed bounded model loading, graph compilation, and execution through the patched MLA, MoE routing, and gather paths. Output correctness remains untested.

Public evidence record for the Apple smoke

The public runbook records the commands, captured output, and known gaps:

Field	Captured value
Run date	2026-07-01
Model snapshot	85864749cd611b4353ce1decdb286193298f64c7
Hardware	Apple M3 Max, 40-core GPU, 128 GB shared memory
MAX build	Source-built 26.5.0.dev2026063006
Local Modular patch commit	957aeded5296d6638386409849b60f82c36146dd
Input and output limits	14-token prompt; output limits of 1 and 8 tokens
Model-level output	! and !!!!!!!!; punctuation only
Focused finiteness evidence	MLA prefill and decode smokes reported zero non-finite outputs
Missing environment fields	Exact macOS and Metal toolchain versions were not captured

The missing macOS and Metal versions make this an incomplete evidence record. The runbook supports inspection of the recorded setup and output. Reproduction, numerical correctness, and performance qualification remain unclaimed.

Scope of the runs

Model	Hardware	Result	Scope
Qwen3-Omni-30B-A3B-Instruct	4x H100 80GB	Generated coherent text through Transformers/PyTorch, ~2.5 tok/s	same-host Transformers text generation; MAX, distributed Fornax, and HTTP serving untested
Qwen3-30B-A3B (BF16 and FP8)	M3 Max	Failed in Metal compiler / KGEN	family-level Apple support was documented; this exact target failed
Kimi-VL-A3B-Instruct	M3 Max	Failed at config (rope_scaling)	blocked above graph compilation
DeepSeek-V2-Lite-Chat	M3 Max	Punctuation-only output after local Apple patches; eight-token local observation ~4.03 tok/s	incomplete source-build smoke; numerical parity, serving, and distributed execution untested

The support ladder begins with a catalog entry, graph compilation, and focused kernel tests. A short generation smoke is the next rung. Numerical comparison, serving behavior, and a target-system run follow it. DeepSeek on the M3 stops at the short-generation rung, and the incomplete environment capture limits that record further.

The decode and MoE implementations are bring-up fallbacks that still need reference comparison and optimization. In the planner terms from Part 3, the smoke supports continued work on an Apple compute role. It leaves Apple ineligible for an authorized placement.

Next gate: a complete Apple stage across two hosts

The next gate has four required pieces of physical evidence:

1. Two physical hosts run the route, with their hardware and software builds captured.
2. The Apple host executes a complete layer stage through the Fornax Stage ABI.
3. The run captures packing, transfer, receiving, and exposed-wait timing at the stage boundary.
4. Boundary activations and final logits match an independent reference within stated tolerances.

This gate qualifies the physical Apple stage path. Serving behavior, longer contexts, batching, cancellation, recovery, and performance qualification remain later gates. Official packaged MAX support is also separate because the current result depends on a locally patched source build.

The current result shows that one bounded DeepSeek-V2-Lite configuration passes the previous compile and runtime gates in that source build. Reference parity and serving utility remain open. The final part examines why the M3 path needed different kernel machinery from the M5-targeted path in the source tree.

---

Previous: Fornax Architecture. Next: Porting an M5-Targeted Apple MLA Kernel to M3. Series index.

Sources: Qwen3-Omni-30B-A3B-Instruct model card; DeepSeek-V2-Lite-Chat model card; DeepSeek M3 Max runbook; Fornax repository; MAX supported models; MAX 26.4 changelog; MAX packages and hardware support.