Part 1 of the series. Fornax’s physical backends are being built on MAX, and MAX exists because of a
specific, expensive problem: making models run across several hardware targets.
This part is a tour of what a new accelerator vendor has to build,
from the driver up to torch.compile, why that job is different for a training
chip than an inference chip, and why taking on all of it without owning any silicon
is the position MAX has put itself in.
This part is the compressed view. The separate series Building Software Support for a New AI Accelerator follows the complete path through hardware architecture, packaging, drivers, runtime and memory, PyTorch tensor semantics, kernel libraries, graph compilation, numerical validation, distributed execution, domain engines, product qualification, and the integration boundaries used by JAX, MAX, CUDA, ROCm, and Intel XPU. This part keeps only the context needed for the Fornax design.
The scope of PyTorch accelerator support
When a chip company says its new accelerator “supports PyTorch,” that phrase hides
a great deal. A researcher’s mental model of PyTorch support is simple:
model.to(device), then the same training or inference script that ran on CUDA runs
here. Delivering that experience is a large software project because PyTorch exposes
a broad API. Its current accelerator documentation counts more than 3,500 built-in operators
including variants, alongside a dispatcher, an autograd engine, a compiler, a
distributed library, and a decade of accumulated behavior that models quietly
depend on.
CUDA has accumulated nearly two decades of work across these layers. Alongside
the GPUs, NVIDIA built
cuBLAS, cuDNN, CUTLASS, cuFFT, NCCL, Thrust, and the ecosystem of libraries that
PyTorch calls into, plus the compiler toolchain (nvcc, ptxas) that turns kernels
into machine code the hardware runs. A new
vendor has to reconstruct functional equivalents of that entire stack, validate it
against real models, and keep it working as PyTorch and the models both move.
Fornax depends on MAX covering these layers.
The stack, from the silicon up
Layer 1: driver, runtime, and memory
At the bottom is the part every vendor has to build regardless of framework: a kernel driver, a user-space runtime, a memory allocator, and a stream or queue model for launching asynchronous work and synchronizing on it. This is the equivalent of the CUDA driver and runtime API. Models never see it directly, but everything above it assumes it exists, is stable, and does not leak or fragment memory under the allocation churn that a large model’s forward and backward passes produce. These are baseline requirements, yet years of real workloads are often needed to expose leaks, fragmentation, synchronization failures, and reset behavior.
Layer 2: the dispatcher backend
PyTorch routes every operator call through a dispatcher. The dispatcher looks at
the tensors involved, in particular their device and dtype, computes a dispatch
key, and sends the call to the kernel registered for that key. CPU, CUDA, and MPS
(Apple’s Metal path) are all dispatch keys. For a brand-new backend, PyTorch’s
recommended entry point is a generic key called PrivateUse1.
The modern path looks roughly like this. You call
torch.utils.rename_privateuse1_backend("npu") so tensors can live on your device
by name, register your kernels against the PrivateUse1 dispatch key with the
TORCH_LIBRARY_IMPL macro, and then supply the surrounding machinery the runtime
expects: a device guard, a random-number generator, autocast support through
AutocastPrivateUse1 for mixed precision, and serialization hooks so saved tensors
know how to round-trip through your device. Huawei’s torch_npu follows this path:
a PrivateUse1 plugin that adapts Ascend hardware to PyTorch without forking
PyTorch itself.
The registration step is small. The multiyear work lies in building correct kernels for the operator shapes real models exercise.
Layer 3: the kernels
The kernels are the cost center. A given model exercises only a subset of PyTorch’s operator surface, but that subset is large and unforgiving. You need dense matmul and batched matmul (the cuBLAS equivalent), attention including fused paths for long-context and decode shapes, convolution and pooling for vision models, layer norm and RMS norm, softmax and log-softmax, activation functions, RoPE and other positional ops, elementwise math across every dtype, reductions, sorting and top-k, gather and scatter and the whole indexing family, and quantized matmul for every low-precision format your hardware claims to support.
Each kernel has to be correct across shapes and dtypes, then fast enough to use the hardware well. Performance requires tiling for the memory hierarchy, exploiting the tensor or matrix units, managing shared or local memory and register pressure, and reaching useful occupancy. NVIDIA encodes decades of this in cuDNN and CUTLASS. AMD reconstructs it in rocBLAS, MIOpen, and Composable Kernel. Every new vendor pays this cost again for its own instruction set, and the attention kernels alone are a research program. Later parts encounter this cost directly in Apple attention kernels.
Layer 4: graph capture and the compiler
Eager execution often leaves the hardware idle between thousands of small
launches. torch.compile adds graph capture and fusion through a separate
backend integration.
torch.compile has two halves. TorchDynamo traces Python execution into an FX
graph, a clean intermediate representation of the model’s computation. Then a
backend compiles that graph. The backend API is compact: a function takes an FX
GraphModule and example inputs and returns a compiled callable equivalent to
the graph. Satisfying that API requires graph lowering, fusion, layout selection,
scheduling, and code generation. The default backend is Inductor, which
generates fused Triton kernels for GPU and C++ for CPU. A new vendor either teaches
Inductor to target its hardware, or writes its own Dynamo backend, or bridges the
graph out to a separate compiler through ONNX, StableHLO, or an MLIR dialect, and
then does fusion, layout selection, and code generation there. This is the work a
graph compiler does: turning a sequence of ops into a schedule of fused kernels that
keep the hardware busy. It can account for a large part of the performance gap
between a mature stack and a new one, alongside kernels, libraries, allocation,
transfers, runtime scheduling, and collectives.
Layer 5: distribution
Many training workloads and large-model inference configurations span devices, so PyTorch’s distributed features assume a collective communication library underneath. NVIDIA has NCCL; AMD has RCCL; Huawei has HCCL. All-reduce, all-gather, reduce-scatter, broadcast, and their performance over whatever interconnect the hardware uses (NVLink, Infinity Fabric, or the vendor’s own fabric) have to exist and be registered as a process-group backend before tensor-parallel or data-parallel training and multi-device inference work at all. This is its own specialized library with its own topology-aware algorithms, and it is easy to underestimate until a model needs it.
Layer 6: numerics and the long tail
Numerical parity and the operator long tail sit above those layers and often dominate production qualification. Reduced-precision accumulation, the exact rounding behavior of softmax and normalization, the dtype coverage across bf16, fp16, fp8, and the newer fp4 and mxfp4 formats, and the endless tail of model-specific operations that break only when you run the actual checkpoint. A model that compiles and runs but produces subtly wrong logits is arguably worse than one that fails loudly, because you ship it before you notice. Closing this tail is a central part of production qualification, measured one model at a time.
An operator name is a poor unit of support. “We support gather” says
nothing until you name the rank, shape, dtype, layout, and tolerance, because a
backend can pass every matmul test and still fail a real model at a rank-specific
gather or a dynamic-shape lowering, as the Apple work later in this series
demonstrates. Validation has to follow the workload, from single operators up through
attention and expert subgraphs to full-model logits against a reference, and any
fallback has to be visible in the result. A silent CPU fallback can preserve
correctness while hiding that the device never ran the kernel at all, which quietly
invalidates every performance number attached to it.
Layer 7: tools, packaging, and compatibility
Backend usability also depends on a profiler that attributes time to host work, transfers, and device kernels; diagnostics for failed compilation and bad numerics; deterministic compilation caches; driver and runtime compatibility rules; installable packages or containers; and a continuous test matrix across framework, driver, compiler, hardware, dtype, and model revisions.
This layer decides whether someone outside the hardware team can reproduce a result, diagnose a regression, and upgrade without losing a working configuration. A framework reaches users through installation, error messages, profiling, documentation, and compatibility policy as much as through dispatcher registration.
How training and inference stacks differ
Training and inference accelerators require different portions of this stack.
A training chip has to do everything above, plus autograd. That means correct backward kernels for every forward op, high-precision accumulation so gradients do not drift, optimizer state and its memory, gradient-reduction collectives at scale, and numerical stability across long runs where small errors compound over millions of steps. Training is where the software stack is deepest and least forgiving. Gaudi 2 and 3, AMD’s MI300 through MI355X, and Huawei’s Ascend 910B and 910C are all built to carry that full weight.
An inference chip can cut a lot of it away. There is no backward pass, so no autograd, no optimizer state, no training-scale gradient collectives, and precision can be pushed lower because only forward-pass output quality has to be preserved. Long-run gradient stability is outside the workload. What matters is low-precision matmul throughput, latency, memory capacity for weights and KV cache, and cost per token. This is a substantially smaller software surface, which is why inference-only silicon keeps appearing.
The inference surface still includes prefill and decode execution, KV paging and eviction, continuous batching and prefix caching, quantization with low dispatch overhead, model loading and compile-cache behavior, and tokenizer and chat-template correctness. Fornax targets inference, so training parity is outside the project requirements, and the per-node surface it needs from MAX is the inference one above. That narrows the job without making any individual model shape or dtype easier: each supported one still has to pass its own tests.
The clearest illustration is Habana’s Goya, an inference-only chip from around 2019, one of the earliest dedicated inference accelerators to post strong ResNet-50 numbers. It is effectively gone from the market. After Intel acquired Habana it consolidated around the Gaudi line, which does both training and inference, and folded pure data-center inference toward Xeon and Gaudi. Goya is out of the market now. Its narrower software stack did not compensate for the ecosystem and roadmap discontinuity.
Qualcomm’s current inference roadmap emphasizes memory capacity and bandwidth. Dragonfly AI200 carries 768 GB of LPDDR per card. Dragonfly AI250 adds the first generation of Qualcomm High Bandwidth Compute, and Dragonfly AI300 is planned with the second generation. Qualcomm’s published software surface includes the AI Inference Suite and Efficient Transformers Library for model onboarding and deployment. These accelerators still require the inference portions of the stack: kernels, compiler and framework integration, runtime behavior, and numerical validation.
How current accelerator stacks compare
The same stack requirements appear in each effort. The table reflects public information available in July 2026.
| Vendor | Silicon | Software stack | PyTorch path | Where it stands (mid-2026) |
|---|---|---|---|---|
| NVIDIA | H100, B200 | CUDA, cuDNN, cuBLAS, CUTLASS, NCCL | native, the reference | the stack everyone else is measured against |
| AMD | MI300, MI325, MI350, MI355X | ROCm 7, HIP, Composable Kernel, rocBLAS, MIOpen, RCCL | native ROCm dispatch | closest challenger; vLLM and SGLang are first-class on ROCm; CDNA4 adds native FP4 |
| Intel | Gaudi 2, Gaudi 3 | Intel Gaudi Software Suite / SynapseAI, gaudi-pytorch-bridge |
bridge to SynapseAI | roadmap reset: the open-source SynapseAI Core repository was archived, and Falcon Shores shifted to an internal test chip |
| Huawei | Ascend 910B, 910C | CANN, HCCL | torch_npu PrivateUse1 plugin |
active stack centered on Ascend hardware and the CANN runtime |
| Qualcomm | Dragonfly AI200, AI250, AI300 | AI Inference Suite, Efficient Transformers Library, hyperscaler software stack | inference-only data-center path | annual accelerator roadmap; AI200 emphasizes capacity, while AI250 and AI300 add successive HBC generations |
AMD has invested heavily and narrowed the gap while ROCm integration in serving engines continues to mature. Intel reset its accelerator roadmap despite substantial resources: Falcon Shores did not become a commercial product, and the open-source SynapseAI Core repository was archived. Silicon is only one part of the difficulty. The software path from the driver to model frameworks is a long-lived product commitment of its own.
The MAX approach
Each vendor in the table builds a software stack for the instruction set it owns. NVIDIA builds CUDA for NVIDIA silicon. AMD builds ROCm for AMD silicon. The software is expensive because it is specific to the hardware, and the hardware business pays for it.
MAX sets out to share the graph compiler, serving runtime, kernel language, and as much kernel source as possible across NVIDIA, AMD, and Apple GPUs. The common compiler can reduce duplication, but each target still needs kernels, runtime integration, packaging, compatibility work, and numerical validation. Its success depends on real-model operator coverage and performance on every target, beyond source-level portability.
MAX is the substrate selected for Fornax’s physical backends. The next part examines its operator support on each platform and follows the lowering path into PTX, AMDGPU code, and Metal shaders.
Previous: Objective and Constraints. Next: MAX Platform Assessment. Series index.
Sources: PyTorch accelerator integration; PyTorch operator registration; PyTorch PrivateUse1 backend integration; PyTorch custom compiler backends; Intel’s Falcon Shores earnings-call comments; Ascend torch_npu; Qualcomm Dragonfly data-center roadmap; Qualcomm AI200 and AI250 software stack; AMD ROCm 7; ROCm and vLLM.