Part 10 · Part 10: Serving and Inference, Where the Bill Arrives
Model-Making Masterclass, Part 10. Written Nov 17, 2026. Engineer voice: plain language, terms defined when they first matter, every load-bearing number tied to a tech report, model card, or flagged as unverified.
Training is a capital expense. Serving is rent.
Every earlier part of this study spends money once. Data curation, the pretraining run, the RL stage: each is a project with an end date. Serving is the opposite shape. From the day a model ships, every token it generates costs memory bandwidth, electricity, and hardware time, and the bill repeats for every user, every session, every agent step, forever. A model that is 10 percent cheaper to serve beats a model that is 10 percent better trained, surprisingly often, because the serving cost is the one that compounds.
That is why this part treats serving as an architecture and release discipline, not an afterthought. The decisions that set the serving bill were mostly made in Part 4 (where the parameters live, how big the KV cache grows, whether prediction heads exist) and in the release artifacts themselves (which precisions ship, which engines run them on day one). Serving is where those decisions come due. The families in this study differ here as sharply as they differ anywhere: one ships five precisions and three engines on launch day, another ships weights that only run on its author’s fork of an engine, and the gap between those two postures is worth more than most benchmark deltas in Part 9.
The serving cost equation
Inference has two phases with opposite economics, and confusing them is the root of most bad serving decisions.
Prefill processes the prompt. All prompt tokens are computed in parallel, the arithmetic is dense matrix multiplication, and the GPU’s compute units are the limit. Prefill is compute-bound. Its cost grows with prompt length, and its user-visible symptom is time to first token (TTFT).
Decode generates the answer, one token at a time. Each new token requires one forward pass, and at batch size one that pass is dominated not by arithmetic but by reading the model’s active weights from memory into the compute units. The arithmetic per token is trivial next to the data movement. Decode is memory-bandwidth-bound, and its user-visible symptom is time per output token (TPOT).
The decode roofline is the single most useful equation in serving:
Maximum decode speed (tokens per second, single stream) is approximately memory bandwidth divided by active weight bytes read per token.
Work it for the hardware in this study’s world. A DGX Spark (GB10, 128 GB unified memory, 273 GB/s of bandwidth) serving a dense 70B model at 4-bit reads about 35 GB of weights per token, so its single-stream ceiling is under 8 tokens per second; a third-party measurement of Llama 3.3 70B in AWQ form on one Spark recorded 6.0 tok/s, close to the roofline once overheads are counted. The same Spark serving a 30B-A3B MoE reads only the active experts, roughly 2 GB per token at 4-bit, and the ceiling jumps past 100 tok/s; the same measurement series recorded 80.9 tok/s for Qwen3-Coder-30B-A3B. Same box, same bandwidth, a 13x speed difference that is entirely explained by active bytes per token. This is Piece 0’s two-number rule (total parameters for the shelf, active parameters for the bill) expressed as a speed limit, and it is why Part 4’s body plans are serving decisions: a dense 30B and a 30B-A3B MoE are the same purchase and completely different rents.
Three consequences follow.
Batching is how you cheat the roofline, once. The roofline above is per stream. When 64 requests decode together, the weights are read once and amortized across 64 tokens, so aggregate throughput climbs until the workload becomes compute-bound and the FLOPs take over as the limit. This is why serving engines are built around continuous batching (vLLM’s signature design, now universal): requests join and leave the running batch at every step instead of waiting for a fixed group to finish, so the GPU never idles on a long generation while short ones queue. The economics invert with scale: at batch one you are buying bandwidth, at large batch you are buying compute, and the same GPU is a different product in the two regimes.
The KV cache is the second tenant, and it pays rent in the same currency. Weights are read; the key-value cache (Part 4) is read and written on every decode step, and it grows with context length and with the number of concurrent sessions. Section 3 prices it properly. For the cost equation, the point is that memory capacity and memory bandwidth are both consumed by two tenants, weights and cache, and the cache tenant’s bill scales with exactly the thing agent workloads maximize: long contexts held open for many steps.
MoE serving has a residency clause. The roofline reads only active experts per token, which is the good news. The fine print is that all experts must be resident in memory (or swapped in over a bus much slower than HBM), because the router can send any token to any expert. Total parameters set the capacity floor even though active parameters set the speed. A trillion-parameter MoE at INT4 is on the order of 500 to 600 GB of residency (parameter count times bytes per weight, plus scales and runtime overhead) before the first token moves, which is why the trillion class is a data-center product no matter how small its active size is. No family in this study occupies that class; the roster tops out at Nemotron 3 Ultra’s 550B.
Quantization: the precision ladder and its evidence
Quantization stores weights (and sometimes activations and the KV cache) in fewer bits, which cuts both terms of the serving bill at once: fewer bytes to store, fewer bytes to read per token. The ladder from the shipping default down:
| Precision | Bytes per weight | Who ships it in this study | Quality evidence |
|---|---|---|---|
| BF16 | 2 | Everyone, as the reference artifact | The baseline everything is measured against |
| FP8 | 1 | Nemotron 3.5 Transcribe; K2-Horizon FP8 builds; most vLLM deployments | Effectively lossless (see below) |
| INT8 | 1 | GGUF Q8_0 builds across the ecosystem | About 1 to 3 percent degradation |
| NVFP4 / FP4 | 0.5 | Nemotron 3 Super and Ultra (trained in it), Lightning NVFP4 builds | Small, measurable, published by NVIDIA itself |
| INT4 (QAT) | 0.5 | No rostered family ships a native INT4 QAT build; the prominent published cases sit in weights-only releases outside this study | The standard it sets (scores measured on the shipped artifact) is covered in the text below |
| INT4 (PTQ) | 0.5 | GGUF Q4_K_M and i-quants, AWQ and GPTQ builds everywhere | Competitive on average, task-dependent tails |
| 2-bit class | about 0.26 | Nemotron 3.5 Bolt 74B-A9B (IQ2_XXS experts) | Vendor card only; see the artifact rule below |
The evidence base for the top of the ladder is unusually good. The most comprehensive public study (Kurtic and coauthors, “Give Me BF16 or Give Me Death?”, over 500,000 evaluations across the full Llama-3.1 family) found FP8 weight-and-activation quantization effectively lossless at every model scale, INT8 weight-and-activation within 1 to 3 percent of the BF16 baseline, and INT4 weight-only quantization more competitive than its reputation, rivaling 8-bit results on many benchmarks. Two qualifications matter more than the averages. First, the losses are not uniform across tasks: multi-step mathematical reasoning degrades earliest and hardest under aggressive 4-bit post-training quantization, because one early wrong step cascades. Averages hide exactly the failures a cyber workload cannot afford, a theme Part 9 established for evals generally. Second, the study’s models were quantized after training. Quantization-aware training (QAT), where the model trains with the quantization in the loop, is a different and stronger claim. The most prominent published QAT case, a native INT4 release whose every published score was measured on the INT4 artifact itself, sits in a weights-only release outside this study’s scope, so it is not used as comparison evidence here; what this study takes from it is the standard, and the standard is artifact discipline: scores measured on the shipped artifact, with no higher-precision original hiding behind the numbers. Inside the roster, Nemotron’s NVFP4 line is the working example (next paragraph): trained in the format, published in both builds, and scored at the precision users actually run.
NVFP4 is the interesting middle case because NVIDIA publishes both sides of the comparison. Nemotron 3 Super and Ultra were pretrained in NVFP4 (Part 5), so their FP4 artifacts are closer to QAT than to afterthought compression. Even so, NVIDIA’s own surfaces show a price: the build.nvidia.com card for Nemotron 3 Ultra lists SWE-Bench Verified at 71.9 for the BF16 build and 69.7 for the NVFP4 build, a gap of about 2 points, while the Hugging Face BF16 card lists 70.7 for the same benchmark, a cross-surface inconsistency the roster flags. Read those three numbers together and you have the honest state of 4-bit serving in 2026: the cost is real, it is small, it is benchmark-dependent, and even the publisher’s own pages do not fully agree on it.
At the bottom of the ladder sits the 2-bit artifact, and Nemotron 3.5 Bolt 74B-A9B is its flagship: a 120B parent compressed by architecture search, retrained, and shipped with its experts stored in IQ2_XXS at 2.0625 bits per weight, producing a GGUF of about 22 GB and a marketing line of a 24 GB memory budget. The card’s benchmark table (vendor-reported) has the 2-bit Bolt beating BF16 builds of Qwen3.6-35B-A3B and Gemma 4 26B-A4B on Terminal-Bench 2.1 and BrowseComp, which, if it reproduces, is a genuinely strong result for the format. But Bolt also teaches the artifact rule twice over. First, the 24 GB figure is a weights budget, not a working configuration: NVIDIA’s own deployment table asks for a 32 GB GPU at 65K context and 48 GB at the full 262K context, because the KV cache and activations were never in the headline. Second, the quality evidence for the 2-bit build is the card and nothing else; there is no independent reproduction in the record this study could find. Bolt’s smaller sibling makes the complementary point about how the ladder is normally climbed: Bolt E4B ships as a Q4_K_M GGUF of about 4 to 6 GB, the llama.cpp ecosystem’s default-quality 4-bit build, aimed at RTX, Jetson, and Spark class devices. Q4_K_M is where the local ecosystem has collectively decided the quality-per-byte curve bends: 8-bit when you have room, Q4_K_M when you do not, and below 4 bits you are making a bet, not a default.
The rule that organizes all of this, and that Part 11 will enforce for LAB2: a benchmark score is a property of a specific artifact at a specific precision, not of a model name. Nemotron Ultra mostly honors it by publishing both builds and scoring the NVFP4 build at its own precision. Any release that benchmarks at BF16 and ships at 2 bits is asking you to grade a different product than the one in the box.
The KV cache: the long-context tax
The KV cache stores, for every token in the context, the key and value vectors every attention layer needs to attend back to it. Its size per token is fixed by architecture: 2 (keys and values) times attention layers times KV heads times head dimension times bytes per element. It is the only part of the serving bill that grows while a session runs, and it is paid per concurrent session, so it, not the weights, usually caps how many sessions a GPU can hold.
Published per-token figures make the spread concrete. DeepSeek-V3’s tech report puts its cache at about 70 KB per token, against 516 KB per token for Llama 3.1 405B in the same table: a 7.4x gap between two frontier models of the same era, decided entirely by attention design. At a 256K context, that is the difference between about 18 GB and about 135 GB of cache for one session. The mechanisms behind the gap are Part 4’s material; here they are priced:
- GQA is the settled baseline. Sharing KV heads across query-head groups cuts the cache by the grouping factor; 8 KV heads is the roster’s standard setting (K2-Horizon family-wide, Nemotron-H, Granite). Against full multi-head attention it is roughly an 8 to 16x cut at typical head counts, at a quality cost most labs now treat as noise.
- MLA compresses the cache itself. DeepSeek-V2’s report claims a 93.3 percent KV cache reduction against its own 67B multi-head baseline, with generation throughput up to 5.76x higher, and V3’s 70 KB per token is what that buys at production scale. The price, from Part 4, is paid elsewhere: the up-projection compute on every step, and design constraints like the inapplicability of QK-norm.
- MoVA spends nothing at cache time. K2-Horizon’s value-expert routing changes how each token’s value vector is computed, then caches only the combined vector, so its cache is exactly the size plain GQA would give. In a part full of trades, this is the rare free one: the capacity MoVA buys is paid in prefill projection compute and shelf parameters, not in the per-session tax.
- Sliding windows cap the tax. OLMo 3’s pattern (three of four layers attending within a 4,096-token window, every fourth layer global) means 75 percent of layers stop accumulating cache beyond the window. The cache grows with the global layers only, at the price Part 4 described: mid-distance retrieval rides on the global layers relaying it.
- Hybrid Mamba mostly abolishes the tax. In Nemotron 3 Nano, 46 of 52 layers are Mamba-2 or MoE layers that carry a fixed-size state instead of a growing cache; only 6 attention layers accumulate KV at all. That is the structural reason behind the family’s long-context serving claims (RULER 86.3 at 1M context, vendor-reported) and behind IBM’s enterprise claim for Granite 4.0 of over 70 percent lower serving memory on long-context multi-session workloads. For multi-session serving, which is the agent case, the hybrid’s cache advantage compounds with session count in a way a single-session benchmark never shows.
Two levers sit on top of the architecture. KV cache quantization (storing the cache in FP8 or lower instead of BF16) halves or quarters the tax for any attention design, at a small quality cost that grows with context length; it is a standard vLLM and SGLang option and the first thing to reach for when sessions do not fit. And prefix caching, covered in the engine section below, avoids recomputing the cache for shared prompt prefixes, which attacks the prefill side of the same bill.
The cautionary counterpoint is Maxwell V1 Flash, from this week’s intake. A reply on the launch post claimed its Block Sparse Attention design carries a KV cache about 4.5x Qwen3.6’s, which would bite exactly in long agent loops at its advertised 256K context. That figure is unverified (a reply claim, not a measurement this study reproduced), and it is flagged as such. But the shape of the risk is real and general: a model can win the parameter ledger and lose the cache ledger, and if the cache numbers are not published, you find out at deployment time, at 200K tokens into a session, on your hardware.
Speculative decoding: spending spare compute to buy time
Speculative decoding exploits the decode roofline. At low batch, decode is bandwidth-bound, which means the GPU’s arithmetic units are mostly idle while they wait on weight reads. A draft mechanism proposes several future tokens cheaply; the target model then verifies all of them in one parallel pass, which costs little more than a single-token pass because the weights were being read anyway. Accepted tokens are free progress. The two founding papers (Leviathan and coauthors, arXiv:2211.17192; Chen and coauthors, arXiv:2302.01318) proved the acceptance rule preserves the target model’s exact output distribution, and reported 2 to 3x speedups on T5-XXL and 2 to 2.5x on Chinchilla 70B respectively. Speculation is the rare serving trick that is provably lossless: with correct rejection sampling, the output is the output the target model would have produced alone, token for token under greedy decoding and distribution-identical under sampling.
Everything since has been a fight over the drafter: who proposes, at what cost, with what acceptance rate. Acceptance is the whole game. Expected tokens per target pass rises with acceptance and draft length, while every rejected token is wasted draft compute, so the optimum is a drafter that is cheap and uncannily aligned with the target on your actual traffic. Three drafter designs matter in this study.
Standalone draft models are the classic form: a small model from the same family drafts, the large model verifies. Simple, composable, and the alignment is only as good as the small model’s similarity to the large one. This is the form llama.cpp implements for local use, which matters for the Mac finding below.
Trained prediction heads (MTP) draft from inside the target. Part 4 covered multi-token prediction as an architecture choice; at serving time it is the best-aligned drafter available, because the heads were trained jointly with the model on the model’s own representations. Nemotron 3’s white paper ablation reports about 97 percent acceptance on the MTP module’s first two predicted tokens, a figure standalone drafters do not reach, and the shipping pattern has become a product line: Nemotron 3 Super and Ultra carry MTP layers, Super VL trained its MTP head as a distinct 100B-token stage, and Nemotron 3.5 Lightning ships dedicated speculative variants (NVFP4-DSpark and DFlash builds) alongside its plain builds. When a lab ships the speculative build as a named artifact, speculation has stopped being a trick and become part of the release. K2-Horizon’s answer is the mirror image: no MTP in the base architecture, and Uno, LoRA-style diffusion adapters fitted after training that emit blocks of tokens per pass for up to 2.2x decoding speedup, keeping the base untouched and the drafter optional.
Parallel drafters attack the draft cost itself. A conventional drafter is still autoregressive, so drafting 8 tokens costs 8 sequential small passes. DSpine (Tencent, arXiv 2609.36173 per the intake brief) restructures the drafter so all positions in a 16-token block draft in parallel, with the causal chain carried through network depth instead of through time. Its published figures (vendor-reported in the brief’s sources) are an acceptance length of 4.77 against 4.00 for DFlash2 on SWE-bench Pro replay, and 2.71x to 3.84x throughput in agent tests. Nemotron’s diffusion lines explore the same territory from the model side: the Labs Sigma 8B card reports 112 tokens per second in self-speculation mode on a DGX Spark against 41.8 autoregressive, via secondary quotation of the card, and flags its own benchmark figures as provisional.
Two limits discipline all of this. First, speculation spends the spare compute that low batch leaves idle; at high batch, decode is already compute-bound, verification stops being free, and gains shrink or invert. Speculation is a latency tool for interactive and agent workloads, not a throughput tool for saturated batch serving, and engines increasingly enable it adaptively for exactly this reason. Second, the drafter must live inside the target’s execution path, and the Mac evidence in the DSpine intake brief is the cleanest demonstration in this study. DSpine itself has no Mac path at all: its reference implementation sits inside SGLang with fused CUDA kernels. A community llama.cpp port exists, but llama.cpp draft-model speculation on Metal is a documented net loss: issue #23752 measured an M1 Max (the exact machine class in Brandon’s estate) running Qwen3.5-9B with every speculative configuration 11 to 24 percent below the plain baseline, because each draft-verify round launches separate Metal kernels and the dispatch overhead exceeds the decode time saved. What works on the same hardware is the model’s own MTP head through MLX: MTPLX reports 1.6x to 2.6x decode gains on Qwen 3.6 27B with exact rejection sampling, and Ollama’s MLX runner enables MTP speculation automatically on Apple Silicon. Same algorithm family, opposite outcomes, decided entirely by whether drafting and verification share one kernel path. On NVIDIA serving stacks that integration is given; on Metal it is the whole ballgame.
Engines: the layer that decides what actually runs
An engine is the runtime that loads the artifact, manages the KV cache, schedules the batch, and executes the kernels. Four open engines cover nearly everything in this study, and they differ less in peak speed than in what they make easy.
vLLM is the default. Its founding contributions (PagedAttention, from Kwon and coauthors, 2023) were memory management: prior systems wasted 60 to 80 percent of KV cache memory to fragmentation and over-reservation, and paging the cache like virtual memory cut the waste to under 4 percent, worth 2 to 4x throughput at the same memory. Continuous batching and the broadest day-one model support in the ecosystem followed. When a lab says a model is supported on vLLM, it means the architecture is implemented, the configs are tested, and the quantization formats are wired. K2-Horizon’s release discipline is the benchmark here: day-zero support across vLLM, SGLang, and Ollama, plus official GGUF and MLX collections, is listed in its roster entry as a shipped feature, not a roadmap item.
SGLang is the structured-workload engine. Its signature is RadixAttention: the KV cache is organized as a radix tree keyed by token sequence, so any requests sharing a prefix (a system prompt, a tool schema block, a conversation history) reuse the same cached entries instead of recomputing them, with cache-aware scheduling that batches requests by shared branches. The SGLang paper reports up to 6.4x throughput on prefix-heavy workloads such as multi-turn chat and tree-structured reasoning. For agent traffic, which is nothing but shared prefixes and branching continuations, this is not an edge case; it is the main case, and the harness section below returns to it.
TensorRT-LLM is NVIDIA’s compiled engine: models are built into optimized execution graphs per GPU architecture, which buys peak throughput and first-class support for NVIDIA’s newest formats (NVFP4 serving on Blackwell is the current example) at the cost of a compilation step, NVIDIA-only hardware, and a release cadence tied to NVIDIA’s. For an all-NVIDIA sovereign deployment it is a legitimate choice; for a mixed estate it is a commitment.
llama.cpp and the MLX stack are the local engines. llama.cpp (GGUF format, CPU and Metal backends) is how the 7B to 32B bands actually reach individual machines, and its quant ecosystem (Q4_K_M as the default, i-quants below it) is the distribution channel for the local tier. On Apple Silicon, MLX-native routes (Ollama’s MLX runner, LM Studio’s MLX backend, MTPLX) beat llama.cpp exactly where kernel integration matters, as the speculation evidence above shows. The honest summary: on NVIDIA hardware the engine choice is about scheduling features; on Apple hardware it is about which runtime owns the kernels.
Against that baseline, the custom-engine risk stands out in full. Maxwell V1 Flash runs only on its author’s custom vLLM build, with current NVIDIA drivers, on a README hardware line of one B300 or two H100s. There is no GGUF, no llama.cpp path, and the launch thread’s “GGUF when?” question went unanswered. Price what that means. The model cannot be independently benchmarked without adopting the author’s stack, which compounds its already unverified benchmark claims (Part 9’s problem, arriving through the serving door). It cannot be served by any engine your team already operates. Its bus factor is one repository with one commit. And its claimed deployment shape contradicts its own pitch: a 73.6 GB BF16 artifact that fits one DGX Spark’s memory is nevertheless specified against data-center GPUs, because the custom build is where the author’s testing lived. A model that only runs on its author’s fork has not been released in any operational sense; it has been demonstrated. The milder version of the same coupling is DSpine living inside SGLang’s CUDA path: a real technique, in a real engine, that still does not exist for anyone outside that engine and that silicon. Engine support is a rung of openness in everything but name, and Part 1’s ladder should be read with a serving footnote: weights you cannot run on a maintained engine are a rung-1 artifact with extra steps.
Harness throughput: the workload that actually pays
Peak throughput numbers are measured on shapes chosen by the publisher: fixed input and output lengths, steady concurrency, no tools, no waiting. Nemotron 3 Nano’s paper claims up to 6x the throughput of Qwen3-8B in reasoning settings; Nemotron 3 Super claims up to 2.2x GPT-OSS-120B and 7.5x Qwen3.5-122B at 1M context; Nemotron-H claims up to 3x inference speed at matched accuracy. These are vendor-reported, shape-specific, and not necessarily wrong. They are also not the workload an agent harness generates, and the difference is structural, not a matter of tuning.
An agent harness (the loop that runs a model through tool calls: read a repo, run a test, edit a file, check the result) produces traffic with four properties no peak benchmark has:
- Many short calls, not one long stream. Each step is a fresh request: a growing conversation prefix, a short new instruction, a bounded generation, then the model stops and waits for a tool.
- The GPU idles on tool round trips. While the harness executes a shell command or a web fetch, the model generates nothing. Wall-clock task time is model time plus tool time plus scheduling gaps, and the model’s share is often the minority.
- Prefixes are enormous and nearly identical. The system prompt, tool schemas (thousands of tokens for a rich tool set), and the accumulated conversation are resent or re-referenced at every step. Prefill, the phase peak benchmarks minimize, is the harness’s dominant compute unless the cache absorbs it.
- Concurrency is bursty and session-shaped. A fleet of agents working in parallel holds many long contexts open at once, which is the KV cache capacity problem from Section 3 wearing a scheduling costume.
Under that workload, the numbers that decide cost per task are TTFT under prefix load, prefix cache hit rate, cache capacity per session, and scheduler behavior at bursty concurrency. Peak decode tok/s barely makes the list. This is why RadixAttention and vLLM’s prefix caching matter more to harness economics than any decode kernel: when 95 percent of every request is a prefix the engine has already computed, the harness’s prefill bill collapses to the new suffix. It is also why the hybrid bodies from Part 4 overperform their benchmark tables in harness service: a session that holds 200K tokens of context open through fifty tool calls pays the cache tax fifty times in attention models and a fraction of it in a hybrid.
Nemotron 3.5 Lightning is the first release in this study designed explicitly for this workload, and its card is refreshingly honest about the trade. In its own benchmark table, Lightning posts SWE-bench Verified 51.56 against 70.12 for Qwen3.6-35B-A3B: it loses the capability comparison in its own launch materials, and sells throughput and cost per task instead. The supporting cast agrees with the positioning: the model ships in BF16, NVFP4, and two speculative NVFP4 variants, was launched alongside NeMo Switchyard, an open-source router for splitting traffic across models, and its one reported downstream use in the roster is CodeRabbit post-training it for code-review routing (80.7 percent routing accuracy on about 10K examples, under 100 dollars of training cost, as reported). That use case is the harness economy in miniature. A routing decision is a short, cheap, high-volume call where the model needs to be adequate and fast, not brilliant; the expensive model is spent only where the router sends the hard cases. The metric that falls out of all of this, and the one Part 11 will build Proof of Defense around, is cost per resolved task: total serving dollars divided by tasks that actually completed. Peak throughput is an input to that number. It is never the number itself.
The cyber read
Serving is where sovereignty stops being a license position and becomes an operational fact. A cyber model that phones an external API is not sovereign regardless of its weights license, so the Counterfleet and LAB2 serving stack has to satisfy four constraints that commercial serving treats as optional.
Everything runs on owned hardware, offline-capable. The engine layer cooperates: vLLM, SGLang, and llama.cpp are all self-hostable open source, models load from local files, and nothing in the serving path needs a vendor cloud. The discipline is in the update path. Model updates should arrive the way Orange’s threat-intel updates do in the fleet design: as versioned, hash-pinned artifacts on a schedule the operator controls, staged on one machine, evaluated, then rolled out, with the previous artifact kept for instant rollback. An air-gapped deployment just makes the transport physical (signed artifacts on media or through an internal mirror) instead of conceptual. Any component that cannot work this way, including a custom engine fork that must be rebuilt from an author’s repo, is a supply-chain decision, and Part 1’s license discipline extends to it naturally: read the provenance before you benchmark the artifact.
Size the fleet by cache and bandwidth, not by parameter count. From Part 4’s bands, the on-prem map onto hardware Brandon’s estate actually has or targets:
| Hardware class | What fits and serves well | The binding limit |
|---|---|---|
| DGX Spark class (128 GB unified, 273 GB/s) | 30B-A3B band at BF16 or NVFP4 (about 60 GB in BF16); K2-Horizon 32B dense in GGUF; Bolt 74B-A9B 2-bit (22 GB weights, but budget 48 GB for full context per NVIDIA’s own table) | Bandwidth: dense models crawl (single-digit tok/s at 70B), MoE active size is what makes this box sing (about 81 tok/s measured on a 30B-A3B) |
| Workstation card, 24 to 32 GB | Bolt E4B (Q4_K_M, 4 to 6 GB), the 7B to 9B band at Q4 or Q8, K2-Horizon 36B community GGUF builds down to about 12 GB | Capacity first, then the same bandwidth rule |
| Apple Silicon, M1 Max 64 GB and up | 27B to 32B class at Q4 through MLX routes, MTP speculation on | Kernel path: use MLX-native engines; llama.cpp draft speculation is a measured net loss on Metal |
| Multi-GPU node (8xH200 class) | K2-Horizon 375B (validated on exactly this shape, tensor parallel 8 with expert parallel, in both vLLM and SGLang per its card); Nemotron 3 Ultra at NVFP4 | Residency and interconnect; the trillion class would live at this tier, but no family in this study occupies it |
Agent fleets multiply the cache problem, and architecture is the answer. Five fleets of agents holding long incident contexts open concurrently is a KV capacity planning exercise before it is anything else. The hybrid and MLA designs from Part 4 (a cache tax paid on 6 of 52 layers, or a 7.4x smaller cache per token) translate directly into how many simultaneous investigations one node can hold. Prefix caching is the second multiplier: every agent in a fleet shares the system prompt, the tool schemas, and usually the threat-intel context, which is precisely the shared-prefix shape RadixAttention exists for. Choose the engine for the fleet’s traffic shape, not for its leaderboard shape.
The shipped artifact is the evaluated artifact. Part 9’s rule lands hardest here. Proof of Defense numbers for a LAB2 model must be measured on the exact build that ships: same precision, same engine, same speculative configuration. A cyber model evaluated at BF16 and deployed at NVFP4 has unmeasured refusal behavior and unmeasured detection quality at exactly the settings its users will run, and the measure-at-shipped-precision discipline is the standard to copy. The corollary for updates: every artifact swap, even a requantization of the same weights, is a new model for evaluation purposes until the eval suite says otherwise. Silent swaps invalidate results; that was BB Arena’s rule for attempts, and it is the right rule for a serving fleet too.
Flagged and unverified items
Per this study’s discipline, what this part could not pin at a primary source stays flagged:
- Vendor-reported throughput multipliers. Nemotron 3 Nano’s up to 6x over Qwen3-8B (paper claim), Nemotron 3 Super’s 2.2x and 7.5x at 1M context (tech report), Nemotron-H’s up to 3x (paper), and SGLang’s up to 6.4x on prefix-heavy workloads (SGLang paper) are publisher figures on publisher-chosen shapes. Directionally consistent with the mechanisms, not independently reproduced in this pass.
- DGX Spark measurements. The 6.0 tok/s (Llama 3.3 70B AWQ) and 80.9 tok/s (Qwen3-Coder 30B-A3B) figures are third-party measurements (GPUwerk, via the Spark bandwidth coverage), not NVIDIA’s and not this study’s. The bandwidth ceiling arithmetic that predicts them is exact; the measurements are illustrative.
- Maxwell’s KV cache claim. The about 4.5x figure comes from a reply on the launch post, quoted in the intake brief. Unverified, no methodology, treated here as a risk shape, not a number.
- DSpine figures. Acceptance length 4.77 and the 2.71x to 3.84x throughput range are vendor-reported from the paper and cards as captured in the intake brief; the llama.cpp port’s numbers are one tester’s RTX 3090 results. The Metal net-loss finding (llama.cpp issue #23752) is a measured result on one model and machine class; it is the best Mac evidence available and it is still a single issue report.
- Sigma self-speculation on Spark. The 112 versus 41.8 tok/s figures reach this study through secondary quotation of the model card, and the card itself marks its benchmark figures provisional. Quoted here as an indicator, not a result.
- Kurtic quantization study scope. The FP8-lossless and INT4 findings are measured on the Llama-3.1 family across academic benchmarks and real-world tasks. They are the best public evidence available and they are still one family’s numbers; MoE and hybrid architectures may quantize differently, and nobody in this roster has published the equivalent 500K-evaluation study for them.
- Nemotron 3 Ultra’s cross-surface scores. SWE-Bench Verified appears as 70.7 (HF BF16 card), 71.9 (build.nvidia.com BF16), and 69.7 (build.nvidia.com NVFP4). The NVFP4 cost estimate of about 2 points in this part is computed from the publisher’s own inconsistent surfaces and should be treated as approximate until one surface is designated canonical.
Sources for this part
Primary: DeepSeek-V2 tech report (arXiv:2405.04434) for the MLA cache reduction and generation throughput; DeepSeek-V3 tech report (arXiv:2412.19437) for the 70 KB versus 516 KB per-token cache table; Leviathan and coauthors, “Fast Inference from Transformers via Speculative Decoding” (arXiv:2211.17192); Chen and coauthors, “Accelerating Large Language Model Decoding with Speculative Sampling” (arXiv:2302.01318); Kwon and coauthors, “Efficient Memory Management for Large Language Model Serving with PagedAttention” (2023) for the fragmentation and throughput figures; Zheng and coauthors, the SGLang paper (arXiv:2312.07104) for RadixAttention and the 6.4x prefix-heavy figure; Kurtic and coauthors, “Give Me BF16 or Give Me Death? Accuracy-Performance Trade-Offs in LLM Quantization” (arXiv:2411.15100) for the FP8, INT8, and INT4 evidence; Nemotron 3 white paper (arXiv:2512.20856) and the Nemotron 3 Super tech report for MTP acceptance and throughput claims. Roster files: research/nemotron-roster.md (Lightning’s positioning, variants, and CodeRabbit report; Bolt and Bolt E4B artifact details and the 24 GB caveat; Ultra’s cross-surface scores; Sigma’s Spark figures), research/k2-horizon-roster.md (day-zero engine support, published quant builds, 375B validated serving shape), research/open-families-roster.md. Intake briefs: x-threads/2026-11-17-teksedge-maxwell-v1-flash.md (custom vLLM requirement, KV claim, missing GGUF) and x-threads/2026-11-08-dspine-mac-check.md (DSpine design and figures, llama.cpp issue #23752, MTPLX and Ollama MLX findings). Hardware figures: NVIDIA DGX Spark specifications (128 GB unified memory, 273 GB/s) with third-party measurements flagged in place.