Model Making, in the Open: A Masterclass

Part 13 · 13. Nemotron Generations

Part 13 · Part 13: The Nemotron Generations

Listen: audio twin
Timeline of seven Nemotron generations, with architecture and license annotated per step
Seven Nemotron generations: the license marches toward OpenMDW-1.1 while the architecture settles into the hybrid Mamba-MoE.

Model-Making Masterclass, Part 13. Written Nov 18, 2026. Engineer voice: plain language, terms defined when they first matter, every load-bearing number tied to a tech report, model card, or flagged as unverified. This part is built from a dedicated source pack compiled Nov 18, 2026 from primary sources (Hugging Face cards and configs, NVIDIA technical reports, arXiv papers) and the study’s Nemotron roster. Every benchmark figure is a vendor claim unless it is explicitly labeled independent. Where two NVIDIA surfaces disagree, both figures are printed with their surfaces named, and the disagreement is flagged rather than resolved.

Why one family gets its own part

The other parts line the families up stage by stage. This one runs the film forward inside a single family, because Nemotron is the rare case where one lab has published seven distinguishable generations of open model making in three years, and each generation changed its mind about something load-bearing: what the license allows, what the layers are made of, where the quality comes from, and what a release even consists of. Read across the generations and you can watch the field’s center of gravity move: from weights, to data, to post-training, to the shape of inference itself.

The discipline is the same at every step. What changed versus the previous generation: architecture, data, training method, context, license. What improved, in the generation’s own numbers, with the harness and conditions named where the sources name them. And attribution: what the generation’s own report or card credits for the improvement, or a plain statement that no attribution was published. Vendor claims are labeled as such. Independent numbers are rare enough in this lineage that each one is marked when it appears.

Step 1: Nemotron-3 8B (2023), the pre-open era

What changed. There is no predecessor, so this step is the baseline everything else departs from. A dense decoder-only Transformer in the GPT pattern, built for NVIDIA’s NeMo framework: 8B parameters, 4,096-token context. The alignment method was SteerLM, which trains the model to respond to attribute knobs at inference time (quality, helpfulness, toxicity and similar), so one checkpoint can be steered along several axes without retraining. Checkpoints shipped as Base, Chat (with SFT, SteerLM, and RLHF variants), and QA, all at 4K context.

Data and license. The QA card states the base was trained on about 3.5 trillion tokens; a secondary compilation says about 3.8 trillion across 53 natural languages and 37 programming languages, and no tech report exists to settle which figure is right. No standalone technical report was published at all. The license was the NVIDIA AI Foundation Models Community License Agreement: a community license with conditions, not an OSI open license. Calling this generation “open” in the sense the later generations earn would be generous; it was downloadable under terms, which in 2023 was already more than most labs offered.

What improved. Nothing measurable is claimed against a predecessor, because there is none. The NGC cards carry no benchmark table. The positioning was enterprise chat and question answering, and the honest summary of this generation’s evidence base is that there barely is one: no report, no scores, a token count that two sources state differently.

Attribution. None published in the sources read. There is no report to credit anything, and this part does not invent one. Step 1’s real contribution to the generations story is negative space: it shows what a model release looked like before the lab decided that the training record was part of the product.

Step 2: Nemotron-4 340B (June 2024), the synthetic-data flagship

What changed. Scale first: 340B dense parameters (331.6B non-embedding), still a dense Transformer, still 4,096-token context, trained on 9 trillion tokens. But the architecture is not the story of this generation; the alignment data is. Over 98 percent of the alignment data was synthetically generated, per the technical report (arXiv:2406.11704), produced with the family’s own Instruct and Reward models. The synthetic data generation pipeline was open-sourced, and the HelpSteer2 preference dataset was released under CC-BY-4.0. The models were sized deliberately to fit a single DGX H100 node (8 GPUs) in FP8. The license moved to the NVIDIA Open Model License Agreement, a step toward real openness from the 2023 community license, though still a custom NVIDIA license rather than an OSI one.

What improved. The release was positioned as a synthetic-data generation stack as much as a model. The Reward checkpoint was reported number 1 on RewardBench at release; two figures circulate in secondary sources, 92.0 and 94.1, and the report’s body table was not re-read in the source pass behind this part, so the exact figure stands as a flagged gap rather than a number this part will assert. No like-for-like benchmark delta against the 2023 8B exists in the sources read, and none should be expected: the two models are different scale classes built for different jobs, and comparing their scores would measure the size difference, not the generation difference.

Attribution. The report itself is explicit, and the credit goes to the pipeline, not the parameters: the headline contribution is that a model family can generate its own alignment data at quality sufficient to top RewardBench, with the pipeline released so others can do the same. The FP8 single-node fit is presented as a design constraint met, not a benchmark gain. This is the first generation where the lab’s claim is about a method outsiders can reuse, and that shift, from artifact to method, is the thread the rest of the lineage pulls on.

Step 3: Llama-Nemotron (2024 to 2025), the derivative era

What changed. This generation builds on someone else’s base, and says so in the name. The 2024 wave was pure post-training on Meta’s Llama: Llama-3.1-Nemotron-70B-Instruct and 70B-Reward (Oct 15, 2024) applied RLHF with REINFORCE on HelpSteer2 preferences to Llama 3.1 70B, with no pretraining of NVIDIA’s own involved. The 2025 reasoning family changed the method completely. Neural architecture search over Llama 3 backbones under NVIDIA’s Puzzle framework (FFN fusion, attention skipping, variable FFN compression ratios) produced compressed architectures, which were then recovered by knowledge distillation and continued pretraining, and finished with SFT and large-scale RL, with reward-aware preference optimization between SFT and RL. Three sizes: Nano 8B (from Llama 3.1 8B), Super 49B (from Llama 3.3 70B), Ultra 253B (from Llama 3.1 405B), all dense, all at 128K context, a 32x context jump over the 2024 flagship. These were the first open models with a dynamic reasoning on and off toggle, switched by the system prompt, so test-time compute became a runtime choice instead of a checkpoint choice. The complete post-training dataset and the codebases (NeMo, NeMo-Aligner, Megatron-LM) were released; NVIDIA’s Megatron Bridge documentation states that training recipes for this family are not available, so the openness is weights plus data, not recipes. The license stacked accordingly: the NVIDIA Open Model License Agreement with the Llama Community License on top, confirmed verbatim on the Super v1.5 card. A derivative’s openness is the minimum of its license chain, a rule Part 1 established and this generation illustrates perfectly.

What improved (vendor claims, paper and card). The 2024 wave’s headline figures, quoted via secondary sources because the card was not re-read in the source pass: Arena Hard 85.0 for the 70B Instruct against GPT-4o (May 2024) at 79.3, and MT-Bench 8.98. Ultra v1 with reasoning on: GPQA-Diamond 76.0 (the best open model on GPQA at the time, per the paper, arXiv:2505.00949), AIME 2024 80.8, AIME 2025 72.5, MATH500 97.0. The paper’s headline claim is parity class with DeepSeek-R1 at superior inference throughput and memory efficiency. Super v1.5 (Jul 25, 2025) extended the post-training (SFT, RPO, RLVR, iterative DPO for tool use, plus multilingual and RAG sets), and its card figures, measured with NeMo-Skills with reasoning on and averaged over up to 16 runs, read: MATH500 97.4, AIME 2024 87.5, AIME 2025 82.71, GPQA 71.97, LiveCodeBench (Oct 2024 to Feb 2025 window) 73.58, MMLU-Pro with chain of thought 79.53.

Attribution. The paper credits the NAS plus distillation pipeline for reaching R1-class reasoning at a fraction of the serving cost, and the reasoning toggle for controllable test-time compute. No independent reproduction of the v1 or v1.5 figures was pinned in the source pass, so every number in this step is a vendor claim, stated once here and not repeated as decoration. The generation’s lesson is structural: when the base is borrowed, the differentiator is entirely post-training and architecture surgery, and the ceiling on openness is set by the upstream license, not by your own generosity.

Step 4: Nemotron-H (April 2025), the hybrid turn

What changed. Architecture, decisively. Nemotron-H (arXiv:2504.03624) replaces most self-attention layers with Mamba-2 state-space layers; only about 8 percent of layers remain attention. Two dense hybrids, 8B and 56B, at 8K context on release. A Mamba-2 layer scans the sequence with a learned, input-dependent state transition and carries a fixed-size state at generation time instead of a growing key-value cache, so per-token cost and memory stay constant in sequence length; the sparse attention layers remain for the jobs state-space layers do badly, chiefly exact content-addressable recall. An FP8 training recipe ships as part of the release. MiniPuzzle pruning plus distillation derives a 47B from the 56B. The license is where this generation breaks the march the other steps follow: Hugging Face metadata tags at least the 8B base repo with the NVIDIA Internal Scientific Research and Development Model License, while redistributors cite the NVIDIA Open Model License, and the card text itself was not read directly in the source pass, so Nemotron-H’s commercial status is unresolved and this part grades it suspended, exactly as Part 1 did.

What improved (paper claims). Accuracy better than or on par with similarly sized open Transformers, with Qwen-2.5-7B and 72B and Llama-3.1-8B and 70B as the named comparators, at up to 3x faster inference. The pruned 47B keeps similar accuracy to the 56B at 20 percent faster inference. The FP8 recipe achieves on-par results with BF16 and was itself used to train the 56B, which makes the recipe a demonstrated artifact rather than a suggestion.

Attribution. The paper credits the hybrid layer pattern for speed at parity, MiniPuzzle for the 47B’s free speedup, and the FP8 recipe for training at BF16 quality. These are the paper’s own claims; no independent verification was pinned in the source pass.

Bridge: Nemotron-Nano v2 (Aug 2025). The dense hybrid line matures here (arXiv:2508.14444): a 12B base pretrained on 20 trillion tokens with an FP8 recipe, Minitron-compressed to the 9B v2 at 128K context. Paper and card claims, via secondary quotation in the roster: AIME25 72.1, MATH500 97.8, GPQA 64.0, IFEval 90.3, and up to 6x the reasoning throughput of Qwen3-8B. Nano v2 matters to this story twice over: it is the dense hybrid’s finished form, and it becomes the backbone of the Labs VoiceChat model in Step 7, which is how a bridge checkpoint turns into infrastructure.

Step 5: Nemotron 3 (December 2025 to June 2026), the hybrid MoE generation

What changed. The two experiments of the previous steps, hybrid layers and mixture-of-experts, merge into one body plan: a hybrid Mamba-Transformer MoE at three sizes, Nano 30B-A3B, Super 120B-A12B, and Ultra 550B-A55B (family paper arXiv:2512.20856). Super and Ultra add LatentMoE, which projects tokens into a lower-width latent space before routing so a much larger expert bank becomes affordable, and multi-token prediction (MTP) layers, and both were pretrained in NVFP4, the 4-bit floating-point format whose training viability Part 5 examined. Context rises to 1M tokens. Post-training becomes multi-environment RL with granular reasoning budget control: not just a reasoning toggle, but a dial. The Nano config, read directly for the source pack, gives the concrete shape: 128 routed experts, 1 shared expert, top-6 routing, 52 layers with only 6 attention layers, vocab 131,072. Training scale: Nano 25 trillion tokens per its card; Super 25 trillion in two phases (20T diversity, 5T high quality) drawn from about 10T unique curated tokens per its tech report; Ultra about 20 trillion per its card, though a blog-derived summary conflicts with that figure and the conflict is carried in the flags below. The family’s stated position is full openness: weights, pre- and post-training software, recipes, and all data with redistribution rights. The licenses, however, diverge per checkpoint inside one generation: Nano ships under the NVIDIA Nemotron Open Model License per its card, Super’s card license rests on secondary summaries rather than a direct card read, and Ultra is card-verified under OpenMDW-1.1.

What improved (vendor claims; harness named where the source names it). Nano: RULER 92.9 at 256K context and 86.3 at 1M, per its card via secondary quotation. Super, from its tech report: SWE-bench Verified 60.47 under the OpenHands harness, AIME 2025 89.0, GPQA without tools 79.23, MMLU-Pro 83.73, RULER at 1M 91.75, and throughput up to 2.2x GPT-OSS-120B and 7.5x Qwen3.5-122B at 1M context. Ultra, from its Hugging Face BF16 card: SWE-bench Verified 70.7, SWE-bench Multilingual 67.7, GPQA without tools 87.0, LiveCodeBench v6 89.0, Humanity’s Last Exam 26.7. The same model’s SWE-bench Verified also reads 71.9 for BF16 on build.nvidia.com and 73.9 in the Super VL card’s comparison table, three figures from three NVIDIA surfaces with harness and condition differences not stated on all of them; the spread is flagged below and is a Part 9 lesson in the wild, that a score is a property of an artifact measured a particular way, not of a model in the abstract. The step’s one independent number: Artificial Analysis scored Ultra 48 on its Intelligence Index, reported as the top US-origin open-weight model at the time.

Attribution. The family paper credits LatentMoE as a novel approach that improves model quality at fixed active compute, NVFP4 for training efficiency, and multi-environment RL for the agentic post-training gains. Ultra’s post-training adds MOPD, multi-teacher on-policy distillation from 10 or more domain-specialized teacher checkpoints built on the Ultra recipe, and those teachers are themselves partially public: one, the 550B coding teacher, was released in Aug 2026. The roster flags that as rare, and it matters for the generations argument: the lab is no longer just releasing models, it is releasing the intermediate artifacts its own pipeline consumes.

Diagram of Nemotron 3 Nano's 52 layers, with the 6 attention slots marked
Nemotron 3 Nano's 52 layers: attention holds 6 slots, Mamba and MoE hold the rest, and the KV cache only grows on the 6.

Step 6: Nemotron 3.5 (August to October 2026), the distillation and compression generation

What changed. The hybrid MoE skeleton stays. The construction method changes, and the change is the generation. The technical report behind this step (“Nemotron 3.5 Super and Lightning,” NVIDIA, Oct 14, 2026, revised Oct 21, 2026, read directly for the source pack) describes both leads as initialized from Nemotron 3 bases: Super from Super 3 Base, Lightning from Nano 3 Base. Each receives a recursively shared MTP head (Super consolidates two non-shared heads into one; Lightning’s head is projected and pruned from Super Base, hidden size 4,096 down to 2,688 and routed experts 512 down to 128), a warm-up on 100B tokens with the backbone frozen, and then distillation on 1T tokens from the next size up: Super from Ultra 3 Base, Lightning from Super 3 Base, using top-128 logits with forward KL at 0.9 plus cross-entropy at 0.1. Super then adds stages the Nemotron 3 recipe did not have as named, separately ablated steps: 100B tokens of Indic continued pretraining, 35B tokens of context extension to 1M, 65.4B tokens of agentic SWE mid-training at 256K context (the Super VL card rounds this to 67B, a small cross-surface disagreement flagged below), and 9B tokens of long-context recovery. Super VL adds vision through a C-RADIO encoder. The Bolt line compresses Super VL by Star Elastic, a router-guided architecture search, cutting expert top-K from 22 to 9 and shipping the artifact with experts at 2.0625 bits via quantization-aware distillation. Transcribe reuses the Lightning decoder behind a Parallel Expert Encoder (a roughly 600M trainable ASR transformer plus the roughly 99M Nemotron-3-Diarization speaker encoder). The whole line sits on OpenMDW-1.1, with upstream terms stacked where the base is not NVIDIA’s own: Bolt E4B is a post-train of Gemma 4 under Apache 2.0, and Content Safety stacks Gemma-3’s terms. One caution from the cards, because it is the Part 7 lesson in miniature: Lightning’s own card lists its post-training as pretraining, MTP continued pretraining, SFT, and GRPO RL, while the technical report describes single-stage text-only SFT plus RLVR for the same model; both descriptions are NVIDIA’s, and the discrepancy is flagged below rather than harmonized.

What improved (vendor claims; conditions named). From the technical report: post-trained Super reaches SWE-bench Pro 51.78, surpassing Nemotron 3 Ultra at roughly 4.6x fewer active parameters, which is the generation’s thesis in one number, a distilled derivative beating its own teacher’s generation on a hard coding benchmark. Test-time scaling, parallel agent execution plus self-verification, moves Super on Terminal-Bench 2.1 from 62.7 to 71.9 and on DeepSWE from 24.6 to 34.5. NVFP4 checkpoints recover 99.62 percent median accuracy against BF16 across 36 benchmarks. Throughput reaches up to 4,856 output tokens per second per GPU on GB200 at 50 or more tokens per second per user on a 20-turn agentic proxy workload, with NVFP4 at 2.43x BF16 throughput on the interactive agentic workload. From the cards, all NVIDIA-measured under NeMo Gym or the NeMo Evaluator SDK: Super VL posts GPQA-Diamond 89.52, LiveCodeBench v6 88.22, SWE-bench Verified 71.10, and Terminal-Bench 2.1 at 60.67 on the current card, where the report’s 62.7 and 71.9 are the same model under report conditions, a three-surface spread flagged below. Lightning posts SWE-bench Verified 51.56, and its own card table shows Qwen3.6-35B-A3B at 70.12 on the same benchmark, an in-table loss the card answers with throughput and cost per task rather than accuracy; Part 10 already filed Lightning as the design that sells cost per task while losing the capability comparison on its own card, and the generations view adds that this honesty is itself a generational trait, unthinkable on the 2023 cards that published no scores at all. Bolt 74B’s shipped 2-bit artifact posts Terminal-Bench 2.1 50.98, BrowseComp 66.75, PinchBench 87.82, and GPQA Diamond 86.43. Transcribe leads its card’s open-weight ASR tables with an English mean WER of 4.71 against the next best open weight, Qwen3-ASR-1.7B at 4.94, in results captured Oct 12, 2026. Independent: Artificial Analysis scored Bolt 74B at 21.4 on its Intelligence Index; an AA figure of 26 for Super VL circulates in secondary reporting but was not pinned to AA directly in the source pass, so it is flagged, not used.

Attribution. This generation’s report is unusually explicit, and the credits are the report’s own, which is precisely why this step can be written with more confidence than Steps 1 and 3. The SWE mid-training data (whole repositories, issue and pull-request pairs, commit histories, build and CI logs, agent trajectories) is credited for the coding jump, shown by a controlled ablation with identical SFT: DeepSWE pass@1 from 6.4 to 10.4, and SWE-bench Verified under the Codex harness from 48.4 to 61.8. The long-context recovery stage is credited for restoring RULER at 1M from 33.5 to 81.0, a number that also confesses how much damage the other stages did before the repair. Hidden-state mixing plus MTP boosting is credited for acceptance-length gains on SPEED-Bench, 4.287 to 4.718 greedy, which is serving economics stated as a training result. The Indic stage is credited for gains of up to 9.82 points across five Indic benchmarks. On the Bolt card, Star Elastic search plus quantization-aware distillation is credited for the 2-bit artifact’s quality. The report points to a companion recipe paper for the mid-training method (Singhal et al. 2026, arXiv:2610.22019), which was not read in the source pass; the ablation numbers above are the report’s, not the companion’s.

Step 7: The Nemotron Labs lines (2026), experiments on the inference axis

What changed. Labs is NVIDIA’s research label, and the Labs models do not chase scale. Each one changes how inference compute is spent, which makes this step less a generation than a set of probes into what the next generations might absorb. Four lines matter.

Nemotron-Labs-Diffusion (NLD), May 2026. A masked-diffusion model on dense Transformer bones, initialized from Ministral 3, trained on about 1T tokens of pure autoregressive objective and then about 300B of joint autoregressive plus block-diffusion training. One checkpoint serves three decoding modes: autoregressive, block diffusion in 32-token blocks, and self-speculation. The claim, from NVIDIA’s research page and Hugging Face blog: the 8B decodes about 6x more tokens per forward pass than Qwen3-8B with slightly better average accuracy (plus 1.2 points across its 10-task instruct eval), and about 4x throughput on SPEED-Bench under SGLang on a GB200. NLD-Uni (card release Oct 27, 2026) extends the line to a unified multimodal model for understanding plus generation, a mixture-of-transformers based on LaVida-O trained on more than 100M image-text pairs; its card publishes an evaluation protocol but no benchmark results, so there is nothing to score it with yet.

Nemotron-Labs-Sigma, October to November 2026. Continuous diffusion: tokens are points in a 16-dimensional embedding space, noised and denoised in blocks, warm-started from pretrained autoregressive weights, with blockwise likelihood training. The paper (arXiv:2610.02665) claims the first large-scale continuous diffusion family, at 3B and 8B, and reports distribution matching as the strongest distillation method studied, giving roughly 8x acceleration on math and 4x on coding (8 and 4 tokens per forward pass) at negligible accuracy loss. The self-speculative distill-spec students (Nov 2026) add a further roughly 3.5x throughput over the autoregressive counterpart on GB200, and their emitted text is exactly the greedy autoregressive output by construction, per the spec-8B card. That card’s vendor-measured acceptance lengths, mean accepted tokens per verification round, are 7.91 on HumanEval, 11.86 on GSM8K with chain of thought, and 13.37 on MBPP.

Nemotron-Labs Dory 1.0 4B, October 16, 2026. Looped latent recurrence: a normalized-GPT whose recurrent block is re-applied for a configurable loop count, so one checkpoint serves 1, 2, and 4 loops as an inference-compute dial, the reasoning toggle of Step 3 rebuilt into the architecture itself. Pretrained on 6T tokens (NVFP4 for roughly the first 4.5T), context extended to 256K, then SFT and GRPO multi-environment RL, per its card. The card’s benchmarks, NVIDIA-measured under NeMo Gym with Terminal-Bench 2.1 via Terminus-2, are the cleanest loop-scaling table in the lineage: SWE-bench Verified 50.3 at 1 loop, 59.5 at 2, 61.8 at 4; Terminal-Bench 2.1 12.5, 18.1, 22.2; GPQA Diamond 69.19, 72.41, 72.16. The monotone gain with loops is the card’s own evidence for recurrence depth as a compute axis; the card offers no further ablation prose for it, and the GPQA flattening between 2 and 4 loops (72.41 to 72.16) is visible in its own numbers for anyone who wants the counterweight.

NemotronLabs VoiceChat 11B, August 3, 2026. Full-duplex speech on the Step 4 bridge: a Fast Conformer encoder, the Nemotron Nano v2 9B backbone, a TTS decoder, and a separate tool-calling output channel, trained on roughly 550K hours of audio, per its card. Card claims: number 2 among open full-duplex models on VoiceBench and Full-Duplex-Bench 1.0, the first open full-duplex model to support tool calling, about 450 ms turn-taking latency, and an AU Harness BFCL-v3 average of 56.1 percent.

Attribution for the step as a whole. Each Labs line carries its own paper or card credit, stated above, and none of them was independently reproduced in the source pass. The honest framing is that Labs is where NVIDIA publishes its inference-axis hypotheses with receipts of varying thickness: Dory’s loop table is a real dose-response curve, NLD-Uni is a protocol with no results, and the distance between those two is the distance between a finding and an announcement.

The cross-generation threads

The license march toward OpenMDW, with exceptions that prove the per-checkpoint rule. The path runs from the 2023 community license, to the NVIDIA Open Model License Agreement (2024 through the Llama-Nemotron era, stacked under Llama’s terms for the derivatives), to the NVIDIA Nemotron Open Model License (Nemotron 3 Nano, and still the license named on the Sigma distill-spec card in Nov 2026), to OpenMDW-1.1 (Nemotron 3 Ultra and the whole 3.5 line). The direction is unmistakable and the uniformity is not: the NLD-Uni card names an NVIDIA Open Source License Agreement, Nemotron-H’s status is unresolved between two conflicting citations, and Super’s card license still rests on secondary summaries. Seven generations in, the rule from Part 1 has not weakened: read the license on the checkpoint you are holding, not the family’s reputation.

Architecture convergence on the hybrid Mamba MoE. Dense Transformer in 2023 and 2024, NAS-compressed Llama derivatives in 2025, then the turn: dense hybrid Mamba-Transformer in Nemotron-H and Nano v2, hybrid Mamba-Transformer MoE with LatentMoE and MTP in Nemotron 3, and distilled, elastic, multimodal derivatives of that same skeleton in 3.5. NVIDIA ran the dense plan at 340B, measured its serving economics, and never shipped a from-scratch dense flagship again. The Labs lines explore the exits from the converged design (diffusion in two flavors, looped recurrence) in parallel, which is what a lab does when it believes the current body plan is a local optimum worth both exploiting and escaping.

Context growth as a release feature. 4,096 tokens in 2023 and again, remarkably, in the 340B flagship of 2024; 128K with the Llama derivatives and Nano v2; 256K native with 1M extension in Nemotron 3 and 3.5. The 3.5 report’s long-context recovery stage, and its RULER confession (33.5 before repair, 81.0 after), show the mature form of this thread: context is no longer a config value but a capability that other training stages can break and a dedicated stage must restore.

Post-training becoming the differentiator. SteerLM’s attribute knobs in 2023; synthetic-data RLHF in 2024, where the pipeline was the product; SFT plus preference optimization plus verifiable-reward RL with a reasoning toggle in 2025; multi-environment RL in NeMo Gym in Nemotron 3; then MOPD from specialized teachers, distillation down the size ladder, and elastic compression as first-class, named, ablated release steps in 3.5. Pretraining scale grew across the same span (3.5T claimed tokens to 25T), but the generations are distinguished by what happens after pretraining, and 3.5’s flagship result, a distilled Super beating Ultra at 4.6x fewer active parameters, is a post-training result from top to bottom.

Precision as a release dimension. FP8 appears first as a serving fit (Nemotron-4 on one node) and a training recipe (Nemotron-H, Nano v2); NVFP4 becomes a pretraining format (Nemotron 3 Super and Ultra, Lightning, Dory’s first 4.5T tokens) with a published recovery figure (99.62 percent median accuracy against BF16 across 36 benchmarks in 3.5); and quantization-aware distillation turns precision into an artifact choice, with Bolt shipping experts at 2.0625 bits and posting real benchmark numbers on the shipped artifact. The Part 10 rule, that a benchmark is a property of an artifact at a precision, gets easier to honor every generation because NVIDIA increasingly measures the artifact it ships.

Data disclosure, the quiet thread. An undisclosed corpus summarized at about 3.5T tokens in 2023; 9T disclosed in 2024 with the alignment pipeline open-sourced; 20 to 25T in Nemotron 3 with the corpora themselves published; and in 3.5, mid-training promoted to a named stage with its own data mix and its own controlled ablation. Each generation publishes one more layer of the thing the previous generation kept private. That, more than any single score, is the trajectory this part exists to document.

What this means for LAB2

LAB2’s Step B, the roughly 30B-active class where the cyber family gets its working size, is currently co-led by Nemotron 3.5 Lightning and Muse Glimmer 30B. The generations view is the evidence base for that bet, and it cuts in Lightning’s favor in four specific ways.

First, lineage support. Lightning is not a one-off release; it is the third size-ladder distillation in a family whose vendor has published, generation after generation, the artifacts a derivative builder actually needs: stage checkpoints, post-training datasets, teacher checkpoints (Ultra’s 550B coding teacher is public), and recipes detailed enough that 3.5’s own report can be audited stage by stage. A base is a dependency, and this lineage’s release behavior across seven generations is the strongest available predictor that the dependency will be maintained, documented, and extended rather than abandoned.

Second, the skeleton’s economics are LAB2’s economics. The hybrid Mamba MoE body plan exists to cut the cost of long generation, and a cyber agent’s working life is long generation: long tool traces, long logs, long exploit chains. The family’s published throughput claims are vendor numbers and should be treated as such, but the architectural reason behind them (constant-cost state for most layers, a small attention minority for exact recall) is a mechanism, not a slogan, and it is the same mechanism LAB2 would be buying.

Third, the distillation pattern is directly reusable, inside the teacher constraint. Nemotron 3.5 is built by distilling down a size ladder from the lab’s own larger bases, with the teachers’ licenses clean because the whole chain is NVIDIA’s. LAB2’s version of that move must respect the standing rule: DeepSeek’s API is the only hosted API whose terms expressly permit training use including distillation, self-served open weights are the second clean line, and GLM output never enters a Counterfleet training set. A Lightning-based Step B keeps the option open of distilling from self-hosted open teachers under licenses that permit it, following the exact shape (frozen-backbone MTP warm-up, 1T-token distillation, then domain mid-training as a named stage) that the 3.5 report ablates. The SWE mid-training ablation is the template for a cyber mid-training stage: whole repositories and agent trajectories in NVIDIA’s case; exploits, patches, and verified tool traces in LAB2’s.

Fourth, the cautions are also generational, and they are specific. Lightning’s own card concedes the capability comparison to Qwen3.6-35B-A3B on SWE-bench Verified (51.56 against 70.12) and answers on cost per task, so a Step B pick of Lightning is a bet that the cyber fine-tune closes a real capability gap, not a claim that the gap is absent. The per-checkpoint license rule applies with full force: the 3.5 line is uniformly OpenMDW-1.1, which is why it is usable at all, but the family’s own history (Nemotron-H’s unresolved conflict, Super’s card license resting on secondary summaries) is the standing reminder to verify the exact checkpoint’s card directly before any LAB2 run, a read still owed for Super. And the score-spread flags below mean every capability figure used in a LAB2 decision should be re-measured on the shipped artifact under LAB2’s own harness, which is already the program’s rule; the generations record is evidence for trajectory and release behavior, never a substitute for the Proof of Defense card.

Open flags carried from the source pack

These are stated once here, in addition to the four that the text above flags in place (the Nemotron-H license conflict in Step 4, Super’s card license resting on secondary summaries in Step 5, Ultra’s three SWE-bench Verified figures in Step 5, and the Terminal-Bench three-surface spread in Step 6). None of them is resolved by this part, because the sources behind it do not resolve them.

  1. Ultra’s pretraining tokens: about 20T per its card, against a developer-blog-derived summary of a 10T base plus 212B new tokens. The card figure is the one cited in the roster; the conflict stands.
  2. Super’s agentic SWE mid-training size: 65.4B tokens in the technical report, 67B on the Super VL card.
  3. Nemotron-4 340B’s RewardBench figure: 92.0 and 94.1 both circulate in secondary sources; the report’s body tables were not re-read in the source pass (abstract only).
  4. Nemotron-3 8B’s pretraining tokens: about 3.5T per the NGC QA card, about 3.8T per a secondary compilation, with no tech report to settle it.
  5. The 2024 wave’s headline figures (Arena Hard 85.0, MT-Bench 8.98) rest on secondary quotation of the 70B card, which was not re-read in the source pass.
  6. Independent reproduction generally: beyond Artificial Analysis (Ultra 48, Bolt 74B 21.4), no independent reproduction of the 3.5 report’s SWE mid-training ablation, its Terminal-Bench self-verification gains, or Dory’s loop-scaling table was pinned in the source pass. All three are vendor-only for now.
  7. Tokenizer changes per generation are thin in the primary sources: the two configs read directly for the source pack, Nemotron 3 Nano and Dory, both use vocab 131,072, and earlier generations’ tokenizer details in circulation come from secondary summaries.
  8. Smaller unread items behind this part: the Rerank 8B card’s numeric results table, the Parse v1.1, v1.2, and 2.0 cards, the Sigma instruct repos’ card-level release dates, VoiceChat’s context length (not stated in the card sections read), and the 3.5 companion mid-training paper (arXiv:2610.22019).

Watch and read

  • NVIDIA, Nemotron 3 family paper. The hybrid Mamba-MoE generation in its own report: LatentMoE, NVFP4 training, multi-environment RL.
  • NVIDIA, Nemotron-H paper. The hybrid turn (Step 4) with the FP8 recipe and MiniPuzzle, and the generation whose license this part grades suspended.
  • Bercovich et al., Llama-Nemotron. The derivative era (Step 3): NAS over Llama backbones, distillation recovery, and the reasoning toggle, with the stacked-license lesson attached.