Model Making, in the Open: A Masterclass

Reference · Glossary

Reference · Glossary Wiki, Model-Making Masterclass

Version 1, built Nov 17, 2026. This is the cumulative glossary for the masterclass. It grows as parts land: every load-bearing term gets one entry with a plain definition, a quick explainer, a note on where the decision shows up in this study, and deep-dive pointers.

Rules for pointers: a specific chapter, lecture, or section is named only where it was verified during the build of this file. Where a pointer is given generally, the entry says so. Pointer verification pass: Nov 17, 2026.

How to read an entry: the first line is the definition. The paragraph after it is the quick explainer. “In this study” ties the term to the anchor families and the part map in Piece 0. “Deep dive” is the homework.

Index

Active parameters; Annealing; Catastrophic forgetting and replay; Checkpoint and safetensors; Context window; Curriculum (data); Decontamination; Distillation; DPO; Embeddings; Eval harness; Expert routing and load balancing; FP8, NVFP4, and quantization; Gradient checkpointing; GRPO; Hybrid Mamba; KV cache; Latent attention (MLA); Learning rate schedule and warmup; LoRA and QLoRA; MFU; Midtraining; Mixture of Experts (MoE); Model souping (weight averaging); MOPD; Muon optimizer (and MuonClip); Multi-token prediction (MTP); On-policy distillation; Parallelism (data, tensor, pipeline); pass@k; Positional encoding (RoPE, NoPE, YaRN); Post-training; Pretraining; RLVR; Scaling laws and tokens per parameter; SFT; Sliding window attention; Speculative decoding; Synthetic data; Tokenizer and BPE.


Active parameters

Definition: The parameters that actually compute on a given token, as opposed to the total parameters stored in the model.

In a dense model the two numbers are the same. In a Mixture of Experts model only a few experts fire per token, so the active count is a small slice of the total. Active parameters drive speed and compute cost per token. Total parameters drive memory: the weights you must load and store, which is why Piece 0 sorts every model into a shelf band (total) and a compute band (active) and never lets a comparison mix the two.

In this study: the spine depends on it. Nemotron 3 Nano (30B total, 3B active) is compared against dense 3B models in the compute band, and Nemotron 3 Ultra (550B total, 55B active) shelves at the top of the chart while computing in a band far below its shelf size.

Deep dive: Stanford CS336, Lecture 4 (Mixture of Experts) and Lecture 2 (PyTorch, Resource Accounting), Spring 2025, both verified in this pass. The DeepSeek-V3 Technical Report (arXiv:2412.19437) states the convention plainly in its abstract: 671B total, 37B activated per token.

Annealing

Definition: The final phase of pretraining, where the learning rate decays toward zero and the data mix usually shifts to the highest-quality slice available.

Think of it as landing the plane. Most of pretraining runs at a roughly steady learning rate on the broad web mix. In the anneal, the trainer spends the remaining budget on the best data (textbooks, curated code, high-grade synthetic) while the shrinking learning rate lets the model settle into the minimum it has been circling. A good anneal buys measurable benchmark points for a small fraction of the total run cost, which is why families guard or publish their anneal mixes as signature decisions.

In this study: Part 6 treats annealing as half of midtraining. OLMo’s Dolmino mix and SmolLM3’s end-of-run data choices are the public exhibits; several Nemotron runs also show anneal-stage data upgrades.

Deep dive: Hugging Face, The Smol Training Playbook: The Secrets to Building World-Class LLMs (Oct 30, 2025), which walks the full SmolLM3 run including its late-stage data decisions; pointer verified generally, the playbook is one long document rather than numbered chapters. The OLMo 2 and OLMo 3 reports describe their anneal mixes in detail; cited generally here, report section numbers not re-verified in this pass.

Catastrophic forgetting and replay

Definition: Forgetting is the tendency of a model to lose earlier skills when trained hard on new data; replay is the standard countermeasure, mixing a slice of the old data back into the new training.

Neural networks overwrite. Fine-tune a general model only on cyber data and it gets better at cyber while quietly getting worse at general reasoning, math, and instruction following, sometimes below the level where the specialization is usable. Replay fights this by keeping a fraction of general data in every later training mix, so the model rehearses what it already knew while learning the new material. The replay fraction is a real budget decision: too little and the base skills rot, too much and the specialization dilutes.

In this study: load-bearing for Part 11 (cyber specialization) and for LAB2’s V0.1 plan, which specifies a general replay slice inside its QLoRA run for exactly this reason. It also explains why post-training sets like Tulu 3 and Nemotron’s mixes are broad rather than narrow.

Deep dive: the replay idea is old continual-learning practice; the classic reference is Kirkpatrick et al. on elastic weight consolidation (2017), cited generally here, not re-verified in this pass. For the applied version, see the post-training data discussions in the Tulu 3 report, cited generally.

Checkpoint and safetensors

Definition: A checkpoint is a saved snapshot of a model’s weights (and sometimes optimizer state) at a point in training; safetensors is the file format most open releases now use to store those weights.

Checkpoints are the unit of openness. A family that releases only the final weights lets you run the model. A family that releases intermediate checkpoints lets you study training itself: where capabilities appear, what the anneal changed, what a stage branch did. Optimizer state checkpoints are far larger and are what you need to resume a run exactly. Safetensors matters because it is a simple, auditable format that cannot execute code on load, unlike older pickle-based formats, and it supports fast partial loading.

In this study: K2-Horizon is the extreme case, about 380 checkpoints including per-stage branches, which is what makes its training record diffable at all. Part 1 grades every family partly on what checkpoint granularity it publishes.

Deep dive: the safetensors library documentation and repository from Hugging Face, cited generally (format spec and rationale are documented there; not re-verified page by page in this pass). For checkpointing as a training-system decision, Stanford CS336, Lecture 7 and Lecture 8 (Parallelism 1 and 2) cover the surrounding systems context, verified in this pass.

Context window

Definition: The maximum number of tokens the model can attend over at once, meaning the longest input plus output it can handle in a single pass.

Context is a budget the architecture and the KV cache have to pay for. Doubling the window roughly doubles the cache and the attention work for full-attention layers, which is why long-context models lean on sliding window attention, latent attention, or hybrid Mamba layers. Models are usually pretrained short (4K to 8K) and extended late in training, because long-context training on the full run would waste compute on a skill most tokens never exercise.

In this study: the anchors sit at the extremes. Every K2-Horizon model ships 512K context, built through a four-stage midtraining extension. SmolLM3’s 100B-token context extension stage is a Part 6 exhibit, and Part 10 prices what 256K and 512K windows cost at serving time.

Deep dive: Hugging Face, The Smol Training Playbook (verified generally), whose “Extending Context” discussion covers staged extension (4K to 32K to 64K and beyond) with a modest token budget late in training. Stanford CS336, Lecture 10 (Inference) covers the serving-side cost of long contexts, verified in this pass.

Curriculum (data)

Definition: A curriculum is a deliberate ordering of training data, usually easy or general first and hard or specialized later, instead of a shuffled uniform mix.

The intuition comes from teaching: foundations before applications. In modern pretraining the strong version of curriculum is less about per-example difficulty and more about staging: broad web data first, then higher-quality and more specialized mixes, then the anneal. Post-training uses it too, sequencing SFT before preference tuning before RL. The honest caveat is that at pretraining scale, ordering effects are smaller than mix proportions, which is why the open reports talk more about mixes and stages than about difficulty ranking.

In this study: K2-Horizon’s four-stage midtrain and its three-phase SFT are curriculum decisions, and Part 2 and Part 7 compare staged mixes across OLMo, SmolLM, and Nemotron. The cyber read in Part 11 asks when domain data should enter the sequence for a specialist model.

Deep dive: the term originates with Bengio et al., “Curriculum Learning” (ICML 2009), cited generally, not re-verified in this pass. For the modern staged-mix version, the data chapters of the OLMo 2 report and The Smol Training Playbook are the working references, both cited generally.

Decontamination

Definition: Removing evaluation material, and near-duplicates of it, from training data so that benchmark scores measure capability instead of memorization.

If a benchmark’s questions sit inside the pretraining corpus, the model can score by recall and the number is worthless. Decontamination pipelines hash and match training documents against benchmark items, usually with n-gram overlap, and drop or scrub the collisions. The rigor varies enormously: some families publish their method and thresholds, others assert cleanliness. Near-duplicate handling is the hard part, since a reworded test item still leaks the answer.

In this study: Part 2 compares decontamination methods per family, and Part 9 returns to it as an evaluation-integrity question. It is also a LAB2 rule: the V0.1 dataset plan requires decontamination against all eval material before training.

Deep dive: Stanford CS336, Lecture 13 and Lecture 14 (Data 1 and Data 2) and the course’s Data assignment, which has students filter and deduplicate Common Crawl themselves; lectures verified in this pass. The FineWeb and DCLM write-ups describe production decontamination and filtering pipelines; cited generally here.

Distillation

Definition: Training a smaller or weaker student model to imitate a stronger teacher model, usually on the teacher’s outputs or output distributions.

The teacher has already paid the cost of learning the skill; the student learns from the teacher’s answers instead of from raw data or sparse rewards, which is a much denser signal. Classic distillation matches the teacher’s full probability distribution over next tokens. The modern workhorse version is simpler: generate a large set of teacher responses, then SFT the student on them. The catch is inheritance: the student absorbs the teacher’s errors and style along with its skill, and licensing terms on teacher outputs decide whether the practice is even allowed for a given teacher.

In this study: everywhere. Nemotron’s specialist line is built by distilling strong teachers into smaller bases, Nemotron-H-47B was created from the 56B model by pruning plus distillation (the MiniPuzzle technique), and Part 8 separates plain distillation from its on-policy variant (next entry but one) and from MOPD.

Deep dive: the origin paper is Hinton, Vinyals, and Dean, “Distilling the Knowledge in a Neural Network” (2015), cited generally. The Nemotron-H report (arXiv:2504.03624, verified in this pass) documents MiniPuzzle pruning plus distillation in a current production recipe.

DPO

Definition: Direct Preference Optimization, a post-training method that teaches a model from pairs of preferred and rejected responses with a simple classification-style loss, no separate reward model and no RL loop.

RLHF’s classic pipeline trains a reward model on human preferences, then runs PPO against it, which is powerful and fiddly. DPO showed the same objective can be rewritten so the model itself is trained directly on the preference pairs: raise the probability of the chosen response, lower the rejected one, stay near the starting model. It is cheaper, stabler, and became the default preference stage in open recipes from 2023 on. Its weakness is that it learns only from fixed pairs, so it cannot explore responses outside the dataset the way on-policy RL can.

In this study: Part 8 lines DPO up against GRPO and RLVR per family. Tulu 3 is the canonical open DPO recipe; several Nemotron specialists use DPO variants before or instead of RL.

Deep dive: Rafailov et al., “Direct Preference Optimization: Your Language Model is Secretly a Reward Model” (arXiv:2305.18290, verified in this pass). Nathan Lambert, RLHF Book (rlhfbook.com), the “Direct-Alignment Algorithms” chapter, which opens with DPO; chapter title verified in this pass, section numbering varies by edition.

Embeddings

Definition: The learned vectors that represent tokens inside the model; the input embedding table maps each token ID to a vector, and in many models the same table, tied, is reused as the output head that scores the vocabulary.

The embedding table is quietly one of the biggest parameter blocks in small models: vocabulary size times hidden dimension. A 256K vocabulary at a 4096 hidden size is a billion parameters before the first layer runs. Tying the input and output embeddings halves that cost and usually loses nothing at small scale, which is why small families tie and large ones often do not. Embeddings are also where vocabulary decisions become parameter decisions, the K2-Horizon “7B that is 9B with embeddings” lesson from Piece 0.

In this study: Part 3 does the vocabulary accounting per family, and Part 4 revisits tying as an architecture decision at small sizes.

Deep dive: Sebastian Raschka, Build a Large Language Model (From Scratch), Chapter 2, “Working with Text Data,” which builds word embeddings and the input pipeline in code; chapter verified in this pass. The Smol Training Playbook’s ablation notes cover tied embeddings as a tested decision for small models, cited generally.

Eval harness

Definition: The software that runs a model against benchmarks in a fixed, repeatable way: same prompts, same scoring, same settings, so two models’ numbers can be compared at all.

A benchmark score is meaningless without the harness that produced it. Prompt formatting, few-shot count, sampling settings, and answer extraction all move scores by points, which is why “beats X on benchmark Y” claims from different harnesses cannot be lined up. The standard open harness is EleutherAI’s lm-eval, and serious families publish their harness or their exact settings alongside the numbers. A holdout is the companion concept: eval data kept private and never trained on or tuned against, so the score cannot be gamed by iteration.

In this study: K2-Horizon ships its own harness (Eval360) with its checkpoints, which is the transparency gold standard in the roster. Part 9 is built around harness discipline, holdouts, and the benchmarketing failure modes from intake, and it is where the pass@k entry below plugs in.

Deep dive: Stanford CS336, Lecture 12 (Evaluation), verified in this pass. The lm-evaluation-harness repository (EleutherAI) and Hugging Face’s evaluation guidebook are the working tools; cited generally, not re-verified page by page in this pass.

Expert routing and load balancing

Definition: Routing is the per-token decision of which experts in an MoE layer process that token; load balancing is the set of tricks that keep the chosen experts evenly loaded instead of letting a few experts take all the traffic.

The router is a small learned network that scores the experts for each token and picks the top few. Left alone, routing collapses: early winners get more training, get better, and get picked more, until most experts are dead weight and the popular ones are a bottleneck. Classic fixes add an auxiliary loss that penalizes uneven assignment. DeepSeek’s auxiliary-loss-free approach instead nudges routing with per-expert bias terms that are adjusted by observed load, keeping the main loss clean. Routing also has a systems cost: tokens must be shipped to the GPUs holding their experts, so balance is a throughput question, not only a quality one.

In this study: Part 4 compares routing designs across DeepSeek-style MoE, Nemotron’s MoE entries, and K2-Horizon’s MoVA model. Expert counts and top-k choices are headline rows in the matched-size tables.

Deep dive: DeepSeek-V3 Technical Report (arXiv:2412.19437), Section 2.1.2, “DeepSeekMoE with Auxiliary-Loss-Free Load Balancing,” verified in this pass. Stanford CS336, Lecture 4 (Mixture of Experts), verified in this pass.

FP8, NVFP4, and quantization

Definition: Quantization means storing and computing with fewer bits per number; FP8 is an 8-bit floating format and NVFP4 a 4-bit one, both used to make training and serving cheaper than the standard 16-bit BF16.

Fewer bits means less memory, less bandwidth, and faster matrix math on hardware built for it, at the price of rounding error that must be managed. For serving, weights are commonly quantized after training (INT4 and FP8 releases, GGUF files for local engines). For training, the frontier moved from BF16 to FP8 mixed precision (DeepSeek-V3, Nemotron-H) and now to NVFP4, where NVIDIA showed a 12B model pretrained on 10T tokens in 4-bit matching an FP8 baseline, using block scaling, stochastic rounding, and a minority of layers kept in high precision. The rule that survives every format: sensitive parts (embeddings, normalization, accumulations, optimizer state) stay in higher precision.

In this study: Part 5 covers FP8 and NVFP4 as pretraining decisions; Part 10 covers post-training quantization for serving, including the KV cache cautionary tale from the Maxwell intake. Nemotron is the family pushing this frontier hardest, which is one reason it anchors the study.

Deep dive: NVIDIA, “Pretraining Large Language Models with NVFP4” (arXiv:2509.25149, verified in this pass). DeepSeek-V3 Technical Report, Section 3.3, “FP8 Training,” verified in this pass. The Nemotron-H report (arXiv:2504.03624) documents an FP8-based recipe at family scale.

Gradient checkpointing

Definition: A memory-saving training technique that discards intermediate activations in the forward pass and recomputes them during the backward pass, trading roughly a third more compute for a large cut in memory.

Activations, not weights, are what blow up GPU memory when sequences get long or batches get big. Checkpointing stores only the inputs at chosen layer boundaries and re-runs the forward computation for each segment when its gradients are needed. It is one of the standard levers, alongside parallelism and low precision, that decides whether a given model and batch size fit on a given cluster at all, and its compute tax shows up directly in MFU.

In this study: background machinery for Part 5’s systems discussion; the Ultra-Scale Playbook treats it as a first-order throughput decision, and every family’s MFU number implicitly includes whether it was on.

Deep dive: Hugging Face, The Ultra-Scale Playbook: Training LLMs on GPU Clusters (hf.co/spaces/nanotron/ultrascale-playbook), the memory and throughput chapters; playbook verified in this pass, cited by topic rather than section number. The original technique is Chen et al., “Training Deep Nets with Sublinear Memory Cost” (2016), cited generally.

GRPO

Definition: Group Relative Policy Optimization, an RL method that samples a group of answers to the same prompt, scores them, and trains the model toward the better-than-average ones, with no separate value model.

PPO needs a critic network estimating expected reward, which doubles the machinery. GRPO replaces the critic with the group itself: sample, say, 16 answers, and each answer’s advantage is just its reward minus the group mean. Simpler, cheaper, and it works especially well with verifiable rewards (see RLVR), where scoring is exact. Popularized by DeepSeek’s math and R1 work, it is now the default RL algorithm across most open post-training recipes, with a small industry of variants fixing its edge cases.

In this study: Part 8’s central algorithm. K2-Horizon’s RL360 code, Nemotron’s RL stages, and the Tulu and OLMo lines all expose their GRPO-style choices for comparison.

Deep dive: Nathan Lambert, RLHF Book, the “Reinforcement Learning” chapter, which treats GRPO in its own section (chapter verified in this pass; section numbering varies by edition). The origin is the DeepSeekMath paper (Shao et al., 2024, arXiv:2402.03300, as cited inside The Smol Training Playbook).

Hybrid Mamba

Definition: An architecture that replaces most attention layers with Mamba state-space layers, keeping a few attention layers, to get constant memory and compute per generated token in the Mamba layers.

Attention is precise but expensive: its cost and KV cache grow with sequence length. Mamba layers process tokens through a recurrent state of fixed size, so generation cost per token stays flat no matter how long the context gets. The hybrid bet is that a minority of true attention layers is enough to recover the recall and in-context learning that pure state-space models lose. Nemotron-H is the reference implementation: Mamba-2, attention, and MLP layers interleaved, claiming up to 3x inference throughput at matched accuracy.

In this study: Nemotron’s main architectural identity, from the Nano 9B and 12B v2 models up through Nemotron 3’s hybrid MoE designs. Part 4 contrasts hybrid Mamba against pure attention (OLMo, K2-Horizon dense) and against sliding window hybrids (SmolLM3) at matched sizes.

Deep dive: NVIDIA, “Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models” (arXiv:2504.03624, verified in this pass). The underlying Mamba and Mamba-2 papers (Gu and Dao) are the primary sources for the state-space layer itself; cited generally here.

KV cache

Definition: The stored keys and values from attention for all tokens processed so far, kept in memory so generating the next token does not recompute the whole sequence.

The KV cache is the real memory budget of serving. Its size scales with layers times context length times batch size, and for long-context models it can exceed the weights themselves. Nearly every serving optimization is an attack on this cache: grouped-query attention shrinks it by sharing keys and values across heads, latent attention compresses it, sliding window attention caps it, quantization stores it in fewer bits, and hybrid Mamba layers avoid growing it at all in most layers.

In this study: Part 10’s central quantity. The Maxwell intake item is the cautionary exhibit: a reply flagged its architecture as needing roughly 4.5x the KV cache of its Qwen rival, which would erase its benchmark story in long agent loops. Nemotron’s hybrid design and DeepSeek’s MLA are the two studied answers.

Deep dive: Stanford CS336, Lecture 10 (Inference), verified in this pass. DeepSeek-V3 Technical Report (arXiv:2412.19437), Section 2.1.1, for the compression approach quantified against GQA baselines.

Latent attention (MLA)

Definition: Multi-head Latent Attention, DeepSeek’s attention variant that compresses keys and values into a small latent vector for the KV cache and reconstructs the per-head tensors on the fly.

Standard attention caches full keys and values per head, which is what makes long-context serving expensive. MLA instead caches one low-rank latent per token, a fraction of the size, and expands it back into keys and values during computation. The trade is extra compute for far less memory, which is the right trade at serving scale, where memory bandwidth is the bottleneck. It is one of the clearest cases in the study of an architecture decision made for inference economics, not for training loss.

In this study: the DeepSeek line’s signature, carried into this roster by Nemotron 3’s hybrid design and discussed in Part 4 against grouped-query attention (the Llama-line answer), sliding window attention (SmolLM3), and K2-Horizon’s Mixture-of-Value Attention, which attacks the same cache problem from the value side.

Deep dive: DeepSeek-V3 Technical Report (arXiv:2412.19437), Section 2.1.1, “Multi-Head Latent Attention,” verified in this pass; the design was introduced and validated in the DeepSeek-V2 report, cited generally.

Learning rate schedule and warmup

Definition: The schedule is the plan for how the learning rate changes over training; warmup is its opening phase, where the rate ramps up from near zero instead of starting at full value.

Warmup exists because a cold model with random weights produces garbage gradients, and full-size steps on garbage can destabilize the run before it starts. The classic main-run shape is cosine decay: peak after warmup, then a smooth fall to near zero, which forces you to know the total token budget in advance. The newer warmup-stable-decay (WSD) shape holds the rate constant for most of the run and decays only at the end, which makes runs resumable, extendable, and branchable, and it pairs naturally with annealing: the decay phase is where the best data goes in. Loss spikes and their recoveries are schedule stories as often as they are data stories.

In this study: Part 5 compares schedules per family, including the NVFP4 validation run’s WSD recipe (constant rate through 80 percent, decay over the final 20). K2-Horizon’s stage branches are only possible because its schedule supports branching.

Deep dive: Hugging Face, The Smol Training Playbook (verified generally), whose run-type tags and hyperparameter sections treat WSD as a tested choice. NVIDIA, “Pretraining Large Language Models with NVFP4” (arXiv:2509.25149), for a fully specified production WSD recipe. The WSD schedule itself was popularized by the MiniCPM work; cited generally.

LoRA and QLoRA

Definition: LoRA fine-tunes a model by training small low-rank adapter matrices added to frozen weights; QLoRA does the same with the frozen base stored in 4-bit, so a large model can be adapted on one modest GPU.

Full fine-tuning updates every parameter and needs optimizer state for all of them, which is where the memory goes. LoRA’s bet is that the update a task needs is low-rank: two thin matrices per target layer capture it, the base never moves, and the adapter is a small file you can swap, merge, or discard. QLoRA quantizes the frozen base to 4-bit NormalFloat and pages optimizer state, which is what makes 30B-class adaptation possible on prosumer hardware. The cost is capacity: an adapter cannot teach as much as full training, and it cannot fix a base that lacks the underlying knowledge.

In this study: LAB2’s V0.1 plan is a QLoRA run, so Part 7 and Part 11 treat adapter rank, target modules, and replay mixing as first-class decisions. Raschka’s appendix is the hands-on entry point.

Deep dive: Sebastian Raschka, Build a Large Language Model (From Scratch), Appendix E, “Parameter-efficient Fine-tuning with LoRA,” verified in this pass. Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models” (arXiv:2106.09685, verified). Dettmers et al., “QLoRA: Efficient Finetuning of Quantized LLMs” (arXiv:2305.14314, verified).

MFU

Definition: Model FLOPs utilization, the share of the hardware’s theoretical peak compute that a training run actually achieves.

MFU is the honest efficiency number. A run at 40 percent MFU is burning 60 percent of a very expensive cluster on communication waits, memory stalls, and pipeline bubbles. It is computed from the model’s known FLOP cost per token (roughly 6 times parameters times tokens for dense training, adjusted for MoE and recompute) divided by the peak FLOPs of the GPUs actually used. Good large runs land roughly in the 30 to 55 percent band depending on model shape; claims far above that deserve suspicion, and claims without a stated MFU cannot be costed at all.

In this study: Part 5 uses MFU to compare the families’ systems competence, from the Ultra-Scale Playbook’s thousands of measured configurations to the anchors’ reported run costs. It is also the number that converts a recipe into a dollar figure for LAB2 planning.

Deep dive: Stanford CS336, Lecture 2 (PyTorch, Resource Accounting) and Lectures 7 and 8 (Parallelism 1 and 2), verified in this pass. Hugging Face, The Ultra-Scale Playbook, whose entire benchmarking program is organized around measured utilization; verified generally.

Midtraining

Definition: Everything trained between the main pretraining run and post-training: annealing, context extension, domain continued pretraining, and stage branches off the stable-phase checkpoint.

Midtraining is the least standardized stage and the one where open families differ most visibly, because it is where a general run gets aimed: longer context, more math and code, a domain tilt, a quality anneal. It is also where openness pays off most for a student of the craft, since several families publish their midtraining mixes and branch checkpoints while keeping almost nothing else about the stage secret. K2-Horizon’s four-stage midtrain is the most documented version in the roster.

In this study: Part 6 is devoted to it: Dolmino and Longmino (OLMo), SmolLM3’s extension stage, Nemotron’s stage structure, and model souping used as an anneal.

Deep dive: the OLMo 2 and OLMo 3 reports for Dolmino and Longmino, and the K2-Horizon tech report (arXiv:2611.21003, as catalogued in Piece 0) for the four-stage design; reports cited generally, section numbers not re-verified in this pass.

Mixture of Experts (MoE)

Definition: An architecture where each layer holds many expert subnetworks and a router sends each token to only a few of them, so total parameters can grow without proportional per-token compute.

MoE decouples the two things a dense model forces together: how much the model can know (total parameters) and how much each token costs (active parameters). The price is complexity: routing and load balancing become training problems, the full weights still occupy memory, and serving needs the whole model resident even though each token touches a slice. Fine-grained experts (many small ones, a handful active) are the current consensus shape, from DeepSeekMoE to Nemotron’s LatentMoE and K2-Horizon’s MoVA designs.

In this study: one of Part 4’s three architecture families, with Nemotron 3 Nano and 3.5 Lightning and K2-Horizon’s 36B MoVA MoE and 375B flagship as the exhibits at four different scales.

Deep dive: Stanford CS336, Lecture 4 (Mixture of Experts), verified in this pass. DeepSeek-V3 Technical Report (arXiv:2412.19437), Section 2.1.2, for the fine-grained expert design in production.

Model souping (weight averaging)

Definition: Averaging the weights of several training runs or checkpoints into a single model, also called weight averaging or a model soup.

It sounds like it should not work, because averaging two different networks usually produces nonsense. It works when the ingredients share a common ancestor checkpoint, so their weights live in the same basin: the average then behaves like a cheap ensemble and often beats every ingredient, with zero added serving cost. OLMo uses souping as its annealing method, averaging late-stage branch runs instead of picking one. K2-Horizon’s five RL experts merged into one model are a post-training cousin of the same idea.

In this study: Part 6 (souping as anneal) and Part 8 (expert merging after RL). It is also a LAB2-relevant trick: nearly free quality, if your pipeline keeps the branch checkpoints to average.

Deep dive: Wortsman et al., “Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time” (arXiv:2203.05482, verified in this pass). The OLMo reports describe souping as a production anneal; cited generally.

MOPD

Definition: Multi-teacher on-policy distillation: a student learns from several teacher models at once, on outputs the student itself generated.

Plain distillation uses one teacher and usually the teacher’s outputs. MOPD combines the two upgrades: the student generates (on-policy, so it learns on its own distribution, including its own mistakes), and a panel of specialist teachers grades it, each in its domain. Nemotron 3 Ultra’s post-training is built on it, distilling from more than ten public teacher checkpoints, which NVIDIA could do openly because it also published the teachers. The method turns a model zoo into a training asset.

In this study: Part 8’s centerpiece for the top of the Nemotron line, and a template LAB2 cares about, since the teacher-roster terms question (whose outputs may legally train a student) decides whether a MOPD-style recipe is even available.

Deep dive: the Nemotron 3 materials in the roster file (research/nemotron-roster.md) are the primary record; the general method is framed by the on-policy distillation entry below. Specific report section numbers were not re-verified in this pass, so this pointer is given generally.

Muon optimizer (and MuonClip)

Definition: Muon is an optimizer that updates weight matrices using an orthogonalized version of the gradient, instead of AdamW’s per-coordinate scaling; MuonClip is the stabilized variant from the Muon scaling literature, adding QK-Clip to stop attention logits from exploding.

AdamW treats every parameter as an independent scalar with its own learning-rate history. Muon treats a weight matrix as a matrix: it takes the momentum update and replaces it with the nearest semi-orthogonal matrix (computed with a short Newton-Schulz iteration), which equalizes the update across directions. The Muon scaling study (Liu et al.) reported roughly 2x compute efficiency over AdamW at compute-optimal settings. Raw Muon destabilizes at large scale because attention logits grow without bound; QK-Clip rescales the query and key weights after steps where logits exceed a threshold. Note Muon only handles 2D hidden weights; embeddings, heads, and scalars still run AdamW.

In this study: the optimizer fight is a headline Part 5 comparison: AdamW (most families, including both anchors) against Muon and MuonClip as method-paper claims with no frontier-scale control run behind them. Optimizer choice is one of the few recipe decisions with a claimed 2x attached to it, so it gets audited hard.

Deep dive: Liu et al., “Muon is Scalable for LLM Training” (arXiv:2502.16982, verified), the method source for Muon and MuonClip. The original Muon write-up by Keller Jordan is the entry-level source; cited generally.

Multi-token prediction (MTP)

Definition: Training the model to predict several future tokens at each position, not just the next one, using extra prediction heads or modules.

Next-token prediction gives one training signal per position. MTP densifies it: the model must also forecast token two and three ahead, which forces richer internal planning and measurably improves benchmark performance in DeepSeek’s ablations. The second life of MTP is at serving time, where the extra heads act as a built-in draft model for speculative decoding, with no separate draft model to train or host. Nemotron 3 carries MTP for the same dual reason.

In this study: Part 4 (architecture) and Part 10 (serving) share this term; it is one of the decisions where a training choice and an inference feature are the same object.

Deep dive: DeepSeek-V3 Technical Report (arXiv:2412.19437), Section 2.2, “Multi-Token Prediction,” including its ablation studies (Section 4.5.1 in the report’s contents), verified in this pass.

On-policy distillation

Definition: Distillation where the student generates its own outputs and the teacher grades those, token by token, instead of the student training on teacher-generated text.

The flaw in ordinary distillation is distribution mismatch: the student learns to imitate the teacher in situations the teacher visits, then fails in the situations it actually reaches, where its own small errors compound. On-policy distillation fixes the sampling side (the student’s own rollouts, like RL) while keeping the dense signal of distillation (a teacher score for every token, instead of one reward per episode). Thinking Machines’ write-up reports matching RL-level reasoning gains at a fraction of the compute, and the method has spread quickly through 2026 recipes, including Nemotron 3 Ultra’s MOPD.

In this study: Part 8, alongside MOPD. For LAB2 it is the most interesting legal and practical route to frontier-grade cyber behavior in a small model, provided teacher terms allow it.

Deep dive: Kevin Lu and Thinking Machines Lab, “On-Policy Distillation,” Connectionism blog, Oct 27, 2025 (thinkingmachines.ai/blog/on-policy-distillation), verified in this pass. The academic origin is Agarwal et al., “On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes” (arXiv:2306.13649, verified). Nathan Lambert’s RLHF Book also treats on-policy distillation in its post-training survey; cited generally.

Parallelism (data, tensor, pipeline)

Definition: The strategies for splitting one training run across many GPUs: data parallelism replicates the model and splits the batch, tensor parallelism splits individual layers across GPUs, and pipeline parallelism splits layers into sequential stages on different GPUs.

No single GPU can hold or feed a modern run, so the question is how to carve it up with the least communication waste. Data parallelism is simple but replicates everything unless paired with ZeRO-style sharding of optimizer state, gradients, and parameters. Tensor parallelism communicates constantly, so it stays inside a node’s fast interconnect. Pipeline parallelism crosses nodes well but introduces bubbles where stages wait idle. Real runs compose all three (plus expert and context parallelism for MoE and long sequences), and the composition is a large part of what MFU measures.

In this study: Part 5’s systems backbone. The Ultra-Scale Playbook’s thousands of benchmarked configurations are the reference data, and the anchors’ framework choices (Megatron-LM for Nemotron, nanotron for Hugging Face) are compared there.

Deep dive: Hugging Face, The Ultra-Scale Playbook: Training LLMs on GPU Clusters, whose early chapters are Data Parallelism (including the ZeRO section) and Tensor Parallelism, with pipeline and combined strategies after; chapters verified in this pass. Stanford CS336, Lectures 7 and 8 (Parallelism 1 and 2), verified.

pass@k

Definition: The probability that at least one of k sampled solutions passes the check, usually unit tests; pass@1 is single-shot accuracy, and large k measures the model’s ceiling under repeated sampling.

pass@k comes from code evaluation, where a checker gives an exact verdict. Two companion disciplines matter. First, the estimate: honest pass@k uses the unbiased estimator from the Codex paper over many samples, not the naive formula. Second, the contrast with pass^k, the probability that all k tries succeed, which measures reliability instead of ceiling. A model with high pass@100 and low pass@1 is a brainstorming partner; a deployment that acts on outputs needs the other direction. Reasoning models are typically reported at pass@1 with averaging over multiple runs, which is yet a third convention, and Part 9 keeps them separate.

In this study: Part 9’s core metric vocabulary, and Part 11’s bridge to cyber: exploit and patch tasks are checkable, so pass@k with execution verification is the natural scoring shape for Proof of Defense style evals.

Deep dive: Chen et al., “Evaluating Large Language Models Trained on Code” (the Codex paper, arXiv:2107.03374, verified in this pass), which defines pass@k and its unbiased estimator. Stanford CS336, Lecture 12 (Evaluation), for the surrounding measurement discipline.

Positional encoding (RoPE, NoPE, YaRN)

Definition: The mechanism that tells attention where each token sits in the sequence: RoPE rotates query and key vectors by position, NoPE drops explicit position signals entirely, and YaRN is a rescaling method that stretches a RoPE model to longer contexts than it was trained on.

Attention by itself is order-blind, so position has to be injected somehow. RoPE became the default because it encodes relative distance elegantly and extrapolates moderately. NoPE is the counterintuitive finding that causal attention can infer position from the mask structure alone, and SmolLM3 adopted it after ablations favored it, pairing it with sliding window layers. YaRN matters at extension time: when a model trained at 8K must serve 128K, rescaling RoPE frequencies with YaRN-style interpolation is the standard, cheap route, usually with a short extension training stage on top.

In this study: Part 3 and Part 4 track the positional choice per family, and Part 6’s context extension stages (K2-Horizon to 512K, SmolLM3, OLMo Longmino) are where YaRN-class methods do their work.

Deep dive: Hugging Face, The Smol Training Playbook, whose ablation tags include NoPE among the tested run types for SmolLM3; verified generally in this pass. The original papers (RoFormer for RoPE, Peng et al. for YaRN, Kazemnejad et al. for NoPE) are the primaries; cited generally, not re-verified in this pass.

Post-training

Definition: Everything trained after pretraining and midtraining to turn a base model into a usable one: SFT, preference tuning, RL, distillation, and the safety and refusal layer.

A base model predicts text; a post-trained model follows instructions, shows its work, uses tools, and refuses or complies according to policy. Post-training uses orders of magnitude less compute than pretraining but decides most of what users experience, and it is where open families diverge most in philosophy: Tulu 3 publishes every dataset and stage, Nemotron publishes datasets and even teacher checkpoints, and the refusal layer is as much a training decision as the capability stages, a point Part 11 leans on for cyber models, where refusal behavior is a measurable property by vulnerability class.

In this study: Parts 7 and 8 split post-training into SFT and then preference, RL, and distillation. The seed terms SFT, DPO, GRPO, RLVR, MOPD, and on-policy distillation all live here.

Deep dive: Nathan Lambert, RLHF Book (rlhfbook.com), the whole arc from instruction tuning through reward modeling, RL, and direct alignment; chapter titles verified in this pass. Stanford CS336, Lectures 15 to 17 (Alignment: SFT/RLHF, then RL in two parts), verified.

Pretraining

Definition: The main training run, where a model learns language and knowledge by predicting the next token over trillions of tokens of broad data, before any instruction tuning.

Pretraining is where the money goes and where the model’s ceiling is set. Everything later is steering and sharpening; almost nothing in post-training adds raw knowledge the base does not have. The run is defined by a handful of giant decisions the rest of this glossary unpacks: data mix (Part 2), architecture (Part 4), optimizer and schedule, precision, parallelism, and token budget relative to size (scaling laws, below). Raschka’s definition is the cleanest mental model: pretraining builds the foundation model; fine-tuning is everything after.

In this study: Part 5 is the pretraining part, comparing the anchors’ runs size by size, including the deliberate overtraining of small models (SmolLM3’s 11T tokens on 3B parameters) against the Chinchilla ratio.

Deep dive: Sebastian Raschka, Build a Large Language Model (From Scratch), Chapter 5, “Pretraining on Unlabeled Data,” verified in this pass. Andrej Karpathy, “Let’s reproduce GPT-2 (124M)” (Zero to Hero series), a full small pretraining run in one video; title verified in this pass, and his nanoGPT and nanochat repositories are the runnable versions.

RLVR

Definition: Reinforcement learning with verifiable rewards: the reward comes from a checker that can prove an answer right or wrong (tests pass, proof verifies, exploit works, patch holds), not from a learned preference model.

Preference rewards are opinions, and models learn to game opinions. A verifiable reward cannot be flattered: the code compiles and the tests pass, or they do not. That makes RLVR both more robust and more scalable, since checkers are free and tireless, and it is the engine behind the reasoning-model era (math, code) and behind GRPO’s rise. Its boundary is the set of tasks with reliable checkers, which is exactly why it matters so much for cyber, where sandboxes can execute an exploit or validate a patch.

In this study: Part 8 compares RLVR recipes per family, and Part 11 builds the cyber eval argument on it: Proof of Defense is RLVR-shaped measurement, attack success and fix survival checked by execution.

Deep dive: Nathan Lambert, RLHF Book, the “Reinforcement Learning” chapter, for RLVR in the modern post-training stack; cited by chapter, verified generally in this pass. Stanford CS336, Lecture 16 (Alignment RL 1) covers RL on checkable answers; verified.

Scaling laws and tokens per parameter

Definition: Scaling laws are the empirical power-law relationships between a model’s loss and its size, data, and compute; the Chinchilla result is their most famous consequence, that compute-optimal training uses roughly 20 training tokens per parameter.

Kaplan’s 2020 laws told the field to grow models faster than data. Hoffmann’s Chinchilla study (400-plus controlled runs) corrected the constants: at fixed compute, parameters and tokens should grow together, and the era’s flagship models were badly undertrained. The modern twist is that almost nobody trains at the Chinchilla point anymore, deliberately: inference cost dominates lifetime cost, so small models are overtrained far past 20 tokens per parameter (SmolLM3 runs about 3,600) to buy a cheaper, faster model at serving time. Tokens per parameter is thus a strategy dial, not a law of nature: low for one-off giants, high for models that will serve billions of queries or run on a laptop.

In this study: Part 5’s budgeting lens for every roster row, from K2-Horizon’s long small-model runs to Nemotron-4 340B’s 9T tokens. The matched-size tables in Piece 0 only make sense with each model’s ratio printed next to its size.

Deep dive: Hoffmann et al., “Training Compute-Optimal Large Language Models” (arXiv:2203.15556, verified in this pass), and Kaplan et al., “Scaling Laws for Neural Language Models” (arXiv:2001.08361, verified). Stanford CS336, Lectures 9 and 11 (Scaling Laws 1 and 2), verified, include fitting laws yourself in the Scaling assignment.

SFT

Definition: Supervised fine-tuning: training a model on example inputs paired with their desired outputs, so it learns the format and behavior of an assistant, a coder, or a specialist.

SFT is the first post-training stage in nearly every recipe and the one that teaches surface behavior: how to answer, how to format, when to show reasoning, which persona to hold. Its data is the recipe: Tulu 3, Dolci, SmolTalk, Nemotron’s post-training sets, and K2-Horizon’s three-phase SFT are all, at bottom, arguments about which examples to show the model and in what order. SFT’s limits are equally consistent across reports: it imitates the data’s ceiling, it can teach style faster than substance, and it cannot by itself produce the exploration gains that RL adds.

In this study: Part 7 is the SFT part, comparing instruction data construction per family, including Nemotron’s specialist SFT on other labs’ bases, which isolates what SFT alone contributes at fixed size.

Deep dive: Sebastian Raschka, Build a Large Language Model (From Scratch), Chapter 7, “Fine-tuning to Follow Instructions,” verified in this pass. Nathan Lambert, RLHF Book, the early post-training chapters on instruction tuning; cited generally.

Sliding window attention

Definition: Attention restricted to a local window of recent tokens in most layers, with occasional full-attention layers kept for long-range recall.

Full attention pays quadratic cost and an ever-growing KV cache. A sliding window caps both: each token sees, say, the last few thousand tokens, which covers most of what language actually depends on, and the interleaved global layers carry the long-range information. SmolLM3’s hybrid pattern is the studied example, and the design pairs naturally with NoPE in its ablations. The failure mode to watch is exact long-range retrieval, which is why families report needle-style long-context tests alongside the speed claims.

In this study: Part 4 counts sliding window hybrids as a third way between full attention and hybrid Mamba, and Part 10 prices the cache savings.

Deep dive: Hugging Face, The Smol Training Playbook, attention architecture sections, for the SmolLM3 design rationale; verified generally in this pass. The pattern descends from Mistral 7B’s sliding window design; cited generally.

Speculative decoding

Definition: A serving technique where a small, fast draft model proposes several tokens and the large target model verifies them in one parallel pass, accepting the longest correct prefix, for output identical to sampling from the large model alone.

Decoding is serial and memory-bound: one token per full pass, with the GPU mostly waiting on weight reads. Speculation converts that waste into parallel verification, typically buying 2 to 3x speed with zero quality change, because the acceptance rule provably preserves the target distribution. The draft can be a separate small model, an early-exit head, or the model’s own MTP heads, which is why MTP-trained families get speculation almost for free.

In this study: Part 10, where it interacts with quantization and KV cache economics to decide real serving cost per token for the anchors, including on the DGX Spark class hardware LAB2 plans around.

Deep dive: Leviathan, Kalman, and Matias, “Fast Inference from Transformers via Speculative Decoding” (arXiv:2211.17192, verified in this pass); the independent DeepMind version is Chen et al., arXiv:2302.01318, verified. Stanford CS336, Lecture 10 (Inference), for placement in the serving stack.

Synthetic data

Definition: Training data generated by models rather than written by humans or scraped from the web: teacher-written instructions, rewritten web text, generated math and code problems, simulated agent trajectories.

Synthetic data is how the field escaped the limits of the public web. It supplies volume where human data is scarce (hard math, tool use, agent traces), quality where web data is dirty (textbook-style rewrites), and control where balance matters (deliberate coverage of a skill). Its risks are equally documented: errors and biases of the generating model get baked in, diversity can collapse toward the teacher’s habits, and the legal terms of the teacher decide whether the data may be used for training at all. Nemotron’s corpora are heavily synthetic by design; OLMo’s Dolma line leans natural with synthetic used surgically, and that contrast is a Part 2 headline.

In this study: Part 2 (pretraining mixes), Part 7 (SFT data), and Part 8 (agent-data synthesis at scale in the Nemotron and K2-Horizon recipes) all turn on where each family draws the synthetic line.

Deep dive: the Nemotron-CC and Nemotron post-training write-ups for the synthetic-heavy position, and the Dolma and FineWeb write-ups for the natural-data position; all cited generally here, with per-family specifics handled in Part 2.

Tokenizer and BPE

Definition: The tokenizer converts text to token IDs and back; BPE (byte pair encoding) is the standard algorithm that learns the vocabulary by repeatedly merging the most frequent byte pairs in a training corpus.

Tokenization is the model’s sensory system, and its decisions are permanent once training starts. Vocabulary size trades embedding parameters (Part 3’s accounting) against sequence length: a bigger vocabulary encodes the same text in fewer tokens, which is cheaper per document and better for code and multilingual text, at the cost of a bigger embedding table and sparser training per token. The roster spans 32K to 256K vocabularies, and that spread alone explains part of several families’ parameter counts and multilingual scores. Tokenizer quirks (how numbers, whitespace, and code indentation split) quietly shape model behavior in ways benchmarks later reveal.

In this study: Part 3 is built on this entry, including the K2-Horizon nameplate lesson that headline sizes often exclude embeddings.

Deep dive: Andrej Karpathy, “Let’s build the GPT Tokenizer” (Zero to Hero), with the companion minbpe repository; title and repo verified in this pass. Sebastian Raschka, Build a Large Language Model (From Scratch), Chapter 2, “Working with Text Data,” which implements BPE and the data pipeline; verified. Stanford CS336, Lecture 1 (Overview and Tokenization), verified.