Model Making, in the Open: A Masterclass

Piece 0 · Piece 0: Map and Spine

Piece 0 · Piece 0: The Map, the Anchors, and the Comparison Spine

Listen: audio twin

Model-Making Masterclass, Piece 0. Written Nov 17, 2026. Engineer voice by design: plain language, every term defined when it first matters, pointers to the deep references at the end.

What this study is

A stage-by-stage comparison of how modern open models are actually made, built from families that publish enough to be diffed: weights, data, training code, checkpoints, and the reasoning in their tech reports. Every part takes one stage of the pipeline (data, tokenizer, architecture, pretraining, midtraining, post-training, evaluation, serving) and lines the families up at that stage: who did what, what the alternative was, what it cost, what it bought, and what the published numbers say the difference was worth.

Two anchor families carry the spine. The supporting cast fills the size bands the anchors leave empty and supplies the cleanest controlled comparisons.

The anchors, corrected on day one

Anchor 1: NVIDIA Nemotron. Not one model, a five-year stack of them: 50 open checkpoints from 1.5B to a 550B flagship, dense models, hybrid Mamba models, MoE models, diffusion experiments, and a long line of post-training specialists built on other labs’ bases (Llama, Qwen, Gemma). NVIDIA publishes tech reports, pretraining datasets (the Nemotron-CC line), post-training datasets, and in the Nemotron 3 generation, even the teacher checkpoints used for distillation. The openness is real but uneven, and license terms vary per checkpoint in ways that matter commercially. The roster file grades every entry.

Anchor 2: K2-Horizon. First correction: this is not Moonshot’s Kimi K2. K2-Horizon is the open fleet from the K2 Team at the Institute of Foundation Models (MBZUAI, Abu Dhabi), the LLM360 group. Six models, all Apache 2.0, all 512K context: 0.9B, 3.7B, 7B, and 32B dense, a 36B MoE built around a new attention design (Mixture-of-Value Attention), and a 375B flagship with 23B active parameters. Behind the weights sits the most complete public training record in the field: about 380 checkpoints including per-stage branches, 58 training runs, training code (xLLM), RL code (RL360), an eval harness (Eval360), and a 134-page tech report published Nov 16, 2026. If Nemotron is the industrial anchor, K2-Horizon is the transparency anchor.

Scope boundary. This study only covers releases open enough to learn from: data, training code, and checkpoints public, not weights alone. Releases that publish weights and a report but keep their data, code, and intermediate checkpoints closed are outside the scope, however strong the models are. One naming warning belongs here because it has already caused confusion once: K2-Horizon (IFM, above) is not Moonshot’s Kimi K2. Different company, different family, no relation beyond the colliding name; Moonshot’s line is one of the weights-and-report releases this study leaves out.

Supporting cast (fully open pipelines): Ai2’s OLMo 2 and OLMo 3 plus the Tulu 3 post-training line, Hugging Face’s SmolLM2 and SmolLM3, Apertus, Marin, LLM360’s earlier models, and the DCLM/FineWeb data line. IBM Granite is included deliberately as the boundary case: weights and recipe public, data only partly, so the edge of “fully open” stays visible.

The comparison spine: matched sizes

Sizes match two ways, and confusing them is the most common error in model comparisons. A dense model’s size is one number. A Mixture-of-Experts model (defined in the glossary seed below) has two: total parameters (what you store and load) and active parameters (what actually computes each token, which is what your speed and cost feel like). A 30B MoE with 3B active sits in the 30B band on the shelf and in the 3B band on the compute bill. Every comparison in this study states both.

The bands, with the fights that matter:

  • Sub-1B to 2B. The fully open world owns this band outright (SmolLM2, OLMo 2 1B, K2-Horizon 0.9B). Neither anchor’s main line plays here. Teaching value: these are cheap enough to retrain, so every claim is rerunnable.
  • 3B compute band. A clean contrast: SmolLM3 (dense 3B) and Granite 4.0 Micro (dense 3B) against an MoE that computes like a 3B model, Nemotron 3 Nano (3B active). Same compute per token, three different answers to where the parameters should live.
  • 7B to 9B dense. The most crowded and best documented band: OLMo 2 7B, OLMo 3 7B, Marin 8B, Apertus 8B, SmolLM3’s bigger siblings, LLM360’s 7B pair, DCLM 7B, Granite 4.1 8B, K2-Horizon 7B, and NVIDIA’s hybrid Nano 9B v2. Eight-plus independent recipes at one size. Inside the Nemotron line alone, this band holds a controlled experiment set: a from-scratch hybrid pretrain, a Llama base with post-training only, and Qwen derivatives that isolate what RL adds over SFT at fixed size.
  • 22B to 32B shelf band. The masterclass sweet spot. OLMo 2 and 3 at 32B (dense, very long training), K2-Horizon 32B dense and its 36B MoVA MoE, Nemotron 3 Nano 30B-A3B and 3.5 Lightning (hybrid Mamba MoE), Granite’s MoE entries.
  • 65B to 70B. Apertus 70B against LLM360’s K2-65B: same size, same openness grade, different eras (about 15T compliance-first tokens vs 1.4T 2023-era tokens). Nemotron’s entry here is a Llama derivative, not an NVIDIA pretrain, and that absence is itself a finding: NVIDIA skipped from-scratch dense at this size entirely.
  • 100B to 120B. A genuine gap. Nemotron 3 Super (120B total, 12B active) stands alone; no fully open family publishes a complete pipeline at this size.
  • 340B and up. Nemotron-4 340B (dense, 9T tokens, the old flagship recipe) against Nemotron 3 Ultra (550B total, 55B active, the modern recipe), with K2-Horizon 375B as the fully open reference between them. Above this band sits the trillion-parameter class, where no family in this study publishes a full pipeline; the study’s ladder tops out at 550B.

The part map

  • Piece 0 (this file). Map, anchors, spine, glossary seed.
  • Part 1. Openness, graded honestly. What “open” means per family: weights, license terms per checkpoint, data, code, checkpoints, logs, eval harness. Includes the license traps found in the rosters (guardrail-termination clauses, internal-R&D tags, inherited upstream licenses) and why LAB2 reads license text before benchmarks.
  • Part 2. Data. Corpus building per family: web crawl processing (Nemotron-CC, Dolma, FineWeb line), quality filtering, dedupe, decontamination, synthetic data shares, domain mixes, and the token budgets each size actually got. Why data decisions explain more performance variance than architecture at these scales.
  • Part 3. Tokenizers and vocabularies. Vocabulary sizes from 32K to 256K across the families, what a bigger vocabulary costs and buys, and the K2-Horizon nameplate lesson (a “7B” that is 9B with embeddings).
  • Part 4. Architecture. Dense vs MoE vs hybrid Mamba-Transformer, attention variants (grouped-query, multi-head latent, Mixture-of-Value, sliding window hybrids, NoPE), multi-token prediction, expert counts and routing. The 30B band as the main exhibit: three architectures at one shelf size.
  • Part 5. Pretraining. Optimizers (AdamW vs Muon), schedules, batch sizes, stability incidents and fixes, parallelism strategies, MFU (model flops utilization, defined in the glossary), precision (BF16, FP8, NVFP4), and the token budgets per size, including who overtrains small models on purpose and why.
  • Part 6. Midtraining and annealing. The stage most releases hide: quality annealing mixes (Dolmino, Longmino), model souping (averaging weights of multiple runs) as an anneal, context extension stages (K2-Horizon’s four-stage midtrain, OLMo’s Longmino, SmolLM3’s 100B-token extension), and continued pretraining for domains.
  • Part 7. Post-training I: SFT. Instruction data construction per family (Tulu 3, Dolci, SmolTalk, Nemotron post-training sets, K2-Horizon’s three-phase SFT), persona and format decisions, and what SFT alone can and cannot fix.
  • Part 8. Post-training II: preference and RL. DPO, GRPO, RLVR (RL with verifiable rewards), Cascade RL, on-policy distillation, multi-teacher distillation (Nemotron 3 Ultra’s MOPD from 10+ public teachers), expert merging (K2-Horizon’s five RL experts merged into one model), and the refusal/safety layer as a training decision, including abliteration and guardrail stripping as documented practices.
  • Part 9. Evaluation. How each family proves its claims: harness choices, contamination discipline, private holdouts, pass@k vs pass^k, the benchmarketing failure modes found in this week’s intake (highlight-reel charts, estimated rival scores), and what execution-verified evaluation means for cyber models specifically.
  • Part 10. Serving and inference. Quantization paths (INT4, FP8, NVFP4, GGUF), KV cache economics (including the Maxwell cautionary tale), speculative decoding and MTP at serving time, and the engine layer (vLLM, SGLang, llama.cpp, MLX).
  • Part 11. The cyber specialization. Applying every prior part to cyber model building: domain data (CVE corpora, patch diffs, exploit traces), refusal measurement by vulnerability class, the Trident Poseidon and CyberQwen case studies from intake, LAB2’s ladder mapped onto the anchors’ recipes, and what Proof of Defense demands that public benchmarks do not.
  • Part 12. Synthesis. The decision tables: if you were building each size class today, which published recipe you would start from, what you would change, and the open questions the field has not answered in public.

Parts land in order where dependencies require it (glossary and openness first) and in parallel where they do not. Each part ships as site pages plus an audio twin on the private podcast feed.

Glossary seed (the wiki grows from here)

  • Active parameters. The parameters that actually compute on a given token. In an MoE model only some experts fire per token, so active is far below total. Active size drives speed and cost; total size drives memory.
  • Annealing. The final phase of pretraining where the learning rate decays toward zero, often while the data mix shifts to the highest-quality slice. Several families treat the anneal mix as their secret sauce; the open ones publish it.
  • Decontamination. Removing evaluation material (and near-duplicates) from training data so benchmark scores mean something. Method and rigor vary; Part 2 compares them.
  • GRPO. Group Relative Policy Optimization, a reinforcement learning method that scores a group of sampled answers against each other instead of training a separate value model. Popularized by DeepSeek, now the default RL recipe in much of the open world.
  • MFU (model flops utilization). The share of the hardware’s theoretical peak compute a training run actually achieves. The honest efficiency number for a training setup; good runs land roughly in the 30 to 55 percent band depending on model shape.
  • Midtraining. Training between the main pretraining run and post-training: annealing, context extension, and domain continued-pretraining all live here. The least standardized stage, and where open families differ most visibly.
  • Mixture of Experts (MoE). An architecture where each layer holds many expert subnetworks and a router sends each token to a few of them. Total parameters grow (knowledge capacity) while per-token compute stays low.
  • Model souping. Averaging the weights of several training runs (or checkpoints) into one model. Used by OLMo as its annealing method; cheap, and it works better than it has any right to.
  • MOPD. Multi-teacher on-policy distillation: a student model learns from several teacher models on the student’s own generated outputs. Nemotron 3 Ultra’s post-training centerpiece.
  • Multi-token prediction (MTP). Training the model to predict several future tokens at once, not just the next one. Improves learning signal density and, at serving time, gives a built-in draft for speculative decoding.
  • RLVR. Reinforcement learning with verifiable rewards: the reward comes from a checker (tests pass, answer matches, proof verifies) rather than a learned preference model. The engine behind most reasoning-model gains, and the pattern cyber evals need.
  • SFT. Supervised fine-tuning: training on example inputs with their desired outputs. The first post-training stage in nearly every recipe.
  • Sliding window attention. Attention restricted to a local window of recent tokens in most layers, with occasional full-attention layers. Trades some long-range recall for large speed and memory wins at long context.

The full wiki adds a long explainer and curated videos, books, and demos per term as the parts that use them land.

Open items the parts must resolve

From the roster verification passes, flagged honestly: a license conflict on Nemotron-H (an internal-R&D tag vs the Open Model License in redistributions) that decides commercial usability; Nemotron 3 Ultra’s pretraining token count (about 20T on the card vs about 10T plus 212B in a blog-derived summary); K2-Horizon’s 32B final-state diff against its stage branches; OLMo 3’s 5.5T vs 5.9T mix naming inside Ai2’s own materials. Each part closes its own items at primary sources or keeps them flagged.

The cyber frame

This study exists because of the 2027 goal: train the best cyber models in the world, in the open, with LAB2 as the vehicle. Every part therefore ends with the cyber read: what this stage’s decisions mean when the domain is vulnerabilities, exploits, and defense, and which anchor recipe LAB2 should steal from, adapt, or avoid. Part 11 collects the full application.

Sources for this piece

Roster files built for this masterclass on Nov 17, 2026, each verified at primary sources with gaps flagged in place: research/nemotron-roster.md, research/k2-horizon-roster.md, research/open-families-roster.md (same folder tree as this file). Key primaries behind them: NVIDIA Hugging Face cards and Nemotron tech reports; IFM K2-Horizon tech report (arXiv:2611.21003) and model cards; Ai2 OLMo 2/3 and Tulu 3 reports; Hugging Face SmolLM2/3 posts and the Smol Training Playbook; Apertus tech report.