Model Making, in the Open: A Masterclass

A stage-by-stage comparison of how modern open models are actually made, built from the families that publish enough to be diffed: NVIDIA Nemotron and IFM K2-Horizon as anchors, and the fully open supporting cast.

Nothing matches that search.

A stage-by-stage comparison of how modern open models are actually made, built from families that publish enough to be diffed: weights, data, training code, checkpoints, and the reasoning in their tech reports. Every part takes one stage of the pipeline and lines the families up at that stage: who did what, what the alternative was, what it cost, what it bought, and what the published numbers say the difference was worth.

Two anchor families carry the spine: NVIDIA Nemotron, the industrial anchor, and IFM K2-Horizon, the transparency anchor (not Moonshot's Kimi K2; a different company and family, outside this study's scope). OLMo, SmolLM, Apertus, Marin, LLM360, and Granite form the fully open supporting cast. Every part ends with the cyber read: what the stage means for building cyber models. Engineer voice throughout; every load-bearing term is in the glossary.

New to the machinery? Start with the prerequisite course
The ground-up companion for software and security engineers: the AI background this study assumes, taught before the diffing starts.
Piece 0
Piece 0: Map and Spine
The map, the anchors, and the comparison spine.
Part 1
1. Openness
What open means per family, graded honestly, license traps included.
Part 2
2. Data
Corpus building, filtering, mixes, and the token budget each size actually got.
Part 3
3. Tokenizers
Vocabulary sizes from 32K to 256K: what a bigger vocabulary costs and buys.
Part 4
4. Architecture
Dense vs MoE vs hybrid Mamba, with the 30B band as the main exhibit.
Part 5
5. Pretraining
Optimizers, schedules, stability incidents, precision, and who overtrains on purpose.
Part 6
6. Midtraining
Annealing mixes, model souping, and context extension: the stage most releases hide.
Part 7
7. SFT
Instruction data construction, persona and format decisions, and what SFT cannot fix.
Part 8
8. RL and Distillation
DPO, GRPO, RLVR, multi-teacher distillation, and refusal as a training decision.
Part 9
9. Evaluation
How each family proves its claims, and what execution-verified evaluation demands.
Part 10
10. Serving
Quantization paths, KV cache economics, speculative decoding, and the engine layer.
Part 11
11. Cyber
Every prior part applied to cyber model building and Proof of Defense.
Part 12
12. Synthesis
Decision tables: which published recipe to start from for each size class.
Part 13
13. Nemotron Generations
The anchor family generation by generation: what changed, what improved, and what each report credits for the gain.
Reference
Glossary
Every load-bearing term: plain definition, quick explainer, long explainer, deep-dive pointers.
Appendix A
Appendix: Nemotron Roster
The complete open Nemotron roster: every size, license, and public training artifact.
Appendix B
Appendix: K2-Horizon Roster
K2-Horizon identified and rostered: six models, the full stage record, and the name-collision note.
Appendix C
Appendix: Open Families Roster
The supporting cast of fully open families, graded on what they actually publish.

The comparison spine: matched sizes

A dense model has one size. A Mixture-of-Experts model has two: total parameters (what you store and load) and active parameters (what computes each token, and what your speed and cost feel like). Every comparison in this study states both. Full treatment in Piece 0.

BandThe fightWhy it matters
Sub-1B to 2BSmolLM2, OLMo 2 1B, K2-Horizon 0.9BThe fully open world owns this band. Cheap enough to retrain, so every claim is rerunnable.
3B compute bandSmolLM3 and Granite 4.0 Micro (dense) vs Nemotron 3 Nano 9B v2 (MoE generation at this compute)Same compute per token, four different answers to where the parameters should live.
7B to 9B denseOLMo 2 and 3, Marin 8B, Apertus 8B, DCLM 7B, Granite 4.1 8B, K2-Horizon 7B, Nemotron Nano 9B v2The most crowded and best documented band, with controlled experiment sets inside the Nemotron line.
22B to 32B shelf bandOLMo 32B, K2-Horizon 32B and 36B MoVA, Nemotron 3 Nano 30B-A3B, 3.5 LightningThe masterclass sweet spot: same shelf size, wildly different recipes.
65B to 70BApertus 70B vs LLM360 K2-65BSame size, same openness grade, different eras. NVIDIA has no from-scratch entry here, which is itself a finding.
100B to 120BNemotron 3 Super 120B-A12B, aloneA genuine gap: no fully open family publishes a complete pipeline at this size.
340B and upNemotron-4 340B vs Nemotron 3 Ultra 550B-A55B, K2-Horizon 375B between themOld flagship recipe against modern recipe. Nothing fully open exists at the trillion class; the ladder tops out at 550B.

Start with Piece 0, or open any part. Search loads a plain-text index of all parts, glossary entries, and rosters on first use.