Model Making, in the Open: A Masterclass

A stage-by-stage comparison of how modern open models are actually made, built from the families that publish enough to be diffed: NVIDIA Nemotron and IFM K2-Horizon as anchors, Kimi K2 as the contrast case, and the fully open supporting cast.

Nothing matches that search.

A stage-by-stage comparison of how modern open models are actually made, built from families that publish enough to be diffed: weights, data, training code, checkpoints, and the reasoning in their tech reports. Every part takes one stage of the pipeline and lines the families up at that stage: who did what, what the alternative was, what it cost, what it bought, and what the published numbers say the difference was worth.

Two anchor families carry the spine: NVIDIA Nemotron, the industrial anchor, and IFM K2-Horizon, the transparency anchor (not Moonshot's Kimi K2; the study corrects that on day one). Kimi K2 is the contrast case, and OLMo, SmolLM, Apertus, Marin, LLM360, and Granite form the fully open supporting cast. Every part ends with the cyber read: what the stage means for building cyber models. Engineer voice throughout; every load-bearing term is in the glossary.

Piece 0
Piece 0: Map and Spine
The map, the anchors, and the comparison spine.
Part 1
1. Openness
What open means per family, graded honestly, license traps included.
Part 2
2. Data
Corpus building, filtering, mixes, and the token budget each size actually got.
Part 3
3. Tokenizers
Vocabulary sizes from 32K to 256K: what a bigger vocabulary costs and buys.
Part 4
4. Architecture
Dense vs MoE vs hybrid Mamba, with the 30B band as the main exhibit.
Part 5
5. Pretraining
Optimizers, schedules, stability incidents, precision, and who overtrains on purpose.
Part 6
6. Midtraining
Annealing mixes, model souping, and context extension: the stage most releases hide.
Part 7
7. SFT
Instruction data construction, persona and format decisions, and what SFT cannot fix.
Part 8
8. RL and Distillation
DPO, GRPO, RLVR, multi-teacher distillation, and refusal as a training decision.
Part 9
9. Evaluation
How each family proves its claims, and what execution-verified evaluation demands.
Part 10
10. Serving
Quantization paths, KV cache economics, speculative decoding, and the engine layer.
Part 11
11. Cyber
Every prior part applied to cyber model building and Proof of Defense.
Part 12
12. Synthesis
Decision tables: which published recipe to start from for each size class.
Reference
Glossary
Every load-bearing term: plain definition, quick explainer, long explainer, deep-dive pointers.
Appendix A
Appendix: Nemotron Roster
The complete open Nemotron roster: every size, license, and public training artifact.
Appendix B
Appendix: K2-Horizon Roster
K2-Horizon identified and rostered, with the Kimi K2 family as the comparison base.
Appendix C
Appendix: Open Families Roster
The supporting cast of fully open families, graded on what they actually publish.

The comparison spine: matched sizes

A dense model has one size. A Mixture-of-Experts model has two: total parameters (what you store and load) and active parameters (what computes each token, and what your speed and cost feel like). Every comparison in this study states both. Full treatment in Piece 0.

BandThe fightWhy it matters
Sub-1B to 2BSmolLM2, OLMo 2 1B, K2-Horizon 0.9BThe fully open world owns this band. Cheap enough to retrain, so every claim is rerunnable.
3B compute bandSmolLM3 and Granite 4.0 Micro (dense) vs Nemotron 3 Nano and Moonlight (3B active, MoE)Same compute per token, four different answers to where the parameters should live.
7B to 9B denseOLMo 2 and 3, Marin 8B, Apertus 8B, DCLM 7B, Granite 4.1 8B, K2-Horizon 7B, Nemotron Nano 9B v2The most crowded and best documented band, with controlled experiment sets inside the Nemotron line.
22B to 32B shelf bandOLMo 32B, K2-Horizon 32B and 36B MoVA, Nemotron 3 Nano 30B-A3B, 3.5 Lightning; Kimi K2 computes here at 32B activeThe masterclass sweet spot: same shelf size, wildly different recipes.
65B to 70BApertus 70B vs LLM360 K2-65BSame size, same openness grade, different eras. NVIDIA has no from-scratch entry here, which is itself a finding.
100B to 120BNemotron 3 Super 120B-A12B, aloneA genuine gap: no fully open family publishes a complete pipeline at this size.
340B and upNemotron-4 340B vs Nemotron 3 Ultra 550B-A55B, K2-Horizon 375B between them, Kimi K2 at 1TOld flagship recipe against modern recipe, where the openness gap is widest.

Start with Piece 0, or open any part. Search loads a plain-text index of all parts, glossary entries, and rosters on first use.