Model Making, in the Open: A Masterclass

Part 12 · 12. Synthesis

Part 12 · Part 12: Synthesis, the Decision Tables

Listen: audio twin

Model-Making Masterclass, Part 12. Written Nov 17, 2026. This part does no new research. It compresses Parts 1 to 11 into build decisions, ranks what the study actually established, lists what it could not, and sets the rules for keeping the whole thing current. Every claim below traces to an earlier part; the part number is cited wherever a decision or a finding rests on one. Anything that would not trace was left out or flagged in place.

How to read the tables

One table per size class on the Piece 0 spine. Rows are pipeline stages. Each cell names the starting recipe this study recommends, drawn from a family that published enough to copy, plus the one change most worth making and why. “Start from” means start from the published recipe, not from the weights, unless the cell says weights. Where a band has no fully open pipeline, the table says so instead of dressing up a derivative as a recipe.

Sub-2B: the teaching band

Stage Start from Change and why
Base and recipe SmolLM2’s published pipeline, end to end (Part 2) None at this size. It is the cheapest complete recipe in the study and every claim in it is rerunnable
Data posture Classifier-filtered blend in the FineWeb-Edu and DCLM style, quality tiers kept as a dial (Part 2) Keep tiers instead of one threshold; the anneal needs the strict cut later (Part 6)
Architecture Dense, tied embeddings, deliberately small vocabulary (49,152 in SmolLM2) (Part 3) Do not import a 250K vocabulary here; the embedding pair would eat a fifth of the model (Part 3)
Midtraining move Short quality anneal on the strictest tier (Part 6) Soup two or three anneal runs if the budget allows; at this size it is nearly free (Part 6)
Post-training shape SmolTalk-style SFT, small and curated (Part 7) Resist scale. Format and persona are what SFT buys at this size; capability was decided upstream (Part 7)
Eval discipline OLMES-style fixed instrument, published configs (Part 9) Report standard errors; at this size most leaderboard gaps are inside the noise floor (Part 9)
Serving artifact Q4-class GGUF or INT8 build, measured at the shipped precision (Part 10) Never quote the BF16 score for the quantized artifact (Part 10)

3B compute band: dense 3B and the 3B-active MoEs

Stage Start from Change and why
Base and recipe SmolLM3 for dense; Granite 4.0 Micro as the enterprise-shaped alternative (Parts 2, 4) Treat Nemotron 3 Nano (3B active) as the MoE reading of this band, not as a 30B model, when comparing compute (Piece 0)
Data posture Staged mixes that specialize as training proceeds; SmolLM3 moves code 12 to 24 percent and math 3 to 13 percent across stages (Part 2) Publish the stage percentages; this band is where mix staging is best documented (Part 2)
Architecture Dense with a NoPE pattern (SmolLM3) or a hybrid ratio (Granite) (Part 4) Pick based on context length you must serve, not on benchmark shape; the cache bill is decided here (Parts 4, 10)
Midtraining move Long-context extension as its own stage, about 100B tokens in SmolLM3’s case (Part 6) Rerun the short-context suite after extension; extension gains can be damaged by later alignment and repaired by weight merging (Part 6)
Post-training shape Single iterated SFT mixture, Tulu style, then light preference work (Parts 7, 8) Keep the safety share deliberate; tiny shares set refusal behavior out of proportion to their size (Part 7)
Eval discipline Dev suite for iteration, unseen suite for reporting, never merged (Part 9) The Tulu IFEval case is the warning: dev up 19 points while unseen went nowhere (Part 9)
Serving artifact The artifact measured is the artifact shipped; INT4 only with quantization-aware evidence behind it (Part 10) Reasoning quality degrades first under aggressive post-training quantization (Part 10)

7B to 9B: the documented band, and LAB2 Step A

Stage Start from Change and why
Base and recipe OLMo 3 7B when attribution matters most; Granite 4.1 8B when the license chain matters most (Part 1) Run both under one recipe if you need to separate recipe effects from base effects; Part 11 makes OLMo 3 the control for exactly this reason
Data posture Dolma-way corpus discipline: named sources, per-source licenses, published dedup and decontamination (Part 2) Budget curation before compute; at this band curation is the binding constraint (Part 6)
Architecture Inherit the base’s body; dense and boring is a feature at this step (Part 4) Spend no design effort here. Architecture selection criteria matter at Step B, not Step A (Part 11)
Midtraining move Dolmino-shaped anneal: roughly half general replay, patch slices upsampled and repeated, micro-annealed per source before committing (Part 6) Soup the anneal across data orders for the gain and the free variance estimate (Part 6)
Post-training shape Taxonomy-first SFT on verified traces, then a short high-quality final phase at decaying rate (Part 7) Keep per-stage checkpoints and per-phase refusal metrics, K2-Horizon style (Parts 7, 8)
Eval discipline Execution-verified suite, training harness and eval harness as separate artifacts (Part 9) Add per-vulnerability-class refusal reporting from the first run, not after the first surprise (Part 9)
Serving artifact QLoRA adapters on the chosen base for iteration; full artifact for the published card (Parts 10, 11) Score the untouched base first so every later number has a denominator (Part 11)

22B to 32B: the sweet spot, and LAB2 Step B

Stage Start from Change and why
Base and recipe The band’s three published body plans are the menu: K2-Horizon 32B dense, MoVA-36B-A4B, Nemotron 3 Nano and 3.5 Lightning (Part 4) Choose by which metric you are buying: loss per token (dense), quality per FLOP (MoVA), throughput and context per serving dollar (hybrid). The study found three different winners (Part 4)
Data posture One shared corpus across sizes if you are running a scaling comparison, K2-Horizon style (Part 2) For a product line, diverge the mixes per size instead; the shared corpus is a study design, not a product design (Part 2)
Architecture Hybrid Mamba-MoE if long agent sessions on owned hardware are the deployment (Parts 4, 10) Add an exact-match retrieval eval with hex and hash needles before trusting any hybrid at this size (Part 4)
Midtraining move Staged context extension with two-thirds general replay in the extension mix (Part 6) Buy a verified 128K before advertising anything larger (Part 6)
Post-training shape Cascade-shaped sequenced RL with execution rewards, MOPD as regression insurance (Part 8) At this size RL stops being a proof of loop closure and becomes the stage where capability is bought; verifier quality becomes the ceiling (Part 11)
Eval discipline Private rotating holdouts must outgrow the public suite at this band (Parts 9, 11) A 30B-class release is the one outsiders benchmark; a public suite everyone trains against measures diligence, not capability (Part 11)
Serving artifact 30B-A3B at BF16 or NVFP4 on Spark-class hardware is the largest shape that serves well there (Part 10) Measure capability on your own tasks, not the vendor card’s framing; Lightning’s card sells cost per task while conceding peak capability (Parts 10, 11)

65B to 70B: the thin band

Stage Start from Change and why
Base and recipe Apertus 70B for a fully open pipeline; LLM360 K2-65B as the era contrast (Piece 0, Part 1) Do not mistake Llama-Nemotron 70B for a recipe at this size; it is a post-training line on someone else’s pretrain (Part 1)
Data posture Apertus’s compliance-first posture (license, toxicity, PII filters; quality by coverage) against K2-65B’s 1.4T 2023-era tokens (Parts 1, 2) The pair is the study’s cleanest same-size, different-era data comparison; use it to price what a decade of data practice bought (Piece 0)
Architecture Dense, both entries (Part 4) Nobody in the study ships a fully open sparse pipeline here; if you need MoE economics at 70B, you are leaving the fully open set (Piece 0)
Midtraining move Apertus’s staged curriculum shape (Part 6) Granite’s decay-stage shares are unpublished, so do not borrow numbers from that direction (Part 6)
Post-training shape Tulu 3’s recipe is the public post-training reference that reaches this size (Part 8) Remember the weights inherit the base’s license; the recipe is open, the chain may not be (Part 1)
Eval discipline Same as the 32B band, with extra suspicion of cross-era comparisons (Part 9) A 2023-budget model and a 15T-token model at one size is a data comparison wearing a size label (Part 2)
Serving artifact Dense 70B crawls on bandwidth-bound hardware, single-digit tokens per second at 4-bit on Spark class (Part 10) If the deployment is one workstation, this band is the wrong band; drop to 30B-A3B (Part 10)

100B to 120B: the gap band

Stage Start from Change and why
Base and recipe Nemotron 3 Super is the only entry, and it is a recipe plus weights, not a full pipeline (Parts 1, 5) Treat every number as single-source. No second family publishes at this size to check it against (Piece 0)
Data posture NVIDIA’s two-phase shape, 20T diversity then 5T quality, at corpus-family granularity (Parts 2, 6) The phase-2 internal mix is not published at ingredient level; plan your own anneal rather than copying a summary (Part 6)
Architecture Hybrid LatentMoE (Part 4) The strongest architecture ablation in the study lives here (quarter-width latent projection, MMLU-Pro up 4.6 at equal size), but the shipped expert counts are not the ablation’s counts (Part 4)
Midtraining move Checkpoint merging credited with about 16 percent FLOP savings in place of part of the decay (Part 5) Worth one experiment at any size; it is nearly free to test at decay time (Part 5)
Post-training shape Multi-environment RL at million-rollout scale, plus MOPD consolidation (Part 8) Budget regression repair as a standing line item, not an exception (Part 8)
Eval discipline Vendor surfaces disagree with each other (three SWE-Bench numbers for Ultra across NVIDIA’s own pages) (Part 10) Designate one surface canonical in your own reporting and say which (Part 10)
Serving artifact NVFP4 build, measured at NVFP4 (Part 10) The BF16 to NVFP4 gap is about 2 points on the publisher’s own numbers; quote the shipped one (Part 10)

340B and up: the flagship band

Stage Start from Change and why
Base and recipe K2-Horizon 375B for the full public record; Nemotron 3 Ultra for the industrial recipe The two flagships split the job: K2-Horizon is the reproducible record, Ultra is the industrial recipe whose internals are partly summary-level (Parts 1, 5)
Data posture K2-Horizon’s stage table (15.13T pretrain, 1.91T midtrain) is the most complete flagship accounting in the study (Part 2) Note the synthetic share question at this scale: K2-Horizon runs roughly 45 percent synthetic in its shared corpus with pipelines released (Part 2)
Architecture MoE with latent attention (Ultra) or MoVA values (K2-Horizon) (Part 4) The openness gap is widest at this size; architecture choice and auditability pull in opposite directions (Piece 0)
Midtraining move K2-Horizon’s four-stage extension ladder, 32K to 512K, priced at about 11 percent of tokens (Part 6) Its per-stage mixes and extension method were not pinned in this study; the ladder’s shape is usable, its recipe is not yet (Part 6)
Post-training shape MOPD from 10 or more teachers, or five-expert RL merged in weight space then patched by phased SFT (Part 8) Public teacher checkpoints (Ultra’s released teacher) make the consolidation rerunnable by outsiders; that is a new asset class (Part 8)
Eval discipline Expect index-version chaos; the flagship’s Artificial Analysis figures conflict (47 vs 31) with the mechanism unestablished (Part 9) Pin index versions in every citation or do not cite the index (Part 9)
Serving artifact Multi-GPU node only; a trillion-class MoE at INT4 would run to roughly 500 to 600 GB of residency before token one, and nothing in this study occupies that class (Part 10) This band does not exist on owned workstation hardware; plan the fleet around the 30B band and treat flagships as teachers (Parts 8, 10)

The ten strongest findings, ranked

Confidence tags: converged means independent labs or independent replications agree; single-source strong means one lab’s controlled work or this study’s own primary-document audit, not yet independently repeated; vendor-claimed means the publisher’s numbers, unverified outside.

  1. Data decisions explain more performance variance than architecture at these scales, and filtering philosophy is a budget decision, not a quality absolute. Classifier-maximal filtering wins short horizons (FineWeb-Edu’s 1.3T subset beat its 15T pool at fixed compute; DCLM gained 6.6 MMLU points at 40 percent less compute), while repair-over-rejection wins long ones (Nemotron-CC matching DCLM with four times the unique tokens). Which is right depends on the token budget being fed. (Part 2.) Converged.
  2. A benchmark score is an instrument reading, not a property of a model. The same weights post a ten-point ARC spread across six published sources from formulation choices alone (OLMES), and one flagship posts three different SWE-Bench numbers across its own publisher’s surfaces at different precisions. Harness, prompt format, sampling settings, scoring rule, and artifact precision are part of the number. (Parts 9, 10.) Converged.
  3. Openness is a ladder with seven rungs, most releases stop at rung 3, and a derivative’s openness is the minimum of its license chain. Weights plus a report is a product you can use; data and configs make it a system you can study; checkpoints, logs, and a runnable harness make it an experiment you can rerun. Termination clauses, unilateral-update rights, and stacked upstream licenses all live below the headline. (Part 1.) Single-source strong (this study’s audit of the rosters at primary documents).
  4. At 30B, the body plan is a business decision: quality per token, quality per FLOP, and throughput per serving dollar have three different winners. On identical data and settings, K2-Horizon’s dense 32B is the quality reference, its MoVA-36B trails by 1.2 to 3.7 points at one-seventh the active parameters, and Nemotron 3 Nano wins on throughput and long-context cost instead. Pick the metric before the architecture. (Part 4.) Single-source strong (one lab’s controlled same-data comparison for the first two; vendor figures for the third).
  5. Midtraining is where domain conversion happens cheapest, and an anneal mix is a portable artifact. Ai2’s Dolmino-shaped anneal (half general replay, patch domains upsampled and repeated) was applied by the independent LLM-jp group to unrelated bases and reproduced, souping included, beating the best single seed on 10 of 12 benchmarks. A published mix transfers across families. (Part 6.) Converged.
  6. On-policy distillation buys teacher-level performance at roughly an order of magnitude less compute than RL, and public teacher checkpoints are a new kind of asset. The mechanism grades every token on the student’s own trajectories instead of one scalar per rollout; cited figures put it near a tenth of RL’s GPU hours for a better score in the Qwen3 case. Its ceiling is the teacher; RL with a real verifier has no such ceiling. (Part 8.) Single-source strong (multiple independent uses of the mechanism; headline efficiency figures partly vendor-reported).
  7. No AdamW challenger has a frontier-scale control run. Muon’s roughly 2x efficiency is a vendor scaling-law fit, and the one independent audit found the advantage shrinking toward 1.1x at 1.2B parameters with a projection it may vanish by 7B. AdEMAMix stands in the same position. Every challenger’s large-scale evidence is that the model trained with it is good. (Part 5.) Converged (as an absence: the same gap in every lab’s record).
  8. Overtraining small models is deliberate, correct, and the norm below 4B; MoE budgets quoted per total parameter are meaningless. Tokens per parameter run into the thousands for small dense models across five independent families, because serving cost dominates lifetime cost. Sparse models must be quoted per active parameter or not quoted. (Part 2.) Converged.
  9. Refusal behavior is trained, stage-locatable, and per-class, and aggregate refusal rates conceal the operative facts. A 31K-example safety slice inside a 30M-example SFT set sets the trigger (Part 7); operator data shows safeguard stop rates from 81 percent (SSRF) to 43 percent (XSS) across classes in the same field (Part 9); stage checkpoints are what make the behavior diffable to the stage that installed it (Parts 1, 8). Single-source strong (family reports plus one operator dataset with small cells).
  10. Serving economics are decided at architecture and tokenizer time, not at deployment. Decode cost is active bytes per token against memory bandwidth (a 13x measured gap between a dense 70B and a 30B-A3B on the same box), the KV design decides how many concurrent long sessions a node holds, and high-entropy cyber text pays roughly double the token tax under general vocabularies. The independent convergence of NVIDIA and IBM on attention in about 8 to 12 percent of hybrid layers is the study’s most trustworthy architecture number. (Parts 3, 4, 10.) Converged for the hybrid ratio and the bandwidth arithmetic; vendor-claimed for the throughput multipliers attached to it.

Open questions the field has not answered in public

Deduplicated from the parts’ flagged sections, sharpened to the question a builder actually needs answered.

  • Is Nemotron-H commercially usable at all? Its card carries an internal research and development tag in one surface and an open model license in redistributions. One direct read of the card text settles it; nobody in this study has done it. (Part 1.)
  • How many tokens did Nemotron 3 Ultra actually pretrain on? About 20T per the card, about 10T plus 212B per a blog-derived summary. The flagship’s budget is a load-bearing number for every comparison built on it. (Parts 2, 5.)
  • Is K2-Horizon’s current 32B the model its stage branches describe? The 32B launched as a Stage 1 checkpoint under the release name; the diff between launch and current artifacts was never performed. (Parts 1, 9.)
  • Does Muon’s advantage survive to frontier scale? No AdamW control exists at that scale from any lab, and the independent audit’s trend line points down. (Part 5.)
  • What MFU did the flagship runs achieve? Neither the K2-Horizon nor the Nemotron 3 Super report states training MFU or total GPU count. Tokens are the flattering unit; the efficiency question is unanswerable from public record. (Part 5.)
  • What are LatentMoE’s shipped configurations? The 512-expert, top-22 figures are a 73B ablation’s, not Super’s or Ultra’s. (Part 4.)
  • How does K2-Horizon balance its routers? The mechanism was not identified in the report sections read, for either the FFN or the value routers. (Part 4.)
  • What is in Granite 4.1’s data, line by line, and what are Granite’s anneal shares? IBM withholds mixture proportions by its own statement; the openness grade rests on the 4.0 pattern carrying over. (Parts 1, 2, 6.)
  • What is the real fertility tax on cyber text? The hex and base64 characters-per-token figures in Part 3 are structural estimates from BPE behavior, not measurements on the study’s tokenizers. The measurement is an afternoon’s work and nobody has published it. (Part 3.)
  • Which index version produced the flagship’s 47, and which produced its 31? Artificial Analysis rebuilt its index twice in the window; the attribution was never established, so the conflict stands as a versioning lesson rather than a resolved discrepancy. (Part 9.)
  • How do MoE and hybrid architectures quantize? The best public quantization study (500K evaluations) covers one dense family. The architectures this study recommends for serving have no equivalent evidence base. (Part 10.)
  • Has anyone independently reproduced a K2-Horizon training claim? As of the tech report’s submission, no. The most transparent fleet in the study is also, so far, an unreplicated one. (Rosters, Part 9.)

The reading path

For an engineer working this study alongside the external spines. The glossary is a companion, not a step: keep it open, and read an entry when a part first uses its term in anger.

  1. Piece 0, then Part 1 (openness). The map and the license habit come before any technique, because they decide which families are worth your hours. No external pairing yet.
  2. Part 2 (data) with CS336’s data lectures and Raschka’s chapter on working with text data. The part gives the comparative evidence; CS336 gives the pipeline mechanics; Raschka grounds tokenization-adjacent basics in code.
  3. Part 3 (tokenizers) with Karpathy’s tokenizer video. Short pairing, high return: the video builds the intuition the part then prices across 18 real configurations.
  4. Part 4 (architecture) with Raschka’s attention chapter and CS336’s MoE and parallelism lectures. Read the part’s 30B comparison twice; it is the study’s centerpiece.
  5. Part 5 (pretraining) with the Smol Training Playbook. The playbook is the practitioner’s version of this part; read them against each other and note where the playbook’s advice is scale-dependent.
  6. Part 6 (midtraining) alone. No external spine covers this stage at comparable depth; that absence is itself one of the study’s findings.
  7. Parts 7 and 8 (SFT, RL, distillation) with Lambert’s RLHF Book. The book supplies method derivations; the parts supply per-family receipts and the refusal-layer treatment the book does not attempt.
  8. Part 9 (evaluation) with the OLMES paper’s instrument framing. Then reread any vendor card you believed last month.
  9. Part 10 (serving) with the Ultra-Scale Playbook’s inference material. The part prices the decisions; the playbook teaches the engineering.
  10. Part 11 (cyber), then this part again. Part 11 is the application the whole study exists for. Rereading the decision tables after it turns them from summary into plan.

The standing update protocol

This study decays unless it is maintained, and the maintenance rules are the same disciplines the parts graded in others.

  • Roster first. A new release enters through the research rosters, not through a part: name, sizes, license read at the artifact (never the summary), and what training material is public. No roster entry, no citation.
  • Grade before use. Every rostered family gets a Part 1 ladder grade before any part may recommend from it. A release that cannot be graded cannot be a table cell.
  • Diff into the parts. A new checkpoint or report updates the specific tables it touches: the vocab table, the token budget table, the band tables, the decision tables here. Each edit names its source part so the trace survives.
  • Corrections propagate. When a checked fact turns out wrong, the correction is written into every part, roster, and table that carries the old claim, in the same pass. A correction that lives in one file is a rumor.
  • Flags carry until closed at a primary source. The open questions above stay open in print until someone reads the card, the LICENSE file, or the report section that settles them. Closing one is a roster event and gets logged with its evidence.
  • Rebuild the surfaces. After any propagation pass, the site and the audio twins rebuild, so the version a reader or listener meets is the corrected one.
  • Version the study. Each maintenance pass stamps the parts it touched with the date, the way the rosters already stamp their verification passes. A study about provenance keeps its own.

Closing: the 2027 goal

This masterclass was commissioned as the study phase of a goal stated plainly: train the most cyber capable models possible, and spend 2027 getting better at the craft of training them, in the open, with the receipts published. Read against that goal, the study’s net message is encouraging and specific. The open families have already paid the pretraining bill and published the recipes, the data pipelines, the anneal mixes, the post-training shapes, and in the best cases the checkpoints and harnesses. What remains scarce, and therefore valuable, is exactly what a small program can own: a documented domain corpus, a verified-trace training set, an execution-grading harness, refusal behavior installed deliberately and measured per class, and scores taken on the artifact that actually ships. Part 11 assembles those into the LAB2 recipe. The decisions that belong to the founder (the budget envelope, the teacher roster, the base pick) are named there and remain his. Everything this study could decide from published evidence, it has decided and cited. The next entry in the craft log is a run.