Model Making, in the Open: A Masterclass

Part 5 · 5. Pretraining

Part 5 · Part 5: Pretraining, the Run Itself

Listen: audio twin

Parts 2 through 4 were about decisions you make before the run: what data, what tokenizer, what shape of model. This part is about the run itself, the weeks or months where those decisions turn into a loss curve, a power bill, and a series of 3 a.m. incidents. Pretraining is where a lab’s engineering quality stops being a matter of taste. A good architecture trained badly loses to a plain architecture trained well, and the difference is rarely one big idea. It is a stack of unglamorous choices: which optimizer, what learning rate shape, how big a batch, how many bits per number, what you do when the loss spikes at step 400,000, and whether your parallelism layout wastes a third of the cluster.

The open families are unusually good teachers here, because several publish the run and not just the result. K2-Horizon publishes its loss spike accounting, episode by episode, including the spikes it rolled back and erased. Marin publishes retrospectives that name its mistakes. NVIDIA publishes the precision experiments behind training Nemotron 3 Super in 4-bit floating point. This part compares those runs: optimizer, schedule, batch and sequence staging, precision, stability, parallelism, and the compute bill. The cyber read at the end translates it all to the scale LAB2 operates at, because almost none of these techniques mean the same thing at 8 billion parameters as at 375 billion.

One honesty note before the numbers. Several figures below come from the labs’ own reports, and labs report what makes them look competent. Where a number is a vendor claim with no independent check, the text says so. Where a report does not publish a number at all, most often total GPU count or training MFU, the text says that too, rather than filling the gap with arithmetic dressed up as fact. Flagged gaps are collected again at the end.

1. Optimizers: AdamW is the default, and the challengers have to prove it at scale

Every family in this study trains its matrix weights with one of two optimizers: AdamW or AdEMAMix. The split is lopsided. AdamW runs K2-Horizon, OLMo 2, OLMo 3, Nemotron 3 Super, Granite, SmolLM, and Marin. AdEMAMix runs exactly one family, Apertus. The third optimizer in this part, Muon, runs in no rostered family at all. It enters as a method, on the strength of its published papers, because it is the most serious AdamW challenger in the current literature and any lab choosing an optimizer today has to have a position on it. That distribution is itself the first finding: inside the open roster the default is total, and the challengers amount to one production family and one paper.

AdamW, in plain terms. Adam keeps two running averages per parameter: a mean of recent gradients (which direction to move) and a mean of squared gradients (how noisy each parameter’s signal is, used to scale the step per parameter). AdamW adds weight decay decoupled from the gradient, shrinking weights toward zero by a fixed proportion each step. The settings barely vary across labs: first-moment decay 0.9, second-moment decay 0.95 or 0.99, weight decay around 0.1, gradient clipping at a global norm of 1.0. K2-Horizon uses betas (0.9, 0.95) and clip 1.0 across all six of its sizes. OLMo 3 uses (0.9, 0.95), weight decay 0.1, and notably applies no weight decay to the embeddings, a stability choice the Smol Playbook also endorses: decaying embedding norms shrinks early-layer inputs, which inflates early-layer gradients through the normalization layers.

What varies is how labs choose the learning rate that AdamW consumes. K2-Horizon’s method is the most principled published in the set: a scaling-collapse procedure. Express each run in terms of the AdamW timescale (the coupling of learning rate, weight decay, and token budget), tokens per parameter, and schedule shape, and the normalized loss curves from different sizes collapse onto one curve. Pick hyperparameters where the collapse holds at small scale, and they transfer, with extra experiments on how the rate should move as routed expert count changes. The payoff: one recipe table covers 0.9B to 375B, peak rates from 8e-4 down to 1.4e-4, chosen by rule rather than per-size grid search.

Muon, in plain terms. Muon ignores the per-parameter scaling that defines Adam. For each weight matrix, it takes the momentum (a running average of gradients, as in plain SGD with momentum) and orthogonalizes it: it replaces the update matrix with the nearest matrix whose singular values are all equal, computed approximately with a few Newton-Schulz iteration steps. Intuitively, Adam asks “how big a step does each individual weight deserve,” while Muon asks “make the update push equally in every direction the matrix can move, with no direction dominating.” Two adjustments make it work at scale, both from the Muon scaling paper (Liu et al., arXiv:2502.16982). First, add weight decay, which vanilla Muon lacked. Second, rescale each orthogonalized update by a factor tied to the matrix shape (the square root of the larger dimension, times 0.2) so that the root-mean-square size of a Muon update matches the RMS size AdamW would produce. Without that matching, Muon’s updates are systematically the wrong size and comparisons against AdamW are meaningless.

Muon also never runs alone. It applies only to two-dimensional hidden-layer weight matrices. Embeddings, the output head, biases, and normalization gains still run AdamW in every Muon deployment. And it is cheaper in optimizer state: Muon stores one momentum buffer per matrix, AdamW stores two moment buffers per parameter, so a Muon run carries meaningfully less optimizer memory.

The evidence published for Muon is strong and should be read as its authors’ claim. The scaling paper’s fits say Muon reaches AdamW’s loss with roughly half the training FLOPs, about 2x compute efficiency in the compute-optimal regime, demonstrated on a 16B total / 3B active MoE trained on 5.7T tokens. Treat the 2x figure with care. An independent audit with a carefully matched comparison protocol (Wen et al. 2025, flagged here as a secondary citation, not re-read in full for this part) found the advantage shrinking with scale: roughly 1.3 to 1.4x at a few hundred million parameters, about 1.1x at 1.2B, and a projection that it may vanish by 7B. The two results do not reconcile yet. The honest summary: Muon is at least as good as AdamW everywhere honestly tested, probably better at small and mid scale, and unproven as a 2x win at frontier scale, because no family in this study has run it at scale with a published AdamW control. That absence is why Muon appears in this part as a method to evaluate, not as a roster result.

MuonClip: Muon plus a guardrail on attention logits. Scaling Muon up surfaced a failure AdamW rarely produces: attention logits exploding. In the MuonClip paper’s mid-scale test (9B active, 53B total MoE) with vanilla Muon, the maximum attention logit quickly passed 1,000 and the run spiked and diverged. The paper’s diagnosis is that Muon’s orthogonalized updates can grow the query and key matrices in ways that inflate logits faster than AdamW’s per-parameter scaling would. The standard fixes did not fit. Logit soft-capping was not enough, and QK-Norm, the usual cure from Part 4, cannot be applied cleanly to Multi-head Latent Attention because the keys are not fully materialized at inference time.

QK-Clip is the fix, and it operates on weights, not activations. After each optimizer step, for each attention head, the trainer tracks the largest attention logit that head produced, Smax. If Smax exceeds a threshold (100 in the paper’s production run), it rescales that head’s query and key weight matrices down by a factor derived from the overshoot, splitting the correction evenly between the two matrices. The current step’s forward and backward passes are untouched; the clip bounds how large logits can grow on future steps. In that reported run the mechanism was a seatbelt that mostly stayed unused: 12.7 percent of heads triggered it at least once in the first 70,000 steps, logits sat at the cap early, and by about 30 percent of training they had decayed into a stable range on their own. The paper’s appendix reports that even an aggressive threshold of 30 costs negligible loss in small-scale tests.

AdEMAMix: Adam with a longer memory. Apertus is the only family in the set that trains on AdEMAMix (Pagliardini et al., arXiv:2409.03137), and it kept the optimizer from version 1.0 into 1.5 unchanged. The idea is one sentence: Adam’s single gradient average forces a tradeoff between reacting to recent gradients and remembering old ones, so AdEMAMix keeps two averages, a fast one at the usual decay and a slow one with a decay around 0.9999 whose weight in the update ramps up over training. The paper’s evidence is at small scale but striking: a 1.3B model trained with AdEMAMix on 101B tokens matched an AdamW model trained on 197B tokens, and the authors show gradients remain useful for tens of thousands of steps. Apertus is the existence proof at production scale: 8B and 70B models, 15T tokens, on thousands of GPUs. What Apertus does not publish is an AdamW control at the same scale, so the same caveat applies as with Muon. The published large-scale evidence for every AdamW challenger is “we ran it and the model is good,” never “we ran both and here is the delta.” That is the state of optimizer science in 2026: the default is AdamW, the challengers have plausible mechanisms and small-scale wins, and nobody with a frontier budget has spent half of it on the control run.

2. Learning rate schedules: cosine commits you, WSD lets you change your mind

The schedule question looks like a curve shape and is actually a management question: do you know, on day one, exactly how many tokens this run will train on?

Cosine decays the learning rate smoothly from peak to near zero across a predeclared horizon. Its weakness is structural. If you extend the run, the rate is already too low to learn much from the new data; if you stop early, you never got the benefit of the final decay, which is where a large share of the quality arrives. OLMo is the cosine camp in this study, and OLMo 2 shows both the shape and the workaround. Its stage 1 used cosine schedules defined over 5T tokens (7B and 13B) or 6.5T (32B), then truncated: the 7B stopped at 4T tokens with the rate still high, and separate stage 2 anneal runs decayed from there on better data. OLMo 3 kept cosine for pretraining proper: the 32B warms up over 2,000 steps to a peak of 6e-4 and decays toward 6e-5, truncated at 5.5T tokens; the 7B peaks at 3e-4. K2-Horizon’s smallest model is its only cosine run: the 0.9B decays to 10 percent of its 8e-4 peak over 5.03T tokens.

Warmup-stable-decay (WSD) warms up, holds the peak rate flat for most of the run, then decays sharply at the end, typically over the last 10 to 20 percent of tokens. The flat phase is the point. A checkpoint taken anywhere in the stable phase can be decayed later, when you know your real budget, your real data supply, and whether the run deserves more tokens. Nemotron Nano v2 used WSD across its 20T token FP8 run. Nemotron 3 Super used WSD at 25T tokens, peak 4.5e-4, with warmup spread over the first 200B tokens.

Two labs pushed the idea further, and their variants are the most interesting schedule designs in this part.

Nemotron 3 Super replaces part of the decay with checkpoint merging. During the stable phase, NVIDIA saved checkpoints every 1,000 iterations (about 25B tokens each) and merged them by weighted averaging over a sliding window; the shipped base model is a merge spanning about 500B tokens of checkpoints. NVIDIA’s recipe documentation credits the approach with roughly 16 percent FLOP savings, because a separate decay run over about 4T tokens never has to happen. Averaging checkpoints along a flat-rate trajectory approximates the settling effect of decaying the rate, at the cost of storing and merging checkpoints instead of compute. Part 6 covers annealing and souping in full; the pretraining point is that Super’s schedule is WSD where the D is partly done in weight space rather than learning-rate space.

K2-Horizon goes further still: for every size except the 0.9B, there is no decay in pretraining at all. The rate warms up (starting from one ten-thousandth of peak) and then holds at peak to the final pretraining step, for up to 1.1 million steps and 21.92T tokens. The decay is deferred entirely: mid-training starts from an average of the last few pretraining checkpoints (four to six of them, spaced 1,500 to 5,000 steps apart), which behaves like an implicit decay, then re-warms and decays for real in mid-training stage 1. The risk of running hot forever is instability, and Section 5 shows K2-Horizon paid for it in loss spikes. The benefit is total schedule freedom: any checkpoint is a valid starting point, and no decay decision ever becomes irreversible.

Marin ran the experiment that explains why most labs do not go that far. Its 8B run began with WSD-S, a cyclic variant that decays briefly every 10,000 to 20,000 steps so you can measure progress. The retrospective records that the team moved away from it after the first phase: plain WSD with an exponential moving average of the weights (decay 0.995, a trick they credit to the DeepSeek-V3 report) gave them the measurable, decayed-quality signal without touching the schedule. Marin’s arc, cyclic WSD to WSD to rewarmed phases with two separate cooldowns, is what schedule freedom looks like in practice: valuable, and mostly spent reacting to hardware and data availability rather than executing a plan.

Peak learning rates, for the record, cluster tightly once you adjust for size: 8e-4 at 0.9B (K2-Horizon), 3e-4 to 4e-4 in the 4B to 36B band (OLMo 3, K2-Horizon, Nemotron 3 Super at 4.5e-4), 1.4e-4 to 1.5e-4 at 32B dense and 375B (K2-Horizon). Marin is the outlier that proves the range is a choice, not a law: it trained its 8B at 1e-3, three times OLMo 2’s 7B peak, following the DCLM finding that you can run hot, and raised it to 1.7e-3 when the batch grew. Apertus sits at the conservative end: its 8B peaks at 1.1e-4, and the Apertus 1.5 report states plainly that the 1.0 run’s 70B peak was likely too low, doubling it for the 1.5 continuation. A lab admitting its learning rate was wrong, in print, is the kind of sentence this masterclass exists to collect.

3. Batch size and sequence length: two staging decisions, mostly settled

Batch size in modern runs is quoted in tokens per step, and the spread across this study is wider than the learning rate spread. Nemotron 3 Super used 3,072 sequences of 8,192 tokens, about 25.2M tokens per step. K2-Horizon scaled batch with size: 8.4M tokens at 0.9B, about 19.9M for the 3.7B through 36B models, and 33.6M for the 375B. OLMo 3’s 32B used 1,024 sequences of 8,192, about 8.4M tokens, with the 7B at half that. OLMo 2’s 7B used 1,024 sequences of 4,096, about 4.2M tokens. Marin started at 4M tokens and grew opportunistically to 12M and then 16M as bigger TPU slices arrived, scaling the learning rate by the square root of the batch increase (a 3x batch took the rate from 1e-3 to 1.7e-3), with no optimizer reset and no rewarm at the transitions.

The pattern: batch grows with model size and cluster size, because the batch is what you parallelize. The countervailing force is that larger batches need more tokens to reach the same loss, so batch size is a throughput purchase paid for in data. Nobody here ramps batch on a schedule as a stability tool, the way older recipes did; learning rate warmup carries that job now. Marin’s mid-run batch growth is the exception, driven by hardware availability, not training theory.

Sequence length staging is more uniform. Almost everyone pretrains short and extends later. K2-Horizon pretrained everything at 8,192 and pushed context extension into mid-training, climbing 32K to 128K to 512K while raising the RoPE base frequency at each step. Nemotron 3 Super pretrained at 8,192, then ran a separate long-context phase at a constant low rate (4.5e-6) on sequences of one million tokens: 34B tokens, followed by 17B tokens alternating 1M and 4K sequences to repair a small math regression the long phase had caused. OLMo 2 and Marin pretrained at 4,096. The logic is arithmetic: attention cost grows with sequence length, most web tokens live in short documents, and long-context ability is teachable late and cheaply. Pretraining at your final context length is paying frontier prices for a skill a short extension phase installs.

4. Precision: BF16 is the floor, FP8 is proven, NVFP4 is the frontier

Precision is where the money is. Halving the bits per number roughly halves memory traffic and, on hardware with native low-precision matrix units, roughly doubles math throughput. The question each lab answers differently is how much numerical damage the model absorbs in exchange.

The BF16 default is more than “train in 16-bit.” The standard recipe, documented cleanly by both K2-Horizon and Marin, keeps master weights in FP32, computes forward and backward passes in BF16, and reduces gradients in FP32. Marin’s variant keeps parameters and optimizer states in FP32 with BF16 compute on TPU. This is the configuration behind most open models ever released, and it is the reference every lower-precision claim is measured against.

FP8 training cuts the matrix math to 8-bit floating point. DeepSeek-V3 is the proof at scale: 671B total parameters trained on 14.8T tokens in FP8, using fine-grained quantization to keep the format honest, with activations scaled in 1x128 tiles and weights in 128x128 blocks, so no single outlier can crush the range of a whole tensor. The V3 report’s appendix compares FP8 against BF16 training directly and puts the relative loss error under 0.25 percent. The economics are the famous part: pretraining cost 2.664M H800 GPU hours, 180K GPU hours per trillion tokens, on a 2,048 GPU cluster. NVIDIA adopted the recipe wholesale for Nemotron Nano v2, whose report states it used DeepSeek’s FP8 training recipe for the entirety of its 20T token pretraining. When a recipe transfers between labs unchanged, that is the strongest endorsement the field offers.

FP8’s boundary is worth marking honestly: the conservative position in the wider literature is to store activations in FP8 to save memory while refusing it for computation on degradation risk. Inside this study’s roster that position has no exhibit: DeepSeek computes in FP8 and NVIDIA adopted the recipe unchanged, and no rostered family publishes a refusal. The difference between the two positions is risk appetite and quantization stack maturity, not a disagreement about the arithmetic, and the roster’s unanimity is itself a measure of how far the format’s credibility has moved.

NVFP4 takes the math to 4 bits, and it only exists on NVIDIA Blackwell hardware. The method paper (arXiv:2509.25149) is a Nemotron paper in all but name: its validation run is a 12B hybrid Mamba-Transformer, the Nano v2 architecture, trained on 10T tokens, the longest 4-bit pretraining run publicly documented. Four techniques make it work. Random Hadamard transforms spread outliers across a block before quantization, applied to the weight-gradient multiplication where outliers concentrate. Two-dimensional scaling keeps the forward and backward views of a tensor consistent. Stochastic rounding is applied to gradients only, because deterministic rounding biases gradient estimates while the same stochastic rounding hurts forward tensors. And about 16 percent of linear layers, the first blocks and the last blocks, simply stay in BF16, chosen by sensitivity analysis. The results: validation loss within about 1 percent of an FP8 baseline during the stable phase, widening slightly during decay, and downstream scores that trade blows (MMLU-Pro 62.58 for NVFP4 against 62.62 for FP8; the coding benchmarks show the largest gap, a few points). A trick worth stealing: switching the forward pass from NVFP4 to BF16 at 8.2T tokens cut the relative loss error from about 1.5 percent to about 0.5 percent, which is precision annealing as a schedule decision.

Nemotron 3 Super then ran NVFP4 in production for the full 25T token horizon on B200 GPUs, with a hybrid layout: NVFP4 by default, the final 15 percent of the network in BF16 for stability, multi-token prediction layers in BF16, attention and latent projections in BF16 or MXFP8, and the Mamba output projection in MXFP8 after NVFP4 underflow showed up at smaller scales. Two production details deserve attention. First, the share of exactly-zero weight gradients grew over training to about 7 percent of parameters, a quiet tax the 4-bit format levies. Second, NVIDIA tested switching everything to MXFP8 before annealing: the loss trajectory improved and downstream accuracy did not, so the shipped model stayed NVFP4 throughout. The claim for the format is 2 to 3x arithmetic throughput over FP8 on Blackwell tensor cores at half the memory. The cost, per the papers, is a loss penalty around 1 to 1.5 percent relative, concentrated in code, recoverable in part by late precision switching. Whether that trade is worth it depends entirely on whether you own Blackwell hardware and a 25T token budget.

5. Stability: the spike is the run’s real adversary

Every long run spikes. The loss jumps, sometimes by a little, sometimes to five times its value, and the run either absorbs it, rolls back, or dies. What separates the families in this study is not whether they spiked. It is whether they published the accounting.

K2-Horizon’s accounting is the most complete ever released for a model family. Their report defines a spike episode precisely (a robust z-score of the loss against a rolling median exceeds 5, and the episode ends when it stays below 2 for three steps), then counts episodes two ways: on the accepted trajectory, and on branches that rollbacks later erased. Read the 7B row slowly. Over 1.1 million steps it recorded 403 accepted episodes and 106 erased ones, needed 43 rollbacks, and its deepest rollback rewound 22,124 steps. The 375B flagship’s first phase was worse: 83 accepted episodes against 184 erased, 29 rollbacks in 210,000 steps, a rollback rate of 13.8 per 100,000 steps against at most 1.2 for the four stable sizes. One erased excursion is documented step by step: at step 84,114 the loss climbed from about 0.9 to 5.4, stayed above 2 for 151 steps, the branch was abandoned, and training resumed from step 82,001. The accepted trajectory shows nothing there. Without the erased-branch accounting, that incident would be invisible, which is exactly the point: every other lab’s smooth loss curve may be hiding the same history.

Three more K2-Horizon findings generalize. First, spikes persist late into training in their runs, unlike their predecessor K2-V2, where most spikes landed in the first 40 percent; their hypothesis is that the long constant learning rate phase keeps the model in a regime where spikes stay possible, the price of the schedule freedom from Section 2. Second, gradient norm is a poor smoke detector: only elevation within the last 10 steps carries signal, and it may be part of the spike rather than a warning of it. Third, they automated the response. A loss monitor reads the last 35 minutes of per-step losses every 30 minutes; if 10 consecutive steps all exceed a loss of 0.97 with the largest above 1.2, the run rolls back automatically to the last checkpoint on a 250-step grid before the flagged window. Checkpoints land every 250 steps precisely so rollback is cheap. The monitor’s stated limitation: it watches training loss only, and nothing stops it from resuming repeatedly from the same checkpoint.

OLMo’s approach is prevention at the architecture and optimizer level, and most of it landed in Part 4: QK-Norm, reordered normalization, z-loss penalizing runaway output logits, and careful initialization, ablated in public. The run-time layer is OLMo 3’s SkipStepAdamW, which skips an optimizer step entirely when the loss or gradient norm sits more than six standard deviations above its recent mean. Skipping is cheaper than rolling back: you lose one batch, not 22,000 steps. OLMo 2’s report also devotes a stability deep dive to a data-side cause, repeated n-grams in the training text, a reminder that some spikes are born in the data pipeline, not the optimizer. The Smol Playbook’s stability chapter reads as the practitioner consensus distilled from these runs: gradient clipping at 1.0, z-loss available, no weight decay on embeddings, QK-Norm for insurance. It also records, honestly, that SmolLM3 skipped z-loss because their implementation added overhead they had not optimized in time. Even the guardrails have engineering budgets.

Marin remains the honesty standard, because its retrospectives publish the failures nobody else would print. The 8B run changed schedule family mid-run, discovered it had been using the wrong rotary embedding settings into its second phase, skipped z-loss and then wrote that in retrospect it should have used it, and discarded an entire first cooldown before reheating to a 12.7T token finish. The 32B run is the better story. Around step 70,000, persistent loss spikes appeared. The team tried clipping, update-norm guards, and skip-step heuristics. None of it worked. Their published conclusion was that the model needed architectural support: they warm-started the run onto a Qwen3-style architecture with QK-Norm added, took a brief loss bump, watched the spikes vanish, and recovered the expected trajectory within about 10B tokens. Then the first cooldown surfaced contaminated data from a cached math dataset and a training-loss phase shift caused by their shuffling algorithm, a linear-congruential shuffle replaced with a Feistel-based one; they cleaned the data and ran the cooldown again. One researcher kept that run alive across a summer of preemptible TPU slices. Marin’s lesson is not any single technique. It is that a mid-run architecture transplant, a discarded cooldown, and a data bug are normal events in a real run, and the labs that publish them are the ones you can learn operations from.

6. Parallelism and MFU: everyone parallelizes, almost nobody reports utilization

A training run spreads one model across hundreds or thousands of accelerators, and the layout is a tax negotiation between memory, communication, and idle time. The vocabulary, briefly: data parallelism replicates the model and splits the batch; tensor parallelism splits individual matrix multiplications across devices; pipeline parallelism splits layers into stages and streams micro-batches through them, losing time to the bubble at fill and drain; expert parallelism places different MoE experts on different devices and routes tokens with all-to-all traffic; ZeRO or FSDP sharding splits optimizer states, gradients, and parameters across the data-parallel group so no device holds a full copy.

K2-Horizon made a clean pipeline call: none. Their framework, xLLM, uses FSDP full sharding for data parallelism with gradients reduced in FP32, tensor parallelism confined to the 8 GPUs inside a node, and expert parallelism tied to the tensor parallel degree so expert all-to-all traffic never leaves the node’s fast interconnect. Pipeline parallelism was evaluated and dropped because, following their K2-V2 experience, the bubble overhead cost more than the memory it saved at their operating point. Context parallelism appears only when sequences get long, in mid-training, reaching a degree of 4 for the 375B at 512K context. Nemotron 3 Super’s long-context phase shows the extreme end of that dial: at one million tokens of context, its layout was context parallelism 64, tensor parallelism 2, expert parallelism 64, a configuration that exists only to shard the sequence.

MFU, model FLOPs utilization, the share of the hardware’s peak math rate the run actually achieves, is where disclosure falls apart. None of the flagship reports read for this part states a training MFU or a total GPU count. K2-Horizon publishes its cluster’s shape (nodes of eight H200s, 3.2 terabits per second of InfiniBand per node) but not per-run device counts or utilization. NVIDIA publishes Super’s inference throughput multipliers prominently and its training utilization not at all. The exception, again, is Marin, whose weekly public summaries report MFU for its later MoE runs (18.6 percent on TPU v4 for a 67B MoE run) and then correct themselves: one summary records that an apparent expert-parallelism win was an artifact of a benchmark running with routing load balance off, silently dropping most token assignments, and restates the honest figure at about 20.7 percent. Treat that episode as the calibration point for every MFU number you ever read, including your own.

7. The compute bill, per size class

Tokens are the common currency. GPU hours and dollars appear only where a lab stated them.

Family and size Pretraining tokens Hardware stated Cost stated
K2-Horizon 0.9B 5.03T H200 nodes (fleet) not stated
K2-Horizon 3.7B and 7B 21.92T each H200 nodes not stated
K2-Horizon 32B and 36B-A4B 21.92T each H200 nodes not stated
K2-Horizon 375B-A23B 15.1T (two phases) H200 nodes not stated
DeepSeek-V3 (671B, 37B active) 14.8T 2,048 H800 2.664M H800 hours
Nemotron Nano v2 (12B) 20T not stated here not stated
Nemotron 3 Nano (30B-A3B) 25T not stated here not stated
Nemotron 3 Super (120B-A12B) 25T B200 (GB200 for long context) not stated
OLMo 2 7B / 13B / 32B 4T / 5T / about 6T not stated here not stated
OLMo 3 7B / 32B 5.93T / 5.5T not stated here not stated
Apertus 8B and 70B 15T shared recipe GH200, up to 4,096 GPUs over 6M GPU hours for the largest run
Marin 8B 12.7T TPU v5e-256, then v4-2048 not stated
Marin 32B about 6.4T TPU v5p, then v4 not stated
SmolLM2 1.7B 11T Hugging Face cluster about $250K, per Hugging Face

Read the table’s silences as data. The two labs that publish GPU hours, DeepSeek and the Apertus consortium (whose CSCS engineering paper reports the largest model’s run at roughly three months and over 6 million GPU hours), are the two whose funding models reward demonstrating efficiency or justifying public infrastructure. Everyone else publishes tokens, the flattering unit: they hide utilization entirely. A 25T token run at 20 percent MFU costs two and a half times the same run at 50 percent, and the token count alone cannot tell you which one happened.

Two structural facts stand out. First, token budgets have decoupled from model size, as Part 2 predicted: the 12B Nano v2 trained on 20T tokens and the 1.7B SmolLM2 on 11T, thousands of tokens per parameter, because inference cost is paid per parameter forever while training cost is paid once. Second, K2-Horizon’s budgets show a deliberate design: every size from 3.7B to 36B trained on the same 21.92T tokens with the same batch and step count, so the family is a controlled experiment in scale, not six separate recipes. The flagship breaks the pattern at 15.1T, split into two phases with an optimizer reset and a data mix change at the boundary, because at 375B the run’s own stability, documented in Section 5, forced the schedule to adapt.

8. The cyber read: what LAB2 actually needs from all this

LAB2 does not pretrain frontier models from scratch, and under its current plan it never will. Its ladder takes open base models (Granite 4.1 8B at Step A, a 30B class mixture-of-experts at Step B) and specializes them for cyber work: domain continued training, supervised fine-tuning, preference work, and Proof of Defense evaluation. That implies continued training in tens to hundreds of billions of tokens on a single-digit number of accelerators, including a dedicated M5 Max, not trillions of tokens on a cluster. Read against that scale, this part’s techniques sort into three piles.

What transfers directly. AdamW with the boring settings (betas 0.9 and 0.95, weight decay 0.1, clip at 1.0, no weight decay on embeddings) is the correct default, and the burden of proof sits on any challenger. The schedule lesson transfers with a twist: LAB2’s budgets are exactly the kind that change mid-flight, when a dataset disappoints or an eval says stop, so WSD thinking applies even at small scale. Hold the rate, checkpoint often, and decay deliberately when a stage ends, rather than marrying a cosine to a token count you guessed. Checkpoint discipline is the cheapest insurance in this entire part: K2-Horizon checkpoints every 250 steps because rollback is its recovery plan, and at LAB2 scale the equivalent is saving often enough that a spike, a bad data shard, or a corrupted run costs hours, not the run. OLMo’s skip-step guard is a few lines in OLMo-core and worth having from day one. And the stability practice that matters most at this scale is Marin’s: keep a run log that records incidents, interventions, and discarded work as they happen. LAB2’s credibility rests on Proof of Defense numbers; a training log with the failures left in is part of that evidence chain.

What is worth one experiment. Muon is the single challenger cheap enough to test honestly at 8B scale: a drop-in for matrix weights that reduces optimizer memory, and a LAB2 ablation (same data, same tokens, AdamW against Muon on a continued-training run, scored on the eval harness rather than loss alone) would produce a real answer at exactly the scale where the independent audit says the advantage may still exist. Checkpoint averaging, the Nemotron Super trick, is nearly free to try at decay time; Part 6 takes it up properly. AdEMAMix is a reasonable second experiment for long continued-training runs, given its forgetting result, but has the thinnest tooling support of the three.

What is frontier-scale only, and should stay there. NVFP4 and FP8 compute are datacenter GPU techniques: they need Hopper or Blackwell tensor cores and a quantization stack that took DeepSeek and NVIDIA person-years to harden. On Apple Silicon and a pair of RTX 3060s they are not available levers, and at LAB2’s token counts the arithmetic they accelerate is not the bottleneck anyway; data quality and eval design are. QK-Clip solves a problem Muon creates at 53B scale; a lab not running Muon from scratch will never need it, though QK-Norm, chosen at architecture time in Part 4, remains cheap insurance. Pipeline parallelism, expert parallelism layouts, and MFU optimization are cluster disciplines. A 30B-A3B model fine-tuned on one machine has a parallelism problem of exactly one device, and the correct response is gradient checkpointing and patience. Finally, the spike machinery: K2-Horizon’s automatic rollback monitor exists because a 375B run cannot afford a human watching the curve. At LAB2 scale a human, or an agent, is watching, and the right kit is frequent checkpoints, a skip-step guard, and a written incident log.

The one number from this part to keep on the wall: DeepSeek trained a trillion tokens for 180K H800 hours at frontier efficiency. LAB2’s entire Step A continued-training budget is a rounding error against that, which is precisely the strategic point. The open families have already paid the pretraining bill and published the receipts. LAB2’s edge was never going to be out-training them. It is out-specializing them, on runs small enough to log honestly, evaluate hard, and repeat.

Sources and flagged gaps

Primary sources read for this part: Liu et al., “Muon is Scalable for LLM Training” (arXiv:2502.16982), the method source for Muon and MuonClip; the K2-Horizon technical report (arXiv:2611.21003), training, infrastructure, and stability sections including its recipe table and spike accounting; NVIDIA, “Efficient Pretraining with NVFP4” (arXiv:2509.25149); the Nemotron 3 Super technical report and NVIDIA’s Nemotron pretraining recipe documentation, including checkpoint merging; the Nemotron Nano v2 report (arXiv:2508.14444) for the FP8 recipe adoption; the DeepSeek-V3 technical report (arXiv:2412.19437) for FP8 quantization and training costs; the OLMo 2 report (arXiv:2501.00656) hyperparameter tables and the OLMo 3 report and OLMo-core configurations for schedules and SkipStepAdamW; the Apertus 1.5 report (arXiv:2610.15542) and the CSCS Alps engineering paper for AdEMAMix, schedule, and GPU hours; the AdEMAMix paper (arXiv:2409.03137); the Marin 8B retrospective and Marin 32B retrospective coverage for phases, incidents, and the QK-Norm warm start; the Smol Training Playbook; and the family rosters in this masterclass’s research folder.

Flagged as unverified or incomplete, deliberately: the Muon efficiency figure (about 2x) is its authors’ scaling-law claim, and the independent audit contradicting it at scale (Wen et al. 2025) is cited secondhand here. Training MFU and total GPU counts are not published for the K2-Horizon or Nemotron 3 Super runs in the sources read, and none is estimated here. Nemotron 3 Super’s warmup and decay iteration counts come from NVIDIA’s recipe documentation and report summaries rather than a single canonical table. The Apertus 1.0 schedule shape is described through the 1.5 report’s account of what it kept and changed. Marin’s discarded-cooldown detail is carried from the project roster.