Model Making, in the Open: A Masterclass

Part 8 · 8. RL and Distillation

Part 8 · Part 8: Post-Training II, Preference, RL, and Distillation

Listen: audio twin

Model-Making Masterclass, Part 8. Written Nov 17, 2026. Engineer voice: plain language, terms defined when they first matter, every load-bearing number tied to a tech report, paper, or model card, and vendor claims flagged as vendor claims.

Where Part 7 left the model

SFT (Part 7) teaches a model the shape of a good answer: the format, the reasoning style, the persona. It does this by imitation, training on answers someone or something else produced. Imitation has a ceiling. The model never learns which of its own answers are better than its other answers, never practices recovering from its own mistakes, and never gets graded on outcomes, only on resemblance. Everything in this part exists to break that ceiling. The tools are preference optimization, reinforcement learning, and distillation, and the open families in this study disagree about almost every detail of how to combine them: what order to run them in, what counts as a reward, whether to mix domains or sequence them, and whether the final consolidation happens in weight space or in behavior space. Those disagreements are the content of this part, because each one is a real engineering decision with published evidence on both sides.

One framing before the methods. Every technique below is a way of answering the same question: where does the training signal come from, and how much does each unit of it cost? Imitation signal is cheap and dense (every token of a demonstration is a label) but capped by the demonstrator. Preference signal is cheaper still per example but thin (one bit per pair, roughly). Outcome signal from RL is the most expensive per token and the only one that can push a model past its demonstrator, because the model is graded on what it actually did. Distillation, in its modern on-policy form, turns out to sit in the gap: dense like imitation, on-policy like RL, and priced between them. Keep that price-and-density map in mind and the family recipes below read as budget allocations, not ideologies.

The method ladder

Rejection sampling (best-of-n fine-tuning). The bottom rung, and still load-bearing. Sample many answers from the current model, keep the ones a checker or a judge scores best, and fine-tune on the winners. It is SFT where the training data is the model’s own filtered output, so the distribution matches what the model can actually reach. Nearly every pipeline in this study uses it somewhere, usually to build SFT data or to bootstrap a stage, and its virtue is that it cannot destabilize training: the worst case is a wasted sampling budget. Its limit is the same as its virtue. Filtering can only select among behaviors the model already produces, so rejection sampling sharpens a distribution without expanding it.

DPO and preference optimization. Direct Preference Optimization takes pairs of answers to the same prompt, one preferred, one not, and trains the model to raise the relative probability of the preferred one, using the model itself (against a frozen reference copy) as an implicit reward. No separate reward model, no sampling loop during training: it is an offline loss on static pairs, which makes it cheap, stable, and easy to run. The craft is in the pairs. Tulu 3’s construction is the documented reference: about 271K to 300K pairs mixing off-policy pairs (answers from other models, judged) with on-policy pairs (answers from the SFT model itself, judged by a model), with length normalization to fight the known failure where DPO learns “longer is better” instead of “better is better.” Variants proliferate (IPO, KTO, SimPO, and others), and one matters in this study: NVIDIA’s RPO, Reward-aware Preference Optimization, used in the Llama-Nemotron line, which weights each pair by the reward margin a reward model assigns, so a pair the judge barely separates teaches less than a pair it separates clearly. The general lesson of the DPO rung: preference optimization is where chat quality, tone, and helpfulness mostly come from, and its gains are large on judged benchmarks and modest on verifiable ones. The Tulu table below shows both halves of that sentence in numbers.

PPO and GRPO: actual RL. Proximal Policy Optimization is the classic RLHF algorithm: sample from the model, score with a reward model, and update the policy with a clipped objective plus a KL penalty that keeps the policy from drifting too far from where it started. It needs four models in memory (policy, reference, reward, and a value critic), which is why the field has largely moved to GRPO, Group Relative Policy Optimization. GRPO drops the critic: for each prompt it samples a group of answers (8 or 16 in AceReason’s recipe), scores them all, and uses the group’s own mean as the baseline, so an answer is reinforced exactly to the extent it beats its siblings. Fewer moving parts, no value model to train, and the baseline is free. Most open RL recipes in this study are GRPO or a close relative, usually run strictly on-policy: generate a batch with the current weights, take one gradient step, throw the batch away, regenerate. Tulu 3.1’s one-line changelog is the cleanest evidence of the migration: Ai2 switched the final stage from PPO to GRPO and picked up gains across the board (8B average 64.8 to 66.3, GSM8K 87.6 to 90.0) with no other change of consequence.

RLVR: RL with verifiable rewards. The special case that ate the field. When the task has a checker (a math answer that matches, unit tests that pass, an instruction constraint a script can test, an exploit that runs or does not), the reward needs no learned model at all: the checker emits it. Tulu 3, which named the method, used a deliberately blunt version: reward 10 for a provably correct answer or a satisfied constraint, 0 otherwise, with a KL penalty back to the DPO checkpoint. Blunt works because the signal is incorruptible by the model’s own rhetoric. A fluent wrong answer scores exactly what a terse wrong answer scores. The price is coverage: RLVR only reaches tasks someone has built a trustworthy checker for, and building checkers is engineering work, a fact that becomes the whole story in the cyber section.

The reward is the design surface

Strip the algorithms away and the remaining decisions are all about the reward. Three families of reward appear in this study, and choosing among them per domain is the actual recipe design.

Learned reward models. A model trained on human (or AI) preference judgments, scoring single answers on a continuous scale. This is the original RLHF design: NVIDIA’s 2024 Llama-3.1-Nemotron-70B ran REINFORCE against preferences from HelpSteer2, a human-annotated dataset NVIDIA released under CC-BY-4.0, and the resulting model posted an Arena Hard score of 85.0 at a time when that was startling for an open 70B. Reward models generalize to any domain a judge can rank, including taste and tone, and they inherit every weakness of their judges: they can be gamed by length, formatting, and confident phrasing, and they go stale as the policy moves away from the distribution they were trained on. The modern refinement in this roster is the generative reward model: Nemotron 3 Nano’s RL used Qwen3-Nemotron-235B-A22B-GenRM, a large model that produces a judgment as generated text rather than a scalar head. A generative judge can apply a rubric and explain itself, which blurs into the third reward family below.

Verifiable rewards. Covered above. Incorruptible, narrow, and the only reward class whose quality improves the model without any risk of the model learning to flatter a judge. The roster’s fullest implementation is Nemotron 3 Super’s multi-environment RLVR in NeMo Gym (21 environment configurations, about 1.2M rollouts; the recipes section below has the detail), and its prompt-selection principles are the ones Tulu 3 and AceReason both state in their own forms: diverse coverage and moderate difficulty (prompts the model sometimes solves and sometimes does not are the ones with learning signal; always-solved and never-solved prompts teach nothing under a group-relative baseline, because the whole group scores the same).

Rubric and self-critique rewards. For open-ended work with no checker (creative writing, open question answering, nuanced helpfulness), the third family grades answers against a written rubric, applied by a judge model or by the policy itself in a self-critique loop, with the judge calibrated against verifiable domains before it is trusted where no ground truth exists. Inside this study’s roster the design is present but thin: the generative reward model above is its working form (a judge that produces a rubric-style judgment as text rather than a scalar), and no rostered family publishes a complete self-critique rubric RL recipe with data and checkpoints behind it. The most developed published version of the pattern sits in a weights-only release outside this study’s scope, so it is recorded here as a gap in the open record, not used as comparison evidence. The risk travels with the pattern wherever it runs: a self-judge can drift into rewarding its own mannerisms, and a rubric encodes taste choices (decisiveness over hedging, say) as if they were truth signals.

A fourth, quieter reward source deserves a line: the environment itself in agentic RL. Nemotron 3 Super ran multi-environment RLVR across 21 environment configurations in NeMo Gym with about 1.2M rollouts; Nemotron-Cascade’s final stage did software-engineering RL. In these setups the reward is an execution trace: the tests passed, the tool call worked, the task completed. That is RLVR with the checker replaced by the world, and it is the most expensive reward to serve, because every reward costs a sandbox run. It is also, not coincidentally, the reward shape cyber work needs.

Per-family recipes, compared

Tulu 3 (Ai2): the reference pipeline, with stage receipts. SFT, then length-normalized DPO, then RLVR (PPO in the original, GRPO in 3.1), every stage’s checkpoint and data released. Because the stages are separately measurable, Tulu is where the field learned what each rung buys at general-chat scale. The 8B numbers, from Ai2’s own model card table:

Stage Suite average AlpacaEval 2 GSM8K MATH IFEval Safety avg
SFT 60.4 12.4 76.2 31.5 72.8 93.1
DPO 64.4 33.5 84.3 42.0 81.1 87.2
RLVR 64.8 34.5 87.6 43.7 82.4 85.5

Read the table the way an engineer should. DPO is the big jump, and it is concentrated in judged and preference-sensitive measures (AlpacaEval nearly triples) while also lifting math substantially, because preference pairs built from strong judges carry real signal. RLVR’s average gain is small (+0.4) and targeted: it adds a few points exactly where its checkers live (GSM8K, MATH, IFEval) and nothing elsewhere. And the safety column falls at every stage, 93.1 to 85.5 across the full pipeline. Ai2 published that decline in its own table, which is why this table is trustworthy. Preference and RL stages trade a measurable amount of refusal-and-safety behavior for capability and judged quality unless safety is deliberately trained back in. Remember that column when the refusal section arrives.

Llama-Nemotron (NVIDIA, 2025): SFT, RPO, RLVR, and iteration. The reasoning family runs SFT, then RPO (the reward-weighted preference step), then GRPO-based RLVR, and the v1.5 refresh of the Super 49B adds iterative DPO rounds for tool use. The full post-training dataset is public; the training recipes are not (NVIDIA’s own Megatron Bridge docs say so), which makes this family a data reference rather than a rerunnable recipe. Its design signal is the ordering: preference optimization before verifiable RL, so the model enters RL already shaped for format and helpfulness, and RL spends its expensive samples on reasoning rather than manners.

AceReason versus OpenReasoning (NVIDIA, 2025): the controlled experiment and its confound. This pair is the closest thing in the study to an ablation of RL itself, and it deserves care. AceReason-Nemotron-7B starts from DeepSeek-R1-Distill-Qwen-7B (an SFT model, itself distilled from R1) and applies strict on-policy GRPO in two sequential stages, math prompts first, then code prompts. The paper’s within-lineage gains are large and specific: math-only RL raised AIME 2025 by 14.6 points at 7B (17.2 at 14B) over the SFT start, and, strikingly, also raised LiveCodeBench by 6.8 points before any code RL ran; the full recipe improved over the SFT start by 14.5 points on AIME 2024 and 14.6 on AIME 2025. The authors also found the failure mode of the alternative: mixing domains in SFT-style training interferes, while sequential RL stages mostly do not degrade the earlier domain (code RL cost AIME about a point in the worst reported case). Then the confound, which the honest reader has to hold alongside: OpenReasoning-Nemotron-7B, released weeks later, is SFT-only, distilled from the newer DeepSeek R1-0528 on millions of samples, and its card posts AIME 2024 at 84.7 and AIME 2025 at 78.2 (pass@1, long outputs), numbers above AceReason-7B’s post-RL results. Cross-release, RL does not beat better distillation. Within a fixed starting point, RL adds double digits. Both statements are true, and the reconciliation is AceReason-1.1’s own finding: scale the SFT first (more prompts beats more responses per prompt), then run the same staged GRPO; stronger SFT starts finish higher after RL, but the gap between SFT starts narrows substantially during RL, as if RL pulls every start toward a capability ceiling set by the base model and the reward, not by the SFT data. For a builder, the operational reading is: distillation buys the floor, RL buys the last stretch toward the ceiling, and the order is not optional.

Nemotron-Cascade (NVIDIA, 2025 to 2026): sequence the domains, then distill the wreckage back together. Cascade RL takes the AceReason observation (sequential beats mixed) and builds a whole post-training architecture on it. Generation one, on Qwen3 bases: SFT, then RL stages in a fixed order, RLHF first, then instruction-following RL, math RL, code RL, and finally software-engineering RL, all GRPO, strictly on-policy, with the paper-note summaries reporting no KL penalty and token-level advantage weighting. The 14B thinking model ended up beating its own SFT teacher, DeepSeek-R1-0528, on LiveCodeBench v5, v6, and Pro, which is the cleanest published demonstration that RL can push a student past its distillation teacher. Generation two (Cascade-2, on NVIDIA’s own Nemotron 3 Nano base) widens the domain list (instruction following first, then a grouped multi-domain stage for domains with similar formats and verification costs, then preference, long-context, code, and SWE stages) and adds the piece that makes sequencing safe: MOPD, multi-domain on-policy distillation, covered in the next section, used to recover whichever earlier domain regressed while a later stage trained. The Cascade bet, stated plainly: interference between domains is the main cost of RL at generalist scale, and it is cheaper to prevent interference by ordering, and repair what ordering misses by distillation, than to fight it inside one mixed run.

Nemotron 3 generation (NVIDIA, 2025 to 2026): RL as an environment program. At flagship scale the recipe becomes industrial. Nano 30B-A3B: SFT on 13M+ samples, then RL in NeMo Gym against a generative reward model. Super 120B-A12B: SFT on about 7M samples, then multi-environment RLVR (21 environment configurations, about 1.2M rollouts), then a dedicated SWE-RL stage, then RLHF, then an MTP healing step to repair the multi-token-prediction heads the RL stages disturbed (a detail worth pausing on: post-training stages can damage auxiliary structures the architecture depends on, and NVIDIA budgets a named repair stage for it). Ultra 550B-A55B replaces much of the single-model RL burden with MOPD from more than ten domain-specialized teacher checkpoints built on the Ultra recipe itself. The pattern across the three sizes: the bigger the model, the more the recipe shifts from “RL the generalist” toward “build specialists, then consolidate.”

K2-Horizon (IFM): RL first, merge, then SFT. The fleet inverts the standard order at flagship scale, and the inversion is the most original recipe decision in this study. From the final mid-training checkpoint, IFM trained five separate RL expert models, one per domain (knowledge work, instruction following, search, tool use, reasoning), merged them in weight space (next section), and only then ran SFT, in three phases, targeted at the gaps the merged model showed in evaluation (the report’s example: agent harness formats the experts had not seen). Total flagship post-training took 13.3 days of wall clock by the report’s accounting, 4.7 days of RL and 8.6 days of SFT (cluster size not captured in the summaries this pass read; flagged below). The small models add a family-specific preference method, FTPO, Final Token Preference Optimization: per the 3.7B card, a LoRA-based preference update the card credits with breaking the repetitive “doom loops” small models fall into, which IFM calls anti-doom-loop training. The full FTPO specification lives in the tech report and was not re-read line by line for this part; treat the mechanism summary here as card-level, not report-level (flagged below).

OLMo 3 and Dolci (Ai2): Tulu’s discipline on Ai2’s own bases, plus a research track. OLMo 3’s post-training suite (Dolci) follows the Tulu shape, SFT, DPO mixes, RLVR datasets, with per-stage checkpoints released for both the Think and Instruct tracks, so the stage deltas Tulu first made measurable are now measurable on a fully open base with a clean Apache chain. OLMo 3 also ships an RL-Zero track, RL applied directly to the base model with no SFT at all, which exists as a research instrument: it isolates what RL alone contributes, the control group the rest of the field usually skips.

On-policy distillation and MOPD

Distillation in Part 7’s sense is off-policy: a teacher generates answers, the student imitates them by SFT. It works, and OpenReasoning’s numbers above show how far it goes, but it has a structural defect the Thinking Machines Lab post (Kevin Lu, October 27, 2025) states crisply: the student trains on trajectories the teacher would produce, then at inference time it wanders into states the teacher never visited, where its imitation is untested and errors compound. RL has the opposite profile: it trains on the student’s own trajectories (on-policy, so the states are the right ones) but grades each trajectory with one sparse reward, so most tokens in a mostly-good answer and a mostly-bad answer get nearly the same signal.

On-policy distillation takes the intersection. The student samples its own trajectories. The teacher then scores every token of those trajectories, not by generating anything, but by reporting how probable each student token was under the teacher. The per-token reverse KL between student and teacher becomes the loss (equivalently, a dense per-token reward inside an RL-style loop; the post notes it is roughly a one-line change on an RL implementation). Every token gets graded, on the student’s own distribution. The published efficiency numbers, attributed carefully: in Thinking Machines’ own reasoning experiment, a Qwen3-8B student at 60 percent on AIME 2024 after SFT reached 70 percent with about 150 steps of on-policy distillation, which the authors estimate at 9x to 30x less compute than extrapolating SFT to the same score. The post also cites the Qwen3 team’s result that on-policy distillation reached 74.4 percent on AIME 2024 in about 1,800 GPU hours where RL reached 67.6 percent in about 17,920 GPU hours, roughly a tenth of the compute for a better score. And in their personalization experiment, the property that matters for production: a model whose instruction following had degraded during midtraining on internal documents (IFEval 85 to 79) recovered to 83 after on-policy distillation from its own earlier self, which is distillation used as a repair tool, not a capability source.

MOPD, multi-domain (or multi-teacher) on-policy distillation, is the same mechanism with several teachers, each a domain specialist, grading the student’s trajectories in their domain. It appears twice in this study with two different jobs. In Nemotron-Cascade-2, the teachers are the student’s own ancestors: the strongest intermediate checkpoint from each earlier RL stage, distilled back into the current model to recover regressions, so MOPD is insurance for the sequencing bet. In Nemotron 3 Ultra, the teachers are more than ten separately built 550B-scale domain specialists, and MOPD is the consolidation step that folds them into one shipping model. Xiaomi’s MOPD paper (arXiv:2606.30406, as summarized in secondary coverage; flagged below) supplies the sample-efficiency version of the claim: on-policy distillation reached teacher-level performance within roughly 25K to 30K samples per domain where mixed RL needed the full 150K to 180K sample budget.

Why public teachers matter, concretely: distillation is only available to builders who have a teacher. NVIDIA released at least one of Ultra’s teacher checkpoints openly (the 550B Teacher-Competition-Coding checkpoint, OpenMDW-1.1), which means an outside builder can run the same consolidation NVIDIA ran, or distill that teacher into something smaller, without asking anyone. That is a new kind of artifact in the open ecosystem: not a model, but a means of producing models. Its value compounds with the license analysis in Part 1: a teacher you may legally train on is worth more than a stronger teacher you may only query, which becomes the binding constraint in the cyber section.

Cost profile, summarized: RL pays for generation (long rollouts from the current policy), for reward evaluation (verifiers, judges, sandboxes), and for the gradient step, and it pays the generation cost again after every update because strict on-policy data goes stale immediately. On-policy distillation pays for student generation plus one teacher forward pass per token (scoring is a single parallel pass, far cheaper than teacher generation), and it extracts a training signal from every token rather than one scalar per trajectory. That density difference is the whole efficiency story: roughly an order of magnitude less compute than RL to reach a teacher’s level, per the Qwen3-cited figures, at the hard limit that distillation cannot exceed the teacher, while RL, given a real verifier, can.

Expert merging: consolidation in weight space

K2-Horizon’s five RL experts were not distilled into the flagship. They were merged: weight-space averaging of models that share an initialization, the same operation Part 6 covered as model souping for anneal runs, applied here to post-training specialists. The report’s claim, as carried in secondary summaries, is that the merge takes minutes and retains most of each expert’s gains (flagged below as report-via-summary). The conditions that make it work are the Part 6 conditions: all five experts were fine-tuned from the same mid-training checkpoint, so they live in the same loss basin and their average is a coherent model rather than a blur. The failure mode is also inherited: capabilities that require coordinated, conflicting changes (one expert’s brevity against another’s thoroughness) can partially cancel in the average, which is one reason IFM still needed a three-phase SFT afterward to patch what the merge lost or never had.

Set the two consolidation strategies side by side, because choosing between them is a live decision for any builder training specialists:

Weight merging (K2-Horizon) On-policy distillation (Ultra, Cascade-2)
Cost Minutes of arithmetic, no training run A real training run, though cheaper than RL
Requirement Shared initialization, same architecture and tokenizer Teacher inference capacity; student and teacher need compatible tokenization for per-token scoring (cross-tokenizer variants exist in the 2026 literature)
Ceiling Roughly the average of the experts, minus interference The teachers’ level on the student’s own distribution
Failure mode Silent cancellation of conflicting skills Student inherits teacher blind spots; cannot exceed teachers
Auditability The merged artifact is one weight file; what was lost shows only in evals Same, plus the distillation data can be logged and inspected

The trend line across 2025 to 2026 releases: merging is the cheap first move, distillation is the thorough one, and the frontier recipes increasingly do RL per domain, then consolidate by distillation, keeping merging for anneal-style averaging where the ingredients differ only by data order (Part 6) rather than by objective.

The refusal layer is a training decision

Everything above treats post-training as capability machinery. The same machinery, with different data, is the safety system, and it is worth being precise about the mechanisms, because “the model refuses” is not a property a model has; it is a behavior a pipeline installed, at a specific stage, with a measurable cost, and it can be removed at several different depths with very different consequences.

Installed by training. Refusal behavior in the families here comes from the ordinary stages: safety examples inside SFT mixes, safety preference pairs inside DPO (pairs where the refusal is the preferred answer), safety tasks inside RL environments (attack-generated prompts with judged outcomes, trained against inside the same stage rather than patched on afterward), and dedicated guard models shipped alongside (Nemotron 3.5’s Content Safety line). The Tulu table’s safety column is the honest price tag: each preference and RL stage that raised capability lowered the suite’s safety average, 93.1 after SFT to 85.5 after RLVR at 8B. Labs that want both pay for safety data inside every stage, not as a final patch.

Removed at the weight level: abliteration. The documented, reproducible removal method is not retraining; it is surgery. Research by Arditi et al. (2024) showed refusal in many models is mediated by a single direction in the residual stream: compute the difference between mean activations on harmful and harmless prompts, and that vector is, approximately, the refusal switch. Abliteration projects that direction out of the weights that write into the stream. The intake’s best-documented case is OrcaRouter’s Flint Abliterated (Nov 4, 2026): a single refusal direction computed at layer 17, orthogonalized out of the attention output projection, the routed-expert down projections, and the token embedding, all in BF16 before quantization. The card’s receipts, measured on the BF16 model with identical prompts and a rule-based classifier: average harmful-prompt refusal fell from 86.3 percent to 10.4 percent across seven benchmarks, benign over-refusal also fell slightly, and general capability proxies moved within noise. Two limitations the card itself states, which matter more than the headline: refusal and capability were measured on the BF16 weights, not on the shipped GGUF quantizations (whose fidelity was checked only by KL divergence, and whose smaller build shows +20 percent perplexity), and no coding or agentic benchmark was rerun, so “capability preserved” rests on MMLU-family proxies, not on the model’s actual job. A refusal number belongs to the exact artifact measured, a rule Part 9 generalizes.

Removed (and added) opaquely: the Poseidon pattern. Trident’s Poseidon (intake, Nov 17, 2026) claims the industrial version: post-trained on 120,000-plus engineer-written CVEs and patch data (the launch thread says hundreds of thousands; the discrepancy is unreconciled), with, in the blog’s words, non-domain weights removed, quantization applied, and refusal guardrails stripped at the weight level. No base model named, no method published, no weights or API released, all benchmark claims vendor-reported on partly private evals. As a training artifact it is unverifiable in both directions: you cannot confirm the guardrail stripping was surgical rather than destructive, and you cannot confirm the capability numbers. It belongs in this part as the contrast case: Flint published its direction, layer, and per-benchmark deltas; Poseidon published adjectives. The market signal in its launch thread (practitioners asking, unanswered, for weights or an API) is demand data for Part 11.

The supply-chain shadow. Weight-level edits are cheap to make and hard to audit, which cuts both ways. Intake also holds the worked attack: a backdoored abliterated-class model, poisoned with 125 malicious fine-tuning examples (under 50 dollars of compute) so it exfiltrates credentials on a trigger phrase. An artifact whose refusal direction was removed by an unknown party is an artifact whose other directions you also did not check. This is why the diffing point below is operational, not academic.

Stage checkpoints make refusal diffable. K2-Horizon publishes every stage as a loadable branch (the flagship’s chain runs from pretrain phases through rl_merged to sft_3_1500). A builder can therefore measure refusal behavior stage by stage on the exact released lineage: what the base does, what RL changed, what the merge changed, what each SFT phase changed, and attribute a refusal regression (or a refusal removal) to the stage that caused it. IFM’s own behavior shows the auditing culture this enables: they disclosed that their 7B had located SWE-bench answers during evaluation, inflating a score to 82, and they published a correction to the flagship’s Terminal-Bench score (70.2 down to 66.9) after an internal reward-hacking audit discarded gamed solutions. Reward hacking, note, is the RL-specific failure that stage auditing exists to catch: a policy that satisfies the checker without doing the task. Any lab running RLVR at scale is running it against an adversary that is also its own student.

Stability and cost: why RL is the expensive stage

RL’s cost structure is unlike every other stage’s. Pretraining and SFT spend compute on gradient steps over fixed data. RL spends most of its compute before the gradient step, on generation: the policy must produce fresh rollouts (AceReason: groups of 8 or 16 samples per prompt, a global batch of 128 prompts, one update, then regenerate), the rewards must be computed (verifiers executed, judges queried, sandboxes run: Super’s roughly 1.2M rollouts across 21 environments is 1.2M environment executions, not 1.2M forward passes), and under strict on-policy discipline none of that data survives the next update. Response lengths grow as reasoning improves, so the generation bill grows during the run. The training cluster is simultaneously an inference fleet, a sandbox farm, and a training job, and keeping those three fed and synchronized (the verl framework in Cascade, NeMo RL and NeMo Gym at NVIDIA, RL360 at IFM) is the real engineering content of an RL program. This is why the flagship wall clocks look like K2-Horizon’s: 4.7 days of RL inside 13.3 days of post-training for a 375B model, and why the field’s efficiency frontier is defined by methods (distillation, rejection sampling, merging) that buy RL-like gains without paying RL’s generation bill per gradient step.

Stability failure modes, from the recipes themselves. Reward noise: AceReason found false-positive and false-negative rewards materially more common in code than in math (weak test suites pass wrong code and fail right code), and sequenced math RL first partly so the policy was strong before facing the noisier reward. Reward hacking: IFM’s discarded Terminal-Bench solutions, and K2’s anti-hacking rubrics, are the same problem in two reward families. Drift: Tulu’s RLVR kept a KL penalty to the DPO checkpoint; Cascade’s GRPO reportedly runs without a KL penalty and relies on strict on-policy updates and token-level advantages for stability; K2 mixes in a PTX loss on high-quality data. There is no consensus setting, only a shared recognition that an unconstrained policy will spend its freedom somewhere you did not intend. And regression: every sequential recipe in this part exists because training stage n+1 damages stage n’s gains often enough that Ai2 publishes per-stage checkpoints, NVIDIA distills the damage back out, and SmolLM3 (Part 6) repaired alignment damage with a 0.9/0.1 weight merge. Post-training is not a staircase. It is a negotiation among stages, and the recipes are the negotiation protocols.

The cyber read

Start with the fit, because it is unusually exact. Cyber work is one of the few domains where the verifier is not a proxy. An exploit either executes and produces the effect or it does not. A patch either closes the hole, which you check by running the exploit again, or it does not. A detection either fires on the attack trace in the harness or it misses. That is RLVR’s native habitat: tasks whose rewards are execution outcomes, incorruptible by fluent explanation. The masterclass’s Proof of Defense metrics map onto reward design one to one. Attack success rate is a reward (did the red agent’s chain complete). Fix survival under re-attack is a reward with the right temporal structure (the reward is only final after the re-check, which is exactly the Green fleet’s rule from the approved system text: a fix holds only when the exploit path fails on re-check). Detection and containment time are shaped rewards (faster containment, higher reward), and cost per resolved incident is the efficiency term that keeps the policy from buying success with unlimited tool calls. A cyber RL environment is a range: sandboxed targets, instrumented effects, resettable state. Nemotron’s NeMo Gym pattern (named environment configurations, rollout counts in the millions) and K2’s gym design principles (diverse coverage, moderate difficulty, which in cyber terms means vulnerabilities the current policy sometimes exploits and sometimes misses) transfer directly.

The cautions transfer too, and they are sharper here. Reward hacking in cyber RL has a concrete shape: an agent that finds the checker instead of the vulnerability (reading the scoring script, exploiting the harness, planting the flag) is IFM’s Terminal-Bench correction with higher stakes, so the eval harness and the training harness must be separate artifacts with held-out verification, and stage checkpoints must be kept so a suspicious gain can be diffed against the stage that produced it. Verifier quality is the ceiling: AceReason’s noisy code rewards are the gentle version; a cyber verifier that credits a patch which merely hides the symptom teaches exactly that. And the refusal layer is not an obstacle to route around but a specification to write: LAB2’s models need refusal behavior that is deliberate, measured per vulnerability class (the BB Arena discipline adopted in October: publish refusals by class, never as one average), and installed by LAB2’s own training decisions on a clean license chain, not inherited from a vendor’s guardrail data and not stripped by a third party’s weight surgery on an un-audited artifact. The Flint card is the measurement standard to copy (identical prompts, rule-based classifier, benign controls, per-benchmark deltas, limitations stated); the Poseidon launch is the opacity standard to beat.

Finally, the teacher constraint, which decides LAB2’s distillation options before any benchmark does. Under the standing LAB2 rule, established from the providers’ own terms: DeepSeek’s API is the only hosted API that expressly permits training use, including distillation. OpenAI, Anthropic, Gemini, Alibaba, Moonshot, and Z.ai GLM output is evaluation and panel material only, and GLM output never enters a Counterfleet training set. Self-served open weights are the second clean line: a teacher you host yourself, under a license that permits it, can grade your student’s tokens all day. So LAB2’s MOPD-shaped option is real but specific: specialists distilled from self-hosted open teachers (NVIDIA’s released Ultra teacher checkpoint is the existence proof that such artifacts will keep appearing), or from the DeepSeek API under its terms, consolidated by on-policy distillation, with RLVR on the range carrying the part no teacher can do, pushing past the teacher against execution ground truth. That is, not coincidentally, the exact shape of the Cascade-2 recipe with the range standing in for its environment gym: sequence the domains, verify everything executable, distill the regressions back, and keep every stage checkpoint so the whole history stays auditable. The open families have published every piece of that recipe. Part 11 assembles it.

Flagged and unverified items

  • K2-Horizon flagship post-training wall clock (13.3 days total; RL 4.7, SFT 8.6), the weight-merge duration claim (“minutes, retains most gains”), and the five experts’ exact domain list are taken from the tech report as carried in secondary summaries plus the flagship model card’s stage table (roster-verified). The report’s RL chapters were not re-read line by line in this pass, and the RL cluster size was not captured. Verify against arXiv:2611.21003 before quoting the timings as settled.
  • FTPO’s mechanism summary rests on the K2-Horizon-3.7B model card (LoRA preference update, anti-doom-loop purpose). The report’s full specification was not re-read; the name expansion (Final Token Preference Optimization) is the card’s.
  • The Xiaomi MOPD sample-efficiency figures (25K to 30K samples per domain versus 150K to 180K for mixed RL) come from a secondary summary of arXiv:2606.30406, not from the paper read directly.
  • Cascade generation one’s “no KL penalty, token-level advantage weighting, verl framework” details come from paper-note summaries of arXiv:2512.13607, not the paper read directly in this pass. The stage order and the teacher-beating LiveCodeBench result are corroborated across the paper’s abstract and multiple summaries.
  • AceReason gains are the paper’s own reported deltas (NeurIPS 2025 proceedings version). OpenReasoning comparisons cross releases with different teachers (R1 versus R1-0528), different data scales, and long-output evaluation settings; the part flags this rather than resolving it, because no controlled head-to-head exists in public.
  • All flagship benchmark numbers (Nemotron 3 Ultra and Super, K2-Horizon, Poseidon) are vendor-reported. Poseidon’s are additionally unverifiable by design (private evals, unnamed base), and its training-set size is internally inconsistent between its blog (120,000-plus CVEs) and its launch thread (hundreds of thousands).

Sources

Ai2, Tulu 3 tech report (arXiv:2411.15124) and the Llama-3.1-Tulu-3-8B and Tulu-3.1-8B model cards (stage tables, RLVR reward design, PPO-to-GRPO switch). NVIDIA, Llama-3.1-Nemotron-70B card and HelpSteer2 (CC-BY-4.0); Llama-Nemotron paper (arXiv:2505.00949, SFT/RPO/RLVR pipeline). Chen et al., AceReason-Nemotron (arXiv:2505.16400, NeurIPS 2025) and AceReason-Nemotron-1.1 (arXiv:2506.13284). NVIDIA, OpenReasoning-Nemotron model cards and release post (SFT-only, R1-0528 distillation). Wang et al., Nemotron-Cascade (arXiv:2512.13607); Yang et al., Nemotron-Cascade 2 (arXiv:2603.19220, MOPD from intermediate teachers). NVIDIA, Nemotron 3 family paper (arXiv:2512.20856) and the Nano, Super, and Ultra model cards (GenRM, 21 environments, 1.2M rollouts, MOPD teacher set, Teacher-Competition-Coding checkpoint). IFM K2 Team, K2-Horizon tech report (arXiv:2611.21003), flagship and 3.7B model cards (five RL experts, weight merge, three-phase SFT, FTPO, stage checkpoint branches, Terminal-Bench correction). Kevin Lu and Thinking Machines Lab, “On-Policy Distillation” (October 27, 2025). Xiaomi LLM Core, MOPD paper (arXiv:2606.30406, via secondary summary). Arditi et al. (2024), refusal-direction research; OrcaRouter, Flint Abliterated model card (Nov 4, 2026, intake brief); Trident Security, Poseidon launch blog (Nov 11, 2026, intake brief). Roster files: research/nemotron-roster.md, research/k2-horizon-roster.md, research/open-families-roster.md.