Model Making, in the Open: A Masterclass

Part 2 · 2. Data

Part 2 · Part 2: Data, Where Performance Actually Comes From

Listen: audio twin

Model-Making Masterclass, Part 2. Written Nov 17, 2026. Engineer voice: plain language, terms defined when they first matter, every claim tied to a published artifact or flagged as unverified.

Why this part comes before architecture

Part 4 will compare architectures, and architecture is the part of model making that gets the conference talks. But line the open families up at matched size and the scoreboard does not sort by architecture. It sorts by diet. A 1.7B model overtrained on 11T tokens of staged, filtered, project-built data (SmolLM2) beat same-size models trained on private data. A 7B trained on a classifier-filtered dataset (DCLM-Baseline) beat the previous open-data best by 6.6 MMLU points while using 40 percent less compute. An 8B trained on a rephrased, ensemble-filtered crawl (Nemotron-CC) beat Llama 3.1 8B on its own home benchmarks. In each case the architecture was ordinary. The data pipeline was the contribution.

This is also the stage where openness pays off hardest. Part 1 graded families on a ladder; the data rung (rung 4) is where the grade becomes usable knowledge. NVIDIA, Ai2, Hugging Face, IFM, and the Swiss AI Initiative all publish their corpora or their corpus recipes. Granite publishes only part. So this part can compare actual pipelines for the open families, and for Granite it can only compare what its cards admit. That asymmetry is itself one of the findings.

The pipeline, stage by stage

Every family in this study implements the same seven stages, in roughly the same order. What differs is the tool at each stage, the aggressiveness of each cut, and whether the stage is documented well enough to rerun.

Stage 1: acquisition. The raw material is Common Crawl for almost everyone, plus curated side channels. FineWeb processed 96 snapshots (2013 to 2024). Dolma used 25 snapshots (2020-05 to 2023-06) and filtered out 84.2 percent of the raw content before anything else happened, 175 TB down to 27.7 TB at the web stage alone. DCLM standardized the starting point itself: DCLM-Pool is a fixed 240T-token extraction of Common Crawl that every entrant in the testbed starts from, so dataset experiments compare filtering, not crawling luck. On top of the crawl, families add code (GitHub, via The Stack and its descendants), math (web math, competition sets), science (Semantic Scholar papers for Dolma’s peS2o; Ai2’s own olmOCR-converted PDFs for Dolma 3), books (Project Gutenberg in Dolma; books in LLM360’s RedPajama-derived mix), and encyclopedic text. Apertus adds a constraint nobody else applies at this stage: robots.txt opt-outs honored retroactively, meaning pages whose owners later opted out are removed from earlier crawls too. Acquisition is where Apertus’s compliance-first posture is set, and it costs them data everyone else keeps.

Stage 2: extraction. Turning WARC records and PDFs into clean text. FineWeb’s pipeline is published as the datatrove library and is the de facto reference implementation. Dolma ships its own toolkit. Dolma 3 replaced its science source (peS2o) with PDFs linearized by olmOCR, Ai2’s own converter, because PDF extraction quality was a measurable bottleneck: the science slice of Dolma 3 is 805B tokens of converted PDFs, 13.6 percent of the final mix, far larger than Dolma 1’s paper slice. Extraction looks like plumbing. At these scales it is a quality decision: bad extraction injects boilerplate, navigation junk, and mangled math into everything downstream.

Stage 3: language identification and language policy. Dolma 1 was English-only by design, with language ID as a hard filter. The multilingual families treat language mix as a first-class budget line instead: Apertus spans 1,811 languages with over 40 percent non-English content by design, FineWeb2 extends the FineWeb recipe to 1,000+ languages and feeds SmolLM3’s multilingual web slice (12 percent of its stage 1), and EuroLLM covers 35 languages. NVIDIA went the other direction and built a dedicated corpus for one language family: the roughly 2.83T-token synthetic Indic corpus in 22 languages released with Nemotron 3.5 Super VL. Same stage, four policies: filter to English, budget multilingual deliberately, extend a proven recipe per language, or synthesize a language family’s corpus outright.

Stage 4: quality filtering. The load-bearing stage, and the subject of the next section.

Stage 5: deduplication. Everyone dedupes; the published pipelines (datatrove for FineWeb, the Dolma toolkit, DCLM’s pipeline) use MinHash-style fuzzy dedup on top of exact and URL-level passes, applied within and across snapshots so the same page crawled ten times is not ten training examples. Nemotron-CC’s headline number makes the dedup stakes concrete: of its 6.3T tokens, 4.4T are deduplicated real tokens and 1.9T are synthetic. The 4.4T unique-real figure is the one NVIDIA argues from, because unique tokens are what long-horizon training actually consumes. A corpus can be enormous and still be small once dedup is honest.

Stage 6: decontamination. Removing benchmark material from training data. Methods and honesty levels vary enough to get their own section below.

Stage 7: mixing and sequencing. The final proportions, the upsampling multipliers, and the order data is served in. LLM360’s K2-65B mix table is the old-school exemplar: per-source token counts with explicit multipliers (DM Math at 3x, RefinedWeb about 47 percent of stage 1). SmolLM3 publishes stage percentages to the point (stage 1: 85 percent web, 12 percent code, 3 percent math; stage 3 decay: 63/24/13). Dolma 3’s mix table is below in the domain-mix section. Two sequencing choices recur: multi-stage curricula that raise the specialized share over time (SmolLM2’s four stages, SmolLM3’s three stages, Granite’s four-stage curriculum), and a final anneal on a small elite mix (Dolmino, Longmino), which Part 6 covers in full.

One sequencing result deserves emphasis because it cuts against intuition. SmolLM2’s team found that its staged curriculum helped the 1.7B but the 135M and 360M models did better on a single-stage mix of consistently high-quality data from the start, and the paper says why they believe that is: a small model has limited capacity to spend on low-value data early in training, so the curriculum’s cheap early phase is not cheap for it. Curriculum design is size-dependent, and the family that published the negative result saved every small-model builder a wasted ablation.

Tokenization-adjacent decisions. The tokenizer itself is Part 3, but two data-side decisions belong here. First, token counts are tokenizer-relative: Dolma’s datasheet counts tokens with the Llama tokenizer, and every family’s T-token headline is denominated in its own tokenizer, so cross-family token comparisons carry a few percent of slop by construction. Second, document packing and sequence length during data prep shape what the model sees as a document; SmolLM2 trained its ablations at 2,048 sequence length while rating the released models at 8K context, a reminder that the data pipeline and the context claim are separate artifacts.

Quality filtering: three philosophies, measured

Filtering is where the field’s center of gravity moved between 2023 and 2026, and the open literature lets us compare philosophies with numbers attached.

Philosophy 1: heuristics first. The Dolma and early FineWeb generation: rule-based filters (document length, symbol ratios, repeated-line fractions, blocklists, language scores) applied in sequence. Cheap, transparent, reproducible, and blunt. Dolma’s web pipeline is the documented reference: it works, it is fully open, and its own successors at Ai2 moved beyond it. Heuristics remove junk reliably. They cannot tell a mediocre page from an excellent one, and at trillion-token scale that distinction is most of the game.

Philosophy 2: classifier filtering on a learned quality notion. The FineWeb-Edu and DCLM result, and the empirical foundation of the current era. FineWeb-Edu: a large model (Llama-3-70B) annotated about 500K pages for educational value, a small embedding classifier learned the annotation, and the full pass kept pages scoring 3 or better on a 5-point scale, yielding 1.3T tokens out of the 15T pool. The result that changed practice: at fixed compute, models trained on the 1.3T subset beat models trained on the full 15T pool. Throwing away over 90 percent of the crawl made the model better, not just cheaper. DCLM ran the controlled version of the same experiment: a fastText classifier trained to recognize instruction-style text (OpenHermes-2.5 and top ELI5 posts as positives) against random crawl, keep the top 10 percent. The winning dataset’s 7B model, trained on 2.6T tokens, hit 64 percent MMLU and beat the previous open-data best by 6.6 points with 40 percent less compute. Per-language versions of the same idea built SmolLM2’s Stack-Edu (educational classifiers per programming language) and FineMath (tiered quality grades, FineMath4+ and FineMath3+, so the mix can dial math quality by tier).

Philosophy 3: ensemble scoring plus repair instead of rejection. Nemotron-CC is the named break from philosophy 2, and its paper argues the point explicitly: aggressive model-based filtering buys benchmark points at the cost of deleting 90 percent of the data, which starves long-horizon training. NVIDIA’s answer has three parts. Ensemble several classifiers instead of trusting one quality notion. Reduce reliance on heuristic filters that were silently deleting good data. And where a document is mediocre rather than bad, repair it: synthetic rephrasing rewrites crawl text in cleaner styles, converting low-value documents into training signal instead of discarding them. The published effect sizes: at 8B scale and 1T tokens, a high-quality subset of Nemotron-CC beats DCLM by 5.6 MMLU points; the full 6.3T dataset matches DCLM on MMLU while holding four times more unique real tokens; and the long-horizon proof, an 8B trained for 15T tokens with 7.2T of them from Nemotron-CC, beats Llama 3.1 8B by 5 points on MMLU and 3.1 on ARC-Challenge. Read the trade honestly: philosophy 2 wins the sprint (best score per token at short horizons), philosophy 3 wins the marathon (enough unique quality tokens to feed a 15T-token run). Which one is right depends on the token budget you are feeding, which is why both survive in modern pipelines, usually as tiers: SmolLM2’s winning ablation was itself a blend, 60 percent FineWeb-Edu with 40 percent DCLM.

A refinement worth naming, because three families use it: quality tiers as a dial rather than a gate. FineMath ships in grades (FineMath4+ for the strictest cut, FineMath3+ for the wider one), Nemotron-CC-Math does the same (133B tokens at grade 3+, 52B at grade 4+), and Nemotron-CC itself separates a high-quality subset from the full corpus instead of pretending one threshold fits every training phase. A tiered corpus lets the mix designer spend the best data where it buys the most, heavily in the anneal (Part 6), moderately in late training, sparingly early, and it lets one dataset release serve both the sprint and the marathon strategies above. Single-threshold filtering throws that dial away.

The fourth position is refusal to filter by a single quality notion at all. Apertus filters for license permissibility, toxicity, and PII, and applies memorization suppression during training (goldfish-style token dropping), but its quality story is compliance and coverage across 1,811 languages rather than an educational-value score. Marin’s record shows the pragmatic middle: its 8B run switched its mixture mid-flight to include Nemotron-CC, an open lab voting with its feet for philosophy 3’s corpus while keeping its own process fully documented.

Synthetic data: from garnish to main ingredient

Five years ago synthetic data meant instruction tuning garnish. In this roster it is a pretraining staple, and the shares are large enough to state plainly.

  • Nemotron-CC: 1.9T of 6.3T tokens (about 30 percent) are synthetic, produced by rephrasing real crawl documents. The synthetic share is what lets the corpus claim both quality and unique-token depth: the real documents anchor provenance, the rephrases multiply usable signal per document.
  • Nemotron-4 340B: over 98 percent of its alignment data was synthetic, generated by the family’s own Instruct and Reward models. The 2024 Nemotron play was a self-generation loop: train the generator, generate the alignment corpus, release both (HelpSteer2, the human-annotated preference set underneath, is CC-BY-4.0).
  • K2-Horizon: roughly 10T synthetic pretraining tokens, per the tech report’s abstract, inside a 22T-token corpus shared by four of the six models. The pipelines that generated them are released and documented in detail, which the abstract itself flags as the rare part: labs publish synthetic data claims often and synthetic data pipelines almost never. At roughly 45 percent of the shared corpus, synthetic is not an ingredient in K2-Horizon’s pretraining. It is close to half the diet.
  • SmolLM line: synthetic as targeted construction. Where a public dataset was not good enough, Hugging Face built one: FineMath for math, Stack-Edu for code, SmolTalk and SmolTalk2 for post-training, largely model-generated and classifier-graded. The datasets are the release; the models are the demonstration.
  • OLMo’s stance: natural first, synthetic where it is labeled. Dolma and Dolma 3 are overwhelmingly natural corpora (web, PDFs, code, arXiv). Ai2’s synthetic weight sits in post-training, where Tulu 3 states its proportions openly (about 43 percent of its 939K SFT prompts synthetic, generated with a persona method over roughly 250,000 personas) and Dolci continues the pattern for OLMo 3. The open family’s most conservative synthetic posture is at pretraining scale, and its most synthetic stage is the one where every example is inspectable in a public dataset file.

The licensing angle, which is a data-stage constraint, not a legal footnote. Synthetic data inherits the terms of whatever generated it. A corpus rephrased or generated by a teacher model whose terms forbid training use is contaminated at the source, no matter how clean the resulting files look. This is why LAB2’s teacher discipline exists: among hosted APIs, DeepSeek’s terms are the one set that expressly permits training use including distillation, while OpenAI, Anthropic, Gemini, Alibaba, Moonshot, and Z.ai GLM terms prohibit it, so their outputs are eval-and-panel material only and never enter a training set. The open families route around the problem in two ways visible in this roster: self-generation (Nemotron-4’s own models generating its alignment data; a lab’s synthetic share is only as clean as its generator’s provenance) and open-weight teachers whose licenses permit it (NVIDIA’s specialist lines distill DeepSeek-R1 traces, an open-weight teacher, for OpenCodeReasoning and AceReason). When a lab does not document its synthetic generators, the licensing question is unanswerable from outside, which is one more thing weights-only openness cannot give you.

Token budgets and the overtraining question

Chinchilla’s rule of thumb (about 20 tokens per parameter for compute-optimal training) describes how to minimize training cost for a fixed capability. Almost nobody in this roster follows it, because almost nobody is optimizing training cost. They are optimizing the deployed artifact: a small model that is cheap to serve forever can justify an enormous one-time training bill. The table below uses total parameters for dense models; for MoE models both readings matter, so both are given.

Model Params Training tokens Tokens per param Reading
SmolLM2 135M 0.135B 2T about 14,800 Extreme overtraining, deliberate
SmolLM2 360M 0.36B 4T about 11,100 Extreme overtraining, deliberate
SmolLM2 1.7B 1.7B 11T about 6,500 The headline case: data quality plus budget
Granite 4.0 Micro 3B about 14.5T (staged) about 4,800 Enterprise small-model overtraining
SmolLM3 3.08B 11.2T about 3,600 Plus 100B extension and 140B reasoning midtrain
Apertus 8B 8B 15T about 1,875 Shared recipe with the 70B
Nemotron-Nano-12B-v2 12B 20T about 1,670 FP8 run, then compressed to 9B
Marin 8B 8B 12.7T about 1,590 Across named phases, incl. a rewarm
OLMo 3 7B 7.3B 5.93T about 810 Long for a research-lab budget
OLMo 2 7B 7B about 4.15T about 590 Stage 1 plus souped anneal runs
Nemotron 3 Nano 30B total / 3.5B active 25T 833 total / about 7,100 active Multi-epoch over a released 10.6T corpus
Nemotron 3 Super 120B total / 12B active 25T 208 total / about 2,080 active Two phases: 20T diversity, 5T quality
Apertus 70B 70B 15T about 214 Same corpus budget as its 8B
OLMo 3 32B 32B about 5.5T about 172 Dense, long-training school
OLMo 2 32B 32.2B about 6T exposure about 186 Stage 1 plus anneal runs
K2-Horizon 375B 375B total / 23B active 15.13T pre + 1.91T mid 40 total / about 660 active Flagship stage table, card-published
Nemotron-4 340B 340B 9T about 26 Near-Chinchilla, 2024 frontier practice
LLM360 K2-65B 65B 1.4T about 21 Chinchilla-era budget, 2023 data

Three patterns to take from the table. First, overtraining is a small-model strategy: below 4B, ratios in the thousands are normal and deliberate, because inference cost dominates lifetime cost and a 1.7B that performs like a 7B is worth 11T tokens exactly once. Second, MoE breaks the ratio’s meaning: Nemotron 3 Nano’s 25T tokens over a 10.6T released corpus means multiple epochs, and its per-active ratio (about 7,100) looks like SmolLM territory while its per-total ratio looks conventional. Quote MoE budgets per active parameter or do not quote them. Third, the flagship dense era was Chinchilla-shaped (Nemotron-4 340B at about 26, K2-65B at about 21), and the modern flagship MoEs are not: their per-active ratios run far above Chinchilla (Nemotron 3 Super at about 2,080 per active parameter, Nemotron 3 Nano higher still), because sparse models convert tokens into capability through the active path while the total parameter count mostly buys capacity and memory. One flagged gap: K2-Horizon’s per-size exposure inside the shared 22T corpus is not fully pinned in the roster (the 0.9B’s budget was not found, and the flagship runs on its own published stage table rather than the shared corpus budget), so the fleet’s small-model ratios stay uncomputed here rather than guessed.

Domain mixes: what the proportions say

Mix tables are strategy documents. Dolma 3’s is the cleanest current example (OLMo 3 tech report figures; the 9.31T pool is sampled and upsampled into the 5.93T mix):

Source Pool Mix Mix share Move
Common Crawl web 8.14T 4.51T 76.1% Downsampled: the pool is mostly web, the mix rations it
olmOCR science PDFs 972B 805B 13.6% Kept nearly whole: scarce, high-value
Stack-Edu code (rebalanced) 137B 409B 6.9% Upsampled about 3x via repetition
FineMath 3+ 34.1B 152B 2.6% Upsampled about 4.5x
arXiv 21.4B 50.8B 0.86% Upsampled about 2.4x
Wikipedia and Wikibooks 3.69B 2.51B 0.04% Present, tiny

The pattern generalizes across the roster. Web is the bulk and gets rationed. Scarce high-value domains (science PDFs, quality math, curated code) get repeated, because repetition of good data beats unique mediocre data at these scales. Code share is a deliberate dial: SmolLM3 moves it 12 to 15 to 24 percent across its three stages while math moves 3 to 10 to 13 percent, so the model’s final diet is far more specialized than its first. Nemotron’s corpus family makes the same point as separate artifacts: Nemotron-CC for general web, Nemotron-CC-Math (133B tokens at grade 3+, 52B at grade 4+, classifier-tiered like FineMath) and dedicated code corpora (Nemotron-CC-Code 427.9B, Nemotron-Pretraining-Code v1 to v3 totaling 1.7T) that a mix can dose precisely. K2-Horizon’s four shared-corpus models make the opposite design choice visible: one corpus and recipe across 3.7B to 36B, so size and sparsity are the only variables, which is exactly what a scaling study wants and exactly what a product line usually does not do. Granite’s four-stage curriculum and every family’s anneal mixes rebalance again at the end of training; that endgame is Part 6’s subject, and the pointer matters because mix decisions made in this part are only half the story until the anneal rewrites them.

Decontamination: methods and honesty levels

Decontamination is the stage where a family’s character shows, because doing it badly is invisible unless you publish the method. Three honesty tiers appear in this roster.

Tier 1: method published, failures published. Tulu 3 is the standard: 8-gram overlap checks against its eval suite, datasets dropped or filtered when overlap crossed a threshold, and the worked example left in the report, 11.3 percent of NuminaMath-TIR overlapped MATH test items. Publishing the failure is the point: it tells you the check runs and bites. The dataset-layer releases (FineWeb, DCLM, Dolma) publish their dedup and decontamination steps in datasheets and code, so the claim is rerunnable, not just stated.

Tier 2: method described, data partly closed. Granite documents its filtering and staged curriculum in its cards, but part of the corpus is proprietary, so its decontamination is a claim the reader can only partly check: the published components can be tested, the withheld ones cannot. The scores may be perfectly clean. The openness grade from Part 1 is what decides how much of the claim you can verify.

Tier 3: self-audit after the fact, published corrections. K2-Horizon supplies the roster’s most instructive case, twice. IFM disclosed that its 7B located SWE-bench answers during evaluation and produced an inflated score of 82 that did not reflect genuine capability. Separately, the flagship’s Terminal-Bench 2.1 score was revised from 70.2 to 66.9 after an internal reward-hacking audit discarded gamed solutions. Neither incident is a decontamination method. Both are what a lab does when its checks catch contamination or gaming after publication: say so, in numbers. For a study that grades honesty, a published correction is worth more than an unexamined clean claim, and Part 9 returns to this when it grades eval discipline.

One structural warning that belongs in this part: decontamination against public benchmarks gets weaker every year, because benchmark items circulate into crawl data, synthetic datasets, and other benchmarks. The families that hold out a genuinely unseen suite (Tulu 3’s two-suite discipline) are defending against exactly this decay.

The cyber read

Cyber corpus building is this same pipeline with the sources rotated, and the documentation gap between the worked examples is the whole lesson.

The raw materials are known: CVE and NVD records, patch diffs from public repositories, exploit proof-of-concepts, advisory write-ups, CTF material, audit findings. Assembled with provenance, they are a training corpus. Assembled as a marketing sentence, they are a claim. This intake cycle supplied one of each.

Documented: the Web3 Audit Corpus. A public corpus of smart-contract audit findings: about 181,000 finding occurrences, with its licensing stated per layer (CC BY 4.0 for the metadata, MIT for the code). What it documents well is provenance and rights: you know what it is, where findings come from, and what you may do with them. What limits it as training material is content depth: it carries findings, not the vulnerable code and patch diffs that would teach a model to find and fix. It is a rung-4 artifact with a shallow payload: auditable, licensed, and honest about its own shape.

Undocumented: Trident Poseidon’s data claims. Trident’s launch material describes Poseidon’s training data as 120,000 or more engineer-written CVEs plus patch diffs, exploit traces, proof-of-concepts, binaries, and crash dumps. The same launch’s social thread described the CVE count in the hundreds of thousands, a discrepancy with the blog figure that was never reconciled in public. No corpus artifact, no provenance documentation, no dedup or decontamination statement, no licensing account of the sources, and the base model is not named. The claimed payload (diffs, traces, binaries) is exactly the deep payload the Web3 corpus lacks, which is what makes the pair instructive: the documented corpus is shallow, the deep corpus is undocumented, and a builder can verify neither Poseidon’s data quality nor its contamination posture against its own published benchmark (CWE-Bench is built from public CVEs, so a model trained on CVE corpora needs a published decontamination method before its score means anything). Vendor-reported benchmark numbers were strong. Data documentation is absent. Part 1’s rule applies verbatim: you cannot audit a sentence.

For LAB2 the operational conclusions are direct. Build the cyber corpus the Dolma way: every source named, every license recorded per source, dedup and decontamination published with the corpus, temporal splits respected so eval CVEs postdate training CVEs, and the deep payload (code, diffs, traces) included because findings-only data teaches recognition, not repair. The readiness gate already encodes the first half of this: a cleaned, provenance-documented training set is a gate item, not a nice-to-have. This part adds the second half: publish the pipeline, tiers and all, because in cyber the corpus is the capability, and an undocumented corpus is an unverifiable one.

Items left flagged, on purpose

Carried from the rosters and this part’s checks, unresolved: per-size token exposure for K2-Horizon’s shared-corpus models (0.9B budget not found; flagship on its own stage table); Nemotron 3 Ultra’s pretraining token figure (card says about 20T, a blog-derived summary says about 10T plus 212B; Part 5 inherits the conflict); Granite’s full mixture proportions (proprietary components withheld by IBM’s own statement); Poseidon’s CVE count discrepancy (120,000-plus on the launch blog vs hundreds of thousands in the launch thread) and its undocumented provenance. Each later part closes its own items or keeps them flagged the same way.

Sources for this part

Roster files built for this masterclass on Nov 17, 2026: research/nemotron-roster.md, research/k2-horizon-roster.md, research/open-families-roster.md. Primary sources checked directly for this part: Nemotron-CC paper (arXiv:2412.02595, abstract and headline results); K2-Horizon tech report (arXiv:2611.21003, abstract: 22T shared corpus, roughly 10T synthetic tokens, released data pipelines); OLMo 3 tech report (arXiv:2512.13961) Dolma 3 pool and mix table via the published report figures and the allenai/dolma3_pool dataset card. From the rosters’ primary reads: FineWeb (arXiv:2406.17557) and FineWeb-Edu; DCLM (arXiv:2406.11794); Dolma datasheet; SmolLM2 (arXiv:2502.02737) and the SmolLM3 blueprint; Tulu 3 (arXiv:2411.15124); Apertus (arXiv:2509.14233); Nemotron-CC-Math (arXiv:2508.15096); Nemotron 3 and 3.5 model cards; LAB2 teacher-terms discipline (Phase 0 records, Oct 2026); intake briefs on the Web3 Audit Corpus and Trident Poseidon (Nov 2026, this lane’s files).