Model Making, in the Open: A Masterclass

Appendix C · Appendix: Open Families Roster

Appendix C · Open Families Roster: The Supporting Cast of Fully Open Model Families

Scope: model families where the recipe, the data, and the code are public, not just the weights. These are the families you can actually learn model making from, because every decision is inspectable and rerunnable. The two anchor families (NVIDIA Nemotron and K2-Horizon from IFM) get their own deep files in this masterclass. This file covers the supporting cast and ends with an openness grading table and matched-size bands against the anchors.

Grading language used throughout:

  • Weights only: you get checkpoints, nothing about how they were made.
  • + recipe: a tech report or blog documents architecture, data mix, and training stages in enough detail to imitate.
  • + data: the actual training datasets (or an exact reconstruction path) are downloadable.
  • + full pipeline: data, training code, configs, intermediate checkpoints, logs, and eval harness are all public. You could rerun the training run, modulo compute.

Primary sources are linked per family. Where a number comes from a secondary source or could not be pinned to a primary artifact, it is flagged in place and again in the gaps section at the end.


1. Ai2 OLMo 2 (Allen Institute for AI)

The reference implementation of full openness at useful scale. If you want to see every moving part of a modern training run, this is the first family to study.

Sizes and specs

Model Params Architecture Context Release License
OLMo 2 1B 1B Dense decoder, Llama-style with stability mods 4,096 2025 (added after launch sizes) Apache 2.0
OLMo 2 7B 7B Dense decoder 4,096 Nov 2024 Apache 2.0
OLMo 2 13B 13B Dense decoder 4,096 Nov 2024 Apache 2.0
OLMo 2 32B 32.2B Dense decoder 4,096 Early 2025 Apache 2.0

Architecture notes that matter for the comparison chapters: OLMo 2 keeps a fairly classic dense decoder but changes the stability recipe, namely QK-norm, reordered (post-norm style) normalization, and z-loss, plus small-init tricks. Those changes are the reason the runs did not blow up with loss spikes, and Ai2 ablated them in public. Vocabulary is 100,278 tokens. Sibling release: OLMoE 1B-7B (6.9B total, 1.3B active MoE, Sept 2024), same openness pattern, useful as the MoE contrast inside the Ai2 line.

What is public, precisely

  • Pretraining data: Stage 1 trains on OLMo-Mix-1124 (about 4T tokens for 1B and 7B, 5T for 13B, roughly 6T total exposure for 32B). The mix is built from Dolma plus StarCoder code data and Proof Pile 2 math data. The exact data files are downloadable (the repo links .csv.gz manifests of the training data), and Dolma itself is a public dataset (ODC-By).
  • Midtraining / annealing: Stage 2 uses Dolmino Mix 1124, a small curated mix (quality-filtered web, math, code, academic text, FLAN-style instruction data). The trick worth studying: they train Stage 2 multiple times from the same Stage 1 checkpoint with different data order (7B: 3 runs of 50B tokens; 13B and 32B: 3 runs of 100B plus 1 run of 300B) and average the weights. Model souping as the anneal, documented with configs per seed.
  • Training code: two repos, allenai/OLMo (7B/13B) and OLMo-core (32B and later). Full YAML configs per stage and per seed are in the repos.
  • Checkpoints and logs: every intermediate checkpoint (at minimum every 1,000 steps) in both OLMo and Hugging Face formats, plus public Weights and Biases runs per model.
  • Post-training: the Instruct models are Tulu 3 applied to OLMo 2 bases (see the Tulu section). SFT, DPO, and RLVR datasets and code are all public.
  • Eval harness: OLMES, Ai2’s standardized evaluation suite, public, with the exact prompt formats used in the report.

Grade: full pipeline. This is the bar everyone else is measured against.

Sources: OLMo 2 tech report (arXiv 2412.15115 line, “OLMo 2: Furious”), github.com/allenai/OLMo, huggingface.co/allenai.


2. Ai2 OLMo 3 and OLMo 3.1

OLMo 3 (Nov 20, 2025) is where Ai2 formalized the release as a “model flow”: every stage of the lifecycle ships as an artifact, including the branches most labs never show (Think, Instruct, RL-Zero). OLMo 3.1 (Dec 12, 2025) refreshed the 32B Think and Instruct models with extra RL.

Sizes and specs

Model Params Architecture Context Release License
OLMo 3 7B 7.3B Dense decoder, hybrid attention 65,536 Nov 20, 2025 Apache 2.0
OLMo 3 32B 32B Dense decoder, hybrid attention 65,536 Nov 20, 2025 Apache 2.0
OLMo 3.1 32B Think / Instruct 32B Same base, extra RL / post-training 65,536 Dec 2025 Apache 2.0

Architecture: the context jump (4K to 65K) comes from a hybrid attention pattern. Three of every four layers use sliding-window attention with a 4,096 window; every fourth layer is full attention, and YaRN scaling is applied only on the full-attention layers (factor 8 over 8,192 base positions). The 7B uses multi-head attention (32 Q and 32 KV heads); the 32B uses grouped-query attention. QK-norm carries over from OLMo 2. Each size ships in four tracks: Base, Think (long chain-of-thought), Instruct (chat and tool use), and RL-Zero (RL applied straight to the base for research).

What is public, precisely

  • Pretraining data: Dolma 3, a roughly 9.3T token corpus (web, science PDFs processed with Ai2’s olmOCR, code, math, encyclopedic text). The actual pretraining mix, Dolma 3 Mix, is about 5.9T tokens (the 32B card lists a 5.5T mix variant; the 7B trained on 5.93T). Mix files are public on the Hub (allenai/olmo-3-pre-training).
  • Midtraining: Dolma 3 Dolmino, a 100B token high-quality mix drawn from a roughly 2.2T token pool (allenai/dolma3_dolmino_pool is public).
  • Long-context stage: Dolma 3 Longmino, about 50B tokens drawn from a 639B token long-document pool, teaches 65K behavior after the main run (pool public).
  • Post-training data: the Dolci suite, split by stage and track: Dolci-Think SFT, Dolci-Instruct SFT, DPO preference mixes, and RLVR datasets (allenai/olmo-3-post-training). Every post-training branch checkpoint (SFT, DPO, final) is released per track, so you can diff what each stage bought.
  • Training code: OLMo-core for pretraining, open-instruct for post-training, both public with configs.
  • Eval and provenance: OLMES harness public; OlmoTrace ships alongside, an infini-gram index over Dolma 3 that lets you trace spans of model output back to training documents. No other family ships output-to-data provenance tooling.
  • Checkpoints: intermediate checkpoints at every stage boundary (base, mid-trained, long-context, each post-training stage).

Grade: full pipeline, and currently the most complete release pattern in the field.

Sources: Ai2, “Olmo 3: Charting a path through the model flow” (Nov 20, 2025), OLMo 3 tech report (arXiv 2512.13961), HF cards for allenai/Olmo-3-7B-Think and Olmo-3-1125-32B.


3. Allen AI Tulu 3 (post-training recipe line)

Tulu 3 is not a base model family. It is the most complete open post-training recipe ever published, and it matters to this masterclass because post-training is where most “open” releases go silent. Released Nov 22, 2024, applied to Llama 3.1 bases at 8B and 70B, with a 405B version added in Feb 2025.

The recipe, stage by stage

  1. Data curation: 939,344 SFT prompts, about 57% from public datasets (WildChat, OpenAssistant, FLAN v2, NuminaMath, OpenMathInstruct, Evol-CodeAlpaca, SciRIFF, Aya, others) and 43% synthetic, generated with a persona method (roughly 250,000 personas). Aggressive decontamination: 8-gram overlap checks against the eval suite, with whole datasets dropped or filtered when overlap passed a threshold. Their worked example: 11.3% of NuminaMath-TIR overlapped MATH test items.
  2. SFT: skill-targeted mixes built per capability, combined, then iterated against a development eval suite. Two epochs at 8B.
  3. DPO: about 271K to 300K preference pairs, mixing off-policy pairs with on-policy pairs generated from the SFT model and judged by a model. Length-normalized DPO.
  4. RLVR: PPO against verifiable rewards (10 for a provably correct math answer or satisfied instruction constraint, 0 otherwise, KL penalty to the DPO model) on the released RLVR mixes (GSM, MATH, IFEval-style constraints).

What is public, precisely

  • Datasets: the full SFT mixture (allenai/tulu-3-sft-mixture) plus per-skill subsets, the preference mixture and per-skill preference sets, and the RLVR mixes. All on the Hub.
  • Code: allenai/open-instruct, the actual training and data tooling, with configs.
  • Checkpoints: SFT, DPO, and RLVR checkpoints released separately at 8B and 70B (405B ships SFT and DPO stages), so stage-by-stage deltas are measurable.
  • Eval: a two-suite discipline worth copying, a development suite used for decisions and an unseen suite used only to check generalization, both public.

License caveat, stated plainly: the recipe, data, and code are Ai2’s and openly licensed, but the Tulu 3 weights sit on Llama 3.1 bases, so the checkpoints inherit the Llama Community License, not Apache 2.0. When Ai2 later applied the same recipe style to its own bases (OLMo 2 Instruct, then Dolci for OLMo 3), the license chain became clean Apache 2.0 end to end.

Grade: full pipeline for the post-training half of model making. Pretraining is out of scope by design.

Sources: Tulu 3 tech report (arXiv 2411.15124), allenai/open-instruct, HF collection allenai/tulu-3-models.


4. Hugging Face SmolLM2

SmolLM2 (Nov 2, 2024; paper Feb 2025, COLM 2025) is the data-centric small-model case study. The headline is not the model, it is that a 1.7B model overtrained on 11T tokens of carefully staged public data beat models trained on private data at the same size.

Sizes and specs

Model Params Architecture Context Training tokens License
SmolLM2 135M 135M Llama-style dense decoder 8K (see note) 2T Apache 2.0
SmolLM2 360M 360M Llama-style dense decoder 8K (see note) 4T Apache 2.0
SmolLM2 1.7B 1.7B Llama-style dense decoder 8K (16K variant exists) 11T Apache 2.0

Note on context: training ran at 2,048 sequence length for ablations and the released models are rated at 8K context; treat exact per-size context as card-level data, flagged in gaps.

What is public, precisely

  • Pretraining data: every component dataset is public and most were built for this project. Web: FineWeb-Edu and DCLM (the winning ablation was 60% FineWeb-Edu, 40% DCLM, about 5.1T unique tokens). Math: FineMath (54B tokens, tiered FineMath4+ and FineMath3+), built because OpenWebMath and InfiMM-WebMath were not good enough. Code: Stack-Edu (about 125B tokens, StarCoder2Data filtered with per-language educational classifiers across 15 languages), plus StarCoderData in early stages.
  • Stage structure: the 1.7B used a four-stage curriculum (0 to 6T web-heavy, 6 to 8T math and code introduced, 8 to 10T higher-quality specialized data, 10 to 11T decay phase upsampling the best math and code). The small models deliberately used a single-stage mix instead, and the paper explains why (small models benefited from consistently high-quality data from the start). That kind of negative-result detail is exactly what closed recipes never give you.
  • Training code: trained with nanotron (Hugging Face’s open trainer); configs and the development log are in github.com/huggingface/smollm. The paper documents the roughly $250K compute cost of the 1.7B run, which is a useful calibration number.
  • Post-training: SmolTalk (the SFT mixture, public, built on Magpie-Ultra plus targeted subsets) and UltraFeedback for DPO. The 135M and 360M used a smaller smol-smoltalk variant. Recipes live in the Alignment Handbook repo.
  • Eval: the paper’s ablation harness setup (lighteval-based) is described and the ablation models are released, so the data decisions can be re-measured.

Grade: full pipeline.

Sources: SmolLM2 paper (arXiv 2502.02737), github.com/huggingface/smollm, huggingface.co/HuggingFaceTB.


5. Hugging Face SmolLM3

SmolLM3 (July 8, 2025) is a single-size release, 3B, built as an “engineering blueprint”: Hugging Face published the architecture reasoning, the exact per-stage data mixtures, the long-context extension recipe, and the hybrid reasoning post-training method.

Specs

Model Params Architecture Context Training tokens License
SmolLM3 3B 3.08B Dense decoder, GQA (4 groups), tied embeddings 64K trained, 128K via YaRN 11.2T pretraining + 100B extension + 140B reasoning midtraining Apache 2.0

Architecture decision to study: NoPE. Every fourth layer drops rotary position embeddings entirely (a 3:1 RoPE to NoPE pattern), which the team found improved long-context behavior without hurting short context. Dual-mode reasoning: the chat template supports /think and /no_think modes in one checkpoint, trained in, not prompted in.

What is public, precisely

  • Pretraining mixtures, by stage, with percentages: Stage 1 (0 to 8T): 85% web (FineWeb-Edu, DCLM, FineWeb2, FineWeb2-HQ, 12% multilingual), 12% code (Stack v2 and friends), 3% math (FineMath3+, InfiWebMath3+). Stage 2 (8 to 10T): web 75%, code 15%, math 10%. Stage 3 decay (10 to 11.1T): web 63%, code 24%, math 13% with instruction and reasoning data (OpenMathReasoning) folded in. Every named component is a public dataset.
  • Long-context extension recipe: 100B extra tokens in two 50B steps, 4K to 32K with RoPE theta raised to 1.5M, then 32K to 64K with theta at 5M, YaRN extrapolation to 128K at inference. The blog reports what did not work too: upsampling specialized long-context data did not beat the NoPE plus theta approach.
  • Reasoning midtraining: 140B tokens (4 epochs over a 35B token set) on OpenThoughts3 and the Llama-Nemotron post-training reasoning data, both public.
  • Post-training: SmolTalk2 (public, with separate think and no-think subsets), then Anchored Preference Optimization (APO) instead of vanilla DPO, then a linear model merge (0.9 APO checkpoint plus 0.1 midtraining checkpoint) to recover long-context performance lost during alignment. Configs are in the Alignment Handbook (recipes/smollm3).
  • Checkpoints: base, midtrained, SFT, and final checkpoints are on the Hub, plus intermediate pretraining checkpoints.

Grade: full pipeline. The one soft spot: there is no formal paper, the blueprint is a long blog post plus cards and configs. Everything needed to reproduce is public, but you assemble it from more places than with OLMo.

Sources: Hugging Face blog “SmolLM3: smol, multilingual, long-context reasoner”, HF model cards, Alignment Handbook recipes.


6. Apertus (Swiss AI Initiative: EPFL, ETH Zurich, CSCS)

Apertus (Sept 2, 2025) is the sovereignty case study: a fully open model built by public institutions, on public compute (the Alps supercomputer, up to 4,096 GPUs), with data compliance treated as a design constraint rather than a legal afterthought.

Sizes and specs

Model Params Architecture Context Training tokens License
Apertus 8B 8B Dense decoder, xIELU activation 65,536 15T (shared recipe) Apache 2.0
Apertus 70B 70B Dense decoder, xIELU activation 65,536 15T Apache 2.0

Distinctive engineering choices: trained from scratch with the AdEMAMix optimizer instead of AdamW; a novel xIELU activation; memorization suppression during training (the tech report describes goldfish-style token dropping); and a corpus spanning 1,811 languages with over 40% non-English content by design. Apertus 1.5 followed later with continued multimodal pretraining (about 4T extra tokens for 8B, 2T for 70B) and context extended to 262,144; treat 1.5 details as a second-generation extension of the same openness pattern.

What is public, precisely

  • Pretraining data: the corpus is built only from openly available sources, filtered for non-permissive licenses, toxic content, and PII, and, uniquely, with robots.txt opt-outs applied retroactively to earlier crawls. The release includes the data preparation code and dataset reconstruction path (swiss-ai/pretrain-data, swiss-ai/pretrain-code), so the corpus is reproducible from its public sources. This is a reconstruction release, not a single downloadable token dump; that distinction matters and is graded accordingly below.
  • Training code: swiss-ai/Megatron-LM fork plus pretraining code, with configs.
  • Post-training: SFT and preference alignment with data released (swiss-ai/posttraining, swiss-ai/posttraining-data). The 1.5 line exposes per-stage repos (base, SFT, DPO, RL), which makes stage deltas inspectable. Nuance: the 1.5 post-trained finals sit behind a Hugging Face gate that records assent to an acceptable-use policy. Weights remain Apache 2.0, but the gate is a real deviation from the ungated pattern elsewhere in this roster.
  • Checkpoints: final and intermediate checkpoints released as branches of the model repos.
  • Eval: evaluation suites released (swiss-ai/evals, lm-evaluation-harness fork), and the alignment charter is published alongside the models.

Grade: full pipeline, with the data leg delivered as documented reconstruction scripts plus preparation code rather than one frozen corpus file.

Sources: Apertus v1 tech report (arXiv 2509.14233), Apertus v1.5 tech report (arXiv 2610.15542), CSCS and ETH Zurich release materials, swiss-ai org on GitHub and HF.


7. IBM Granite 4.x

Granite is the enterprise case, and the honest answer to “how open is it” is: the weights are clean Apache 2.0, the recipe is documented at the level of a good tech report, the data is only partly public, and the training code for pretraining is not released. Granite does not belong in the fully open tier, and this roster includes it precisely so the boundary is visible.

Granite 4.0 (Oct 2, 2025)

Model Params Architecture Context Training tokens (staged total) License
Granite 4.0 Micro 3B Dense decoder 128K class about 14.5T Apache 2.0
Granite 4.0 H-Micro 3B Hybrid Mamba-2 / Transformer (9:1) 128K class about 17.5T Apache 2.0
Granite 4.0 H-Tiny 7B total, 1B active Hybrid MoE 128K class about 22.5T Apache 2.0
Granite 4.0 H-Small 32B total, 9B active Hybrid MoE 128K class about 22.5T Apache 2.0

The model card publishes the four-stage curriculum table (general mix, code and math upweighting, then two high-quality decay stages with different LR schedules) and the infrastructure (GB200 NVL72 on CoreWeave). The card also states the data is “a mix of open source and proprietary data”, which settles the openness question by itself.

What is public around the model: GneissWeb, IBM’s 10T token web dataset, is a real public dataset with its filtering recipe and the open-source Data Prep Kit that built it. Named public components of the mix include DCLM and TxT360 subsets. What is not public: the full mixture proportions with the proprietary components, the complete pretraining data, and the pretraining code and configs.

Granite 4.1 (Apr 29, 2026)

Granite 4.1 went back to plain dense decoders: 3B, 8B, and 30B, GQA with RoPE, configs at 131,072 positions and marketed context up to 512K, trained on roughly 15T tokens per IBM’s materials. The 8B is the headline: IBM reports it matching the 4.0 H-Small 32B MoE on enterprise benchmarks. Openness pattern is unchanged from 4.0: Apache 2.0 weights, documented recipe summaries, GneissWeb and Data Prep Kit public, full training data and pretraining pipeline not released. (Granite 4.2 followed in Aug 2026; not covered here, flagged in gaps.)

Grade for the Granite 4.x line: weights + recipe (documented, partial data). One full grade below the fully open tier. For LAB2-style work it remains a strong base candidate on license cleanliness, but you cannot study its training run the way you can study OLMo’s.

Sources: Granite 4.0 HF model cards (ibm-granite/granite-4.0-micro-base and siblings), IBM Granite 4.0 announcement, IBM Research Granite 4.1 announcement (Apr 29, 2026), GneissWeb dataset card (ibm-granite/GneissWeb), Data Prep Kit (github.com/data-prep-kit).


8. LLM360 (Amber, CrystalCoder, K2-65B)

LLM360 (MBZUAI, Petuum, CMU and partners) started the fully transparent movement in Dec 2023, before OLMo 2 made it fashionable. The artifacts are older and weaker as models, but the release discipline, hundreds of checkpoints plus the exact data sequence, is still a teaching resource nobody else matched at 65B scale for a long time.

Models

Model Params Architecture Context Training tokens Release License
Amber 6.7B Llama-style dense decoder 2,048 1.25T Dec 2023 Apache 2.0 (data ODC-By)
CrystalCoder 6.7B Llama-style dense decoder 2,048 1.4T Dec 2023 Apache 2.0 (data ODC-By)
K2 (Diamond) 65B Llama-style dense decoder 2,048, extended to 8,192 in stage 2 1.4T 2024, tech report Jan 2025 Apache 2.0 (data ODC-By)

What is public, precisely

  • Data: the exact pretraining data sequence is published (LLM360/K2Datasets on the Hub, plus per-model dataset repos). K2’s mix table is public down to per-source token counts and upsampling multipliers (RefinedWeb about 47%, RedPajama books, StarCoder, DM Math at 3x, and so on, 1.3T in stage 1).
  • Checkpoints: Amber ships 360 intermediate checkpoints, CrystalCoder about 143 to 152 (sources differ slightly by counting method), K2 saved 360 and uploaded 120 (each checkpoint is over 100 GB, storage was the binding constraint, and they say so).
  • Code and logs: training code on GitHub (github.com/LLM360), full Weights and Biases logs, and prompt and evaluation galleries that show generations from intermediate checkpoints.
  • Post-training: K2 Chat’s SFT datasets are named and public (OpenHermes-2.5, FLAN subsets, MathInstruct, plus a small safety set built from Do-Not-Answer), with the fine-tuning code released.
  • Follow-ons: the line continued with TxT360 (a documented corpus with upsampling methodology), K2-Think (RL-focused reasoning release), and a K2-V2 generation with named midtraining (TxT360-Midas) and SFT (TxT360-3efforts) datasets. The follow-ons are only partially verified at primary sources for this file; see gaps.

Grade: full pipeline for Amber, CrystalCoder, and K2-65B.

Sources: LLM360 paper (arXiv 2312.06550), K2 tech report (arXiv 2501.07124), huggingface.co/LLM360, wandb.ai/llm360.


9. Marin (Stanford CRFM open lab)

Marin is an open lab, not just an open model: every experiment is a public GitHub issue, preregistered, with failures kept in the record. The models are trained in JAX on TPUs (Google TPU Research Cloud) with the Levanter framework, which also makes Marin the roster’s one serious non-PyTorch, non-GPU data point.

Models

Model Params Architecture Context Training tokens Release License
Marin 8B Base 8B Llama 3.1 architecture config, dense 4,096 12.7T total across phases May 2025 Apache 2.0
Marin 8B Instruct 8B Same base, SFT only 4,096 SFT about 5.3B tokens (card figure, see gaps) 2025 Apache 2.0
Marin 32B Base 32B Llama-3-style dense 4,096 class (see gaps) about 6.4T Late 2025 Apache 2.0

What is public, precisely

  • The run itself, phase by phase: Marin 8B’s retrospective names every phase (Kestrel, Ocelot, Jellyfish, Phoenix, Starling, Deeper Starling), including a discarded cooldown and a mid-run decision to rewarm and switch the mixture to Nemotron-CC. Final mix components are all public datasets: DCLM Baseline, StarCoder Data, Proof Pile 2, Nemotron-CC, FineMath, and OLMo 2’s Dolmino mix in the cooldown. This is the best public record of what a real training run looks like when it does not go in a straight line.
  • Checkpoints: a named checkpoint per phase on the Hub (44 intermediate checkpoints remain after a Nov 2026 storage pruning, with all named milestones kept).
  • Code: the whole stack is open, Levanter (trainer), Haliax, and the marin repo where experiments are tracked as issues with W&B links. Data mixture pipelines in the repo deterministically reproduce the mixes from the public source datasets.
  • Scaling science: the Delphi suite publishes scaling experiments from 3e18 to 1e23 FLOPs with fitted laws, which is the kind of artifact you need before choosing a training budget.
  • Post-training: thin by design so far. The 8B Instruct is SFT-only, and the 32B has no post-trained release. Marin is a pretraining teacher, not a post-training one. A frontier MoE program (a 67B-A2B run and a larger hero run) is in progress in the open; not graded here.

Grade: full pipeline for pretraining, with the caveat that the artifact trail lives across issues, docs, and W&B rather than in one tech report.

Sources: Marin 8B retrospective (marin.readthedocs.io), Marin 32B retrospective, Google Open Source Blog on training Marin 32B (Dec 2025), github.com/marin-community/marin, HF marin-community/marin-8b-base.


10. DataComp-LM (DCLM) baselines

DCLM (June 2024, NeurIPS 2024) is a testbed, not a product family: fix the training recipe and compute, vary only the dataset, and measure. It belongs in this roster because half the families above train on its outputs, and because its central finding (model-based filtering beats heuristic filtering, by a lot) is one of the load-bearing results in open data work.

What it ships

  • DCLM-Pool: a standardized 240T token corpus extracted from Common Crawl, public.
  • Fixed recipes: OpenLM-based training configs at five scales, 412M to 7B parameters, with a 53-task evaluation suite (CORE and extended sets).
  • DCLM-Baseline: the winning filtered dataset, built by a fastText classifier trained to separate instruction-style text (OpenHermes-2.5 and top ELI5 posts as positives) from random web pages, keeping the top 10%. The classifier, thresholds, and full filtering pipeline are public.
  • Baseline models: the flagship 7B trained on 2.6T tokens reached 64% MMLU (5-shot), beating the previous open-data best (MAP-Neo) by 6.6 points with 40% less compute, and landing near Mistral-7B and Llama 3 8B at a fraction of their compute. Honest caveat from the paper’s own follow-up analysis: the final 7B run added StarCoder and Proof Pile 2 and a long-context cooldown, so the headline number is not a pure single-variable result.

Grade: full pipeline as a dataset testbed. The models are baselines, deliberately ordinary architectures, because the dataset is the variable under study.

Sources: DCLM paper (arXiv 2406.11794), datacomp.ai/dclm, github.com/datacomp-lm.


11. FineWeb line (Hugging Face) and the dataset-first releases

FineWeb (May 2024) is a dataset release with ablation models attached, and it is the substrate under SmolLM, much of Marin’s ecosystem, and indirectly half this roster.

  • FineWeb: 15T tokens from 96 Common Crawl snapshots (2013 to 2024), with the full filtering and dedup pipeline published as the datatrove library and every major decision ablated in the paper (arXiv 2406.17557). License ODC-By 1.0.
  • FineWeb-Edu: a 1.3T token subset kept by an educational-value classifier (Llama-3-70B annotated 500K samples, a small embedding classifier did the full pass, threshold 3 of 5). At fixed compute it trains better models than the full 15T pool. That result is the empirical foundation of the quality-filtering era.
  • FineWeb2: the same recipe extended to 1,000+ languages (2025), and the source of SmolLM3’s multilingual web data and part of Apertus’s lineage.
  • Ablation models: Hugging Face released the small models trained for the filtering ablations, so the dataset claims are checkable against actual checkpoints.
  • Adjacent: the Common Pile (EleutherAI and partners, June 2025), about 8 TB of public-domain and permissively licensed text with demonstration models, is the provenance-first counterpart to FineWeb’s crawl-first approach. Details of its model releases were not fully verified for this file; see gaps.

Grade: full pipeline at the dataset layer. Not a model family, but no roster of open training is complete without it.

Sources: FineWeb paper (arXiv 2406.17557), HuggingFaceFW dataset cards, datatrove repo.


12. EuroLLM (EU consortium)

EuroLLM (Instituto Superior Tecnico, Unbabel, University of Edinburgh, and partners; EU and EuroHPC funded) is the boundary case that keeps the grading honest. It calls itself fully open, and parts of that claim hold.

Models

Model Params Architecture Context Training tokens Release License
EuroLLM 1.7B 1.7B Dense decoder, GQA 4K about 4T recipe family Feb 2025 Apache 2.0
EuroLLM 9B 9.2B Dense decoder, GQA 4K 4T 2025 Apache 2.0
EuroLLM 22B 22.6B Dense decoder, GQA, 54 layers 4K (32K claimed via RoPE scaling in secondary sources) about 4T Dec 2025 Apache 2.0

Trained from scratch on MareNostrum 5 (400 H100s for the 22B), 35 languages, all 24 official EU languages plus 11 more.

What is public, precisely

  • Weights and checkpoints: Apache 2.0, ungated, including per-phase Megatron-Core checkpoints for all three sizes.
  • Recipe: the tech reports (9B: arXiv 2506.04079; 22B: arXiv 2602.05879) document tokenizer, architecture, data mixture, and training procedure in real detail, and the EuroFilter classifier is released.
  • Data, partial: EuroWeb-2512 (the multilingual web subset) and EuroBlocks-SFT-2512 (instruction data) are public. The rest of the roughly 4T token mix (parallel data, code and math components, synthetic math, long-context data) is documented in the report but not released, and the released datasets carry no explicit license tag on their cards.
  • Code, weak: the EuroLLM-specific code is the evaluation harness; the cited pretraining repo is a generic Megatron-LM fork without EuroLLM configs, so the training run is not reproducible from released code.

Grade: weights + recipe, data partial. Stronger than any weights-only release, clearly short of OLMo-class openness. The gap between the “fully open” label and the artifact list is itself a teaching example.

Sources: EuroLLM tech reports, utter-project org on HF, os-ai-map openness review (currentai-org/os-ai-map PR 494, Sept 2026).


Openness grading table

Family Weights Recipe documented Data public Code + configs Checkpoints + logs Eval harness Grade
OLMo 2 Yes Yes Yes (exact files) Yes Yes, every 1K steps + W&B Yes (OLMES) Full pipeline
OLMo 3 / 3.1 Yes Yes Yes (Dolma 3 mixes + pools) Yes Yes, per stage and branch Yes (OLMES, OlmoTrace) Full pipeline
Tulu 3 (post-training) Yes (Llama license) Yes Yes (SFT/DPO/RLVR) Yes (open-instruct) Yes, per stage Yes (dev + unseen suites) Full pipeline, post-training only
SmolLM2 Yes Yes Yes (all components) Yes (nanotron, configs) Yes Yes (ablations) Full pipeline
SmolLM3 Yes Yes (blog blueprint) Yes (all components, exact mixes) Yes (configs, handbook recipes) Yes Partial (blog-reported) Full pipeline
Apertus Yes Yes Reconstruction scripts + prep code Yes Yes Yes Full pipeline (data via reconstruction)
LLM360 (Amber, CrystalCoder, K2) Yes Yes Yes (exact sequence) Yes Yes (120 to 360 per model) Yes (galleries) Full pipeline
Marin Yes Yes (retrospectives + issues) Yes (public components, deterministic mixes) Yes (Levanter, marin) Yes (per phase) Yes Full pipeline (pretraining)
DCLM baselines Yes Yes Yes (pool + filtered set) Yes (OpenLM) Yes Yes (53 tasks) Full pipeline (testbed)
FineWeb line Ablation models Yes Yes (the dataset is the release) Yes (datatrove) Ablation checkpoints Yes (in paper) Full pipeline (dataset layer)
EuroLLM Yes Yes Partial (web + SFT only) Weak (generic fork) Yes (per phase) Yes Weights + recipe, data partial
IBM Granite 4.x Yes Partial (stage tables, mix summary) Partial (GneissWeb + named public parts; proprietary parts withheld) No (pretraining) No Partial Weights + recipe, data partial

Matched-size bands vs the anchors

Two ways to match sizes, and the difference is the lesson. Dense families match on total parameters. MoE anchors (Nemotron 3, K2-Horizon’s MoVA line) have two sizes at once: total parameters (memory, storage) and active parameters (compute per token). A 30B-A3B Nemotron sits in the 30B band on the shelf and in the 3B band on the compute bill.

Anchor reference points used below: Nemotron line, Nano 4B class and Nano 9B v2 at the small end, Nemotron 3 Nano 30B-A3B, Nemotron 3.5 Lightning 30B-A3B, Nemotron 3 Super 120B-A12B, Nemotron 3 Ultra 550B-A55B, plus the older Nemotron 4 340B. K2-Horizon line: 0.9B, 3.7B, 7B, and 32B dense, MoVA-36B-A4B, and the 375B-A23B flagship, all Apache 2.0 with data, code, and checkpoints public (full detail in research/k2-horizon-roster.md).

Band Supporting cast (fully open) Nemotron anchor K2-Horizon anchor Notes
Sub-1B to 2B SmolLM2 135M / 360M / 1.7B, OLMo 2 1B, EuroLLM 1.7B None this small in the current line K2-Horizon 0.9B The fully open world owns this band outright. Best teaching sizes: cheap to rerun.
3B SmolLM3 3B, Granite 4.0 Micro and H-Micro 3B Nemotron 3 Nano by active params (3B active) K2-Horizon 3.7B Cleanest contrast in the roster: dense 3B models trained several different ways against MoEs that compute like 3B models.
7 to 9B dense OLMo 2 7B, OLMo 3 7B, Marin 8B, Apertus 8B, EuroLLM 9B, LLM360 Amber and CrystalCoder 7B, DCLM 7B, Granite 4.1 8B Nemotron Nano 9B v2 (hybrid, nearest total-params match) K2-Horizon 7B The most crowded band and the best documented. Eight independent recipes at one size is a gift for comparison chapters.
13 to 16B OLMo 2 13B None current None Thin band. OLMo 2 13B is the only fully open dense entry.
22 to 32B OLMo 2 32B, OLMo 3 32B, Marin 32B, EuroLLM 22B, Granite 4.0 H-Small 32B-A9B, Granite 4.1 30B Nemotron 3 Nano 30B-A3B, Nemotron 3.5 Lightning 30B-A3B (total params) K2-Horizon 32B dense, MoVA-36B-A4B The masterclass sweet spot. Same shelf size, wildly different recipes: dense long-training (OLMo), hybrid MoE (Nemotron), and IFM’s dense-versus-MoVA controlled pair.
65 to 70B Apertus 70B, LLM360 K2 65B, Tulu 3 70B (post-training only) Llama-3.1-Nemotron-70B (derivative fine-tune, not an NVIDIA from-scratch run) None Apertus vs K2-65B is the cleanest same-size, different-era comparison: 15T tokens and compliance-first data vs 1.4T tokens and 2023-era data at the same openness grade.
100 to 120B None fully open Nemotron 3 Super 120B-A12B None A real gap. No supporting-cast family publishes a full pipeline at this size.
340 to 550B Tulu 3 405B (post-training recipe only, Llama base) Nemotron 4 340B, Nemotron 3 Ultra 550B-A55B K2-Horizon 375B-A23B Full-pipeline pretraining openness stops at this band. Only the post-training layer (Tulu) reaches it from the supporting cast.
1T class None None None The trillion class has no fully open entry. Nothing in this study occupies it; the ladder tops out at Nemotron 3 Ultra’s 550B.

Gaps and unverified items

Read this before citing anything above in a build decision.

  1. OLMo 2 release dates per size: 7B and 13B in Nov 2024 are solid. The 32B is “early 2025” per secondary sources; the exact announcement date was not pinned to an Ai2 primary post in this pass. The 1B’s addition date is likewise approximate.
  2. OLMo 3 mix sizes: Ai2 materials give Dolma 3 Mix as about 5.9T tokens in prose and 5.5T in the 32B mix name (dolma3_mix-5.5T-1125), with the 7B at 5.93T. The discrepancy is in the primary sources themselves (mix name vs rounded prose), not a transcription error here.
  3. SmolLM2 context length: ablation training ran at 2,048 sequence length; cards and secondary sources variously rate the released models at 8K, with a 16K variant for the 1.7B. Per-size context was not verified card by card.
  4. Apertus data leg: whether every corpus component is directly downloadable versus reconstructible from scripts and source lists varies by component. The tech report’s framing is full release of preparation scripts, checkpoints, eval suites, and training code. Treat “data public” for Apertus as “data reproducible from released artifacts”, which is how it is graded above. The goldfish-loss detail is from the tech report’s memorization discussion and was not independently re-verified line by line.
  5. Apertus 1.5 and the 4B claim: one secondary source claimed a 4B v1.1 size (June 2026). Not confirmed at a primary source in this pass, so it is excluded from the tables.
  6. Granite 4.1 data specifics: IBM’s 4.1 materials confirm sizes, architecture, and rough token count (about 15T) via secondary reporting of IBM’s announcement and cards. A primary-source, line-item data statement for 4.1 (equivalent to the 4.0 card’s open-plus-proprietary wording) was not read directly in this pass. The openness grade assumes the 4.0 pattern carries over; verify against the 4.1 cards before relying on it. Granite 4.2 (Aug 2026) is out of scope here.
  7. Granite 4.0 token totals: staged totals are summed from the 4.0 Micro card’s stage table (10/2/2/0.5 for Micro, 15/5/2/0.5 for the H MoE models). IBM’s own press framing rounds differently (“22T dataset”). Use the card table for engineering purposes.
  8. LLM360 follow-ons: K2-V2, TxT360-Midas, TxT360-3efforts, and K2-Think details come from a mix of the AI Alliance state-of-open-training survey and LLM360 author pages, not from reading each primary artifact end to end. CrystalCoder checkpoint counts differ slightly between sources (143 vs 152).
  9. Marin figures: the 8B Instruct SFT token figure (about 5.3B) comes from the base model’s HF card text and may describe the instruct run approximately. Marin 32B context length was not pinned to the retrospective in this pass; the 8B’s 4,096 is verified. Marin’s newer MoE runs (67B-A2B, larger hero run) are in progress and ungraded.
  10. DCLM model license: the DCLM datasets, code, and baseline models are public via datacomp.ai and the datacomp-lm org, but the exact license tags on the baseline model repos were not verified in this pass.
  11. Common Pile models: the dataset (v0.1, June 2025, about 8 TB permissive text) is confirmed; the demonstration model sizes and training details were not verified at primary sources, so no model rows were included.
  12. EuroLLM 22B tokens and context: the roughly 4T token figure is documented for the 9B and carried across the family in secondary sources; the 22B’s exact token count and the 32K context claim (vs the 4K card figure) need the 22B tech report read directly before use.
  13. Trillion-class gap: no fully open family publishes at 1T total parameters or above. The largest fully documented entries in this study are Nemotron 3 Ultra (550B) and K2-Horizon (375B); any trillion-class comparison would have to rest on weights-only releases, which this study excludes by scope.