Part 3 · Part 3: Tokenizers and Vocabularies, the Quiet Tax
Model-Making Masterclass, Part 3. Written Nov 17, 2026. Engineer voice: plain language, terms defined when they first matter, every number tied to a config file, a tech report, or flagged as an estimate.
The thesis: the first layer is a budget decision
Before a model sees a single transformer layer, the tokenizer has already spent part of its parameter budget, set the price of every document it will ever read, and decided which languages and which kinds of text get first-class treatment. The tokenizer is a fixed, frozen artifact: a vocabulary of subword pieces, learned once from data, that maps text to integer IDs. Each ID indexes a row in an embedding matrix, and in most models a second matrix of the same shape turns the final hidden state back into scores over the whole vocabulary. Those two matrices are sized vocabulary times hidden dimension, and they are not small. At the sizes this study cares about, the vocabulary matrices run from 6 percent of a small model to 26 percent of a mid-size one, and the share is regressive: the tax is a fixed number of parameters, so it falls hardest on exactly the small models that can least afford it.
Nobody puts the tokenizer on the launch slide. This part does the accounting anyway: the vocabulary table across every family in the study, what a large vocabulary buys, what it costs, how each lab built or inherited its tokenizer, and what all of this means when the text is logs, hashes, and shell history instead of prose.
The vocabulary table
Vocabulary sizes, hidden dimensions, and tying status below were read from the models’ own config.json files on Hugging Face for this part, except where a tech report is the source (Nemotron-4, whose report publishes its own embedding counts) or a paper is (EuroLLM). “Vocab matrices” means the input embedding plus the output head where the two are untied, or the single shared matrix where they are tied. Shares are computed against the released total parameter counts from the rosters and cards, rounded, and marked as computed.
| Model | Vocab | Hidden | Tied | Vocab matrices | Share of total |
|---|---|---|---|---|---|
| SmolLM2 135M | 49,152 | 576 | yes | 28M | about 21% |
| SmolLM2 1.7B | 49,152 | 2,048 | yes | 101M | about 6% |
| SmolLM3 3B | 128,256 | 2,048 | yes | 263M | about 9% |
| Granite 4.0 Micro 3B | 100,352 | 2,560 | yes | 257M | about 9% |
| K2-Horizon 0.9B | 64,256 | 1,536 | no | 197M | about 20% |
| K2-Horizon 3.7B | 250,624 | 2,560 | no | 1,283M | about 26% |
| K2-Horizon 7B | 250,624 | 4,096 | no | 2,053M | about 23% |
| K2-Horizon 32B | 250,624 | 5,120 | no | 2,566M | about 7% |
| K2-Horizon MoVA-36B-A4B | 250,624 | 2,560 | no | 1,283M | about 3.5% |
| K2-Horizon 375B-A23B | 250,624 | 6,144 | no | 3,079M | about 0.8% |
| OLMo 2 7B | 100,352 | 4,096 | no | 822M | about 11% |
| OLMo 3 7B | 100,278 | 4,096 | no | 821M | about 11% |
| Marin 8B | 128,256 | 4,096 | no | 1,051M | about 13% |
| Apertus 8B | 131,072 | 4,096 | no | 1,074M | about 13% |
| Nemotron-H 8B | 131,072 | 4,096 | no | 1,074M | about 13% |
| Nemotron 3 Nano 30B-A3B | 131,072 | 2,688 | no | 705M | about 2.2% |
| Nemotron-4 15B | 256,000 | 6,144 | no | 3,200M (report figure) | about 20% |
| Nemotron-4 340B | 256,000 | 18,432 | no | 9,400M (report figure) | about 2.8% |
Three more rows are inherited tokenizers, where the model in the study did not choose the vocabulary at all. The Llama-Nemotron derivatives carry Llama’s 128,256. The Qwen-derived Nemotron specialist lines carry Qwen’s 151,936. Nemotron 3.5 Bolt E4B, a post-train of Google’s Gemma 4, carries Gemma’s 262,144, the largest vocabulary in the study, chosen by a different lab for a different model (upstream figure, not re-read from the Bolt config in this pass). LLM360’s Amber, CrystalCoder, and K2-65B sit at the other end of history with the Llama tokenizer’s 32,000. EuroLLM’s paper reports a 128,000 vocabulary, sized by experiment; its 9B config was not readable in this pass, so no share is computed for it.
The spread is the story. Vocabularies in one study run from 32,000 to 262,144, an 8x range, and the parameter tax runs from under 1 percent at the flagship end (K2-Horizon’s 375B) to a quarter of the whole model at 3.7B.
The worked lesson: K2-Horizon’s nameplates
The K2-Horizon roster flagged a naming puzzle: the “7B” model holds about 9B parameters in its safetensors files, the “3.7B” about 5B, the “32B” about 35B. The configs solve it exactly, and the solution is this part in miniature.
The fleet from 3.7B up shares one vocabulary of 250,624 pieces, untied. Run the arithmetic per model. At 7B, hidden size 4,096: one matrix is 250,624 times 4,096, about 1.03B parameters, and the untied pair is about 2.05B. Subtract from the roughly 9B total and the transformer core is about 6.95B. The nameplate is the core. At 3.7B, hidden 2,560: the pair is about 1.28B, and 5B minus 1.28B leaves about 3.7B, which is why the model’s own card calls it a “3.7B-core” model. At 32B, hidden 5,120: the pair is about 2.57B, and 35B minus 2.57B leaves about 32.4B. Three nameplates, three exact reconciliations. IFM names the engine and bills the vocabulary separately, and anyone doing hardware math from the nameplate alone underbuys memory by the size of the tax.
The same table shows the tax’s regressivity inside one fleet. The identical 250,624 vocabulary costs the 3.7B model about 26 percent of its parameters, the 7B about 23 percent, the 32B about 7 percent, and the 375B flagship under 1 percent. IFM clearly did the same math, because the one fleet member exempted from the big vocabulary is the smallest: the 0.9B drops to a 64,256 vocabulary, and even then its untied pair is about 197M of a roughly 1B model, about 20 percent. There is no size at which a serious vocabulary is free. There is only a size at which it stops being the dominant line item.
Nemotron-4 makes the same point across generations. NVIDIA’s report publishes the split directly: the 340B has 9.4B embedding parameters against 331.6B non-embedding, about 2.8 percent. Its 15B sibling, same 256,000 vocabulary, carries 3.2B of embedding in a roughly 15.7B model, about 20 percent. Same tokenizer, same lab, and the tax differs by 7x purely because the denominator moved.
What a large vocabulary buys
Compression. A bigger vocabulary holds longer pieces: whole common words, common word pairs in well-covered languages, frequent code idioms. Longer pieces mean fewer tokens per document, and tokens are the unit everything is billed in. Fewer tokens per document means more documents per context window, fewer forward passes per page at inference, and a training corpus that covers more raw text per trillion tokens. Part 2’s token budgets are all denominated in the local tokenizer, which is why cross-family token comparisons carry slop: a trillion K2-Horizon tokens and a trillion SmolLM2 tokens are different amounts of text.
Multilingual coverage. Vocabulary slots are a zero-sum allocation across scripts. A 32,000-piece vocabulary built mostly from English text has almost nothing left for Devanagari or Arabic after the Latin core is served, so those languages fragment into bytes and short pieces. The 128K to 256K generation of vocabularies is, in large part, the field buying its way out of that: Gemma’s 256,000 and K2-Horizon’s 250,624 exist so that many scripts get real pieces instead of confetti. EuroLLM’s paper is the roster’s cleanest statement of the trade: they trained BPE tokenizers at several sizes for 35 languages, measured fertility (pieces per word) per language, and concluded that 128,000 pieces was the point where multilingual compression stopped justifying the embedding growth.
A shorter path for frequent strings. Every domain has strings that recur verbatim: JSON keys, HTML tags, shell commands, log prefixes, license headers. If the tokenizer’s training data contained them often enough, they exist as single pieces or short sequences, and the model reads and writes them cheaply and reliably. This is the benign version of a general rule that cuts both ways in the fertility section below: the tokenizer is a compression dictionary for the corpus it was trained on, and it compresses everything else worse.
What it costs
Parameters, twice if untied. Covered above, with one addition: the output head is not optional spending. Every generated token is scored against the full vocabulary, so the head matrix is also compute, about hidden times vocab multiply-adds per token. For K2-Horizon 7B that is roughly 1.03B multiply-adds per token spent just producing logits, on the order of a dozen transformer layers’ worth of arithmetic, before any sampling happens. Large vocabularies tax every token twice: once in memory, once in the output projection.
Training memory for logits. Cross-entropy training materializes scores for every position in the batch against every vocabulary entry. Batch times sequence times 250,624 in floating point is a memory line item that shapes real training configs, and it is why large-vocab training stacks lean on fused or chunked cross-entropy implementations. The vocabulary decision reaches into the training infrastructure, not just the model card.
Dead and undertrained rows. A vocabulary entry only learns from the tokens that use it. In a 250,624-piece vocabulary trained on a corpus dominated by a few languages, the tail pieces (rare scripts, rare strings, pieces that made the cut on a small subsample) may appear so rarely that their embedding rows are barely trained: near their initialization, semantically empty, and liable to behave strangely if a prompt ever lands on them. The extreme version is the glitch token phenomenon, pieces that exist in the vocabulary but were effectively absent from training data and produce erratic behavior when invoked. Vocabulary size is therefore also a data requirement: every piece you mint is a promise to show the model enough examples of it. K2-Horizon can keep that promise to a 250K vocabulary because its corpus is 22T tokens. A lab training on 1T tokens with a 250K vocabulary is minting rows it cannot train.
Slower tokenization and bigger artifacts. Minor but real: larger merge tables cost encode time and ship in every tokenizer file, every GGUF, every deployment. Nobody in the study treats this as decisive. It is listed because the tax metaphor should be complete: the vocabulary charges rent at every stage, not just in the parameter count.
How these tokenizers were made
Every tokenizer in the study is byte-pair encoding (BPE) or a close relative: start from bytes or characters, repeatedly merge the most frequent adjacent pair, stop at the target vocabulary size. The differences are in the implementation frame, the training sample, and whether the lab trained one at all.
Trained on the model’s own diet. Nemotron-4’s report is explicit: a SentencePiece BPE tokenizer trained on data randomly sampled from the final training dataset, vocabulary 256,000. The principle is that the merge table should mirror the corpus the model will actually read, and sampling it from the final mix (not an early draft of it) keeps the dictionary honest. EuroLLM trained its SentencePiece BPE with byte fallback on its multilingual mix and sized the vocabulary by experiment, as above. IFM’s K2 line documents the most deliberate tokenizer design in the study in the K2-V2 tech report, the direct predecessor of the Horizon fleet: a byte-level BPE trained on a weighted multilingual mixture sampled specifically to optimize fertility for English and Arabic, the vocabulary initialized with Unicode ranges spanning Arabic, Latin, Cyrillic, Devanagari, and CJK scripts, and a custom pre-tokenization regex built to handle code syntax, whitespace consolidation, and punctuation isolation. One honest flag: that description is from the K2-V2 report. The Horizon configs confirm the fleet’s vocabulary sizes directly (250,624, and 64,256 for the 0.9B), but Horizon’s own report section on tokenizer construction was not read line by line in this pass, so the method is credited to the family’s documented practice, not re-verified for the fleet itself.
Built on a standard ID space. OLMo 2 and OLMo 3 share a 100,278-piece vocabulary whose IDs sit on the GPT-4 cl100k layout (the configs’ end-of-text ID is cl100k’s 100257, with Ai2’s padding token at 100277). Granite 4.0’s config shows the same 100,2xx ID neighborhood and a 100,352 vocabulary. Building on a famous ID space buys ecosystem compatibility: tooling, token counters, and intuitions transfer. The tokenizer stops being a research artifact and becomes infrastructure.
Inherited whole. Marin’s config is a stock Llama configuration, 128,256 vocabulary included: Stanford’s open lab spent its novelty budget on process transparency and took Meta’s tokenizer off the shelf. Every derivative line does the same by necessity. A post-training lab cannot change the tokenizer without retraining the embedding matrices, which is pretraining money, so Llama-Nemotron models speak Llama’s 128,256, the Qwen specialists speak Qwen’s 151,936, and Bolt E4B speaks Gemma’s 262,144. When you adopt a base model, its tokenizer is part of the adoption, as locked as its license and considerably harder to notice.
Deliberately small. SmolLM2 is the counter-design: a 49,152 vocabulary, roughly a third of the contemporary standard, on models from 135M to 1.7B. At those sizes the embedding arithmetic dominates the design, and Hugging Face chose compression losses over parameter losses. Then SmolLM3, at 3B, moved to the standard 128,256 space (its token IDs are the Llama-3 layout, start token 128000). Read together, the two releases are a size threshold made visible: below about 2B, a bespoke small vocabulary wins; at 3B, the standard vocabulary’s ecosystem and compression win. (Flag: SmolLM2’s config confirms the 49,152 size and tying; the tokenizer’s own training provenance was not separately verified in this pass.)
Special tokens: the training surface nobody budgets for
A slice of every vocabulary is not learned from data at all. Special tokens are reserved IDs for structure: start and end of text, padding, fill-in-the-middle markers, chat role markers, tool-call delimiters, reasoning delimiters. They are added by hand, and they have three properties that matter.
First, they are cheap exactly once. Reserved slots designed into the vocabulary before pretraining cost nothing extra. Slots added later are new embedding rows in an already trained model, starting from initialization or from an average of existing rows, and they only become real behavior if later training exercises them heavily. A special token is a promise that training data will use it; an unexercised one is a dead row with a official-sounding name.
Second, post-training runs on them. The chat template is a string of special tokens, and the model’s entire sense of turn-taking, tool invocation, and reasoning boundaries is keyed to those IDs. The K2-V2 report describes adding special tokens at the SFT stage for reasoning and tool use, on top of the fill-in-the-middle and repository-context tokens added at pretraining: the vocabulary grows a new control layer at each stage, and each layer has to be trained in that stage or it does not exist behaviorally. K2-Horizon ships a dedicated parser in vLLM and SGLang for its reasoning format, which is the serving-side shadow of the same fact: the format is part of the model artifact, not a prompting convention. SmolLM3’s dual-mode reasoning works the same way, with the chat template’s think and no-think markers trained in as mode switches.
Third, there is a real alternative: do not use a token. The Llama-Nemotron reasoning family toggles reasoning on and off with a system-prompt instruction in plain text, no dedicated control token. The cost is reliability (a text instruction is a request; a trained control token is closer to a switch) and the benefit is that the mechanism survives any tokenizer and needs no vocabulary surgery. The study’s families split on this, and Part 8 returns to it, because refusal behavior lives on this same surface: how a model is told to stop, to refuse, or to switch modes is partly a tokenizer-and-template decision made long before the safety training starts.
Fertility: who pays more per word
Fertility is the exchange rate between text and tokens, usually measured as tokens per word for spaced languages or tokens per byte where words are the wrong unit. Everything in this section is that one number moving around.
Across languages. A tokenizer compresses the languages its training sample emphasized and fragments the rest. The literature’s benchmark finding (Petrov and coauthors, 2023) is that for equivalent content, some languages cost several times, in the worst pairings an order of magnitude, more tokens than English under widely used tokenizers. The roster’s families respond at three levels of seriousness. EuroLLM measured fertility per language and sized its vocabulary to the measurement. IFM’s K2 line optimized its tokenizer sample for English and Arabic fertility explicitly and published the comparison figure. Apertus, covering 1,811 languages with about 40 percent non-English pretraining data, is the extreme case: at 131,072 pieces, its vocabulary cannot give every language long pieces, so its low-resource languages necessarily tokenize closer to bytes. That is not a failure of the design; it is the arithmetic of a fixed vocabulary divided by 1,811 languages, and Apertus’s bet is that coverage with fragmentation beats exclusion. A builder should nonetheless read Apertus’s multilingual scores knowing its per-language token costs are the highest in the study.
Across code. Source code tokenizes worse than prose in English-centric vocabularies: indentation is long whitespace runs, punctuation density is high, and identifiers in camelCase and snake_case split at their seams. The pre-tokenization regex is where this is fought. GPT-style regexes split numbers into short digit groups and keep contractions and word chunks intact; IFM’s custom regex was designed with code syntax as an explicit requirement. The other response is data-side rather than tokenizer-side: code-heavy families (SmolLM with Stack-Edu, Nemotron with its dedicated code corpora) accept the fertility and feed more code tokens. Nobody in the study retrained a tokenizer primarily for code fertility, which is itself a finding: code is expensive per token in every one of these models, and the field has decided to pay it.
The general rule. Fertility is where the tokenizer’s training corpus becomes a permanent pricing policy. Text that resembled the tokenizer’s training sample is cheap forever; everything else pays a surcharge on every read, every generation, every context window, for the life of the model. Which brings us to the text this study ultimately cares about.
Tied versus untied: the small-model lever
Tying means the input embedding and the output head are the same matrix: the vector that represents a token on the way in is the vector scored against on the way out. It halves the vocabulary tax at a stroke. The table shows who reaches for it. SmolLM2 ties at every size. SmolLM3 ties at 3B. Granite 4.0 Micro ties at 3B. Nobody above the small band ties: OLMo, K2-Horizon, Nemotron, Apertus, and Marin all run untied pairs, and the Gemma line is the exception that proves the size rule is a tendency rather than a law (Gemma ties even at large sizes, an upstream design choice Bolt E4B inherits).
The arithmetic explains the pattern. For SmolLM2’s 135M, tying is the difference between a 28M vocabulary line (about 21 percent of the model) and a 57M one (about 42 percent): untied, the smallest SmolLM2 would be nearly half dictionary. At 1.7B the same choice saves about 101M, 6 percent, still material. At flagship scale the same choice is noise: at K2-Horizon’s 375B the untied pair is under 1 percent of the model, and untied heads buy a real if modest quality edge, because the output projection is free to specialize for prediction rather than doubling as the input representation. So the field’s rule of thumb is legible in the table: tie when the vocabulary tax is a double-digit share of your model, untie when it is noise. The crossover in this study sits around 3B, exactly where SmolLM3 (tied, 9 percent) and the untied 3B-class designs bracket it.
One subtlety worth keeping: tying also couples training dynamics. A tied matrix gets gradients from both the input side (every occurrence of the token in context) and the output side (every position’s prediction), so rare tokens still receive output-side gradient signal at every step. For small models with thin data per token, that extra signal is part of why tying costs less quality than the parameter saving suggests.
The cyber read
Cyber text is where general vocabularies go to be expensive, and the reasons are structural, not incidental.
The payloads are high-entropy. Hex dumps, base64 blobs, hashes, encrypted and compressed content, memory addresses, and UUIDs share one property: near-zero redundancy. BPE is a frequency machine; it has long pieces for strings that recurred in its training sample and almost nothing for strings that are random by design. A SHA-256 hash is 64 hex characters that no tokenizer in this study holds as a unit, or as a handful of units. It fragments into short pieces, at boundaries determined by merge ranks rather than meaning, and the same is true of the base64 blocks in certificates, tokens, and encoded payloads, and of the hex in shellcode and packet dumps. Structural estimate, flagged as such rather than measured on these tokenizers in this pass: English prose runs about 4 characters per token under GPT-class vocabularies, ordinary code about 3, and hex or base64 closer to 1.5 to 2.5 characters per token, because merges rarely fire beyond pairs on near-random strings.
The costs compound. At half the characters per token, a fixed context window holds half as many log bytes. A 512K window that swallows a respectable incident’s worth of prose holds a much thinner slice of its hex dumps and encoded artifacts. Training and inference are billed per token, so the same corpus costs roughly twice the compute to read. And fragmentation is not just a price: a hash split into twelve pieces at merge-determined boundaries presents the model with a different segmentation problem than the same string seen as a unit, and exact-match behavior (have I seen this indicator before, is this the same hash as three pages ago) has to be reconstructed across pieces. Add the digit-grouping rules in GPT-style pre-tokenization, which split numbers into one-to-three-digit chunks by position rather than value, and timestamps, ports, and IP addresses all arrive pre-fragmented too. None of this makes cyber modeling impossible. All of it means a cyber model’s effective context and effective budget are materially smaller than its card numbers, by a factor set at tokenizer time.
What LAB2 should do with this. Four conclusions, in order of confidence.
First, the tokenizer is a base-selection criterion, not a post-training variable. LAB2’s ladder (Granite 4.1 8B as the recommended Step A base, OLMo 3 7B as control) inherits a 100K-class vocabulary either way, and no amount of SFT or RL changes the fertility of a hex string under it. The tokenizer read belongs in the same base-selection pass as Part 1’s license read: before benchmarks, read the license, then tokenize your actual corpus with the candidate’s tokenizer and look at the exchange rates.
Second, measure before surgery. The measurement is an afternoon: run the candidate tokenizers over a representative cyber corpus (logs, advisory text, exploit code, hex and base64 samples, CVE descriptions) and compute characters per token per text class. If prose-like classes dominate the token spend, the general vocabulary is fine and the fertility tax is a rounding error on the real budget. If high-entropy classes dominate, the tax is a first-order cost and the next two options come into play. Decide from the measurement, not from the intuition that cyber text is special.
Third, vocabulary extension is a continued-pretraining decision, not a fine-tuning trick. Adding domain pieces (whole recurring artifacts: common log-format prefixes, protocol keywords, standard headers) can reclaim real compression, but each new piece is an untrained embedding row, and rows become behavior only through training volume on data that uses them. The honest version of the surgery is: extend, then run enough domain continued pretraining to train the new rows, accepting that you have forked the lineage and invalidated direct token-denominated comparisons with the base. The dishonest version, adding tokens before a light SFT, leaves dead rows that the model routes around, and buys nothing but a bigger config file.
Fourth, do not replace the tokenizer. A from-scratch tokenizer is a from-scratch model decision. For LAB2’s ladder it is never on the table below a full pretraining run, and the study’s evidence says it does not need to be: every family here post-trains successfully on inherited vocabularies, including NVIDIA’s terminal and security specialists on Qwen’s. The achievable wins are selection (pick the base whose vocabulary taxes your corpus least), budgeting (plan context and compute in cyber-corpus tokens, not prose tokens), and, only if the measurement demands it, extension with real training behind it. The template surface deserves the same conservatism: refusal and control behavior are keyed to special tokens and chat templates, and disturbing that surface after safety training is a way to buy subtle breakage at the exact layer Part 8 has to measure.
Items left flagged, on purpose
Unresolved here, kept visible: the K2-Horizon tech report’s own tokenizer section was not read line by line; the fleet’s method is described from the K2-V2 report (same team, predecessor fleet) and its vocabulary sizes from the Horizon configs directly. SmolLM2’s tokenizer training provenance was not separately verified (size and tying are from its configs). Apertus’s tokenizer is documented in its tech report’s tokenizer section and its multilingual design goal is stated there, but the section’s construction details were not extracted in this pass; its vocabulary, hidden size, and tying are from the 8B config. EuroLLM’s 9B config read failed in this pass; its 128,000 vocabulary is from the EuroLLM paper, which predates the 9B, and the roster’s report coverage treats the tokenizer as shared across the family. Characters-per-token figures for hex and base64 are structural estimates from BPE behavior, not measurements run on the study’s tokenizers. Gemma’s 262,144 vocabulary and tying are upstream Gemma-line facts, not re-read from the Bolt E4B config. OLMo 2’s 32B and Granite 4.1’s configs were not read, so no embedding shares are computed for them.
Sources for this part
Config files read directly for this part (Hugging Face, Nov 17, 2026): IFM/K2-Horizon-0.9B, 3.7B, 7B, 32B, MoVA-36B-A4B, and 375B-A23B; HuggingFaceTB/SmolLM2-135M, SmolLM2-1.7B, and SmolLM3-3B; allenai/OLMo-2-1124-7B and Olmo-3-7B-Think; marin-community/marin-8b-base; swiss-ai/Apertus-8B-2509; nvidia/Nemotron-H-8B-Base-8K and NVIDIA-Nemotron-3-Nano-30B-A3B-BF16; ibm-granite/granite-4.0-micro-base. Tech reports and papers: Nemotron-4 340B Technical Report (arXiv:2406.11704, embedding parameter counts and SentencePiece tokenizer training) and the Nemotron-4 15B report (arXiv:2402.16819); K2-V2 Technical Report (IFM, arXiv:2512.06201, tokenizer construction and fertility method); EuroLLM (arXiv:2409.16235, tokenizer sizing experiment and fertility comparison). Roster files built for this masterclass on Nov 17, 2026: research/nemotron-roster.md, research/k2-horizon-roster.md, research/open-families-roster.md. Literature cited for the cross-language fertility finding: Petrov et al., “Language Model Tokenizers Introduce Unfairness Between Languages” (2023).