Part 4 · Part 4: Architecture, Where the Parameters Live
Model-Making Masterclass, Part 4. Written Nov 17, 2026. Engineer voice: plain language, terms defined when they first matter, every load-bearing number tied to a tech report, model card, or flagged as unverified.
Why architecture is a storage and routing problem
Strip the mystique away and an architecture is two decisions. First, where do the parameters live: in one dense stack that every token passes through in full, in a bank of experts that each token visits selectively, or split between attention layers and state-space layers with different memory behavior. Second, how does information move: which tokens can see which other tokens, at what cost, over what distance.
Everything else in this part is a variation on those two decisions, and every variation is a trade you can price. Dense costs compute per token and buys simplicity. Experts cost memory and routing complexity and buy capacity per unit of compute. State-space layers cost exact recall and buy constant-cost generation over long sequences. The families in this study have now run all three body plans at production scale, in public, at matched sizes. That almost never happens, and it is what makes this part possible.
One vocabulary note before the tables. A layer in most of these models is a block with a sequence mixer (attention or a state-space scan) followed by a feed-forward network (FFN). NVIDIA’s dense hybrids break that convention: in Nemotron-H, mixers and FFNs are separate layers in one long alternating sequence. When this part gives layer counts, it says which grammar the count uses.
The three body plans
Plan 1: the dense Transformer
Every token passes through every parameter. Attention for sequence mixing, a gated FFN (usually SwiGLU) for per-token computation, rotary position embeddings (RoPE) or a variant, pre-norm RMSNorm. This is the Llama-shaped default, and it is what OLMo 2 and 3, SmolLM2 and 3, Apertus, Marin, LLM360, EuroLLM, and the K2-Horizon dense models (0.9B through 32B) all ship. Nemotron-4 340B was dense too; NVIDIA’s own flagship abandoned the plan in the next generation, which tells you where the pressure is.
Why teams still choose it: the scaling behavior is the best understood in the field, the training dynamics are the most forgiving, every inference engine supports it optimally, and for a fully open release it is the easiest architecture for outsiders to reproduce and audit. K2-Horizon’s dense models make the point about auditability in an unusual way: IFM adopted the depth, width, and head configurations of Qwen3-4B, Qwen3-8B, and Qwen3-32B for its 3.7B, 7B, and 32B respectively, deliberately standing on established baselines instead of inventing new shapes. When your contribution is the training record, a boring body is a feature.
What it costs: per-token compute equals total parameters, and the key-value (KV) cache grows linearly with context. At 512K context and up, a dense model is a memory system that also computes.
Plan 2: the MoE Transformer
Replace most FFNs with a bank of smaller expert FFNs and a router that sends each token to a few of them. Attention stays standard (or latent, see below). DeepSeek-V3 set the template in December 2024: fine-grained experts, one shared expert, Multi-head Latent Attention, auxiliary-loss-free balancing. K2-Horizon’s two MoE models follow the DeepSeekMoE pattern with their own additions. Nemotron 3.5 Super VL’s 512 experts per MoE layer show where the count goes when a latent projection makes experts cheap (LatentMoE, below). OLMoE is Ai2’s small proof that the fully open pipeline works for this plan too.
Why teams choose it: capacity decouples from compute. At fixed active parameters (fixed FLOPs), a larger expert bank holds more knowledge for the same per-token cost, so you are buying capacity with memory instead of with FLOPs. The roster prices that exchange where controlled runs exist: IFM’s same-data comparison of its dense 32B against its MoVA-36B (Section 7) and NVIDIA’s LatentMoE ablation (below) are the two exhibits, and both are single-lab results. An honest gap: the cleanest published sparsity scaling study in the wider literature sits in a weights-only release outside this study’s scope, so the roster’s own evidence for the sparsity payoff is thinner than the field’s, and this part does not borrow the outside number.
What it costs: all the weights must be stored and, in distributed serving, moved. Routing is a new failure surface (collapse, imbalance, dropped tokens). Training needs expert parallelism and all-to-all communication, which is why MoE infrastructure is its own discipline (Part 5 touches the training side; Part 10 the serving side).
Plan 3: the hybrid Mamba-Transformer
Replace most attention layers with Mamba-2 state-space layers. A Mamba-2 layer processes a sequence by scanning it with a learned, input-dependent state transition; at generation time it carries a fixed-size state instead of a growing KV cache, so its per-token cost and memory are constant in sequence length. Attention layers are kept, sparsely, for the jobs state-space layers do badly, chiefly exact content-addressable recall.
NVIDIA is the plan’s main industrial backer: Nemotron-H (dense hybrid, 2025), Nemotron-Nano v2 (dense hybrid), then Nemotron 3 (hybrid with MoE FFNs). IBM’s Granite 4.0 line runs the same plan at enterprise sizes, dense and MoE variants. Note what “hybrid” names: a ratio, not a single mechanism. Which layers stay attention, and how many, is the design variable.
Why teams choose it: inference economics for long generation. Nemotron-H reports up to 3x inference speed against similar-size dense Transformers at matched or better accuracy; Nemotron 3 Nano claims 3.3x the throughput of Qwen3-30B-A3B at 8K input and 16K output tokens, with the gap growing at longer sequences (both vendor-reported). Reasoning workloads are the target: a model that generates 16K thinking tokens per answer pays the KV cache bill 16K times.
What it costs: exact recall is structurally weaker (the state is a lossy compression of history), the kernels and framework support are younger, and the design space (which layers stay attention, in what pattern) is still being mapped in public. Section 4 prices the ratios the field has converged on.
| Dense | MoE Transformer | Hybrid Mamba-Transformer | |
|---|---|---|---|
| Compute per token | All parameters | Active experts only | All (dense hybrid) or active experts (MoE hybrid) |
| Generation memory | KV cache grows with context | KV cache grows with context | Constant state for Mamba layers; small KV cache for the few attention layers |
| Capacity lever | Width and depth | Expert count at fixed active size | Layer ratio and state size |
| Main risk | Serving cost at long context | Routing and infrastructure complexity | Exact recall; younger tooling |
| Roster exemplars | OLMo 3, SmolLM3, Apertus, K2-Horizon dense | DeepSeek-V3 pattern, K2-Horizon MoE, Nemotron 3.5 Super VL | Nemotron-H, Nemotron 3, Granite 4.0 |
Attention: the main design axis
Architecture debates sound like they are about the whole network. In practice the sharpest differences in this roster sit inside the attention block: how keys and values are produced, stored, and mixed. Five variants matter.
MHA to GQA: the settled compression
Multi-head attention (MHA) gives every query head its own key and value head. Grouped-query attention (GQA) shares one KV head across a group of query heads. The mechanism is trivial; the effect is not: the KV cache, the dominant memory cost of long-context serving, shrinks by the grouping factor. Nearly everyone in this roster runs GQA with 8 KV heads (K2-Horizon across all sizes, Nemotron-H, Nemotron-Nano v2, Granite 4.1). OLMo 3’s 7B is the outlier that kept full MHA (32 query and 32 KV heads); its 32B uses GQA. Read that as a size-dependent judgment: at 7B the cache is affordable, at 32B it is not.
What it saves: KV cache memory and memory bandwidth at decode, roughly proportional to the grouping. What it risks: a small quality cost versus MHA at fixed size, which most labs now judge to be below the noise floor of their other decisions. Multi-query attention (one KV head total) survives mostly in NVIDIA’s Megatron Bridge documentation language for Nemotron-H; the shipped configs use 8 KV heads, which is GQA.
MLA: compress the cache itself
Multi-head Latent Attention, from DeepSeek-V2 and carried into V3, attacks the cache differently. Instead of storing keys and values per head, the model compresses each token’s keys and values jointly into one low-rank latent vector, caches only that vector, and re-expands it into per-head keys and values on the fly during attention. A small separate key component carries the rotary position information, shared across heads (the “decoupled RoPE” trick), because a compressed latent cannot be rotated per head after the fact.
What it saves: the largest KV cache reduction of any variant here, an order of magnitude class reduction versus MHA in DeepSeek’s accounting, which is what makes very long context on a very large model servable at all. What it risks: the up-projections add compute and the mechanism constrains other choices. One constraint is structural: QK-norm, the query-key normalization several families use for stability, does not apply cleanly to MLA because the key matrices are not fully materialized during inference. The architectural lesson stands on its own: pick MLA and some of your other tools stop fitting. Note also that no family in this study’s roster ships MLA in a fully open pipeline; the design is documented here from the DeepSeek reports as the reference design the roster’s variants answer to, and the roster’s own cache answers are GQA, MoVA, and the hybrid state approach below.
MoVA: experts inside the value path
Mixture-of-Value Attention is K2-Horizon’s signature contribution, shipped only in the 36B-A4B, and it is now verified from the tech report’s architecture section rather than from GGUF metadata (the roster flagged the layer counts as third-party; the report’s Table 1 confirms them: 48 layers, hidden size 2,560, 32 query heads, 8 KV heads, head dimension 128).
The mechanism: in standard attention, a token’s value vector is one linear projection of its representation. MoVA replaces that single projection with a bank of 64 value projections (“value experts”). A router picks 4 of the 64 per token, each selected projection’s output passes through a SiLU nonlinearity, and the results are combined with the router’s weights into the single value vector attention then uses. The SiLU is load-bearing: without it, a weighted sum of linear projections collapses into one input-dependent linear map and the bank buys almost nothing. Two properties make the design cheap to adopt. It only changes how each token’s own value vector is computed, so it composes with FlashAttention, GQA, and sparse attention unchanged. And because only the combined value vector is cached, the KV cache is exactly the size it would be under plain GQA. The sparsity is free at cache time; you pay for it in projection compute at prefill and in the extra parameters on the shelf (64 value projections per layer, of which 4 fire).
Paired with MoVA, the 36B uses softplus attention gating: a learned gate vector multiplies the attention output element-wise. The report’s argument against the usual sigmoid gate is crisp. A sigmoid gate is bounded in (0, 1), starts near 0.5 at initialization (halving the attention output at birth), and can only ever attenuate. Their softplus gate is unbounded above, so it can amplify as well as attenuate, and with the scaling coefficient set to log 2 it equals 1 at zero input, so gated attention starts life behaving exactly like standard attention. Whether MoVA and the gate generalize beyond this one model is an open question (one fleet, one size, no independent replication yet), but as a design pattern it is the first serious attempt to put conditional capacity inside attention rather than only in the FFN.
What it saves: nothing directly; it spends a little compute to buy capacity in a place FFN experts cannot reach, the content that attention moves around. What it risks: a second routing system to balance and debug (the 36B carries two independent routers per layer, one for values, one for FFN experts), and single-source evidence.
Sliding-window hybrids: attention with a horizon
OLMo 3’s pattern: three of every four layers attend only within a 4,096-token sliding window; every fourth layer attends globally, and YaRN position scaling is applied only on those global layers. SmolLM3 and Granite use relatives of the same idea (Granite’s horizon is set by its Mamba layers instead; see below). The windowed layers are cheap and their cache can be capped; the periodic global layers restore cross-document reach.
What it saves: attention compute and cache on 75 percent of layers, with long-context quality largely preserved in Ai2’s measurements. What it risks: information that must travel farther than the window depends on the global layers relaying it, and tasks that need precise retrieval from the middle distances are exactly the stress case. The pattern’s honesty is appealing: instead of claiming one mechanism does everything, it budgets two mechanisms explicitly.
NoPE: position information is optional
NoPE (no positional embeddings) drops RoPE entirely, on the theory that a causal attention layer can infer position from the mask structure itself, and that removing explicit position codes removes the main obstacle to extrapolating beyond trained context. Three distinct doses appear in this roster. SmolLM3 runs a 3:1 pattern: every fourth layer is NoPE, the rest keep RoPE, and HF credits the pattern (with RoPE base increases during extension) for its 64K trained, 128K extrapolated context. Granite 4.0 removes positional encodings entirely, relying on its Mamba-2 layers to carry order. Nemotron-H and Nemotron-Nano v2, per their tech reports, use no position embeddings at all: the Mamba layers are inherently order-sensitive, and NVIDIA lets them carry position for the whole network, attention layers included. One documentation wrinkle is flagged at the end of this part: NVIDIA’s Megatron Bridge docs describe RoPE for the Nemotron-H family, contradicting the papers. The papers are the primary source for the shipped design; treat the docs line as unresolved.
What it saves: context extension pain. Models that never hard-code position tend to degrade more gracefully past their trained length. What it risks: precise position-dependent behavior (counting, exact offset arithmetic, some retrieval patterns) has less scaffolding, and the evidence base is younger than RoPE’s.
The MoE design space
An MoE layer has more knobs than any other component in these models, and the roster’s settings differ enough to teach from. The knobs: how many experts, how big each expert is, how many fire per token, whether some experts always fire, how tokens choose, and how the load is kept even.
Granularity: many small experts beat few large ones
DeepSeekMoE’s core finding, now the industry default: split the FFN into many fine-grained experts and activate proportionally more of them. Fine granularity lets the router compose specialists (one for a language, one for a syntactic pattern, one for a domain) instead of forcing each expert to be a generalist. The roster’s numbers:
| Model | Routed experts | Active per token | Shared experts | Expert FFN width | Notes |
|---|---|---|---|---|---|
| DeepSeek-V3 | 256 | 8 | 1 | 2,048 | Template; first 3 layers dense; node-limited routing |
| K2-Horizon 375B | 192 | 8 | 1 | 1,792 | 61 layers; first 3 layers dense |
| K2-Horizon 36B-A4B | 100 | 8 | 1 | 768 | Plus MoVA value routing (64 total, 4 active) |
| Nemotron 3 Nano 30B-A3B | 128 | 6 | 1 | Not pinned here | Hybrid: experts live in MoE layers between Mamba layers |
| Nemotron 3.5 Super VL | 512 | Not pinned here | Not pinned here | Not pinned here | Count inflated by LatentMoE design (below) |
| Granite 4.0 H-Small | Fine-grained, count not in IBM’s announcement | Not pinned here | Yes, always active | Not pinned here | 32B total, 9B active |
Expert counts are not comparable across rows until you know the expert width and the hidden size they feed. A 512-expert layer of narrow experts can hold fewer total parameters than a 128-expert layer of wide ones. The honest summary statistic is the pair (total parameters, active parameters), which Piece 0’s spine already enforces.
Shared experts: the always-on lane
Every MoE in the table keeps at least one shared expert that processes every token. The rationale from the DeepSeekMoE work: without a shared lane, common knowledge (grammar, frequent patterns) gets duplicated across routed experts, wasting the capacity the routing was supposed to specialize. The shared expert is the model’s commons; the routed experts are its specialists. Granite’s announcement calls out the same design (“shared experts that are always activated”) in its first MoE generation, a sign the pattern has fully converged.
Routing and load balancing: three generations of answer
A router that is free to choose will collapse: a few experts get most tokens, train faster, get chosen more, and the rest starve. The field has answered three times.
Generation one, the auxiliary load-balancing loss: add a penalty term that rewards even expert usage. It works, and it taxes the main objective, because the model spends gradient on traffic management instead of on prediction. Switch Transformer and most early MoEs, and in a small-weight form OLMoE and many current models, live here.
Generation two, auxiliary-loss-free balancing (DeepSeek-V3’s contribution): give each expert a bias term that is added to its routing score for selection purposes only, not for the weighting of its output. After each step, nudge the bias up for under-used experts and down for over-used ones, based on measured load. Balance is achieved by a control loop on the router’s preferences, with no gradient interference in the main loss. V3 keeps a very small sequence-level balance loss as a backstop against extreme imbalance within a single sequence, but the bias loop does the work. This is currently the reference design at the frontier end of the roster.
Generation three is really a placement decision rather than a new mechanism: LatentMoE (below) shrinks the routed dimension so that more experts and more active experts fit the same budget, which softens the cost of any imbalance that remains. For K2-Horizon’s two MoE models, the balancing mechanism was not pinned down in this pass and is flagged at the end rather than guessed.
Two routing details deserve a line each. Token-choice top-k with sigmoid scoring (each expert scored independently, top k taken) is the DeepSeek-line default, replacing the older softmax-over-experts that forced scores to compete. And V3’s node-limited routing, which caps how many devices a token’s experts may span, is a reminder that at scale the router is also a network-traffic scheduler, and routing choices are systems choices whether the model card says so or not.
LatentMoE: shrink the road, multiply the experts
Nemotron 3 Super and Ultra (and the 3.5 Super VL) replace the standard MoE block with LatentMoE. The mechanism, from the Nemotron 3 white paper: before routing, project each token from the model hidden dimension d down to a smaller latent dimension (about d/4 in the published ablation). Experts live entirely in that latent space; their outputs are projected back up. Routing decisions, the gating network, and the shared expert stay in the full dimension.
The savings are structural. In distributed serving, an MoE layer’s costs are dominated by reading expert weights (latency-bound serving) and by all-to-all traffic proportional to active experts times hidden dimension (throughput-bound serving). Both shrink by the d-to-latent ratio, about 4x. NVIDIA reinvests the saving in the two places the ablation says pay: total experts and active experts both scale up by the same factor. The published controlled comparison, two roughly 73B-total, 8B-active models trained on 1T tokens with identical hyperparameters, is the strongest single architecture ablation in this part: standard MoE (128 experts, top-6, hidden 4,096) against LatentMoE (512 experts, top-22, latent 1,024). LatentMoE won every aggregate: MMLU-Pro 52.87 against 48.30, MMLU 72.11 against 70.10, code 55.14 against 51.95, math 80.19 against 78.32, commonsense 82.10 against 81.73. Same shelf, same compute class, better model, purely from where the expert parameters sit relative to the projection.
The risk is the projection itself: everything a routed expert can express must pass through a bottleneck one quarter the model’s width. The ablation says the extra expert diversity more than pays for the bottleneck at this scale. Whether that holds at other scales is exactly the kind of question an open recipe lets you test, and NVIDIA published the recipe.
Depth, width, and layer allocation
Two allocation questions decide a model’s shape: how deep versus how wide, and in a hybrid, which layers get which job.
On depth and width, the roster is less experimental than you might expect. Frontier shapes cluster: K2-Horizon’s flagship runs 61 layers at hidden 6,144; its 32B dense runs 64 layers at 5,120. K2-Horizon’s smaller dense models borrow Qwen3’s shapes outright. The interesting deviation is the MoVA-36B: only 48 layers and hidden 2,560, much narrower than the 32B dense it is compared against, because its capacity lives in the expert banks (100 FFN experts plus 64 value experts per layer) rather than in width. Depth and width buy per-token computation; experts buy shelf capacity. The 36B is what a model looks like when the budget moves from the first ledger to the second.
On hybrid allocation, two grammars exist and the difference matters when you read configs.
NVIDIA’s dense hybrids (Nemotron-H, Nano v2) treat mixer and FFN as separate layers in one sequence. Nemotron-H-8B: 52 layers total, of which 24 are Mamba-2, 24 are FFN, and 4 are attention (about 8 percent), with attention layers evenly dispersed, the first layer always Mamba-2, the last always FFN, and every attention layer immediately followed by an FFN layer. The 56B keeps the same proportions at 118 layers (10 attention). Nano v2’s 12B base: 62 layers, 28 Mamba-2, 28 FFN, 6 attention. The ratio is the design: roughly one attention layer in twelve, placed like load-bearing columns through the stack.
Nemotron 3’s MoE hybrids fold the FFN into MoE layers and interleave those with Mamba-2 layers: Nano’s 52 layers hold 6 attention layers, with the remaining 46 alternating Mamba-2 mixers and MoE layers. Granite 4.0 uses the block grammar instead: each block pairs a mixer with an FFN or MoE, in a 9:1 Mamba-to-attention block ratio, so about one block in ten attends.
Either way, the field has converged on a narrow band: attention gets 8 to 12 percent of the sequence-mixing slots in a hybrid, no matter whose logo is on the report. That convergence, reached independently by NVIDIA and IBM within months, is one of the more trustworthy numbers in this part. Below that band, exact recall suffers in the published ablations; above it, the inference economics that justified the hybrid evaporate.
Multi-token prediction: architecture that pays twice
Multi-token prediction (MTP) adds auxiliary heads (in DeepSeek’s design, full lightweight modules) that predict not just the next token but the tokens after it, trained jointly with the main objective. It is an architectural choice with two separate payoffs, and families adopt it for different mixes of the two.
Payoff one is training signal: predicting several positions ahead forces richer internal planning, and the published ablations show quality gains, not just speed. DeepSeek-V3 shipped one MTP module (predicting one extra token) and reported gains across its eval suite. Nemotron 3’s white paper ablates it on an 8B-active MoE base at 1T tokens: adding MTP improved every reported category, about 2.4 percent on average (MMLU-Pro 45.05 to 47.84, GSM8K 82.49 to 84.46, MMLU 70.06 to 71.26).
Payoff two is serving speed: at inference, the MTP heads act as a built-in draft model for speculative decoding, proposing tokens the main model verifies in parallel. Nemotron 3’s lightweight MTP module hit about 97 percent acceptance on its first two predicted tokens in the ablation study, which is draft quality most standalone draft models do not reach. Super and Ultra ship MTP layers; Super VL trained its MTP head on 100B tokens as a distinct stage; Nemotron 3.5 Lightning carries one MTP layer and even ships dedicated speculative-decoding variants (DSpark and DFlash builds).
The counterpoints are as instructive. K2-Horizon trains no MTP heads; its answer is Uno, LoRA-style diffusion adapters fitted after training that emit blocks of tokens per pass for up to 2.2x decoding speedup, keeping the base architecture untouched. And the dense fully open families (OLMo, SmolLM, Apertus, Marin) ship none either. The pattern: MTP is a frontier-scale, serving-driven choice. If your model’s serving economics dominate its lifecycle cost, train the heads. If your model is a research artifact or an on-premises workhorse at moderate query volume, a bolt-on drafter or plain decoding is one less coupled system to train.
The 30B band: three body plans at one shelf size
This is the comparison the whole study is built around, and it only works because IFM trained two of the three entries on identical data with identical settings.
| K2-Horizon 32B | K2-Horizon MoVA-36B-A4B | Nemotron 3 Nano 30B-A3B | |
|---|---|---|---|
| Body plan | Dense Transformer | MoE Transformer plus MoVA | Hybrid Mamba-MoE |
| Total params | 34.8B | 37.4B | 31.6B |
| Active per token | 34.8B (all) | 4.67B backbone | 3.5B (card headline) |
| Layers | 64 | 48 | 52 (6 attention, rest Mamba-2 and MoE) |
| Hidden size | 5,120 | 2,560 | Not restated here |
| Attention | GQA, 64 Q / 8 KV heads | GQA 32 Q / 8 KV, MoVA values, softplus gate | GQA in the 6 attention layers |
| Experts | None | 100 routed, top-8, 1 shared; 64 value experts, top-4 | 128 routed, top-6, 1 shared |
| Context | 512K (extended in mid-training) | 512K | Up to 1M (262K default config) |
| Training tokens | 22T (shared corpus) | 22T (same corpus, same batches) | 25T over a released 10.6T corpus (multi-epoch) |
What each plan bought, in the publishers’ own numbers:
The dense 32B is the control. On the shared corpus it sets the loss curve the other two are judged against, and nothing at this shelf size in the family beats it outright. That is the honest baseline fact about sparsity at 30B: at equal data, dense is still the quality reference; the sparse models win on cost, not on the scoreboard.
The MoVA-36B nearly erases the gap at one-seventh the active parameters. IFM’s report: its pretraining loss closely tracks the 32B’s, and at 1.1M steps it trails by only 1.2 to 3.7 points across MMLU, GSM8K, HellaSwag, and HumanEval, with the final evaluation section reporting it matches or stays within a few points of the dense model on every benchmark they ran. You give up a few points and in exchange your per-token compute drops by about 7x, which at serving time is the difference between a model a workstation can run interactively and one it cannot. Note also what the 36B’s shape says: it reached that result with a narrower, shallower backbone (2,560 hidden, 48 layers) plus two routing systems. Capacity moved out of the backbone and into the banks.
Nemotron 3 Nano plays a different game on the same shelf. Its published claims are throughput and context, not a same-data loss race: 3.3x Qwen3-30B-A3B’s throughput at 8K-in, 16K-out, RULER 92.9 at 256K and 86.3 at 1M context (vendor-reported, from the model card via the roster). The hybrid body is why: 46 of its 52 layers never build a KV cache at all, so the long-context serving bill is set by 6 attention layers instead of 52. The trade it accepts is the one flagged throughout this part: exact recall rides on those 6 layers, and its quality comparison against a dense 30B trained on the same data does not exist in public, because nobody has run it.
Three shelves, three different definitions of “better”: best loss per token of data (dense), best quality per unit of compute (MoVA MoE), best throughput and context per dollar of serving hardware (hybrid). Part 12’s synthesis turns this into build decisions; the architectural point for now is that at 30B the body plan is a business decision wearing a lab coat.
Activations and normalization: the small print
These choices rarely headline a release, and they decide whether a training run survives its own scale. Three divergences matter in this roster.
The FFN activation. SwiGLU, a gated activation using two input projections and a SiLU gate, is the near-universal default (K2-Horizon family-wide, DeepSeek-V3, OLMo, SmolLM, most others). Two families deliberately deviate. Nemotron-H and Nano v2 use squared ReLU in their FFN layers: cheaper (one projection path, no gate), and in NVIDIA’s ablations competitive at their shapes; it pairs naturally with their separate-FFN-layer grammar. Apertus goes furthest: xIELU, a trainable piecewise activation derived by designing its gradient first (squared-ReLU-like growth for positive inputs, a trainable negative-gradient branch borrowed from the ELU family), used as a plain ungated activation. The xIELU paper reports lower perplexity than both squared ReLU and SwiGLU at matched compute in 1.1B and 3B tests on FineWeb-Edu. Apertus is the only production-scale adopter in this roster, so the claim currently rests on small-scale evidence plus one 15T-token run whose ablations are Apertus’s own. Flag it as promising, single-source at scale.
Normalization placement. The default is pre-norm RMSNorm: normalize before each sublayer, keep the residual path clean. OLMo 2 famously reordered this, using a post-norm-style placement (normalize after the sublayer, inside the residual stream) plus QK-norm and a z-loss, and ablated the package in public as its stability recipe; OLMo 3 keeps QK-norm. The lesson is not that one placement wins universally; it is that normalization placement is a stability decision that interacts with learning rate and scale, and OLMo’s public ablations are the best evidence available either way.
QK-norm and its discontents. Normalizing queries and keys before the attention dot product caps logit growth and is cheap insurance against attention entropy collapse. OLMo 2 and 3 carry it; NVIDIA’s Megatron Bridge configs for Nemotron-H list QK LayerNorm as well. MLA designs cannot use it, since their keys are never fully materialized at inference; that is one of the standing costs of the latent-attention choice above. K2-Horizon’s answer is different again and worth naming precisely: Grouped RMSNorm, applied family-wide. Plain RMSNorm lets a single normalized element grow as large as the square root of the hidden dimension (about 90 at width 8,192), which the report blames for outlier-driven instability in wide models. Grouped RMSNorm normalizes within fixed-size groups of dimensions instead, capping each element at the square root of the group size. The group counts scale with the model (from the report’s Table 1: 1 group at 0.9B, 2 at 3.7B and 36B, 4 at 7B and 32B). It is the same instinct as QK-norm, applied to every normalized tensor rather than only to attention, and IFM credits it with substantial stability gains across the fleet’s scales.
The cyber read
Architecture choices land differently when the workload is security work. Four implications LAB2 should price in.
Context is the working set. Cyber tasks are document tasks: a full repository, a day of logs, a packet capture decoded to text, a threat-intel corpus, an incident timeline. The useful context window is not a benchmark number; it is whether the whole artifact fits and stays affordable through a long agent session. This is where the body plans diverge hardest. A dense or MoE Transformer pays KV cache on every layer for every token of that artifact. A hybrid pays full attention cache on 6 of 52 layers (Nemotron 3 Nano) or about one block in ten (Granite), and constant state elsewhere. MLA compresses the cache that remains by an order-of-magnitude class factor. For on-premises cyber serving, where the GPU is a fixed budget and the sessions are long, the hybrid and MLA designs are not optimizations; they are what makes the deployment shape possible at all. IBM’s enterprise claim for Granite 4.0, over 70 percent lower serving memory on long-context multi-session workloads against Transformer-only models, is the commercial version of the same arithmetic.
Exact recall is the hybrid’s weak joint, and cyber is a recall workload. Finding one indicator of compromise string in 400K tokens of logs, or the one line in a codebase where a check was dropped, is a needle task. State-space layers compress history lossily; sliding windows see only their horizon; the retained attention layers are the mitigation, and published long-context suites (which skew toward semantic retrieval) can overstate needle performance. Any LAB2 candidate architecture needs an exact-match retrieval eval at realistic log lengths before the body plan is trusted, and the eval should use hex, hashes, and base64-shaped needles, because tokenization of those strings is where retrieval quietly dies (Part 3 owns the tokenizer side).
The 30B-A3B class is the on-prem sweet spot, and its body plan is a real choice. LAB2’s Step B lives at this shelf size. The three-way comparison above is the decision: a dense 30B serves like a 30B; a 30B-A3B MoE serves like a 3B with a 30B’s knowledge; a hybrid 30B-A3B serves like a 3B with a 30B’s knowledge and a small model’s long-context bill. For a sovereign deployment on one workstation-class GPU, the sparse plans are the difference between shipping and not shipping. The offsetting risk is routing behavior on out-of-distribution token streams: shellcode bytes, encoded payloads, and minified scripts are exactly the inputs a router trained mostly on natural text and clean code has seen least, and expert collapse on weird inputs will not show up in MMLU. It will show up in a cyber eval. Build one (Part 9 and Part 11).
Refusal and safety behavior has an architecture address. K2-Horizon’s report closes the loop this part opened: its safety evaluation measures refusal and jailbreak robustness per model, and finds the MoE pair tracking the flagship closely (the 36B within 0.006 of the 375B on its overall safety figure). Safety behavior is trained in (Part 8), but it is worth knowing that at these scales it does not appear to be a function of body plan. What architecture does decide is your ability to inspect and modify behavior: open weights on a standard body plan can be fine-tuned, ablated, and re-evaluated with commodity tooling today; exotic bodies (LatentMoE projections, MoVA routers, diffusion heads) narrow the tooling that understands them. For a program whose core move is retraining behavior, boring and inspectable is a genuine architectural virtue, and it belongs in the Step A decision (Granite-class dense) even if Step B goes sparse.
Flagged and unverified items
Per this study’s discipline, what this part could not pin at a primary source stays flagged:
- K2-Horizon MoVA internals: resolved. The roster carried the 36B’s layer counts (48 layers, hidden 2,560, 4-of-64 value experts) from GGUF metadata quoted by a third-party blog. The tech report’s Table 1 and Section 2.2 confirm them and add the FFN expert configuration (100 routed, top-8, 1 shared, expert width 768). The flagship’s internals, absent from the roster, are also now pinned from the same table (61 layers, hidden 6,144, 192 routed experts, top-8, expert width 1,792, 23.59B active backbone parameters).
- K2-Horizon load balancing: not pinned. The report’s architecture sections describe the routers but the balancing mechanism for its FFN and value routers was not identified in this pass. Do not assume it matches DeepSeek’s bias method.
- Nemotron-H position encoding: documentation conflict. The Nemotron-H and Nano v2 tech reports state the models use no position embeddings. NVIDIA’s Megatron Bridge documentation for the same family describes RoPE. The papers govern the shipped design as far as this study is concerned, but the conflict is on the record.
- Nemotron 3 Nano routing detail. The card text is internally inconsistent (top-6 versus 5 active experts in different lines; 1 versus 2 shared experts). This part uses top-6 and 1 shared, following the HF-side descriptions in the roster; the config file would settle it and has not been read.
- LatentMoE shipping configurations. The 512-expert, top-22 figures come from the white paper’s controlled ablation at 73B scale, not from the shipped Super and Ultra configs. Super VL’s card reports 512 routed experts per MoE layer; per-model expert widths and active counts for Super and Ultra were not pinned in this pass.
- Granite 4.0 MoE specifics. IBM’s announcement confirms fine-grained MoE with always-active shared experts and the 9:1 hybrid ratio, but does not publish expert counts or active-expert numbers for H-Tiny and H-Small; those cells are marked “not pinned here” rather than filled from secondary coverage.
- Vendor-reported performance. Throughput multipliers (Nemotron-H up to 3x, Nemotron 3 Nano 3.3x), Granite’s memory reduction, and all benchmark figures in this part are publisher-reported unless a sentence says otherwise. The controlled comparisons (IFM’s same-data 32B versus 36B, NVIDIA’s LatentMoE ablation) are single-lab results awaiting independent replication.
Sources for this part
Primary: K2-Horizon tech report (arXiv:2611.21003), architecture Table 1 and Sections 2.1 to 2.3; Nemotron 3 white paper (arXiv:2512.20856), Sections 2.1 to 2.3 (hybrid MoE, LatentMoE ablation, MTP ablation); Nemotron-H paper (arXiv:2504.03624), architecture section (layer patterns, squared ReLU, no position embeddings); DeepSeek-V3 tech report (arXiv:2412.19437) for the template (MLA, DeepSeekMoE, auxiliary-loss-free balancing, MTP); IBM Granite 4.0 announcement (ibm.com) for the 9:1 ratio, NoPE, and shared experts; xIELU paper (Huang and Schlag, arXiv:2411.13010) for the activation’s design and small-scale results. Roster files: research/nemotron-roster.md, research/k2-horizon-roster.md, research/open-families-roster.md (OLMo 3 sliding-window pattern and YaRN placement, SmolLM3 NoPE pattern, Granite roster entries, Nemotron 3 Nano card figures).