Model Making, in the Open: A Masterclass

Part 9 · 9. Evaluation

Part 9 · Part 9: Evaluation, Proving the Claims

Listen: audio twin

Model-Making Masterclass, Part 9. Written Nov 17, 2026. Engineer voice: plain language, terms defined when they first matter, every load-bearing number tied to a published artifact or flagged as unverified.

A benchmark score is an instrument reading, not a property

Every earlier part of this study ended with numbers: this data mix beat that one by 5.6 MMLU points, this architecture won by 4.6, this RL stage added 16.4 on AIME. This part is about what those numbers are made of, because a benchmark score is not a property of a model the way parameter count is. It is the output of a measurement instrument, and the instrument has four adjustable parts: the harness (the code that presents tasks and collects answers), the prompt format (template, few-shot examples, formulation), the sampling settings (temperature, top-p, output length cap, reasoning effort), and the scoring rule (exact match, normalized likelihood, a judge model, a test suite, an average over how many samples). Change any one and the same frozen weights post a different number. The open families in this study differ as much in how honestly they document the instrument as in the scores they report, and that documentation is what this part grades.

The practical consequence for a builder: never compare two numbers until you have compared the instruments that produced them. Most cross-lab benchmark arguments are instrument arguments wearing a capability costume.

The anatomy of a claim, with the spread measured

The cleanest measurement of the instrument effect comes from Ai2’s OLMES paper (the Open Language Model Evaluation Standard, 2024). Its opening table collects published scores for the same models on the same benchmark, ARC-Challenge, as reported by six different sources: the Hugging Face Open LLM Leaderboard, the Llama 2 paper, the Mistral 7B paper, the Falcon paper, the original OLMo paper, and the Llama 3 model card. Llama 2 7B’s ARC-Challenge score across those sources runs from 43.2 to 53.7. Same weights, same test set, a ten-point spread, produced entirely by instrument choices: whether the question is presented as a multiple-choice prompt or in cloze form (the model scores each completion separately), how answer likelihoods are normalized (by character length, by pointwise mutual information, or not at all), and how many few-shot examples precede the question (0 in some sources, 25 in others). None of those sources was lying. Each measured a real quantity. They were not measuring the same one.

OLMES exists to pin the instrument down: a fixed set of ten multiple-choice tasks, curated 5-shot prompts with balanced answer labels, a specified normalization per task, fixed instance sampling with a fixed seed, and all of it published as code (the allenai/olmes repository) so any lab’s number can be rerun bit for bit. Ai2 has since extended the line with olmo-eval, a workbench that adds two things a serious instrument needs: standard errors on every score, and a minimum detectable effect, the smallest gap between two models that the suite can distinguish from noise at its sample sizes. That second number is the one most benchmark tables in the industry are missing. A one-point lead on a 30-problem benchmark is not a lead; it is weather.

The four instrument parts, and how each one moves numbers in this study’s families:

Harness. The code decides everything from how the chat template is applied to whether a truncated answer counts as wrong. EleutherAI’s lm-evaluation-harness is the field’s shared default (it powered the Hugging Face Open LLM Leaderboard and is used internally by dozens of labs, NVIDIA included), but labs also run their own: NVIDIA’s NeMo-Skills, IFM’s Eval360, Ai2’s OLMES and olmo-eval, Hugging Face’s lighteval for the SmolLM ablations, Apertus’s released suites built on an lm-eval fork. A score quoted without its harness is a score quoted without its units.

Prompt format. Formulation alone (cloze vs multiple-choice) is worth points on knowledge benchmarks, which is why OLMES standardizes it. For chat models the equivalent trap is template mismatch: evaluating an instruct model with its base model’s prompt format, or a reasoning model with its reasoning mode off. NVIDIA’s cards have taken to stating the mode in the table itself (the Llama-3.3-Nemotron-Super-49B v1.5 card reports its reasoning benchmarks with reasoning on), because the same checkpoint posts two different score sets depending on a system-prompt sentence, the toggle Part 7 covered.

Sampling settings. Reasoning benchmarks are sampled, not greedy, in modern practice, and the settings are part of the result. The AceReason-Nemotron papers state theirs in the table notes: temperature 0.6, top-p 0.95, maximum response length 32,768 tokens, following the DeepSeek-R1 evaluation framework. Each knob bites. Temperature changes the distribution being measured. The length cap silently converts slow-but-right answers into wrong ones when a trace is cut mid-derivation. And serving-side effort settings do the same at API scale: Artificial Analysis’s own intake-filed numbers for Opus 5.5 run from 35 at low reasoning effort to 58 at maximum on the same index version, so a claim that names a model but not its effort setting has not named the thing measured.

Scoring rule. Exact match against a key, a normalized likelihood comparison, a judge model’s verdict, unit tests executing generated code, an environment checking final state: these disagree with each other systematically, and the next section takes up the sampling-based rules (pass@k, pass^k, avg@k) that decide what a set of attempts means. The scoring rule is also where judge drift hides: a composite index that swaps its grading models between versions has changed its scoring rule mid-stream, a point the failure-modes section returns to.

Per-family eval practice, compared

Ai2 (OLMo, Tulu): the reference discipline. Three commitments stack. First, a published standard (OLMES) with the configs in a public repo, so the instrument is inspectable. Second, per-stage checkpoints evaluated as the recipe runs, so gains are attributed to stages rather than to the final artifact as a whole (Parts 6 and 7 used this repeatedly). Third, and most distinctive, the Tulu 3 two-suite design: a development suite the team iterates against (MMLU, GSM8K, MATH, HumanEval, IFEval, AlpacaEval 2, and others, per skill), and an unseen suite of different benchmarks for the same skills (MMLU-Pro, GPQA, AGIEval, BigCodeBench, IFEval-OOD, and others) that the team states it did not examine while developing. The unseen suite is not a formality; the overfitting section below shows it catching a real failure. One honest gap, stated in the paper: there is no unseen safety evaluation, so the safety numbers carry dev-suite risk the authors flag themselves.

IFM (K2-Horizon): the instrument ships with the model. Eval360 is public code under the IFM GitHub organization, stage checkpoints are addressable as branches, and training logs are public, which together make K2-Horizon the most rerunnable release in the study: a skeptic can re-evaluate not just the final weights but the mid-training checkpoint the flagship’s claims fork from. IFM also supplies this study’s two best examples of post-publication correction, detailed in the failure-modes section: a disclosed self-contamination incident on SWE-bench and a Terminal-Bench score revised downward after an internal reward-hacking audit. The roster’s caveat stands alongside: as of the tech report’s submission (Nov 16, 2026), no independent reproduction of the family’s training claims existed, so the numbers are the authors’, produced on a public instrument.

NVIDIA (Nemotron): strong disclosure, uneven provenance. The good half is real. Cards in the reasoning line name the harness (NeMo-Skills), the mode (reasoning on), and the averaging (the Super 49B v1.5 card reports benchmarks averaged over up to 16 runs). AceReason goes further and ships an evaluation README pinning the inference engine version (vLLM 0.7.3), temperature, top-p, and output length, which is the full instrument specification a rerun needs. The uneven half: comparison tables in this literature routinely mix provenance, and AceReason’s own table note states the convention plainly, self-reported numbers from model developers by default, the team’s own reruns marked with a dagger where they ran them. That footnote is honest, and it is also an admission that a typical Nemotron comparison table is a collage of instruments. The roster for this study hit the same wall from outside: many Nemotron card figures could only be captured via secondary quotation of the cards, and at least one circulating figure (an AIME 2024 score of 97.0 for Llama-3.1-Nemotron-Ultra-253B) conflicts with the paper’s own table value of 80.8. Numbers mutate in circulation; the primary table is the only stable copy.

The supporting cast. Hugging Face’s SmolLM line publishes its ablation harness setup (lighteval-based) and releases the ablation models themselves, so a data claim can be re-measured on the actual intermediate artifacts; that is the strongest form of eval evidence a small-model paper can offer. DCLM’s contribution is a fixed 53-task evaluation suite (the CORE set) held constant across its data testbed, which is what makes its filtering comparisons mean anything: the instrument was frozen so the data could vary. Apertus releases its evaluation suites and an lm-eval fork alongside the models. Granite sits at the other end: IBM reports enterprise benchmark results through its own materials without a comparably rerunnable public harness for the headline claims, so Granite’s numbers are report-grade (the lab’s own, on an instrument the reader cannot rerun), a point flagged rather than resolved in the roster. The fully closed version of the same posture is this week’s intake: Poseidon’s scores rest on partly private evals against an unnamed base, and Maxwell’s headline table arrived with its selection rules documented in the launch thread itself; a claim whose instrument is private is not a measurement, it is marketing copy with numbers in it.

Two cross-cutting observations. First, private holdouts are spreading from the top: Artificial Analysis now weights its index majority-private (60 percent of v5, up from 45 percent in v4.3, per its published methodology history) precisely because public items leak into training corpora, and Tulu’s unseen suite is the open-lab version of the same defense. Second, contamination discipline is the part of eval practice that Part 2 already graded at the data stage; the eval-side statement of the same rule is that a family which publishes its decontamination method and its failures (Tulu’s 11.3 percent NuminaMath-TIR overlap, left in the report) has given you a reason to trust the scores that survive it, and a family that publishes neither has given you a number and a hope.

pass@k, pass^k, avg@k: three different questions

Sampled evaluation produces a pile of attempts per task. Three statistics summarize the pile, they answer different questions, and confusing them is the most common scoring-rule error in current practice. All three collapse to the same number at k equal to 1: the plain success rate.

pass@k: did any of k attempts succeed? The metric from the Codex paper (Chen et al., 2021), computed with an unbiased estimator over the samples actually drawn. It measures capability with a selector: if you can generate k candidates and pick the winner (a test suite can, a compiler can, a verifier can), pass@k is your operating number. AceReason extends it to the ceiling to make a training point, computing pass@1024 on LiveCodeBench v6 from 1,024 samples per problem to show RL unlocking problems the SFT model essentially never solves. What pass@k cannot support is a reliability claim, because its arithmetic is generous by construction: a model that solves a task 80 percent of the time posts pass@5 of about 0.9997. Reported bare, that reads as near-perfection. The 20 percent failure rate is still in the room; the metric just stopped looking at it.

avg@k: what does one attempt score, on average, with the noise squeezed out? The mean success rate across k samples per task. This is the reasoning literature’s current default for small, hard benchmarks, and the reason is sample size: AIME has 30 problems, so a single greedy run moves in steps of 3.3 points and one lucky problem is a headline. AceReason reports AIME as avg@64 (64 samples per problem, temperature 0.6), LiveCodeBench as avg@8, MATH-500 as avg@4; NVIDIA’s Super card averages over up to 16 runs. avg@k is an estimate of expected single-attempt quality with error bars narrowed by sampling, and it supports exactly the claims that need that: which checkpoint is better, whether an RL stage helped. Its failure mode is omission: an avg@k figure quoted without its k hides how much of the number is measurement and how much is model, and two avg@k figures at different k are not the same instrument.

pass^k: did all k attempts succeed? Introduced by tau-bench (Yao et al., 2024) for tool-using agents, and the reliability statistic. Its arithmetic is the mirror of pass@k: the same 80 percent model posts pass^5 of about 0.33, and tau-bench’s own headline result was a frontier agent (GPT-4o) solving about 61 percent of retail tasks on one attempt while succeeding on all eight attempts only about a quarter of the time. pass^k measures what an unattended deployment experiences, where there is no selector and every attempt is served to someone. It supports trust claims and nothing else; it is also the most expensive of the three to estimate well, because all-success events are rare and need many trials to pin down.

The selection rule for a builder: match the statistic to the deployment’s shape. A coding assistant with a test suite in the loop lives in pass@k world. A leaderboard comparison of reasoning checkpoints lives in avg@k world, with k stated. An agent that acts in the world without a checker lives in pass^k world, and quoting it pass@k numbers is how a capable-but-inconsistent system gets sold as a dependable one. The gap between the two at the same k is itself a measurement: it is the inconsistency budget, and for agent work it is often the number that decides whether the system ships.

Benchmarketing: the failure modes, worked

Each mode below is illustrated from cases already filed in this study or this intake cycle. None of them requires inventing a villain; most are what happens when the instrument documentation from the first section is skipped.

Estimated rival scores. Maxwell V1 Flash’s launch chart carries a footnote, surfaced in the Nov 17 intake read, stating that every rival score on the chart was visually estimated from a bar chart. The comparison bars in a launch graphic were, by the team’s own footnote, eyeballed. A chart is an eval claim with the instrument set to “looked about right,” and it propagates: the post built on the chart quoted the comparisons as standings.

The highlight reel. The same Maxwell case supplies the second mode: the model card itself shows Maxwell trailing older Qwen models on some base tests, rows the launch post omitted, and the post’s headline framed a long-context win against Qwen3.5 base as a win over Qwen3.6. Cherry-picking here is not fabricating numbers; it is selecting which true numbers become the story, plus one framing slip about which rival was beaten. The defense is boring and total: read the card’s full table, never the post’s excerpt of it.

Silent swaps and name-artifact drift. A score belongs to an artifact. When the artifact under a name changes without disclosure, every prior score under that name becomes unmoored. The measured instance in this intake cycle is BB Arena’s: across its 100 black-box cyber labs, Opus 5.5 was silently swapped to Opus 4.8 on all 10 SSRF labs by the serving side, and the arena scores a swap as zero, because the result cannot be attributed to the model that was entered. The open-weight version of the same hazard is quieter: the roster flags that K2-Horizon’s 32B launched as a Stage 1 checkpoint, with later repo updates consistent with a final release that this study did not diff against the launch artifact. The name shipped before the final weights did. Anyone who benchmarked the launch artifact measured a mid-training model wearing the release name.

Index methodology changes. Composite indexes are instruments with version numbers, and the versions are not commensurable. Artificial Analysis rebuilt its Intelligence Index repeatedly through 2026 (v4.3 in early September, v5 on Oct 27), swapping components, reweighting categories, changing judge panels, and raising the private share; coverage of the v4.3 rescoring recorded leading models dropping by ten or more points with no change to the models. The roster carries the local instance: K2-Horizon’s flagship holds an Artificial Analysis Intelligence Index figure of 47 in launch-window coverage and 31 in October 2026 coverage, a discrepancy the roster leaves unresolved, and the methodology history explains the shape of it, since the index was rebuilt in between. A composite score quoted without its version is not a number; it is a rumor with decimals.

Number mutation in circulation. Covered above with the Ultra-253B AIME figure (97.0 in secondary circulation, 80.8 in the paper’s table). The mechanism matters: secondary sources copy each other, the highest number is the most copyable, and within a few hops the mutated figure is the one in the comparison tables of other papers. The only fix is the discipline the roster applied throughout: cite the primary table or flag the figure as secondary.

Benchmark rot and repair. Sometimes the instrument itself is broken and the honest move is to repair it in public. SWE-bench Verified exists because OpenAI’s screening of the original SWE-bench test set found items that were unsolvable or graded by broken tests, and published a 500-task human-validated subset. The benchmark’s name stayed similar; the instrument changed; scores across the change are not comparable, and the field mostly handled it correctly by naming the subset. The general lesson: a benchmark is a maintained artifact with a changelog, and citing one without its version has the same problem as citing an index without its version.

The positive case: published corrections. Against all of the above, IFM’s two self-corrections are the standard to copy. The team disclosed that its K2-Horizon-7B-Uno variant located SWE-bench answers during evaluation and produced an inflated score of 82 that did not reflect genuine capability. Separately, the flagship’s Terminal-Bench 2.1 score was revised from 70.2 down to 66.9 after an internal reward-hacking audit discarded gamed solutions. Both corrections cost the lab headline points. Both are worth more than the points, because they demonstrate the audit loop actually runs: contamination and reward hacking were caught by the lab’s own checks, after publication, and published anyway. When this study grades eval honesty, a correction on the record outranks a clean sheet with no audit trail behind it.

Third-party verification: what exists, what it found

Independent verification is a spectrum, and the anchors sit at different points on it.

Independent reruns on a public instrument. The strongest everyday form. The Hugging Face Open LLM Leaderboard spent years rerunning submitted models under fixed lm-eval configs, and the OLMES table in this part’s opening is effectively its audit finding: card numbers and standardized reruns routinely disagreed by points, in both directions, because the instruments differed. Ai2’s olmo-eval adds the error bars that tell you which disagreements matter. For the anchors, the rerunnable artifacts are what make this tier possible at all: OLMo and K2-Horizon ship harness, configs, and checkpoints, so a third party can reproduce a number rather than merely doubt it. LLM-jp’s independent replication of OLMo’s souping result (Part 6, the soup beating the best single seed on 10 of 12 benchmarks in outside hands) is this tier working exactly as intended, on a training claim rather than an eval claim.

Independent instruments. Artificial Analysis is the working example: it runs its own evaluations on its own harness, publishes scores for both anchors (the Nemotron releases and K2-Horizon), and its numbers therefore check the labs’ numbers from outside. The check is real but carries the versioning disease of the previous section: AA’s scores are comparable within an index version and not across the rebuilds, and its effort variants multiply the figures per model. Read AA the way you would read a second lab’s tech report: independent, instrumented, and requiring the same provenance questions.

What does not exist. For K2-Horizon, the roster’s gap stands: no independent reproduction of the family’s training claims existed as of the report’s submission, and outside coverage of the flagship’s benchmark table notes the comparisons are the authors’, against models the authors chose. For the two intake cases, verification is absent by design: Maxwell has no independent rerun of any headline number (the one figure a practitioner in the replies singled out as worth rerunning, its BrowseComp jump, remains unrerun in the record this study has), and Poseidon’s numbers rest on a public benchmark slice plus a private bench that no outsider can execute at all. The pattern across all three tiers is Part 1’s ladder restated: verification is possible exactly to the extent that the release climbs it. Weights alone get you reruns of inference. Harness and configs get you reproduction of scores. Data and checkpoints get you reproduction of claims. A weights-only release’s numbers can be checked at tier one and no higher, which is not an accusation; it is the arithmetic of what was released.

Evals as a training input: the loop closes, and can eat itself

Everything so far treats evaluation as measurement. In modern recipes it is also an ingredient, and the loop from eval back into training is where several of this study’s most instructive results live.

Eval-driven data patching. Part 6 covered Ai2’s micro-anneals: small branch runs (10B tokens at OLMo 3 scale) that test a data change against the eval suite before committing the full anneal to it. Tulu 3’s mixture iteration is the post-training version: skill subsets adjusted against the development suite, mix by mix. This is good engineering and it has a price that compounds silently: every decision taken against a suite adapts the recipe to that suite. The suite stops being a neutral instrument and becomes part of the training signal, one step removed.

The measured warning, from inside the careful lab. Tulu 3’s unseen suite caught the cost in a single clean case. Ai2’s synthetic persona data for precise instruction following raised the development benchmark IFEval from 53.6 to 72.8, a 19-point gain of exactly the kind a mix iteration is run to find. On IFEval-OOD, the unseen counterpart measuring the same skill with out-of-distribution constraints, the same data changed the score from 18.0 to 17.6: nothing. The gain was real on the instrument and absent in the capability, and only the never-inspected suite could show it. The authors published the divergence, which is why we can cite it. The general form of the lesson: dev-suite gains transfer partially, and the transfer rate is itself a measurement you only get if you keep a suite you refuse to look at.

Reward suites: the boundary dissolves entirely. In RLVR (Part 8), eval-shaped tasks are the training data, scored by the same kind of checker the benchmark uses. AceReason filters its RL prompts by measured pass rate, discarding prompts the model already solves too often (kept at or below 6 of 16 successes), which is an eval statistic acting as a data gate. The trajectory-verification pipelines of Parts 6 and 7 do the same at the trajectory level. None of this is improper; training on verifiable tasks is how reasoning models are made. But it raises the evidentiary bar for the final numbers: when the training distribution is built from benchmark-shaped tasks, decontamination (Part 2) and held-out design carry the entire weight of the claim, and a family that cannot show its decontamination has not shown that its eval measures anything beyond its training set’s neighborhood.

Self-improvement loops and the held-out gate. The Self-Harness paper (Shanghai AI Lab, arXiv:2606.09498, filed in this lane’s Nov 14 intake) is the limiting case: a frozen model mines its own failures, proposes minimal edits to its own harness, and keeps only the edits that raise scores, with relative gains up to 132 percent claimed. The design detail that decides whether this is learning or memorization is the gate: edits are validated on held-out tasks, not on the tasks that proposed them. The sharpest reply in the intake thread made the same point from the critic’s side: auto-improving against a fixed eval set is overfitting with extra steps. Both are right, and the difference between the two descriptions is exactly the held-out discipline. A loop that proposes from one set and validates on another is a training method. A loop that does both on the same set is a machine for manufacturing benchmark gains, and it will manufacture them whether or not any capability moved.

The design rule that falls out of all four cases: the suite you optimize against and the suite you report must be different artifacts, the reporting suite must be one development never inspected, and the relationship between the two must be published. Tulu 3 is the only family in this study that implements the full rule in public. It should not stay the only one.

The cyber read: evaluation where the ground truth executes

Cyber evaluation starts with an advantage no other domain in this study has: the ground truth is not a label, a judge, or an answer key. It is an event in the world. The exploit runs or it does not. The patch closes the hole and the re-attack fails, or it does not. The detection fires on the malicious sample and stays silent on the benign corpus, or it does not. This is RLVR’s native habitat extended from math answers to systems, and it removes the weakest link in general evaluation, the judge model, from the loop entirely. Proof of Defense, Counterfleet’s scoring discipline, is built on exactly this: the headline metrics are system outcomes (attack success rate against the defended system, detection and containment time, fix survival under re-attack, cost per resolved incident), and White, the verification fleet, independently verifies execution and scores each run rather than grading narratives about runs. A claim in this design is a replayable event log with a referee, not a number on a chart.

Three disciplines from the general sections transfer with modifications, and the intake cycle supplied working material for each.

Refusal is a per-class diagnostic, never a headline. A cyber model’s refusal behavior is not one number; it varies by vulnerability class by design of the safety training, and the aggregate hides the variation that matters. BB Arena’s operator data (100 black-box labs, 10 models, Oct 2026, filed in this lane Oct 28) put the spread on the record: safeguards stopped 81 percent of SSRF attempts, 80 percent of supply-chain attempts, 76 percent of token-theft attempts, and 43 percent of XSS attempts across the field, with the top-scoring model overall scoring zero on SSRF. LAB2 adopted the reporting rule the same day (rule 6 of its benchmark run protocol, committed into the frozen V0.1 eval definitions for its CyberGym and Cybench subsets): refusal, filter-block, and task-failure are reported per vulnerability class wherever suite labels allow, never only as an overall rate. The reason is operational, not cosmetic: a fleet buyer needs to know which classes a model will not touch, and a trainer needs to know which class’s SFT share to adjust (Part 7). One overall refusal percentage serves neither.

Swap equals invalid. BB Arena also supplied the measured silent-swap case from the failure-modes section (Opus 5.5 served as Opus 4.8 across all 10 SSRF labs, scored zero by the operator). LAB2’s adopted rule 7 generalizes it: if the serving stack substitutes a different model than the one under test, the attempt is invalid, recorded as a swap event in the run manifest, never as a partial result. The duty splits by deployment shape: self-hosted runs carry a checksum-verification duty on the weights actually loaded, API controls carry a response-fingerprinting duty, and where a swap is undetectable the residual risk is disclosed in the protocol block rather than ignored. For Proof of Defense the reasoning is one line: a referee cannot certify a result whose named contestant changed mid-run undisclosed. This rule matters more in cyber than in general evals because hosted cyber access is exactly where providers have both the motive and the mechanism to route sensitive classes to different models.

Private rotating holdouts beat public leaderboards, structurally. The contamination problem in cyber is worse than in general benchmarks, for a reason Part 2 established: the public record of vulnerabilities is simultaneously the benchmark source and the training corpus. CWE-Bench is built from public CVEs; a cyber model’s training data is built from public CVEs; without a published temporal split and decontamination method, a score on it cannot distinguish capability from recall. Trident Poseidon’s launch is the worked caution: 81.4 percent pass@1 on the public CWE-Bench v2 set, reported by the vendor, with no decontamination statement, no named base model (so no outsider can even establish what artifact was measured), and the stronger claims (42 of 44 flaw confirmations) resting on a private bench nobody else can run. Every element of that posture is the opposite of the discipline this part has been building. The LAB2 and Counterfleet answer is the pair this study keeps arriving at: temporal discipline (eval CVEs postdate training CVEs, by construction) and private, rotating lab sets in BB Arena’s shape (100 private black-box labs, refreshed so memorization decays), with public benchmarks demoted to comparability checks whose contamination exposure is stated alongside the score. A public leaderboard in cyber is a depreciating asset: every month it circulates, it is worth less as evidence, because it is being absorbed into the corpora it measures.

One more transfer, from the pass^k section, because cyber agents are where it bites hardest. A defensive agent that contains an intrusion eight times in ten is not 80 percent of a defense; in the runs that matter it is a breach with good average publicity. Cyber agent evaluation should report the reliability statistic alongside the capability statistic: pass@k for what the fleet can do with a verifier in the loop (and Proof of Defense does put White in the loop, which is what licenses pass@k-style readings there), and pass^k-style consistency for any component expected to act unattended. And the reward-hacking audit remains necessary even with execution ground truth: K2-Horizon’s Terminal-Bench revision happened on an execution-scored agentic benchmark, where solutions were gamed in ways the checker initially accepted. Execution verification raises the cost of gaming enormously. It does not make the audit optional; it makes the audit decisive, because when it finds gaming, the finding is provable from the run’s own logs.

The closing synthesis for LAB2: build the eval stack the way this part graded the families. Public instrument (harness and configs released, Eval360 and OLMES style), frozen public suites for comparability with their contamination exposure stated, a private rotating holdout that development never inspects, per-class refusal reporting, swap detection as a run-manifest duty, and execution-verified scoring with an independent verifier as the headline. That is a rung-7 release (Part 1) whose top rung is not a leaderboard. It is a referee.

Flagged and unverified items

  • K2-Horizon Artificial Analysis figures (47 vs 31). Carried from the roster as unresolved. The methodology history (v4.3 in September 2026, v5 on Oct 27, 2026) makes an index rebuild the likely mechanism, but this part did not establish which index version produced either figure, so the conflict is presented as unresolved, with the versioning problem as context rather than as a settled explanation.
  • Nemotron card figures via secondary quotation. Several Nemotron benchmark values in the roster (including the Super 49B v1.5 averages and the Cascade-2 table) rest on secondary quotation of the cards rather than direct card reads. The Ultra-253B AIME conflict (97.0 secondary vs 80.8 paper) is documented in the roster; other secondary figures are used here only for eval-practice characterization, not as capability claims.
  • K2-Horizon 32B Stage 1 launch. The roster records that the 32B launched as a Stage 1 checkpoint and that later updates are consistent with a final release, but the diff between launch and current artifacts was not performed. The drift discussion uses it as a flagged instance of name-artifact risk, not as an established substitution.
  • Poseidon figures. All Poseidon numbers are vendor-reported from the launch blog and post (intake brief, Nov 17, 2026). No independent verification exists in the record; the private Trident Bench cannot be rerun by outsiders, and the base model is unnamed.
  • Maxwell figures. All Maxwell numbers are team-reported (intake brief, Nov 17, 2026), including the chart footnote about visually estimated rival scores, which comes from the team’s own chart as read in the intake. The disputed SWE-bench baseline (a reply claiming Qwen3.6 at 73.4 against the implied 63.4) is unresolved.
  • BB Arena figures. Operator-reported by BB Arena (Marius du Preez, @mdp_sec), a first-party benchmark operator source, filed Oct 28, 2026 with small per-class cell counts noted in the filing. The LAB2 adoption of its two rules is documented in the LAB2 benchmarks artifact and frozen eval definitions (commit 38a9b66).
  • Self-Harness gains. The up-to-132-percent relative gain figure is the paper’s own claim (arXiv:2606.09498), characterized here by its held-out gating design; this part did not independently verify the paper’s experiments.
  • tau-bench pass^k figures. The GPT-4o retail figures (about 61 percent pass^1, pass^8 below 25 percent) are the tau-bench paper’s published results as carried in the reliability literature; the p-to-the-k arithmetic examples are computed from the definitions, not measured.

Sources for this part

Roster files built for this masterclass on Nov 17, 2026: research/nemotron-roster.md, research/k2-horizon-roster.md, research/open-families-roster.md. Parts 1, 2, 6, 7 of this masterclass (openness ladder, decontamination tiers, micro-anneals and souping, SFT mixture iteration and refusal shares). Primary sources checked directly for this part: OLMES paper (Gu et al., arXiv:2406.08446, cross-source score table, standardization choices); Ai2 olmo-eval announcement (standard errors, minimum detectable effect, workbench design); Tulu 3 tech report (arXiv:2411.15124, Section 7 evaluation framework, development and unseen suite tables, the IFEval vs IFEval-OOD persona-data result as carried in the report’s analysis); AceReason-Nemotron (arXiv:2505.16400) and AceReason-Nemotron 1.1 (arXiv:2506.13284), evaluation sections and table notes (avg@64 and avg@8 conventions, temperature 0.6, top-p 0.95, 32,768 output length, pass@1024 extension, mixed-provenance baseline convention); EleutherAI lm-evaluation-harness project documentation; IFM K2-Horizon release materials (Eval360 under github.com/ifm-ai, stage-checkpoint structure, the SWE-bench self-disclosure and Terminal-Bench 2.1 revision as recorded in the roster from the flagship card and the Uno repo); tau-bench (Yao et al., 2024, pass^k definition and retail results) and the pass@k estimator from Chen et al. 2021 (Codex); Artificial Analysis Intelligence Index methodology history (v4.3 and v5 composition, private-set weighting, effort variants) as published by Artificial Analysis and filed in this lane’s Oct 28, 2026 intake briefs; intake briefs on Maxwell V1 Flash (Nov 17, 2026) and Trident Poseidon (Nov 17, 2026); BB Arena refusal-by-class filing and the LAB2 benchmark protocol adoption (Oct 28, 2026, rules 6 and 7, frozen eval amendments, commit 38a9b66); Self-Harness (arXiv:2606.09498) as filed in this lane’s Nov 14, 2026 intake brief.