A Benchmark Is a Question, Not a Crown

A score answers a designed question

A benchmark is a collection of tasks and a scoring procedure. It can help compare systems when the tasks resemble the work people care about and the measurement is reliable. But the score is not a direct reading of a model’s entire capability. It is evidence about performance under a particular design.

The first question is therefore not simply “What score did it get?” Ask what the benchmark selected, what it left out, and what successful performance is supposed to represent.

Task selection changes the conclusion

If a benchmark emphasizes mathematics, code generation or multiple-choice knowledge, it says more about those sampled tasks than about customer support, long-horizon planning or a specialized business workflow. A broad label such as “reasoning” can hide important differences in task construction and scoring.

Research on evaluation practice has identified recurring concerns including limited task coverage, insufficient documentation and a mismatch between measured performance and the wider capability people infer from it. The LiveBench project responds to some known problems by refreshing questions and using tasks with automatically verifiable answers where possible. That is a design strategy, not a universal solution: each benchmark still has a scope and assumptions that readers should inspect.

Contamination and saturation

When test questions or close equivalents appear in training data, a model may benefit from prior exposure. Researchers cannot always know the full contents of training corpora, so contamination checks are difficult. A benchmark can also become saturated: once many systems score near the top, it may no longer distinguish their practical strengths well.

Fresh questions, hidden test sets and contamination analysis can reduce some risks, but none guarantees that a benchmark predicts performance on unseen real work. A high score is a reason to investigate further, not proof of transfer.

Scoring is part of the experiment

Exact-match scoring is easy to reproduce but may overlook valid answers expressed differently. Human or model-based judges can handle richer responses, but introduce their own variation: rubrics may be underspecified, judges may favor certain styles, and different evaluators can disagree. The choice of prompt, tools, allowed retries and time budget also affects the result.

These observations summarize issues discussed in evaluation research. TST LAB’s practical interpretation is straightforward: ask how much of the reported result comes from the model, and how much from the benchmark’s prompts, tools, graders and operating conditions.

A practical checklist

Before relying on a model claim, ask:

1. What exact tasks and user needs does the benchmark represent?
2. Which model version, configuration, tools and time limits were used?
3. How were answers scored, and who or what judged them?
4. Is there a credible contamination or data-leakage discussion?
5. Is the benchmark saturated or able to separate current systems?
6. Were results independently reproduced, and what limitations were disclosed?
7. Have you tested representative examples from your own workflow, including failure cases?

Use the score to choose the next test

Benchmarks are most useful when they help narrow a question. A published result can suggest which systems deserve a closer look and which failure modes to probe. The final decision should come from task-specific testing with the actual workflow, constraints and acceptable error defined in advance. A benchmark is a question—not a crown.