Skip to content
EN

Lesson 1 of 6 · 6 min · +100 XP

How are AI language models tested and scored?

New language models are tested on benchmarks: fixed sets of questions with checkable answers, such as maths problems, coding tasks with tests, or multiple-choice exams. Labs add safety tests and blind human comparisons. A score only means something if the questions weren't in the training data and every model is tested under the same rules.

▶ Try it

Why it matters

Every launch comes with big numbers. Knowing how the tests work lets you tell a real improvement from a leaky test or an unfair comparison.

In 6 minutes you'll be able to

  • Benchmarks are fixed sets of questions with checkable answers, from maths to coding to safety.
  • Scores mislead when test questions leaked into training or the rules differ between models.
  • Look for blind comparisons, outside results and tests on tasks like yours.

What you'll practise

  1. Fair test or leaky test?

    Can you trust the score from this test?

    • Questions written after the model's training data was collected → Fair test
    • A maths test posted online years ago, answers included → Can't trust the score
    • Coding tasks checked by actually running the code → Fair test
    • The team tweaks the model until it aces this one test → Can't trust the score
    • People compare two answers without knowing which model wrote them → Fair test
    • Our model gets 10 tries per question, the rival gets 1 → Can't trust the score

  2. Check the launch claims

    Halcyon AI, an invented rival lab, just announced Halcyon 2. Pick the right check for each claim, see what it turns up, then stamp it.

    You: What's new in Halcyon 2?
    Halcyon AI: Meet Halcyon 2, our best model yet!It scores 92% on MathBench, a public test of 1,000 school maths problems.It beats Vireo 3 at coding: 71% vs 64%.In blind tests, people preferred its answers 6 times out of 10.It is the safest model ever built.It scores 89% on a new set of questions written after its training ended.Try it today!
    1. It scores 92% on MathBench, a public test of 1,000 school maths problems.

      Check for leaks · Read the test details · A popular public test may have leaked onto the web, and from there into the training data. Check that first.

      Halcyon 2 tech report · data section: MathBench has been online since 2021, and the training data includes forum pages that copy its questions and answers.

      → Misleading: If the model saw the questions and answers in training, 92% may be memory rather than maths skill.

    2. It beats Vireo 3 at coding: 71% vs 64%.

      Compare like with like · Read the test details · Two scores side by side only mean something if both models took the test the same way.

      Halcyon 2 tech report · footnote 4: Halcyon 2 got 10 tries per task, counted as solved if any try passed. Vireo 3's 64% is for a single try.

      → Misleading: Ten tries against one isn't a fair race. With the same rules, the gap might shrink or vanish.

    3. In blind tests, people preferred its answers 6 times out of 10.

      Find outside results · Read the test details · Preference claims are easy to check if others ran the comparison too. Look for independent results.

      An independent rating site · blind votes: 4,200 blind votes: Halcyon 2's answer was preferred in 58% of comparisons.

      → Holds up: Blind, lots of votes, run by someone else, and 58% is close to the claim. This one holds up.

    4. It is the safest model ever built.

      Common sense · Read the test details · A sweeping claim with no number should make you pause. What test could even prove “ever”?

      Halcyon 2 tech report · safety section: No safety tests named, no scores, no outside testers mentioned.

      → Can't verify: Nothing shows it's false, but nothing backs it either. Without named tests and results, it can't be checked.

    5. It scores 89% on a new set of questions written after its training ended.

      Read the test details · Check for leaks · Read how the test was made: who wrote the questions, when, and whether they are public.

      Halcyon 2 tech report · evaluation section: Questions written by hired experts after the training data was collected, and kept private.

      → Holds up: Fresh, private questions can't have leaked into training. That's what makes a fair test.

  3. Reading the report card

    1. Every new model now scores above 95% on an old exam. What does that tell you?

      • The exam has become too easy to tell models apart (Answer)
      • AI can now do everything that exam covers, perfectly

    2. Why do some test makers keep their questions secret?

      • So the questions can't leak into training data (Answer)
      • So nobody can complain about the questions

    3. Vireo aces a hard coding benchmark. What should you still check before trusting it with your project?

      • How it does on tasks like yours, and test its code (Answer)
      • Nothing: the benchmark already proved it

Remember

A benchmark score is only as good as the test: fresh questions, the same rules for every model, and results others can check.

A benchmark score is only as good as the test: fresh questions, the same rules for everyone, and results others can check.

Key facts

  • MMLU, a widely used knowledge benchmark, has multiple-choice questions across 57 subjects, from maths to law. · Source: Hendrycks et al., Measuring Massive Multitask Language Understanding, ICLR 2021
  • HumanEval tests coding with 164 hand-written programming problems, each checked by running unit tests on the model's code. · Source: Chen et al., Evaluating Large Language Models Trained on Code, OpenAI, 2021
  • GSM8K is a set of about 8,500 grade-school maths word problems used to test step-by-step reasoning. · Source: Cobbe et al., Training Verifiers to Solve Math Word Problems, OpenAI, 2021
  • The GPT-3 paper checked how many benchmark questions had leaked into its training data, a problem called contamination. · Source: Brown et al., Language Models are Few-Shot Learners, OpenAI, 2020
  • Chatbot Arena ranks models from crowdsourced, blind side-by-side votes: people compare two anonymous answers and pick the better one. · Source: Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, 2023

Updated:

Questions people ask

What is an LLM benchmark?

A benchmark is a fixed set of tasks with checkable answers, such as exam questions, maths problems or coding tasks. Running different models on the same benchmark gives comparable scores.

What is benchmark contamination?

Contamination means test questions or answers ended up in a model's training data. The model may then score well by remembering instead of reasoning, so the score overstates its skill.

Are AI benchmark scores reliable?

They're useful but limited. Trust them more when the questions are fresh or private, every model follows the same rules, and independent groups get similar results.

How do AI leaderboards rank models?

Some average scores across benchmarks. Others collect blind votes: people compare two anonymous answers to the same prompt and pick the better one, and the votes add up to a ranking.

Related terms