Skip to content
EN

Lesson 3 of 6 · 6 min · +100 XP

How does next-token prediction train an LLM?

A large language model is pretrained by playing one game over and over: it reads real text, hides the next token, bets on what comes next, then checks the real answer and adjusts its weights. Repeated trillions of times, this simple game forces it to pick up grammar, facts, reasoning patterns and even code.

▶ Try it

Why it matters

Pretraining has no teacher writing answers. The text itself is the answer key, which is why a model can learn from the whole web, and why it picks up so much more than grammar.

In 6 minutes you'll be able to

  • An LLM learns by guessing the next token in real text and adjusting after every guess.
  • The text is its own answer key, so no human has to label anything.
  • Guessing well rewards grammar, facts, reasoning and code, which is why models pick them all up.

What you'll practise

  1. Take Vireo's seat

    Each line comes from the training text. Bet on the next token, then see how a trained model spreads its bets. The percentages are illustrative.

    1. She opened the door and walked ___

      • in: 38% (most likely)
      • out: 20%
      • away: 12%
      • slowly: 6%
      • door: 0.1%
      • other words: 24%
      To guess well here, the model has to learn grammar: after “walked” comes a direction or a manner, almost never another noun.

    2. Water freezes at 0 degrees ___

      • Celsius: 72% (most likely)
      • .: 10%
      • and: 5%
      • Fahrenheit: 3%
      • other words: 10%
      Getting this right means soaking up a fact. Nobody taught Vireo physics: the fact simply appears in the text again and again.

    3. Ana has 3 apples and buys 4 more. Now she has ___

      • 7: 64% (most likely)
      • seven: 12%
      • 4: 3%
      • lots: 1%
      • other words: 20%
      Simple arithmetic wins this round. Predicting text well quietly rewards anything that helps, including step-by-step reasoning.

    4. for i in range( ___

      • 10: 30% (most likely)
      • len: 26%
      • n: 20%
      • apples: 0.1%
      • other words: 24%
      Code is text too. Practising on millions of programs teaches the patterns of programming languages along the way.

    5. English: thank you. French: ___

      • merci: 82% (most likely)
      • thank: 3%
      • bonjour: 2%
      • other words: 13%
      Pairs like this appear all over the web, so the model learns to translate as a side effect of guessing the next token.

    6. My neighbour's cat is called ___

      • Max: 5% (most likely)
      • Luna: 5% (most likely)
      • Mr: 4%
      • Tiger: 2%
      • other words: 84%
      No text can tell you this. The model learns to spread its bets thinly, yet it will still pick a name, one root of confident made-up answers.

  2. One step of pretraining

    Put one turn of the training game in order.

    1. Take a stretch of real text from the training data
    2. Hide the token that comes next
    3. Vireo bets on every possible next token
    4. Reveal the real token and measure how surprised Vireo was
    5. Nudge the weights so the real token gets a higher chance
    6. Slide one token along and play again

  3. What does this round teach?

    Each gap rewards a different skill. Sort them.

    • The cats ___ asleep. (are) → Grammar
    • The largest planet is ___ (Jupiter) → Facts
    • 12 + 30 = ___ (42) → Reasoning
    • Yesterday I ___ to school. (went) → Grammar
    • Water boils at ___ °C at sea level. (100) → Facts
    • Mo is taller than Kai, Kai is taller than Lu, so Lu is the ___ (shortest) → Reasoning

Remember

Pretraining is one game: guess the next token, check, adjust. Played trillions of times, it teaches far more than words.

One game, guess the next token, teaches grammar, facts, maths and code all at once.

Key facts

  • In 1951, Claude Shannon measured how predictable English is by asking people to guess the next letter of a text. · Source: C. E. Shannon, Prediction and Entropy of Printed English, Bell System Technical Journal, 1951
  • Llama 3 was pretrained on over 15 trillion tokens of text. · Source: Meta, The Llama 3 Herd of Models, 2024
  • Much of GPT-3's training text came from a filtered version of Common Crawl, an open archive of the web. · Source: Brown et al., Language Models are Few-Shot Learners, OpenAI, 2020
  • A model trained only to predict the next word could already attempt tasks like translation and summarising without being trained on them specifically. · Source: Radford et al., Language Models are Unsupervised Multitask Learners, OpenAI, 2019

Updated:

Questions people ask

What is next-token prediction?

The training task behind large language models: given some text, predict the token that comes next. The model scores every possible token and learns from how wrong it was.

What is self-supervised learning?

Learning where the data provides its own answers. A language model hides the next token of real text and tries to guess it, so no one has to label examples by hand.

How can predicting words teach a model facts?

Many next tokens can only be guessed by knowing something about the world. To lower its error on those, the model has to store patterns that work like facts in its weights.

How many tokens are LLMs trained on?

Modern large language models are pretrained on trillions of tokens. Meta reported over 15 trillion tokens for Llama 3, for example.

Related terms