Skip to content
EN

Lesson 4 of 6 · 6 min · +100 XP

How does an LLM generate a reply, one token at a time?

A language model writes its reply one token at a time. For each step it scores every possible next token, turns the scores into odds and draws one, so likely tokens win most often. The token is added to the text and the loop repeats until a stop token. Temperature makes the draw safer or more adventurous.

▶ Try it

Why it matters

Every reply you read was drawn token by token from a list of odds. Knowing that explains why answers stream in, why they vary and how one dial changes the tone.

In 6 minutes you'll be able to

  • Language models write by scoring every next token, drawing one and repeating.
  • Different answers to the same question come from different draws, not a change of mind.
  • For facts and maths, steady settings help; for ideas, warmer settings bring variety.

What you'll practise

  1. The generation loop

    You send Vireo a question. Put the steps of writing the reply in order.

    1. Your message is split into tokens
    2. The model scores every possible next token
    3. The scores become odds, shaped by the temperature
    4. One token is drawn and added to the text
    5. The loop repeats until a stop token is drawn

  2. Run the generation machine

    Press Next token and watch the wheel draw each word. Then set the dials for each mission and test them. The odds are made up to show the idea.

    1. A maths answer must be reliable: the right number, every single time.

      You: What is 7 × 8?

      Next token · odds

      1. “7 × 8” 70% · “Seven times eight” 22% · “The result” 8%
      2. “is” 90% · “equals” 8% · “makes” 2%
      3. “56” 84% · “54” 7% · “58” 5% · “65” 3% · “purple” 1%
      4. “.” 90% · “!” 10%

      ✓ At low temperature the top token wins almost every draw, so all three replies say 56. For facts, maths and code, cold is what you want.

    2. Brainstorm: the robotics club wants lots of different team names to vote on.

      You: Suggest a name for our robotics team.

      Next token · odds

      1. “Team” 100%
      2. “Circuit” 50% · “Bolt” 20% · “Gear” 14% · “Pixel” 10% · “Turbo” 6%
      3. “Owls” 55% · “Rockets” 20% · “Wizards” 12% · “Foxes” 9% · “Sparks” 4%

      ✓ Warmer odds give the less likely words a real chance, so every draw brings a new name. For ideas, variety is the point.

    3. A story opener should surprise the reader, but it still has to make sense.

      You: Start a short story about a lighthouse.

      Next token · odds

      1. “The lighthouse keeper” 100%
      2. “found” 48% · “heard” 26% · “painted” 18% · “the” 5% · “ate” 3%
      3. “a” 66% · “an old” 20% · “one” 10% · “glorp” 4%
      4. “message” 40% · “whale” 25% · “door” 20% · “sock” 10% · “of of” 5%
      5. “in the fog.” 50% · “under the stairs.” 28% · “in a bottle.” 17% · “banana banana” 5%

      ✓ High temperature for surprise, top-k 3 so only the three likeliest words at each step can be drawn. Creative, and the nonsense tail is cut off.

  3. What it means for you

    1. You ask Vireo the same question twice and get two different wordings. What happened?

      • Each token was drawn again from the odds (Answer)
      • Vireo learned something new between the two questions

    2. Why does a long reply appear on screen word by word?

      • Each token is generated only after the one before it (Answer)
      • It's an animation to make the app feel friendly

    3. A developer sets the temperature to 0 for a tool that fills in tax forms. Good idea?

      • Yes: the top token wins every time, so answers are steady (Answer)
      • No: temperature 0 switches the model off

Remember

A reply is drawn one token at a time from a list of odds. Low temperature is steady, high temperature is creative.

A reply is drawn one token at a time from odds. Low temperature is reliable, high is creative, and top-k cuts off the nonsense.

Key facts

  • The transformer, the design behind today's large language models, generates its output one token at a time, feeding each new token back in as input for the next. · Source: Vaswani et al., Attention Is All You Need, Google, 2017
  • Researchers found that always picking the most likely word tends to produce bland, repetitive text, and proposed nucleus (top-p) sampling, which draws only from the smallest set of tokens that covers most of the odds. · Source: Holtzman et al., The Curious Case of Neural Text Degeneration, ICLR 2020
  • Top-k sampling, drawing only from the k most likely tokens, was used to generate more varied stories than standard decoding. · Source: Fan, Lewis and Dauphin, Hierarchical Neural Story Generation, ACL 2018
  • Temperature divides the model's raw scores before they become odds: below 1 the favourite gets stronger, above 1 the odds even out.

Updated:

Questions people ask

What is temperature in an LLM?

Temperature is a setting that reshapes the odds before each token is drawn. Low values make the most likely token win almost every time, for steady answers. High values flatten the odds, so replies get more varied and, if pushed too far, less sensible.

Why do AI chatbots give different answers to the same question?

Chatbots usually draw each token from a list of odds instead of always taking the top one. A different draw early in the reply leads to different wording, and sometimes a different answer.

What is top-k sampling?

Top-k sampling lets the model draw only from the k most likely next tokens. It keeps some variety while cutting off the long tail of unlikely, often nonsensical tokens.

How does an LLM know when to stop writing?

During training it learned a special stop token that marks the end of a reply. When that token is drawn, or a length limit is reached, generation ends.

Related terms