Skip to content
EN

Lesson 4 of 6 · 6 min · +100 XP

What are tokens in AI, and why does a model read words in pieces?

Tokens are the small pieces of text a language model reads and writes. A tokenizer cuts text into common chunks: frequent words stay whole, while rare words, new names and long numbers are split into several pieces. Each token is swapped for a number, its ID, so the model can compute with it. Length and cost are counted in tokens.

▶ Try it

Why it matters

A model never sees letters or whole words, only token IDs. That explains why it struggles to count letters, why long numbers trip it up and why some languages cost more.

In 6 minutes you'll be able to

  • When a model miscounts letters or fumbles a long number, remember it sees pieces, not characters.
  • Writing in a language with fewer tokens per word can make the same request cheaper and faster.
  • When a tool shows limits or prices “per token”, think of roughly word-sized pieces, not whole words.

What you'll practise

  1. Be the tokenizer

    Tap the gaps between letters to cut each word into tokens. Then press Tokenize and compare. The splits and IDs are illustrative, like a typical tokenizer.

    1. the → [the] #279
      Very common words get a token of their own. “the” appears so often that splitting it would waste space.
    2. unbelievable → [un] #403 [believ] #33246 [able] #481
      Long words are built from familiar pieces: “un”, “believ” and “able” each show up in thousands of other words.
    3. Vireo → [Vir] #53413 [eo] #78
      A new name isn't in the vocabulary, so the tokenizer builds it from pieces it does know. Any word can be spelled this way.
    4. 1234567 → [123] #4513 [456] #10961 [7] #22
      Many tokenizers chop long numbers into chunks of up to three digits. The model never sees the whole number at once, one reason long sums can trip it up.
    5. Schmetterling → [Sch] #32065 [met] #4164 [ter] #466 [ling] #2785
      German for “butterfly”. Tokenizers learn mostly from English text, so many words in other languages are split into more pieces.
    1. Same meaning, two languages. Which one costs Vireo more tokens?
      • A: I love cats. → [I] [·love] [·cats] [.] (4 tokens)
      • B: Ich liebe Katzen. → [Ich] [·lie] [be] [·K] [atzen] [.] (6 tokens)
      The German sentence needs more pieces because the tokenizer saw less German. More tokens means more computing, and often a higher price, for the same meaning.
    2. Which message costs more tokens?
      • A: See you at 7. → [See] [·you] [·at] [·7] [.] (5 tokens)
      • B: See you at 19:45:30. → [See] [·you] [·at] [·19] [:] [45] [:] [30] [.] (9 tokens)
      Digits and punctuation often cost a token each. Times, codes and long numbers fill up a model's budget surprisingly fast.

  2. Why pieces, not words?

    Tap each card to see what's behind it.

    🔢 Why turn text into numbers?
    A model only does maths. Each token is swapped for its ID, so a sentence becomes a list of numbers it can compute with.
    📕 Why not one token per word?
    There are too many words: names, typos, new slang. A word list is never complete, but pieces can build any word, even a brand-new one.
    🔤 Why not single letters?
    Then every sentence becomes very long and the model must work much harder to see meaning. Word pieces are the middle ground.
    🧮 Where does the token list come from?
    Before training, a program scans lots of text and keeps merging the most frequent pairs of pieces until it has a fixed list, often tens of thousands of tokens long.

  3. What tokens explain

    Select every statement that's true.

    • A model reads token IDs, not letters
    • Common words are often a single token
    • Rare and new words are split into pieces
    • The same text can cost more tokens in another language
    • Every word is exactly one token
    • Tokens are just syllables
    • Counting letters in a word is easy for a model

Remember

A model reads text as numbered pieces called tokens. Common words stay whole, rare ones are built from parts.

Text becomes a list of token IDs. Common words stay whole, rare ones are built from pieces.

Key facts

  • Splitting rare words into frequent subword pieces with byte pair encoding was introduced for neural translation in 2016 and is the basis of most LLM tokenizers today. · Source: Sennrich, Haddow & Birch, Neural Machine Translation of Rare Words with Subword Units, ACL 2016
  • GPT-2's tokenizer had a vocabulary of 50,257 tokens. · Source: Radford et al., OpenAI, Language Models are Unsupervised Multitask Learners, 2019
  • Llama 3 uses a tokenizer with a vocabulary of 128,000 tokens, which encodes text more efficiently than its predecessor. · Source: Meta, Introducing Meta Llama 3, 2024
  • The same text translated into different languages can need very different numbers of tokens, so some languages cost more to process. · Source: Petrov et al., Language Model Tokenizers Introduce Unfairness Between Languages, NeurIPS 2023

Updated:

Questions people ask

How many words is a token?

It depends on the text and the tokenizer. In English a token is often a short word or part of a longer one, so 100 tokens is usually a bit less than 100 words. Other languages, code and numbers often need more tokens.

Why can't AI count the letters in a word?

The model never sees individual letters, only token IDs for chunks like “straw” and “berry”. To count letters it has to recall how each piece is spelled, which is easy to get wrong.

What is a tokenizer?

A tokenizer is the program that cuts text into tokens and swaps each one for its ID before the model sees it. It is built once, before training, from the most frequent pieces in a large sample of text.

Why are AI models priced per token?

The work a model does grows with the number of tokens it reads and writes. Counting tokens measures that work more fairly than counting words or characters.

Related terms