How LLMs Are Made / Glossary

Tokenizer

A tokenizer is the tool that cuts text into tokens, the pieces a language model reads, and turns each piece into a number. It also turns the model's output numbers back into text.

A tokenizer has a fixed vocabulary of pieces. Frequent words stay whole, while rare words, names and long numbers are split into several smaller pieces. Each piece has an ID number, and the model only ever sees those numbers.

Most LLM tokenizers are built with byte pair encoding, a method that starts from single characters and keeps merging the pairs that appear most often. GPT-2's tokenizer had 50,257 tokens; Llama 3's has about 128,000.

The tokenizer explains some odd behaviour. Models can struggle to count letters because they see pieces, not letters, and the same sentence can need far more tokens in some languages than in others, which makes them slower and more expensive to process.