AI Literacy / Glossary

Token

A token is the small unit of text a language model reads and writes: a whole word, part of a word, a punctuation mark or a space. The model predicts one token at a time.

Before a chatbot can work with your message, it splits the text into tokens. Common short words are often one token, while long or unusual words are split into several pieces.

When it answers, the model chooses the next token, adds it to the text and then predicts the one after that. This step-by-step prediction is why answers appear word by word.

Tokens also explain some limits: services count usage in tokens, a model can only take in a certain number of tokens at once, and tasks like counting letters can go wrong because the model sees pieces, not single letters.