How LLMs Are Made / Glossary

Pretraining

Pretraining is the first and biggest stage of building a language model: it reads a huge amount of text and learns by predicting the next token, over and over, without any human labels.

During pretraining, the model sees real text with the next token hidden, makes a guess, checks the real answer and adjusts its weights a little. The text itself provides the answers, so no one has to label anything.

Repeated across trillions of tokens, this simple game forces the model to pick up grammar, facts, writing styles, reasoning patterns and code. Llama 3, for example, was pretrained on over 15 trillion tokens.

Pretraining uses most of the computing power spent on a model and gives most of its knowledge. The result is a base model that continues text well but doesn't yet follow instructions; fine-tuning and feedback training come next.