How LLMs Are Made / Glossary

Loss function

A loss function is the formula that measures how wrong a model's prediction was. Training tries to make this number, the loss, as small as possible.

When a language model predicts the next token, it gives every possible token a probability. The loss looks at the probability it gave to the token that really came next: a high probability means a low loss, and a low probability means a high loss.

This kind of loss is called cross-entropy, and it is what almost every language model is trained to lower. Gradient descent uses the loss to work out which way to nudge each weight.

Loss is also how labs track progress. Scaling laws describe how loss falls predictably as models get more parameters, data and compute. A low loss is not the same as a helpful or truthful assistant, though; that takes further training and testing.