How LLMs Are Made / Glossary

Scaling laws

Scaling laws are patterns showing that a language model's loss falls smoothly and predictably as you add parameters, training data and compute. Labs use them to plan how big to build.

Research in 2020 found that loss follows simple power laws: double the model size, data or compute and the loss drops by a predictable amount. That let labs forecast how good a model would be before spending the money to train it.

A 2022 study refined the recipe: for a fixed compute budget, grow the model and its data together, at roughly 20 training tokens per parameter. Its 70-billion-parameter model Chinchilla, trained on 1.4 trillion tokens, beat the 280-billion-parameter Gopher, which used the same budget.

Scaling laws predict loss, not specific skills, so some abilities still surprise. Many recent models also train on far more than 20 tokens per parameter, because a smaller model trained longer is cheaper to run afterwards.