What are scaling laws in AI?
Scaling laws are patterns showing that a language model's loss falls predictably as you add parameters, training data and compute. They also tell labs how to split a fixed compute budget: a well-known 2022 study found the best results come from about 20 training tokens per parameter, growing the model and its data together.
▶ Try it
Why it matters
Training a big model costs a fortune in chips and electricity, so labs plan it in advance with scaling laws. The surprise: for a fixed budget, the biggest model is not the best one.
In 6 minutes you'll be able to
- Model quality depends on three ingredients that scale: data, parameters and compute.
- For a fixed budget, a giant model with too little data does worse than a balanced one.
- Scaling laws let labs forecast big training runs from small, cheap experiments.
What you'll practise
Three things that scale
Sort each line from Vireo's training plan into the ingredient it grows.
- 15 trillion tokens of text → Data
- Wider layers, and more of them → Parameters
- Thousands of chips running for months → Compute
- 70 billion weights → Parameters
- Adding books, code and more web pages → Data
- A record electricity bill → Compute
Spend the compute budget
Drag the slider to split the budget between model size and training data. Find the lowest loss, then lock it in.
Same budget, two ways to spend it: a bigger model, or more text to train it on. Slide to try different splits and watch the predicted loss. Lower is better.
- Budget 1×: The best split here: 1B parameters on 20B tokens.
- Budget 10×: The best split here: 3.2B parameters on 63B tokens.
- Budget 100×: The best split here: 10B parameters on 200B tokens.
≈ 20 training tokens per parameter Illustrative: a Chinchilla-style formula, not real measurements
Plan Vireo's next run
Same budget, two plans. Which model will probably come out better?
- A model a quarter the size, trained on 4× more text (Answer)
- The biggest model the budget allows, on whatever text is left
Vireo Labs trains a small model on far more than 20 tokens per parameter. Why might that be smart?
- Small models are cheaper to run for millions of users, so extra training pays off (Answer)
- It's a mistake, more than 20 tokens per parameter can't help
What do scaling laws let a lab do before spending months of compute?
- Predict roughly how good a bigger model will be from smaller test runs (Answer)
- Guarantee exactly which skills the model will have
Remember
For a fixed compute budget, grow the model and its data together. The classic sweet spot is about 20 training tokens per parameter.
For a fixed budget, grow model and data together: about 20 training tokens per parameter.
Key facts
- Language model loss falls smoothly and predictably, following power laws, as model size, data and compute grow. · Source: Kaplan et al., Scaling Laws for Neural Language Models, OpenAI, 2020
- For a fixed compute budget, the best results came from scaling model size and training tokens equally, about 20 tokens per parameter. · Source: Hoffmann et al., Training Compute-Optimal Large Language Models, DeepMind, 2022
- The 70-billion-parameter Chinchilla, trained on 1.4 trillion tokens, outperformed the 280-billion-parameter Gopher, which used the same compute budget. · Source: Hoffmann et al., DeepMind, 2022
- GPT-3, with 175 billion parameters, was trained on about 300 billion tokens: fewer than 2 tokens per parameter. · Source: Brown et al., Language Models are Few-Shot Learners, OpenAI, 2020
- Llama 3 was pretrained on over 15 trillion tokens, far more than 20 per parameter even for its largest model. · Source: Meta, The Llama 3 Herd of Models, 2024
Updated:
Questions people ask
What is the Chinchilla scaling law?
A 2022 finding that, for a fixed amount of training compute, model size and training data should grow together. The rule of thumb that came out of it is about 20 training tokens for every parameter.
Do bigger AI models always perform better?
Not for the same budget. A bigger model only helps if it also gets enough training data. A smaller model trained on more text can beat a much bigger, undertrained one.
What does compute mean in AI training?
The total number of calculations used to train a model. It is paid for in specialised chips, the time they run and the electricity they use.
Why do labs train small models on so much data?
Small models are cheaper and faster to run for every user. Training them on far more data than the compute-optimal amount squeezes out extra quality that pays off over millions of uses.
Related terms