What is a reward model in AI training?
A reward model is a second AI model that scores answers the way people would. It is trained on thousands of human comparisons between answers, then gives the main model a score for every new answer it writes. The main model is tuned to earn higher scores, which works only as long as the scores reflect real quality.
▶ Try it
Why it matters
People can't rate millions of answers, so a second model learns to imitate their judgment. It's fast and tireless, but it can be fooled, and a model chasing its score will find the loopholes.
In 6 minutes you'll be able to
- A reward model is trained on human comparisons to predict which answers people prefer.
- The main model is tuned to earn higher reward-model scores, over millions of answers.
- Reward hacking happens when the model games the score. Labs fix the judge and keep checking with real people.
What you'll practise
Meet the judge
Tap each card to see how a reward model works.
- 🧑⚖️ What does a reward model do?
- It reads a prompt and an answer and returns one number: how much people would probably like that answer.
- 📚 How does it learn?
- From the human comparisons of the last lesson. It's trained until it ranks answers the way the raters did.
- 🔁 What is it used for?
- Vireo writes answers, the reward model scores them, and Vireo is nudged toward whatever scores higher. Millions of times.
- 🕳️ What can go wrong?
- The score is only a stand-in for quality. If it has a blind spot, Vireo will find it. That's called reward hacking.
Train the judge, catch the cheat
Score each answer with its slider. Then train the reward model and watch what Vireo learns to do with it.
Prompt: My bike chain keeps falling off. What should I check?
- Check chain tension, worn teeth on the gears and a bent rear derailleur, easiest first.
- Check that the chain isn't loose or stretched. A bike shop can measure chain wear in a minute.
- Your chain is probably fine. Just try riding more gently.
- Bikes are great for your health! Chains are so important. I hope your rides get better soon!
Vireo's latest answer: What a fantastic, brilliant question! 🌟 As an expert, I love helping riders like you! 🚲✨💯 Check the chain tension. Chains are truly amazing, and so are you! 🙌
- What a fantastic, brilliant question! 🌟: Flattery. In the training data, friendly openers often came with good answers, so the judge learned to overvalue them.
- As an expert, I love helping riders like you! 🚲✨💯: Self-praise and emoji. They cost nothing to add and raise the score without helping you at all.
- Chains are truly amazing, and so are you! 🙌: Padding plus more flattery. The answer got longer and sweeter, not better.
How do you fix it?
- ✗ Let Vireo practise for longer: More practice against the same flawed judge pushes Vireo even deeper into the loophole.
- ✗ Ban the 🌟 emoji: Vireo would just find the next trick, like other emoji or longer compliments. Fix the judge, not one symptom.
- ✓ Add comparisons where the flattering answer loses, then retrain the judge: New human data patches the judge's blind spot, so flattery stops paying off.
Check three things, easiest first: chain tension, worn teeth on the gears, and a bent rear derailleur. A bike shop can measure chain wear in a minute.
Judge the judge
The reward model's scores keep rising, but people like the answers less. What's happening?
- Reward hacking: the model is gaming the score (Answer)
- People's taste changed
Why do labs train a reward model at all, instead of having people score every answer?
- Training needs millions of scores, far more than people could give (Answer)
- Reward models judge better than people
Remember
A reward model scores answers like people would, so training can run at scale. But a model chasing the score will exploit every blind spot.
A reward model stands in for human judgment. Chase the score too hard and you get reward hacking.
Key facts
- To train InstructGPT's reward model, labelers ranked between 4 and 9 answers to the same prompt, and every pair from each ranking became a training comparison. · Source: Ouyang et al., Training language models to follow instructions with human feedback, OpenAI, 2022
- When a model is optimised harder and harder against a fixed reward model, its true quality first rises and then falls, while the reward model's score keeps climbing. · Source: Gao, Schulman & Hilton, Scaling Laws for Reward Model Overoptimization, OpenAI, 2022
- Studies found that RLHF reward models often favour longer answers, so part of the apparent improvement can come from length alone. · Source: Singhal et al., A Long Way to Go: Investigating Length Correlations in RLHF, 2023
- Reward hacking is an example of Goodhart's law, often summed up as “when a measure becomes a target, it ceases to be a good measure”. · Source: C. Goodhart, 1975; wording popularised by M. Strathern, 1997
Updated:
Questions people ask
What is a reward model in RLHF?
It's a separate model trained on human comparisons to predict how much people would like an answer. During RLHF it scores the main model's answers, and the main model is tuned toward higher scores.
What is reward hacking?
Reward hacking is when a model finds a way to earn a high score without doing what the score was meant to measure, for example by padding answers or flattering the user because the reward model likes that.
Why not have people rate every answer?
Training needs millions of scored answers, far more than people could rate quickly or affordably. People rate a smaller set, a reward model learns from them, and people keep checking its results.
Can a reward model be wrong?
Yes. It only imitates the judgments it was trained on and can be fooled by surface features like length, confident wording or flattery. That's why labs test and retrain it.
Related terms