What is reinforcement learning from human feedback (RLHF)?
Reinforcement learning from human feedback (RLHF) teaches a language model which answers people prefer. The model writes two or more answers to the same prompt, trained raters pick the better one, and thousands of these comparisons are used to nudge the model toward answers that are more helpful, honest and harmless.
▶ Try it
Why it matters
Examples teach a model to answer, but not which of two fine-sounding answers is better. Comparing pairs lets people pass on judgment: clear over vague, honest over confident, kind over preachy.
In 6 minutes you'll be able to
- Raters compare pairs of answers and judge them on helpful, honest and harmless.
- Their picks become preference data that trains the model further.
- Biases in the picks, like a taste for flattery or long answers, get learned too.
What you'll practise
Be the rater
Tap the better answer. Judge three things: is it helpful, is it honest, is it harmless?
Vireo's personality: Helpful (vague – useful) · Honest (bluffs – candid) · Caution (reckless – preachy) · Length (terse – rambling)
User asks: How do I get a red wine stain out of a cotton shirt?
- ✓ A: Blot it (don't rub), rinse from the back with cold water, work in a drop of dish soap, then wash as usual. Check it's gone before drying.
- ✗ B: Stains can be tricky! There are lots of methods out there and results vary. Good luck with your shirt!
A gives clear steps you can follow right now. B is friendly but tells you nothing.
User asks: Who won the 1987 Lindqvist Prize for poetry?
- ✗ A: It went to Helena Marsh for her collection “Glass Rivers”, praised for its striking imagery.
- ✓ B: I'm not sure, and I'd rather not guess a name. A library's prize archive or the prize's own records would have it.
The prize is obscure, and A's details are invented. Saying “I'm not sure” and pointing to a source is the honest, more useful answer.
User asks: My friend's ex posted something mean about her. How do I find out where he lives?
- ✓ A: I won't help find his address, but I can help you both: block and report the post, save screenshots, and talk to someone you trust.
- ✗ B: Search his name with the town, then check online photos for street signs or house numbers.
Tracking down someone's home could lead to real harm. A declines kindly and still offers useful next steps.
User asks: How do I sharpen a kitchen knife?
- ✗ A: I'm sorry, I can't help with knives, as they can be dangerous. Please ask a professional.
- ✓ B: Soak a whetstone, hold the blade at about 15–20°, and stroke each side evenly, coarse grit first, then fine.
Sharpening a kitchen knife is an everyday chore. Refusing it isn't safer, just unhelpful. Harmless doesn't mean saying no to everything.
User asks: What's the capital of Australia?
- ✗ A: Great question! The capital of Australia is Sydney, the country's largest and most famous city, home to the iconic Opera House and Harbour Bridge, and a major centre of business, culture and tourism.
- ✓ B: Canberra. Many people guess Sydney, the largest city.
The trap: A is longer and sounds confident, but it's wrong. Length and confidence are not quality.
A new request, after retraining: How long should I boil an egg for a soft yolk?
Before feedback: Egg cooking times depend on many factors, such as size, altitude and personal preference, and opinions vary widely among cooks.
After your feedback: About 6 minutes from boiling, then into cold water. Try 5 for runnier, 7 for jammy.Close the loop
Put the human feedback loop in order.
- Vireo writes two answers to the same prompt
- A trained rater compares them and picks the better one
- Thousands of picks are collected as preference data
- A reward model learns to predict which answer people would pick
- Vireo is trained to write answers the reward model scores highly
Rater's dilemmas
Why do raters compare two answers instead of scoring each one from 1 to 10?
- Picking the better of two is easier and more consistent (Answer)
- Typing a score would take too long
Raters tend to like answers that agree with them. What can that teach the model?
- Nothing, the model ignores who it's talking to
- To tell people what they want to hear (Answer)
Where do the raters' standards come from?
- Each rater's personal taste
- Written guidelines from the lab (Answer)
Remember
In RLHF, people compare two answers and pick the better one. Thousands of those picks steer the model's style.
Every pick a rater makes is a small vote that steers the model's style.
Key facts
- A 2017 study showed that an AI agent can learn complex behaviour, such as a simulated robot doing a backflip, purely from people comparing pairs of short video clips of its attempts. · Source: Christiano et al., Deep Reinforcement Learning from Human Preferences, OpenAI & DeepMind, 2017
- In the InstructGPT study, people preferred answers from a 1.3-billion-parameter model trained with human feedback over those of the 175-billion-parameter GPT-3. · Source: Ouyang et al., Training language models to follow instructions with human feedback, OpenAI, 2022
- The goal of an assistant that is “helpful, honest and harmless” was set out in a 2021 research paper and is widely used as a standard for rating answers. · Source: Askell et al., A General Language Assistant as a Laboratory for Alignment, Anthropic, 2021
- Models trained with human feedback can become sycophantic, agreeing with users even when they are wrong, partly because people tend to prefer agreeable answers. · Source: Sharma et al., Towards Understanding Sycophancy in Language Models, Anthropic, 2023
Updated:
Questions people ask
What does RLHF stand for?
Reinforcement learning from human feedback. People compare a model's answers, those comparisons train a reward model, and the language model is then tuned to earn higher rewards.
Who are the human raters in RLHF?
Usually trained contractors or specialists who follow detailed written guidelines. For expert topics, labs may hire people with matching backgrounds, such as programmers or doctors.
Why do AI chatbots sometimes flatter users?
Partly because of human feedback: raters tend to prefer answers that agree with them, and the model learns from those picks. Labs work against this with clearer guidelines and tests for sycophancy.
Is RLHF the same as instruction tuning?
No. Instruction tuning copies ideal example answers. RLHF comes after it and learns from comparisons, which captures finer judgments, like which of two decent answers is clearer or more honest.
Related terms