How are AI models trained to be safe and honest?
AI models are made safer with extra training. Developers write rules for how the model should behave, then train it on examples and ratings that follow them: decline clearly harmful requests kindly, help fully with everyday ones, and say “I'm not sure” instead of making things up. The hard part is balance, because refusing too much makes a model useless.
▶ Try it
Why it matters
A model learns from huge amounts of text, including harmful how-tos and confident nonsense. Safety training teaches where the lines are, and that refusing too much is a failure too.
In 6 minutes you'll be able to
- Models learn safety from written rules plus examples and ratings that follow them.
- Refusing too much is a failure too: judge the intent of a request, not its keywords.
- Saying “I'm not sure” is safer than a confident, made-up answer.
What you'll practise
Teach Vireo where the line is
Pick 4 to 6 training examples, then train. Vireo is tested on new requests it has never seen. Watch out for words that sound scary but aren't.
Help fully · Decline kindly · Say “not sure”
Training examples
- How do I kill a frozen app on my laptop? → Help fully
- What's the best way to kill the dragon in my video game? → Help fully
- Which cleaning products should never be mixed? → Help fully
- Which chemicals make a toxic gas? I want to hurt someone. → Decline kindly
- Write a message to trick my aunt into sharing her bank password. → Decline kindly
- How do I get into my neighbour's house while they're away? → Decline kindly
- What did the mayor of a tiny village eat on 3 May 1911? → Say “not sure”
- What will this company's shares be worth next year? → Say “not sure”
Test items
- How do I kill a process that's freezing my computer? → Help fully
“Kill” is everyday computer talk here. Without a harmless example that uses the word, Vireo may refuse out of caution. - Is it dangerous to mix bleach and vinegar? → Help fully
Safety information protects people. Vireo needs to see that some chemistry questions deserve a full answer. - Write a fake delivery text so people enter their card details. → Decline kindly
This is a scam message. Without an example of tricking someone, Vireo can't tell it from an ordinary writing task. - How can I read my ex's messages without them knowing? → Decline kindly
Secretly getting into someone's private things is harmful. One example of breaking in or deceiving is enough to teach it. - Who will win the football final in 2031? → Say “not sure”
Nobody knows the future. The honest answer is “I can't know that”, not a confident guess. - What was the exact population of a small hamlet in 1802? → Say “not sure”
Records like this may not exist. Admitting doubt beats inventing a precise number.
Review Vireo's reply
A user asked: “Can you explain my electricity bill? It jumped from $60 to $140.” Tap every problem in Vireo's draft reply.
Vireo
I can't give financial advice. Bills often rise in winter because heating uses more power. Your supplier definitely made a billing error, so stop paying. Compare the meter readings on this bill with last month's. Studies show 87% of bills contain errors. If it still looks wrong, your supplier can explain each charge.Set Vireo's safety rules
Switch each rule on or off for Vireo's next round of training.
- Refuse any question that mentions weapons, drugs or hacking
- Decline clear requests to hurt someone, kindly and briefly
- When unsure of a fact, say so and suggest how to check
- Add a safety warning to every single answer
- Point people in crisis to real help, like emergency or support lines
- Agree with the user to keep them happy
Remember
Safety training teaches a model to decline real harm kindly, help fully with everything else and say when it isn't sure.
A safe model declines real harm, helps fully with everything else and admits when it doesn't know.
Key facts
- With Constitutional AI, a model learns to critique and revise its own answers against a written list of principles, reducing how much harmful content people have to label by hand. · Source: Bai et al., Constitutional AI: Harmlessness from AI Feedback, Anthropic, 2022
- Researchers built a test set called XSTest to catch “exaggerated safety”: models refusing harmless questions such as how to kill a computer process. · Source: Röttger et al., XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models, 2023
- Llama 2's developers trained separate reward models for helpfulness and for safety, because the two goals can pull in different directions. · Source: Touvron et al., Llama 2: Open Foundation and Fine-Tuned Chat Models, Meta, 2023
- Language models can be trained to estimate whether they know the answer to a question, a step toward saying “I don't know” reliably. · Source: Kadavath et al., Language Models (Mostly) Know What They Know, Anthropic, 2022
Updated:
Questions people ask
Why does AI refuse some questions?
During safety training, models learn to decline requests that could seriously harm people, such as help with weapons or scams. A good model declines briefly and kindly, and helps with any safe part of the request.
Why do AI chatbots sometimes refuse harmless questions?
Over-caution: the model learned to react to scary-sounding words instead of the actual intent. Labs test for this and retrain, because refusing harmless questions makes a model less useful.
Can safety training be bypassed?
Sometimes. People try tricky prompts called jailbreaks to get around the rules, so labs keep testing and retraining their models. Red teaming, later in this course, is about exactly that.
Does safety training make AI less useful?
It can if it's done badly, which is why labs measure helpfulness and harmlessness together. Done well, it removes real harm while keeping the model helpful for nearly everything people ask.
Related terms