How LLMs Are Made / Glossary

Red teaming

Red teaming is deliberately attacking your own AI system before release, trying to make it misbehave so its weak spots can be found and fixed before real users meet them.

The name comes from military and security exercises, where a red team plays the attacker. AI red teamers write tricky prompts: role-play tricks, harmful requests split into harmless-looking pieces, or instructions hidden in documents the model reads.

Each weak spot is rated by how likely and how harmful it is, then fixed with training or filters and tested again. Language models can red-team each other too, with one model generating test prompts that find failures in another.

Red teaming is becoming a duty, not just good practice: under the EU AI Act, providers of the most capable general-purpose AI models must carry out and document adversarial testing.