How LLMs Are Made / Glossary

RLHF (reinforcement learning from human feedback)

RLHF, reinforcement learning from human feedback, is a training method that teaches a model which answers people prefer, using many human comparisons between pairs of answers.

The model writes two or more answers to the same prompt, and trained raters pick the better one. These comparisons train a reward model that scores answers the way the raters would, and the main model is then tuned to earn higher scores.

In the InstructGPT study, people preferred answers from a 1.3-billion-parameter model trained this way over those of the 175-billion-parameter GPT-3. The aim is often summed up as an assistant that is helpful, honest and harmless.

RLHF has side effects. Because people tend to like agreeable answers, models trained with human feedback can become sycophantic and agree with users even when they are wrong. Newer methods learn from the same comparisons more directly or use AI feedback alongside human ratings.