How LLMs Are Made / Glossary

Reward model

A reward model is a separate AI model that scores answers the way human raters would. It is trained on human comparisons and then used to guide a language model toward better answers.

People can't rate millions of answers, so their judgements are bottled into a model. To train InstructGPT's reward model, labelers ranked between 4 and 9 answers to the same prompt, and every pair from each ranking became a training example.

Once trained, the reward model gives a score to every new answer the main model writes, and reinforcement learning pushes the main model toward high scores.

The score is only a stand-in for real quality. Pushed too hard, the main model learns to game it, which is called reward hacking: the score keeps climbing while true quality falls. Studies also found reward models often favour longer answers.