How LLMs Are Made / Glossary

Knowledge distillation

Knowledge distillation trains a small “student” model to copy the outputs of a large “teacher” model, so the student keeps much of the teacher's skill at a fraction of the size.

The student does not just learn the teacher's top answer. It learns the teacher's whole spread of odds, such as “probably cat, a bit like dog, not car”, which carries more information than a single right answer.

The idea was popularised in a 2015 paper. A well-known example, DistilBERT, was 40% smaller and 60% faster than the model it learned from while keeping 97% of its language understanding.

Many small and fast language models today are trained partly on answers written by larger models. Distillation is often combined with quantization to fit models onto phones and laptops.