How LLMs Are Made / Glossary

Gradient descent

Gradient descent is the method that trains most AI models: it works out which direction lowers the loss for every weight, then nudges all the weights a small step that way, again and again.

Picture standing on a foggy hillside and wanting to reach the valley. You feel which way the ground slopes and take a small step downhill. Gradient descent does the same with the loss, across billions of weights at once.

The slope for every weight is calculated by backpropagation, a method popularised in a 1986 paper in Nature. The size of each step is called the learning rate: too big and training becomes unstable, too small and it crawls.

Large language models are usually trained with Adam or a close variant, a refined form of gradient descent that adapts the step size for each weight. Training runs typically warm the learning rate up at the start and lower it over time.