How LLMs Are Made / Glossary

Parameters (model weights)

Parameters, also called weights, are the adjustable numbers inside an AI model. Training tunes them so the model's predictions get better, and together they store everything the model has learned.

A language model is a huge calculation. Text goes in as numbers, gets multiplied and added with the model's parameters, and a prediction comes out. Before training the parameters are random; training nudges each one until the predictions are good.

Model size is usually counted in parameters. GPT-3 had 175 billion of them, and many models you can run on a laptop have a few billion. More parameters can store more patterns, but they also need more memory, more computing power and more training data.

Parameters take up real space: at 16 bits each, a model with 7 billion weights needs about 14 GB just to load. That is why tricks like quantization, which stores each weight with fewer bits, matter for running models on phones and laptops.