How LLMs Are Made / Glossary

Quantization

Quantization makes an AI model smaller and faster by storing each of its weights with fewer bits, for example 4 instead of 16, accepting a small amount of rounding error.

Each weight is a number, and storing it precisely takes space. Quantization rounds the weights to a smaller set of values. A model with 7 billion weights needs about 14 GB at 16 bits per weight, and about 3.5 GB at 4 bits.

Done carefully, it costs surprisingly little quality: researchers ran models with up to 175 billion parameters with 8-bit weights without a loss in performance. Rounding too far, to very few bits, does start to hurt.

Quantization is a main reason capable models now run on laptops and phones. It is often combined with distillation, which trains a smaller model to copy a bigger one.