How do you make a large language model small enough for a phone?
Two main tricks shrink a language model. Quantization stores each of its billions of weights with fewer bits, for example 4 instead of 16, so the file gets several times smaller and faster with little loss in quality, until rounding goes too far. Distillation trains a small student model to copy a big teacher model's answers.
▶ Try it
Why it matters
The biggest models need racks of chips. Shrinking them is how AI runs on a laptop or a phone, cheaper, faster and even offline, and it explains why small models can still be good.
In 6 minutes you'll be able to
- A model's size is mostly its weights times the bits used to store each one.
- Down to about 4 bits, quality usually drops only a little; much lower and answers break.
- Small on-device models can work offline and keep your text on your device.
What you'll practise
Why shrink a model?
Tap each card.
- 📦 How big is a model?
- A model is billions of numbers called weights. At 16 bits each, a model with 7 billion weights fills about 14 GB.
- 📱 Why run it on a device?
- It works offline, answers without a trip to a server, and your text can stay on your device.
- ⚡ Why is smaller faster?
- For every token, the chip has to read the weights from memory. Fewer bytes to read means more tokens per second.
- 💸 Why do labs care?
- Serving millions of people costs chips and electricity. A smaller model that answers almost as well is much cheaper to run.
Shrink Vireo for a phone
Slide the precision down and watch the weights, the size and the quality. Ship when it fits and still works. Sizes, speeds and scores are illustrative.
Vireo · 7B
- 32-bit: 28 GB · 2 tok/s · Quality 100% · The capital of Australia is Canberra.
- 16-bit: 14 GB · 5 tok/s · Quality 100% · The capital of Australia is Canberra.
- 8-bit: 7 GB · 9 tok/s · Quality 99% · The capital of Australia is Canberra.
- 4-bit: 3.5 GB · 16 tok/s · Quality 96% · Australia's capital is Canberra.
- 2-bit: 1.75 GB · 21 tok/s · Quality 38% · The capital capital of Austral is is Sydberra berra
Vireo Mini · 1B
- 32-bit: 4 GB · 12 tok/s · Quality 89% · Canberra is the capital of Australia.
- 16-bit: 2 GB · 24 tok/s · Quality 89% · Canberra is the capital of Australia.
- 8-bit: 1 GB · 44 tok/s · Quality 88% · Canberra is the capital of Australia.
- 4-bit: 0.5 GB · 70 tok/s · Quality 85% · Canberra is Australia's capital.
- 2-bit: 0.25 GB · 84 tok/s · Quality 31% · Canberra the the of capital Aus Aus
Fit Vireo into the 4 GB the phone has free, with quality of 90% or more.
✓ 4 bits per weight: an eighth of the original size and still almost as good. That's why 4-bit models are popular for laptops and phones.
Even at 2 bits, the 7B model is 1.75 GB, and broken. How do you get a much smaller model that still answers well?
- ✓ Train a small student to copy the big model's answers: That's distillation. The student learns from the teacher's full odds for every word, not just right or wrong, so it learns a lot from little.
- ✗ Delete most of the big model's layers at random: Cutting layers at random breaks what the model learned. Distillation keeps the knowledge by teaching a smaller model on purpose.
- ✗ Train a small model from scratch on a tiny dataset: A small model trained alone on little data learns far less. Copying a big teacher's answers gives it a head start.
A cheaper phone has only 0.8 GB free. Ship the student with quality of 80% or more.
✓ Distillation, then quantization: a model that started at 28 GB now answers well in half a gigabyte. Labs combine both tricks all the time.
Which trick is it?
Is this quantization, distillation, or does it make a model bigger?
- Storing each weight with 4 bits instead of 16 → Quantization
- A 1B model trained on a 70B model's answers → Distillation
- Rounding 0.73914 to 0.74 for every weight → Quantization
- Adding twice as many layers → Makes it bigger
- Copying the teacher's odds for every next word → Distillation
- Training on ten times more text with more weights → Makes it bigger
Remember
Quantization rounds every weight to fewer bits; distillation teaches a small model to copy a big one. Both make AI fit on smaller devices.
Rounding weights to fewer bits shrinks a model a lot for little loss, until it goes too far. A student model can learn from a teacher.
Key facts
- Knowledge distillation, training a small model on the softened outputs of a large one, was popularised in a 2015 paper. · Source: Hinton, Vinyals and Dean, Distilling the Knowledge in a Neural Network, Google, 2015
- DistilBERT was 40% smaller and 60% faster than the BERT model it learned from, while keeping 97% of its language understanding. · Source: Sanh et al., DistilBERT, Hugging Face, 2019
- Storing weights in 8 bits let researchers run models with up to 175 billion parameters without a loss in performance. · Source: Dettmers et al., LLM.int8(), 2022
- With 4-bit weights, researchers fine-tuned a 65-billion-parameter model on a single 48 GB graphics card. · Source: Dettmers et al., QLoRA, 2023
- A model with 7 billion weights needs about 14 GB at 16 bits per weight, and about 3.5 GB at 4 bits.
Updated:
Questions people ask
What is quantization in LLMs?
Quantization stores a model's weights with fewer bits, for example 4 instead of 16. The model becomes several times smaller and faster, usually with only a small drop in quality.
What is model distillation?
Distillation trains a small student model to imitate a large teacher model's outputs, including how likely it finds each possible next word. The student ends up far better than a small model trained alone.
Can you run an LLM on a phone?
Yes. Small models with a few billion weights or fewer, often quantized to 4 bits, can run on modern phones and laptops, even without an internet connection.
Does quantization make a model worse?
A little. Down to about 8 or 4 bits the loss is usually small, but at very low precision such as 2 bits the rounding errors pile up and answers can break.
Related terms