How LLMs Are Made / Glossary

Data deduplication

Data deduplication means finding and removing repeated copies of the same text from a dataset, so a language model does not see the same passage again and again during training.

The web is full of copies: the same article on many sites, the same cookie notice on millions of pages, the same terms of service everywhere. Deduplication tools spot exact copies and near-copies and keep just one.

Repeated text wastes training time and teaches the model to memorise. Researchers found that deduplicating training data made language models repeat memorised text far less often and train more efficiently.

Deduplication is one step in a longer cleaning line that also strips page clutter, drops spam and low-quality text and removes private details such as email addresses and phone numbers.