How LLMs Are Made / Glossary

Web crawler

A web crawler is a program that visits web pages automatically, saves their content and follows their links to find more pages. Crawlers collect much of the text used to train language models.

A crawler starts from a list of addresses, downloads each page, picks out the links and adds them to its to-do list. Repeated millions of times, this maps and copies a large part of the public web. Search engines have used crawlers since the 1990s.

For AI, crawled pages are the biggest single source of training text. Common Crawl, a non-profit, has archived billions of web pages since 2008 and shares them for free, and filtered versions of it feed many language models.

Well-behaved crawlers name themselves and follow the rules a site sets in its robots.txt file. Raw crawled text is messy, so it is cleaned, filtered and deduplicated before training, and most of it is thrown away.