How LLMs Are Made / Glossary

robots.txt

robots.txt is a small text file on a website that tells crawlers which pages they may and may not visit. Site owners can use it to ask AI crawlers not to collect their content.

The file sits at the top of a website, for example example.com/robots.txt. It lists crawlers by name and the parts of the site each one should stay out of. The idea dates back to 1994 and became an internet standard in 2022.

Many AI companies publish the names of their crawlers so sites can block them one by one, and researchers found that sites restricted AI crawlers far more often in 2023 and 2024 than before. In the EU, robots.txt is one machine-readable way for rights holders to reserve their works from text and data mining.

robots.txt is a request, not a lock: polite crawlers follow it, but nothing technically forces them to. Blocking a crawler also only stops future collection; it does not remove text from models that were already trained.