2026-10-31 / Signal #4
The Biggest AI Labs Just Agreed on One robots.txt Line to Opt Out of Training
“Four AI labs just invented a single line of robots.txt that tells all of them at once: don’t train on this.”
The Story
OpenAI, Google, Microsoft, and Anthropic said they will honor a single new robots.txt policy token (usage_ai-training) so a site can block AI training crawlers in one step instead of listing every bot. Anthropic has started implementing; the others plan support in coming months or 2027. It does not affect already-trained data or non-training uses.
Why It Matters
Internet culture, institutions, and money—the old robots.txt protocol getting a new sacred object for the AI era. Concrete, usable artifact for publishers and builders; weird that the labs themselves are shipping the opt-out.
Evidence
The Verge, company statements, IETF draft. Primary URL: https://www.theverge.com/tech/1023763/openai-google-microsoft-ai-scraping-standard-opt-out
Sources
Daily scan: 2026-10-31