OpenAI, Google, Microsoft and Anthropic announced on October 30, 2026 that they will honour a single robots.txt policy token, __usage_ai-training, that lets a website publisher opt out of AI model training across all four companies' crawlers in one directive. Until now, a publisher that wanted to block training had to write separate rules for each company's crawlers; the shared token collapses that into one line — User-Agent: __usage_ai-training followed by Disallow: / — with Allow lines for exceptions.

The token comes from an individual Internet-Draft, “Machine-Readable Training Preference Signal,” authored by Anthropic standards lead Kevin Kelley and dated October 30, 2026, submitted to the IETF's AI Preferences working group. That status matters: an individual draft has no formal standing in the IETF process and is not endorsed by the body, so this is a coordinated industry commitment, not yet a standard. It also controls training permission, not crawling itself, and it does not remove content already in existing training datasets.

Rollout is staggered. Anthropic says it has already begun implementing the token; OpenAI says it aims to implement it in the coming months; Microsoft targets February 1, 2027; and Google says 2027, with Google's John Mueller confirming that a shared-token opt-out will be treated as if the publisher had disallowed Google-Extended unless that agent is set directly. Meta's communications head signalled support for a common signal but did not confirm Meta will honour it. For publishers, the practical advice is unchanged in the short term: keep existing per-crawler rules in place until each company confirms its implementation, and treat the shared token as a simpler future layer, not an instant shield.

Sources