PolicyLaw, regulation & governance
OpenAI, Google, Microsoft and Anthropic back a shared robots.txt token for opting out of AI training
A new IETF Internet-Draft from Anthropic's Kevin Kelley defines one robots.txt token, __usage_ai-training. Publishers can use it to opt out of AI training at four major AI companies at once. Anthropic already uses it, and OpenAI, Google and Microsoft say they will follow.
From the The Verge · AI article of October 30, 2026 and cited sources: Rollout is uneven. [1] [5] [6]
SOURCES[1] The biggest names in AI are backing a new way to opt out… theverge.com[5] PPC Land: Google, Microsoft, OpenAI and Anthropic back one… ppc.land[6] AI Weekly: OpenAI, Google, Microsoft, Anthropic Back One… aiweekly.co
AT Pulse research deskPublished · Updated · 5 min read · AI-assisted analysisAI-assisted, not human-edited. Methodology Corrections Who owns this site
- Original source
- The Verge · AI article, independent perspective
- Brief by
- AT Pulse research desk AI-assisted; publisher and contact on the desk page
- When
- October 30, 2026 article published
- Covered
- Oct 31 edition
- Evidence
- 9 sources across 8 sites
- 1The biggest names in AI are backing a new way to opt out…theverge.com
- 2IETF Datatracker: Machine-Readable Training Preference…datatracker.ietf.org
- 3IETF blog: Progress on AI Preferencesietf.org
- 4Search Engine Journal: Cloudflare lets sites disallow AI…searchenginejournal.com
- 5PPC Land: Google, Microsoft, OpenAI and Anthropic back one…ppc.land
- 6AI Weekly: OpenAI, Google, Microsoft, Anthropic Back One…aiweekly.co
QuietHow it’s measured
Hacker News · 4 likes
Reach blends public engagement (views, likes, comments, shares, stars) from the platforms where the story or its original source lives: YouTube, Hacker News, GitHub, Substack and similar. A video and the article written from its transcript share one score, taken from the video. Platforms with no data are left out rather than counted as zero, and the age of a story never lowers its score.
Rollout is uneven. Anthropic already honors the token. Microsoft plans to start on February 1, 2027, and Google is aiming for 2027. OpenAI says only that support will come in the coming months. Meta said it backs the idea but did not commit to honoring the token. [1] [5] [6]
The draft uses the existing robots.txt syntax (RFC 9309) and adds no new grammar. A blanket User-Agent: * group does not count as matching the token, so sites need to add an explicit group for it. Token matching ignores case, and more specific path rules win. The token covers only training, meaning changes to a model's learned parameters. It does not cover fetching, search indexing or ranking. [2] [5]
one token could replace separate rules for each company's crawler, but existing blanket blocks don't trigger it. Implication: publishers should add an explicit group and keep vendor tokens until every rollout is done.
Read the full assessment
Compliance stays voluntary, and payment terms are handled separately.
Status: writing the dossier from the IETF draft and reporting reviewed.
Executive brief
OpenAI, Google, Microsoft and Anthropic say they will honor one robots.txt token, __usage_ai-training. A publisher could use it to opt out of AI training at all four companies at once, instead of writing separate rules for each crawler (The Verge). The spec is an individual IETF Internet-Draft by Anthropic's Kevin Kelley, so it has no formal standing at the IETF (IETF Datatracker). It also relies on voluntary compliance. Only Anthropic has started using it so far. Meta voiced support but has not said it will honor the token.
Read the full section
OpenAI, Google, Microsoft and Anthropic say they will honor one robots.txt token, __usage_ai-training. A publisher could use it to opt out of AI training at all four companies at once, instead of writing separate rules for each crawler (The Verge). Two caveats matter. The spec is an individual IETF Internet-Draft by Anthropic's Kevin Kelley, so it has no formal standing at the IETF (IETF Datatracker). It also relies on voluntary compliance. Only Anthropic has started using it so far. Meta voiced support but has not said it will honor the token.
What changed and event timeline
IETF AIPREF sets the groundwork
The working group's co-chairs said it had adopted drafts for a shared preference vocabulary and for ways to attach it through robots.txt and HTTP headers. They stressed that preferences are not technically enforced ().
Cloudflare adds a no-training toggle
Cloudflare's "Disallow AI Training" setting writes vendor-specific no-training rules into robots.txt while search crawlers keep access ().
Kelley publishes the draft
"Machine-Readable Training Preference Signal" puts a policy token in robots.txt User-Agent lines. It is an individual submission and expires May 3, 2027 ().
Four companies commit and Meta hedges
Anthropic has already adopted the signal. OpenAI says it will follow in the "coming months," and Google and Microsoft plan support next year. Meta's Andy Stone backed the idea without committing to honor it ().
(scheduled): Microsoft support begins
Microsoft's Webmaster Tools transition opens Dec 15, 2026. Its NoArchive/NoCache controls are set to retire after Mar 1, 2027 ().
Capabilities and access
- Syntax:
User-Agent: __usage_ai-trainingfollowed byDisallow:and optionalAllow:paths (PPC Land). - Scope: the token covers using content to change a generative model's learned parameters, meaning training.
- Microsoft: offers separate controls for training and for grounding (pointing a model at a source at answer time), and says Bing search inclusion is unaffected (The Verge).
Read the full section
- Syntax:
User-Agent: __usage_ai-trainingfollowed byDisallow:and optionalAllow:paths (PPC Land). - Scope: the token covers using content to change a generative model's learned parameters, meaning training. It does not cover fetching, search indexing or ranking (PPC Land).
- Microsoft: offers separate controls for training and for grounding (pointing a model at a source at answer time), and says Bing search inclusion is unaffected (The Verge).
- Rollout: Anthropic is live; Microsoft starts Feb 1, 2027; Google targets 2027; OpenAI gives no date (AI Weekly).
Technical analysis for researchers and developers
- No new grammar.
- **
User-Agent: *does not count.** Token matching ignores case. More specific rules should win over general ones (IETF Datatracker). - A vendor token such as Google-Extended may be treated as more specific than the shared token. Private licensing deals can override stated preferences (PPC Land).
Read the full section
- No new grammar. The token reuses RFC 9309 syntax, and consumers must read ordinary RFC 9309 signals exactly as they would without the draft (IETF Datatracker).
- **
User-Agent: *does not count.** Wildcard groups must not be read as matching the token, so an existing blanket block does not opt a site out of training. Token matching ignores case. More specific rules should win over general ones (IETF Datatracker). - Interaction with vendor tokens. A vendor token such as Google-Extended may be treated as more specific than the shared token. Private licensing deals can override stated preferences (PPC Land).
- Link to the IETF working group. The draft borrows its definition of "training" from the working group's vocabulary draft (vocab-08). It is pitched as input to the group's attachment work, which had stalled (PPC Land).
Claims and evidence
- Four-company commitment: these are vendor statements reported by The Verge. No independent compliance testing has been published.
- Broad drafting input: the draft's acknowledgments thank contributors from Google, Microsoft, OpenAI, Cloudflare, the Internet Archive, publishers and civil-society groups (IETF Datatracker).
- Date conflict: The Next Web dates the draft Oct 25. Datatracker shows Oct 30.
Read the full section
- Four-company commitment: these are vendor statements reported by The Verge. No independent compliance testing has been published.
- Broad drafting input: the draft's acknowledgments thank contributors from Google, Microsoft, OpenAI, Cloudflare, the Internet Archive, publishers and civil-society groups (IETF Datatracker). Acknowledgments show who took part, not that they endorse it.
- Date conflict: The Next Web dates the draft Oct 25. Datatracker shows Oct 30.
Context and prior work
- Google introduced Google-Extended in 2023 to opt out of model training without affecting Search (TechCrunch).
- The IETF's own track has moved slowly. Its attachment draft was at -05 by August 2026 (IETF).
- Really Simple Licensing (RSL) lets publishers charge crawlers, and Cloudflare blocks AI crawlers by default (The Verge).
Read the full section
- Vendor-specific tokens came first. Google introduced Google-Extended in 2023 to opt out of model training without affecting Search (TechCrunch).
- The IETF's own track has moved slowly. AIPREF had aimed to send its drafts to the IESG (the IETF's approval body) in August 2025 (IETF blog). Its attachment draft was at -05 by August 2026 (IETF).
- Alternatives exist. Really Simple Licensing (RSL) lets publishers charge crawlers, and Cloudflare blocks AI crawlers by default (The Verge).
Limitations, safety and contested findings
Read the full section
- No enforcement. The draft has no enforcement mechanism. PPC Land cites TollBit data that about 15% of AI crawlers in Europe reached disallowed URLs anyway (PPC Land).
- Not retroactive. The token does not remove content already in training sets (AI Weekly).
- Uneven coverage of grounding. Google will treat the shared token as a Google-Extended disallow, which reportedly also covers Gemini grounding. The draft itself covers training only. Google has not said how it will translate path-level rules (PPC Land).
- Generated configs can fail. A later Cloudflare change reportedly stripped AI opt-outs from 118 sites' robots.txt files (PPC Land).
Business and practitioner implications
- Publishers: add an explicit
__usage_ai-traininggroup, because*blocks won't trigger it. - Crawler operators: parse the token case-insensitively and apply longest-match path rules (IETF Datatracker).
- Visibility trade-off: Microsoft warns that blocking grounding significantly reduces visibility in AI-generated experiences (The Next Web).
Read the full section
- Publishers: add an explicit
__usage_ai-traininggroup, because*blocks won't trigger it. Keep vendor tokens in place until every company's rollout is done, and check generated robots.txt files after CDN changes. - Crawler operators: parse the token case-insensitively and apply longest-match path rules (IETF Datatracker).
- Visibility trade-off: Microsoft warns that blocking grounding significantly reduces visibility in AI-generated experiences (The Next Web).
- Payment: the signal sets no licensing or payment terms, so RSL-style deals remain separate.
The source trail.
Sources (9)
The biggest names in AI are backing a new way to opt out of scraping
Article text retrieved; extracted text may omit tables or interactive elements.
theverge.com