PolicyLaw, regulation & governance
Meta's new commercial terms let it crawl business-linked websites, including news publishers' sites, to train its products
From October 30, businesses using Meta's platforms commercially consent, by continued use, to Meta crawling their associated websites and using that content to train Meta products. Legal experts say publishers' robots.txt blocks may not protect them; Meta says its crawling is unchanged.
From the The Verge · AI article of October 16, 2026 and cited sources: The revised Commercial Terms authorize scrapers, bots and crawlers. [1] [7] [8]
SOURCES[1] Meta’s new terms may allow it to train AI on news sites… theverge.com[7] PPC Land: Meta's new terms let it crawl business websites… ppc.land[8] Gizmodo: Meta to Literally Every Business User: Nice… gizmodo.com
AT Pulse research deskPublished · Updated · 5 min read · AI-assisted analysisAI-assisted, not human-edited. Methodology Corrections Who owns this site
- Original source
- The Verge · AI article, independent perspective
- Brief by
- AT Pulse research desk AI-assisted; publisher and contact on the desk page
- When
- October 16, 2026 article published
- Covered
- Oct 17 edition
- Evidence
- 11 sources across 11 sites
- 1Meta’s new terms may allow it to train AI on news sites…theverge.com
- 2Nieman Lab: Meta's new commercial terms will allow the…niemanlab.org
- 3Meta Commercial Terms (current)facebook.com
- 4Press Gazette: People Inc signs AI licensing deal with Metapressgazette.co.uk
- 5ProPakistani: Meta One brings a new subscriptionpropakistani.pk
- 6Metricool: Meta One subscriptionmetricool.com
Reach: not measured yet
The revised Commercial Terms authorize scrapers, bots and crawlers. They also let Meta create test credentials to access domains associated with a business and to re-pull their content periodically. Commercial use is defined broadly: it covers advertising, selling products, building apps and managing a business Page. Businesses consent by continuing to use Facebook, Instagram, WhatsApp or Messenger, and the only way to refuse is to stop using them commercially. [1] [7] [8]
Three lawyers told Nieman Lab the wording leaves Meta room to argue that the contract overrides robots.txt blocks. One, Cornell's James Grimmelmann, says it permits AI training on any content from publishers' sites. Meta replies that the update merges its commercial and self-serve ad terms, is about product information for ads, and leaves robots.txt in force. That assurance comes from Meta's statements and documentation, not from the contract text, which never mentions robots.txt. [2] [1] [7] [8]
Meta's crawler documentation lists separate user agents. Meta-ExternalAgent (foundation model training), Meta-WebIndexer (Meta AI search) and Meta-ExternalAds (advertising) honor robots.txt. FacebookExternalHit (link previews) and Meta-ExternalFetcher (user-initiated fetches) may bypass it. The new terms don't tie any purpose to a particular agent, so blocking only the training crawler leaves the others free to crawl. [9] [7]
The clause lands as Meta pays some publishers for AI content. In December 2025 it signed multi-year licensing deals with People Inc, CNN, Fox News, USA Today, Le Monde and others, with undisclosed terms. Its new Meta One plans charge businesses to post clickable links in organic posts and Reels, so publishers would pay to link to content the terms let Meta crawl for free. [4] [10] [5] [6] [1]
What the evidence shows: the contract text is broad and silent on robots.txt. The implication, if the lawyers' reading holds, is that publishers could lose leverage in AI licensing, a market where Meta already pays some outlets for content.
Sources are gathered and the dossier follows.
Executive brief
Starting October 30, 2026, any business that uses Meta's platforms commercially agrees, just by continuing to use them, to let Meta crawl websites "associated" with its products and use that content to "train" Meta products. That group includes news publishers with Pages or ad accounts. Three legal experts told Nieman Lab that the wording could let Meta argue that the contract overrides robots.txt blocks (Nieman Lab). Meta says nothing changes about how it crawls and that the clause is about product information for ads (The Verge). Businesses that refuse can only stop using Meta for commercial purposes.
What changed and event timeline
Current terms take effect
The Commercial Terms in force today carry this effective date and have no clause on crawling, scraping or AI training ().
Meta pays publishers for AI content
Meta signed multi-year AI licensing deals with People Inc, CNN, Fox News, USA Today, Le Monde and others. Financial terms were not disclosed ().
Meta One launches
Paid business and creator plans run from $14.99 to $499 a month. Clickable links in organic posts and Reels start at the $49.99 tier (;).
Nieman Lab reports the clause
Three lawyers say the new terms could permit scraping news sites to train AI. Meta had told business users about the update in early October ().
Meta pushes back
Spokesperson Daniel Roberts calls claims of any crawler change "false." Communications VP Andy Stone says the clause is about product information, "not about scraping news sites" ().
Terms take effect
Commercial users of Facebook, Instagram, WhatsApp and Messenger consent by continuing to use them ().
Capabilities and access
- This is a contract change, not a model release. No specific Meta model or version is named.
- The clause authorizes "web scrapers, bots, robots, spiders, crawlers, and user agents."
- "Commercial use" is defined broadly. It covers advertising, selling products, building apps, managing a business Page and using Meta Business Suite (Gizmodo).
Read the full section
- This is a contract change, not a model release. No specific Meta model or version is named.
- The clause authorizes "web scrapers, bots, robots, spiders, crawlers, and user agents." It also lets Meta create "test credentials" to access and periodically re-pull content from associated domains (The Verge).
- "Commercial use" is defined broadly. It covers advertising, selling products, building apps, managing a business Page and using Meta Business Suite (Gizmodo).
Technical analysis for researchers and developers
- Meta's crawler documentation lists separate user agents. Meta-ExternalAgent is used for "training foundation AI models." Meta-WebIndexer serves Meta AI search, and Meta-ExternalAds serves advertising.
- The new contract never mentions robots.txt and does not tie any purpose to a particular user agent (PPC Land).
- "Test credentials" raise the possibility of getting past logins or paywalls (Gizmodo).
Read the full section
- Meta's crawler documentation lists separate user agents. Meta-ExternalAgent is used for "training foundation AI models." Meta-WebIndexer serves Meta AI search, and Meta-ExternalAds serves advertising. All three honor robots.txt. FacebookExternalHit (link previews) and Meta-ExternalFetcher (user-initiated fetches) may bypass it (Meta for Developers).
- The new contract never mentions robots.txt and does not tie any purpose to a particular user agent (PPC Land).
- "Test credentials" raise the possibility of getting past logins or paywalls (Gizmodo).
- Practical step: block each agent separately. Blocking only Meta-ExternalAgent leaves the others free to crawl.
Claims and evidence
- Legal reading: Cornell professor James Grimmelmann says the clause "clearly" permits AI training on any content from publishers' sites. These are interpretations.
- Meta's position: this "consolidates" its commercial and self-serve ad terms, and robots.txt still applies (The Verge).
- No independent evidence has been reported of Meta crawling news sites under the new clause.
Read the full section
- Legal reading: Cornell professor James Grimmelmann says the clause "clearly" permits AI training on any content from publishers' sites. Attorney Matt Topic says it gives access to material that could cost "many millions" in a licensing market (Nieman Lab; Gizmodo). These are interpretations. No court has ruled.
- Meta's position: this "consolidates" its commercial and self-serve ad terms, and robots.txt still applies (The Verge). That assurance comes from Meta's statements and documentation, not the contract.
- No independent evidence has been reported of Meta crawling news sites under the new clause.
Context and prior work
- Meta stopped paying publishers for news in 2022, shut Facebook's News tab in 2024, then began paying for AI content in 2025 (Reuters via StreetInsider).
- Meta-ExternalAgent first drew attention in August 2024 as a quietly launched crawler for collecting AI training data (Fortune).
- People Inc already had similar licensing deals with OpenAI and Microsoft (Press Gazette).
Read the full section
- Meta stopped paying publishers for news in 2022, shut Facebook's News tab in 2024, then began paying for AI content in 2025 (Reuters via StreetInsider).
- Meta-ExternalAgent first drew attention in August 2024 as a quietly launched crawler for collecting AI training data (Fortune).
- People Inc already had similar licensing deals with OpenAI and Microsoft (Press Gazette).
Limitations, safety and contested findings
- The central dispute is unresolved: does the contract override robots.txt? Meta says it does not.
- Law professor Eric Goldman calls the consent model a Hobson's choice (Gizmodo).
- Nieman Lab points out that Meta One charges users to post links to the same content Meta will be allowed to crawl for free (The Verge).
Read the full section
- The central dispute is unresolved: does the contract override robots.txt? Meta says it does not. The lawyers say the text leaves Meta room to argue that it does (Nieman Lab).
- Law professor Eric Goldman calls the consent model a Hobson's choice (Gizmodo).
- Nieman Lab points out that Meta One charges users to post links to the same content Meta will be allowed to crawl for free (The Verge).
Business and practitioner implications
- Publishers: check how the terms interact with existing licensing contracts, and get legal advice before October 30.
- Web and infra teams: block each Meta user agent explicitly in robots.txt and keep server-side logs.
- AI builders: using platform terms of service to get training data, instead of licensing it, is becoming a regulatory and reputational risk.
Read the full section
- Publishers: check how the terms interact with existing licensing contracts, and get legal advice before October 30.
- Web and infra teams: block each Meta user agent explicitly in robots.txt and keep server-side logs. Robots.txt is a stated policy, not a contractual guarantee (Meta for Developers).
- AI builders: using platform terms of service to get training data, instead of licensing it, is becoming a regulatory and reputational risk.
- Advertisers: an e-commerce product feed falls under the same clause as editorial content.
The source trail.
Sources (11)
Meta’s new terms may allow it to train AI on news sites for free
Article text retrieved; extracted text may omit tables or interactive elements.
theverge.com