OpenAI, Google, Microsoft, and Anthropic Unite Behind a Universal AI Scraping Opt-Out Standard
OpenAI, Google, Microsoft, and Anthropic are backing a new IETF-proposed 'policy token' standard that lets publishers opt out of AI training data scraping via a single robots.txt signal. The initiative aims to replace the fragmented, crawler-by-crawler opt-out system that has left many publishers — including those in India — with little practical control over how their content is used.
The Web’s Consent Problem Just Got a Proposed Solution
For years, the relationship between AI companies and the publishers whose content they consume has been one of the most contentious fault lines in the technology industry. Millions of websites, news outlets, blogs, academic repositories, and creative platforms have watched their carefully produced content get hoovered up by AI training pipelines — often with little recourse beyond individually blocking each crawler one at a time. That dynamic may be about to shift in a meaningful way.
As reported by The Verge (https://www.theverge.com/tech/1023763/openai-google-microsoft-ai-scraping-standard-opt-out), four of the most powerful names in artificial intelligence — OpenAI, Google, Microsoft, and Anthropic — have announced they will adopt a new shared standard that gives publishers a single, unified mechanism to opt out of AI data scraping. Instead of playing a never-ending game of whack-a-mole with individual crawlers, website owners will be able to express their data-use preferences once, in a standardised way, and have those preferences respected across all participating AI companies simultaneously.
What Is the ‘Policy Token’ Signal?
At the heart of this initiative is something called a shared “policy token” signal. The concept was proposed by Kevin Kelley, a technical lead for standards at Anthropic, and submitted through the Internet Engineering Task Force (IETF) — the venerable standards body responsible for much of the foundational architecture that keeps the internet functional and interoperable.
The policy token would live inside a website’s robots.txt file. If you are not familiar with robots.txt, it is a plain-text file that has existed since the earliest days of the web and sits at the root of virtually every website. Historically, it has been the primary way site owners communicate with automated bots — telling search engine crawlers which pages to index, which to ignore, and which archives are off-limits. The file is a kind of digital etiquette agreement: bots that respect it behave as good citizens of the web; those that don’t are considered bad actors.
The new proposal extends this long-standing convention into the AI era. By embedding a policy token in robots.txt, a publisher can now declare their preferences about AI training data use in one location, and participating AI companies — those that have pledged to adopt the standard — will read and honour that signal without the publisher needing to maintain separate entries for each company’s crawler.
Why This Matters: The Scale of the Problem
To appreciate the significance of this development, it helps to understand just how fractured the current opt-out landscape is. Until now, if a publisher wanted to prevent OpenAI from scraping their content, they had to block GPTBot specifically. If they also wanted to exclude Google’s AI training crawler, they needed a separate rule. Microsoft’s crawlers required yet another entry. Anthropic’s Claude-related bots, another. And that list grows as new AI companies and new crawlers proliferate.
For large publishers with dedicated engineering teams, maintaining this ever-expanding blocklist is an operational headache. For small publishers — independent journalists, regional news outlets, solo bloggers, academic researchers sharing findings — it is frequently an insurmountable burden. Many simply do not know how to update a robots.txt file, let alone track which crawler strings correspond to which AI company’s training pipeline.
The cumulative effect has been a web environment where a publisher’s stated preferences are inconsistently enforced at best and largely ignored at worst. Legal challenges have proliferated, with multiple high-profile lawsuits filed by news organisations and authors against AI companies for using copyrighted content without consent or compensation.
A Single Signal, Multiple Participants
The elegance of the proposed standard is its simplicity. Rather than requiring publishers to maintain a separate blocklist entry per AI company, the policy token creates a shared vocabulary. A publisher sets their preference once using the standardised signal, and every AI company that has pledged to participate reads that signal and acts accordingly.
This is structurally similar to how the “Do Not Track” signal worked in web browsers — a user-set preference that, in principle, communicated data-use intentions to websites across the board. The difference here is that the four companies backing the AI scraping standard represent a substantial share of the most powerful AI training operations on the planet. Their collective buy-in gives the standard real teeth in a way that voluntary browser signals historically struggled to achieve.
The proposal being routed through the IETF is also significant. The IETF is a rigorous, consensus-driven body. Submitting the proposal there signals an intent to create something durable, technically sound, and potentially adoptable by additional AI companies and web platforms in the future. It is not a proprietary solution controlled by any single company — it is an open standard in the making.
What Publishers in India Should Know
For publishers based in India — from major Hindi-language news portals to English-language digital media startups and independent content creators — this development carries particular relevance. Indian digital publishing has grown dramatically, producing vast volumes of original journalism, analysis, creative writing, and educational content. Much of this content is in English, making it especially accessible to AI crawlers trained primarily on English-language corpora.
Indian publishers who have been concerned about their content being used to train AI models without consent or compensation now have — or will soon have — a more practical lever to pull. Once the standard is finalised and adopted, updating a robots.txt file to include the appropriate policy token should be a straightforward technical task, achievable even for sites without large engineering teams.
It is worth noting that adopting the opt-out signal does not guarantee financial compensation for past use, nor does it retroactively address content that has already been scraped. Those questions remain the subject of ongoing legal and regulatory debate globally, including in India where questions around data localisation and digital rights are increasingly prominent in policy discussions.
Limitations and Open Questions
No standard is a silver bullet, and this one comes with important caveats worth considering carefully.
- Voluntary compliance: The standard works only because the companies backing it have pledged to honour it. Any AI company that chooses not to participate — or that emerges after the standard is established — is under no legal obligation to respect the signal.
- Enforcement gap: Unlike, say, GDPR in the European Union, there is currently no regulatory framework in most jurisdictions that mandates compliance with
robots.txt-based opt-out signals for AI training purposes. Honouring the token is a matter of corporate policy, not law. - Scope of ‘scraping’: The standard as described addresses AI training data collection. It may not cover all forms of AI-related data use, such as real-time retrieval augmented generation (RAG) systems that pull live web content to answer queries — a distinction that could matter enormously for publishers.
- Verification: Publishers currently have limited ability to verify whether their opt-out preferences are actually being respected, since AI training pipelines are largely opaque.
Despite these limitations, the initiative represents meaningful progress. Having four of the most significant AI developers in the world commit to a shared, IETF-backed standard is a materially better situation than the fragmented, individually-negotiated status quo.
The Broader Trajectory
This announcement arrives at a moment when the question of how AI companies interact with the broader information ecosystem is under intense scrutiny from regulators, courts, and civil society alike. Lawsuits from news publishers and authors, proposed legislation in multiple countries, and growing public awareness of how AI models are trained have collectively created pressure on AI companies to demonstrate more responsible data practices.
The shared policy token initiative can be read as a response to that pressure — an attempt by major AI developers to establish industry norms before regulators impose them. Whether that is viewed as genuine progress or strategic pre-emption depends on your perspective. What is less debatable is that a workable, widely-adopted technical standard for AI scraping preferences would be a meaningful improvement for the many publishers — large and small, in India and globally — who have felt powerless in the face of AI data collection at scale.
The IETF process is deliberate and can take considerable time. Watching how this standard evolves, which additional companies sign on, and crucially, how consistently it is enforced in practice, will tell us a great deal about whether the AI industry is genuinely prepared to rebuild trust with the content creators whose work underpins the intelligence of these models.
