OpenAI, Google, Microsoft, and Anthropic Unite Behind a Universal AI Scraping Opt-Out Standard
Reading Time: 5 minutesOpenAI, Google, Microsoft, and Anthropic are backing a new IETF-proposed ‘policy token’ standard that lets publishers opt out of AI training data scraping via a single robots.txt signal. The initiative aims to replace the fragmented, crawler-by-crawler opt-out system that has left many publishers — including those in India — with little practical control over how their content is used.