OpenAI, Google, Microsoft, and Anthropic Back a Single Opt-Out Signal for Web Scraping
The four largest AI companies have agreed to honour a new shared standard that lets publishers block AI training crawlers with a single line in their robots.txt file — the text file websites use to tell web crawlers what they can and cannot access, according to The Verge.
Currently, blocking each company's crawler requires separate, individual instructions. The new "policy token" signal, proposed through the IETF (Internet Engineering Task Force — the body that sets internet standards), would cover all participating companies at once. Anthropic has already adopted it. Google and Microsoft plan to follow in 2025; OpenAI says it will implement it "in the coming months." Meta signalled support but has not confirmed it will comply.
If you run a website and care about this, the robots.txt change is what to watch for.
Sources

