Anthropic, OpenAI, Google and Microsoft have announced support for a shared signal that would make it easier for publishers to communicate that they do not want their websites scraped for AI training, Nieman Lab’s Laura Hazard Owen reported.
Publishers currently must address companies’ crawlers individually. The proposed approach uses a common instruction in a website’s robots.txt file, allowing participating providers to recognize the same preference. Anthropic engineer Kevin Kelley proposed the signal through work at the Internet Engineering Task Force.
Implementation will be staggered. Anthropic already supports the signal, while OpenAI said it would introduce support in the coming months. Google and Microsoft said they would follow next year. The announcement concerns participating companies’ handling of publishers’ preferences; it does not mean every AI crawler will immediately stop collecting content.
Comments
No comments on this item Please log in to comment by clicking here