AI giants promise publishers new way to block some scraping

SAN FRANCISCO, California – OpenAI, Google, Microsoft and Anthropic pledged on Oct 30 to give media outlets and other websites a single way to prevent their content from being scraped to train artificial intelligence models, amid growing criticisms and legal battles from news publishers.Publishers currently have to implement multiple blocking commands for every bot that crawls the web to collect data, but the new proposed solution will allow them to enter a single, standardised instruction that is recognised and honoured by any AI company that adopts the protocol.The solution will not fully take effect for months, though.Microsoft will implement it beginning Feb 1, 2027, and Google sometime in 2027.ChatGPT-maker OpenAI said it will roll it out “in the coming months”, while Claude-maker Anthropic said it has “already started” rolling it out.The announcement comes as publishers are facing a surge in bots that scrape their articles to feed AI models. Some, like the New York Times, have taken the matter to court.In December 2023, the Times sued OpenAI and Microsoft for allegedly using millions of its articles without authorisation or payment to train ChatGPT.The two tech companies are invoking an exception to copyright law known as fair use.Meanwhile, internet infrastructure provider Cloudflare has blocked AI bots by default on new customer sites since July 2025, and a standard called Really Simple Licensing already allows publishers to charge bots that visit their pages.The new setting will apply only to future training of models but does not prevent bots from crawling otherwise, and content scraping will be allowed by default until the setting is changed by a publisher, Microsoft said in a blog post.Opting out of AI training could reduce the chances that a publisher’s content will surface in AI apps and experiences, Microsoft and Google said in separate blog posts.“We believe creating a common signal that conveys such preferences to multiple providers at once is an important step. And we look forward to working with other AI companies on the standards and their implementation across the industry,” Meta spokesman Andy Stone wrote on X, conveying his support for the effort. AFP