Nearly 80% of the largest news websites in the UK and US are now taking a firm stance against AI training bots by implementing crawling restrictions. Research revealed that 79% of nearly 100 top news sites surveyed are blocking at least one major crawler used for training large language models (LLMs), such as OpenAI’s GPTBot, ClaudeBot, and Anthropic-ai.
Furthermore, 71% are actively preventing AI bots from performing live searches by blocking services like ChatGPT-User, Claude-Web, and Perplexity-User. This sweeping shift was discussed earlier this week at a conference in London, where industry leaders stressed the importance of blocking bots to maintain control over proprietary content.
Anthony Katsur, CEO of IAB Tech Lab, suggested publishers band together to block bots collectively, emphasizing this as a strategic move to maintain their market influence. Katsur proposed that publishers could establish a content marketplace where LLM developers must pay for accessing and using data. “It’s never too late to start blocking bots,” he remarked, pointing out that LLMs need continuous recrawling to ensure the accuracy and relevance of their outputs.
This industry-wide trend reflects growing concerns over the use of proprietary content to train AI models without appropriate compensation or permission. By limiting bot access, publishers are not only protecting their intellectual property but also preparing for a future where content licensing models may become standard practice in the AI ecosystem.
The implications of this development extend to AI developers, content marketers, and product managers, as businesses relying on AI functionalities like chatbots or live search will need to navigate these new access restrictions. Similarly, WordPress-focused web developers and SEO specialists must now accommodate strategies that ensure compliance with these new measures.
For publishers and small businesses focusing on content optimization, tools like the Content Freshness Engine can help maintain web content relevancy while managing the evolving landscape of AI-driven insights.
Source: Press Gazette
Source: Press Gazette