×
Anthropic’s Aggressive Data Scraping Is Causing Problems for Sites It Targets
Written by
Published on
Join our daily newsletter for breaking news, product launches and deals, research breakdowns, and other industry-leading AI coverage
Join Now

Anthropic’s aggressive data scraping practices raise ethical concerns as the company disregards website permissions in its quest to train its Claude language model.

Anthropic’s data scraping tactics: The artificial intelligence company Anthropic has been engaging in aggressive data scraping practices to gather training data for its Claude language model, often disregarding website permissions and robots.txt files:

  • Ifixit.com reported that Anthropic’s ClaudeBot hit their site a million times in a single day, highlighting the intensity of the company’s data scraping efforts.
  • Freelancer experienced 3.5 million hits from ClaudeBot in just four hours, with the bot’s activities triggering alarms and waking up system administrators in the middle of the night, ultimately forcing the website to block Anthropic.
  • Anthropic ignores robots.txt files, which are used by websites to specify which data can be accessed by web crawlers, and continues to scrape data even when explicitly instructed not to do so.

Ethical concerns and industry practices: Anthropic’s actions raise questions about the company’s commitment to responsible AI development, especially given its origins and stated mission:

  • The company was founded by former OpenAI researchers who aimed to develop “responsible” AI systems, but their current practices seem to contradict this goal.
  • Egregious plagiarism and disregard for website permissions have long been common in AI model training, but Anthropic’s behavior stands out as particularly aggressive and unethical.

Broader implications for AI development: Anthropic’s actions underscore the need for greater oversight and regulation in the AI industry to ensure that companies respect intellectual property rights and adhere to ethical data collection practices:

  • As AI models become increasingly sophisticated and require vast amounts of training data, it is crucial to establish clear guidelines and enforce consequences for companies that engage in unethical data scraping.
  • The AI community must work together to develop best practices and hold each other accountable to maintain public trust and ensure the responsible development of artificial intelligence technologies.
Anthropic is scraping websites so fast it's causing problems

Recent News

AI-powered computers are adding more time to workers’ tasks, but there’s a catch

Early AI PC adopters report spending more time on tasks than traditional computer users, signaling growing pains in the technology's implementation.

The global bootcamp that teaches intensive AI safety programming classes

Global bootcamp program trains next wave of AI safety professionals through intensive 10-day courses funded by Open Philanthropy.

‘Anti-scale’ and how to save journalism in an automated world

Struggling news organizations seek to balance AI adoption with growing public distrust, as the industry pivots toward community-focused journalism over content volume.