<ul><li>Wikimedia Commons, the repository of images, videos, and audio files, has seen a 50% surge in bandwidth consumption due to AI scrapers since January 2024. Bots account for 65% of the most resource-intensive traffic, though they make up only 35% of overall pageviews.</li><li>Frequently-accessed content stays closer to the user in the cache, while less popular content resides in the more expensive core data center, which is often the target of scrapers. The Wikimedia Foundation's site reliability team is spending significant resources to block crawlers and maintain service for regular users.</li><li>The rise in AI crawlers ignoring 'robots.txt' files meant to deter automated traffic is threatening the open internet. Companies like Cloudflare have introduced AI measures to slow down scrapers, but the cat-and-mouse game between developers and crawlers continues.</li><li>The increasing challenges posed by crawlers may lead many publishers to resort to logins and paywalls, negatively impacting the user experience on the web.</li></ul>

AI crawlers cause Wikimedia Commons bandwidth demands to surge 50%

Discover more