Over time, especially recently, I have seen thousands of diverse IP addresses scraping the site. They use random/varying user-agents. I originally was blocking Brazil /16s since it appeared that most of the traffic was coming from there, but over the past few weeks the IPs come from everywhere. Each IP makes only a few requests, trying to stay under the radar. Right now, I have set some scripts to block and log the IPs as they come in.
I am blocking between 50 and 100 unique IP addresses per minute, and this is after I already blocked the main Chinese LLM scrapers and several /16s. Few of the IPs belong to obvious providers. Many just seem to be home users. Many are from countries that do not have the money to build LLMs. There are even wireless phone company IPs.
None of the requests are particularly malicious. They are just downloading pages.
Am I missing something? Is there a new botnet scraping the web ? A quick grep through my logs shows I have blocked 15,000 requests in the past 90 minutes, but only 1300 of them are repeats of IPs that have been added to my block list. Yesterday, I blocked 220,000 requests and only 13,000 of them were repeats.
Different devices also enable this, as mentioned in the comments. Smarthome / IoT devices, cable-brand router/APs, etc. There are also services for rotating residential proxies, that are essentially breaking the ToS of the companies in charge of them, but they trade/buy new IPs constantly.
The larger scale a site is indexed at, the more you'll see this traffic pick up. CloudFlare has rules that can help with it, and you can get stricter with them if you know your audience / customer base via whitelists; geo (can be circumvented ofc, but quiets the noise), user-agent, http version, etc. For the more broad ones, just immediately prompt a challenge if you don't want to outright drop/block them.
It's been this way for a while, LLMs have made it worse, but there was already a ton of garbage requests / scanning going on.
If you have AI-generated content, expect an AI-generated audience.