I had upwards of 3000 AI bots before implementing Cloudflare. Cloudflare without 'attack mode' brings that down to about 50-100; cloudflare with attack mode brings it down to virtually nothing other than legit search engines and registered users. I'm sure every site is different, but for me chasing them through .htaccess was futile. It was a whack-a-mole game I'd never win...Search your access logs for DuckDuckGo's user agent and you can see what the status code is.
If you have mod_security I'd remove the sections for QUERY_STRING and REQUEST_URI, it's going to be a lot of duplication of what mod_security is already doing.
The other hosts I have no idea. You could for example be blocking other services like VPN's which you may or may not want to block. Bots are going to come from wide range of hosts, I keep an eye on the IP's in awstats and only block entire networks as required after trying to determine I'm not blocking something like a VPN, DuckDuckGo etc.
Statistics: Posted by CPTOM — Wed May 14, 2025 3:50 am