- cross-posted to:
- [email protected]
- cross-posted to:
- [email protected]
You must log in or register to comment.
In my experience, the single biggest bully on the internet are the servers controlled by Meta which in my experience literally perform DDoS attacks whilst crawling, hitting sites several orders of magnitude more than all the others combined.
Actively blocking them was the only option left.
We need something better than Anubis that can run with JS disabled
Then it’s literally down to things like statistics, rate limiting certain ip addresses if one tries to access completely different pages at the same time, etc. All of those can be worked around easily
Dread has something that works without JS. Don’t know much about it though
Are there any good log monitoring programs that will automatically blacklist the IP of any crawler that ignores robots.txt?
Yeah, I’ve been curious if you could explicitly block a page in robots.txt, hide an invisible link to the same page in your footer, then kinda have it act like an immediate IP block when someone requests it.
There are systems that will use a hidden hyperlink (which only a bot would see and use) which directs them to an infinitely long/wide junk link tree. It means they end up trapped in bot-purgatory and stop crawling the rest of your site.
The issue is that it means you end up consuming resources just to keep the bot trapped.
