- cross-posted to:
- [email protected]
- cross-posted to:
- [email protected]
You must log in or register to comment.
In my experience, the single biggest bully on the internet are the servers controlled by Meta which in my experience literally perform DDoS attacks whilst crawling, hitting sites several orders of magnitude more than all the others combined.
Actively blocking them was the only option left.
Jeez, don’t these fucksticks have enough data already? People are literally handing it to them hand over fist and they’re still like “no, we need to forcibly suck the data out of you until your servers burst into flames”
How else would they get complete profiles on people who don’t use their products?
In my opinion, pretty much spot on.
Yep same thing. I have some small servers and was getting hammered by openai ip controlled ai crawlers not respecting robots.txt. had to block all their IP addresses and create an AI black hole in order to stop them ddos ing my tiny site(s).
Man, this current age of AI really sucks.
You licensed your comment? You just saw first hand that AI crawlers don’t care about legal barriers.
Some info about that …
I suppose the goal isa kind of poisoning the well, in case at any time in the future it becomes enforceable somehow
Did you not pay your protection money to CloudFlare?
From the article …
GNOME sysadmin Bart Piotrowski shared on Mastodon that only about 3.2 percent of requests (2,690 out of 84,056) passed their challenge system, suggesting the vast majority of traffic was automated.
I put a rate limit on my nginx docker container. No clue if it worked but my customers are able to use the website now. I get a Alton of automated probing and SQL injection requests. Pretty horrible considering I built my app for very minimal traffic and use session data in places rather than pulling from DB and the ddos basically attacks corrupt sessions
The Internet has always been like that even before the AI stuff got up to stream. If you expose anything to the public Internet it takes about 5s for things to start port scanning if they can it try WordPress/Drupal exploits.
Even mainly text-mode sites like LWN are feeling the strain and finding it hard to support all this parasite bots.
So don’t support them. Block them!
Sure, but the challenge is how to block them without putting undue load on humans.
In the olden days, you’d just host a webserver and be done with it. Today you need elaborate setups to trick bots. It’s a losing proposition.
What if we start throtling them so we make them waste time? Like, we could throttle contiguous requests, so if anyone is hitting the server aggresively they’d get slowed down.
They can just interleave requests to different hosts. Honestly, someone spidering the whole Web probably should be doing that regardless.
The tricky bit is recognizing that the requests are all from the same source. Often they use different IP addresses and to even classify requests at all you have to keep extra state around that you wouldn’t need without this anti-social behavior.
Btw, Cloudflare supports Privacy Pass for their captchas. Using Silk let’s me log-in to Nexusmods on my hardened Firefox profile.
