Ask HN: How do deal with scrappers/bots?
I'm having an issue with a random individual who has built script/bot to scrape content off of my startup's site. We have a public API with the same content, but this individual is using private calls at a rate of 30+/sec and he's essentially taking down the site. I've blocked his IP via the firewall, but I'd like to figure out a system to prevent future individuals from doing the same thing.
So in your startups/api's how do you deal with bots/scrappers? I found that just blocking him at the backend didn't help since the requests were just coming faster and faster (less to parse).
Also, any recourse? Almost taking down someone's servers is not cool. Not to mention the costs incurred from having to serve 53,000 extra queries in a 4 hour block (bandwidth and CPU).
No comments yet.