Make your data accessible or you’re fired
medium.comWeb scraping is such an underappreciated art. Unfortunately, it's harder and harder to do little hobby projects since many sites will quickly ban you.
I was working on a music project, trying to use Rate Your Music as the data source. Got banned before even hitting 50 requests (less than one a minute). I should have checked their TOS first ...
Some secrets:
- use a VPS (throw away IP) - rate limit yourself - cache all the things