Ask HN: Who uses web scraping?
A few survey questions about collecting data via the web:
Do you have experience with people using web scraping commercially?
What field or industry was it being used in?
Do you think it fell in a 'grey' area, or was it explicitly disallowed or allowed?
Thanks for any info! Yes I had done web scrapping commercially, sometime back when I had just left college. I had to scrap website(s) for getting data about all pharma products available in india. So basically market research. But as soon as I realized this was against TOS of the website, i deleted all data and communicated this to client. I had to discard the contract and return even the previous payments.
Although I am happy that I did. :) Eventbrite used scraping in the early days. They first argued it was a publicly available website. "Public" in the sense that the site was browse-able without restriction. The plaintiff responded that it had anti-scraping language in its user agreement. The court said "the Terms of Use are not displayed on the website in any way in which a reasonable user could be expected to notice them." [1] - Yes [2] - Lead generation, selling data, consumer + market research [3] - I think it is always a gray area with regards to another site's TOS. But the burden of proof is on them, not the scraper. That and masking your scrapers is too easy.