Ask HN: How would you build a custom web scraper?
I'm trying to build a web scraper that can scrape similar websites. For example, if I wanted to scrape 10 different fast food sites and return the locations the stores are in, it would be able to identify different elements and do so. Any ideas on how to implement this? 1. crawl pages looking for more pages on the site 2. scrape text 3. find address using something like this: https://stackoverflow.com/questions/9397485/regex-street-add... Use Scrapy to get at least some measure of maintainability.