Ask HN: Crawl the entire web?
If you were tasked with crawling the entire web, how would you go about doing it? I'd grab the CommonCrawl data and keep moving on what my actual problem is.
If you were tasked with crawling the entire web, how would you go about doing it? I'd grab the CommonCrawl data and keep moving on what my actual problem is.