Ask HN: What do you use (AI based) web crawlers for?
I am curious, what are people and organisations building with web crawlers, especially since now all of them seem to have LLM support for further refining the output from crawled pages.
Open source projects like crawl4ai[1] have a crazy amount of stars (~59k). This got me curious.
https://github.com/unclecode/crawl4ai
No comments yet.