Show HN: estela, a modern elastic web scraping cluster
github.comestela is an elastic web scraping cluster running on Kubernetes. It provides mechanisms to deploy, run and scale web scraping spiders via a REST API and a web interface.
It is a modern alternative to the few OSS projects available for such needs, like scrapyd and gerapy. estela aims to help web scraping teams and individuals that are considering moving away from proprietary scraping clouds, or who are in the process of designing their on-premise scraping architecture, so as not to needlessly reinvent the wheel, and to benefit from the get-go from features such as built-in scalability and elasticity, among others.
estela has been recently published as OSS under the MIT license:
https://github.com/bitmakerla/estela
More details about it can be found in the release blog post and the official documentation:
https://bitmaker.la/blog/2022/06/24/estela-oss-release.html
https://estela.bitmaker.la/docs/
estela supports Scrapy spiders for the moment being, but additional frameworks/languages are on the roadmap.
All kinds of feedback and contributions are welcome!
Disclaimer: I'm part of the development team behind estela :-)
No comments yet.