Ask HN: What is the correct way to deal with pipelines?
I have an application where once a job succeeds it needs to go through a series of steps, some are CPU intensive (ffmpeg, etc.), others require communicating with an external API that may fail (Discord), etc.
This is not a massive data-engineering project. I just need to ensure the steps get run to completion.
My current plan to do this is to just use Redis lists as queues, and then have steps go from one queue to the next, but I'm wondering if there's a better way. "correct" is a value judgement that depends on lots of different things. Only you can decide which tool is correct. Here are some ideas: - https://github.com/huginn/huginn Your idea about a queue (in redis, or postgres, or sqlite, etc) is also totally valid. These off-the-shelf tools I listed probably wouldn't give you a huge advantage IMO. I agree there are many options in this space. Two others to consider: - https://github.com/spotify/luigi There are also many Kubernetes based options out there. For the specific use case you specified, you might even consider a plain old Makefile and incrond if you expect these all to run on a single host and be triggered by a new file showing up in a directory… I like Airflow because you can give access to the web UI to operators and they can kick/run/stop tasks or graphs of tasks. Both Airflow and Luigi expect you to express your workflow as a DAG in Python code. It seems like the modern solution is to use something like temporal.io -- (related to windmill). Alas, surely people have been solving this problem for ages now without temporal.io Buddy.works is my favorite CI tool