Better Airflow with Metaflow
outerbounds.comI have used Airflow as a product for over a year and I can say for sure that Airflow sucks for orchestrating ML.
The ability to pass state in Airflow absolutely sucks. It's interesting that Metaflow is trying to compile into Airflow but I am skeptical about it's scalability on Airflow especially with DAGs that have a larger task footprint.
Metaflow passes state using an object store (s3, azure blob store, etc.) even within Airflow - short circuiting Airflow's xcom machinery. But agreed, Airflow presents many more scalability challenges - this integration addresses a few in-place as well as preserves the ability to swap out Airflow with a more scalable workflow orchestrator if needed.
Totally! One intended use case of this integration is that it allows you to move easily to a more scalable orchestrator if you hit limits of Airflow