Use SQL for data pipelines, Airflow Sucks for MLOps, Draw plots on the terminal
ingestthis.substack.com
2 threads
I know it’s not the focus of the article, but a personal pet-peeve of mine is when pandas gets mentioned in articles and discussions regarding alternatives to sql and data pipeline tools. Pandas was never meant to be a replacement for sql, it was meant to be a replacement for excel in highly dynamic quickly changing financial and econometric excel models. It just so happens there’s a lot of overlap between this and more proper data pipeline tools. But for its original purpose I don’t know if anything that still comes close to the useability of pandas.
You're right, but pandas today is used a bit everywhere, and in data pipelines too, even if it wasn't its main goal.
And Data-Centric AI!