Data needs its DevOps moment
render.comHi Everyone! (I wrote this article)
I've been working in data space for ~7 years now. I drank the ELT koolaid and then painful learned about its inefficiencies. I'm trying to build tools that take an SRE/DevOps approach to data infrastructure!
I'm building a pipeline tool SQLFlow that aims to enable DevOps for data:
- Configuration Driven - CI Test framework integrated - First class metrics and monitoring
https://github.com/turbolytics/sql-flow
I wrote about some of these while launching SQLFlow on the render platform.
I'm happy to answer any questions or just talk modern data!
Thank you!
-----
I'm trying to build a whole ecosystem. SQLFlow is the stateful stream processor.
dbhealth is a way to monitor postgres and table freshness and schema changes.
control is a way to monitor both of these :)