State of data 2023

In the past 2 years, the data ecosystem has been evolving rapidly. New tools have been emerging every month in the modern data stack. In a hype cycle, it becomes hard to distinguish the signal from the noise. Which of those tools would eventually become simple features or actual products that we would be using in a few years?

In addition to our growing number of tools, we've seen a few new trends, such as declarative approaches appearing everywhere (from Kubernetes where we have code as infra, orchestration as code, and even integration as code)

Other trends include the rise of the Semantic Layer, Rust becoming the future of performance-intense applications in data (potential replacing Spark eventually), or even data modeling coming back with the exposing of the modern data stack. All this is without mentioning AI and vector-based engines being used for small data, such as DuckDB along with newer ones especially supporting the AI wave behind the curtains with Pinecone, Qdrant, etc.

So much is going on that we had to take a step back on the evolution of the data engineer

To make sense of it all, we need all the data we can get. Fortunately, this State of Data 2023 survey is the largest data engineering survey made to date. It will help us take a step back and understand what the community is using and feeling excited about, what is noise or signal in the modern data stack.

The research will first give details on the demographics of the survey participants. Then, it will go through the data stack, but also the blogs, podcasts, newsletters that we follow most. 

The best insights are usually discovered when using the filters at your disposal, per company size and per experience, so you can drill down on the information that matters most to you.

Now, let's see what we discovered with the survey.

This is the largest data engineering survey made to date.

This is the largest data engineering survey made to date.

This is the largest data engineering survey made to date.