What UI do you use on top of data engineering tools to look at data?
Tools like DuckDB Wasm and data engineering platforms like Iceberg leverage Parquet’s built-in indexing to very efficiently query files over the network. But as I’ve been building data tools myself, the stack gets complicated fast, especially once you try to visualize or explore the data instead of just querying it. I’m intrigued by some of the modern tricks people are using to do more data engineering client-side.
With OPFS + Parquet + Wasm, the browser already has everything it needs to handle multi-GB LLM datasets client-side.
Is the world of data UIs evolving? Are there new data tools and best practices beyond notebooks and DuckDB? Having tried a variety of tools, I usually return to Snowflake.
Within Snowflake, I don't use notebooks; instead, I prefer to create database Views to contain my code. I may have Views that call other Views etc. I like Snowflake because for every query in its UI, it automatically profiles the results it returns. e.g., for each column it gives a frequency distribution histogram of the values within the column. In short, I like the ability to store my code in something I can easily backup and re-use without adding a lot of other tools. Makes sense. I'm not currently in snowflake because I'm mostly working with local parquet files. Would prefer not to have to pay for snowflake just to explore my data. I'm interested in better data UIs though so I might need to check it out. To keep costs down. I use standard edition with x-small data warehouse. The biggest cost driver is CPU or how long queries run.