This is the coolest thing we've ever done - and probably the least sexy demo This has been my white whale ever since the first release of WishfulSearch, to know if this is even possible. Let me explain - or you can just watch it work👇
Post
Post
Hrishi on X: "This is the coolest thing we've ever done - and probably the least sexy demo This has been my white whale ever since the first release of WishfulSearch, to know if this is even possible. Let me explain - or you can just watch it work👇"
Modern AI systems being able to do natural-language to SQL queries opened up a huge world of possibilities, and a massive industry that's trying to let you 'talk to your data'. The problem most people and businesses encounter is that your data doesn't fit.
Most information today is best considered semi-structured, where it sits in Excel sheets, NoSQL JSON/XML, old databases, or exposed at the whim of providers that host that data. Very little is in a Postgres db you control. Little of it has defined typespecs, relationships.
Therein lies the problem - structured-search AI systems presume you're starting near the finish line, or are willing to spend time and money migrating to their hosting. Someone needs to build the ETL pipelines - and for the last four years in shipping, that's been our job.
Near 40% of our dev time has gone towards building these things. It takes us weeks to do what you just watched happen in less than a few seconds. Thing of the past. We can now build completely automated data import pipelines that understand and extract structure, flatten,
and import it to be queried by AI and joined together as tools. Drag-and-drop. Oh and one more thing: we can build these pipelines without executing model-written code. This is huge, for 1: Security. Model-written code will not be safe for the next few years at least,
especially not when it handles your data. 2: Control. Model-written code is a black-box where error-free execution of the code is the only feedback you're getting. Here you have tons of intermediate steps - sDDLs, typespecs, schema where you can inspect, modify and direct.
I'm pretty excited what we can do with this. Open-source is some time away - my first step is to delete all of this code before anyone sees it. It's the worst-written thing you've ever seen, existing only to prove to us that it can be done. Give me a week to write better code 🧘
This is pure gold! Great work
@hrishioa- this is the future. Happy to contribute code and help test. We’ve got around 20+ hand built ETL pipelines and close to 200+ models built on dbt used by our business daily. Would love to accelerate adding new sources and high impact use
