Engineering over AI
martinantos.comVery true!
I do RAG for other types of structured data and this is fundamental to get relevant objects in your context.
My approach for code would be to create a graph structure with relationships between the different codepaths and expose a retrieval api through tools/function calling so that the LLM can query the codebase structure on top of doing semantic embedding similarity search and text similarity search.
You could also add a graph search for related elements for each element returned by the other search pipelines to increase the chance of having all the pieces of the puzzle in the context before using the LLM to solve the problem.
The other crucial thing to do would be to inspect dependencies (and their types, when possible) and maybe download documentation to offer tips that are accurate and not hallucinated.
Nowadays I get hallucinations for code generation as soon as things get hard, making LLM coding useful only for trivial code writing.
Analysing the code structure and dependencies would require plenty of work for each specific language, so it won't be a easy win like "just throwing RAG" - which is what the current players are doing to raise money - with mediocre results.
I feel like I just read the introduction to an interesting blog post.
At the bottom of the page, in the footer, there's this text:
> Half-baked ideas for a better future.
Maybe the article is intentionally supposed to read like that?
https://philip.greenspun.com/writing/changed-by-web-and-webl...
What was missing from the 1990 Web? The original Web did not effectively support the one-paragraph idea. A reader who encountered a Web page containing only one paragraph would have been startled and wondered if the page had been left unfinished.
How startling would that really be? Seems like half the internet claimed to be under construction.
If you had a dialup modem, like mine, that only ever got 1-2 kilobytes per second despite being theoretically 56k, pages did sometimes stop loading in the middle without any
> If you had a dialup modem, like mine, that only ever got 1-2 kilobytes per second despite being theoretically 56k, pages did sometimes stop loading in the middle without any [cuts off]
Are you still using it?
But this isn't a one-paragraph idea, it's 10 paragraphs suggesting that there will be an idea, then one paragraph suggesting that there has been an idea.
Maybe let AI finish the article.
I feel like this is both very right but also the million dollar question?
I don't think others necessarily quote-unquote "lost focus" on this problem, but it's not exactly easy to solve correctly, so in the meantime it's easier to create something with the next best approximation.
I think this is the idea behind sourcegraph's cody, trying to take their expertise in understanding codebases and ASTs and using it to guide the llm
Where's the rest of the article?
I finished reading it, wondering the same thing!
Typo of "codebase" as codebae: This is also the reason why the higher context window doesn’t matter. Even if you could feed your whole codebase into an LLM, you’d still face the same problem of missing the structural relationships of the codebae.
Tbh, "codebae" describes the most compelling use case for AI. Relevant Stallman story:
You would need to have smaller agents negotiate on behalf of their functional units.