Faster Retrieval for Better, More Efficient RAG
machinelearning.apple.comJust yesterday I saw a talk from Spotify where they presented their own fast retrieval mechanism "Parallel Fusion Router" [1]. It's also in the "learned retrieval" paradigm - in their case they use a teacher LLM to train a much smaller router LLM how to retrieve, using an LLM-as-a-judge based reward function.
From my own experience, these learned methods from Apple, Spotify and others, are highly relevant to hyperscale only - when you are dealing with hundreds of millions of items to choose from (or more), and require super high relevance in the top-5. For all other use cases (up to a few million items) I found that just building a BM25 index, and giving the LLM a BM25 "search tool" and casting a "wide net" wins every single time compared to learned methods (though the latter are more publishable in recsys and icml).
[1] https://research.atspotify.com/2025/9/you-say-search-i-say-r...
It seems, that the amount of information that "typically" an person would want to work with, can be handled with pretty much any technique by the compute that most of us have, at a performance level that is satisfying.. It's only when you start to increase the information volume, decrease the compute, or increase the performance expectations, that it is really impactful to start to find methods that really make the most of the deep structures of that information to query it. Complexity of the original information and what accuracy means for your use case also comes into play I suppose.
I've been experimenting with hybrid memory retrieval using SQLite FTS5/BM25 for lexical search and dense embeddings from Nomic Embed Text v1.5 (768D), all-MiniLM-L6-v2 (384D), and Qwen3-Embedding (0.6B), with NumPy cosine similarity, Matryoshka-style dimensional truncation (64D → 768D reranking), and optional graph-based candidate expansion. In one 40-query benchmark, combining retrieval channels improved candidate recall from 30/40 with compact vectors alone to 40/40, although Top-1 accuracy was only 33/40. Another metric I was looking at was number of final candidates found to filter. If its very high, you can get better accuracy, but at the expense of resource use, and it can also lead to worse accuracy if your tying to choose top result efficently.. it makes a needle in a needle stack.
> I found that just building a BM25 index, and giving the LLM a BM25 "search tool" and casting a "wide net" wins every single time compared to learned methods
Funny, I did exactly this for my last RAG project. Works great for a small dataset.
Same here. Up to a few millions of documents, this is already great. Combine this with a few other simple techniques and extra metadata during ingestion, and you already have a system that practically useable.
I have tried this by ingesting all of cloudflare's highly technical complex documents (more than 3000 A4 sized pages) and then comparing results with that of cloudflare's own chatbot in their dashboard.
In all cases my system either matched, or performed marginally better than cloudflare's own bot.
I do use a few more techniques though, but, not RDF triplets, and definitely no custom trained neural network.
For some reason I thought everyone was using BERT vectors and cosine similarity to do this RAG stuff. This is not my jam, but I do dabble. Neat!
> found that just building a BM25 index, and giving the LLM a BM25 "search tool" and casting a "wide net" wins every single time compared to learned methods
I've been saying this for a while now. Lexical is approximately the same thing as semantic when a frontier LLM is driving the search terms.
I'm not sold; that's not to say I can't be swayed, but I've spent a lot of time building RAG solutions as a "one-size-fits-all" against codebases and I am not convinced embedding models or any of the myriad BERT implementations work well enough that they can beat out grep.
Most embedding/BERT models are far too small to truly grasp the nettles when an AI ask questions to them; and if you condition the AI to only search for really basic things, then... you're basically back to grep + regex.
You need to prune all manner of useless detritus that might gum up your matches like test files, documentation, and sidecars that semantically overlap or you're just going to match your tests. That's fine if you're building a document store for a business, but if you want a "one-and-done" solution for your codebase it's rarely good enough.
And if you do use a very large embedding model (which means it'll require running in the cloud as most company computers won't run it well), then... well, you're better off just using a language model as a subagent to scout ahead for you and elide the embedding model to begin with.
I feel like retrieval approaches, despite how good your ranking mechanisms are, end up biasing LLMs into what a heuristics-based system with way worse understanding of the domain says. I guess is a trade off between speed and quality, but letting the LLM explore without a bias (based on my personal experience) seems to generally provide much better results in terms of quality for the final output.
The trade-off between speed and quality is everything though. I've found that semantic searching everything that might be relevant and dump the top N into the model is way faster at getting a workable result.
Good point and it’s an often overlooked.
RAG can take the intelligence of an LLM and dumb it down to old school Google search results.
With the caveat that I am not the average user, my most philosophically-pleasing system would be an LLM which generates and iterates "normal" search parameters. Then the user has the option to view, tweak, and and--just maybe--learn the underlying query language.
Apple has a fascinating history of waiting and seeing how a market develops before entering with a better product and eventually dominating, the iPod and iPhone are two examples. It seems they may be building a strong position to pull that off again in the personal assistant AI category ChatGPT is trying to dominate.
I enjoy watching the slow burn approach Apple is deploying as they quietly gear up to offer some % of local inference on iPhones to provide most of the AI services consumers want. Local inference could enable a very low price point for AI services that Apple would leverage alongside their massive ecosystem advantages and privacy claims to compete very effectively for personal assistant AI subscriptions.
> the personal assistant AI category
Didn’t Apple sort of already enter this market early-on with their subpar Siri?
I’ve been an iPhone and MacBook Pro user for years, but each time when I set up a new Apple device the first thing I do is say no to anything Siri, and also turning off additional Siri related settings. To the best of my knowledge, Siri is worse than useless, and I haven’t regretted staying away from it.
Agreed, but I wonder how that plays out without them entering the frontier model space. I just downloaded the new foundation models cli and ran some inference on my Mac, it wasn't great, but it was neat to see. I asked which model it was using and it said gpt 4o mini.
An on-device, 100% private AI assistant seems like it could be so big that it could be a legitimate threat to the frontier model providers they are partnering with right now, imho.
> I wonder how that plays out without them entering the frontier model space
bet could be that this space will be commoditized: many good enough open weight models and multiple competing provides.
I think it'd be foolish to assume that they aren't already - they have a reputation of doing a lot more internally than ever comes to light. Smaller companies like mistral or thinking machines show that you don't even need that much resources to be semi-competitive, and comparatively Apple is a vertically integrated behemoth.
That said in a pinch they could also acquire / acquihire both of those companies without much thought.
Apple or Steve Jobs? It seems like that was Steve Jobs doing as Apple hasn't done it without him.
as a recovering RAG-enthusiast, I will share a quote from a peer that has really stuck with me:
"why not use grep?"
You definitely should use grep wherever you can (i.e. simple, unranked term searches) to 1) exclude things you are definitely not care about 2) to include things you definitely do care about. The reason is that this is cheap and easy and it reduces the overall results you still need to dig through. And because everything else is much more expensive this is going to help reduce the amount of compute needed from that. And results are cacheable as well which helps you with repeat searches for similar things.
For ranked searches, you can do more than one and combine their results. Reciprocal rank fusion (RRF) is a common approach to do that. A simple ranked lexical query and one or more vector searches. Can be combined to amplify results that perform well across multiple of these.
RRF is really easy: for each search result, take its position in each ranked list, add 60, and divide 1 by that number. Add up those values across all lists to get the final score. Results that rank highly in multiple lists get higher scores. The magical number of 60 dampens the effect of highly ranked results biasing the outcome to much. It's not going to be a good approach for all data sets. But it works surprisingly well and it's stupidly easy to implement.
For term search, expending some AI and other cleverness at indexing time to e.g. calculate categorizations is very helpful. If you can reduce your result set with some simple term searches, it might be small enough to pass to an LLM directly and skip ranked searches entirely. Using vector search as magic pixie dust is generally going to work a lot better if you can filter and combine it with a lexical search. Using e.g. a decision model at index time to calculate some categorizations and classifications is relatively cheap and easy relative to calculating vector embeddings.
Grep is not semantic like embeddings, although the llm can expand a concept into multiple literal searches with a similar effect.
There have been some attempts at "semantic grep" but don't know its current status
Your understanding was applied too strictly and you missed the intent of the joke...
What's the joke?
Considering the exemple they have provided, it seems they used grep...
Grep is the fastest tool for many things, but for production RAG it cannot search with meaning, different languages or typos like hybrid vector search can. And it won't scale on terabytes of documents across distributed servers if you want to say under 1000ms.
all true for a single grep invocation, however a capable LLM can drive grep using similar terms (`grep (red|maroon|scarlet|crimson)`, moving the measure of similarity from embedding/indexing time to inference time. There's benefits to that.
> red|maroon|scarlet|crimson
still doesn't include typo'ed variants for any of these
but a good combination of prompt, harness and model will easily close that gap. a more advanced (but equivalent) form of grep could also offer fuzzy matching/levenshtein distance function to augment that.
It will, but it does so by burning tokens. I'm sure you've also seen the output of an overly eager grep by an LLM.
I think we're verging on the point where people will care more about the most token-efficient ways to do a task than the lowest human effort version. Humans are expensive, humans using half as many tokens are cheaper.
This sounds like AOT vs JIT weights.
If the data is this large then why not ask the llm to build a query
That's more-or-less what RAG is; a tool to surface relevant data so that AI can make informed decisions. Efficient retrieval is important so that AI doesn't need to ingest the whole corpus for each query.
latency is often the reason, distilling interactions into significantly better retrieval is the second.
My case:
With a very large dataset of totally unnormalized data, semantic folding is a silver bullet.
Retrieval is a whole subject of study. For well-structured plaintext datasets, basic filtering may be as good as semantic and lexical searches. Takes measurement to know what you need
the amount of times the clanker has used grep badly and nuked my changes should be illegal, god bless git, god bless AI hook gates, and god bless RAG
Can't wait to see how they leverage this research to make Siri even worse!
The questions and examples are so laughable for anyone who played with embedding search that you wonder if this paper is real or not.