- Am I wrong to be somewhat peeved by the use of "RAG" in these contexts? I always read things like this, and wonder if instead the author should be saying "Semantic Retrieval" or something something Vector, etc. Retrieval augmented generation captures tool-use, and; semantic search of course is really just a tool under the hood.
To make an anology, in my mind, this is akin to saying "fuel air mixture system" when referring to direct fuel injection specifically, when of course, a carburetor also lives in that category.
- To be honest as someone working in this space for the past two years, the problem with the “RAG” and semantic search community is it’s mostly vendors and solutions people selling simple, general stuff to product teams.
If you really are into search you probably implement something bespoke for your use case and integrate it into a product directly, and engage with models/infra tools directly rather than through the products in the space.
If you understand how “semantic retrieval” and other search tools are implemented in practice they feel almost embarrassingly primitive to give such fancy names, or pay for through tools that just implement really basic post-filtering. The entire space had the rug pulled out from under it once “agentic search” took off and most major LLM vendors started integrating web search and tool calling into their products. There is still a lot more interesting stuff you could do with customized rerankers/embedding models, and search algorithms, or small models specialized for agentic search/retrieval, etc but the userbase is big companies that realistically don’t need anything more than a list of tech support document titles that a cheap LLM can select from. So “RAG” is basically a sales shibboleth for that type of stuff now.
- Yes you are wrong. RAG means retrieval-augmented generation. If you’re generating something and that generation is augmented by some retrieval you’re doing RAG. The retrieval doesn’t need to be from a vector db or even based on semantic similarity.
- Calling the framework "Retrieval-Augmented Generation" is actually an oversimplification. It's defensible, but inaccurate at best.
For RAG to succeed at scale, it relies heavily on sophisticated orchestration layers, data transformation engines (indexing) and logical loops (augmented retrieval).
Arguing semantics is important at times but being absolute about it is black and white thinking - cheap effort for a complex topic. Don't throw out the baby with the bath water!
Check out Lume: https://github.com/DeepBlueDynamics/lume. If it weren't obvious by the commits, I work on it. Probably a good reference for HOW to do good RAG, but not the ONLY way to do it.
- Don't blame me, blame the people who invented the concept and wrote the original RAG paper.
- RAG is a fancy acronym that basically boils down to: let's give ai agents the super power of information retrieval (aka. search) and "augment" the generation with a list of results by adding that to the context.
The narrow interpretation of this is usually some kind of vector search. Which some people naively treat as magic pixie dust that will make search quality amazing without any tuning whatsoever.
This does not actually work all that well beyond really simple use cases. A well tuned traditional search engine can be surprisingly competitive. And I know people that do pretty complicated things with vector search that usually involve training their own models and spending a lot of effort on testing and validating those are any good.
I've been doing stuff with search for a bit over two decades. AI use cases makes information retrieval more relevant than ever. It's a key ingredient to answering questions for complex, proprietary data. And especially when that data is very complex and unstructured, naive approaches tend to have their limitations. In other words, it can pay off to to sit down and do it properly and think about things like data ingestion pipelines, transforming & enriching data, testing search quality, etc. Most of the success of a good search system usually boils down to getting your data right for indexing and optimizing it for how you are going to query your data.
The good news is that with large context windows, precision (best results are at the top) matters a bit less than recall (the search returns what you need when you search for it) these days. You can compensate for imprecise search by just fetching more results. As long as what you needed appears somewhere in the top 500 or so, you'll be fine. The flip side is of course that you end up adding a lot of noise to your context which might throw the LLM off and in general wastes a lot of tokens. That's why precision is still important.
What the article is proposing is post processing imprecise results to filter out the noise with an LLM to compensate for what is basically not a great search implementation. That can work of course (provided your recall doesn't suck). But it's going to add some cost and latency to searches. And usually, agents do multiple searches.
But if your search is so poor, why bother with vector search at all? Especially dense vector search at scale is not cheap. If you are going to fetch lots of results, just use some cheap lexical searches. Sparse vector search might be a good compromise (higher cost to index but similar performance to lexical search).
- As a real dummy on the subject, maybe you could help me understand where vector search tends to fall over?
I use it to retrieve tool functions by description and it has worked very well for me, but I expect I'm in the "very simple use cases" category that you mentioned.
- Processing 500 retrieved chunks here (probably with an additional LLM) will also add a bit of latency (not shown in these graphs!).
So for user facing apps, that scenario is probably not feasible (more like filter 10 chunks). Which as the parent of this comment suggests is fine to add in extra context given the current size of context windows.
- I agree. We're seeing more variants of "RAG" that aren't semantic at all (e.g. coding agents or simple memory systems that feed summary indexes directly into context).
I think, over time, it's going to become a SQL / NoSQL sort of divide. There will be the right kind of RAG for the job and lots of forcing the wrong kind because the developer doesn't understand the nuances.
- I read “RAG Context” as “the retrieved content injected into the context window”
- So when an agent does "cat file.txt" that's RAG to you?
- It is to me. And I agree that the term is losing value because it's becoming ubiquitous but it's the differentiation from the first versions of ChatGPT etc., which were purely user input -> LLM -> output driven.
- In the context of the title of the hn submission, I read “RAG Context” as “the retrieved content injected into the context window” .... "read" read /rε d/ not /riːd/
> So when an agent does "cat file.txt" that's RAG to you?
No, that might be "RAG Context" though.
- The answer would be yes. It's about using stored knowledge to increase the accuracy of the answer and evidence surfacing.
It doesn't have to be a vector database.
Kapa is one of the few companies doing RAG right.
- RAG, as far as I understand, is a term that came about before LLM tool-calling was as prevalent. Your options were to have an LLM hallucinate up a response, or instead do a [document -> chunk -> embedding -> vector db -> query -> context window] pipeline. I haven't heard anyone talk of LLMs + web search or other tool calls as RAG, even though if you pull apart the semantics the term is applicable. In fact I don't hear people talk about RAG much at all. I suppose much of what people were trying to solve with document chunking/embedding pipelines has been solved with bigger models and tool calls. And along with that change in tooling we have left behind the term "RAG", which leaves it attached to the concept of those pipelines.
- I do wonder if the term will make a comeback, Retrieval Augmented Generation as a concept is a fairly fundamental idea, or maybe it's considered too generic which is why tool calling is used in favour if it?
The problem with tool calling is it's too generic in my mind, maybe RAG will make a return when we get around to having different flavours of it, digging into the rich vein that is Information Retrieval
Also is it just me that doesn't like this sort of wording?
> agent has to hold
I find it very generic, I'd much prefer process or recall or any term that indicates what the agent is doing with tokens in that context
- I think it’ll just become “agentic search” and “information retrieval” again because RAG is too intertwined with a particular kind of implementation/use case of basic document scoring + first gen vector dbs that is IMO undesirable for more sophisticated approaches to associate themselves with.
You need a lot more unstructured data than most typical “RAG” users doing document search are dealing with for it it to not be a solved problem, IMO (just give a tool calling agent your sql schema/directory structure). Even that is still an interesting problem for more typical use cases, but only at large scales where you start needing to do multiple passes or fan-out or convert data that could be structured like that into data that already is. I’m interested in large scale code search, coding agent context/conversation search, and network/trace analysis which has a lot of domain-specific considerations that make it interesting but definitely not structured like a typical “document chunking with cosine similarity” RAG implementation.
- > Am I wrong to be somewhat peeved by the use of "RAG" in these contexts?
Probably? The ship has sailed on this one, like it has on "AI" or "crypto". RAG has almost exclusively been used to describe some variation on vector-search + bm25 pushed into the prompt for the last few years.
- "Three knobs matter:"
I can't help reading articles with the radar on for signs of AI-generation nowadays. I have noticed that Claude sometimes uses the word knob for parameter, so here I get suspicious.
- I am being constantly mocked for using there term 'knob fiddling', but I am a British child of the 70's :-)
- [dead]
- The contribution here is the technique (which looks like a nice alternative to reranking) rather than the article or it being some kind of think piece, so does it really matter?
I was able to skim it and go from "lol they reinvented re-ranking" to "oh, that's something more interesting, I should try this one day", and the writing didn't get in the way.
- Would be cool to see a retrieval comparison to IE a claude code agent trace for the same query, even a cherry picked one.
- Did you experiment with the Pruner completely replacing the Reranker?
- Replacing the reranker with the pruner completely was not feasable since it would have to run on the 150-200 chunks we retrieve before reranking.
- Bad or only loosely related context can make the final answer worse than having less context.
- I think this is really where energy saving comes into play. Context is so incredibly energy and processing time sensitive.
- i wouldn't trust the small llm there.
it will be an intellectual bottleneck when it comes to processing the very information your just arduously fished out of the ocean!!
- I've had good success using a local model for preprocessing (tag extraction) and tuning the search parameters (hybrid search).
- The risk in relevance-based pruning is the same one summarization has: it's tuned to drop whatever's rare, and in a lot of domains the rare chunk is the whole answer. A contraindication, an "except when," the single row that contradicts the other forty. Those score low against the query precisely because they're phrased in the exception's language, not the question's, so a similarity cut throws them out first. The strongest version of the pro-pruning case is "prune by relevance to the query, not blindly," and I still don't trust it where being wrong is expensive, because relevance-to-the-query is exactly the signal that misses the buried caveat. If I ask whether drug A interacts with drug B and the warning is written under B's contraindications without naming A, semantic pruning helps me lose it. What's worked better for me is pruning by structure instead of by score: keep whole records or whole sections as units and drop by type (this entire category of source isn't relevant) rather than by ranking individual sentences. You give up some of the token savings of aggressive sentence-level pruning, but you stop silently deleting the one clause the answer depended on. If you must prune fine-grained, the honest test is recall on a set where the answer hinges on a rare chunk, not average-case QA where there's lots of redundancy to spare.
- "relevance-to-the-query is exactly the signal that misses the buried caveat."
You are exactly hitting the nail on the head. The underlaying problem of a pruner is to determine document relevance to a user query. And document relevance comes in different flavours:
1. Direct relevance: a document can be relevant to the user query because it directly answers it. 2. Subquery-Conditional Relevance: a document can be relevant because it answers a decomposed subquery of the user query. 3. Document-Interdependent Relevance: a document is relevant only because another document provides bridge context, domain knowledge, disambiguation, a definition or a constraint.
This means the right question is therefore not "is this document relevant to the original query in isolation?", but:
Given the original user query, the decomposed sub-queries, and all other retrieved documents, does this document contribute to form a complete and sufficient set of documents for answering the original query?
This is exactly why we tuned our pruner at kapa based on recall against this Document Relevance Model.
- [dead]
- [flagged]
- [flagged]
- [flagged]
- [flagged]
- tl,dr: They used a rubric to have the LLM grade the chunks on a Likert scale. I think this is a good way to coax numbers out of an LLM.
- Likert scale just doesn't work in LLM evals. My idea of 3/5 is different from your 3 and definitely different from an non-deterministic system's 3.
- That's what the rubric is for; it reduces the problem to NLP, which is the LLM's forte. The more objective you can make your rubric, the better.
- My dissertation used a Likert scale with ChatGPT-3 era LLMs, and it was both internally consistent over 5-6 runs on a given statement, and consistent with human raters. I don't think you can bat it away as simply "doesn't work"
- Cliche topic - from a few years ago (the "RAG is dead" vs "All You Need Is Advanced RAG" BS - it came in waves and cycles, spread by bots on social media networks).
"Pruning RAG Context" is trying to recycle the old stuff (again), presuming the reader is naive (implies kapa.ai is not going anywhere). The current cycles were "openclaw" (I think that died), now we are on "harnesses" - when that dies the paid social media bots will give you something else. Shell game.
Just declare / define dictionary as a variable in your prompt to carry forward (when you decide to continue using LLMs for certain things). Also either summarize or truncate history. 3-4 year old concept. Not a big thing.
- "RAG is dead" was declared because of ever expanding context windows.
However, resources consumed expand so much with context that I think there is a huge practical barrier there. In the beginning AI services were basically free so nobody cared but this is rapidly shrinking.
Personally I see more in a combination of RAG and live querying during the thinking process (e.g. by tools).
Also I don't think dumping any context that might be relevant into the model really helps accuracy. In my experience models just get lost when they get an overload of irrelevant stuff in their context and start overlooking the relevant parts even if it does fit the window.
- Shallow dismissal
139 points by emil_sorensen 22 hours ago | 39 comments