0
0
Latest models are shipped with ever increasing context windows. Just for context, a 1 million token translates to about 750 thousand words. Yeah! that’s like 10-15 books. Obviously, with that much of context, what is the purpose of RAG (retrieval augment generation) and you would frequently hear folks in AI engineering declare:
RAG is dead
It does sounds reasonable at first. If an LLM can hold millions of tokens in context, why bother building retrieval pipelines? Why chunk documents, create embeddings, maintain indexes, and retrieve relevant information when you can simply give the model everything?
But taken to its logical conclusion, that argument gets strange very quickly.
If fitting more information into context makes retrieval obsolete, why stop at your company’s documentation? Why not put the entire internet into the prompt and kill search engines too? Now, that sounds absurd.
The ability to load more information into memory does not eliminate the need to find the right information.
Long context changes what retrieval systems can do. It does not make retrieval unnecessary.
Bigger context is not the same as better knowledge
The conversation around long context often treats context windows as if they were databases but a context window is temporary working memory. A database is a system for storing, organizing, filtering, updating, ranking, and attributing knowledge. Those are fundamentally different jobs.
For a small set of documents, long context can absolutely simplify an application. Instead of retrieving five chunks from a 20-page document, you might just send the entire document to the model.
That is useful.
But most serious AI applications don’t operate on 20 pages of information.
They operate on enormous and continuously changing knowledge bases.
A financial system might need years of filings, earnings transcripts, market data, research reports, and news. A software engineering assistant might need millions of lines of code, documentation, tickets, pull requests, and architectural decisions. Legal systems work across contracts, statutes, regulations, opinions, and case law. Medical applications may need clinical literature, guidelines, patient records, and drug information.
In these environments, the question isn’t: Can the model accept more tokens?
The question is: Which information should the model see for this particular task?
That is a retrieval problem.
There are few other considerations.
Latency: Long context creates a user-performance problem
Users notice when systems take too long to respond, particularly in interactive workflows such as search, coding, customer support, research, and productivity applications. Passing enormous contexts into a model means the system has more tokens to process before it can produce an answer.
Even if inference continues to get faster, sending 500,000 tokens when only 5,000 are relevant is still unnecessary work.
A good retrieval system narrows the problem before asking the model to reason about it.
Instead of saying:
Here are 10,000 documents. Figure it out.
the system says:
Here are the 10 pieces of information most likely to matter.
That difference translates directly into better responsiveness.
And responsiveness is not merely an infrastructure concern. It is a product feature.
Cost: Long context is expensive
The economics matter, unless you are the company behind the frontier model.
One of the dominant assumptions behind the rapid adoption of AI is that inference costs will continue falling. But falling token prices don’t mean token efficiency stops mattering. If models become 10x cheaper but applications respond by sending 100x more tokens, the economics don’t necessarily improve.
Retrieval lets applications spend their inference budget where it matters. Consider two approaches to answering a question against a large enterprise knowledge base. One system sends hundreds of thousands of tokens to the model on every request. The other performs retrieval first and sends only the most relevant evidence.
Even with extremely cheap models, the second architecture has an inherent efficiency advantage. Even if you save only a few cents, multiply that to enterprise scale it quickly becomes large infrastructure costs.
In many cases, RAG makes good retrieval even more economically valuable.
Scalability: Data growth will outpace context growth
Context windows will continue getting larger, so will data.
Companies generate enormous amounts of new information every day: documents, emails, code, support tickets, meeting transcripts, telemetry, research, customer interactions, contracts, and internal knowledge.
Scientific literature grows. Legal precedent grows. Financial information grows. Software repositories grow.
There is no obvious end state where a context window becomes so large that an organization can simply place its entire information universe inside every prompt. Even if such a model existed, doing so would rarely make architectural sense.
Retrieval is how bounded reasoning systems interact with unbounded information.
Agents: The future is agentic, and agents need retrieval
Future of AI engineering is ever evolving nature of information processing and acting on it. Agents execute tasks and tasks need sharp context, not the entire information load. An agent investigating a production outage might begin with monitoring data, inspect application logs, search a codebase, read recent deployments, look at related incidents, and then examine documentation.
It cannot know in advance which information it will need. Trying to preload every possible source would defeat the entire purpose of an adaptive system. A right architecture is to provide the agents an ability to ask progressive questions, including figuring out what information it needs next to make a decision.
This turns RAG from a preprocessing technique into a core reasoning primitive. Future retrieval systems may therefore look less like today’s basic vector-search pipelines and more like intelligent information navigation.
An agent might decide dynamically whether to:
- Search semantically,
- Query a database,
- Inspect structured records,
- Search source code,
- Browse documentation,
- Retrieve historical conversations,
- Call an API,
- Recursively investigate related evidence.
In an agentic world, RAG is more sophisticated.
Needle in the haystack: Bigger haystacks make it harder
There is another problem with the “just put everything in context” strategy. More context also means more noise. LLMs are remarkably capable of reasoning across long inputs, but their ability to identify and use relevant information is not perfect. In my own experiments, I see some models hallucinating more when the context window grows. Even when a model has to select the right tool to call, it matters if it has 20 tools at its disposal or 2000. In fact, I wrote a RAG pipeline, just to preselect best tool candidates before presenting it to model.
When critical information is buried among enormous amounts of unrelated text, models can overlook it, underweight it, or become distracted by irrelevant evidence.
This is the classic needle-in-the-haystack problem. Making the haystack larger doesn’t eliminate the problem.
But wait: RAG isn’t perfect
Yes, there are legitimate criticism hidden inside many “RAG is dead””” arguments.
RAG systems rely on naive chunking and cosine similarity. There is zero understanding of the domain. Semantic similarity isn’t a replacement for structured search. What about document hierarchies, metadata, entities, relationships, recency, permissions, or source quality.
They return arbitrary top-k chunks and hope the model figures out the rest.
And that’s where I propose an evolved architecture. RAG pipeline can navigate the knowledge base starting with some sort of “table of contents”, navigate to find semantic similarity of sections, follow that up with decision tree that offers the entire section or even entire report instead of chunked fragments.
If retrieval identifies the correct repository, contract, research paper, or case, the model may then consume the entire source instead of another layer of chunk retrieval. I won’t call this a death of RAG. It’s evolution. Retrieval is more about selecting the right information boundary rather than squeezing information into tiny context windows.
Instead of retrieving passages, retrieve documents. Instead of retrieving documents, retrieve datasets. Instead of blindly querying an embedding index, agents may dynamically choose between search, SQL, knowledge graphs, APIs, code search, and other retrieval mechanisms.
Long context makes retrieval systems better because they can retrieve larger, more coherent units of information.
RAG isn’t dead but its legacy architecture might be
Long context windows are one of the most important improvements happening in language models. They will eliminate unnecessary chunking, reduce architectural complexity, and allow models to reason over much richer sources of information. But they don’t eliminate retrieval.
The winning AI systems will likely combine both approaches: retrieval to identify the right information and long context to reason deeply over it and for agents interacting with massive, dynamic information environments, retrieval is indispensable.
