Context retrieval is the process of selecting and fetching the pieces of information, such as passages, records, conversation history or tool results, that are placed into a language model’s context window to help it answer a particular request. A model can use only what is in its training and in its context, and the context window is limited and costly. Deciding what to put there, so that the model sees what it needs and little else, is one of the most important design tasks in applications built on large language models.
Relation to other concepts
Context retrieval is the practical core of retrieval-augmented generation, but it is broader. It includes fetching documents through document retrieval, recalling memories and earlier conversation turns, loading user profiles and settings, calling tools to obtain live data and selecting examples for few-shot prompting. The overall craft of choosing what the model sees is sometimes called context engineering, an extension of contextual prompting.
Sources of context
- Document collections: manuals, policies, reports and web pages in a knowledge base.
- Structured data: database rows, spreadsheets and records from a knowledge graph.
- Conversation history: earlier turns, summarized when long.
- Long-term memory: facts about the user or project stored between sessions.
- Tool and application programming interface results: calendars, search engines, code execution and business systems.
- Environment information: the date, location, device or the state of a task.
How the process works
- Understand the request: interpret the question, resolve references to earlier turns and decide what information is needed.
- Generate queries: rewrite the request into one or several search queries, or choose which tools to call.
- Retrieve candidates: use keyword, semantic or hybrid search over indexed content.
- Rank and filter: rescore results with reranking, apply permissions and freshness rules and remove duplicates.
- Assemble the context: place the selected material into the prompt in a clear structure, with labels and source identifiers.
- Generate and cite: the model answers from the supplied context and references its sources.
- Evaluate and iterate: check whether the answer was supported and adjust retrieval.
Deciding how much to retrieve
More context is not always better. Too little leaves gaps and invites guessing, while too much adds noise, raises cost and delay and can dilute attention. Studies show that models may overlook details in the middle of very long inputs. A practical approach is to retrieve a generous candidate set, rerank it and keep only the passages that fit a token budget, placing the most important near the start or end. Adaptive systems retrieve more when a question is complex or when the first pass seems insufficient, and skip retrieval when the model can answer reliably alone.
Techniques to improve quality
- Query rewriting and expansion: produce several phrasings to catch different wording.
- Contextual chunk enrichment: add a short note about where a chunk comes from, as in contextual retrieval, which can reduce missed results.
- Parent-child retrieval: match on small chunks, then supply the larger section for understanding. See chunking.
- Metadata filtering: restrict by date, product, language or user permissions.
- Iterative retrieval: let the model ask follow-up queries after reading initial results, common in AI agents.
- Compression: summarize or extract only the relevant sentences from retrieved passages.
- Diversity: avoid filling the window with near-duplicates.
- Source quality scoring: prefer authoritative and recent sources.
Risks
- Missing context: retrieval fails to find the key passage, leading to a wrong or invented answer, a form of hallucination.
- Conflicting context: retrieved sources disagree, and the model blends them.
- Stale information: outdated documents are returned.
- Privacy and access leaks: content is shown to a person who should not see it.
- Injected instructions: retrieved text contains commands that try to control the model, see prompt injection.
- Cost and latency: each retrieval step adds delay and expense.
Evaluation
Teams evaluate retrieval separately from generation. For retrieval, they check whether the passages needed to answer appear in the results, using recall at k and ranking metrics on questions with known sources. For the final answers, they check faithfulness to the supplied context, completeness and citation accuracy. They also track the proportion of requests where retrieval returned nothing useful, and review those cases to improve content or search settings.
A worked example
A support assistant for a software product receives the question: why does my export fail after the update. The system rewrites the question into two queries, one about export errors and one about the latest release notes, retrieves the top results from the help center and the release notes, reranks them and keeps four passages that total about 1,500 tokens. It also loads the customer’s product version from the account system. The model answers that a known issue in version 4.2 affects large exports and that the fix is in version 4.2.1, citing the release note. Without retrieval, the model would have offered generic troubleshooting.
Memory and personalization
Assistants that serve the same person over time keep a store of facts and preferences and retrieve the relevant ones for each request. A good memory system decides what is worth saving, avoids storing sensitive details without permission, lets users view and delete entries and retrieves only items that change the answer. Showing every stored fact in every conversation would waste context and could feel intrusive, so selection is as important for memory as for documents.
Context retrieval for agents
Agents that work over many steps accumulate tool outputs, plans and intermediate results that quickly exceed the window. They use context retrieval to decide which earlier results to keep in view, which to summarize and which to write to external storage and fetch later on demand. Well-designed agents fetch information just in time through search and file tools, instead of loading everything up front, which keeps the window focused on the current step.
Design checklist
Teams building a context pipeline benefit from a short checklist: define the questions the system must answer, inventory the sources and their owners, set freshness and permission rules, choose chunking and indexing methods, add reranking and a token budget, log what was retrieved for every answer, test with real questions and review failures weekly. Logging is especially valuable, because it lets a team see the exact context the model received when an answer turns out wrong.
FAQs
Is context retrieval the same as RAG? RAG is the best-known pattern. Context retrieval also covers memory, tools and other information supplied to a model.
Does a longer context window remove the need for retrieval? Rarely. Selecting relevant information is usually cheaper, faster and more accurate than including everything.
How do I know whether retrieval or generation is the problem? Check whether the needed passage was retrieved. If it was and the answer is still wrong, the problem lies in generation or prompting.
More in Retrieval and Knowledge Systems
Key facts
- Lewis and colleagues introduced retrieval-augmented generation in 2020, in which a retriever fetches passages that a generator then uses to produce its answer.1
- Liu and colleagues found in 2023 that language models use information placed in the middle of a long context less reliably than information at the beginning or end.2
- Anthropic described a contextual retrieval technique in 2024 in which each chunk is prefixed with a short explanation of its place in the document before indexing, reducing failed retrievals.3
Related concepts
Quick checkWhat is context retrieval?Show answer
It is selecting and fetching the information, such as passages, records, history or tool results, that is placed into a model's context window so that it can answer a specific request accurately.
Sources
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, https://arxiv.org/abs/2005.11401
- Liu et al., Lost in the Middle: How Language Models Use Long Contexts, https://arxiv.org/abs/2307.03172
- Anthropic, Introducing Contextual Retrieval, https://www.anthropic.com/news/contextual-retrieval
Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue
