Retrieval-Augmented Generation: How AI Applications Use External Knowledge

A language model can produce useful responses from what it learned in training. It knows nothing about your contracts, your policies or last week's support tickets.

Retrieval-augmented generation — usually shortened to RAG — is one architecture for closing that gap: find the relevant material first, put it in front of the model, and let the model answer from it.

What is retrieval-augmented generation?

The workflow is short to state. A question arrives. The system retrieves the material most likely to be relevant. That material becomes part of the model's context. The model generates a response grounded in it.

The model is unchanged. Nothing is retrained. The only thing that differs from a bare request is the information supplied alongside the question — which is why retrieval is usually a better first move than fine-tuning anything.

Why is retrieval useful?

Organisations hold information in documents, knowledge bases, databases, internal systems, websites and structured content. A retrieval layer selects from that body the part relevant to one particular request.

The alternative — putting everything into the model's context — is limited by the context window, costs more per request, and makes answers worse rather than better as irrelevant material crowds out the relevant passage.

How does retrieval actually find anything?

Many retrieval systems represent content as numerical vectors called embeddings. Text is converted into a representation that places similar meanings close together, so search can match on meaning rather than only on shared keywords. Asking about "time off" can then find a document about "annual leave".

Embeddings are not magic. They inherit the assumptions of whatever produced them, and a query phrased in unusual language can still miss.

Chunking is the other half. Large documents are split into smaller sections before indexing, and the size and boundaries of those sections shape retrieval quality more than most teams expect. Poor chunking produces missing context when a section is cut mid-argument, irrelevant retrieval when sections are too large to be specific, and incomplete answers when the material needed spans a boundary.

Why is retrieval not the same as truth?

Retrieving a document does not make its contents current, correct, complete or applicable to the question asked.

A system that confidently answers from a superseded policy is worse than one that finds nothing, because the answer carries the authority of a citation. Source quality, versioning and expiry are part of the architecture, not a content problem to solve later.

What does the architecture look like?

Documents are ingested and split into chunks. Each chunk is embedded and stored in a search index. At query time the question is embedded too, the index returns the closest matches, those become the context, and the model generates a response — ideally citing which chunks it used.

Two halves, two failure modes: retrieval can return the wrong material, and generation can misuse the right material. Measure them separately or you will not know which one is broken.

1. 01DocumentsIngested into the system.
2. 02ChunksSplit into smaller pieces.
3. 03EmbeddingsEach chunk is embedded.
4. 04Search indexWhere the embeddings are stored.

1. 01QuestionEmbedded the same way.
2. 02RetrieveThe index returns the closest matches.
3. 03ContextThose matches become the context.
4. 04GenerateThe model responds, ideally citing its chunks.

What should an engineer consider?

A production system has to address:

• Ingestion — how documents arrive, and what happens when one changes.
• Indexing — how often, and what it costs to rebuild.
• Retrieval quality — measured, not assumed. Does the right chunk appear in the results at all?
• Access control — the retrieval layer must respect the permissions the asker actually has. A system that can read everything will happily answer a question the user was not entitled to ask.
• Freshness — how stale the index is allowed to be.
• Source attribution — so an answer can be checked rather than trusted.
• Evaluation — of retrieval and generation independently.
• Latency — retrieval adds a step before the model starts.
• Security — retrieved text is untrusted input. A document can carry instructions aimed at the model.

The practical conclusion

RAG is not "put the documents into the AI". It is an information retrieval architecture with a generation step on the end, and its output is only as good as the retrieval beneath it.

---

Related reading: generative AI from models to applications, and AI agents.