AWS Certified AI Practitioner Training Course

Retrieval Augmented Generation is the most important architectural pattern in applied generative AI. It solves the two limits established in module 2 — a model knows nothing about your private data, and it fabricates when it does not know — by retrieving relevant content at query time and supplying it to the model as context.

The pattern

RAG has two phases. The first happens ahead of time:

1. Ingest the source documents from wherever they live.
2. Chunk them into passages small enough to embed meaningfully and large enough to stand alone.
3. Embed each chunk into a vector using an embedding model.
4. Index the vectors, along with the original text and metadata, in a vector store.

The second happens per query:

1. Embed the user's question with the same embedding model.
2. Search the index for the chunks whose vectors are nearest the query vector.
3. Assemble a prompt containing those chunks as context, the question, and an instruction to answer only from the provided material.
4. Generate the response, ideally with citations back to the source chunks.

The model is unchanged throughout. Nothing is trained. The knowledge lives in the index, which is why updating the system means updating documents rather than retraining anything.

Why RAG is usually the right answer

Five properties explain its dominance:

• Currency. Update a document and the next query reflects it. There is no retraining lag.
• Attribution. Because the answer came from identified passages, you can cite them, and a user can verify. This matters enormously for trust and for regulated use.
• Reduced hallucination. Grounding the model in supplied text and instructing it to stay within that text substantially reduces fabrication.
• Access control. Retrieval is a point where permissions can be enforced, filtering to content the requesting user is entitled to see. Fine-tuning offers no equivalent — knowledge baked into weights cannot be selectively withheld per user.
• Cost. Indexing is cheap compared with training, and changes are incremental.

Where RAG systems actually fail

The common assumption is that RAG failures are model failures. Usually they are retrieval failures, and diagnosing correctly is the practical skill.

• The right passage was never retrieved. Poor chunking split it, the embedding model did not capture the domain vocabulary, or the query phrasing did not align with the document's. No model can answer from context it never received.
• Chunking destroyed the meaning. A table separated from its header, a procedure cut mid-sequence, a pronoun whose referent was in the previous chunk. Chunk size and overlap are real tuning parameters.
• Retrieval returned plausible but wrong passages. Semantically similar documents about a different product, region, or policy version. Metadata filtering — by product, date, or document type — is the usual remedy.
• Too much context was retrieved. Twenty marginally relevant chunks dilute the signal, cost more, and push relevant material into the middle of the context where it is attended to less reliably. Fewer, better passages beat more.
• The source itself was wrong or outdated. RAG faithfully surfaces bad documentation. Content quality is now a system property.
• The model ignored the context and answered from its own knowledge anyway. The remedy is an explicit instruction to answer only from the provided material and to state when it cannot.

Techniques that improve retrieval

Several refinements appear in practice and occasionally in exam wording. Hybrid search combines semantic vector search with keyword search, which recovers exact matches on product codes, error numbers, and proper nouns that embeddings handle poorly. Reranking retrieves a larger candidate set and then applies a more precise model to reorder it, sending only the top results to the generator. Query rewriting reformulates a vague or conversational question into a better search query. Metadata filtering constrains retrieval by attributes before similarity is considered. Contextual chunking attaches document-level context to each chunk so it remains interpretable in isolation.

RAG versus fine-tuning

This comparison is examined constantly, and the discriminator is what kind of gap you are closing. RAG supplies knowledge: facts, documents, current information, anything that changes. Fine-tuning shapes behavior: format, tone, style, domain-specific response patterns, and task-specific consistency. When a scenario says the model does not know something about your business, the answer is RAG. When it says the model knows the material but responds in the wrong shape, the answer is fine-tuning. When both problems are present, do both — they are complementary, not alternatives.

On AWS

You can assemble RAG yourself from an embedding model, a vector store, and orchestration code, which gives maximum control. Amazon Bedrock Knowledge Bases provides the whole pattern as a managed capability, and Amazon Q Business packages it further with enterprise connectors and permission-aware retrieval. The next two lessons take up the vector store choices and the managed service in turn.

What RAG does not fix

Two limits are worth stating so you do not over-apply the pattern. RAG does not make the model better at reasoning. If the answer requires synthesizing several retrieved passages into a judgment, that is a model capability question, and supplying more context does not compensate for a model that cannot do the reasoning. And RAG does not change the model's behavior — its tone, its output format, its adherence to your conventions. Those remain prompting and fine-tuning concerns.

There is also an honest cost. Retrieval adds a step before generation, so latency increases. Retrieved passages consume context window and are paid for as input tokens on every call. And the system now has an index that must be built, kept current, secured, and governed, which is real operational work rather than a one-time setup. None of this outweighs the benefits for knowledge-grounded applications, but a design that adds RAG to a use case with no knowledge gap has paid all of those costs for nothing — which is why the diagnosis in the previous lesson comes first.