Embeddings, Vector Stores, and Why RAG Works

An agent gets asked a question its documents clearly answer, and it comes back with nothing. Or worse, it comes back confident and wrong, having pulled a chunk that's topically close but actually irrelevant. Nobody touched the model. Nobody touched the prompt. The retrieval step just quietly handed the model the wrong material to work with.

RAG (retrieval-augmented generation) is the practice of pulling relevant documents into a model's context at query time, using embeddings, numeric vectors that capture meaning, stored in a vector database and matched by similarity search. It grounds an agent in real, current documents instead of only whatever it memorized during training. Here's the mechanism underneath that sentence, and why most "the model is bad at this" complaints are actually chunking problems.

Key Takeaways

• RAG retrieves documents at query time instead of baking knowledge into the model's weights, which makes updates as cheap as re-indexing a file.
• Embeddings are vectors: numbers that place similar meanings close together in space, so a query and a relevant chunk end up near each other even without sharing exact words.
• A vector store's job is fast similarity search over millions of embeddings, not storage alone.
• Most retrieval failures trace back to chunking, not the model or the embedding model itself.
• RAG and a bigger context window aren't rivals; they solve different problems and increasingly get used together.

What Is RAG, and When Does It Beat a Bigger Context Window?

RAG retrieves the specific documents relevant to a question and hands them to the model at generation time, instead of relying on the model's training data or requiring every possible document to already sit in the context window. That distinction matters more than it sounds.

A bigger context window is tempting: paste everything in, let the model sort it out. And for a fixed, small set of documents that fits comfortably, that works fine. But three things break down as your document set grows.

Cost climbs, since you pay for every token in the context on every single call, whether or not it's relevant to this question. Freshness suffers, since anything pasted into context is a snapshot; the moment a document changes, you're re-pasting the whole thing again. And precision suffers too: models don't attend evenly across a long context, so a fact buried in the middle of ten thousand tokens is genuinely more likely to get missed than the same fact retrieved as one focused chunk.

The 2026 consensus among people building production agentic AI systems isn't "RAG replaces long context" or the reverse, it's that the two are complementary. Long context is good at holding a conversation's own history, where you already know everything in there matters. RAG is good at pulling in outside knowledge, at exactly the moment it's needed, from a corpus too large to ever fully paste in, a point Atlan's 2026 breakdown of context engineering versus RAG makes well. You'll see this pattern again in Module 19, where an agent decides what to fetch just-in-time rather than front-loading everything either way.

What Are Embeddings?

An embedding is a list of numbers, a vector, generated by a model to represent the meaning of a piece of text, such that texts with similar meaning end up as vectors that sit close together in that numeric space. That's the whole trick: turn meaning into geometry, so "finding related content" becomes "finding nearby points."

This is why a query like "how do I cancel my plan" can retrieve a document titled "ending your subscription" even though the two share almost no exact words. Older, word-level embedding methods assigned one fixed vector per word and had no way to represent that connection well. Modern transformer-based embedding models generate a vector for a whole chunk of text at once, accounting for context, so the same word can contribute differently to the vector depending on what surrounds it.

You don't need to train an embedding model yourself for a typical RAG pipeline. You call one (through an API or a small local model), get a vector back for each chunk of your documents, and store those vectors for later comparison against a query vector generated the same way. Pick a mismatched embedding model between indexing and querying, though, and every comparison silently breaks: the two vector spaces aren't guaranteed to line up, so nothing retrieves correctly even though nothing throws an error.

How Embeddings and Vector Stores Find the Right Chunk

A vector store's job is to take a query's embedding and, out of possibly millions of stored embeddings, find the handful that are actually closest to it, fast. Comparing a query against every single stored vector one by one would work correctly but scale terribly, so vector stores build an index, a structure designed specifically to make that nearest-neighbor search fast without checking every vector individually.

Two vectors count as "close" by a similarity measure, most commonly cosine similarity, which looks at the angle between them rather than raw distance. The result of a query is a ranked list: the chunks whose embeddings sit nearest to the query's embedding, typically the top three to ten, depending on how much context you want to hand the model. Skip building a real index and just loop over every vector at query time, and it still works on a hundred documents; it grinds to a noticeable crawl once that corpus reaches the size RAG actually gets used for.

Chunking Is Where RAG Actually Breaks

Most "my RAG is bad" complaints are actually chunking problems wearing a model complaint's clothes: how you split documents before embedding them determines whether retrieval can ever succeed, regardless of which embedding model or which language model you use downstream.

The naive approach splits documents into fixed-size blocks, say five hundred tokens each, regardless of where sentences or ideas actually end. That's fast and requires no judgment calls, but it routinely slices a sentence in half or crams two unrelated topics into one chunk, a failure mode Atlan's survey of chunking strategies confirms shows up consistently across production RAG pipelines. When that happens, the resulting embedding becomes a kind of blurry average of everything in the chunk, matching neither topic particularly well, and a query that should have found it easily comes back empty or ranks it too low to surface.

Tuned chunking respects the document's actual structure, splitting at paragraph or section boundaries and keeping each chunk to one coherent idea, sometimes with a little overlap between neighboring chunks so a fact right at a boundary doesn't get orphaned. I've watched the same document set, indexed both naive and tuned, retrieve identical queries at meaningfully different precision, and the giveaway every time was the same: the naive version's near-misses were chunks that technically contained the right words but split them across an awkward boundary, not chunks that lacked the answer entirely.

That's the practical implication worth sitting with: before you blame the language model for a bad RAG answer, check what chunk it actually received. Frequently the right information genuinely wasn't in the chunk it got handed, and no amount of prompting the generation step fixes a retrieval step that already lost the plot.

When RAG Is Overkill

RAG earns its cost when your documents are large, change often, or need to stay attributable to a real source, but it's genuine overhead for anything smaller or more static than that.

Two clear wins: a support agent answering questions against a product's documentation that gets updated weekly, where re-indexing a changed page is far cheaper than retraining or manually re-pasting everything into every prompt; and a research assistant working over a corpus of hundreds of documents too large to fit in any context window at all, where retrieval isn't a nice-to-have, it's the only way the task is possible.

One clear skip: a small, fixed set of five or six reference documents that rarely change and easily fits inside a modern context window. Here, RAG adds an indexing pipeline, a vector store to run, and a retrieval step that can fail, all to solve a cost and freshness problem you don't actually have yet. Pasting the documents directly is simpler, and simpler is correct when it's genuinely sufficient.

Where RAG Fits Next

RAG is one context strategy among several, not the default answer to every "the model doesn't know this" problem. Static RAG, retrieving once per query, is the version covered here, and it's exactly what you'll build in the next lesson. The retrieved chunks you hand back to the model are just another input, so the same discipline behind validating structured output applies here too: a retrieval step that silently hands back the wrong chunk is no different from a pipeline that silently accepts a malformed field. Later in this module, that same pipeline gets upgraded so the agent decides for itself whether to retrieve, what to retrieve, and whether one retrieval pass was even enough, which is a meaningfully different and more capable design than what naive RAG can do alone.

Your Lab: Measure Naive vs. Tuned Chunking