Understanding RAG: Retrieval-Augmented Generation from Scratch

Large language models are impressive, but they have two stubborn limitations: they only know what was in their training data, and they happily make things up when they don't. Retrieval-Augmented Generation (RAG) fixes both by giving the model the right context at question time — so instead of answering from memory, it answers from your documents.

In this post I'll walk through what RAG actually is, why it works, and how to build a pipeline end to end.

The core idea

A RAG system answers a question in two steps:

Retrieve — find the chunks of your data most relevant to the user's question.
Generate — hand those chunks to the LLM as context and ask it to answer using only that context.

That's it. The "intelligence" is still the LLM; RAG just makes sure it's reading the right page of the right book before it speaks.

Question ─▶ [Retriever] ─▶ relevant chunks ─▶ [LLM + prompt] ─▶ grounded answer

Step 1: Chunking

You can't embed a 40-page PDF as one vector — you'd lose all the detail. So you split documents into chunks (say 300–800 tokens each), usually with a small overlap so ideas that straddle a boundary aren't cut in half.

function chunkText(text: string, size = 800, overlap = 100) {
  const chunks: string[] = [];
  for (let i = 0; i < text.length; i += size - overlap) {
    chunks.push(text.slice(i, i + size));
  }
  return chunks;
}

Good chunking matters more than people expect. Split on semantic boundaries (paragraphs, headings) where you can — a chunk that contains one coherent idea retrieves far better than one that ends mid-sentence.

Step 2: Embeddings

An embedding turns a piece of text into a vector — a list of numbers that captures its meaning. Texts about similar things end up close together in vector space, even if they share no exact words.

const { embedding } = await embed({
  model: google.textEmbeddingModel("text-embedding-004"),
  value: chunk,
});
// embedding -> [0.021, -0.44, 0.13, ...]  (hundreds of dimensions)

You embed every chunk once, up front, and store the vectors in a vector database (pgvector, Pinecone, Upstash Vector, etc.) alongside the original text.

Step 3: Retrieval

When a question comes in, you embed the question with the same model, then ask the vector store for the nearest chunks — typically by cosine similarity.

const { embedding: queryVec } = await embed({
  model: google.textEmbeddingModel("text-embedding-004"),
  value: userQuestion,
});

const topChunks = await db.query(
  `SELECT text FROM documents
   ORDER BY embedding <=> $1
   LIMIT 5`,
  [queryVec]
);

Because both the question and the chunks live in the same vector space, "how do I reset my password?" will match a chunk titled "Account recovery" — no keyword overlap required. That's the magic semantic search gives you that plain LIKE '%...%' never could.

Step 4: Grounded generation

Now assemble a prompt that includes the retrieved chunks and a firm instruction to stay inside them:

const prompt = `Answer the question using ONLY the context below.
If the answer isn't in the context, say "I don't know."

Context:
${topChunks.map(c => c.text).join("\n---\n")}

Question: ${userQuestion}`;

const { text } = await generateText({
  model: google("gemini-2.5-flash"),
  prompt,
});

That single instruction — "if it isn't in the context, say you don't know" — is what turns a confident hallucinator into a trustworthy assistant.

What actually makes RAG good

Getting a demo working is easy. Making RAG reliable is where the real engineering lives:

Chunking strategy. Too big and retrieval gets noisy; too small and you lose context. Tune it against real questions.
Retrieval quality. Add a re-ranker to reorder the top-k, or hybrid search (combine semantic + keyword) for queries with exact terms like error codes.
Metadata filtering. Store tags with each chunk (tenant, doc type, date) and filter before the vector search so users only retrieve what they're allowed to see.
Citations. Return the source chunk with the answer. It builds trust and makes hallucinations obvious.
Evaluation. Keep a set of question/answer pairs and measure retrieval hit-rate and answer quality every time you change the pipeline.

Conclusion

RAG isn't a model — it's an architecture. You embed your knowledge once, retrieve the relevant slice at question time, and let the LLM reason over it. The result is an assistant that answers from your data, cites its sources, and admits when it doesn't know.

Start simple: chunk, embed, retrieve top-5, ground the prompt. Then iterate on the parts that move your accuracy — usually chunking and retrieval, rarely the model itself.