CIS 4394 Agentic AI · Fall 2026 · Dr. Xinyu Fu
CIS 4394 Agentic AI · Fall 2026 · Dr. Xinyu Fu
The model was trained on text that stopped at a cutoff date and never included your documents. RAG closes that gap without touching the weights: find the passages that answer this question, paste them into the prompt, and require the answer to come from them — with a citation (Lewis et al., 2020).
Everything the model "knows" on its own is compressed into its weights during training. Three consequences follow, and none of them are fixable by prompting harder:
It is frozen. Whatever changed after the training cutoff is invisible. It never saw your data. Your contracts, tickets, and policies were not in the training set, and you would not want them to be. It has no sources. Parametric knowledge is a blend, not a filing cabinet, so the model cannot tell you which document a claim came from — and when it is unsure, fluent guessing looks exactly like knowing.
Lewis et al. (2020) paired a generator with a retriever over an external corpus, so the knowledge lives in documents you control rather than in frozen parameters (arXiv:2005.11401 ↗). Dense passage retrieval — matching questions to passages by learned embeddings rather than keywords — is the retrieval half of that design (Karpukhin et al., 2020, arXiv:2004.04906 ↗).
Fresh: update a document, and the next answer changes. Private: the corpus stays yours. Attributable: the answer can name the chunk it used, so a human can check it.
Retrieval has to find the right passage when the user's words are not the document's words. A customer types "can I send this back?"; the policy says "return window." Zero words in common — and a keyword search returns nothing useful.
An embedding model converts a piece of text into a list of numbers — a vector, typically several hundred values long. The model is trained so that text with similar meaning lands in similar directions, which turns a language problem into a geometry problem: to find related text, find nearby vectors.
Nearness is usually measured with cosine similarity — the angle between two vectors, reported on a scale where 1.0 means pointing the same way and 0 means unrelated. "Retrieve the top k" simply means: embed the question, then return the k stored chunks whose vectors sit at the smallest angle to it.
Two practical consequences you will meet in the exercises. First, the question and the documents must be embedded by the same model — vectors from two different models are not comparable, so swapping the embedding model means re-embedding the entire corpus. Second, similarity is not truth: the nearest chunk is the one that talks about the topic most, which may well be last year's version of the policy. Freshness has to be enforced by metadata, not by geometry.
Six stages, split across two moments in time. The top lane runs offline, once per document. The bottom lane runs live, on every question. Click any box.
A fictional teaching example: Lakeside Supply Co., a retailer with a small policy corpus. A support rep asks about a return. Press Step and watch the question travel through the pipeline. Similarity scores are illustrative.
Retrieval alone does not stop a model from inventing things. Three instructions turn retrieved text into an actual constraint, and they belong in the augmented prompt every single time:
Then verify: a groundedness check compares each claim in the answer against the retrieved text, and a low-confidence result routes to a human gate — the same gate you placed on irreversible actions in Week 4, now placed on assertions.
Classic RAG retrieves on every question, whether or not retrieval helps. Agentic RAG makes retrieval a tool, and hands the decision to the model inside the loop you built in Week 4.
"Do I need to look this up?" A greeting, a formatting request, or arithmetic needs no search — and searching anyway costs latency and drags irrelevant text into the context. A question about your refund policy always needs a search, and the model should refuse to answer it from memory.
If the first retrieval comes back weak, the agent can rewrite the query and try again — dropping jargon, adding a synonym, or narrowing by metadata. That is the Observe → Reason → Act loop pointed at the index instead of at the outside world.
Multi-hop questions need more than one retrieval: find the customer's region, then find the policy for that region. One shot at top-k cannot do this; a loop with a stopping condition can. Each hop is a step you can cap and log.
These are not competitors — they change different things. Retrieval changes what the model sees. Fine-tuning changes what the model is.
| RAG (retrieval) | Fine-tuning | |
|---|---|---|
| What it changes | The input: relevant passages are added to the prompt at question time. The weights are untouched. | The weights: continued training on your examples bakes behavior into the model itself. |
| Best for | Knowledge — facts, policies, documents, anything the model must be correct about. | Behavior — tone, format, house style, a narrow repeated task where you want the output shape without a long prompt. |
| Freshness | Immediate. Replace the document, re-index that document, done. | Frozen at training time. New facts require a new training run. |
| Traceability | The answer can cite the chunk and version it used. | None. A fine-tuned claim cannot be traced to a source document. |
| Cost to update | Low and incremental — an indexing job, not a training job. | High and lumpy — curated examples, compute, and a full evaluation pass to check nothing else regressed. |
| Access control | Enforceable at retrieval time: filter by the user's permissions before anything enters the prompt. | Not enforceable. Whatever went into the training data is in the model, for every user. |
On the reasoning patterns page you ran the weights visualizer: prompting left all twenty edges of the toy network untouched, while fine-tuning rewrote them. That picture is the whole argument on this page. Retrieval is a very sophisticated way of changing the input — which is why it can be fresh, private, and citable, and why fine-tuning can be none of those things.
Retrieved chunks are tokens, and they compete with everything else in the packing game. Raising k from 3 to 20 does not make answers twenty times better; it dilutes the prompt with near-misses and pushes you toward the recall problems of an overstuffed window. Retrieve narrow and relevant, not wide and hopeful.
The canonical whiteboard explainer, about 6½ minutes: the two failure modes RAG addresses — no source, and out-of-date knowledge — and the retrieve-then-generate loop.
About 8½ minutes on the step this page ends with: retrieval as a tool an agent chooses to call, re-query, and chain — the bridge from a fixed pipeline to the loop you built in Week 4.