CIS 4394 Agentic AI · Fall 2026 · Dr. Xinyu Fu
CIS 4394 Agentic AI · Fall 2026 · Dr. Xinyu Fu
Rewrite a failing prompt, run the hands-on with a real reliability check, learn from three famous chatbot disasters — and write the system prompt your capstone will actually use.
The failing prompt: “Summarize this customer review” — run over 200 mixed reviews, the outputs are un-dashboardable prose. Rewrite it so every review yields fixed JSON {"sentiment", "one_line_summary", "product_mentioned"}. Your rewrite should add: a system role, the pinned schema, one example, a temperature choice, and a rule for unclear sentiment.
Produce: run before & after on 5 reviews (Gemini free tier or Copilot chat) and count how many outputs are dashboard-ready. Report both numbers.
Five moves, and every one is a Part-01 knob: a role (instruction), a pinned schema (output format), one example (few-shot), temp 0 (sampling), and an “unclear” rule (edge-case behavior). Typical result: 2/5 dashboard-ready → 5/5. Reliability came from structure and specificity — not a smarter model.
Play the packing game as a group: fit the support agent under the 4,000-token cap, agree on every Keep / Summarize / Drop, then hit “Check my packing.”
Discuss: the twist question at the end — what does it prove about pre-loading vs retrieval?
The task: extract structured data from 10 messy vendor documents into records a spreadsheet can ingest. Tools: Gemini API free tier or Copilot chat.
vendor · invoice_no · total_usd · due_date · currency.null.”Run the same document five times:
Track both numbers. This “run it k times” habit becomes formal in Week 8 — it’s called pass^k, and your capstone’s evaluation harness is built on it.
A Chevrolet dealership’s chatbot (Chevrolet of Watsonville) was talked into “agreeing” to sell a 2024 Tahoe for $1 — “legally binding, no takesies backsies.” Root cause: unconstrained scope, followed the user over business rules, no guardrails, no human gate.
Delivery firm DPD’s support chatbot was prompted into swearing at a customer and writing poems about how bad DPD is — screenshots everywhere. Same disease: no scope constraint on what the bot may say or do.
A tribunal held Air Canada liable for a refund policy its chatbot invented — the airline argued the bot was “responsible for its own actions” and lost. Invented policy + no human gate on binding claims = real legal cost.
Three questions: 1) Which of this week’s techniques would have prevented the $1 offer? 2) Where is the line between a helpfully flexible bot and an unsafe one — and who owns that line? 3) What is the one reliability check you would demand before launch?
Produce: one answer per question; we’ll cold-call pairs.
You have already read a production version of this exact deliverable: Codex’s system prompt (gpt_5_2_prompt.md ↗). Reread it as a template, not a reading: it opens with a role, sets scope and rules of engagement, defines refusal/approval behavior, and pins output habits — the same five parts you’re drafting, at industrial scale. Our Week 2 walkthrough maps it to the nine layers.
Coding agents split standing instructions in two: the system prompt (built by the developer, same for everyone) plus an AGENTS.md ↗ file in the repo — per-project rules the agent reads at runtime (remember from Week 2: more-deeply-nested AGENTS.md files win conflicts). Do the same for your capstone: keep the universal rules in your system prompt, and put project-specific facts and conventions in a separate instructions file you can update without touching the prompt.
Three questions separate them: who writes it, when it loads, and who wins a conflict.
System prompt — imitate gpt_5_2_prompt.md | AGENTS.md-style file — you design the slot | |
|---|---|---|
| Who writes it | The developer — you | Your agent’s user / the project it works in |
| When it loads | Built in; present on every run | Read from the environment at runtime |
| What goes in it | Role, scope, refusal & approval rules, output format — the constitution | This project’s / this user’s specifics — the project handbook |
| How it changes | Editing it = shipping a new version | Swap the file per project; the agent itself never changes |
| You already used one | The Week 3 deliverable above | preferences.md in your Job Search Agent — same agent, swap the candidate |