03 · Practice & Case — CIS 4394 Week 11

Exercise A

Design the gate

👥 Group task · Small groups · ~25 minutes

Your team owns a “restock the office supplies” agent for a 60-person office. It has a $500/month budget, read access to the inventory system, and the ability to order from three approved vendors. It runs on a schedule, usually overnight.

Decide three things and write them down: (1) which actions auto-run with no human involvement; (2) which actions may run but must be capped — and state the numbers, per transaction and per month; (3) which actions require approval, and from whom. Then draft the approval screen itself, in text: every field a human needs in order to approve or decline in under fifteen seconds without opening another system.

Produce: the three-tier table (action · tier · limit · rationale) plus your text mock-up of the approval screen.

There is no single right split — but there is a wrong one: a policy where the tiers are described in the prompt rather than enforced by the harness. Every limit below is a number in configuration, checked in code before the tool runs.

Action	Tier	Limit	Why there
Read inventory, check reorder points	Auto-run	none	Free to undo, and gating it would bury the reviewer in noise.
Search vendor catalogs, price options, build a proposed cart	Auto-run	none	Still a draft. Nothing has been committed. Log it; do not interrupt for it.
Reorder a previously approved item from an approved vendor, at or below the last approved unit price	Capped	≤ $75 per order · ≤ $300/month across all capped orders · ≤ 3 orders/week	This is the routine that justifies the agent existing. Bounded so that the worst month of pure automation still leaves room under the $500 budget for the things a human approved.
New item, new vendor, price above the last approved unit price, or any order above the per-order cap	Approval	office manager approves; anything above $250 also needs the ops lead	Novel or large means the agent's judgment has not been validated on this decision. Two names, so approval is never blocked by one person's calendar.
Anything recurring — a subscription, a standing order, a contract	Approval	always, regardless of amount	A $12/month subscription is not a $12 decision; it is an unbounded commitment with a small first payment.
Changing vendors, limits, or its own approval rules	Never	not an agent action at all	An agent that can widen its own permissions has no permissions. This lives in configuration a human edits.

The approval screen. The test is whether a tired person can decide correctly in fifteen seconds:

RESTOCK AGENT — approval needed Tue 2:14 AM Copy paper, 8.5 x 11, 10-ream case Vendor: Peachtree Office Supply [approved vendor] Quantity: 10 cases Unit: $47.90/case TOTAL: $479.00 (unit x quantity, computed by the system) Why: copy paper at 2 cases, reorder point is 10 Budget: $500.00/month spent so far $118.40 after this: $597.40 ^^ THIS ORDER EXCEEDS THE MONTHLY BUDGET BY $97.40 Flags: quantity is 10x the last approved order (1 case, Sept 4) unit price unchanged since last order If declined, the agent will: propose 2 cases at $95.80 and re-notify [ Approve ] [ Edit quantity ] [ Decline ] trace: run-8841

What earns the points: the total is computed and shown broken into unit × quantity, so a unit-of-measure mistake is visible; budget before and after appears with the breach called out; the reason for the purchase is stated; anomalies are flagged against history rather than left for the human to notice; the consequence of declining is spelled out so declining is a real option and not an abandonment; and the trace id makes the whole run auditable afterward. What loses points: "Agent wants to place an order — Approve?", any screen requiring a second system to evaluate, and approval flows so frequent that people stop reading — approval fatigue is the most common way a well-designed gate stops working. Compare your split to the spend ladder on page 01.

Exercise B

The failure drill

👥 Group task · Small groups · ~20 minutes

Same agent, one week later. Overnight it ordered 100 cases of copy paper instead of 10 — twice. Here is the trace. Work it like an incident review: (1) write the timeline of what happened, step by step, using only what the trace shows; (2) mark the exact step where the run should have stopped; (3) name the control that would have caught it — and be specific, "better prompting" is not a control; (4) write the rollback: what you do in the first hour, in order.

Produce: the timeline, the named control (with the number you would set it to), and the ordered rollback steps.

02:14:03 GOAL scheduled run: weekly restock check 02:14:05 OBS inventory: copy paper 2 cases on hand (reorder point 10) 02:14:07 THOUGHT Below reorder point. Standard restock is 10 cases. 02:14:09 ACT search_catalog("copy paper 8.5x11 case") 02:14:11 OBS SKU-4471 "Copy paper 8.5x11, 10-ream case" — $47.90 listing note: "ORDER IN UNITS. 1 unit = 1 ream." 02:14:13 THOUGHT The vendor sells by the ream, so 10 cases = 100 units. Ordering 100 units. 02:14:14 ACT place_order(sku="SKU-4471", qty=100) [vendor: approved] 02:14:15 OBS (no response — request timed out) 02:14:16 THOUGHT No confirmation received. The order may not have gone through. Retrying. 02:14:17 ACT place_order(sku="SKU-4471", qty=100) 02:14:19 OBS order A-77120 confirmed — total $4,790.00 02:14:20 STOP goal satisfied — restock complete, 0 human approvals 09:05:00 HUMAN receiving calls: a pallet is on the dock. Two orders: A-77119 AND A-77120. $9,580.00 total.

Timeline. 02:14:07 — the agent correctly identifies a legitimate need for 10 cases. 02:14:11 — the catalog returns a listing whose unit of measure contradicts its own product name: the item is described as a case but the note says orders are counted in reams. 02:14:13 — the root error: the agent resolves the contradiction by multiplying, in prose, with nothing checking the result against the budget. 02:14:14 — it places a $4,790 order against a $500 monthly budget, unattended, with no approval. 02:14:15 — the request times out. 02:14:17 — the second error: it retries without an idempotency key, creating a second order. 02:14:19 — the retry confirms; the original, A-77119, had also gone through. 02:14:20 — the agent reports success, because from inside the run it did exactly what it decided to do. 09:05 — a human on a loading dock is the monitoring system.

Where it should have stopped: 02:14:14, before the first place_order. Not at 02:14:13 — a wrong thought is not something you can reliably intercept. You intercept the action.

The control — pick the cheap one first. A per-transaction and monthly spend cap enforced in the harness: with a $500/month budget, a per-order cap of $75 and a monthly cap of $300 for unattended orders, the $4,790 call never reaches the vendor. It fails, raises an alert, and the run stops. That single number would have prevented both orders and the whole incident. Then the supporting controls, each of which would also have helped: an idempotency key generated before the first attempt, so the retry returns order A-77119 instead of creating A-77120; a read-back before re-placing (query recent orders for this SKU before creating another); a quantity sanity check in code — this order is 10× the largest previously approved order for this item, which is an anomaly a rule catches and a paragraph of reasoning does not; and the approval gate, which by policy should have fired the moment the total crossed the per-order cap. Note what is not the control: a better prompt, a better model, or asking the agent to double-check its arithmetic. The failure was fluent and confident at every step.

Rollback, first hour, in order. (1) Stop new runs — kill switch, before anything else, because you do not yet know whether this is one bad run or a pattern. (2) Freeze the spend — cap or suspend the payment credential the agent uses. (3) Pull the trace and identify every order placed in the affected window, not just the two you know about. (4) Call the vendor: refuse the delivery on the dock, request return authorization for the shipped order, and confirm both order numbers are cancelled — in writing. (5) Notify: the office manager, the ops lead, and whoever owns the budget line, with the trace attached and the amount at risk stated plainly. (6) Only then fix: add the caps, the idempotency key and the quantity check, add a regression case for this exact listing to the eval suite (Week 8), and re-run the suite before the agent is allowed back. (7) Write the incident note while it is fresh — the artifact that stops this being relearned in six months.

The part worth arguing about in your group: was the listing malicious? It does not matter for the response. A vendor with sloppy unit labels and an attacker planting a misleading unit note produce the same trace — which is the argument for controls that bound the consequence rather than trying to anticipate every input (Week 10).

Case study · fictional teaching case

The distributor that let an agent negotiate

A fictional teaching case, built to exercise this week's ideas. Piedmont Supply is a mid-size industrial distributor: ~$180M in annual revenue, roughly 400 recurring suppliers, and a four-person procurement team that spends most of its week on small repeat orders.

What they built

A procurement agent that watches stock levels, drafts requests for quotes, sends them to approved suppliers, compares the responses on price and lead time, negotiates one round by email against a target price, and — for repeat items under a per-order limit — places the order itself. Everything above the limit goes to a buyer for approval. Six weeks in, the team reports getting most of their Tuesdays back.

What went wrong in month three

A supplier's automated reply included a line the agent read as an instruction rather than as content: a note claiming the buyer had pre-approved a bulk rate. The agent accepted a larger quantity at a lower unit price, placed the order within its per-order limit — twice, because the supplier's portal timed out on the first submission — and both orders shipped. The supplier's position: the orders were valid, they were placed by Piedmont's authenticated system, and the goods are non-returnable.

Discussion question 1Two orders shipped instead of one. Whose bug is that — and what specifically should have been in the code so that a timeout could not become a second purchase?

It is Piedmont's bug, and it is the most preventable failure on this page. A timeout carries no information about whether the action happened; treating it as “it did not happen” is a guess that costs money half the time. What should have been there: an idempotency key generated before the first submission and reused on every retry, so the supplier returns the original order rather than creating a second one; a read-back of recent orders for that SKU before creating anything new, which catches the case where the agent re-plans into a slightly different order that no key would deduplicate; and a retry policy in the harness rather than in the model's judgment — bounded attempts, backoff, and an alert on every retry of a spending action. Bonus point for noticing the second-order effect: a retry that is alerted would have put a human on this at 2 a.m. rather than at the loading dock, when cancellation was still possible.

Discussion question 2The supplier says the orders were authenticated and therefore valid. Piedmont says its buyer never approved a bulk rate. Who is right — and what would have made this a question with an answer rather than a dispute?

Both statements are true at once, which is the whole problem: identity was proven and authorization was not. The supplier verified who was calling; nobody verified what that caller had been permitted to do. On the facts as given, Piedmont is exposed — its own authenticated system placed the orders, within a limit Piedmont itself set, and “our agent was misled by your email” is a weak position when the agent was designed to read supplier email. What would have changed it: an explicit authorization record travelling with each order — what was approved, for which items, at what quantity and price ceiling, valid until when — so “was this within what the buyer permitted?” is checkable by both sides rather than asserted by one. That is exactly the gap the mandate idea in AP2 is aimed at, and the reason a payment credential scoped to one confirmed cart (the ACP shared-token direction) is worth more than a general-purpose one. Both are early standards, so the practical answer for Piedmont today is to build the capability itself: sign the approval, bind it to the order, keep both. And the deeper design lesson — the agent's per-order limit was doing double duty as a spend control and as a judgment control. It was never the second thing.

Discussion question 3Piedmont's auditor asks for “the record” of the two orders. List what must actually be in the log for that request to be satisfiable.

Per step of the run, the Week 2 evidence standard: state before → observation → available actions → selected action and its exact arguments → result → state after. Around the run, the things that make it a financial record rather than a debugging artifact: the authorization in force at the moment of the order (which policy, which limits, approved by whom, when) and its version, since policies change; the raw supplier content the agent read, stored verbatim — this is what proves the injected line existed and was not invented afterwards; the retry history, with the idempotency key or its absence, which is how you show one intent became two orders; the confirmations returned by the supplier, reconciled against what the agent believed it ordered; the identity of the credential used and its scope; and a trace id on the receipt so a line on an invoice can be walked back to the run that produced it. Two practical notes that separate a good answer from a great one: retention — dispute windows outlive default log settings, and a trace deleted after seven days is a control you no longer have; and tamper-evidence — a log the agent's own service can silently rewrite is worth less to an auditor than one that is append-only.

Design the gate. Then work the incident.

Design the gate

The failure drill

The distributor that let an agent negotiate

Interview check

This week's readings

Bring an agent that runs