03 · Practice & Case — CIS 4394 Week 9

Exercise A

Decompose — or not?

👥 Group task · Pairs · ~20 minutes

You are handed a conference-planning agent. Its job: for a two-day departmental conference with a fixed total budget, choose a venue, arrange catering, invite and confirm speakers, and keep the whole thing inside budget. On paper or a whiteboard, draw it twice: (1) as a single agent with tools, and (2) as a supervisor with four workers. On both drawings, mark in one colour every piece of shared mutable state (anything two parts of the system both read and write) and in another colour every hand-off point (anywhere information has to travel from one box to another). Then apply the test out loud: do these subtasks run in parallel, and can each one be verified on its own?

Produce: both sketches, with shared state and hand-offs marked, plus a one-line verdict that names the test — e.g. "single agent, because X and Y are coupled through the budget and neither is verifiable alone."

The verdict most groups should reach: mostly do NOT split — but split one specific slice.

Where the test fails. The four "subtasks" are not independent; they are four views of one coupled plan. Budget is shared mutable state that all four write to — every speaker fee you commit is catering money you no longer have. Headcount and dates are shared too: the venue's capacity caps the invite list, the venue's kitchen rules constrain catering, and a speaker's availability can force a date change that invalidates the venue booking. Nor are the outputs independently verifiable: a catering plan is not "correct" in isolation — it is correct relative to a headcount, a room, and a remaining balance. Both halves of the test fail, which is the textbook shared-mutable-state case from page 02.

Where a split is genuinely justified. The read-only research inside each area does pass the test: "survey twelve candidate venues for capacity, date availability, and price", "look up availability and speaking fees for these ten people", "get quotes from six caterers." Those are parallel, and each result is verifiable against its own source. That is a fan-out of lookups, not a team of decision-makers.

The design that follows. One planning agent owns the plan and the budget — the mutable state stays in one place, with one owner and one trace. Underneath it, parallel research workers (or simply parallel tool calls, which at this scale is often the better engineering) return options with prices, never commitments. Commitments — booking, signing, spending — run through the single owner, behind a human gate, because they are irreversible and they move money (Week 4’s gate placement: after the action is chosen, before it executes).

Common misses to check your sketch for: drawing four workers with no line back to a budget box (the money then exists in four places at once); marking the hand-off arrows without writing what travels on them; and forgetting the date constraint, which is the coupling that surprises people — it links the venue and speaker "workers" that look most independent on the page.

Exercise B

Assign the blame

👥 Group task · Groups of three · ~15 minutes

Below is a trace from a fictional three-agent quarterly-reporting system. Read it carefully, then answer three things. (1) Which agent malfunctioned? (2) Which MAST category is the root failure — specification & system design, inter-agent misalignment, or task verification & termination — and which category is a secondary contributor? (3) Name the one log line you wish this system had recorded that would have made the failure obvious in seconds.

Produce: your category call with the exact line of the trace that proves it, plus your one missing log, written in the form "for every X, record Y."

Fictional teaching case. User request: "Summarize Q3 revenue by region for the board deck. Use the finalized figures only — the preliminary numbers in the shared drive are not approved for external use."

SUPERVISOR decompose → [1] retrieve Q3 revenue by region [2] write a 150-word summary HAND-OFF → data_agent: "Pull Q3 revenue broken out by region." DATA_AGENT search_drive("Q3 revenue") → Q3_revenue_prelim.xlsx (modified Oct 2, all 5 regions) → Q3_revenue_final.xlsx (modified Oct 9, all 5 regions) reasoning: "prelim file is complete and listed first; both have all regions." read_file("Q3_revenue_prelim.xlsx") → returns: total $42.7M · NA 19.1 · EMEA 11.4 · APAC 7.8 · LATAM 2.9 · Other 1.5 HAND-OFF → writer_agent: "Write 150 words summarizing these figures. {figures}" WRITER_AGENT drafts 148 words: "Q3 delivered a record $42.7M, led by North America…" self-check: length 148 ≤ 150 OK · all five regions mentioned OK SUPERVISOR review: "Summary is complete, on length, covers all regions." → DONE ACTUAL: finalized Q3 revenue was $39.4M. The board deck shipped with unapproved preliminary figures and the word "record", which was not true of the final numbers.

(1) Which agent malfunctioned? None of them. That is the whole point of the exercise, and it is the single most useful instinct to build this week. Given the instruction it received — "pull Q3 revenue by region" — the data agent's choice was defensible; it even explained itself. Given the figures it received, the writer's draft was accurate and on-spec. Given the summary it reviewed, the supervisor's review was true. Three correct agents, one wrong system.

(2) Root category: specification & system design (MAST category 1). The user's binding constraint — finalized figures only, preliminary not approved — appears in the user request and then in no payload anywhere in the trace. The proving line is the first hand-off: "Pull Q3 revenue broken out by region." The supervisor decomposed the task and dropped the constraint, so no downstream agent could have honoured it. A subagent's clean context window is clean of your requirements too.

Secondary: task verification & termination (category 3). The supervisor's review checked the summary against the summary's properties — length, region coverage — not against the user's requirement or the source of record. "Looks complete" is not verification; it is a rubber stamp. The writer's self-check has the same shape: both checked form, neither checked provenance.

Why it is not inter-agent misalignment (category 2) — the tempting answer. Category 2 is about agents contradicting each other, withholding information from each other, or derailing each other. Here the agents were perfectly consistent with one another; they were consistently wrong, because the same missing constraint was missing everywhere. A useful discriminator: category 2 failures show a disagreement somewhere in the trace. There is none in this one, and that is what makes it dangerous — nothing in the system's own output looks off.

(3) The one log you wish you had. The strongest single answer: "For every hand-off, record the payload verbatim alongside the user's original constraints, and flag any constraint that appears in the request but in no payload." That one line turns this failure into an automatic alert instead of a board-meeting discovery. Two other good answers, in order: "for every file read, record the alternatives that were available and the reason this one was selected" — the data agent's rationale was in its reasoning but was never carried anywhere a human could see it — and "for every 'done' verdict, record what was checked and against which source of record." All three are the Week 2 evidence standard applied at the seams: state before → observation → available actions → selected action → result → state after, logged per hand-off rather than only per agent.

The fix, for completeness. Carry constraints explicitly in every payload (the hand-off contract from page 01), verify against the source of record rather than against the artifact's shape, and note that a single agent reading the same folder in one context would very likely have seen both files and asked which one to use — a fair reminder that the split created the seam that lost the constraint.

Case study · fictional teaching case

Five agents, two memos, one problem

Meridian Ridge Capital is a fictional mid-sized investment-research firm. Last quarter it replaced its single research agent with a five-agent pipeline, and the demo was spectacular.

The architecture

An intake supervisor takes an analyst's question about a company and fans it out to four specialists: a filings agent (reads the last four quarterly reports), a macro agent (sector and rate environment), a news agent (press coverage and announcements from the last 90 days), and a memo writer that turns whatever comes back into a two-page investment memo in house style.

Throughput went up roughly fivefold. The memos are clean, well-organized, and confident. Analysts loved them for six weeks.

What broke

Two analysts asked about the same company two days apart and got two memos with opposite conclusions. One described margin expansion and a strengthening position; the other described margin compression and rising input costs. Both cited real evidence. Both read as authoritative.

The reconstruction: the filings agent had used the most recent filed quarter; the news agent had picked up a pre-announcement covering the quarter after it; the macro agent had summarized input costs at sector level while the filings agent had company-specific figures. Nothing in the pipeline ever compared the four specialists' outputs to each other. The memo writer's job was to write well — and it did, twice, from two different sets of inputs, with no idea the first memo existed.

The compliance question that followed was worse than the technical one: which memo did we send to a client, and can we reconstruct why it said what it said?

Discussion question 1Both memos cited real evidence and neither agent was wrong about its own slice. Why did the contradiction arise — and why did the system not notice it?

Context fragmentation. Each specialist held a different fragment of the picture and each drew a locally reasonable conclusion from the fragment it held: the filings agent's quarter and the news agent's quarter were not the same quarter, and the macro agent's cost picture was at a different level of aggregation than the filings agent's. Every conclusion was valid relative to its own inputs. There was no shared definition of the basic parameters the four disagreed about — which period, whose numbers, what level of aggregation — and none of them was written down anywhere, so no agent could have detected the mismatch even in principle.

The system did not notice for a structural reason, not a quality reason: nothing in the pipeline ever put two specialists' outputs side by side. The memo writer received four inputs and treated them as compatible because its job was writing, not reconciliation. Fluency then hid the seam — a well-written memo is more convincing, not less, when its premises conflict. Note the second-order failure too: the two memos were separated by two days, and nothing in the system carried memory of the first, so the contradiction was only ever visible to a human who happened to read both. This is exactly Cognition's argument in page 02: fragments produce conflicting decisions, and the conflicts are plausible.

Discussion question 2You may add exactly one verification gate. Where does it go, what does it check, and why is "add a reviewer agent that reads the memo" the wrong answer?

Put it at the join — before the memo writer, on the four specialists' outputs together. That is the only point in the pipeline where the contradiction is visible at all; everywhere upstream the fragments are individually consistent, and everywhere downstream the contradiction has been dissolved into fluent prose.

What it checks is the boring part, and it is not a vibe check. Reconcile the parameters the fragments disagree on: the as-of period each input covers, the source and its date, the level of aggregation (company vs sector), and the direction of the claim on any metric two agents both touch. Most of that is a deterministic comparison of structured fields, not a language judgement — which is exactly why the hand-off contract from page 01 asks each payload to carry evidence with sources rather than conclusions alone. On mismatch, the gate does not average the two views or pick one: it halts and surfaces the conflict to a human, with both sides shown. A conflict that a human resolves is a good outcome; a conflict silently resolved by a model is the failure repeating in a new place.

Why a reviewer agent reading the memo is the wrong answer: by then the evidence is gone. A reviewer sees a polished narrative, not the four sets of inputs, so it can only assess coherence — and the memo is perfectly coherent; that is the problem. An LLM asked "does this look right?" with no independent source of truth mostly agrees with what it is shown, which is MAST's task-verification failure mode precisely: verification that is present on the org chart and absent in substance. It also adds a full model call per memo and a comforting sense of safety, which is the worst combination available: cost with no signal.

Discussion question 3Meridian Ridge's CTO asks whether to fix the pipeline or collapse it back to one agent. What evidence decides it — and what would you tell the board?

Run the test, then run the ablation. Start with the test: parallel and independently verifiable? The four research pulls genuinely are parallel and each is checkable against its own source — that half holds. But the memo is a single coherent artifact whose correctness is a property of the whole, so the final stage was never independently verifiable, and that is the stage that failed. That asymmetry is the answer in miniature: keep the fan-out, collapse the judgement. Parallel retrieval into one context; one agent that reconciles and writes, holding all four fragments at once; the gate from question 2 in front of it.

The evidence that decides it is an eval set of real analyst questions with known-correct conclusions, run against three designs — the old single agent, the five-agent pipeline, and the collapsed hybrid — scored on conclusion accuracy and on contradiction rate across repeated runs of the same question (the pass^k intuition from Week 8: an architecture that gives different answers to the same question twice has a reliability problem no average score will show). Ablate one specialist at a time; any agent whose removal does not move the score is coordination cost with no payoff. Report cost per memo alongside, at real volume.

What to tell the board: the failure was not that the AI was wrong, it was that the system had no owner of the conclusion — four correct fragments and nobody responsible for whether they added up. Fluent output is not evidence, and throughput measured without a contradiction rate measured the wrong thing. The remedy is one accountable owner of the judgement, a reconciliation gate with a human on conflicts, and a per-memo trace that answers "why did it say that" — which, incidentally, is also the answer to the compliance question, and the reason the trace has to exist before the incident rather than after it.

Decompose it. Blame it. Defend it.

Decompose — or not?

Assign the blame

Five agents, two memos, one problem

Interview check

This week's readings

Quiz 4 — multi-agent systems, security & governance

Five sample questions