CIS 4394 Agentic AI · Fall 2026 · Dr. Xinyu Fu
CIS 4394 Agentic AI · Fall 2026 · Dr. Xinyu Fu
Two halves, one lesson. First: computer-use agents — the most general thing an agent can do, and the least reliable, according to the benchmarks built to measure it. Then: what actually changes when a working notebook becomes something that runs unattended, costs money per run, and has to be turned off at 2 a.m. by someone who did not write it.
A clean contract with the world. If the vendor never built an API, the agent cannot reach them.
The agent perceives pixels and acts with a mouse and keyboard — no API required, and no schema to catch a mistake either.
Anthropic's own short demo: the model looking at a screen, moving a cursor, filling fields. About two minutes, and worth watching for the rhythm — look at how much of the run is the agent checking whether its last action worked.
Source: Anthropic (official channel).
A longer third-party walkthrough (Developers Digest, roughly 16 minutes) of OpenAI's Operator browsing and acting on live sites. Watch it for the failure texture: hesitation, re-reads, and the places a human has to step in.
Third-party review, not an official OpenAI channel — useful as a demonstration, not as documentation.
Two research benchmarks put agents in real computer environments and score whether the task actually got done. Both are worth knowing by name, because both are the standard reply to “we'll just let it run.”
369 real tasks in real operating-system environments, executed and verified rather than graded on plausible-looking output. As reported in the paper at publication: humans completed above roughly 72% of the tasks, while the best model setup the authors evaluated finished around 12%.
812 long-horizon tasks in a realistic, self-hosted web environment — shopping, a forum, a code host, a CMS — again scored on whether the end state is correct. As reported at publication: a GPT-4-based agent succeeded on roughly 14% of the tasks.
The benchmarks do not say “never ship.” They say the safe envelope is narrower than the demo suggests. Pick the two properties of your task and see where it lands.
The rule underneath this widget is a single sentence worth memorizing: autonomy is earned by reliability and bounded by reversibility. An unreliable agent can still be shipped — on reversible work, with someone reviewing. A reliable one still should not run unattended on irreversible work.
Not “what does the model cost per million tokens” — that is a price, not a cost. The number a manager needs is the cost of one completed task, which is a different and larger thing.
Every step re-sends the conversation so far. Step 6 pays for steps 1 through 5 all over again, so tokens per run grow faster than steps do — the reason a multi-step agent can cost many times a single call for the same question. Add failed attempts, and the true denominator is completed tasks, not runs.
The honest way to get this number: instrument a real run and read it off, then multiply by expected volume. Every estimate assembled from published prices and guessed step counts is a story; the meter is the fact.
A multi-step agent is a sequence of network calls to a large model, each one waiting on the last. What feels instant in a notebook becomes tens of seconds in a product, and the user has no way to tell “thinking” from “broken.” Latency is a product decision before it is an engineering one: stream partial progress, show which step it is on, and pick a smaller model for the steps that do not need the big one.
Your provider caps requests and tokens per minute. One agent at a time never notices; a hundred concurrent runs discover it at once — as errors, in production, usually on the busiest day. Handle it deliberately: a queue, exponential backoff with jitter, and a concurrency limit you chose rather than one the provider imposes on you mid-run.
Networks fail halfway. That is not an edge case; it is the normal condition of distributed systems. The question is only what your code does about it.
A caller-generated identifier for this intended action, sent with the request. If the same key arrives twice, the server returns the original result rather than performing the action again. It has to be generated before the first attempt and reused on every retry — a key generated inside the retry loop is a new key each time, which is the same bug wearing a hat.
A retrying agent is not a retrying HTTP client. It may re-plan, decide the order “did not go through,” and place a subtly different one — different quantity, different vendor — which no idempotency key will catch, because it is a genuinely new request. So the loop needs the other half too: read back the state of the world before acting again. Check for an existing order before creating one.
Log every run as a trace, and alert on the signals that mean something is wrong before a customer notices: cap-hits (an agent that keeps hitting the step limit is failing, not economizing), spend approaching the monthly limit, retry rate, approval-rejection rate, and success rate against your eval suite. Rising rejections at the gate is the most useful early warning you will get — humans are telling you the agent's judgment has drifted.
A prompt change is a deploy. So is a model version change, and that one can happen without you touching anything. Keep prompts, tool definitions and model versions pinned and versioned, re-run the eval suite before promoting, and be able to return to the previous version in minutes. "Roll back the prompt" should be as ordinary as rolling back code.
One flag that stops new runs immediately, reachable by someone who is not you, documented where they will look at 2 a.m. Then name the on-call person, and tell them what to do first: stop new runs, freeze the spend, pull the trace, notify whoever is affected. An unowned agent in production is an incident waiting for a volunteer.
If you can answer these four in a meeting without hedging, you are ready to deploy. If you cannot, the missing answer is your next task.
1 · What does a run cost?
2 · What happens when it fails?
3 · Who is accountable?
4 · How do we turn it off?