CIS 4394 Agentic AI · Fall 2026 · Dr. Xinyu Fu
CIS 4394 Agentic AI · Fall 2026 · Dr. Xinyu Fu
pass@k and pass^k look almost identical on a slide and mean nearly opposite things. One is the most generous number you can quote about an agent; the other is the number your customer actually lives in. Learning to tell them apart is the single most useful thing on this site.
Introduced for evaluating code generation on HumanEval (Chen et al., 2021). You let the model generate k candidate solutions for one problem and count the problem as solved if any single one of them passes the unit tests. It answers: if I can sample k times and cheaply check which one is right, how often is a correct answer somewhere in the batch?
Rises with k. Bigger k always looks better. That is legitimate when a cheap, reliable verifier exists — unit tests, a compiler, a checksum — because the verifier is what lets you pick the winner out of the batch.
Chen et al., 2021 — Evaluating LLMs Trained on Code (arXiv:2107.03374) ↗
Introduced by τ-bench (Yao et al., 2024) precisely because pass@k was flattering agents that could not be relied on. Run the same task k independent times; it counts only if every single run succeeds. It answers: can I hand this to a customer who will use it k times?
Falls with k — fast. There is no verifier to rescue you: in production nobody generates eight refunds and picks the good one. The customer gets the run they get.
Set the per-run success rate p and the number of trials k. The blue curve is pass@k (at least one succeeds). The red curve is pass^k (all of them succeed). Same agent. Same p. Opposite conclusions.
Three agents you might describe with the same word — “reliable” — and what happens to each when a real user runs it more than once. Every cell is pk, rounded.
| pass@1 (one run) | pass^2 | pass^3 | pass^5 | pass^8 | What it means in a sentence |
|---|---|---|---|---|---|
| 70% | 49% | 34% | 17% | 6% | Fails more often than it works by the third use. This is a prototype, not a product. |
| 90% | 81% | 73% | 59% | 43% | The number everyone celebrates. Most repeat users still hit a failure. |
| 99% | 98% | 97% | 95% | 92% | Survives repeated use — and going from 90% to 99% is usually far more than 9% more work. |
Moving 70% → 90% is usually prompt work, a better tool, one retry. Moving 90% → 99% means finding the long tail: the malformed inputs, the empty tool results, the rate limits, the adversarial cases. Each of those is a separate engineering project, and each buys a fraction of a point. Budget accordingly — and notice that the same money spent on narrowing the task often buys more reliability than the same money spent on the agent.
“Our agent is 90% accurate” is not a lie, but it is not a deployment statement either. The deployable version names the metric, the k, the dataset, and the date: “On our 40-case eval set, pass@1 = 0.90 and pass^3 = 0.73, measured 10/14. The 27% of cases that fail at least once in three runs are listed here.” That sentence is the deliverable of your capstone eval harness.
Public benchmarks are the closest thing the field has to shared evidence — and the most misquoted numbers in any AI vendor deck. Three you will meet constantly, what each actually stresses, and how the score gets used misleadingly.
| Benchmark | Domain | What it stresses | Metric usually quoted | Watch out for |
|---|---|---|---|---|
| SWE-bench Jimenez et al., 2023 | Real GitHub issues, Python repos | Navigating an unfamiliar codebase, long edits, making the repo's own tests pass | % of issues resolved, single attempt | Contamination (public issues and fixes); which subset was used; harness differences |
| τ-bench Yao et al., 2024 | Retail and airline customer tasks | Tool use with a simulated user, while following a written domain policy | pass^k — and pass@1 alongside it | Domain policies are specific; simulated users are not real ones; airline is harder than retail |
| WebArena Zhou et al., 2023 | 812 tasks on self-hosted website clones | Long-horizon, realistic web work across multiple sites | Task success rate | Clones are not the live web; the headline numbers are from launch and are dated |
If the benchmark's problems and their solutions have been public on the internet for years, some of them may sit in the model's training data. A high score can then reflect recall rather than capability — and it will not transfer to your private codebase or your internal tickets.
The same model scores differently depending on the scaffolding around it: which tools it gets, how many steps it is allowed, how many attempts, whether a retry counts. Two “SWE-bench scores” are only comparable if the harnesses were. Ask what the scaffold was.
“SWE-bench” names a family: the full 2,294-issue set, the human-validated Verified 500-issue subset, and smaller subsets. A number quoted without its subset is unreadable — and the flattering subset is the one that ends up on the slide.
Most of what an agent produces has no exact right answer, so you need a grader for open-ended output. The obvious move is to have a strong model grade it against a rubric or a reference (Zheng et al., 2023). It scales beautifully. It is also a fallible evaluator, and the failure mode is that it looks rigorous.
| Kind | Good for | Cost |
|---|---|---|
| Rule / exact match | Anything checkable: IDs, numbers, required fields, forbidden actions, schema validity, tests passing | Nearly free, perfectly consistent |
| LLM-as-judge | Open-ended quality: clarity, tone, helpfulness, faithfulness to a source | Cheap per item, but biased and needs validating |
| Human review | High-stakes and ambiguous edges: legal risk, safety, anything you will be asked to defend | Expensive, slow, and the ground truth everything else is calibrated against |
Worked example — one document-QA agent, four checks: the cited document ID → rule (it either matches or it does not). The extracted dollar amount → rule. Whether the answer reads clearly and stays faithful to the source → LLM-judge. Anything flagged as a legal-risk edge case → human. Do not reach for a judge where a rule is exact and free.
The concept, the rubric, and the pitfalls in six minutes. Watch it, then read the four biases below and see how many the video names.
When comparing two answers, judges can favor the one in a particular position rather than the better one. Documented in Zheng et al., 2023. Your ranking can flip when you swap A and B.
Longer, more elaborate answers tend to score higher, whether or not they are more correct. This is the bias students game accidentally: pad the response and the rubric rewards it.
A judge can prefer text in its own style or from its own model family. If the judge and the agent are the same model, you are partly grading a family resemblance.
Agreeable, flattering, confident-sounding output scores well. Confidently wrong answers are exactly the ones a judge is worst at catching — and exactly the ones that hurt in production.
Zheng et al., 2023 — Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv:2306.05685) ↗