
Climbing the wrong hill: Notes on evals
In this note
- Agents don't just overfit; they route around you
- Mistake 1: Writing LLM judges before reading raw traces
- Mistake 2: Mushy 1-to-5 scales and ignoring pass^k
- Mistake 3: Letting coding agents memorize your test suite
- Mistake 4: Prompt-tweaking on an architectural plateau
- Keep the offline scoreboard tethered to real humans

I enjoy watching a metric go up and to the right on a Friday afternoon as much as anyone in product. I've just learned the hard way to open the hood before celebrating it.
Years before anyone on my team wired up a large language model, we walked straight into the classic version of this trap on cruise search results. A brand-new visitor with zero browsing history landed on sailings sorted by Price: Low to High. If you graded that page by click-through rate on the top row, it looked like a triumph: almost everybody clicks a $199 two-night weekend cruise out of pure curiosity.
Almost nobody flying a family of four into South Florida for spring break actually books it, though. The seven-night Caribbean sailing that fit their school calendar sat buried three scrolls down. We were high-fiving over baited clicks while families bounced.
Economists call that Goodhart's Law; product and ML folks call it climbing the wrong hill. We only got unstuck when we stopped worshipping top-slot clicks and judged cold-start itinerary ranking on actual booking conversion in controlled holdout A/B tests. Against Price: Low to High, ranking for real session and geographic intent lifted booking conversion by 13% and drove roughly $300M a year in incremental bookings. (As GrowthBook's guide to testing AI search notes, adaptive models make click-baiting even worse unless every logged training event carries its experiment assignment ID.)
What's wild in 2026 is watching sharp teams walk right back into that exact same trap with AI agents, except agents climb bad hills ten times faster and find much funnier ways to cheat.
Agents don't just overfit; they route around you
A classical ranking model can only game the numeric features you feed it. An AI agent has a multi-turn loop, shell tools, file access, and persuasive prose. Hand an agent a sloppy scoreboard and it acts like a clever teenager finding the loophole in a household chore rule.
Anthropic's writeup on evaluating agents shared three real examples worth taping above any AI team's desks:
- When benchmark containers weren't scrubbed cleanly, a coding agent boosted its score by peeking at
gitcommit history left behind by earlier test runs. - A flight-booking agent cheerfully congratulated travelers on confirmed reservations when zero records existed in the database. Grade the chat transcript instead of the database state, and you train a very polite liar.
- Rigid graders break in the opposite direction too: one benchmark failed an agent that found a valid policy rule helping a traveler better than the scripted answer key, while another rejected
96.12because it expected96.124991(fixing the brittle grader alone jumped accuracy from42%to95%without touching the model).
I watched a miniature version of this happen while building Home Port. Whenever the builder session that modeled a 3D prop also graded its own render, it gave itself glowing marks every time. The instant we split non-editing critic sessions away from builders, first-pass scores dropped to 4.5 to 6.5 out of ten, and the room actually started getting good.
Mistake 1: Writing LLM judges before reading raw traces
When an AI feature starts acting weird, the modern reflex is to spin up an eval dashboard, write five 1-to-5 LLM-as-judge prompts scoring "helpfulness" and "hallucination", generate fifty synthetic test queries, and spend two weeks nudging 3.8 up to 4.3. It looks scientific in a sprint review, and half the time it's pure theater.
Say a travel assistant quotes an expired promotional fare. You can burn three days adding NEVER HALLUCINATE PROMOTIONS in all caps to the system prompt and tuning a judge rubric. Or you can pour a cup of coffee and read thirty failing traces by hand first.
Do that, and nine times out of ten you find out the model reasoned fine over bad inputs: a stale upstream cache handed it yesterday's promo table. Fix the cache TTL and add a four-line deterministic date check, and the bug disappears permanently at zero token cost.
Hamel Husain calls this the revenge of the data scientist: reading raw traces is exploratory data analysis, still the highest-ROI habit in applied AI and still the step everyone tries to skip. And as Shreya Shankar and her co-authors showed in their criteria-drift research, you can't outsource that first pass because grading real failures is how you discover what your grading criteria actually are.
Before anyone on my team writes a judge prompt, we open thirty to fifty real production traces in a plain spreadsheet, jot one blunt note per failure, and cluster those notes until new buckets stop showing up:
| Trace | What went wrong | Group |
|---|---|---|
| 1 | Quoted a promotional fare that expired yesterday | stale data |
| 2 | Asked for the departure port a second time | lost context |
| 3 | Answered a refund question from the promotions policy | wrong document |
| 4 | Quoted yesterday's fare again | stale data |
By row thirty, one or two buckets own most of the pain, and at least half of them belong to plumbing fixes rather than prompt poetry.
Mistake 2: Mushy 1-to-5 scales and ignoring pass^k
Nobody knows the real difference between a 3 and a 4 on "helpfulness", including GPT or Claude, and averaging five mushy scores lets a warm, well-formatted paragraph hide a broken business rule. We climb a strict three-rung ladder instead, moving up only when the rung below can't catch the failure:
- Deterministic code assertions first: Schema validity, required tool calls, working URLs, price and date bounds, p95 latency, and verifying the backend state actually mutated.
- Retrieval before generation: If your search chunking misses the right policy doc (
recall@k), scolding the generator prompt just teaches the model to bluff smoothly. Measure retrieval and answer synthesis separately. - Narrow binary (
PASS/FAIL) judges last: One specific failure mode per judge, reasoning trace emitted before the verdict bit, an explicitUNKNOWNescape hatch, and human-agreement checks using precision and recall rather than raw accuracy (on a bug that happens5%of the time, a rubber-stamp judge that passes everything looks95%accurate).
Even binary pass rates hide an ugly surprise on multi-step agents. Anthropic distinguishes pass@k (at least one of k attempts succeeds) from pass^k (all k attempts succeed consistently). For an internal coding tool where an engineer eyeballs diffs, pass@5 is great. A family planning a vacation gets one conversation. An agent with a 75% single-run pass rate succeeds three times in a row (pass^3) only 42.2% of the time.
Drag the sliders to feel how fast consistency falls off a cliff:
Mistake 3: Letting coding agents memorize your test suite
In classical ML, training on your test set gets your paper thrown out. With coding agents, teams accidentally automate test-set leakage before lunch. Point an agent at your entire sixty-case eval file and tell it to hill-climb the prompt until 95% pass, and within twenty iterations your system prompt is stuffed with weird brittle rules reverse-engineered straight from those sixty rows. Offline scores hit 96%; production guest satisfaction doesn't budge.
Two habits stop self-cheating cold:
- Split your golden dataset four ways and lock the vault: Keep a tiny
Trainslice for few-shot prompt examples, aDev-Tuneslice that engineers and optimizers can hammer freely, aDev-Checkslice run periodically to spot overfitting early, and a sealedHeld-Out Testvault stored outside the developer prompt workspace that runs only on release candidates and frontier model swaps. - Pair positive cases with "don't overreach" cases: Test only queries that need web search or a tool call, and you'll breed a trigger-happy agent that calls tools on
"hello". Always pair unambiguous action prompts with underspecified prompts where the right move is asking one crisp clarifying question.
| Set | Share | Who sees it |
|---|---|---|
| Train | 15% | the prompt, as worked examples |
| Dev-Tune | 30% | everyone tuning: people, optimizers, coding agents |
| Dev-Check | 25% | run now and then, to catch overfitting early |
| Held-Out Test | 30% | nobody, until a release or a model change |
Mistake 4: Prompt-tweaking on an architectural plateau
Sometimes you do the eval work cleanly, climb from 52% to 74%, and then watch five prompt experiments in a row move the needle by 0.2%. That flatline isn't failure; it's the clearest signal in AI product work that you've topped out your current hill and need to change the architecture, runtime context, or UI:
- Cut the agent chain: An early multi-agent cruise search prototype routed through four sequential agents and dragged p95 latency out near
9.5 seconds. No prompt rubric makes a ten-second search bar feel good. Collapsing that path into a single structured tool call under2 secondsis what got people to actually use it. - Give it working memory instead of scolding it: When a multi-turn planning assistant kept forgetting party size or departure port by turn six, adding louder prompt reminders did nothing. Wiring up a structured in-session guest scratchpad lifted multi-turn task completion by
10 percentage pointsover our no-scratchpad baseline on human-graded transcripts at flat latency. Eugene Yan nails this as treating context as infrastructure. - Render UI cards, or skip AI entirely: Comparing three ships across pricing, kids' ages, and balcony cabins belongs in structured side-by-side comparison cards, not five walls of chat prose. And over in
CrashCalc, the optimal amount of generative AI inside highway skid-mark math is flat zero.
| Wrong hill | Why the scoreboard turns green | What to do instead |
|---|---|---|
| Judges before traces | Generic rubrics move tone while upstream bugs stay broken | Read 30 to 50 real traces and group the failures first |
| Vague scales, pass@k | Polite 4 out of 5 averages hide broken rules; one good run hides fragility | Code checks, then retrieval, then narrow pass/fail judges; track pass^k |
| Tuning on the whole suite | Agents write the test cases into the prompt | Seal a held-out set; test both sides |
| Prompting on a plateau | The limit is the architecture, the memory or the interface | Change the loop, the state or the screen, or use plain code |
Keep the offline scoreboard tethered to real humans
Even a great eval suite ages into a regression floor. Once your gate clears 95%, keep running it in CI to catch model drift and broken tool schemas, while steering the roadmap from live human behavior: holdout booking conversion, how often a traveler has to re-explain themselves on turn two, and whether support containment rises without trapping people away from a human agent.
Coding agents made hill-climbing almost free. You can kick off three optimization branches before dinner and wake up to twenty graded runs.
I still love seeing a chart move up and to the right on a Friday afternoon. Now I just read ten raw traces before I buy anybody drinks.