During the agentic toolchain rollout at Visa, a model produced one confidently wrong summarization. It wasn't a production outage, and I won't dress it up as one. But it stalled a team for weeks, because once people catch a tool being confidently wrong, they stop trusting it.
I took two lessons from that, and they shaped everything after.
Trust is asymmetric
One bad output costs more than a hundred good ones buy back. You can't talk a team back into trust with a deck or a better demo. The only lever you really have is when the model is allowed to reach a person, and what it has to prove before it does.
So we moved the gate. Evals stopped being something we ran after a launch to see how it went. They became the price of admission. No model or skill touched a user until it cleared our test track, and inside the product pipeline an adversarial critic reviewed every stage's output before the next stage could start. A weak artifact simply couldn't travel forward.
A test beats a title
The second lesson was about people, not models. Leadership got excited about a new model most weeks, and some of the people excited were more senior than me. Arguing taste against rank is a losing game, and it should be. So I stopped arguing and built a rule: beat the incumbent on our test track, or no keys.
That turned a mandate into a one-week benchmark. Several shiny things failed. Sometimes the new candidate won, and we adopted it happily. The point was never to say no. The point was a test both sides trusted, so the decision stopped being personal.
If you can't write an eval for it, you can't safely delegate it. Everything else is hope.
What "the track" actually means
People hear "evals" and picture a leaderboard. A useful test track is smaller and meaner than that:
- Frozen cases drawn from real work, including the exact cases that already burned you.
- Hard assertions that must pass every time: the right fields, no secrets, no invented numbers.
- A graded rubric for quality, scored the same way on every run.
- The incumbent run through the same harness, so "better" means better than what you already have, including on cost.
- A gate sized to blast radius. An internal summarizer is not the same as anything that can touch a transaction. The closer to money, permissions, or customers, the more a person has to approve.
What I'd do differently
Install the harness before the first disagreement, not during it. We built the rule in the middle of an argument, and it worked, but it would have been cheaper to have it waiting.
And watch the cost of your own safety net. Critic loops burn tokens, and humans reviewing agent output get tired in ways no dashboard shows until change-failure moves. Budget the critics per stage, and measure reviewer load next to throughput.
The same rule, on my own laptop
I run an open-weight model locally, and before I trusted it with anything I put it through 31 scored calls with a frontier grader checking every answer. It extracted every deal term from two real SEC filings and solved every logic puzzle. It also invented a fact with total confidence and attached a false tax rule to math it got right. Same lesson, smaller scale: the model is a great reader and a poor source of facts, and I would only know which is which because I tested it first.
Evals before exposure isn't a slogan. It's a scheduling decision about when a model gets to meet a human. Make it early.