Why your agent works in staging and fails in production

Every agent demo works. That is the problem — the demo is run by the person who built it, on inputs they chose, in a session that lasts ninety seconds. Production is a different animal entirely.

The three failure modes

The first is drift in the input distribution. Your staging prompts were written by engineers; your production prompts are written by a customer who is angry, typing on a phone, and pasting in half a spreadsheet. The second is tool flakiness — a retrieval call that times out once in every four hundred requests will look invisible in testing and catastrophic at scale. The third is silent quality decay, where nothing errors and everything is slightly worse.

None of these show up in your error rate. That is why teams keep getting blindsided: the dashboard is green while the support queue fills up.

What actually closes the gap

Trace everything, not just failures. Sample production traffic into your eval suite continuously rather than in a quarterly review. Set a hard budget on latency and cost so degradation surfaces as an alert instead of an invoice. And put a guardrail in front of the response, because the cheapest bad output is the one the customer never sees.

None of this is exotic. It is the same discipline you already apply to any other service in production — it just has not been available for agents until recently.