How we think about trust in autonomous systems

We get asked fairly often what our position is on agent autonomy. The short version: we are not against it, we are against unearned confidence.

Confidence should track evidence

An agent that answers from a retrieved document should sound different from one answering from parametric memory. An agent that has run a task ten thousand times should be allowed more rope than one on its first day in production.

Most of the harm we see is not a model being wrong. It is a model being wrong in a register that sounds right, to a user who had no way to check.

What that means for the product

It is why every trace carries its evidence, why guardrails default to observe rather than block, and why we will always show you the run that produced a score instead of just the score.

Trust in a system is built the same way it is built in a colleague: by being legible when it matters.