The approve button is not a control

Every team that builds its first agent arrives at the same safeguard, and they arrive at it within about ten minutes: ask the human first. Put a prompt in front of anything consequential. Nothing happens without a click.

It feels obviously right. Anthropic measured what people actually do with those prompts across their own products, and published the number in how they contain Claude: users approve roughly 93% of permission prompts, and their attention declines over time.

A control that is satisfied 93% of the time by reflex is not a control. It is a log entry with a button on it.

What real deployments actually look like

Before prescribing anything, it is worth knowing what is already out there. Anthropic's research on measuring agent autonomy in practice sampled 998,481 random tool calls across their public API and classified each one.

Two numbers from it are worth memorising. 80% of tool calls carried at least one safeguard — restricted permissions, an approval requirement — and 73% had a human involved in some way. Agents in the wild are not running unsupervised.

The third number is the one that should reorganise how you think about the problem. Of those 998,481 calls, the share that appeared irreversible — the email that has been sent, the record that is gone — was 0.8%.

Under one percent of what agents do cannot be undone. Almost everything an agent does is a draft, a query, a read, a file it can rewrite. The risk is concentrated in a sliver — sending the email, moving the money, dropping the table, posting the thing — and that sliver is where a human's attention is worth something.

Spend approvals there. Spend them nowhere else. Every prompt you put in front of a reversible action is a small withdrawal from the same account you will need when the irreversible one appears.

And the window in which a click means anything is getting longer. The same research tracked how long agents run per turn, and at the top of the range it nearly doubled in four months.

The median turn stayed at roughly 45 seconds, so this is not everything getting slower — it is the long tail getting much longer. An approval granted at the start now governs a stretch of unattended work that can run past forty minutes. Whatever is protecting you at minute thirty is not that click.

The distinction that does the work

Anthropic's engineering guidance draws a line that most teams never draw, and almost everything else follows from it:

supervise what the agent does via a human-in-the-loop — versus — supervise what it's able to do

The first is a prompt. The second is a boundary. Their stated preference is unambiguous, and containment is where they say Anthropic engineering has devoted the most effort.

The practical difference: a permission prompt asks a human to be right every single time. A boundary has to be set up correctly once. Only one of those scales to a system that runs for months.

This is not an argument for removing humans. It is an argument for the framing Anthropic uses — effective oversight does not require approving every action, but being in a position to intervene when it matters.

Three layers, in order

The guidance names three places a guardrail can live, and the ordering is the advice.

The environment. Where the agent executes — sandboxes, virtual machines, egress controls. This is the layer to design first, because it is the only one that holds when the model is confused or the instructions are hostile.

The model. System prompts, classifiers, training. Real, useful, and second — you steer behaviour here after you have bounded capability below.

External content. Everything the agent can read or call: tools, plugins, connected servers. This is the layer most teams forget entirely, and it is where the interesting attacks live.

The principle to hold onto: design for containment at the environment layer first, then steer behaviour at the model layer. Defences should overlap.

Two failure modes worth naming

Anthropic published their own failures, which is unusual and worth reading directly. Two generalise beyond their products.

The agent reads something hostile

The live attack on agents is not someone typing malicious instructions into your chat box. It is instructions hidden in content the agent reads — a document, a web page, an email, a ticket, an API response.

In Anthropic's own adversarial testing of this against Claude Cowork, a direct phishing attempt succeeded in exfiltrating data in 24 of 25 attempts. Model training did not stop it. Prompts did not stop it. The thing that stopped it was an egress control — the boundary that decides where data is allowed to go, regardless of what the agent decided to do.

Note what that means for the approval button: the agent was doing something it had been persuaded was legitimate. A human approving that action would have approved it too.

Every tool you connect is a capability grant

The second lesson is blunter, and it is the one to carry into vendor conversations. Anthropic's finding is that tool allowlists function as capability grants — every reachable function becomes attackable, not just the destination domain.

Their own case: attackers used legitimate approved API endpoints to move data out. The domain was on the allowlist. The endpoint was real. The traffic looked correct.

So "we only allow it to talk to approved services" is a weaker statement than it sounds. What you have allowed is every function those services expose.

Match the boundary to the person

One more principle, and it is the one that keeps this from becoming a lecture about locking everything down: match isolation strength to the user's capacity for oversight.

The measurements make the case. Experienced users — around 750 sessions — run in full auto-approve mode more than 40% of the time, against roughly 20% for newcomers. But the same experienced users interrupt more often, about 9% of turns against 5%.

That is not carelessness. It is a different oversight model: they are not checking each step, they are watching the shape of the work and stopping it when it goes wrong. That model requires knowing what wrong looks like.

So the honest read is that autonomy should be earned per operator and per task, not set globally in a config file. The same agent that should run unattended for one person should be gated for another, and no vendor default gets that right.

What to ask if someone is building this for you

"What can it do if everything it reads is hostile?" This is the only version of the security question that matters, and it is answered at the environment layer or not at all.

"Which actions are irreversible, and what gates those specifically?" If the answer is that everything is gated, nothing is — you have built approval fatigue and called it safety.

"Where does data get out, and what enforces that?" Egress was the control that worked in the one published test where everything else failed.

"Is the isolation something you built?" Battle-tested hypervisors, syscall filters and container runtimes beat proprietary alternatives. In Anthropic's own words: the weakest layer is the one you built yourself.

The honest summary

The permission prompt is the guardrail everyone builds first, and it is the one that fails quietly — not with an alarm, but with a person clicking approve for the four hundredth time while thinking about something else.

What holds instead is boring and structural: decide what the agent is able to do, enforce that in the environment rather than in an instruction, reserve the interruptions for the 0.8% of actions that cannot be taken back, and let a human watch the shape of the work rather than rubber-stamp each step of it.

Which is the same lesson as the rest of this site, arriving from the direction of risk: the model is not where the outcome is decided. The harness around it is.

Frequently asked questions