Agent Reliability: Why 99% Per Step Is Not a Service Guarantee

Cover: Generated editorial illustration of linked tool stations interrupted by a broken coupling. It depicts workflow dependencies, not measured agent performance.

TL;DR: A workflow with 20 independent steps that each succeed 99% of the time finishes without error in 81.79% of attempts, a calculated example rather than a model measurement. The original June 2024 τ-bench introduced pass^k to ask whether an agent succeeds on every repeated trial, a different question from finding one success in k attempts.^[1] The commercial opportunity lies in verified completion and safe recovery.

The agent industry sells completed work, but it often demonstrates successful attempts. Those are different commercial units. A customer buying account maintenance, order changes or an internal operations assistant cares whether the intended state was reached, whether the change was authorized and what happened when a dependency failed.

This analysis revisits research first released between 2023 and 2025. It is not a new ranking of November 2026 models. The durable question is how an attractive individual capability becomes a dependable workflow once tools, policies and users share control of the outcome.

The Metric: One Success Is Different From Reliable Service

τ-bench made a useful distinction. Pass@k asks whether at least one of k trials succeeds; pass^k asks whether all k trials succeed, averaged over tasks. A buyer should demand the metric that matches the service promise.^[1]

Consider an explicitly simplified task with a fixed 90% independent success probability. Four attempts give a 99.99% chance of at least one success, but only a 65.61% chance that every attempt succeeds. Neither number is dishonest. They answer different questions. Selecting a correct result also requires an effective verifier; an operator cannot treat an attractive transcript as proof.

A research assistant searching for one usable candidate can benefit from multiple attempts. An assistant executing every customer's requested change must account for all customers. The second business cannot charge the first business's success metric without explaining who detects and fixes the unsuccessful attempts. Retry budgets, review labor and unobserved damage belong in the economics.

The Arithmetic: Small Errors Accumulate Across Dependencies

Suppose every required step succeeds independently with probability p and the task requires n steps. Then the chance of an entirely successful sequence is pⁿ. This is a teaching model, not an estimate of any deployed agent. Real errors can be correlated, retries can repair them and branching workflows can avoid unnecessary steps.

The implication is not to minimize tool use blindly. A verification call can increase call count while reducing the chance of a wrong final action. The right target is unnecessary exposure: duplicated decisions, ambiguous intermediate state and operations that have no clear completion signal. An extra check can be commercially valuable when it prevents expensive repair.

The Outcome: Inspect State, Then Inspect the Path

WebArena evaluates functional completion in realistic web environments, while OSWorld supplies execution-based checks across desktop tasks.^[4][5] Both point toward a better product question: what changed outside the conversation?

Outcome checking still has limits. τ-bench's authors explicitly note that a correct final database state can coexist with a policy violation, such as skipping a required confirmation.^[1] A refund with the correct amount can still be the wrong action if the customer never authorized it. Final-state correctness and process compliance should therefore be separate fields in an evaluation.

WorkArena's enterprise setting adds another boundary: performance on a consumer website does not establish performance on a company's administrative workflow.^[6] AgentBench's eight environments make the broader point that agent competence has multiple dimensions.^[7] Buyers need representative tasks from their own system, including unusual but costly cases, rather than a blended score whose failure categories remain invisible.

The User: Coordination Creates Another Failure Surface

The June 9, 2025 τ²-bench paper introduced a dual-control telecom environment, where the simulated user can act on the shared problem as well as communicate. The authors report that guiding users introduces challenges beyond an agent operating alone.^[2]

That distinction matters for support businesses. An instruction to restart a device is not evidence that the restart happened. A request for an identifier is not evidence that the identifier refers to the right account. The agent needs observable checkpoints and a clear account of what remains uncertain. Otherwise the workflow builds confident decisions on assumed actions.

A useful evaluation separates failures of reasoning, communication, tool execution and user follow-through. It also records whether the agent noticed the problem. An unsuccessful attempt that safely escalates can be a better service outcome than an apparently successful attempt that corrupts state. Those outcomes should have different costs and different release gates.

Recovery: Build a Transaction Before Selling Autonomy

Amazon's guidance on idempotent APIs explains why repeating a request needs a stable expression of intent: a timeout does not reveal whether the original action happened.^[3] An agent retrying a payment or reservation encounters the same distributed-systems problem as any other client. Better language reasoning does not remove it.

A practical rollout should record an operation identifier, its authorization, the expected state change and the result observed afterward. Reconciliation should inspect current state before repeating a consequential action. Retries need limits and a route to human repair. Google's overload guidance provides the complementary lesson: clients and services must handle capacity restrictions explicitly instead of amplifying load.^[8]

The uncomfortable truth is that an agent can become more capable while its product becomes harder to operate. More tools create more possible state changes, and longer workflows create more opportunities for ambiguity. The valuable supplier will make those boundaries legible, price the review burden and prove that the system can recover. Autonomy earns its margin when failure has a controlled destination.

Last updated: November 9, 2026