# METR Time Horizons Explained: What 50% Reliability Actually Buys
**Plutonous** | September 20, 2026 | 11 min read
> METR's time horizon measures human task difficulty, not an agent's unattended runtime. Here is how to read the uncertainty and design a business acceptance test.
Tags: METR, AI Agents, AI Benchmarks, Reliability, Software Engineering, AI Evaluation, Automation, Enterprise AI
---
**TL;DR: METR's 50% time horizon is the human task duration at which an agent is predicted to succeed half the time, not the number of hours it can safely run unattended.[1] Its current dashboard warns that estimates above 16 hours are unreliable with the existing task suite; choosing an agent for production still requires measuring accepted outcomes, review effort and recovery cost on your own work.[2]**
A chart expressed in hours is unusually easy to turn into a staffing promise. If an agent has an eight-hour horizon, the tempting interpretation is that it can handle a working day. That interpretation assigns the metric a job it was never designed to do.
Our [earlier METR reporting](/news/who-really-runs-metr-board-funding-benchmarks) examined who runs the organization. Here, the question is what its measurement means. The real story isn't whether benchmark progress matters. It does. The commercial question is how much of that progress survives the move from a scored task to an accepted deliverable. This September 20 analysis revisits research first released in March 2025 and subsequent methodological updates. It does not announce a new model result.[1]
### Why This Matters Now
Use the horizon to identify systems worth evaluating. Use a local acceptance test to decide what to delegate. The dashboard reviewed for this article lists May 8, 2026 as its latest update, so its model coverage should not be mistaken for a complete September leaderboard.
*Cover: generated editorial artwork. The clock and calibration apparatus illustrate a distinction, not METR's equipment or measured results.*
## The Denominator: Human Work, Not Agent Runtime
The original paper defines a success threshold over tasks calibrated by human completion time. A fitted relationship connects that duration to agent success. Reading its 50% crossing gives a measure of task difficulty in familiar units.[1]
Consider a hypothetical two-hour horizon. That does not specify a two-hour model session, two hours of paid inference or a guaranteed two-hour saving. Those are separate operational measurements. Nor does it tell you that every task below the threshold is safe to assign. Our recommendation is to write those distinctions directly into any procurement brief that cites the chart.
METR's January limitations note explicitly rejects converting a horizon into an unattended-work allowance. It also explains that a longer horizon does not translate mechanically into proportionally less human intervention.[3] A more capable agent can fail later, after making changes that take longer to untangle. That possibility belongs in the business case alongside its successful runs.
## The Reliability Threshold: Read the Curve Before the Headline
The dashboard offers both 50% and 80% success views. It is based primarily on software engineering, machine learning and cybersecurity tasks, not every kind of office work.[2] A buying decision needs both a task match and a reliability requirement.
| Question | Relevant measurement | What a horizon cannot supply |
| --- | --- | --- |
| Can the system solve harder tasks? | Time horizon and uncertainty | Acceptance on your specific backlog |
| Can it finish before a deadline? | Actual elapsed runtime | Runtime inferred from human duration |
| Does it save staff time? | Preparation, review and recovery effort | Salary savings inferred from the chart |
| Can a result be used immediately? | Local acceptance rubric | A guarantee from a benchmark pass |
Treat the two reliability views as related estimates. METR says they come from the same fitted model, rather than independent measurements. Its limitations note also says 99% and higher reliability horizons need substantially larger, better benchmarks.[3] Do not extrapolate a purchasing guarantee into that unmeasured region.
For an exploratory code suggestion, an unsuccessful attempt may be cheap. For a change that silently corrupts a report, it may be expensive. The acceptable failure rate follows the workflow's consequences and detectability. It does not follow the model's position on a public chart.
### Same Headline, Different Useful Range
METR's fitted relationship can be written as `p(t) = 1 / (1 + (t/h)^β)`: `t` is human task duration, `h` the 50% horizon, and `β` the slope parameter. Its 80% horizon is `h × (1/4)^(1/β)`.[5] The following arithmetic is our hypothetical illustration, not two measured models.
| Hypothetical fitted curve | 50% horizon, h | Slope, β | Calculated 80% horizon |
| --- | ---: | ---: | ---: |
| A | 120 minutes | 1 | 30 minutes |
| B | 120 minutes | 2 | 60 minutes |
Both curves cross 50% at 120 minutes. Yet B reaches 80% success at a task duration twice A's. At 60 minutes, A predicts `1/(1+0.5)`, or two-thirds success; B predicts `1/(1+0.25)`, or 80%. Reporting only the shared two-hour headline discards information relevant to a higher-reliability workflow. These calculations illustrate the model's shape; they do not validate its fit to any real workload.
Now separate success probability from statistical confidence. “80% horizon” identifies a point on the fitted success curve. A “95% confidence interval” concerns uncertainty in an estimated parameter under the interval procedure's assumptions. It does not mean 95% of tasks finish inside that time, or that an 80% horizon becomes 95% reliable. Neither number measures the chance of meeting your next release deadline.
## The Error Bars: Changes in Measurement Matter
TH1.1, released January 29, 2026, expanded the suite from 170 to 228 tasks. Tasks estimated at eight hours or longer increased from 14 to 31, but only five of those 31 had measured human baseline times; the remainder used estimates.[4] That is useful improvement with a visible measurement constraint.
The March 20 modeling note adds another caution. Correcting a regularization mistake reduced recent models' 50% horizon estimates by up to 20%. The author identifies task distribution as the most important uncertainty and shows that reasonable alternative fits can change the point estimates.[5]
These are reasons to preserve the version, date and interval when copying a result. They are not proof that the entire research program is worthless. A procurement slide that removes those details manufactures confidence the source did not provide. Compare like methodology with like methodology; avoid presenting a revised historical score as a new improvement in the model itself.
## The Task Boundary: A Benchmark Pass Is Not a Finished Product
The underlying benchmarks clarify what is being measured. HCAST's March 2025 paper describes 189 tasks and 563 human baselines. Its limitations include restricted scope, automatic scoring and solitary software work.[6] RE-Bench, introduced in November 2024, uses seven research-engineering environments, with explicit objectives such as improving a GPU kernel.[7]
Those are useful tests of real technical ability. They do not recreate every dependency of maintaining a product: deciding what the request means, preserving unwritten conventions, finding the correct owner and handling a disputed requirement. Our interpretation is that deployment evaluation should preserve these difficult parts instead of cleaning them away to imitate a benchmark.
METR's August 2025 holistic-evaluation update studied 18 tasks from two repositories using early-2025 agents. It found that functionally correct output could still require work on tests, formatting or code quality. The authors describe preliminary results, not an upper bound on future agents.[8] The lesson is to measure acceptance separately from passing existing tests, not to apply that study's failure pattern to every current model.
## The Business Test: Measure Accepted Work and Its Cleanup
The following is our proposed evaluation protocol, not an experiment we have run. Start with a representative sample of actual work, including tasks that require clarification. Preserve the starting state, instructions and success rubric so a later model can face the same test.
Record the exact model version, agent tools, permissions, prompt, retry policy and spending limit. Changing those between candidates changes the system under evaluation. Keep a held-out set out of prompt tuning. Have the person who would normally accept the work review the result against the same standard used for a human submission.
| Record for every attempt | Why the business needs it |
| --- | --- |
| Accepted without edits, accepted after edits, or rejected | Separates production value from plausible output |
| Human preparation, review and repair minutes | Reveals labor transferred into supervision |
| Elapsed time and inference/tool charges | Captures deadline and cash costs |
| Failed attempts and repeated calls | Prevents reporting only the winning run |
| Escaped defects and rollback effort | Exposes costs discovered after initial approval |
Set the stop rule before the pilot. A task should return to a person when it exhausts its budget, encounters an unresolved requirement or needs an unapproved action. Report results by task family, with sample counts and uncertainty, rather than hiding a failing category inside a favorable average.
### A Cost Ledger That Keeps the Failures
Suppose a hypothetical pilot starts 20 distinct tasks. Twelve are accepted unchanged, four after repair, and four rejected. Across all attempts, inference and tools cost $40. Preparation and review take 200 human minutes; repairs to the four salvageable results take another 120 minutes. Two subsequently discovered defects require 80 additional minutes of rollback and restoration. Assume those incidents are resolved and all 16 accepted outputs remain accepted at the reporting cutoff.
At an assumed fully loaded labor rate of $60 per hour, the 400 human minutes cost $400. The pilot therefore spends **$440 for 16 accepted outcomes, or $27.50 each**. The $40 machine bill alone would suggest $2.50 per accepted outcome. That elevenfold difference comes from our stipulated labor costs, not an empirical claim about agents.
| Hypothetical accounting item | Calculation | Cost |
| --- | --- | ---: |
| Inference and tools, including failed attempts | Given pilot total | $40 |
| Preparation and review for all 20 tasks | 200 minutes × $1/minute | $200 |
| Repair of salvageable submissions | 120 minutes × $1/minute | $120 |
| Post-acceptance rollback and restoration | 80 minutes × $1/minute | $80 |
| Total / 16 accepted outcomes | $440 / 16 | $27.50 each |
The four rejected tasks are still unfinished. If a person completes them in 120 additional minutes, the combined workflow costs $560 for all 20 deliverables, or $28 each. Do not count those four as agent successes. For a deliberately simplified comparator, assume completing the same 20 tasks manually would take 30 minutes each, including review, at the same labor rate: $600 total. Under these assumptions, the combined workflow saves $40, or 6⅔%, before fixed setup costs. A $60 setup cost would reverse the advantage.
Replace every assumed input with measured local values. Actual task difficulty, reviewer rates and incident severity will differ. Record elapsed time separately: labor savings do not imply shorter customer wait times. Also state the defect-observation window. A report closed immediately after acceptance cannot include next week's failures, and assigning them a zero cost would be misleading.
### A Pilot Scorecard That Survives Scrutiny
Choose the decision and sample before seeing outputs. For example, a pilot might ask whether one agent can handle a defined family of reversible maintenance tasks within a spending ceiling. Sample from the eligible queue rather than selecting memorable successes. Keep exclusions with reasons, and do not remove an inconvenient task after a failed run.
| Scorecard field | Reporting rule |
| --- | --- |
| Coverage | Count distinct tasks and report task families and exclusions |
| Acceptance | Separate unchanged, repaired and rejected outcomes |
| Repetition | Keep every attempt linked to its original task |
| Labor and cash | Include failures, interventions and later recovery |
| Reliability | Report denominators, uncertainty method and observation window |
| Comparison | Preserve starting conditions; disclose any tuning or tool changes |
Twenty successful runs on one easy task are not twenty independently sampled tasks. Shared repositories, similar requirements and repeated prompts can make outcomes dependent. When comparing systems, reuse the same starting snapshots, prevent one system from seeing the other's solution, and record results as paired tasks. For uncertainty analysis, account for meaningful task groups rather than treating every retry as independent evidence.
Even a perfect small sample warrants restraint. In an illustrative independent-trial model with constant 95% success probability, the probability of 20 successes in 20 trials is `0.95^20`, or 35.85% rounded to two decimals. A spotless 20-run pilot is therefore compatible with a system whose underlying failure probability is 5%. This calculation is not a confidence interval or a claim that production tasks meet the independence assumption.
NIST describes proportion-interval methods and warns that symmetric normal approximations can be inaccurate with few observations or failures.[9] Preselect an appropriate method and stopping rule; report counts alongside the interval. A narrow interval on an unrepresentative sample still answers the wrong business question. Keep a separate holdout for the final decision after tuning, and revisit acceptance when the model or workflow changes.
### The Key Insight
An agent can improve on the benchmark and still be the wrong choice for a workflow with expensive, difficult-to-detect failures. The business unit of value is an accepted outcome after review and recovery costs.
The uncomfortable truth is that a bigger horizon does not remove the obligation to define good work. METR offers a valuable signal about capability. Your acceptance process determines how much of that capability becomes dependable output. Buy the latter.
## Sources
1. [Measuring AI Ability to Complete Long Software Tasks](https://arxiv.org/abs/2503.14499)
2. [Task-Completion Time Horizons](https://metr.org/time-horizons/)
3. [Clarifying limitations of time horizon](https://metr.org/notes/2026-01-22-time-horizon-limitations/)
4. [Time Horizon 1.1](https://metr.org/blog/2026-1-29-time-horizon-1-1/)
5. [Impact of modeling assumptions on time horizon results](https://metr.org/notes/2026-03-20-impact-of-modelling-assumptions-on-time-horizon-results/)
6. [HCAST: Human-Calibrated Autonomy Software Tasks](https://arxiv.org/html/2503.17354v1)
7. [Evaluating frontier AI R&D capabilities](https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/)
8. [Algorithmic vs. Holistic Evaluation](https://metr.org/blog/2025-08-12-research-update-towards-reconciling-slowdown-with-time-horizons/)
9. [Confidence intervals for proportions](https://itl.nist.gov/div898/handbook/prc/section2/prc241.htm)
*Last updated: September 20, 2026*
---
*Source: [LLM Rumors](https://www.llmrumors.com/news/metr-time-horizons-explained-reliability)*