TL;DR: In the original Tree of Thoughts study, GPT-4's Game of 24 success rose from 4% with chain-of-thought prompting to 74% with breadth-five search, using different amounts of computation.[5] A separate process-supervision study reported 78% on its representative MATH test subset; neither result establishes a universal cost advantage.[2] The commercial question is how much generation, checking and delay each accepted result consumes.
A reasoning system can improve without changing its underlying weights. It can draft alternatives, inspect intermediate steps and spend another round on a difficult question. That makes inference a resource-allocation decision, with a budget that someone must set and a result that someone must judge.
The real story isn't that longer answers are smarter. It is that a service can decide which questions deserve expensive work. This November 9, 2026 analysis revisits research published between 2021 and 2025. These are historical experiments, not a ranking of today's models or a report of a new release.
Generated editorial illustration: mechanical drafting arms produce candidate drawings and a checking gauge selects one. The artwork represents allocation and verification, not measured model performance or proof that a verifier is reliable.
Why This Matters Now
A buyer evaluating a reasoning service needs a cost per independently accepted task. A token bill alone cannot show whether additional reasoning bought a useful answer, repeated an error or merely increased waiting time.
Allocation: the same budget can buy different work
Charlie Snell and colleagues' August 6, 2024 paper studied search with process verifiers and learned answer revision. Its difficulty-aware strategy reported more than 4× better test-time compute efficiency than a best-of-N baseline in its studied MATH setting. The base was PaLM 2-S*, with capability-specific training. The paper also notes that estimating difficulty can itself be costly.[1]
That qualification matters commercially. An allocator that spends heavily discovering whether a problem is difficult can erase the saving it claims to create. A production comparison must include that decision cost, rejected attempts and the final check. The relevant boundary is the complete service, from incoming request to accepted outcome.
Three budgets are useful to keep separate. The first is total work across every branch. The second is elapsed time on the critical path. The third is simultaneous capacity reserved while branches run. Parallel search may reduce waiting while consuming more total computation. Sequential revision may conserve concurrency while keeping a customer waiting. Neither trade can be resolved by counting the visible answer's words.
Verification: a second model creates a second failure mode
Cobbe and colleagues introduced GSM8K and trained verifiers to rank candidate solutions in October 2021. Their setup separates producing a candidate from selecting it.[3] That separation is the foundation of a useful economic model: generation buys options, while verification determines which options deserve delivery.
Lightman and colleagues' May 2023 work compared outcome supervision with feedback on intermediate steps. It reported 78% on a representative MATH test subset and released 800,000 step-level feedback labels as PRM800K.[2] Verification was an explicitly trained capability with a data cost, not a free consequence of asking the generator to be careful.
A business should therefore test the checker against the generator it actually deploys. A correct final answer can contain invalid reasoning; a persuasive chain can conceal a wrong result. A learned score is evidence about predicted success. It is not equivalent to an executable proof, a passing independent test or a confirmed external fact.
Search breadth: more success is not equal-cost success
The May 2023 Tree of Thoughts paper provides a compact example. Its authors reported GPT-4 Game of 24 success of 4% for chain-of-thought prompting, 45% for breadth-one search and 74% for breadth-five search. These are the same study and task, with different search procedures and compute budgets.[5]
Evidence graphic
Game of 24 success in the original study
Author-reported GPT-4 results on 100 relatively hard games, Table 2. Experiments ran May 5–16, 2023, at temperature 0.7; search budgets differ.
Chain-of-thought prompt
Tree of Thoughts, breadth 1
Tree of Thoughts, breadth 5
Source: Yao et al., Tree of Thoughts, 2023. Author-reported task success, not an equal-cost or equal-latency comparison. Historical GPT-4 setup; no claim about present models or production throughput.
The useful lesson is to preserve the budget beside the accuracy. Self-consistency, introduced in March 2022, instead samples reasoning paths and aggregates their answers.[4] Agreement can be useful, but an operational acceptance rule should still ask whether those paths share the same mistaken premise. More agreeing candidates do not automatically provide more independent evidence.
Evaluation: name the task before quoting the score
The original MATH dataset, introduced in March 2021, contains 12,500 competition mathematics problems.[8] A score on a selected subset needs that subset named. The question set, prompt, answer grader, sampling budget and verifier determine what a result actually measures.
Liu and colleagues' February 2025 test-time scaling study evaluated MATH-500 and AIME24 and emphasized dependence on the policy model, process reward model and problem difficulty.[6] Its small-model comparisons are research configurations. They do not establish that a smaller system will be cheaper, faster or more reliable on an enterprise workload.
There is also a measurement trap in treating generated tokens as synonymous with thought. Geiping and colleagues' February 2025 recurrent-depth work scales computation in latent space.[7] A procurement scorecard should ask for measured service cost and latency rather than assume every architecture exposes its work through the same token counter.
Unit economics: pay for accepted outcomes
Consider a deliberately hypothetical workload. A service spends $0.02 on generation and $0.01 on checking per task, with 60% of tasks independently accepted. Dividing $0.03 by 0.60 gives $0.05 per accepted task. A heavier configuration spending $0.06 with 80% accepted costs $0.075 per accepted task. These are arithmetic assumptions, not vendor prices or measured results.
The higher-success configuration can still be the right purchase when failures are costly. The point is to make that decision visible. Add the cost of human review, rework, abandoned requests and deadline misses. Then decide whether another search branch creates more value than escalating to a person or declining the request.
| Record for each configuration | Why the buyer needs it |
|---|---|
| Fixed task set and independent acceptance rule | Prevents the denominator changing with the method |
| Generation, verification and allocation work | Includes effort spent on rejected candidates |
| Median and tail completion latency | Shows the waiting cost hidden by averages |
| Escalations and unresolved tasks | Keeps difficult cases in the business result |
The verifier belongs in the bill
A research efficiency claim does not automatically include every production expense. Keep difficulty estimation, verifier training, failed branches, checking and waiting time inside the comparison before calling a reasoning configuration cheaper.
Let's be clear: extra computation is an investment in a particular answer. The strongest reasoning product will not be the one that thinks longest on every request. It will be the one that can show why another unit of work was worth buying.
Sources & References
Key sources and references used in this article
| # | Source | Outlet | Date | Key Takeaway |
|---|---|---|---|---|
| 1 | Snell et al. | 2024-08-06 | Difficulty-aware allocation improves efficiency in the studied MATH setting. | |
| 2 | Lightman et al. | 2023-05-31 | Process supervision and PRM800K make verification a trained capability. | |
| 3 | Cobbe et al. | 2021-10-27 | Candidate generation and answer selection are separate operations. | |
| 4 | Wang et al. | 2022-03-21 | Voting aggregates multiple sampled reasoning paths. | |
| 5 | Yao et al. | 2023-05-17 | Game of 24 success changes with search breadth; compute is unequal. | |
| 6 | Liu et al. | 2025-02-10 | MATH-500 and AIME24 outcomes depend on policy, verifier and difficulty. | |
| 7 | Geiping et al. | 2025-02-07 | Recurrent latent computation is distinct from producing more visible tokens. | |
| 8 | Hendrycks et al. | 2021-03-05 | The original MATH dataset contains 12,500 competition problems. |
Last updated: November 9, 2026
