Generated editorial illustration: a letterpress reuses engraved plates beside a typesetting bench. This is a metaphor for prompt reuse, not a measurement of an inference system.
TL;DR: Under a specified cache policy of 1.25× for a write and 0.1× for a read, ten identical prefixes cost 2.15 input-price units when the first writes and the next nine hit, versus 10 units without caching.[1] That is a calculated 78.5% reduction for the repeated prefix, not the entire application bill. The strategic asset is a workload that reuses stable context before its cache expires.
The AI industry's price sheet misses a variable that product teams control: how often they pay to process the same information. A support agent may reread the same policy manual for every customer. A coding assistant may repeatedly ingest its tool definitions. A research workspace may ask different questions about one unchanged report. These are different products with the same economic opportunity.
This is a November 9, 2026 analysis of an established systems idea, not a new launch report. Prompt Cache was first submitted on November 7, 2023; SGLang followed on December 12, 2023. Both investigated reuse of attention state, before caching became a familiar API billing category.[2][3] Their lasting lesson is architectural: repeated context can become reusable computational inventory.
Why This Matters Now
Agents turn one user request into several model calls. Stable instructions and shared documents can recur across those calls. Teams that measure only total tokens cannot tell whether they are buying new reasoning, repeated input processing, or expensive cache rewrites.
The Mechanism: Reuse Computation Before Generating the Answer
A prompt cache stores intermediate attention state for reusable input. It does not store a finished answer that every user receives. Different questions can share a document prefix and still require different output. Prompt Cache's research design used explicitly defined reusable modules; SGLang's RadixAttention managed reusable KV state in a runtime.[2][3] Those are related approaches, not interchangeable implementation guarantees.
The distinction matters commercially. A cached response trades on identical answers. A cached prefix trades on identical groundwork. The latter can support diverse questions over common information while continuing to pay for new input after the prefix and generated output. A provider's discount is therefore a price for avoiding part of the request's work, not proof that the whole request becomes proportionally cheaper.
PagedAttention's September 12, 2023 paper addressed allocation waste and KV sharing in serving.[4] Better allocation and cross-request reuse solve different problems. Buying a larger accelerator does not automatically create either one. The software must identify reusable state and keep enough of it available to make reuse worthwhile.
The Arithmetic: A Discount Needs Enough Hits
Use one explicit policy to make the calculation inspectable. Anthropic's documentation, reviewed for this analysis, lists standard multipliers of 1.25 for five-minute cache writes and 0.1 for reads, alongside model-specific read exceptions.[1] The chart selects that standard policy. It is not a quotation for every Claude model, a current dollar offer, or a comparison with another provider.
Let one complete uncached prefix cost one unit. For ten equal-sized prefixes, with every miss creating a cache entry, cost is 1.25 × writes + 0.1 × reads. New suffix tokens and outputs are excluded. With five writes and five reads, the cost is 6.75 units. With one write and nine reads, it is 2.15 units. With ten writes and no reads, caching costs 12.5 units, 25% more than the uncached baseline.
Calculated cost scenario
Ten equal prefixes: billed input cost under one specified policy
Each uncached prefix costs 1 unit. Every cache miss writes at 1.25 units; each hit reads at 0.1 units. All prefixes have the same token count.
Uncached baseline
10 writes, 0 reads
5 writes, 5 reads
1 write, 9 reads
LLM Rumors calculation using the standard 1.25× write and 0.1× read multipliers documented by Anthropic, accessed November 9, 2026 [1]. Selected policy only; model exceptions apply. Excludes outputs, changing suffixes, storage, taxes, platform modifiers and retries. This is not measured application spending.
For a token-weighted hit fraction h across these equal-sized eligible prefixes, average cost is 1.25 − 1.15h units. Break-even occurs at h = 5/23, or 21.7391% rounded to four decimal places. One write followed by one read already costs 1.35 units instead of 2. That result assumes the entry survives, every hit qualifies, and there are no extra charges outside this simplified prefix calculation.
The Product Design: Stable Context Is an Economic Resource
The real story isn't a discount percentage. It is that prompt organization becomes a financial decision. Place reusable instructions, stable schemas and appropriate shared documents before changing request details. A timestamp inserted ahead of a reusable prefix can make work that looks identical to a human differ at the machine's matching boundary.
The selected API documentation requires exact matching through the cache boundary and describes isolation between organizations, with workspace behavior varying by platform.[1] A product should therefore partition by legitimate access boundaries first, then optimize reuse within them. A higher hit rate cannot justify allowing one customer's confidential context to become another customer's input.
vLLM's automatic prefix caching documentation describes reuse when requests share a prefix.[7] In a self-hosted deployment, the benefit accrues through avoided processing and available capacity rather than an API line-item discount. In a hosted API, the bill follows the provider's contract. Similar underlying computation does not imply identical customer economics.
The Infrastructure: A Hit Has a Location
What's often overlooked is that a cache entry can exist somewhere expensive to reach. CacheGen, first submitted on October 11, 2023, examined compression and transfer of KV state because moving cached tensors over a network can itself delay context loading.[5] Saving computation is useful only when retrieval and restoration cost less than the work avoided.
SGLang's HiCache documentation makes the hierarchy concrete: GPU memory, host memory and a storage backend have different scopes and movement costs. Its local host cache is instance-private; cross-instance reuse depends on the configured storage tier.[8] The managerial consequence is direct. A dashboard that reports one undifferentiated hit rate can hide a fast local hit and a slower remote restoration behind the same label.
DistServe, first submitted on January 18, 2024, separated prefill and decoding resource planning around their different latency requirements.[6] That supports a measurement discipline, not a speed ranking: report time to first token separately from output generation and end-to-end completion. This article does not compare throughput claims across these papers because their systems and workloads differ.
The Margin: Measure the Completed Task
A prefix-input saving is not a gross-margin result. In an illustrative bill containing $4 of reusable prefix input and $6 of all other unchanged costs, reducing that input component by 90% yields $6.40 total, a 36% reduction. Those dollars are an arithmetic example, not a vendor price or observed customer invoice. If extra retries or longer outputs consume the saving, the product has optimized an intermediate metric.
The operating dashboard needs eligible prefix tokens, cache writes, cache reads, uncached input, output cost, entry age and completed-task rate. Compare equal tasks before and after the change. Record model version, prompt composition, output lengths, concurrency, latency distribution and routing policy so a change in workload does not masquerade as a cache improvement. Cache-specific accounting must also remain distinct from total input volume.
The Key Insight
A cache-write premium makes poorly reused context more expensive. A cache-hit discount applies only to the portion covered by that policy. Treat hit rate as a workload property and completed-task cost as the business outcome.
Here's the genius in designing around reusable context: the same policy manual or tool schema can serve more useful work before the system pays to process it again. The durable advantage belongs to products that make repetition deliberate, measurable and safe. Cheap input is a price. Reusable work is a design capability.
Sources & References
Key sources and references used in this article
| # | Source | Outlet | Date | Key Takeaway |
|---|---|---|---|---|
| 1 | Anthropic | Accessed 2026-11-09 | Standard write/read multipliers, model exceptions, exact matching and isolation. | |
| 2 | Gim et al. | 2023-11-07 | Explicit prompt modules reuse attention state; research prototype. | |
| 3 | Zheng et al. | 2023-12-12 | RadixAttention manages reusable KV state in structured LM programs. | |
| 4 | Kwon et al. | 2023-09-12 | Allocation waste and KV sharing are serving-system design problems. | |
| 5 | Liu et al. | 2023-10-11 | Transporting cached state creates a context-loading tradeoff. | |
| 6 | Zhong et al. | 2024-01-18 | Prefill and decoding have distinct resource and latency requirements. | |
| 7 | vLLM | Accessed 2026-11-09 | Requests with common prefixes can reuse cached state. | |
| 8 | SGLang | Accessed 2026-11-09 | GPU, instance-private host memory and storage have different sharing scopes. |
Last updated: November 9, 2026
