Generated editorial illustration: conversation tapes fill separate drawers in a shared server cabinet. The image is a conceptual capacity metaphor, not a photograph or measured hardware layout.
TL;DR: In an illustrative full-attention decoder with 32 layers, 8 KV heads, a head dimension of 128 and 16-bit cache elements, 65,536 retained tokens require 8 GiB of raw KV payload per sequence. At 131,072 tokens, that becomes 16 GiB.[1][3] Those are calculated storage requirements, excluding weights and runtime overhead; long-context availability and affordable concurrency are different promises.
A model can fit on a GPU and still make a poor multiuser service. Its weights occupy memory before the first customer arrives. Each active conversation then brings additional state, and that state grows as the conversation grows. The commercial mistake is to budget the first expense and sell as if the second were negligible.
This analysis is dated November 9, 2026 and revisits foundational research rather than reporting a new model release. Multi-query attention was published on November 6, 2019; GQA followed on May 22, 2023; PagedAttention on September 12, 2023; and KIVI on February 5, 2024.[2][1][3][4] Together they explain why the memory behind an advertised context window deserves its own business case.
Why This Matters Now
An agent can retain instructions, retrieved documents, tool results and its own generated history. Supporting a long sequence says little about how many such sessions a service can sustain at an acceptable latency and cost. The missing product specification is usable concurrency under the intended workload.
The Equation: Context Length Becomes Live Inventory
For a conventional decoder that retains keys and values at every full-attention layer, raw cache bytes equal 2 × layers × retained tokens × KV heads × head dimension × bytes per element × independent sequences. The initial 2 counts keys and values. This is tensor-storage accounting, not a throughput model.[1][3]
Use a hypothetical configuration: 32 layers, 8 KV heads, head dimension 128 and 2 bytes per element. That is 131,072 bytes, or 128 KiB, per retained token per sequence. Multiply by 4,096 tokens and the payload is 0.5 GiB. At 16,384 tokens it is 2 GiB; at 65,536 it is 8 GiB. A GiB is 1,073,741,824 bytes, so the units here are binary, not the decimal GB often used in hardware marketing.
Calculated memory requirement
Raw KV payload grows with retained sequence length
One independent sequence; 32 full-attention layers, 8 KV heads, head dimension 128 and 2 bytes per cache element. Values are calculated, not benchmarked.
4,096 retained tokens
16,384 retained tokens
65,536 retained tokens
131,072 retained tokens
LLM Rumors tensor-size calculation using the full-attention KV structure described in GQA and PagedAttention [1,3]. Hypothetical configuration, not a named model or GPU benchmark. Excludes model weights, activations, allocator metadata, quantization metadata, fragmentation and other runtime memory. Assumes no prefix sharing, sliding-window eviction or latent cache compression.
The number includes both retained prompt tokens and generated history. It is not simply the maximum input length printed on a model card. Hugging Face's cache documentation distinguishes dynamic, static, quantized and offloaded strategies because a runtime can reserve or place memory differently.[5] Architecture and implementation can also invalidate the simple equation: sliding-window layers, latent attention and other state representations need their own accounting.
The Architecture: Count KV Heads, Not Query Heads
A common spreadsheet error substitutes query-head count for KV-head count. In standard multi-head attention they match. In grouped-query attention, several query heads share a smaller number of key and value heads. Multi-query attention takes sharing to a single KV head.[1][2] Those architectural decisions change the amount of state retained for every token.
Hold the other hypothetical dimensions fixed at 65,536 tokens. A 32-KV-head version needs 32 GiB of raw payload. An 8-KV-head version needs 8 GiB. A 1-KV-head version needs 1 GiB. The storage ratios are exactly 4:1 and 32:1 relative to the 32-head case. These are shape calculations across hypothetical architectures, not evidence that converting an existing model preserves quality or produces those speedups.
The GQA paper describes uptraining a checkpoint, not a universal inference switch. Its authors studied the quality and efficiency tradeoff of sharing attention state.[1] The uncomfortable truth is that “same parameter count” is an inadequate procurement comparison. Two models with similar weight sizes can impose very different per-session memory demands, and different answers, because their state architectures differ.
The Allocator: Empty Space Still Consumes Capacity
Even a correct tensor equation can understate allocated memory. Conversations have different lengths and finish at different times. Reserving a large contiguous region for every possible future token can strand capacity. PagedAttention treated that fragmentation and duplication as a memory-management problem, using fixed-size blocks and sharing where appropriate.[3]
That does not make the information payload disappear. It makes the physical allocation follow useful state more closely. vAttention, first submitted on May 7, 2024, explored a different approach that separates physical allocation from a contiguous virtual address layout through CUDA virtual-memory facilities.[7] The shared lesson is to inspect occupied and reserved bytes separately. Neither design turns a 16 GiB payload into zero bytes.
FlashAttention's May 27, 2022 paper addressed IO-aware exact attention with tiling.[6] Avoiding large temporary attention intermediates and retaining a conversation's KV tensors are separate memory concerns. A serving stack can benefit from efficient attention kernels and still run out of persistent cache capacity. Product claims should identify which memory component improved.
The Precision: Fewer Bits Have a Quality Contract
A raw 2-bit element uses one eighth of the payload bits of a 16-bit element. That arithmetic is not an end-to-end eightfold memory saving. Weights, higher-precision residual state, scales, packing, metadata and temporary buffers remain. KIVI examined different key and value distributions and proposed per-channel key quantization with per-token value quantization.[4] The method's asymmetric design is the substantive contribution, not the headline bit count alone.
Hugging Face warns that quantizing the cache can harm latency for short contexts when enough GPU memory is already available.[5] Less stored data can introduce conversion work. Acceptance therefore needs task-quality checks as well as memory traces, with the same model, prompts, decoding settings and concurrency. This article does not combine KIVI, PagedAttention or kernel speed claims into a leaderboard.
Moving state elsewhere creates another tradeoff. CacheGen, first submitted on October 11, 2023, studied compressed KV transfer because network loading can become a significant component of context delay.[9] A cache that fits in host memory or remote storage is not automatically ready for the next GPU step. Capacity and availability have a transport bill.
The Business Case: Sell Concurrency With a Latency Budget
Suppose a service has an explicitly hypothetical 32 GiB budget available solely for raw KV payload after every other reservation. At 65,536 tokens in the 8-head example, arithmetic permits four independent 8 GiB sequences. At 131,072 tokens it permits two 16 GiB sequences. These are storage-only upper bounds; runtime overhead and required operating headroom lower practical capacity. Shared prefixes may change the result, while generated tokens can push a session beyond its initial reservation.
Even staying within that memory bound does not establish a service-level objective. DistServe's January 18, 2024 research separated prefill and decoding because their resource needs and latency goals differ.[8] A commercial evaluation should report completed requests that meet both time-to-first-token and inter-token requirements, with tail latency and queueing visible. The supported context limit alone cannot answer that question.
The Key Insight
A longer context window is an entitlement to consume state, not proof of economical multiuser serving. Calculate raw payload, measure actual allocations, validate quality after compression, then test concurrency against explicit latency targets.
Let's be clear: memory is not a footnote to model intelligence. It sets how much of that intelligence a product can keep available at once. The winning service will know the cost of a live conversation as precisely as it knows the cost of its model weights.
Sources & References
Key sources and references used in this article
| # | Source | Outlet | Date | Key Takeaway |
|---|---|---|---|---|
| 1 | Ainslie et al. | 2023-05-22 | KV-head sharing changes state size; conversion involves uptraining. | |
| 2 | Shazeer | 2019-11-06 | Multi-query attention shares keys and values across query heads. | |
| 3 | Kwon et al. | 2023-09-12 | KV allocation, fragmentation and sharing affect batch capacity. | |
| 4 | Liu et al. | 2024-02-05 | Per-channel keys and per-token values require different quantization treatment. | |
| 5 | Hugging Face | Accessed 2026-11-09 | Cache strategies trade memory, placement and latency; quantization can hurt short-context latency. | |
| 6 | Dao et al. | 2022-05-27 | IO-aware tiling targets attention intermediates and memory movement. | |
| 7 | Prabhu et al. | 2024-05-07 | Virtual and physical allocation can be separated using CUDA memory facilities. | |
| 8 | Zhong et al. | 2024-01-18 | Prefill and decoding have separate latency constraints and resource needs. | |
| 9 | Liu et al. | 2023-10-11 | Moving cached state can add network delay to context loading. |
Last updated: November 9, 2026
