# A Bigger Context Window Is Not a Better Working Memory
**Plutonous** | November 9, 2026 | 6 min read
> Lost in the Middle, RULER and NoLiMa show why usable context depends on evidence position, task difficulty and a stated acceptance test.
Tags: Long Context, AI Benchmarks, NoLiMa, RULER, AI Evaluation, Enterprise AI, Document AI, AI Strategy
---
*Cover: Generated editorial illustration of a long folded manuscript with a crimson bookmark in its middle. The image is a conceptual metaphor, not an attention map.*
**TL;DR: A context window specifies how much text a model can accept, while effective context depends on what it must do with that text. NoLiMa's 2025 GPT-4o evaluation reported 98.1% accuracy at 1K tokens and 69.7% at 32K on the same association-retrieval benchmark, a 28.4-percentage-point difference.[3] Procurement should test evidence position, distractors and reasoning requirements before treating advertised capacity as usable working memory.**
A model can accept an entire document collection and still fail to use the paragraph that matters. Lost in the Middle, first released on July 6, 2023, demonstrated that moving relevant information within the input could change performance. Information at the beginning or end was often easier to use than information in the middle.[1]
This analysis is published on November 9, 2026. It revisits research from 2023 through 2025 to examine an enduring evaluation problem, rather than ranking today's models. The commercial issue is simple: buying a larger input allowance does not specify how reliably the system will complete a difficult task inside it. The uncomfortable truth is that a capacity specification can be accurate while the buyer's interpretation of it is wrong.
### Why This Matters Now
Whole-repository assistants, research workspaces and document tools invite users to supply more material. The acceptance test must match the actual work: finding a literal identifier, reconciling distant facts and summarizing a narrative require different capabilities even when their inputs have the same token count.
## The First Trap: Capacity Is Not an Accuracy Contract
Lost in the Middle controlled where the useful information appeared. That matters because ordinary application tests often change several things at once: the question, documents, evidence location and number of distractions. A disappointing answer then says little about which change caused the failure.[1]
A fair position test starts with an answerable question and preserves its evidence. Move that evidence through the beginning, middle and end while holding the surrounding material constant. Score the answer with the same rule at every placement. If the result changes, the product has discovered a reliability condition that a single maximum-context number did not describe.
This is commercially useful even without a universal scientific explanation. A team can reduce unnecessary material, improve document ordering or change the workflow. It can also reject the assumption that merely increasing the window will solve the problem. The test identifies a product constraint; it does not establish that every long-context model has the same weakness.
## The Second Trap: Literal Retrieval Can Flatter the System
RULER, first released in April 2024, expanded the needle-in-a-haystack idea into 13 tasks, including multi-hop tracing and aggregation. Its authors evaluated 17 long-context models.[2] The methodological advance was to make the question harder than locating one conspicuous string. A system that finds an exact phrase may still struggle to combine several dispersed facts.
NoLiMa, first released in February 2025, reduced the lexical overlap between the question and the hidden evidence. The required association made direct matching less useful. The chart below preserves a single model row across increasing lengths, rather than constructing a leaderboard from incompatible experiments.[3]
### GPT-4o accuracy by input length in NoLiMa (2025)
Researcher-reported Table 3 results for the same model on association retrieval with minimal literal overlap. Each bar is a tested length, not a different product.
- label: 1K tokens; value: 98.1; displayValue: 98.1%
- label: 2K tokens; value: 98; displayValue: 98.0%
- label: 4K tokens; value: 95.7; displayValue: 95.7%
- label: 8K tokens; value: 89.2; displayValue: 89.2%
- label: 16K tokens; value: 81.6; displayValue: 81.6%
- label: 32K tokens; value: 69.7; displayValue: 69.7%
100
Accuracy
0%
100%
Source: Modarressi et al., NoLiMa v3, Table 3 (2025). Historical synthetic association-retrieval evaluation using the paper's instruction templates and greedy decoding for standard instruction-tuned models. These are not current API guarantees, general reading-comprehension scores or speed measurements.
NoLiMa defined effective length using a threshold set at 85% of each model's base score. That base score uses best short-length performance per example, not simply the result at 1K. Consequently, “effective length” is a benchmark definition with a tolerance rule. It is not a physical boundary beyond which the model suddenly stops understanding.[3]
## The Third Trap: One Benchmark Cannot Represent Every Job
LongBench, released in August 2023, covered 21 datasets across six task categories in English and Chinese. These included question answering, summarization and code completion.[4] Task variety matters because a retrieval failure and a weak summary are different product defects. Combining their scores without showing the task mix can conceal the weakness that matters to a particular buyer.
L-Eval, released in July 2023, also examined how evaluation metrics corresponded to human judgments. Its work warned against assuming common text-overlap measures adequately represented long-answer quality.[5] The implication is an evaluation-design problem: teams must decide what counts as a correct, complete answer before comparing systems.
∞Bench, released in February 2024, targeted contexts averaging more than 100K tokens and included tasks where retrieving a handful of passages was insufficient.[6] This prevents an equally simplistic conclusion that retrieval always beats a full window. Some questions require distributed narrative understanding or dependencies across the material. The best architecture depends on which evidence must interact.
## The Evaluation Plan: Preserve the Difficulty That Matters
Start with real tasks and a stable scoring guide. For a code review, the decisive information might connect a distant caller to an implementation. For a document comparison, it might connect an exception clause to a general rule. Preserve those relationships as the input grows. Padding a prompt with irrelevant text measures resistance to distraction; adding necessary evidence measures something else.
Separate exact-match lookup, association, multi-step synthesis and global summarization in the results. Record evidence placement, total tokens, output allowance, prompt template, decoding settings and model identifier. Keep latency and cost alongside accuracy, but do not let a faster response count as a correct one.
The official NoLiMa repository supplies dataset material, model configurations and evaluation code.[8] Reproducibility still requires recording the version used. A repository can gain models and settings after a paper's experiment; combining an old paper row with a newer evaluation variant silently changes the comparison. Save the test definition with the result, not just the headline score.
## The Buying Decision: Useful Context Is a Workload Property
Streaming introduces another distinction. StreamingLLM, first released in September 2023, retained initial attention-sink tokens alongside a bounded recent window to support stable continued processing.[7] Continuing through a long stream does not mean every old fact remains available for later arbitrary retrieval. Conversation duration, retained memory and reliable access to earlier evidence are separate requirements.
A procurement brief should therefore specify the job: input sizes, evidence dependencies, acceptable error, cost ceiling and review process. The vendor's maximum window is one constraint in that brief. The workload test determines whether it provides useful capacity. A smaller window with better evidence selection can be adequate for one task; another task may require broad context and careful synthesis.
### An effective length needs a stated test
A number such as 8K has no universal reliability meaning without the benchmark, model version, task and acceptance threshold. Historical failures justify better evaluation. They do not prove that every current model fails at the same length.
The strategic advantage belongs to teams that know which evidence their applications need and can measure whether it was used. Buying room for more text is easy. Buying reliable work requires a test that survives the extra pages.
## Sources
1. [Lost in the Middle: How Language Models Use Long Contexts](https://arxiv.org/abs/2307.03172)
2. [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://arxiv.org/abs/2404.06654)
3. [NoLiMa: Long-Context Evaluation Beyond Literal Matching](https://arxiv.org/html/2502.05167v3)
4. [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://arxiv.org/abs/2308.14508)
5. [L-Eval: Instituting Standardized Evaluation for Long Context Language Models](https://arxiv.org/abs/2307.11088)
6. [∞Bench: Extending Long Context Evaluation Beyond 100K Tokens](https://arxiv.org/abs/2402.13718)
7. [Efficient Streaming Language Models with Attention Sinks](https://arxiv.org/abs/2309.17453)
8. [NoLiMa official evaluation repository](https://github.com/adobe-research/NoLiMa)
*Last updated: November 9, 2026*
---
*Source: [LLM Rumors](https://www.llmrumors.com/news/long-context-position-bias-evaluation)*