How a Phone Chip Actually Runs AI: The A20 Pro Mental Model

TL;DR: A 3-billion-parameter model needs 1.5 GB for raw weights at 4-bit precision, or 6 GB at 16-bit precision, before the app and conversation cache consume memory. CPU, GPU and Neural Engine share the work according to the model and runtime.^[2] A20 Pro’s extra bandwidth and cooling address different bottlenecks: moving data fast enough and sustaining the work without overheating.^[1]

The A20 Pro is best understood as a small, shared workshop. The CPU schedules work. The GPU attacks parallel math. The Neural Engine executes model operations that Core ML has compiled for specialized hardware. None is an all-purpose “AI button”: the runtime’s route changes with format, iOS version, operator support, memory pressure, and developer configuration.^[2] Apple has disclosed processor blocks and relative claims, but not Duo’s RAM, GB/s, clocks, power draw, or a model-and-prompt test harness. Those omissions prevent a speed leaderboard.^[1]

For the launch facts, see iPhone Duo and AirPods 5’s on-device AI story and the A20 Pro comparison. For the model-market context, see our Liquid AI LFM2.5 analysis.

Compute: Three Processors, Different Jobs

The CPU is the coordinator. It handles app code, audio/file preparation, scheduling, tokenization, interface updates, and the small decisions around inference. It is flexible at branching logic and can run small models directly. GPUs and dedicated neural processors can improve throughput or energy use for supported parallel operations; the best route depends on the workload.

The GPU is the wide workbench. It runs the same math across many pieces of a tensor at once, which suits dense matrix multiplication, attention, image processing, and some model formats. A local runtime such as llama.cpp supports Apple’s Metal stack alongside ARM NEON and Accelerate,^[6] and its iPhone SwiftUI sample demonstrates that local inference can be embedded in an app rather than routed to a web service.^[7]

The Neural Engine is the specialized assembly line. Core ML can select CPU, GPU, Neural Engine, or combinations according to what the compiled model can execute. Unsupported operators may run elsewhere, so a whole-graph Neural Engine conversion is a model-by-model achievement. Desert Ant says its Voz transcription graph does that on its tested iPhones; this is evidence about Voz’s conversion and runtime, not every iOS app.^[8]

Memory: Capacity Is Not Bandwidth

Capacity answers, “Can the job fit?” Bandwidth answers, “How quickly can the chip repeatedly fetch what it needs?” Both matter because a language model must read much of its weight data for each generated token. A fast accelerator waiting on memory does not feel fast to a person watching a reply arrive.

The clean weight calculation is deliberately simple. A 3B-parameter model at 4 bits per weight needs 3,000,000,000 × 4 ÷ 8 = 1,500,000,000 bytes, or 1.5 decimal GB, for raw weights. At 16 bits, the same arithmetic is 6 GB. Neither figure includes metadata, quantization scales, the tokenizer, executable code, temporary compute buffers, the operating system, the host app, or the conversation cache. The calculation establishes why bits matter. It does not establish that a 3B model fits or performs well on iPhone Duo, whose RAM capacity is not published.^[1]

Apple’s “50% more unified-memory bandwidth” claim is therefore meaningful but incomplete. It says a shared-memory pipeline may move model data faster than the prior generation under Apple’s comparison, not how much capacity Duo has or how any particular LLM will perform.^[1] A model can fit and still feel slow if repeated weight reads saturate bandwidth. Conversely, a smaller model with a shorter context can feel immediate while a larger model fails to allocate a second buffer.

Quantization and Context: A Moving Memory Budget

Quantization stores a number with fewer bits. Reducing weight precision from 16 bits to four cuts the raw-weight calculation by 75%, but a smaller file is not a lossless model. The runtime must decode it efficiently and the model must retain enough task quality. Apple’s 2024 research used a mixed 2-bit/4-bit strategy because operations tolerate compression differently.^[4]

For full-attention transformer layers, the KV cache is a second, growing store of memory. After the model reads a prompt token, it saves attention “keys” and “values” so the next token can refer back without recomputing the entire conversation. That is why a long conversation can be slower or fail even though the weight file has not changed. Cache size rises with context length and depends on architecture, layers, hidden dimensions, attention-head design, precision, and runtime configuration. It should not be inferred from parameter count alone. Sliding-window attention caps cache growth in the affected layers, while hybrid convolution/attention models do not carry a full attention cache in every layer.^[10][11]

Consider a hypothetical full-attention model with 32 layers, eight KV heads per layer, a head dimension of 128 and 16-bit cached values. Keys plus values consume 2 × 32 × 8 × 128 × 2 = 131,072 bytes per token: 128 KiB. At 4,096 tokens, that is 512 MiB; at 32,768 tokens, 4 GiB. This calculation excludes overhead and describes an illustrative architecture, not A20 Pro, Apple’s model or Liquid’s 230M model.

Some optimizations target this budget directly. Apple’s 2024 research describes efficient KV-cache updates on its Neural Engines; its 2025 technical report adds KV-cache sharing as an architectural optimization.^[5] llama.cpp supports quantized K/V cache options across its backends, but its feature matrix treats capabilities as backend-specific, not a promise that every format is equally fast everywhere.^[4][9] The practical rule is simple: ship the smallest context that serves the feature, then measure the actual memory footprint on a physical device.

Prefill and Decode: Two Different Waits

An LLM session has two jobs. Prefill reads the supplied prompt and constructs the KV cache. It is usually the main contributor to time-to-first-token for a long prompt. Decode produces one new token at a time, repeatedly consulting the saved cache and weights. Its rate is the number people commonly call tokens per second. These are not interchangeable measures.

This changes product design. A local note-summary feature can prefill a bounded document once, generate a short answer, then release the model. A chat assistant with a sprawling prompt and long history pays a larger prefill bill before saying anything. A stable prefix enables cache reuse only when that prefix repeats; new context still has to be read.^[10]

Cold loading is a third wait. Before prefill starts, a phone may need to read the model from storage and prepare it for a hardware backend. Desert Ant reports 20 seconds for Voz’s first Core ML specialization and 0.2 seconds for subsequent loads.^[8] That is why a product should report cold start, time-to-first-token and generation rate separately. A fast answer after a warm-up does not tell the user what the first launch feels like.

Cooling and Cloud: The Limits of a Phone

A phone cannot sustain peak computation indefinitely. It has a thin enclosure, a battery, an operating system that protects responsiveness, and a user who notices a hot device. Apple’s iPhone 18 Pro announcement describes an A20 Pro package with memory beside silicon, clearing the chip’s direct path to a vapor chamber.^[12] Duo also uses A20 Pro with a custom vapor chamber. That is credible evidence of a design aimed at sustained workloads, but it is still not a measured A20 Pro AI result.^[1]

The sensible local workload is narrow and frequent: transcription after a recording, offline text classification, image understanding on a selected photo, private retrieval over a small set of documents, or constrained extraction into a known schema. Liquid’s 230M LFM2.5 explicitly targets data extraction and lightweight tool-routing, while advising against reasoning-heavy math, code generation, and creative writing. That division of labor is a more honest mobile-AI future than pretending every request belongs on a phone.^[11]

For harder work, hybrid is a feature, not a failure. Apple describes on-device processing for many requests and Private Cloud Compute for larger ones. A well-designed app should make the same choice: handle private, low-latency, bounded work locally; ask permission and use a remote model for broad knowledge, difficult reasoning, or work that exceeds the device’s memory and heat budget.^[3]

Return to the workshop inside the phone. The CPU coordinates the job, the GPU and Neural Engine execute the operations they support, memory keeps the materials within reach, and cooling determines how long the work can continue. Improving one part helps only when another part is not already setting the limit. That is why a smaller model, a shorter context or a better runtime can sometimes matter more to an app than another generation of silicon.

A private note-summary feature is a useful way to apply the model. First, fit the weights and working memory within the device’s budget. Then measure cold loading, prompt processing and answer generation separately. Finally, repeat the task on a warm phone and decide which requests need a server. Each step connects a part of the chip to something the user can feel: whether the app opens promptly, responds smoothly and finishes without becoming a battery problem.

A20 Pro’s bandwidth and thermal changes expand the engineering room for those choices. They do not choose the workload or build the experience. Once that distinction is clear, on-device AI becomes easier to judge: follow the data, identify the limiting resource, and measure the completed task. The value of the chip is the useful work it lets the phone finish. The value of the app is knowing which work belongs there.

Last updated: September 10, 2026