Cover: Generated editorial illustration of archive cabinets that all occupy space while only a few drawers are open. It is a metaphor for stored capacity and selective use, not a model diagram or measured server configuration.
TL;DR: The original DeepSeek-V3 report specifies 671 billion total parameters and 37 billion active per token; Qwen's Qwen3-30B-A3B card specifies 30.5 billion total and 3.3 billion active.[5][6] An ideal 16-bit representation of Qwen's full 30.5-billion-parameter weight set is 61 decimal GB before cache, runtime and other overhead. Sparse activation changes which weights do work, not whether the other weights need a storage plan.
A mixture-of-experts model arrives with two attractive numbers. One advertises the capacity of a large network. The other suggests the computation of a smaller one. Both can be accurate while a buyer still reaches the wrong conclusion about what will fit on a server.
This November 9, 2026 analysis uses historical releases to explain that distinction. Mixtral's announcement was published December 11, 2023; its paper followed January 8, 2024. The DeepSeek-V3 report first appeared December 27, 2024, and the Qwen3 report May 14, 2025. None is presented here as a new release or the latest model.[3][4][5][7]
The real story isn't that sparsity eliminates cost. It redistributes cost across arithmetic, storage, routing and communication. An infrastructure buyer needs to understand all four before turning a model card into a capacity plan.
Why This Matters Now
An active-parameter label can make an open-weight model appear smaller than the checkpoint a team must place in memory or storage. Capacity planning must start with total weights, then add the actual serving workload. Otherwise a correct architecture fact becomes an incorrect purchasing assumption.
The Architecture: A Router Selects Work, Not Permanent Ownership
The sparsely gated MoE paper of January 2017 describes conditional computation: a gating network selects a subset of expert networks for an input.[1] The useful intuition is a large workshop that sends each job to selected stations. Keeping some stations idle for one job does not remove them from the workshop.
Mixtral makes the distinction concrete. Its paper describes eight feedforward experts in each MoE layer, with two selected for each token. Shared parts of the network remain outside that selection.[4] This is why “two of eight experts” should not be interpreted as an exact whole-model memory fraction.
Mistral's release specifies 46.7 billion total parameters and 12.9 billion used per token. These are vendor architecture figures.[3] They describe different accounting scopes. The active count is useful for understanding the sparse computation path. The total count is necessary for reasoning about full weight storage.
| Historical model | Total parameters, vendor-reported | Active per token, vendor-reported | What the figures do not establish |
|---|---|---|---|
| Mixtral 8x7B | 46.7B | 12.9B | Minimum server memory or measured latency.[3] |
| Original DeepSeek-V3 | 671B main model | 37B | Minimum server memory or measured latency.[5] |
| Qwen3-30B-A3B | 30.5B | 3.3B | Minimum server memory or measured latency.[6] |
Sources: the vendors' historical release, report and model card. Architecture inventory only; this table is not an inference-performance ranking. The V3 main-model count does not specify the size of a download that also includes an auxiliary prediction module.
The Storage: Total Weights Set the Starting Point
For a simple theoretical calculation, multiply parameter count by bits per weight and divide by eight. If all 30.5 billion Qwen weights were represented uniformly at 16 bits, the raw payload would be 61 billion bytes, or 61 decimal GB. Using the active count instead would yield 6.6 GB, which does not describe storage for the full model.
At an ideal uniform 8 bits, the same total weights would occupy 30.5 GB. At an ideal packed 4 bits, they would occupy 15.25 GB. These are arithmetic representations, not measured checkpoints, supported quantization recipes or demonstrated hardware requirements. Real formats include scales, metadata and sometimes mixed-precision tensors.
Calculated weight payload
The same 30.5B total weights have a 61 GB ideal 16-bit payload
Qwen's vendor-reported total parameter count held fixed. Only the hypothetical uniform representation changes; decimal GB means 1,000,000,000 bytes.
Uniform 16-bit weights
Uniform 8-bit weights
Ideal packed 4-bit weights
LLM Rumors arithmetic: Qwen model card's 30.5B total parameters × bits / 8. Excludes quantization metadata, mixed precision, KV cache, activations, workspaces, replication and runtime. Not measured VRAM, a deployment recommendation, or evidence of equal quality at each precision.
This starting point is deliberately incomplete. A model that fits its weights into aggregate memory can still lack enough headroom for concurrent requests. The deployment question is how the total serving state fits and moves, under the workload the business intends to run.
The System: Sparse Arithmetic Creates Scheduling Work
Switch Transformers, first submitted in January 2021, treats routing complexity, communication and training instability as central obstacles to sparse-model adoption.[2] That history is useful because it explains why an attractive parameter ratio never guaranteed an equally attractive runtime ratio.
MegaBlocks, released in November 2022, addresses the dynamic work shapes of MoE training through block-sparse operations. Its motivation includes the trade-off between dropping tokens and padding computation.[8] The narrow lesson for procurement is that software and scheduling matter. A training-system result is not a universal serving-speed claim.
In a distributed deployment, expert placement can introduce transfers between devices. In an offloaded deployment, weights may move from slower storage or host memory. A selected expert must become available when its token reaches it. Such designs can be useful, but their capacity advantage must be evaluated alongside transfer delays and concurrency. “The weights need a storage plan” does not mean every weight must reside on one GPU at every instant.
The Capacity: Context and Concurrency Add Another Bill
DeepSeek-V2's May 2024 report introduces latent attention to compress the KV cache, alongside its expert architecture.[9] This is a separate mechanism. Weight sparsity does not automatically shrink conversation memory, and a cache optimization does not erase the full weight inventory.
For a business, that separation changes the acceptance test. A short single-user demonstration measures a different operating point from many simultaneous long sessions. Reserve capacity for the actual context lengths, output lengths and concurrency. Include temporary buffers and the serving framework's placement behavior. Treat an out-of-memory failure as a system result, rather than evidence that the model card's active count was wrong.
Qwen3's technical report also describes a thinking-budget mechanism.[7] Additional generated reasoning can change time and token consumption without changing the named checkpoint. Architecture, request policy and workload together determine the deployment, so they belong together in the evaluation record.
The Purchase: Evaluate the Deployed Workload
Ask for an inventory of the total weights, numerical format, expert placement, device count, cache budget and concurrency limit. Then measure the accepted business task. Record model revision, hardware, prompt and output lengths, decoding settings, batch size, time to first token, tail latency and benchmark harness. If speculative decoding is used, include its acceptance rate.
Without those conditions, a headline speed comparison is just a pair of deployment signals. This article deliberately provides no throughput leaderboard. A team can use the same parameter count in very different systems and obtain different economics. The architectural label cannot settle a purchasing decision that depends on utilization and reliability.
Active parameters are not a memory specification
Total parameters describe the full learned weight inventory. Active parameters describe the subset used along a token's computation path. Neither alone specifies end-to-end VRAM, throughput or cost per accepted task. Keep weight storage, runtime state and measured service behavior separate in the capacity model.
The uncomfortable truth is that sparse models reward operators who understand systems. Open weights provide control, but control includes owning the storage and scheduling problem. A model's smaller active number can improve the computation bargain. It cannot sign the infrastructure bill on its own.
Sources & References
Key sources and references used in this article
| # | Source | Outlet | Date | Key Takeaway |
|---|---|---|---|---|
| 1 | Shazeer et al. | 2017-01-23 | Conditional computation expands capacity without using every expert for every input. | |
| 2 | Fedus, Zoph and Shazeer | 2021-01-11 | Sparse models bring routing, communication and training-stability challenges. | |
| 3 | Mistral AI | 2023-12-11 | Mistral specifies 46.7B total and 12.9B active parameters. | |
| 4 | Jiang et al. | 2024-01-08 | Each MoE layer routes each token to two of eight experts. | |
| 5 | DeepSeek-AI | 2024-12-27 | The original V3 report specifies 671B total and 37B active parameters. | |
| 6 | Qwen | Reviewed 2026-11-09 | Model card specifies 30.5B total, 3.3B active, 128 experts and eight selected. | |
| 7 | Yang et al. | 2025-05-14 | Qwen3 includes dense and MoE models, with a thinking-budget mechanism. | |
| 8 | Gale et al. | 2022-11-29 | Block-sparse kernels address dynamic routing and padding trade-offs. | |
| 9 | DeepSeek-AI | 2024-05-07 | Latent attention addresses KV-cache storage separately from expert activation. |
Last updated: November 9, 2026
