# LLM.txt - MoE Models Have Two Sizes. Your Memory Bill Uses the Bigger One. ## Article Metadata - **Title**: MoE Models Have Two Sizes. Your Memory Bill Uses the Bigger One. - **URL**: https://www.llmrumors.com/news/mixture-of-experts-active-total-memory - **Publication Date**: November 9, 2026 - **Reading Time**: 7 min read - **Tags**: Mixture of Experts, Model Memory, Open Weights, DeepSeek, Qwen, Mixtral, AI Infrastructure, Inference Economics - **Slug**: mixture-of-experts-active-total-memory ## Summary Active parameters explain only part of sparse AI economics. Historical Mixtral, DeepSeek-V3 and Qwen3 examples separate routing, weight storage and production capacity. ## Key Topics - Mixture of Experts - Model Memory - Open Weights - DeepSeek - Qwen - Mixtral - AI Infrastructure - Inference Economics ## Content Structure This article from LLM Rumors covers: - Data acquisition and training methodologies - Financial analysis and cost breakdown - Comprehensive source documentation and references ## Full Content Preview Cover: Generated editorial illustration of archive cabinets that all occupy space while only a few drawers are open. It is a metaphor for stored capacity and selective use, not a model diagram or measured server configuration. TL;DR: The original DeepSeek-V3 report specifies 671 billion total parameters and 37 billion active per token; Qwen's Qwen3-30B-A3B card specifies 30.5 billion total and 3.3 billion active.[5][6] An ideal 16-bit representation of Qwen's full 30.5-billion-parameter weight set is 61 decimal GB before cache, runtime and other overhead. Sparse activation changes which weights do work, not whether the other weights need a storage plan. A mixture-of-experts model arrives with two attractive numbers. One advertises the capacity of a large network. The other suggests the computation of a smaller one. Both can be accurate while a buyer still reaches the wrong conclusion about what will fit on a server. This November 9, 2026 analysis uses historical releases to explain that distinction. Mixtral's announcement was published December 11, 2023; its paper followed January 8, 2024. The DeepSeek-V3 report first appeared December 27, 2024, and the Qwen3 report May 14, 2025. None is presented here as a new release or the latest model.[3][4][5][7] The real story isn't that sparsity eliminates cost. It redistributes cost across arithmetic, storage, routing and communication. An infrastructure buyer needs to understand all four before turning a model card into a capacity plan. An active-parameter label can make an open-weight model appear smaller than the checkpoint a team must place in memory or storage. Capacity planning must start with total weights, then add the actual serving workload. Otherwise a correct architecture fact becomes an incorrect purchasing assumption. The Architecture: A Router Selects Work, Not Permanent Ownership The sparsely gated MoE paper of January 2017 describes conditional computation: a gating network selects a subset of expert networks for an input.[1] The useful intuition is a large workshop that sends each job to selected stations. Keeping some stations idle for one job does not remove them from the workshop. Mixtral makes the distinction concrete. Its paper describes eight feedforward experts in each MoE layer, with two selected for each token. Shared parts of the network remain outside that selection.[4] This is why “two of eight experts” should not be interpreted as an exact whole-model memory fraction. Mistral's release specifies 46.7 billion total parameters and 12.9 billion used per token. These are vendor architecture figures.[3] They describe different accounting scopes. The active count is useful for understanding the sparse computation path. The total count is necessary for reasoning about full weight storage. | Historical model | Total parameters, vendor-reported | Active per token, vendor-reported | What the figures do not establish | | --- | --- | --- | --- | | Mixtral 8x7B | 46.7B | 12.9B | Minimum server memory or measured latency.[3] | | Original DeepSeek-V3 | 671B main model | 37B | Minimum server memory or measured latency.[5] | | Qwen3-30B-A3B | 30.5B | 3.3B | Minimum server memory or measured latency.[6] | Sources: the vendors' historical release, report and model card. Architecture inventory only; this table is not an inference-performance ranking. The V3 main-model count does not specify the size of a download that also includes an auxiliary prediction module. The Storage: Total Weights Set the Starting Point For a simple theoretical calculation, ... [Content continues - full article available at source URL] ## Citation Format **APA Style**: LLM Rumors. (2026). MoE Models Have Two Sizes. Your Memory Bill Uses the Bigger One.. Retrieved from https://www.llmrumors.com/news/mixture-of-experts-active-total-memory **Chicago Style**: LLM Rumors. "MoE Models Have Two Sizes. Your Memory Bill Uses the Bigger One.." Accessed November 10, 2026. https://www.llmrumors.com/news/mixture-of-experts-active-total-memory. ## Machine-Readable Tags #LLMRumors #AI #Technology #MixtureofExperts #ModelMemory #OpenWeights #DeepSeek #Qwen #Mixtral #AIInfrastructure #InferenceEconomics ## Content Analysis - **Word Count**: ~1,229 - **Article Type**: News Analysis - **Source Reliability**: High (Original Reporting) - **Technical Depth**: General - **Target Audience**: AI Professionals, Researchers, Industry Observers ## Related Context This article is part of LLM Rumors' coverage of AI industry developments, focusing on data practices, legal implications, and technological advances in large language models. --- Generated automatically for LLM consumption Last updated: 2026-11-10T07:29:41.404Z Source: LLM Rumors (https://www.llmrumors.com/news/mixture-of-experts-active-total-memory)