Every model covered throughout this glossary so far — GPT-3, Llama, the Transformer architecture itself — has shared one quiet assumption: every single parameter gets used on every single token. This phase opens by breaking that assumption entirely, with an architecture called Mixture of Experts.
The simple definition
A Mixture of Experts model splits its feed-forward layers, covered in the Feed-Forward Network article, into many separate sub-networks called “experts,” and for any given token, only activates a small number of them — rather than running every single expert on every single token. A model might have 8, 16, or even hundreds of experts total, but a lightweight router decides, for each individual token, which handful of experts should actually process it — commonly just 2 out of 8, or 2 out of 16.
Why this idea matters, connecting directly to Parameters
Recall from the Parameters article’s discussion of scale — more parameters generally mean more capacity to learn complex patterns, but also more compute required at both training and inference. MoE offers a genuinely clever way out of this trade-off: a model can have an enormous total parameter count — giving it broad capacity and specialized knowledge spread across many experts — while only actually computing through a small active subset of those parameters for any single token. You get much of the capacity benefit of a huge model, without paying the full computational cost of running all of it, every time.
flowchart LR
A[Token arrives] --> B[Router decides which experts are relevant]
B --> C[Expert 3: activated]
B --> D[Expert 7: activated]
B --> E[Experts 1, 2, 4, 5, 6, 8: skipped entirely]
C --> F[Combined output]
D --> FANALOGY vs. TECHNICAL REALITY
Analogy: Think of a large hospital with dozens of specialist doctors — a cardiologist, a neurologist, a dermatologist — rather than one single doctor who has to personally handle every kind of case. When a patient arrives, a triage nurse (the router) quickly directs them to just the one or two specialists actually relevant to their specific problem, rather than making every single doctor in the building examine every single patient.
Where this breaks down: A hospital’s triage nurse applies genuine medical judgment. The router in an MoE model is itself a small, trained neural network — producing a probability distribution over which experts seem most relevant, exactly the mechanism covered in the Probability Distribution article, then selecting the top few — a learned, statistical decision, not a judgment call grounded in real understanding of what each expert actually specializes in.
How this actually gets done, step by step
This connects directly to the Feed-Forward Network article’s place inside a Transformer layer. In a standard Transformer, every token passes through one shared feed-forward network at each layer. In an MoE Transformer, that single feed-forward network gets replaced with several parallel expert networks, plus a small router network. For each token: the router calculates a score for every expert, selects the top-k highest-scoring ones (commonly top-2), sends the token only to those, and combines their outputs — typically weighted by the router’s own confidence scores — into the token’s final representation for that layer.
The real, published origin, and the model that shocked the entire market
This deserves to be told as a genuine, dramatic story, because it happened recently and its consequences were enormous. Google researchers published the Switch Transformer in 2021, demonstrating MoE could scale language models efficiently, and Mistral AI’s Mixtral 8x7B later became a real, widely used open-weight example — an MoE model with 8 experts per layer that matched or beat much larger dense models on many benchmarks. But the story that made MoE a household concept even outside AI circles happened in January 2025: the Chinese startup DeepSeek released its R1 model, built on a Mixture of Experts architecture, and reported training it for roughly 593 billion in market value — the largest one-day market capitalization loss for any single company in U.S. stock market history. Nvidia itself publicly called DeepSeek “an excellent AI advancement,” while independent analysts later noted DeepSeek’s true costs, including hardware and prior research, were likely far higher than the headline $5.6 million figure — but the market’s reaction to a genuinely efficient MoE architecture was real, immediate, and historic regardless.
A concrete example, layered
For a simple beginner example: a small MoE model with 8 experts might learn, through training alone with no explicit instruction, that certain experts become especially good at handling code-related tokens while others specialize in conversational language — an emergent division of labor, discovered automatically rather than assigned by a human designer. For a production example: DeepSeek’s V3 and R1 models, and Mistral’s Mixtral family, are real, published, publicly available MoE architectures — with DeepSeek-V3 reportedly having 671 billion total parameters but activating only around 37 billion per token, a concrete illustration of the “huge total capacity, modest active compute” trade-off this article has described.
Why MoE isn’t a strictly free upgrade
It’s worth being honest about genuine, real engineering costs here. Even though only a few experts activate per token, all the experts’ parameters still need to be loaded into GPU memory, since different tokens in the same batch can route to different experts — meaning MoE models save on compute but not necessarily on memory, a real, documented trade-off. Training also introduces its own challenge: routers can develop a bias toward a small number of “favorite” experts, under-training the rest, a known issue researchers address through techniques that explicitly encourage more balanced expert usage.
Follow one token through a real MoE layer
Suppose an MoE layer has eight experts and uses top-2 routing. The token “photosynthesis” reaches the router, which produces these simplified teaching scores:
Expert: E1 E2 E3 E4 E5 E6 E7 E8
Router score: .04 .08 .61 .03 .17 .02 .03 .02
Selected: ✓ ✓Only Experts 3 and 5 process this token. Their output vectors are mixed using routing weights derived from their scores, and the combined vector continues to the next part of the Transformer. The next token can be sent to a different pair; routing is repeated per token and per MoE layer.
Verified model numbers
| Model | Experts or routing | Total parameters | Parameters active for one token |
|---|---|---|---|
| Mixtral 8x7B | 8 experts, top 2 | about 47B accessible | about 13B |
| DeepSeek-V3 | DeepSeekMoE routing | 671B | 37B |
Mixtral’s paper states that every token selects two of eight feed-forward experts. DeepSeek-V3’s technical report states 671 billion total parameters and 37 billion activated for each token. These are architecture figures, not claims that every 37B parameter subset behaves like a separately trained 37B dense model. Sources: Mixtral of Experts and the DeepSeek-V3 Technical Report.
Common misconception
A frequent beginner assumption: that an MoE model’s “experts” are pre-assigned to specific human-understandable topics (one expert for math, one for history) the way a hospital assigns specialists by medical field. As this article’s analogy breakdown explained, this isn’t accurate — experts specialize purely through the statistical pressures of training, and while some interpretable patterns sometimes emerge, there’s no guarantee any single expert maps cleanly onto a topic a human would recognize.
Where this fits in what comes next
You now understand the specific architecture that made “sparse activation” a mainstream, headline-making concept. The next article, Sparse Model, generalizes this idea beyond MoE specifically — covering the broader category of models where not every parameter is used for every computation.
In one sentence
Mixture of Experts splits a model into many specialized sub-networks and activates only a handful per token through a learned router, delivering much of a huge model’s capacity at a fraction of its computational cost — a genuinely consequential architecture, dramatically proven when DeepSeek’s MoE-based R1 model triggered the largest single-day stock market loss in history.
Related Terms
- Author
- TechByteByByte Editorial Team
- Reviewed by
- TechByteByByte Admin
- Published
- Last reviewed