The Sparse Model article covered architectures that deliberately leave most of their capacity unused per input. This article covers the direct opposite, and the historical default: a dense model.
The simple definition
A dense model is one where every parameter participates in processing every input — no routing, no selective activation, every single weight genuinely contributes to every single computation. Recall from the Dense Vector article’s core idea, back in the Data Representation phase — nearly every position holding a meaningful value. A dense model applies that same “everything participates” principle to its parameters: GPT-3, BERT, and every Transformer architecture covered earlier in this glossary, before this phase introduced sparsity, are dense models.
Why dense was the default for so long, and still often is
Recall from the Feed-Forward Network article’s description of a standard Transformer layer — one shared feed-forward network, applied to every single token, every single time. This is the simplest, most predictable way to build a neural network: every parameter is trained on every example, every parameter contributes to every prediction, with no routing decisions or specialization to manage. Dense models remain the right choice for the vast majority of real applications precisely because of this simplicity — easier to train reliably, easier to reason about, and without the routing-imbalance challenges covered in the MoE article.
flowchart LR
A[Token arrives] --> B[Every parameter processes it]
B --> C[Full model capacity engaged, every single time]ANALOGY vs. TECHNICAL REALITY
Analogy: Think of a small, close-knit team where every single team member reviews and contributes to every project, regardless of their specific individual expertise — nobody sits out, everyone weighs in on everything, a genuinely simple, predictable way of working, even if it’s not always the most efficient use of any one person’s specific specialty.
Where this breaks down: A human team’s “everyone contributes” involves actual collaborative discussion. A dense model’s every-parameter-participates structure is the exact same weighted-sum-plus-activation calculation covered throughout the Node and Neural Network articles, applied identically and mechanically to every input — no discussion, just a fixed mathematical computation running through every single weight, every single time.
The direct, quantified trade-off against sparse models
This is worth stating precisely, tying together the last two articles. A dense model’s compute cost scales directly and predictably with its total parameter count — a 70-billion-parameter dense model always uses all 70 billion parameters, for every token, meaning training and inference cost track total size exactly. A sparse model, as covered in the previous two articles, can have a far larger total parameter count while keeping per-token compute much lower — but at the real cost of routing complexity and the memory overhead of holding every parameter available, even the unused ones. Dense models trade scaling efficiency for architectural simplicity; sparse models make the opposite trade.
A concrete example, layered
For a simple beginner example: a small, 100-million-parameter dense model used for a simple sentiment-classification task runs every single one of its 100 million parameters on every input sentence — straightforward, predictable, and entirely sufficient for a task that doesn’t need enormous specialized capacity. For a production example: GPT-3, as confirmed throughout OpenAI’s published paper and referenced constantly across this glossary, is a dense model — every one of its 175 billion parameters activates on every single token it processes, a real, well-documented illustration of the traditional, still-dominant architecture the vast majority of widely deployed language models were built on, well before MoE architectures like DeepSeek’s became prominent.
Why dense models still make sense even as sparse ones gain attention
It’s worth being direct about this, since the DeepSeek story covered in the MoE article might suggest dense models are simply being phased out. Many production use cases don’t need the enormous total capacity sparse architectures are built to provide — a dense model sized appropriately for its task remains simpler to train, deploy, and debug, with none of the routing or load-balancing challenges sparse architectures introduce. The choice between dense and sparse is a genuine engineering trade-off based on scale and requirements, not a simple “sparse is strictly newer and better” progression.
See dense and sparse computation side by side
Dense layer with 8 blocks:
token → [1][2][3][4][5][6][7][8] → output
all blocks participate
Sparse MoE layer with 8 experts, top 2:
token → router → [3] and [7] → combined output
six experts skipped for this tokenGoogle’s Gemma 2 family provides concrete dense-model sizes of 2B, 9B, and 27B parameters. The 27B architecture has 46 decoder layers, a hidden width of 4,608, and feed-forward projections reaching 36,864 dimensions. Those structural numbers show why “27B” is only a summary: the model is an organized stack of embeddings, attention, feed-forward networks, and normalization. Source: Google’s Gemma 2 architecture explanation.
Common misconception
A frequent beginner assumption: that dense models are an outdated approach being actively replaced by sparse, MoE-style architectures across the entire industry. As this article has explained, dense models remain the standard, sensible default for the great majority of model sizes and use cases — sparse architectures like MoE become genuinely advantageous specifically at the very largest scales, where the total-capacity-versus-active-compute trade-off starts to matter enormously, not a general replacement for dense architecture at every scale.
Where this fits in what comes next
You now understand both fundamental architectural approaches to how parameters get used. The next article, Parameter Count, covers a specific, closely related concept this whole comparison depends on — and the genuine ambiguity that MoE models introduced into a number that used to have one simple, unambiguous meaning.
In one sentence
A dense model uses every one of its parameters on every input, trading the scaling efficiency sparse architectures offer for genuine architectural simplicity and predictability — and it remains the right, standard choice for the overwhelming majority of real-world models, even as sparse architectures capture headlines at the very largest scales.
Related Terms
- Author
- TechByteByByte Editorial Team
- Reviewed by
- TechByteByByte Admin
- Published
- Last reviewed