Model Architecture

Every specific structural choice covered so far in this glossary — the Transformer’s attention mechanism, MoE’s routing, dense versus sparse activation — is really a decision made at one particular level. This article names that level directly: model architecture.

The simple definition

Model architecture refers to the overall structural design of a model — how its layers, components, and connections are organized — as distinct from its parameters (the specific learned values) or its training data (what it learned from). Recall from the Transformer article’s core architecture: encoder or decoder, attention mechanism, feed-forward networks, arranged in a specific, repeated pattern. That entire structural blueprint is the model’s architecture; the specific numbers filling in that blueprint, learned through training, are its parameters.

Why this distinction matters, and why architecture is a separate design layer

Recall from the Weights article that a model’s actual “knowledge” lives in its learned parameter values, discovered through training. Architecture is different — it’s the fixed structure those parameters get organized into, decided by human engineers before training ever begins. Two models can share the exact same architecture (say, both decoder-only Transformers) while having completely different learned parameters, because they were trained on different data. Conversely, two models trained on similar data can behave very differently if their underlying architectures differ — dense versus MoE, as covered throughout this phase, being a genuine, consequential example.

flowchart LR
    A[Model Architecture: fixed structural design, decided before training] --> C[Training process fills in specific parameter values]
    B[Training Data] --> C
    C --> D[A specific, trained model]

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of a building’s architectural blueprint — how many floors, where the load-bearing walls go, how rooms connect — versus the specific furniture, paint colors, and decorations that eventually fill that structure. The blueprint is decided once, upfront, by an architect; what actually fills the rooms can vary enormously even within the exact same underlying building design.

Where this breaks down: A building’s blueprint, once constructed, is essentially fixed and expensive to change. A model’s architecture, while genuinely fixed once training begins, is a much cheaper thing to experiment with before that commitment — researchers can and do test many different architectural variants at small scale before deciding which one to actually scale up and invest enormous compute into training, a genuine, deliberate research process this article’s examples will illustrate.

What architectural choices actually cover, concretely

This is worth naming directly, tying together decisions covered throughout this entire glossary. Architecture includes: whether a model is encoder, decoder, or both, as covered in the Encoder and Decoder articles; how many layers deep it is, and how wide each layer’s hidden dimension is, as covered in the Hidden Layer article; how many attention heads each layer uses, as covered in the Multi-Head Attention article; and, as this entire phase has covered, whether the model is dense or sparse, and if sparse, how many experts and how the router works. Every one of these is a genuine, deliberate design decision, made independently of what data the model will eventually train on.

Why architecture research has become its own genuine competitive battleground

This is worth stating directly, connecting to the real stories covered throughout this phase. Recall from the MoE article’s DeepSeek story — the market’s reaction wasn’t primarily about DeepSeek having better training data or a bigger budget; it was fundamentally about DeepSeek achieving strong results through smarter architectural choices, getting more capability per unit of compute spent. This is exactly why frontier AI labs invest heavily in architecture research specifically, alongside data and scale — a genuinely better architecture can deliver capability improvements that no amount of additional training data or compute, applied to a less efficient structure, would match.

A concrete example, layered

For a simple beginner example: choosing between a 12-layer, narrow architecture and a 6-layer, wide architecture for a small model, holding total parameter count roughly constant, is a genuine architectural decision that can affect the model’s final capability — a real, empirical question researchers test rather than something with one obviously correct answer. For a production example: GPT, Llama, DeepSeek, and Mixtral, all referenced throughout this glossary, represent genuinely different architectural families and choices — decoder-only dense Transformers, MoE Transformers with different expert counts and routing strategies — each a real, documented, published design decision, not simply different amounts of the same underlying structure.

Why architecture and scale are genuinely separate levers

It’s worth being direct about this, tying together the Scaling Laws article coming next in this phase. A model can be improved by scaling up an existing architecture (more layers, more parameters, more data) or by adopting a genuinely different, more efficient architecture at the same scale — these are two distinct paths to better performance, and real research progress in the field has come from both, not just from making existing architectures bigger.

Read a real architecture instead of seeing one black box

Google documents Gemma 2 27B as a decoder-only model with 46 decoder layers, a token embedding size of 256,000 × 4,608, and grouped-query attention. Its query projections use 32 heads while key and value projections use 16 heads. Its feed-forward section expands from 4,608 dimensions to 36,864 before projecting back.

Token ID
   ↓
256,000-word embedding table → 4,608-number token vector
   ↓
46 × [attention → normalization → feed-forward → normalization]
   ↓
vocabulary scores → next-token probabilities

These choices are architecture. The billions of learned numbers inside the matrices are parameters. Training changes those values, but it does not normally turn 46 layers into 47 or replace grouped-query attention with another mechanism. Source: Google’s Gemma 2 architecture explanation.

Common misconception

A frequent beginner assumption: that “architecture” and “model size” mean roughly the same thing — that a bigger model is automatically a different architecture. As this article has explained, architecture describes structural design choices, independent of scale — you can have two models with the exact same architecture at very different sizes (a small GPT-style model and a large one share the same fundamental design), or two models of similar size with genuinely different architectures (a dense model versus a sparse one), and conflating the two obscures a real, important distinction.

Where this fits in what comes next

You now understand model architecture as the overall structural design layer sitting above specific parameters and training data. The next article, Scaling Laws, covers what happens when you hold architecture roughly fixed and systematically study how performance changes as you scale up parameters, data, and compute together — a genuine, published, and surprisingly dramatic area of research.

In one sentence

Model architecture is the overall structural blueprint of a model — layer count, attention design, dense versus sparse activation — decided before training and independent of the specific learned parameter values that training later fills in, and it has become a genuine, real competitive battleground in its own right, proven dramatically by architectural innovation alone reshaping the AI industry’s market expectations.

Related Terms

• Transformer
• Mixture of Experts (MoE)
• Weights
• Hidden Layer
• Scaling Laws