Inside an LLM: From Your Prompt to Its Reply

You Typed Five Words. The LLM Performed Billions of Calculations.

Debugging one request—from the Send button to the final streamed answer

You type:

Tell me a political joke

Then you press Send.

A moment later, words begin appearing on the screen.

Perhaps the answer starts like this:

Why did the politician bring a ladder to the debate? They wanted to take their arguments to a higher level.

It feels like one simple action:

question → answer

That is what the interface is designed to make you feel.

Underneath, the request travels through two very different worlds:

1. a production software system that accepts, validates, routes, schedules, checks and streams the request;
2. a Transformer neural network that repeatedly transforms numbers and predicts one token at a time.

flowchart TD
    User["User types a request"] --> App["Production AI system"]
    App --> Model["Transformer inference"]
    Model --> App2["Output checks and streaming"]
    App2 --> User2["Words appear on screen"]

This article will debug that journey as if we could pause the system after every major step.

We will connect the steps to the foundations you are learning:

• software and APIs;
• numbers and vectors;
• matrices;
• machine learning;
• neural networks;
• deep learning;
• dot products;
• softmax;
• attention;
• probability;
• GPU computation;
• inference and generation.

We will also answer a common question:

Where do cosine similarity and embeddings belong—and where do they not belong?

---

Before we begin: this is a representative trace

AI companies do not publish every internal detail of their production systems. Different assistants use different:

• model architectures;
• tokenizers;
• safety systems;
• routing rules;
• prompts;
• serving infrastructure;
• decoding settings.

Therefore, this is not a claim about the hidden internals of one specific commercial product.

It is a technically grounded trace of how a modern assistant built around a decoder-style Transformer can process the request.

All token IDs, vectors, attention scores, logits and probabilities shown below are invented teaching values. Their purpose is to make each transformation visible.

Reading the examples: Some sections introduce a fresh toy example rather than continuing the previous numbers. The calculation notes below distinguish those examples explicitly. “Billions of calculations” in the original opening title describes large-model scale, not a measured count for this request.

---

The complete journey first

flowchart TD
    A["1. Send request"] --> B["2. Gateway validates it"]
    B --> C["3. Conversation is assembled"]
    C --> D["4. Input safety is assessed"]
    D --> E["5. Model and server are selected"]
    E --> F["6. Text becomes tokens"]
    F --> G["7. Token IDs become vectors"]
    G --> H["8. Transformer layers process context"]
    H --> I["9. Next-token probabilities are produced"]
    I --> J["10. One token is selected"]
    J --> K{"Finished?"}
    K -->|No| H
    K -->|Yes| L["11. Output is checked and streamed"]

This diagram hides a lot, but it gives us the route. Now let us walk through it slowly.

Streaming clarification: The overview groups output handling at the end to keep the route readable. In a streaming implementation, output checks and delivery can happen during generation, not only after the finished-answer decision.

---

Part I — Before the words reach the model

Stage 1: the browser creates a network request

The text begins as characters stored by the application:

Tell me a political joke

When you press Send, the browser or mobile app creates a structured request. A simplified version could look like this:

{
  "conversation_id": "conversation-123",
  "messages": [
    {
      "role": "user",
      "content": "Tell me a political joke"
    }
  ],
  "stream": true
}

The exact format differs among products, but the software idea is familiar:

• the text says what you want;
• the role says who supplied it;
• the conversation ID connects it with earlier turns;
• stream: true asks the server to send partial output as it becomes available.

The browser serializes this structure into bytes and sends it over an encrypted network connection.

At this stage, no neural network has interpreted the joke request.

This is ordinary software engineering: user interface, JSON or another protocol, HTTP, encryption, authentication and networking.

Debug snapshot

Field	State
Input	Characters typed by the user
Output	Structured network request
Main concept	API communication
Has the LLM started?	No

---

Stage 2: the gateway checks whether the request can enter

The request usually reaches an API gateway or similar front-door service.

The gateway may check:

• Is the user authenticated?
• Is the request correctly formed?
• Is it too large?
• Has this account exceeded a rate limit?
• Which region should handle it?
• Is the requested model available?
• Should cached conversation data be loaded?

If authentication fails, the system can reject the request before spending expensive model computation.

flowchart TD
    Request["Incoming request"] --> Valid{"Valid structure?"}
    Valid -->|No| Error["Return request error"]
    Valid -->|Yes| Auth{"Authorized?"}
    Auth -->|No| Deny["Return access error"]
    Auth -->|Yes| Continue["Continue processing"]

The gateway may attach operational metadata such as a request identifier. That identifier helps engineers follow one request through logs and services.

This is why the word debugging is appropriate: production teams do not merely inspect the generated sentence. They trace latency, errors and decisions across the entire path.

Debug snapshot

Field	State
Input	Network request
Output	Validated, identified request
Main concept	Distributed systems
Possible failure	Authentication, validation or rate-limit error

---

Stage 3: the system assembles the real prompt

The sentence you typed may not be the only text sent to the model.

A conversational system may assemble:

1. a system or developer instruction;
2. relevant earlier messages;
3. tool results or retrieved documents, if used;
4. your new message;
5. special formatting that marks roles and boundaries.

A simplified prompt might look conceptually like this:

<system>
You are a helpful assistant. Follow the application safety rules.
</system>

<user>
Tell me a political joke
</user>

<assistant>

The final <assistant> marker means:

Continue by generating the assistant’s message.

The real representation may use special tokens rather than readable XML-style tags. The important point is that role information becomes part of the context the model processes.

What if this conversation is long?

Models have a finite context window. If the assembled input is too long, the application may need to:

• remove older messages;
• summarize part of the conversation;
• retrieve only relevant history;
• reject the request;
• use a model with a larger context window.

The application must make this decision before or during prompt preparation.

Debug snapshot

Field	State
Visible user text	Tell me a political joke
Actual model context	Instructions + history + user text + role markers
Main concept	Prompt construction
Important lesson	The model answers the assembled context, not the latest sentence in isolation

---

Stage 4: safety systems interpret the request category

“Political” is a topic. It is not automatically harmful.

A production assistant may apply one or more safety checks before generation, during generation or after generation. These checks may be classifiers, rules, smaller models, the main model itself, or a combination.

For our request, a conceptual classifier might produce scores like:

ordinary political humour      0.91
targeted harassment            0.05
incitement or threat           0.01
other                          0.03

These numbers are illustrative, not real system values.

The system may decide that a harmless, non-targeted political joke can proceed. If the request instead asked for threats, targeted abuse or manipulation, the response path could change.

Possible decisions include:

• allow normally;
• allow with constraints;
• produce a safe redirection;
• refuse;
• send the request for a different review path.

Is this cosine similarity?

Not necessarily.

A safety classifier may use a neural network that ultimately outputs class scores. Another system might compare an embedding with known unsafe examples. The external behaviour does not tell us which technique was used.

Do not assume every “AI comparison” uses cosine similarity.

Debug snapshot

Field	State
Input	Assembled request or user message
Output	Safety labels, scores or control decision
Main ML concept	Classification
Result for our example	Continue, possibly with harmless-humour constraints

---

Stage 5: the router chooses where the request runs

A production platform may host several models or several versions of one model.

A router may consider:

• requested capability;
• input length;
• expected complexity;
• latency target;
• cost;
• server load;
• available hardware;
• region and data-handling requirements.

The request then reaches an inference server. A scheduler decides when it will run.

Why might it wait for a moment?

GPUs are efficient when they process several requests together. A serving system may combine requests into a batch.

Traditional batching waits for a batch to finish before replacing it. Modern LLM servers often use continuous batching: completed requests leave and new requests can join as generation proceeds.

This improves total throughput, though the scheduler must still protect latency and memory.

flowchart LR
    R1["Request A"] --> Scheduler["Inference scheduler"]
    R2["Our joke request"] --> Scheduler
    R3["Request C"] --> Scheduler
    Scheduler --> GPU["Shared model execution"]

Debug snapshot

Field	State
Input	Approved request
Output	Assigned model, server and execution slot
Main concept	Model serving
Important metric	Queue time before computation begins

---

Part II — Turning language into numbers

Stage 6: the tokenizer splits text into tokens

The neural network does not receive raw words directly.

First, a tokenizer converts text into smaller units called tokens.

Tokens may be:

• complete words;
• parts of words;
• punctuation;
• spaces joined with neighbouring text;
• bytes or byte-derived units;
• special control markers.

Different tokenizers split the same sentence differently.

For teaching purposes, imagine:

"Tell me a political joke"

↓

["Tell", " me", " a", " political", " joke"]

This split is illustrative. A real tokenizer may produce a different result.

Why not use one token per word?

Human languages contain huge numbers of words, names, spellings and word forms. A word-only vocabulary would be enormous and would still encounter unknown words.

Subword tokenization creates a compromise:

• common pieces can remain together;
• uncommon words can be built from smaller pieces;
• the vocabulary stays manageable.

For example, an unfamiliar word might become:

"unpredictability"

↓

["un", "predict", "ability"]

The actual split depends on the tokenizer’s learned vocabulary and algorithm.

---

Stage 7: tokens become token IDs

The tokenizer has a vocabulary—a mapping between tokens and integer identifiers.

A toy vocabulary might say:

"Tell"       → 4512
" me"        → 502
" a"         → 264
" political" → 7821
" joke"      → 9134

Our request becomes:

[4512, 502, 264, 7821, 9134]

These IDs are labels, not quantities.

Token 9134 is not “more meaningful” than token 502. We should not add or average token IDs and expect useful language relationships.

The IDs tell the model which rows to look up in an embedding table.

Debug snapshot

Representation	Example
Characters	Tell me a political joke
Tokens	["Tell", " me", " a", " political", " joke"]
Token IDs	[4512, 502, 264, 7821, 9134]
Main concept	Encoding categories as indices

---

Stage 8: token IDs become vectors

A token ID points to one row of a learned embedding matrix.

Imagine a very small vocabulary with six tokens and four numbers per embedding:

$$E= \begin{bmatrix} 0.2 & -0.1 & 0.7 & 0.4\\ -0.5 & 0.8 & 0.1 & 0.3\\ 0.6 & 0.2 & -0.4 & 0.9\\ 0.1 & 0.5 & 0.6 & -0.2\\ -0.3 & 0.7 & 0.2 & 0.5\\ 0.8 & -0.4 & 0.3 & 0.1 \end{bmatrix}$$

Each row belongs to one token.

Table-size clarification: This six-row matrix is a new miniature illustration, not a lookup table indexed by the earlier IDs such as 7821. A real embedding table needs a row for every supported token ID.

If the ID for " political" points to the fifth row, its initial token vector is:

$$[-0.3, 0.7, 0.2, 0.5]$$

Real LLM vectors have far more dimensions.

A vector is simply an ordered list of numbers. Inside a neural network, it is the working representation that layers repeatedly transform.

The model now has one vector for each token position.

"Tell"       → [ ... many numbers ... ]
" me"        → [ ... many numbers ... ]
" a"         → [ ... many numbers ... ]
" political" → [ ... many numbers ... ]
" joke"      → [ ... many numbers ... ]

These initial vectors come from parameters learned during training.

---

Token embeddings are not yet the final meaning

The same token can mean different things in different contexts.

Consider:

The bank approved my loan.
We sat on the river bank.

The initial lookup for bank may be the same. After attention and later layers process the surrounding tokens, the contextual representation of bank becomes different in the two sentences.

This distinction matters:

• initial token embedding: learned starting vector for the token ID;
• contextual hidden state: vector after the model incorporates surrounding context.

The Transformer does most of its reasoning-like representation building in those contextual transformations.

---

Stage 9: the model adds position information

Without position, these would contain the same tokens:

Dog bites person.
Person bites dog.

But their meanings differ because order differs.

Transformers therefore incorporate position information. Architectures do this in different ways. The original Transformer added sinusoidal positional encodings. Many later models use learned positions or rotary position mechanisms.

Conceptually, the model needs to distinguish:

position 0 → "Tell"
position 1 → " me"
position 2 → " a"
position 3 → " political"
position 4 → " joke"

You can imagine each token entering the network with two kinds of information:

token identity + position/context-order information

Do not treat this plus sign as proof that every modern architecture literally adds a separate position vector. The implementation varies. The purpose remains: preserve order.

---

Part III — Inside one Transformer layer

The model is deep because the transformation repeats

A large language model contains many Transformer blocks stacked on top of one another.

flowchart TD
    X["Token vectors"] --> L1["Transformer block 1"]
    L1 --> L2["Transformer block 2"]
    L2 --> L3["Transformer block 3"]
    L3 --> More["Many more blocks"]
    More --> Final["Final hidden states"]

Each block generally contains two major operations:

1. attention, where token positions exchange relevant information;
2. a feed-forward network, where each position transforms its representation.

Residual connections and normalization help information and gradients move through the deep network.

We will debug one simplified attention head first.

---

Stage 10: create Query, Key and Value vectors

For each token representation $x$, the model calculates three new vectors:

$$Q=xW_Q$$ $$K=xW_K$$ $$V=xW_V$$

$W_Q$, $W_K$ and $W_V$ are learned weight matrices.

This is matrix multiplication. The model learned these weights during training.

The names can be understood with a search analogy:

• Query: What information is this position looking for?
• Key: What kind of information does this position offer?
• Value: What information should be passed along if this position is relevant?

The analogy is not a literal database query. Q, K and V are numerical projections learned by the network.

For the token position representing joke, one head might need information from political to understand what kind of joke is requested.

---

Stage 11: use dot products to calculate attention scores

Suppose the query vector for joke is:

$$Q_{joke}=[1,2]$$

The key vectors for two earlier tokens are:

$$K_{political}=[1,1]$$ $$K_{me}=[-1,0]$$

Calculate the dot products.

joke query against political key

$$[1,2]\cdot[1,1]=(1\times1)+(2\times1)=3$$

joke query against me key

$$[1,2]\cdot[-1,0]=(1\times-1)+(2\times0)=-1$$

The higher raw score suggests that, for this head and this layer, the political position aligns more strongly with what the joke position is seeking.

These are teaching values. Real attention heads use much larger vectors and compute scores for many positions in parallel.

---

What does the dot product measure here?

Mechanically, dot product means:

1. multiply matching dimensions;
2. add the results.

Geometrically:

$$A\cdot B=\|A\|\|B\|\cos(\theta)$$

It combines:

• directional alignment;
• vector magnitudes.

In attention, Q and K are learned specifically so their dot product becomes a useful compatibility score.

The model does not attach an English label such as “political relevance = 3.” The training process shapes the numerical space so useful relationships help next-token prediction.

---

Is attention using cosine similarity?

In the standard Transformer formula, no.

Scaled dot-product attention uses:

$$\text{Attention}(Q,K,V)= \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}+M\right)V$$

The core comparison is a scaled dot product, not cosine similarity.

Cosine similarity divides by both vector magnitudes:

$$\cos(\theta)=\frac{Q\cdot K}{\|Q\|\|K\|}$$

Standard attention does not perform that exact normalization.

Where is cosine similarity common?

Cosine similarity is often used outside the LLM’s token-by-token attention loop for:

• semantic search;
• RAG retrieval;
• clustering;
• duplicate detection;
• recommendation systems;
• comparing sentence or document embeddings.

Both attention and RAG can involve vectors, but they use vectors for different jobs.

System	What is compared?	Common comparison
Transformer attention	Query and key vectors inside a layer	Scaled dot product
RAG retrieval	Query embedding and document embeddings	Cosine, dot product or distance

This distinction prevents a major beginner misunderstanding.

---

Stage 12: scale the attention scores

Dot products can become large when the key/query dimension is large. Very large scores can push softmax into extremely sharp regions and make learning unstable.

The Transformer divides scores by:

$$\sqrt{d_k}$$

If our key dimension is 2:

$$\sqrt{d_k}=\sqrt{2}\approx1.414$$

The score 3 becomes:

$$\frac{3}{1.414}\approx2.12$$

The score -1 becomes:

$$\frac{-1}{1.414}\approx-0.71$$

Scaling does not change which score is larger. It controls their numerical range before softmax.

---

Stage 13: apply the causal mask

During generation, a token position must not look at future tokens that have not been generated yet.

Suppose positions are:

0 Tell
1 me
2 a
3 political
4 joke

When calculating the representation at position 3, the model may attend to positions 0 through 3, but not position 4 if position 4 is considered future in that computation.

A causal mask assigns an effectively impossible score to forbidden future positions before softmax.

Allowed attention pattern

Tell       → Tell
me         → Tell, me
a          → Tell, me, a
political  → Tell, me, a, political
joke       → Tell, me, a, political, joke

This causal structure makes next-token prediction possible: the model learns to predict what follows using only what came before.

---

Stage 14: softmax converts scores into weights

Assume the scaled, allowed scores for one query are:

Tell       0.20
me        -0.10
a          0.05
political  2.12
joke       0.70

These scores are not probabilities. They can be negative and do not add to 1.

Softmax converts them into positive weights that sum to 1:

$$\text{softmax}(z_i)=\frac{e^{z_i}}{\sum_j e^{z_j}}$$

A toy result could be:

Tell       0.08
me         0.06
a          0.07
political  0.59
joke       0.20
           ----
total      1.00

Calculation check: The weights just shown are a separate simplified distribution for the next weighted-sum example. Applying softmax to the displayed scores [0.20, -0.10, 0.05, 2.12, 0.70] actually gives approximately [0.0903, 0.0669, 0.0777, 0.6161, 0.1489]. Also, Stage 14 starts a fresh score example: its me score is -0.10, not the -0.71 calculated in Stage 12.

Now political receives the largest attention weight.

Softmax does not prove that political is objectively the most important word. It says this attention head, at this layer and this position, produced that distribution.

Another head or layer may focus elsewhere.

---

Stage 15: calculate a weighted sum of Value vectors

Each token also has a Value vector.

For a tiny example:

V_Tell       = [ 0.2,  0.1]
V_me         = [-0.1,  0.3]
V_a          = [ 0.0,  0.1]
V_political  = [ 0.8,  0.6]
V_joke       = [ 0.5, -0.2]

The attention output is the weighted combination:

$$0.08V_{Tell}+0.06V_{me}+0.07V_a+0.59V_{political}+0.20V_{joke}$$

For the first dimension:

$$(0.08\times0.2)+(0.06\times-0.1)+(0.07\times0)+(0.59\times0.8)+(0.20\times0.5)$$ $$=0.016-0.006+0+0.472+0.100=0.582$$

For the second dimension:

$$(0.08\times0.1)+(0.06\times0.3)+(0.07\times0.1)+(0.59\times0.6)+(0.20\times-0.2)$$ $$=0.008+0.018+0.007+0.354-0.040=0.347$$

The output is:

$$[0.582,0.347]$$

Keeping the examples consistent: This weighted sum is correct for the deliberately simplified weights [0.08, 0.06, 0.07, 0.59, 0.20]. Using the actual softmax of Stage 14’s scores instead would give approximately [0.5787, 0.3768] with these same Value vectors.

This new vector contains a context-dependent mixture of information from allowed positions.

That is the heart of attention:

Use learned comparisons to decide how much information to gather from other token positions.

---

Stage 16: multi-head attention repeats this in parallel

One attention head has one set of Q, K and V projections.

Models use multiple heads so several learned relationship patterns can be processed in parallel.

For our sentence, different heads might—purely for intuition—be useful for:

• request structure: Tell relates to the expected response action;
• topic: political modifies joke;
• speaker relationship: me indicates who receives the response;
• phrase structure: a political joke forms the requested object.

We should be cautious about assigning a clean human purpose to every real head. Learned representations are distributed and individual heads do not always have simple interpretations.

The outputs from the heads are combined and projected back into the model’s working dimension.

flowchart TD
    X["Token representations"] --> H1["Attention head 1"]
    X --> H2["Attention head 2"]
    X --> H3["Attention head 3"]
    X --> H4["Attention head 4"]
    H1 --> Combine["Combine and project"]
    H2 --> Combine
    H3 --> Combine
    H4 --> Combine

---

Stage 17: residual connection and normalization

Instead of replacing the old token representation completely, the block adds the attention output back to it through a residual connection.

Conceptually:

$$\text{new state}=\text{old state}+\text{attention update}$$

Why preserve the old state?

Imagine editing a document by receiving suggested changes. You want the original plus a useful update—not a complete rewrite at every tiny step.

Residual connections also help very deep networks train because information and gradients have shorter paths through the network.

Normalization keeps activation scales manageable. Exact ordering and normalization type vary by architecture.

The safe mental model is:

• residual connections preserve and update information;
• normalization stabilizes the scale of the representations.

---

Stage 18: the feed-forward network transforms each position

After attention mixes information across positions, a feed-forward neural network transforms the representation at each position.

A simplified form is:

$$\text{FFN}(x)=W_2\,\sigma(W_1x+b_1)+b_2$$

where:

• $W_1$ and $W_2$ are learned matrices;
• $b_1$ and $b_2$ are biases in architectures that use them;
• $\sigma$ is a nonlinear activation function.

Why is nonlinearity required?

If every layer performed only linear matrix multiplication, many layers could collapse mathematically into one larger linear transformation. Nonlinear activation functions allow the network to model more complex relationships.

You can think of the two major sublayers as:

• attention: gather relevant context from token positions;
• feed-forward network: transform what was gathered.

Then the updated vectors enter the next Transformer block, where the process repeats with different learned weights.

---

What changes across layers?

The first layer does not write an English explanation and hand it to the second layer.

Every stage remains numerical.

initial token vectors
       ↓
context-adjusted vectors from layer 1
       ↓
richer transformed vectors from layer 2
       ↓
...
       ↓
final hidden vectors

Earlier layers may capture useful local or syntactic patterns. Later layers can build representations useful for the final prediction. But this is an approximate interpretive picture, not a strict rule assigning one human-understandable job to every layer.

At no point must the model explicitly create a variable named:

intent = "political humour"

Its internal state remains distributed across many numbers.

---

Part IV — Turning the final vector into the next word

Stage 19: use the final hidden state to produce logits

After the context passes through all Transformer layers, the model uses the hidden state at the final relevant position to predict the next token.

The vector is projected to one score for every token in the vocabulary:

$$\text{logits}=hW_{vocab}+b$$

If the vocabulary contains 100,000 tokens, the output contains 100,000 scores.

These raw scores are called logits.

A tiny example:

"Why"       8.2
"Here"      6.1
"I"         4.8
"Politics"  3.5
"Banana"   -1.2

Higher logit means the model currently favours that token more strongly. Logits are not probabilities yet.

---

Stage 20: softmax creates a probability distribution

Softmax converts vocabulary logits into probabilities that sum to 1.

"Why"       0.54
"Here"      0.18
"I"         0.09
"Politics"  0.05
all others   0.14
             ----
total        1.00

Again, these values are illustrative.

Probability clarification: This distribution is another independent illustration, not the softmax of Stage 19’s displayed logits. At temperature 1, treating those five logits as the entire toy vocabulary gives approximately 85.84%, 10.51%, 2.86%, 0.78%, and 0.0071%. A real full-vocabulary distribution also depends on all omitted logits.

The model has not yet produced the full joke. It has produced a probability distribution for one next token.

This is the fundamental objective behind an autoregressive language model:

$$P(\text{next token}\mid\text{all previous tokens})$$

---

Stage 21: the decoding strategy selects one token

The application or model server must choose a token from the distribution.

Greedy decoding

Always choose the highest-probability token.

In our example, choose Why.

This is predictable but can produce repetitive or overly safe text.

Sampling

Select randomly according to the distribution. A token with probability 0.54 is more likely than one with 0.05, but is not guaranteed.

Sampling can create variety.

Temperature

Temperature adjusts how sharp or flat the distribution is before selection.

$$P_i=\text{softmax}\left(\frac{z_i}{T}\right)$$

• Lower temperature makes high-scoring tokens more dominant.
• Higher temperature distributes more probability to alternatives.
• At or near deterministic settings, implementations commonly approximate greedy behaviour.

Top-k and top-p

• Top-k: sample only from the $k$ highest-scoring candidates.
• Top-p: keep the smallest candidate set whose cumulative probability reaches a chosen threshold.

These settings influence variation, but they do not add new knowledge to the model.

---

Why can the same prompt produce different jokes?

If sampling is used, more than one continuation can be plausible.

The first token might be:

Why...
Here...
Sure...
Politics...

Once the first token changes, the context for the second prediction changes. The paths can quickly diverge.

flowchart TD
    Prompt["Tell me a political joke"] --> A["Why"]
    Prompt --> B["Here"]
    Prompt --> C["Politics"]
    A --> A2["did the politician..."]
    B --> B2["is a light one..."]
    C --> C2["is like..."]

The model is not retrieving one permanently stored joke response. It is constructing a continuation one token at a time from conditional probabilities.

---

Stage 22: append the selected token and repeat

Suppose the selected token is Why.

The context now becomes conceptually:

<user> Tell me a political joke
<assistant> Why

The model performs the next-token process again:

P(next token | prompt + "Why")

Perhaps it selects did, then the, then politician, and so on.

Generation is autoregressive:

predict token 1
append token 1
predict token 2
append token 2
predict token 3
append token 3
...

It stops when:

• the model produces an end-of-message token;
• a maximum output limit is reached;
• a stop sequence appears;
• a safety or operational system interrupts generation;
• the client disconnects.

---

Stage 23: the KV cache avoids repeating all attention work

Without caching, the model would repeatedly recalculate keys and values for every earlier token at every generation step.

That would waste computation.

During the first pass over the prompt—called prefill—the server calculates internal states for the input tokens. It stores attention Keys and Values in a KV cache.

During each later decode step, the model calculates the new token’s information and reuses cached K and V tensors for earlier positions.

flowchart LR
    Prompt["Prompt tokens"] --> Prefill["Prefill once"]
    Prefill --> Cache["Store K and V"]
    Cache --> D1["Decode token 1"]
    D1 --> D2["Decode token 2"]
    D2 --> D3["Decode token 3"]

Why does KV cache matter?

It reduces repeated computation, but consumes GPU memory. Long contexts and many simultaneous users require large caches.

Serving engines carefully manage this memory. Techniques such as paged attention and prefix caching are infrastructure optimizations around this problem.

Prefill versus decode

Phase	Work
Prefill	Process the input prompt, usually many tokens in parallel
Decode	Generate new tokens one at a time per sequence

Decode is sequential across the tokens of one answer because token 20 depends on tokens 1–19.

Different user requests can still be batched together on the GPU.

---

Part V — Why GPUs and matrices appear everywhere

The Transformer sees matrices, not floating words

After token embedding, the input can be represented as a matrix.

If we have five tokens and each token vector has four dimensions, the input shape is:

$$5\times4$$

                 dimension
              1     2     3     4
Tell        [ 0.2, -0.1,  0.7,  0.4 ]
me          [-0.5,  0.8,  0.1,  0.3 ]
a           [ 0.6,  0.2, -0.4,  0.9 ]
political   [-0.3,  0.7,  0.2,  0.5 ]
joke        [ 0.8, -0.4,  0.3,  0.1 ]

Weight matrices transform all these vectors. Attention calculates many dot products. Feed-forward layers perform more matrix multiplications.

Large models repeat these operations across many layers and tokens.

GPUs are valuable because matrix operations contain huge numbers of smaller multiply-and-add operations that can run in parallel.

The GPU is not “understanding politics” as a separate human activity. It is executing the numerical operations that produce context-sensitive representations and token probabilities.

---

Where did the weights come from?

During inference, the model uses learned weights. It normally does not update them for your one request.

The weights were shaped during training.

For the numerical learning process, continue with How LLMs Learn: One Training Step, Explained.

A simplified training example is:

Input:  "The sky is"
Target: " blue"

The model predicts probabilities. If it assigns low probability to the actual next token, a loss function produces an error signal.

Backpropagation calculates how each parameter contributed to the error. An optimizer slightly updates the weights.

flowchart TD
    Data["Training text"] --> Predict["Predict next token"]
    Predict --> Loss["Compare with actual token"]
    Loss --> Gradients["Backpropagate gradients"]
    Gradients --> Update["Optimizer updates weights"]
    Update --> Predict

This repeats over vast amounts of data.

The final weights encode statistical patterns useful for prediction: language structure, associations, formats, styles and information present in training.

Additional training stages can shape instruction-following and safer behaviour.

Training versus our request

Training	Inference
Model learns from examples	Model answers a request
Calculates loss against targets	Predicts continuations
Uses backpropagation	Usually no backpropagation
Updates weights	Keeps weights fixed
Very compute-intensive	Smaller per request, but repeated at scale

Your political-joke request is an inference request.

---

Part VI — What happens after tokens are generated

Stage 24: generated token IDs become text again

The model outputs token IDs.

The tokenizer’s decoding process converts them back into token strings and joins them into readable text.

[5191, 750, 279, ...]

↓ decode

"Why did the ..."

Subword boundaries and whitespace are handled by tokenizer-specific decoding rules.

The final text did not exist inside one vector waiting to be unpacked. It was generated as a sequence of selected token IDs.

---

Stage 25: output safety may inspect the generation

A production system may inspect partial or completed output.

For a political joke, it might check that the generated humour does not become:

• a threat;
• targeted dehumanization;
• prohibited persuasion in a sensitive context;
• disclosure of private personal information;
• another disallowed form of content under the product’s rules.

The response can be allowed, modified through a new generation path, stopped or replaced with a safer answer.

The exact mechanism varies. Safety is not necessarily one single classifier before the LLM. It can be a layered system.

flowchart LR
    Draft["Generated tokens"] --> Check{"Output acceptable?"}
    Check -->|Yes| Stream["Send to user"]
    Check -->|No| Control["Stop, revise or refuse"]

For our harmless example, the response continues.

---

Stage 26: the server streams the response

The server does not always wait for the full answer.

It can send small pieces as they become available:

chunk 1: "Why"
chunk 2: " did the"
chunk 3: " politician"
chunk 4: " bring a ladder"
...

The browser receives each event and appends its text to the message component.

This improves perceived latency. The full answer may take several seconds, but the user sees progress after the first token or text chunk arrives.

Useful serving metrics include:

• time to first token: delay before output begins;
• inter-token latency: time between generated tokens;
• tokens per second: generation rate;
• end-to-end latency: time until the response finishes.

What feels like “the model is typing” is the interface rendering streamed output.

The model is not using a keyboard.

---

Stage 27: the response is stored and observed

Depending on the product and its privacy settings, the application may record operational information such as:

• request ID;
• model version;
• token counts;
• latency;
• error codes;
• safety decisions;
• user feedback;
• resource usage.

Production systems need observability to detect problems such as:

• a tokenizer or prompt change increasing input length;
• one model server becoming slow;
• GPU memory pressure;
• abnormal refusal rates;
• output streaming failures;
• quality regressions after a model update.

Appropriate logging must also protect user privacy and sensitive data.

At this point, our request’s journey is complete.

---

Part VII — Where embeddings, RAG and cosine similarity would enter

Our request did not require RAG

“Tell me a political joke” does not need private or current factual knowledge. A production router may send it directly to the language model without retrieval.

But imagine the request were:

Tell me a joke based on our company’s latest leave policy.

The model may not know that private document. The application could use RAG.

flowchart TD
    Q["User question"] --> QE["Create query embedding"]
    Docs["Policy chunks + embeddings"] --> Search["Vector search"]
    QE --> Search
    Search --> Context["Retrieve relevant policy text"]
    Context --> Prompt["Add text to model prompt"]
    Prompt --> LLM["Generate grounded joke"]

The RAG-specific steps are:

1. split documents into chunks;
2. create one embedding vector for each chunk;
3. store vectors with their original text;
4. create an embedding for the new query;
5. compare query and document vectors;
6. retrieve nearby chunks;
7. add their original text to the LLM prompt.

Cosine similarity might be used in step 5.

Once the retrieved text enters the prompt, the LLM tokenizes it and processes it through attention like the rest of the context.

---

Three different things called “embeddings”

The word embedding appears in related but distinct places.

1. Token embeddings inside the LLM

Each token ID selects a learned starting vector from the model’s embedding matrix.

token ID → token vector

2. Contextual token representations

After Transformer layers, each token position has a vector influenced by surrounding context.

token vector + context transformations → contextual hidden state

3. Sentence or document embeddings for retrieval

A separate embedding model may produce one vector representing a query, sentence or chunk for similarity search.

whole passage → retrieval vector

They are all vectors, but their purposes and spaces differ.

Do not take a random internal token vector from one model and compare it with a document embedding from another model. Vector coordinates are meaningful only in the compatible space that produced them.

---

Dot product versus cosine similarity one last time

Suppose:

$$A=[1,2]$$ $$B=[2,3]$$

Dot product:

$$A\cdot B=(1\times2)+(2\times3)=8$$

Cosine similarity:

$$\frac{A\cdot B}{\|A\|\|B\|} =\frac{8}{\sqrt{5}\sqrt{13}} \approx0.992$$

In attention

Learned Q and K vectors commonly use scaled dot products. Magnitude remains part of the score.

In retrieval

Query and document embeddings may use cosine similarity, dot product or a distance metric depending on how the embedding model was trained.

The practical rule

Do not choose a metric because its name sounds familiar. Use the metric intended for that model and validate it on your real task.

---

Part VIII — A debugging table for the entire request

Stage	Input representation	Operation	Output representation	Field it connects to
1	Characters	Serialize request	Network bytes	Software engineering
2	Request	Validate and authenticate	Approved request	Backend systems
3	Messages	Assemble instructions and history	Full prompt	Prompt engineering
4	Text/features	Classify safety or intent	Scores and decision	Machine learning
5	Request metadata	Route and schedule	Model execution slot	AI infrastructure
6	Text	Split into subwords	Tokens	NLP
7	Tokens	Vocabulary lookup	Token IDs	Encoding
8	Token IDs	Embedding lookup	Vectors	Linear algebra
9	Vectors + order	Add/rotate positional information	Position-aware states	Transformer design
10	Hidden states	Linear projections	Q, K and V vectors	Matrix multiplication
11	Q and K	Dot products	Attention scores	Vector mathematics
12	Scores	Scale and mask	Controlled scores	Numerical stability
13	Controlled scores	Softmax	Attention weights	Probability
14	Weights and V	Weighted sum	Context vector	Attention
15	Context states	Feed-forward transformations	New hidden states	Neural networks
16	Final hidden state	Vocabulary projection	Logits	Deep learning
17	Logits	Softmax	Token probabilities	Probability
18	Probabilities	Sample/select	Next token ID	Decoding
19	Previous + new tokens	Repeat with KV cache	Complete token sequence	Autoregressive inference
20	Token IDs	Decode	Text	Tokenization
21	Generated text	Safety/output handling	Approved chunks	Responsible deployment
22	Text chunks	Network streaming	Visible response	Frontend engineering

---

Part IX — What each subject contributes

If you are learning AI from the foundations, this one request shows why the subjects connect.

Mathematics

You need:

• vectors to represent token states;
• matrices to transform many vectors;
• dot products to create attention scores;
• functions such as softmax to turn scores into distributions;
• probability to understand token selection;
• calculus and gradients to understand training.

Machine learning

You need:

• training examples;
• loss functions;
• generalization;
• evaluation;
• classification for some safety and routing tasks;
• awareness that learned patterns can fail.

Neural networks

You need:

• layers;
• weights and biases;
• activation functions;
• forward propagation;
• backpropagation during training.

Deep learning

You need:

• many stacked learned transformations;
• embeddings;
• attention;
• residual connections;
• normalization;
• large-scale optimization.

NLP

You need:

• tokenization;
• vocabulary design;
• language modelling;
• context and sequence order;
• decoding.

AI engineering

You need:

• APIs;
• prompt assembly;
• inference servers;
• batching;
• GPU memory management;
• KV caching;
• safety layers;
• latency monitoring;
• streaming;
• cost and reliability controls.

The user sees one answer. Producing it is a collaboration among all these fields.

---

Part X — Common misconceptions corrected

“The model searches a database for the joke.”

Not necessarily. A plain LLM request can generate a joke from its learned next-token distribution without external retrieval. RAG or web search is a separate application step when external knowledge is needed.

“The sentence is converted into one vector and decoded into an answer.”

Inside a decoder LLM, the prompt becomes a sequence of token representations. Layers transform them, and the model predicts an answer one token at a time.

“Attention uses cosine similarity.”

Standard Transformer attention uses scaled dot products between Query and Key vectors. Cosine similarity is common in embedding retrieval but is not the default attention formula.

“The highest-probability full sentence is calculated first.”

The model usually produces a distribution for the next token, selects one, appends it and repeats. It does not enumerate every possible full answer.

“Softmax makes the model factual.”

Softmax only converts scores into a distribution. It does not verify truth.

“The GPU understands the sentence.”

The GPU performs numerical operations efficiently. Useful language behaviour emerges from the trained model architecture and weights executed through those operations.

“Inference trains the model on my request.”

Ordinary inference uses fixed model weights. Whether product data is later retained or used for improvement is a separate policy and system question.

“The model decides the whole answer at once.”

Autoregressive generation builds it token by token. Early choices affect later probabilities.

---

The one thing to remember

You typed:

Tell me a political joke

The system did not pass that sentence into a box containing human-style thoughts.

It performed a chain of representation changes:

characters
→ network request
→ assembled prompt
→ tokens
→ token IDs
→ vectors
→ contextual vectors across many layers
→ vocabulary logits
→ probabilities
→ selected token
→ repeat
→ decoded text
→ streamed response

At the application level, software validated, routed, checked and streamed the request.

At the model level, learned matrices transformed vectors. Attention used scaled dot products to move information among token positions. Feed-forward networks transformed the resulting states. Softmax produced next-token probabilities. A decoding strategy selected one token, and the loop continued.

If RAG were needed, embeddings and a similarity measure such as cosine similarity could help retrieve external text before the LLM ran. For this simple joke request, retrieval may add no value.

The final response feels like one act of intelligence because the interface hides the machinery.

Debug the machinery, and you see something more interesting:

A modern AI response is not one operation. It is a carefully engineered journey in which text repeatedly becomes numbers, numbers influence other numbers, and probability turns those transformations back into language—one token at a time.

---

Sources and further reading

• Vaswani et al.: Attention Is All You Need
• Hugging Face: Tokenization algorithms
• PyTorch: Scaled dot-product attention
• PyTorch: Transformer building blocks
• vLLM documentation: inference serving, PagedAttention and continuous batching