Before a Neural Network Can Learn, It Must Make a Prediction

Following one forward pass, number by number

A note on the teaching model: This is a small feed-forward classifier with three word labels, not a complete language model. The vector [1, 2] is supplied for teaching; this example does not learn a tokenizer or an embedding. Also, a model’s probability is not a guarantee of real-world correctness or calibrated confidence. Here we measure loss against one known training target.

This is the first article in our three-part journey through one training step:

Forward pass → Backpropagation → Gradient descent

We begin with the only thing a model can do before it learns: make a prediction using the weights it currently has.

Suppose we give a tiny language model this incomplete sentence:

The sky is …

We want it to predict the next token.

It has three possible answers:

blue
green
runs

Now imagine pausing the model at the exact moment it makes its prediction. On our tiny network, we can inspect every number involved. The result will be:

blue:  45.73%
green: 38.58%
runs:  15.69%

How did text become these probabilities?

The answer is forward propagation, also called the forward pass.

Forward propagation is the journey of information from the model’s input to its output.

flowchart LR
    Input["Input"] --> Hidden["Hidden layer"]
    Hidden --> Output["Output scores"]
    Output --> Prob["Probabilities"]
    Prob --> Prediction["Prediction"]

Nothing moves physically through a network like water through a pipe. “Forward” means that each operation uses the result of the previous operation:

input numbers
→ weighted sums
→ activation
→ new numbers
→ more weighted sums
→ output

In this article, we will calculate every step using actual numbers. We will not hide the arithmetic inside a library call. By the end, you will be able to look at a forward pass and see a chain of small, understandable operations—not one large piece of AI magic.

First, where does the forward pass fit?

A complete training step contains:

1. forward propagation;
2. loss calculation;
3. backpropagation;
4. optimizer update.

flowchart TD
    Forward["1. Forward propagation"] --> Loss["2. Calculate loss"]
    Loss --> Backward["3. Backpropagation"]
    Backward --> Update["4. Update weights"]
    Update --> Forward

Forward propagation answers:

Given the model’s current weights and this input, what does the model predict?

Backpropagation answers a different question:

How did each parameter contribute to the prediction error?

This article follows only the forward direction. We will stop after the model produces a prediction and its loss is measured.

That boundary is intentional. Backpropagation deserves its own article because it introduces a different mental model: gradients, the chain rule and assigning responsibility for an error. Mixing all of that into our first journey through the network would make both ideas harder to see.

For now, remember the handoff:

The forward pass makes the prediction and measures the error. The backward pass uses that error to work out how the weights should change.

Does forward propagation happen only during training?

No.

It happens during both training and inference.

During training

forward pass → prediction → loss → backward pass → update weights

During inference

forward pass → prediction

When a deployed model answers your request, it performs forward passes. It normally does not run backpropagation or update its weights for that request.

Meet our tiny neural network

Our input represents:

the sky is

The expected next token is:

blue

To keep the mathematics visible, we will assume that earlier text-processing steps have already produced this two-number input vector:

A vector is simply an ordered list of numbers. “Ordered” matters: [1,2] is not the same vector as [2,1]. Each position carries its own part of the representation. In a real language model, a token is represented using hundreds or thousands of numbers rather than only two.

These two values are teaching values. A real language model uses sequences of much larger vectors.

Our small network has:

• two input values;
• two hidden neurons;
• a ReLU activation;
• three output neurons—one for each candidate token;
• softmax to create probabilities.

flowchart LR
    X1["x₁ = 1"] --> H1["Hidden neuron 1"]
    X1 --> H2["Hidden neuron 2"]
    X2["x₂ = 2"] --> H1
    X2 --> H2
    H1 --> Out["blue / green / runs"]
    H2 --> Out

What is a neuron?

An artificial neuron performs three basic operations:

1. multiply inputs by weights;
2. add the results and a bias;
3. apply an activation function.

where:

• $x_1,x_2$ are inputs;
• $w_1,w_2$ are weights;
• $b$ is a bias;
• $z$ is the value before activation;
• $f$ is the activation function;
• $a$ is the neuron’s output.

The neuron does not independently “understand” a concept. It performs a learned numerical transformation.

What is a weight?

A weight controls how strongly an input affects a neuron.

Suppose:

With weight 0.5:

With weight 3:

With weight -1:

A positive weight pushes in one direction. A negative weight pushes in the opposite direction. A larger absolute value creates a stronger effect.

During training, these weights are adjusted.

During one forward pass, they remain fixed.

What is a bias?

A bias is an additional learned value added after the weighted inputs.

It allows a neuron to shift its response instead of being forced through zero.

Think of weights as deciding the slope and the bias as shifting the starting point.

The numbers enter the network

Our input vector is:

Therefore:

Both values connect to both hidden neurons.

The first hidden neuron receives both numbers

Hidden Neuron 1 has:

weight from x₁ = 0.5
weight from x₂ = 0.3
bias           = 0.1

Its weighted sum is:

Substitute the inputs:

Calculate each part:

Add them with the bias:

Before activation, Hidden Neuron 1 contains:

The second hidden neuron does the same job—with different weights

Hidden Neuron 2 has:

weight from x₁ = -0.2
weight from x₂ =  0.8
bias           = -0.1

Its weighted sum is:

Substitute:

Calculate:

Add everything:

Before activation, Hidden Neuron 2 contains:

Two neurons give us a new vector

The two neuron results form a vector:

This is the hidden layer’s output before applying its activation function.

The word pre-activation simply means “calculated before activation.”

The same calculation as matrix multiplication

So far, we calculated each neuron separately.

Computers arrange the weights into a matrix and calculate the layer together.

Using a row-vector convention:

Then:

Multiply:

For the first output position:

For the second:

Therefore:

The per-neuron calculation and matrix multiplication are not different methods. Matrix multiplication performs all the dot products together.

A dot product is simply the operation we have already performed: multiply matching numbers, then add the results.

Add the bias 0.1, and we get 1.2. The name sounds mathematical, but the operation itself is only multiply, multiply, add.

Check the shapes

Shapes help detect mistakes.

x   has shape [1 × 2]
W₁  has shape [2 × 2]
b₁  has shape [1 × 2]
z   has shape [1 × 2]

Matrix rule:

The inner dimensions match, and the outer dimensions describe the result.

The network now makes one small nonlinear decision

We will use the ReLU activation function:

Its rule is simple:

• if the value is positive, keep it;
• if it is negative, replace it with zero.

Examples:

ReLU(3.2)  = 3.2
ReLU(0.5)  = 0.5
ReLU(0)    = 0
ReLU(-1.7) = 0

Our values are positive:

The activated hidden vector is:

Why do we need activation functions?

Without nonlinear activations, stacking linear layers does not provide the full expressive benefit we expect from a deep neural network.

Suppose:

and:

Substitute the first equation:

Because matrix multiplication can be grouped:

The two linear transformations can collapse into one combined linear transformation.

A nonlinear activation between them prevents this simple collapse and allows the network to model more complex relationships.

What if a hidden value were negative?

Suppose the second pre-activation had been:

After ReLU:

That neuron would contribute zero to the next layer for this input.

This does not permanently delete the neuron. Another input may produce a positive value and activate it.

From hidden values to possible words

The output layer has three neurons:

• one score for blue;
• one score for green;
• one score for runs.

The activated hidden vector is:

Output weights and biases are:

Each output neuron calculates:

Giving blue a score

Calculate:

Add the bias:

Giving green a score

Giving runs a score

Three raw scores—but not probabilities yet

Mapped to tokens:

blue   0.84
green  0.67
runs  -0.23

The highest logit belongs to blue.

The model currently prefers the correct candidate.

Why is a negative logit allowed?

Logits are raw scores, not probabilities.

They may be:

• negative;
• zero;
• positive;
• larger than 1.

Only their relative values matter before softmax.

The logit -0.23 does not mean negative probability. Softmax will convert it into a positive probability.

Output layer as one matrix operation

Arrange candidate weights as columns:

Biases:

Then:

Shape check:

h   [1 × 2]
W₂  [2 × 3]
b₂  [1 × 3]
o   [1 × 3]

The three output values are calculated together.

Softmax turns the scores into probabilities

We want probabilities that:

• are all positive;
• are between 0 and 1;
• sum to 1;
• preserve the ranking of logits.

Softmax does this:

Our logits are:

First, make every score positive

Exponentiation makes every result positive.

Next, find the total

This total becomes the denominator.

Finally, divide each value by the total

Probability of blue

Probability of green

Probability of runs

Final distribution:

The probabilities sum to 100%.

The model can finally choose a word

If we use greedy selection, choose the highest-probability token:

The model predicts:

The sky is blue.

During text generation, a model may sample rather than always choose the maximum. During ordinary supervised training, we usually calculate loss directly from the logits and known target instead of sampling a token first.

During training, the prediction is not the end

The target is:

blue

The model assigned it probability:

The prediction is correct under argmax, but the model is not very confident. green still has 38.58% probability.

Training should reward blue and reduce competing probability.

Turning “how wrong?” into one number

For one correct target:

Therefore:

The forward pass ends with:

prediction = blue
probability of correct token = 45.73%
loss ≈ 0.782

No weight has changed yet.

Backpropagation would begin from this loss.

Why calculate loss when the prediction was correct?

Because training cares about the full probability distribution, not only which token ranked first.

Compare two models:

Both choose blue, but Model B assigns much more probability to the correct target.

Cross-entropy gives Model B a lower loss.

The complete forward pass in one table

And as one expression:

The complete journey as a debug trace

INPUT
x = [1.0, 2.0]

HIDDEN LINEAR LAYER
z₁ = (1.0 × 0.5) + (2.0 × 0.3) + 0.1 = 1.2
z₂ = (1.0 × -0.2) + (2.0 × 0.8) - 0.1 = 1.3
z  = [1.2, 1.3]

RELU
h = [max(0, 1.2), max(0, 1.3)]
h = [1.2, 1.3]

OUTPUT LINEAR LAYER
blue  = (1.2 × 0.4)  + (1.3 × 0.2) + 0.1 =  0.84
green = (1.2 × 0.1)  + (1.3 × 0.5) - 0.1 =  0.67
runs  = (1.2 × -0.3) + (1.3 × 0.1)       = -0.23

SOFTMAX
P(blue)  = 45.73%
P(green) = 38.58%
P(runs)  = 15.69%

PREDICTION
blue

CROSS-ENTROPY LOSS
-ln(0.4573) ≈ 0.782

This is forward propagation with every intermediate value visible.

The same journey in Python

import math


x = [1.0, 2.0]

# Hidden-layer parameters
w_hidden_1 = [0.5, 0.3]
w_hidden_2 = [-0.2, 0.8]
b_hidden = [0.1, -0.1]


def dot(a, b):
    return sum(left * right for left, right in zip(a, b))


def relu(value):
    return max(0.0, value)


def softmax(values):
    # Subtracting max improves numerical stability.
    maximum = max(values)
    exponentials = [math.exp(value - maximum) for value in values]
    total = sum(exponentials)
    return [value / total for value in exponentials]


# 1. Hidden linear layer
z_hidden = [
    dot(x, w_hidden_1) + b_hidden[0],
    dot(x, w_hidden_2) + b_hidden[1],
]

# 2. Hidden activation
h = [relu(value) for value in z_hidden]

# 3. Output-layer parameters
output_weights = {
    "blue": [0.4, 0.2],
    "green": [0.1, 0.5],
    "runs": [-0.3, 0.1],
}

output_biases = {
    "blue": 0.1,
    "green": -0.1,
    "runs": 0.0,
}

# 4. Output logits
tokens = list(output_weights)
logits = [
    dot(h, output_weights[token]) + output_biases[token]
    for token in tokens
]

# 5. Probabilities
probabilities = softmax(logits)

# 6. Prediction
best_index = max(range(len(tokens)), key=lambda index: probabilities[index])
prediction = tokens[best_index]

# 7. Training loss for the known target
target = "blue"
target_index = tokens.index(target)
loss = -math.log(probabilities[target_index])

print("hidden pre-activation:", z_hidden)
print("hidden activation:", h)
print("logits:", logits)
print("probabilities:", probabilities)
print("prediction:", prediction)
print("loss:", loss)

Expected values are approximately:

hidden pre-activation: [1.2, 1.3]
hidden activation:     [1.2, 1.3]
logits:                [0.84, 0.67, -0.23]
probabilities:         [0.4573, 0.3858, 0.1569]
prediction:            blue
loss:                  0.782

Our network is tiny. What changes inside a real LLM?

Our tiny network is deliberately simple. A Transformer LLM has a much longer forward path.

flowchart TD
    Text["Text"] --> Tokens["Tokens and IDs"]
    Tokens --> Emb["Token embeddings"]
    Emb --> Position["Position information"]
    Position --> Blocks["Many Transformer blocks"]
    Blocks --> Norm["Final normalization"]
    Norm --> Projection["Vocabulary projection"]
    Projection --> Logits["Logits for every token"]

Token embeddings

Every token ID retrieves a learned vector.

Positional information

The model needs to distinguish token order.

Attention

Query and Key dot products create attention scores. Softmax converts them into weights. Weighted Value vectors move relevant context among positions.

Feed-forward networks

Each Transformer block includes learned transformations applied at token positions. Modern architectures may use activations such as GELU or SiLU-family gated variants rather than the simple ReLU in our example.

Residual connections and normalization

These help preserve information and keep deep computation stable.

Vocabulary projection

The final hidden state is transformed into one logit per vocabulary token.

Softmax or cross-entropy

During inference, logits guide token selection. During training, cross-entropy compares logits with known target tokens.

The architecture is larger, but forward propagation still means:

Apply every operation in order using the current input and current parameters.

A real language model processes a sequence

A language model does not normally compress the entire prompt into our manually supplied [1,2] vector.

Suppose the tokens are:

[the, sky, is]

Each position has a vector:

the → [many numbers]
sky → [many numbers]
is  → [many numbers]

The sequence is represented as a matrix:

Attention and feed-forward layers transform the complete matrix.

During causal language-model training, outputs can predict several targets in parallel:

the       → sky
the sky   → is
the sky is→ blue

The causal mask prevents each position from reading future targets.

During generation, the model uses the final position’s logits to choose the next token, appends it and performs the next decode step.

Training adds one more dimension: the batch

Training normally processes multiple sequences together.

If:

batch size       = 32
sequence length  = 128
hidden dimension = 768

the hidden-state tensor may have shape:

[32, 128, 768]

Meaning:

• 32 sequences;
• 128 token positions per sequence;
• 768 values representing each position.

Layers operate over these tensors using highly optimized matrix operations.

GPUs are effective because many multiply-and-add operations can run in parallel.

Forward propagation is conceptually simple, but its scale creates enormous computation.

Common places where the forward pass goes wrong

Mistake 1: incompatible matrix shapes

If input width does not match weight-matrix height, multiplication is undefined.

Always write the shapes.

Mistake 2: forgetting the bias

A linear layer is commonly an affine transformation:

not only $xW$.

Mistake 3: applying activation in the wrong place

The output layer for classification or language modelling usually produces logits. Do not automatically apply ReLU to vocabulary logits.

Mistake 4: applying softmax twice

Common cross-entropy implementations expect raw logits and internally perform the stable normalization required for the loss.

Mistake 5: confusing logits with probabilities

Logits can be negative and need not sum to 1. Probabilities are normalized.

Mistake 6: thinking the forward pass learns

The forward pass calculates a prediction. Weights change only after gradients are calculated and an optimizer performs an update.

Mistake 7: believing one neuron has one guaranteed meaning

Learned representations are distributed. We should be cautious about assigning a clean human concept to an individual neuron.

Mistake 8: ignoring numerical stability

Direct exponentiation of very large logits can overflow. Stable softmax implementations subtract the maximum logit before exponentiation without changing the final probabilities.

How engineers inspect a real forward pass

When a network produces unexpected output, inspect intermediate values.

Useful questions include:

• What is the input shape?
• Are values finite?
• Did activations become all zero?
• Are logits extremely large?
• Does softmax contain NaN?
• Was the correct model checkpoint loaded?
• Is the model in training or evaluation mode?
• Are tensors on the correct device?
• Are input and weights using expected data types?

A debug table might look like:

Debugging forward propagation means observing how representations change, not merely checking the final prediction.

The one thing to remember

Forward propagation is not one mysterious AI operation.

It is an ordered sequence of ordinary mathematical transformations.

Our example began with:

The hidden layer calculated:

ReLU produced:

The output layer produced:

Softmax converted those logits into:

The model selected:

During training, the correct-token probability gave us loss:

No weight changed during these steps.

The forward pass answered only:

With the model exactly as it is now, what output does this input produce?

Backpropagation can now travel through the recorded calculations in reverse, and the optimizer can update the weights. That is where our next article should begin: with this exact loss, this exact network and one question—which weight was responsible for how much of the error?

But before a model can learn from a mistake, it must first make the prediction.

That prediction-making journey is forward propagation.

The next article begins with our loss of 0.782 and asks: which weight was responsible for how much of that error? That is the problem solved by backpropagation.

Sources and further reading

• PyTorch: Linear layer
• PyTorch: ReLU
• PyTorch: Softmax
• PyTorch: CrossEntropyLoss
• Vaswani et al.: Attention Is All You Need

Continue the series

1. Forward Pass: Before a Neural Network Can Learn, It Must Make a Prediction
2. Backpropagation: The Model Was Wrong. Which Weight Should We Blame?
3. Gradient Descent: The Gradient Knows the Direction. But How Far Should the Model Move?

x = [1, 2]

h₁ before ReLU = 1 × 0.5 + 2 × 0.3 + 0.1 = 1.2
h₂ before ReLU = 1 × (−0.2) + 2 × 0.8 − 0.1 = 1.3

h = [max(0, 1.2), max(0, 1.3)] = [1.2, 1.3]

blue:  1.2 × 0.4 + 1.3 × 0.2 + 0.1 = 0.84
green: 1.2 × 0.1 + 1.3 × 0.5 − 0.1 = 0.67
cat:   1.2 × (−0.3) + 1.3 × 0.1 = −0.23

pᵢ = exp(scoreᵢ) / Σ exp(scores)
blue ≈ 45.73%   green ≈ 38.58%   cat ≈ 15.69%

target = blue
loss = −ln(0.4573156788) ≈ 0.782381