Activation Function

Every article in this phase has referenced this step without fully explaining it — mentioned in the Node article as the last thing that happens after the weighted sum and bias, mentioned again in the Output Layer article as what shapes a prediction into the right format. It’s time to explain it properly: the activation function.

The simple definition

An activation function is a small mathematical function applied to a node’s weighted-sum-plus-bias result, deciding how strongly — and in what form — that value gets passed forward to the next layer. Recall the exact calculation from the Node article: weighted sum, then add bias, then activation function. That last step isn’t a minor technicality — it’s arguably the single most important ingredient that makes deep neural networks capable of what they do.

Why this step exists at all: the problem with skipping it

This is worth understanding precisely, because it’s genuinely the whole reason activation functions matter. If every node simply passed its weighted-sum-plus-bias value straight through, unmodified, something mathematically important would break: stacking multiple layers of pure weighted sums, with nothing else in between, turns out to be mathematically equivalent to just one single layer — no matter how many layers you stack. Combining linear calculations (weighted sums) with more linear calculations just produces another linear calculation; you never actually gain the ability to represent more complex, curved, or intricate patterns, no matter how deep the network gets.

An activation function fixes this by introducing non-linearity — a deliberate bend or kink in the calculation that breaks this mathematical collapse. With non-linearity inserted between every layer, stacking layers genuinely does build up more and more representational power, exactly as the Hidden Layer article’s edges-to-shapes-to-parts progression described. Without activation functions, none of that progressive, increasingly abstract pattern-building would be mathematically possible at all — a network of any depth would be no more capable than the simplest single-layer model.

flowchart LR
    A[Without activation functions] --> B[Stacked layers collapse into one equivalent linear layer]
    C[With activation functions] --> D[Each layer genuinely adds new representational power]

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of a dimmer switch on a light. A weighted sum alone is like a switch that just adds up voltage in a straight, proportional line — twice the input, exactly twice the output, forever. An activation function is like installing an actual dimmer mechanism that reshapes that relationship — dimming weak signals down to nothing, letting strong signals through more sharply, or capping things off at a maximum — introducing a genuine “bend” in how input relates to output, rather than a plain straight line.

Where this breaks down: A physical dimmer switch is adjusted by a person turning a knob. An activation function’s specific shape is chosen once, by an engineer, before training even begins — it’s a fixed mathematical formula (the topic of the very next article, ReLU, being one common choice), not something adjusted dynamically for each individual signal the way a person might turn a dimmer knob differently each time.

What common activation functions actually look like

Different activation functions reshape their input differently, and it’s worth knowing a couple by name and behavior, since they’ve shaped real deep learning history:

Sigmoid — squashes any input, no matter how large or small, into a smooth curve between 0 and 1, historically popular for its clean probability-like output, but prone to a serious problem covered in the next article.
Tanh — similar to sigmoid but squashes input into a range between -1 and 1 instead, sharing many of sigmoid’s same historical strengths and weaknesses.
ReLU — the modern default for most hidden layers, covered in full detail in the very next article, which takes a strikingly simple approach: pass positive values through completely unchanged, and turn any negative value into exactly zero.
Softmax — used specifically in output layers for multi-class classification, as covered in the Output Layer article, converting a set of raw numbers into a full probability distribution that sums to 100%.

Before activation and after activation

A node first calculates a weighted sum called the pre-activation value, often written as z. The activation function transforms z into the node’s output, often written as a.

z = (inputs × weights) + bias
a = activation(z)

Using ReLU:

Pre-activation `z`	Activation `a = ReLU(z)`
-2.0	0
0.0	0
0.59	0.59
3.2	3.2

The weighted calculation decides what signal the node has assembled. The activation function decides how that signal is passed onward.

Why stacked linear layers collapse into one

Suppose one layer only multiplies an input by 2 and the next only multiplies by 3:

Layer 1: y = 2x
Layer 2: z = 3y = 3 × 2x = 6x

The two layers are equivalent to one layer that multiplies by 6. Stacking more purely linear layers still creates only another linear transformation. Nonlinear activation functions break this collapse and allow a network to form bends, boundaries, and complex combinations that one straight transformation cannot represent.

Choose the activation for its role

ReLU and its variants are common in hidden layers because they are simple and train efficiently.
Sigmoid is useful for an output representing one probability or several independent probabilities.
Softmax converts a group of output scores into probabilities that sum to 1 for mutually exclusive classes.
Linear output is common for regression when the prediction should not be restricted to 0–1.

Activation choice is an architectural decision. Using softmax in every hidden layer, or ReLU for a probability output without an appropriate conversion, can give the network the wrong behavior.

A concrete example, layered

Simple example: sigmoid changes an unbounded score

Suppose a node’s weighted sum plus bias equals 5.4.

Before activation: z = 5.4
After sigmoid:     a ≈ 0.9955

Sigmoid squashes the large positive number into a smooth value close to 1. The result is bounded between 0 and 1 and can be useful when the output needs a probability-like interpretation.

Production example: GPT-style feed-forward networks

GPT-style language models use nonlinear activations inside their Transformer feed-forward networks. GPT-2’s public implementation uses GELU, a smooth activation related in purpose—but not identical in formula—to ReLU.

The industry moved away from using sigmoid or tanh throughout very deep hidden networks because their saturated regions can produce very small gradients. ReLU-family and smoother modern activations made deep optimization more practical.

Attention output → linear expansion → GELU → linear projection

Focused infographic: where activation sits in a hidden transformation

flowchart LR
    A[Incoming representation] --> B[Linear weighted projection]
    B --> C[Activation function]
    C --> D[Nonlinear transformed representation]
    D --> E[Next operation or layer]

The activation is not a separate thinking module. It is a mathematical operation inside the layer, applied repeatedly to many values.

Real-model connection: Transformers use activations too

Transformer blocks contain feed-forward networks as well as attention. Those feed-forward parts need nonlinear activation functions; otherwise stacking their linear projections would not provide the same expressive power.

OpenAI’s public GPT-2 implementation uses GELU in its multilayer perceptron. GELU is smoother than ReLU: instead of turning every negative value sharply into zero, it gradually scales values based on their magnitude.

flowchart TB
    A[Transformer block] --> B[Attention transformation]
    A --> C[Feed-forward network]
    C --> D[Linear expansion]
    D --> E[GELU activation]
    E --> F[Linear projection back]

The exact activation varies by architecture. Understanding ReLU first is still valuable because it makes the purpose of nonlinearity easy to see before studying GELU, SwiGLU, and other modern choices.

Common misconception

A frequent beginner assumption: that activation functions are a minor implementation detail, interchangeable and not worth much thought. As the “why this step exists” section explained, this badly undersells their importance — without non-linear activation functions, a deep network’s entire layered structure would be mathematically pointless, collapsing into something no more powerful than a single layer, regardless of how many layers were stacked on top of each other. Choosing the right activation function is a genuine, consequential architectural decision, not an afterthought.

Where this fits in what comes next

You now understand why activation functions exist and what problem they solve. The final article in this phase, ReLU, covers the single most widely used activation function in modern deep learning in full detail — including exactly why it replaced sigmoid and tanh as the default choice, and the real trade-offs that come with it.

In one sentence

An activation function is the small but essential step, applied inside every node, that introduces the non-linearity making it mathematically possible for stacked layers to build up genuinely more representational power than any single layer alone — without it, depth itself would be meaningless.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026