Node

The Neural Network article described the whole structure at a glance — layers, transformation, increasing abstraction. Now it’s time to zoom all the way in, to the single smallest piece that structure is actually made of: the node.

The simple definition

A node (also called a neuron or a unit) is a single computational element that takes in several numbers, combines them using weights and a bias, and produces one output number. A neural network with millions or billions of parameters is really just an enormous number of these individually simple nodes, connected together in layers. Understanding one node fully is understanding the entire mechanism the whole network is built from — everything else is repetition and scale.

What a single node actually calculates, step by step

This is worth walking through with real numbers, since “combines inputs using weights” can sound abstract until you see it happen. Say a node receives three input numbers: 2, 5, and 1 — perhaps representing normalized versions of square footage, bedrooms, and age from the house-price example used throughout this glossary. Each incoming connection has its own weight, learned through training, as covered in the Weights article — say w1 = 0.4, w2 = 0.9, and w3 = -0.2. The node’s first step is a weighted sum:

weighted_sum = (2 × 0.4) + (5 × 0.9) + (1 × -0.2)
             = 0.8 + 4.5 - 0.2
             = 5.1

Next, the node adds its own bias — say b = 0.3, exactly the kind of fixed offset described in the Bias article:

5.1 + 0.3 = 5.4

Finally, that number passes through an activation function — a topic getting its own full article shortly in this phase, but worth naming here since it’s the last actual step. For now, know that this step decides whether, and how strongly, the node’s calculated value gets passed forward — some activation functions might pass 5.4 through mostly unchanged, others might reshape it. Whatever comes out of that last step is the node’s final output — a single number, sent forward to every node in the next layer.

flowchart LR
    A[Input 1: 2] -->|× 0.4| D[Weighted Sum]
    B[Input 2: 5] -->|× 0.9| D
    C[Input 3: 1] -->|× -0.2| D
    D --> E[+ Bias: 0.3]
    E --> F[Activation Function]
    F --> G[Node's Output]

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of a single judge on a talent show panel, scoring a performance based on several specific factors — vocal ability, stage presence, song choice — each weighted differently according to that judge’s personal priorities (maybe vocal ability counts double for this particular judge). The judge combines all these weighted impressions into one final score, which then gets passed along to be combined with the other judges’ scores.

Where this breaks down: A judge applies genuine subjective taste and experience to each factor. A node’s “weighing” is pure, fixed arithmetic — the exact same weighted-sum-plus-bias-plus-activation calculation, applied identically every single time, with the specific weight values having been discovered through training rather than personal judgment. And where a judge considers a handful of factors at most, a single node in a real neural network layer might combine hundreds or thousands of incoming numbers at once.

Why one node alone can’t do much, and why that’s fine

A single node, on its own, is genuinely limited — it can really only detect one specific kind of weighted combination of its inputs. The real power described in the Neural Network article comes from having many nodes side by side in a layer, each with its own different weights, each detecting a different combination of the same inputs — and then stacking layers of these nodes on top of each other, letting later nodes combine the outputs of earlier ones into increasingly complex, abstract patterns. No individual node needs to be smart; the network’s capability emerges from how many simple nodes work together, echoing the Weights article’s point that meaning in a large network is distributed across many components rather than concentrated in any single one.

Node, edge, weight, bias, parameter, feature, and label

These words describe different parts of the same learning system. They are related, but they are not interchangeable.

Term	What it is	Does training learn it?	Role in one node calculation
Node	A computational unit that produces an output value.	The node’s existence is designed; its parameters are learned.	Combines incoming values, weights, bias, and activation.
Edge	A connection carrying a number from one node or input to another node.	The connection pattern is usually designed.	Carries an incoming value and is associated with a weight.
Weight	A number controlling the strength and direction of one incoming connection.	Yes.	Multiplies the value arriving along an edge.
Bias	One adjustable offset belonging to a node.	Yes.	Shifts the node’s weighted sum before activation.
Parameter	Any model number updated during training.	Yes.	Weights and biases are common parameter types.
Feature	An input measurement or useful representation.	Raw features are supplied; hidden features are learned.	Provides a value that a node can combine.
Label	The correct target answer for a training example.	No. It comes from the training data.	Used after the forward pass to calculate error; it is not stored inside the node.

The shortest relationship is:

Node = computation
Edge = connection
Weight = learned strength on a connection
Bias = learned offset inside a node
Parameter = category containing learned weights and biases
Feature = information entering the network or learned inside it
Label = correct answer used to measure training error

What an edge means in a network diagram

flowchart LR
    A[Input node<br/>x = 0.8] -->|edge carries 0.8<br/>weight = 0.6| B[Hidden node]
    C[Input node<br/>x = 0.3] -->|edge carries 0.3<br/>weight = -0.5| B
    D[Bias<br/>-0.1] --> B
    B --> E[Activation output<br/>0.23]

An arrow does not mean that a complete human idea travels through the network. It carries a number. For an ordinary dense layer, every incoming edge to a node has its own weight.

value carried by edge 1 = 0.8
weighted contribution 1 = 0.8 × 0.6  = 0.48

value carried by edge 2 = 0.3
weighted contribution 2 = 0.3 × -0.5 = -0.15

node pre-activation = 0.48 - 0.15 - 0.10 = 0.23

The edge and weight are related but different: the edge is the connection; the weight is the learned number controlling that connection.

Where the label enters—and where it does not

flowchart LR
    A[Features] --> B[Network nodes and parameters]
    B --> C[Prediction]
    D[Label: correct answer] --> E[Loss function]
    C --> E
    E --> F[Backpropagation updates weights and biases]

During inference, the label is normally unavailable because the model is predicting an answer for a new case. Nodes still run using their already-trained weights and biases.

A node from the loan network

One hidden node receives [0.8, 0.3, 0.9], uses weights [0.6, -0.5, 0.4], and has bias -0.1.

Step 1: multiply each input by its weight
0.8 ×  0.6 =  0.48
0.3 × -0.5 = -0.15
0.9 ×  0.4 =  0.36

Step 2: add the products and bias
z = 0.48 - 0.15 + 0.36 - 0.10
z = 0.59

Step 3: apply ReLU
output = max(0, 0.59) = 0.59

flowchart LR
    A[Inputs] --> B[Multiply by weights]
    B --> C[Add products]
    C --> D[Add bias]
    D --> E[Apply activation]
    E --> F[Node output 0.59]

The value 0.59 now becomes an input to nodes in the next layer. During training, the three weights and bias can change. The input values belong to this applicant; the learned weights and bias belong to the model.

A concrete example, layered

Simple example: should you carry an umbrella?

In a tiny umbrella-prediction network, one node might combine:

• Cloud cover, using a positive weight.
• Humidity, also using a positive weight.

When both inputs are high, the node produces a high output. It effectively becomes a small, learned rain-risk detector, discovered automatically through training rather than programmed by hand.

Production example: a node-like calculation inside GPT-3

Inside one of GPT-3’s 96 layers, hidden units perform the same basic weighted-sum, bias, and activation calculation while receiving inputs from—and sending outputs to—thousands of other units.

A single calculation is simple. Multiplied across a network containing 175 billion learned parameters, the combined transformations help GPT-3 produce fluent, contextually appropriate text.

Focused infographic: one node’s complete job

flowchart LR
    A[Receive numbers] --> B[Weight each one]
    B --> C[Add them]
    C --> D[Add bias]
    D --> E[Apply activation]
    E --> F[Send one output number]

One node receives several numbers but emits one number. A layer contains many such computations, so it emits a vector of numbers.

Real-model connection: what is a “node” inside GPT?

In a traditional diagram, a circle is conveniently called a node or neuron. In a Transformer such as GPT, engineers more often talk about hidden dimensions, attention heads, and feed-forward units. Each scalar unit in a feed-forward part still performs learned weighted computations, but drawing billions of individual circles would hide the architecture rather than explain it.

The GPT-2 technical report lists a hidden representation size (dmodel) of 1,600 for its largest 1.542-billion-parameter model. This means each token is represented by a vector containing 1,600 numbers as it passes between Transformer blocks. It does not mean the whole model contains only 1,600 nodes.

Common misconception

A frequent assumption, especially given the “neuron” terminology: that a single node “understands” or “represents” one specific, nameable concept — like one node meaning “cat” or one node meaning “question mark.” As the Weights article explained, this is rarely how it actually works in large, real networks — a node’s specific contribution is usually a small, hard-to-interpret piece of a much larger, distributed pattern, not a clean, human-readable concept, except sometimes in very small, simple networks like the beginner umbrella example above.

Where this fits in what comes next

You now understand the smallest working piece of a neural network in full detail. The next article, Layer, covers how many nodes get grouped together to work in parallel, and how those groups stack on top of each other to build the full network described in the previous article.

In one sentence

A node is the smallest computational unit in a neural network — a weighted sum, plus a bias, passed through an activation function — and while any single node’s calculation is genuinely simple, the combined effect of millions or billions of them working together is what gives neural networks their real power.

Related Terms

• Neural Network
• Weights
• Bias (model parameter)
• Layer
• Activation Function