Activation Functions

The bend after every weighted sum. Without it, a hundred layers are one straight line; with it, a layer of units can draw a curve. Compare six of them, their slopes, and what they let a real network learn.

Builds onThe Perceptron The Derivative

Every number computed in your browser

Training the tour's networks…

At a glance

function	range	steepest
Step	0 or 1	0 (∞ at the jump)
Sigmoid	(0, 1)	0.25 at z = 0
Tanh	(−1, 1)	1 at z = 0
ReLU	[0, ∞)	1 for every z > 0
Leaky ReLU	(−∞, ∞)	1 for z > 0, 0.01 below
GELU	[−0.17, ∞)	1.13 near z = 1.5

A common misconception

“More layers make a network more powerful, with or without activation functions.”

Without them, every extra layer multiplies into the same single matrix. On 200 seeded wave datasets, a network of two 16-unit layers with no activation finished within a millionth of the least-squares line's error every time (mean 0.0536). One 16-unit layer with ReLU averaged 0.0042 — a median 13.3× lower, and better on every one.

Check yourself

Question 1 of 3

A network has two layers and no activation function: h = W₁x + b₁, then y = W₂h + b₂. What can it compute?

DSAverse

Activation Functions

See also

Activation Functions

See also