Lesson 5: Simple neural networks (Perceptron & MLP)

Now that Lesson 4 has shown how PyTorch manages tensors and computes derivatives automatically, we are ready to assemble our first real deep-learning model: a Multi-Layer Perceptron (MLP).

This lesson covers the mathematics of a single artificial neuron, how those neurons stack into a multi-layer network, why non-linear activation functions are not optional, and how to initialise weights so the network can actually start learning.

5.1 The artificial neuron: from linear to non-linear

An artificial neuron takes a vector of inputs $X = [x_1, x_2, ..., x_n]$, multiplies them by matching weights $W = [w_1, w_2, ..., w_n]$, adds a bias $b$, and pushes the total through a non-linear activation function $f$:

If that formula looks familiar, it should: drop $f$ and it is exactly linear regression. A single neuron like this has its own name — the Perceptron, proposed by Frank Rosenblatt in 1958 — and for years it was expected to be the foundation of artificial intelligence.

Why do we need the non-linear activation $f$ at all?

Real datasets have complicated decision boundaries — the classic XOR problem, or two concentric circles. Without a non-linear $f$, no matter how many linear layers you stack, the whole thing is mathematically equivalent to a single linear multiplication:

Without a non-linearity, a neural network loses all ability to approximate non-linear functions.

Here are the three activation functions you will meet most often. Note the red bands at both ends of Sigmoid and Tanh — those are the saturation regions, where the curve goes flat, and a flat curve means a derivative of zero:

The trade-off is visible in the picture. Sigmoid and Tanh are smooth in the middle but dead flat at both ends — and flat means zero gradient, which means no more learning. ReLU is just two straight segments, far cruder, but its right half has a slope of exactly 1 everywhere: a gradient can pass through any number of layers without shrinking. The price is the left half, completely flat — the source of the "Dying ReLU" problem below.

In detail:

• Sigmoid: $f(x) = \frac{1}{1 + e^{-x}}$. Squashes every output into $(0, 1)$, which suits binary classification. Its big weakness is vanishing gradients: when the input is very large or very small, the derivative approaches zero.
• Tanh (hyperbolic tangent): $f(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}$. Squashes into $(-1, 1)$. It usually behaves better than Sigmoid in hidden layers because its output is centred on zero, which speeds up optimisation in the following layer.
• ReLU (Rectified Linear Unit): $f(x) = \max(0, x)$. Extremely simple — zero for negatives, unchanged for positives — and the default for every hidden layer in modern networks. It avoids vanishing gradients because the derivative is always 1 for positive inputs.
  Weakness: Dying ReLU. If a neuron's input stays negative, it always outputs 0 and its gradient is permanently 0, so the neuron is effectively "dead". The usual alternative is Leaky ReLU: $f(x) = \max(\alpha x, x)$ with a small slope $\alpha \approx 0.01$.

Some concrete numbers show how badly Sigmoid "vanishes" gradients. Its derivative is $f'(x) = f(x)(1 - f(x))$, which peaks at exactly $0.25$ at $x=0$ and shrinks fast as $|x|$ grows:

import math

def sigmoid(x):
    return 1 / (1 + math.exp(-x))

def sigmoid_derivative(x):
    s = sigmoid(x)
    return s * (1 - s)

for x in [0, 2, 5, 10]:
    print(f"x={x:>3} | f'(x) = {sigmoid_derivative(x):.8f}")

# x=  0 | f'(x) = 0.25000000   best case: the gradient passes through at 1/4 strength
# x=  2 | f'(x) = 0.10499359
# x=  5 | f'(x) = 0.00664806
# x= 10 | f'(x) = 0.00004540   effectively zero

# What 10 stacked sigmoid layers do to a gradient:
print(f"best case,  10 layers: {0.25 ** 10:.3e}")   # 9.537e-07
print(f"x=2 inputs, 10 layers: {0.1049 ** 10:.3e}")  # 1.613e-10

The last two lines are the alarming part. Note that 0.25 is the largest value the Sigmoid derivative can ever reach — it happens only at exactly $x = 0$. So even in the best possible case, every Sigmoid layer cuts the gradient to a quarter. Across 10 layers, the gradient reaching the first layer is $0.25^{10} \approx 9.5 \times 10^{-7}$ — roughly a million times smaller than it started. And that is the optimistic scenario; shift the inputs slightly away from zero, say to $x = 2$, and the number falls to $1.6 \times 10^{-10}$.

With a learning rate around $10^{-3}$, a gradient that small changes the first layer's weights in the tenth decimal place — which is to say not at all within 32-bit float precision. The first layer sits frozen while the last layers train normally. This is the vanishing gradient problem, and it is why ReLU — derivative exactly 1 for positive inputs, no shrinkage with depth — became the default for every hidden layer in modern deep networks.

5.2 Assembling an MLP with PyTorch

A Multi-Layer Perceptron (MLP) stacks fully connected (linear) layers, where every neuron in one layer connects to every neuron in the next. The forward computation at each layer is a matrix multiplication: $$Z^{[l]} = W^{[l]} A^{[l-1]} + b^{[l]}$$ $$A^{[l]} = f(Z^{[l]})$$

In Lesson 4 we declared every weight by hand with torch.randn((), requires_grad=True) — fine for four parameters. The network below has 105, and a real one has millions. This is where torch.nn comes in.

torch.nn (short for neural networks) is PyTorch's package of ready-made building blocks. Three of them matter right now:

• nn.Linear(in, out) — a fully connected layer. Writing nn.Linear(2, 8) means "take 2 numbers in, produce 8 out", and PyTorch creates for you an $8 \times 2$ weight matrix and an 8-element bias vector, already flagged with requires_grad=True. You declare no weights at all.
• nn.ReLU(), nn.Sigmoid() — the activations from section 5.1, wrapped as layers so they are easy to stack. They hold no learnable parameters.
• nn.Sequential(...) — a container that runs layers back to back, each one's output feeding the next. If you know Express, it is exactly a middleware chain.

nn.Module is the base class every PyTorch model inherits from. Why a class rather than a function? Because a model is not only a computation — it also carries state: all of its weights. Inheriting from nn.Module gives you that for free: model.parameters() lists every learnable weight (Lesson 6 hands that list straight to an optimizer), model.to(device) moves the whole network to a GPU in one line, and model.state_dict() saves and loads it.

That leaves you two jobs: declare the layers in __init__, and describe the order data flows through them in forward():

import torch
import torch.nn as nn

class SimpleMLP(nn.Module):
    def __init__(self, input_dim=2, hidden_dim=8, output_dim=1):
        # Must come first — it sets up the bookkeeping nn.Module needs.
        super(SimpleMLP, self).__init__()

        # Layers run top to bottom: output of one feeds into the next.
        self.network = nn.Sequential(
            nn.Linear(input_dim, hidden_dim),   # 2 -> 8
            nn.ReLU(),                          # non-linearity, no parameters
            nn.Linear(hidden_dim, hidden_dim),  # 8 -> 8
            nn.ReLU(),
            nn.Linear(hidden_dim, output_dim),  # 8 -> 1
            nn.Sigmoid()                        # squash to [0, 1] for a probability
        )

    def forward(self, x):
        # You never call forward() yourself — write model(x) and PyTorch calls it.
        return self.network(x)

model = SimpleMLP()
print(sum(p.numel() for p in model.parameters()))  # 105

That 105 breaks down like this: layer 1 has $8 \times 2 = 16$ weights and 8 biases; layer 2 has $8 \times 8 = 64$ weights and 8 biases; the output layer has $1 \times 8 = 8$ weights and 1 bias. Total $16 + 8 + 64 + 8 + 8 + 1 = 105$ numbers, and every one of them is created by PyTorch, already tracking gradients, ready for the training loop in Lesson 6. Compare that with Lesson 4, where you wrote requires_grad=True by hand for exactly four parameters.

One detail that trips people up: you define forward() but never call model.forward(x) — you write model(x). nn.Module implements __call__ to invoke forward() for you, along with internal hooks. Calling forward() directly still returns the right answer but skips those hooks, so the convention across the whole PyTorch ecosystem is model(x).

Here is the shape of the network we just declared, with the data dimensions at each stage:

5.3 Weight initialisation

The network has a shape now, but its 105 numbers have to start somewhere. "What value should the weights begin at" sounds like a trivial detail — in fact getting it wrong means the network never learns, and this section proves that with code rather than assertion.

If we initialise every weight to zero (or to any single constant), then during the forward pass every neuron in a hidden layer produces an identical output. During backpropagation their gradients are therefore identical too, and they receive identical updates. The multi-layer network degenerates into a single neuron — it has failed to achieve symmetry breaking.

Let us PROVE that with code, rather than take the theory on trust:

import torch
import torch.nn as nn

torch.manual_seed(0)
layer = nn.Linear(2, 4)  # 4 neurons in one hidden layer

# Initialise every weight to the SAME constant.
nn.init.constant_(layer.weight, 0.5)
nn.init.constant_(layer.bias, 0.0)

x = torch.tensor([[1.0, 2.0]])
output = layer(x)
print("4 neurons, constant init:", output)
# tensor([[1.5, 1.5, 1.5, 1.5]]) — ALL FOUR NEURONS PRODUCE THE SAME VALUE.
# Same input, same weights -> same output -> same gradient on the backward pass
# -> identical updates forever -> 4 neurons behaving as 1.

# Random init (Kaiming here) breaks that symmetry.
layer2 = nn.Linear(2, 4)
nn.init.kaiming_normal_(layer2.weight, nonlinearity="relu")
output2 = layer2(x)
print("4 neurons, Kaiming init:", output2)
# tensor([[1.6014, -0.2386, 0.0344, -3.3811]]) — four different values, so each
# neuron can now learn a different feature.

The printed output says it all: the first line is tensor([[1.5, 1.5, 1.5, 1.5]]) — four neurons producing four identical values. Because they match on the forward pass, their gradients match on the backward pass, so their updates match, and they will go on matching forever. A 4-neuron layer initialised this way really is one neuron copied four times — you pay for 4 and receive the power of 1.

The second line, with Kaiming initialisation, gives [1.6014, -0.2386, 0.0344, -3.3811]: four genuinely different values. From the very first training step each neuron heads in its own direction and can learn its own feature. That is all "symmetry breaking" means.

But "random" alone is not enough — random at what scale is the real question. Initial weights that are too large make the signal grow layer by layer (exploding gradients); too small and the signal shrinks away to nothing (the same vanishing gradient as in section 5.1, this time caused by initialisation rather than by the activation). The two formulas below exist to pick that scale:

• Xavier (Glorot) initialisation: weights drawn from a distribution whose variance is inversely proportional to the number of input and output neurons: $$\text{Var}(W) = \frac{2}{n_{\text{in}} + n_{\text{out}}}$$ This works very well alongside linear or S-shaped activations such as Tanh and Sigmoid, keeping the variance of both the data and the gradients stable as they pass through layers.
• He (Kaiming) initialisation: variance tuned specifically for ReLU: $$\text{Var}(W) = \frac{2}{n_{\text{in}}}$$ Why a factor of 2 rather than 1? Because ReLU zeroes out the entire negative half, halving the variance of whatever passes through. Multiplying by 2 compensates for exactly that loss.

5.4 Hands-on project: classifying non-linear data (two concentric circles)

Now we combine all three sections into a program that runs. The project generates a two concentric circles dataset — outer ring labelled $0$, inner ring labelled $1$ — builds a three-layer MLP in PyTorch, initialises it with Kaiming, then runs one forward pass to see what the network predicts before any training.

Why two concentric circles? Because this is precisely the kind of data the single Perceptron of section 5.1 cannot handle: no straight line separates the inner ring from the outer one. If an MLP can do it, that is direct evidence for everything this lesson has argued. In this lesson we only build the network — training it to actually classify correctly is Lesson 6.

Here is the full Python source for the project:

import torch
import torch.nn as nn
import numpy as np


# Two concentric circles: the classic dataset that a single straight line cannot
# separate. Written by hand rather than imported from scikit-learn so the file
# runs with nothing but torch and numpy installed.
def generate_concentric_circles(n_samples=1000, noise=0.05, factor=0.5):
    n_samples_out = n_samples // 2
    n_samples_in = n_samples - n_samples_out

    # Outer ring, radius 1.0, label 0.
    theta_out = np.linspace(0, 2 * np.pi, n_samples_out)
    x_out = np.cos(theta_out) + np.random.normal(0, noise, n_samples_out)
    y_out = np.sin(theta_out) + np.random.normal(0, noise, n_samples_out)
    X_out = np.vstack((x_out, y_out)).T
    y_out_label = np.zeros(n_samples_out)

    # Inner ring, radius `factor`, label 1.
    theta_in = np.linspace(0, 2 * np.pi, n_samples_in)
    x_in = factor * np.cos(theta_in) + np.random.normal(0, noise, n_samples_in)
    y_in = factor * np.sin(theta_in) + np.random.normal(0, noise, n_samples_in)
    X_in = np.vstack((x_in, y_in)).T
    y_in_label = np.ones(n_samples_in)

    X = np.vstack((X_out, X_in))
    y = np.concatenate((y_out_label, y_in_label))

    # Shuffle, so the two classes are not handed to the model in blocks.
    indices = np.arange(n_samples)
    np.random.shuffle(indices)
    return X[indices], y[indices]


class SimpleMLP(nn.Module):
    def __init__(self, input_dim=2, hidden_dim=8, output_dim=1):
        # super().__init__() must come first: it sets up the bookkeeping that
        # lets PyTorch find and track every layer you assign below.
        super(SimpleMLP, self).__init__()

        # Three linear layers with a non-linearity between each pair. Remove the
        # ReLUs and the whole stack collapses into a single linear layer.
        self.network = nn.Sequential(
            nn.Linear(input_dim, hidden_dim),  # hidden layer 1
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),  # hidden layer 2
            nn.ReLU(),
            nn.Linear(hidden_dim, output_dim),  # output layer
            nn.Sigmoid(),  # squash to [0, 1] so the output reads as a probability
        )

        self._initialize_weights()

    def _initialize_weights(self):
        for m in self.modules():
            if isinstance(m, nn.Linear):
                # Kaiming is the right choice here because the hidden layers use
                # ReLU. Xavier would under-scale the variance for that activation.
                nn.init.kaiming_normal_(m.weight, nonlinearity="relu")
                if m.bias is not None:
                    # Bias can safely start at 0 — the random weights already
                    # break the symmetry between neurons.
                    nn.init.constant_(m.bias, 0.0)

    def forward(self, x):
        return self.network(x)


if __name__ == "__main__":
    print("=== Generating the two-circles dataset ===")
    X_np, y_np = generate_concentric_circles(n_samples=10, noise=0.05, factor=0.5)
    print(f"Shape of X: {X_np.shape} (10 samples, each an (x, y) coordinate)")
    print(f"Labels y: {y_np} (0 = outer ring, 1 = inner ring)\n")

    X_tensor = torch.tensor(X_np, dtype=torch.float32)

    print("=== Building the MLP ===")
    model = SimpleMLP(input_dim=2, hidden_dim=8, output_dim=1)
    print(model)
    n_params = sum(p.numel() for p in model.parameters())
    print(f"Trainable parameters: {n_params}")

    print("\n=== Forward pass on the untrained network ===")
    # eval() switches off training-only layers. This model has none, but making
    # it a habit costs nothing and prevents a whole class of bug later.
    model.eval()
    with torch.no_grad():
        predictions = model(X_tensor)

    print("Raw probabilities from the untrained model:")
    for i in range(len(X_np)):
        print(
            f"Point: [{X_np[i][0]:6.3f}, {X_np[i][1]:6.3f}] | "
            f"P(class 1) = {predictions[i].item():.4f} | true label: {int(y_np[i])}"
        )

    print("\nThe network has not been trained, so these probabilities carry no")
    print("information yet — they hover around 0.5 regardless of the true label.")

Running it and reading the output

Run python classify_circles.py and the tail of the output looks like this:

=== Building the MLP ===
SimpleMLP(
  (network): Sequential(
    (0): Linear(in_features=2, out_features=8, bias=True)
    (1): ReLU()
    (2): Linear(in_features=8, out_features=8, bias=True)
    (3): ReLU()
    (4): Linear(in_features=8, out_features=1, bias=True)
    (5): Sigmoid()
  )
)
Trainable parameters: 105

=== Forward pass on the untrained network ===
Raw probabilities from the untrained model:
Point: [-0.008, -1.007] | P(class 1) = 0.5014 | true label: 0
Point: [ 0.488,  0.098] | P(class 1) = 0.5190 | true label: 1
Point: [-0.075,  0.432] | P(class 1) = 0.5355 | true label: 1
Point: [ 0.543, -0.078] | P(class 1) = 0.5087 | true label: 1
Point: [ 0.980, -0.013] | P(class 1) = 0.5239 | true label: 0
Point: [-1.033,  0.026] | P(class 1) = 0.4318 | true label: 0
Point: [ 0.027,  1.010] | P(class 1) = 0.5977 | true label: 0
Point: [ 0.074, -0.512] | P(class 1) = 0.5050 | true label: 1
Point: [-0.482, -0.061] | P(class 1) = 0.4659 | true label: 1
Point: [ 0.919, -0.052] | P(class 1) = 0.5199 | true label: 0

Two things are worth reading here. First, print(model) prints back exactly the structure you declared, with the real dimensions of each layer: 2 → 8, 8 → 8, 8 → 1. This is the fastest way to check the network has the shape you think it has — and with more complex models it usually catches a dimension mistake before you can even complete one training round.

Second, every predicted probability sits close to 0.5 (lowest 0.4318, highest 0.5977), and none of them correlate with the true label: points labelled 0 and points labelled 1 produce the same sort of numbers. That is exactly what should happen. The weights are random Kaiming values that have never seen the data, so the model knows nothing beyond a coin flip. If you ever see an untrained network predicting correctly, you almost certainly have data leaking in somewhere.

Put differently: this lesson has built the skeleton and confirmed that it runs, but has taught it nothing. Turning those numbers around 0.5 into correct predictions needs two more pieces, both of which Lesson 6 supplies — a function that measures the error, and a loop that updates the weights along the gradient.

Summary and what comes next

Bridge to the next lesson: the MLP is designed but cannot learn yet. In Lesson 6 we write the training loop, using backpropagation to update the network's weights.