Lesson 2: Linear Algebra & Derivatives from the command line

Welcome back to the Practical AI Engineer roadmap. In Lesson 1 we moved successfully into the Python environment. Before we can train complex deep learning models — Multi-Layer Perceptrons (MLP), CNNs, Transformers — we have to understand the shared language of the AI world: linear algebra and calculus.

For a web developer with no advanced maths background, the formulas taught at university tend to be dry and hard to picture. This lesson is built to change that completely: we start from the most visual, geometric meaning of each idea, then implement matrix multiplication and derivative loops in raw Python so you can see exactly how a machine "learns" from data.

2.1 The geometric meaning of vectors & matrices

Let's start by turning the real world into numbers. In computing, every kind of input — an image, an audio file, a piece of writing — is represented as an array of numbers. In mathematics we call those arrays vectors and matrices.

• Vector: a one-dimensional array of numbers. Through a programmer's lens, a 3-dimensional vector is simply a list with 3 elements: v = [x_1, x_2, x_3]. Geometrically, a vector is an arrow pointing from the origin $O(0, 0, 0)$ to the point with those coordinates in space.
• Matrix: a two-dimensional array of numbers — a table of rows and columns. A $2 \times 3$ matrix (2 rows, 3 columns), for instance, is written in Python as a list containing lists: A = [[1, 2, 3], [4, 5, 6]].

The number of elements in a vector sets the dimensionality of the space it lives in. The vector [3, 1] lives in the two-dimensional plane, so we can draw it on paper; a 768-dimensional vector — the real size of an embedding in Lesson 8 — cannot be drawn, but every operation below stays exactly the same. That is why the two-dimensional case is worth understanding properly: it is the only one you can see, and all the others differ from it only in the number.

So what is the core geometric meaning of a matrix? Don't think of it as static cells holding numbers, like a spreadsheet. In machine learning, a matrix represents a linear transformation of space. When we multiply a matrix $A$ by an input vector $x$, we are rotating, stretching or compressing that vector in space to produce a new output vector $y = Ax$:

The easiest way to see this is to watch one specific transformation act on one specific vector. Take $A = \begin{bmatrix} 2 & 0 \\ 0 & 0.5 \end{bmatrix}$ and $x = [2, 2]$ — this matrix stretches the horizontal axis by two and squashes the vertical axis to a half:

The solid yellow arrow on the left is $x$; the cyan arrow on the right is $y$; the dashed yellow arrow on the right is where $x$ used to be, for comparison. The matrix doesn't "contain" any data — it does one thing: it pushes every vector in the plane to a new place. Multiply a different vector by it and that vector gets stretched wide and squashed flat by exactly the same ratios.

This is the principle behind a layer of artificial neurons: take an input vector $x$ and push it through a transformation so that things which started out tangled become separable, easier to classify. In the neural-network context that transformation matrix is usually written $W$ and called the weights — the same object as $A$ above, under a different name. The crucial point: the numbers inside $W$ are not chosen by a human, the machine finds them itself. The whole second half of this lesson is about how it finds them.

The dot product — the operation you need before anything else

Before we get to matrix multiplication, there is a much smaller operation to master. Once you have it, matrix multiplication has nothing left to memorise, because it is just this operation repeated.

The dot product of two vectors is: multiply each pair of elements in matching positions, then add everything up. The result is a single number — not a vector. That's where the name comes from in the alternative term "scalar product": two arrows with direction go in, one plain number comes out.

An everyday way to picture it: you have a basket [2, 1, 3] — 2 pens, 1 notebook, 3 folders — and a price list [5, 20, 8]. The bill is $2 \times 5 + 1 \times 20 + 3 \times 8 = 54$. You just computed a dot product. It is everywhere in AI because it is exactly the operation "weigh up several factors and settle on one number": each input multiplied by how much it matters, then summed.

Two examples to show where its boundary lies.

• Works: two people rating the same 3 films out of five, an = [5, 1, 4] and binh = [4, 2, 5]. Their dot product is 42 — high, because both gave big scores to the same films. Against chi = [1, 5, 1], An's dot product is only 14. That number is a measure of "how alike are these two people", and that is genuinely how recommender systems work.
• Doesn't work: [1, 2] and [1, 2, 3]. Nothing pairs up with the 3, so the operation does not exist — it is not zero. The two vectors must have the same number of dimensions. Remember this condition: it is the root of the shape-mismatch error coming up in the next section, and of the single most common mistake people make writing their first AI code.

Written in pure Python it is as short as the description — one line for the whole operation:

def dot(a, b):
    # Two vectors must have the same length, or there is nothing to pair up.
    if len(a) != len(b):
        raise ValueError(f"length mismatch: {len(a)} vs {len(b)}")
    # zip() walks both lists in step, giving (a[0], b[0]), (a[1], b[1]), ...
    return sum(x * y for x, y in zip(a, b))

basket, price = [2, 1, 3], [5, 20, 8]
print("total bill:", dot(basket, price))          # 54

an, binh, chi = [5, 1, 4], [4, 2, 5], [1, 5, 1]
print("an . binh =", dot(an, binh))               # 42 -> similar taste
print("an . chi  =", dot(an, chi))                # 14 -> different taste

print(dot([1, 2], [1, 2, 3]))                     # ValueError: length mismatch: 2 vs 3

That last line deliberately blows up. Run it once — meeting the length-mismatch error while you are causing it on purpose is far more pleasant than meeting it later inside a 20-layer model.

2.2 Decoding matrix multiplication

With the dot product in hand, matrix multiplication is no longer a rule to memorise. It is just this: take the dot product of each row on the left with each column on the right. The cell at row $i$, column $j$ of the result is the dot product of row $i$ of the left matrix and column $j$ of the right matrix. Nothing more.

And because a dot product demands two vectors of the same length, the shape condition for matrix multiplication follows on its own, with nothing to remember: the number of columns in the left matrix must equal the number of rows in the right matrix — because however many elements a row on the left has, a column on the right must have exactly that many to pair up with.

If matrix $A$ is $m \times n$ and matrix $B$ is $n \times p$, the result $C = A \cdot B$ is $m \times p$, and each element $C_{ij}$ is given by: $$C_{ij} = \sum_{k=1}^n A_{ik} B_{kj}$$

Look closely at that sum: it is the dot product formula you just wrote, with the variables renamed. If the formula looks forbidding, read it aloud as a sentence: "this cell is the dot product of row $i$ with column $j$".

Written in pure Python it needs three nested for loops: one walking the rows, one walking the columns, and one — the innermost — which is the dot product being carried out by hand:

def naive_matrix_multiply(A, B):
    m, n = len(A), len(A[0])      # rows and columns of the left matrix
    n_B, p = len(B), len(B[0])    # rows and columns of the right matrix

    # Columns of A must match rows of B, or the row/column pairs cannot be zipped.
    if n != n_B:
        raise ValueError(f"cannot multiply: A is {m}x{n} but B is {n_B}x{p}")

    # Start with a result matrix full of zeros, then accumulate into it.
    C = [[0.0 for _ in range(p)] for _ in range(m)]

    for i in range(m):                          # walk the rows of A
        for j in range(p):                      # walk the columns of B
            for k in range(n):                  # ← this innermost loop IS the dot product
                C[i][j] += A[i][k] * B[k][j]
    return C

A = [[1, 2], [3, 4]]    # 2x2
B = [[5, 6], [7, 8]]    # 2x2
print("naive matrix product:", naive_matrix_multiply(A, B))
# Output: [[19.0, 22.0], [43.0, 50.0]]

Check the first cell by hand, to trust the code: row 1 of $A$ is [1, 2], column 1 of $B$ is [5, 7], and their dot product is $1 \times 5 + 2 \times 7 = 19$ — exactly the number in the top left corner. Those three loops do nothing but repeat that calculation for all 4 cells.

One small detail that can throw you when you run it: the output prints 19.0 rather than 19, even though every input was a whole number. The cause is the 0.0 on the initialisation line — add an integer to a float in Python and you get a float back. Change it to 0 and you get integers. Nothing is wrong, but this is the kind of discrepancy that makes a beginner think they mistyped something, so it saves time to know about it.

In practice, getting shapes wrong is the single most common error when writing your first AI code, and NumPy reports it with a fairly confusing message. Knowing how to read that message is worth much more than memorising the rule, so let's cause it on purpose once.

import numpy as np

A = np.array([[1, 2, 3], [4, 5, 6]])   # shape (2, 3)
B = np.array([[1, 2, 3], [4, 5, 6]])   # shape (2, 3) — same shape as A, which does NOT fit

# Columns of A = 3, rows of B = 2. 3 != 2, so the row/column pairs cannot be zipped.
try:
    C = A @ B
except ValueError as e:
    print("error:", e)

# The fix used most often: .T transposes B, turning (2,3) into (3,2).
C = A @ B.T   # (2,3) @ (3,2) -> (2,2), valid
print("after transposing B:\n", C)

# Best habit of all: print both shapes BEFORE the line that fails.
print(A.shape, B.shape)   # (2, 3) (2, 3)

The message printed is, word for word (NumPy 2.4) — long, but only three parts need reading:

error: matmul: Input operand 1 has a mismatch in its core dimension 0,
with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 2 is different from 3)

Decoding those three parts:

• "operand 1" — NumPy counts from 0, so operand 0 is $A$ and operand 1 is $B$. The culprit being named is the matrix on the right.
• "core dimension 0" — dimension 0 of $B$, meaning its number of rows.
• "size 2 is different from 3" — $B$ has 2 rows, but 3 are needed to match $A$'s 3 columns. Exactly the "left columns = right rows" rule.

The gufunc signature (n?,k),(k,m?)->(n?,m?) in the middle is that shape rule written in symbols: the letter k appears in both input slots, meaning the columns of the first and the rows of the second are forced to be the same number. Once you can read that line you don't need any explanation at all.

The practical tip that condenses all of the above: whenever you hit a matmul ValueError, print the .shape of both arrays on the line immediately before the failing one. Almost every time you will spot which dimension is off, faster than reading the message.

2.3 Derivatives — the compass pointing towards less error

We now have the tools to represent a neural network: matrices and matrix multiplication. But as section 2.1 said, the numbers inside the weight matrix $W$ are not set by a human — the machine has to find them. The rest of this lesson answers that, and the tool is the derivative.

Geometrically, the derivative of a function $f(x)$ at a point is the slope of the line tangent to the graph at that point. It tells you whether the function is rising or falling, and how fast, if we shift $x$ by a tiny amount.

An everyday way to picture it: you are standing on a hillside in thick fog. You cannot see where the valley is, but you can feel which way the ground under your feet tilts, and how steeply. The derivative is that sensation, written as a number. And it is also exactly all the information a training algorithm ever gets — it never sees the answer, it only feels the tilt where it currently stands.

Two examples of what that number says. One: if $f$ is the position of a car over time, its derivative is the speed — a large derivative means moving fast. Two: at the very bottom of the valley the ground is flat, so the derivative is 0. That zero is the signal "we have arrived, stop" — and you will watch the algorithm in section 2.4 stop on exactly that signal.

The theoretical limit definition of the derivative:

Read the formula aloud to make it land: nudge $x$ to the right by a very small amount $h$, see how much $f$ changes, then divide by $h$ — that is "change vertically divided by change horizontally", which is precisely what a slope is. The $\lim_{h \to 0}$ notation only adds that $h$ has to be infinitely small.

A computer cannot compute an infinitely small limit, but it doesn't need to: just pick an $h$ small enough and evaluate that same expression. This approach is called finite difference, and $h$ is usually taken as $10^{-5}$:

def f(x):
    return x**2

def numerical_derivative(func, x, h=1e-5):
    # The limit formula, written out literally. No calculus rules needed.
    return (func(x + h) - func(x)) / h

slope = numerical_derivative(f, 3)
print(f"approximate derivative at x=3: {slope:.5f}")
# approximate derivative at x=3: 6.00001
# By hand, the exact derivative of x**2 is 2x, so at x=3 the true answer is 6.

What prints is 6.00001, not 6. That 0.00001 discrepancy is not a bug — it is the leftover $h$, because we used a small finite step in place of "infinitely small". This is worth remembering: this way of computing derivatives is always slightly wrong, and if you shrink $h$ to reduce the error you hit the opposite problem — subtracting two nearly equal numbers destroys the significant digits of a float. That is why real frameworks don't use this method but use autograd, which gives exact derivatives; you'll meet it in Lesson 4. In this lesson we accept a small error in exchange for seeing the principle in bare code.

With a real loss function, depending on millions of weights $W$ rather than one variable $x$, we take the derivative with respect to each weight in turn, treating all the others as constants. A derivative taken that way is called a partial derivative, written $\frac{\partial f}{\partial w_i}$. Its practical meaning is very concrete: "if I nudge only this one weight, how does the error change?"

Collecting all of those partial derivatives into a single vector gives the gradient vector, written $\nabla f$ (pronounced "nabla f"): $$\nabla f = \left[ \frac{\partial f}{\partial w_1}, \frac{\partial f}{\partial w_2}, \dots, \frac{\partial f}{\partial w_n} \right]$$

Geometrically, the gradient always points in the direction of steepest ascent at that point. So to find the minimum of the error function — the bottom of the valley — we must travel in the opposite direction, $-\nabla f$. That is the core idea of the legendary optimisation algorithm gradient descent.

2.4 Hands-on project: writing gradient descent yourself

Now let's put it all together into a working algorithm. We'll write gradient descent ourselves to find the minimum of: $$f(x) = x^2 - 4x + 4$$ This function can be rewritten as $f(x) = (x-2)^2$, so we already know the answer: the minimum value is 0, at $x = 2$.

Knowing the answer in advance is deliberate. The algorithm will not be told that number — all it may use is the derivative, exactly like the feet-in-the-fog sensation from the previous section. With the answer in hand, we can check whether it really arrives, instead of having to take it on trust.

We put the starting point far away, $x_{\text{init}} = 10.0$, then let the loop crawl its own way back using the update rule: $$x_{\text{new}} = x_{\text{old}} - \eta \cdot f'(x_{\text{old}})$$ where $\eta$ (pronounced "eta") is the learning rate, which decides how long each step is.

The thing worth pausing on for a second is the minus sign. The derivative points uphill; we want to go down, so we subtract. That is the entire content of the word "descent" in the algorithm's name, and it is also the easiest part to get muddled about when you are new — the hand-traced table further down shows this minus sign automatically doing the right thing even when $x$ is on the left of the bottom.

Look at the path it takes on the graph before reading the code — the first four steps, with $\eta = 0.1$:

Notice how the early steps are long, then get shorter as it nears the bottom. Nobody programmed it to slow down: near the bottom the slope is small, and step length is proportional to slope, so it brakes by itself. This is a rather beautiful property of the algorithm — move fast where you are far away, move carefully where you are already close.

Two more words in the coming code are worth knowing first. An epoch is one update iteration — in this lesson, one step downhill (with real data, one epoch means the model has been through the entire dataset once). A hyperparameter is a value we choose and the algorithm never adjusts — such as learning_rate or epochs. Keep it distinct from a parameter, meaning the weights the algorithm finds by itself; here the only parameter is x. Choosing hyperparameters is your job, finding parameters is the machine's.

def f(x):
    """The loss function we are minimising."""
    return x**2 - 4 * x + 4

def numerical_derivative(func, x, h=1e-5):
    """Approximate the derivative with a finite difference."""
    return (func(x + h) - func(x)) / h

# Hyperparameters — values WE choose, which the algorithm never changes.
x = 10.0             # starting guess, deliberately far from the answer
learning_rate = 0.1  # how big a step to take each iteration
epochs = 100         # maximum number of steps
tolerance = 1e-6     # stop once x barely moves any more

for epoch in range(1, epochs + 1):
    grad = numerical_derivative(f, x)

    # Step AGAINST the slope. This one line does all the learning.
    x_new = x - learning_rate * grad

    # Converged: x stopped moving, so more iterations would change nothing.
    if abs(x_new - x) < tolerance:
        print(f"Converged early at iteration {epoch}. x = {x_new:.6f}")
        x = x_new
        break

    x = x_new
    print(f"Iteration {epoch:02d}: x = {x:.6f} | loss = {f(x):.6f}")

print(f"\nResult: x = {x:.6f} | minimum value: f(x) = {f(x):.6f}")

Run it and you'll see exactly this (with the middle trimmed):

Iteration 01: x = 8.399999 | loss = 40.959987
Iteration 02: x = 7.119998 | loss = 26.214382
Iteration 03: x = 6.095998 | loss = 16.777196
...
Iteration 65: x = 1.999999 | loss = 0.000000
Converged early at iteration 66. x = 1.999998

Result: x = 1.999998 | minimum value: f(x) = 0.000000

Three things to read out of this result. One: the algorithm reached $x = 1.999998$ — about two parts in a million away from the true answer of exactly $2$, and it was never told the number 2. Two: it stopped on iteration 66 rather than running all 100, because the tolerance branch caught the moment $x$ had all but stopped moving. Three: the value epochs = 100 was not picked at random — with $\eta = 0.1$ it takes until iteration 66 for the step to become small enough to stop; set epochs = 50 and the loop ends before convergence and you would never see that "Converged early" line at all.

Finally, to see the relationship between the sign of the derivative and the direction $x$ moves — the easiest thing to muddle when starting out — here are the first three iterations traced by hand with $x_{\text{init}} = 10$, $\eta = 0.1$:

The invariant rule throughout: positive derivative → $x$ must DECREASE to go downhill (since we subtract a positive number); conversely if the derivative is NEGATIVE (we are to the left of the minimum at $x=2$), subtracting a negative number makes $x$ INCREASE — so the algorithm always crawls in the right direction towards the minimum at $x=2$ without knowing the answer beforehand, purely from the sign of the derivative at each step.

A small detail so nothing confuses you when comparing: the table uses the exact derivative $f'(x) = 2x - 4$, which is why it gives round values like $16$ and $8.4$. The code uses a finite difference, so it prints $16.000010$ and $8.399999$. That discrepancy is the $h$ discussed in section 2.3 — the table computes by mathematics, the code computes by measurement. Both are correct, they just differ in precision.

How to run this project on your own machine

1. Download gradient_descent.py at the end of the lesson, or retype the code above into a new file.
2. Open a terminal in the folder containing the file. No virtual environment needed, nothing to install — this script uses pure Python only.
3. Run it: python3 gradient_descent.py (on Windows, python gradient_descent.py).
4. Compare the last line with the result above. Then do the part that pays off most: change learning_rate to 0.5, then 1.0, then 1.5, running it each time, and watch the algorithm converge fast, then oscillate, then explode. Reading about those four behaviours is easy to forget; watching them appear in your own terminal is much harder to forget.

Lesson summary & bridge to what's next

Bridge to the next lesson: to push matrix computation up to millions of operations per second without choking the CPU, we need to learn how to parallelise data with the two core libraries NumPy and Pandas, in Lesson 3.