Up to now we have optimised LLM applications by shaping prompts, building RAG pipelines to supply outside knowledge, and constructing tool-using agents. But what if you need the model to follow one very specific output structure absolutely, to speak in your company's house voice, or to teach a small 7B model the deep domain reasoning of a much larger one?
That is when you have to change the neural network's weights directly, through fine-tuning. This lesson dissects the practical difference between RAG and fine-tuning, decodes the mathematics behind the extremely common LoRA (Low-Rank Adaptation) technique, covers the standard instruction-tuning data format, and implements a LoRA training loop from scratch in NumPy.
pip install numpy. No GPU and no model download — the project
implements LoRA on a single 8×8 linear layer so you can see every matrix multiplication. Knowledge you need: Lesson 6 for loss and backpropagation — the training loop here is exactly that loop, differing only in that just two matrices get updated. Lesson 3 for matrix multiplication and what the dimensions mean.
19.1 When should you fine-tune? Telling it apart from RAG
There is a classic line in the field: "RAG is like letting the model sit an open-book exam; fine-tuning is like sending it to university for several years to change how it thinks."
- Purpose: RAG supplies outside knowledge and continuously changing data. Fine-tuning teaches new skills, shapes voice and output structure.
- Mechanism: RAG retrieves relevant passages into the prompt context. Fine-tuning runs backpropagation to update the model's weights.
- Hallucination: RAG is excellent (the evidence is right there in the prompt). Fine-tuning is moderate — the model only remembers as a probability distribution over weights.
- Prompt token cost: RAG is high (large raw context attached every time). Fine-tuning is low (the skill is baked in, so the prompt stays short).
- Data freshness: RAG is excellent (just update the vector database). Fine-tuning is poor — every knowledge update means training again.
19.2 PEFT & LoRA (Low-Rank Adaptation)
Tuning every parameter of a large language model (full fine-tuning) demands enormous compute. To train a Llama-3 8B you must store weights, gradients and optimiser states, which needs hundreds of gigabytes of enterprise GPU memory.
LoRA solves this by completely freezing the original weight matrix $W_0 \in \mathbb{R}^{d \times k}$ and instead adding a parallel branch holding a weight-correction matrix $\Delta W$, decomposed into the product of two low-rank matrices $A \in \mathbb{R}^{r \times k}$ and $B \in \mathbb{R}^{d \times r}$ with rank $r \ll \min(d, k)$:
\[\Delta W = B \cdot A\]
For example, with a hidden dimension of $d = 4096$ and $k = 4096$, the original matrix holds $4096 \times 4096 \approx 16.7$ million parameters. At rank $r = 8$, matrix $A$ holds $8 \times 4096 \approx 32{,}768$ parameters and $B$ holds another $32{,}768$. LoRA therefore trains $65{,}536$ parameters — over 250 times fewer.
During the forward pass, for an input vector $x$, the linear layer's output $h$ becomes:
\[h = x \cdot W_0 + \frac{\alpha}{r} (x \cdot B \cdot A)\]
where $\alpha$ is a scaling constant (LoRA alpha) controlling how strongly the newly trained LoRA weights affect the output. Note the order carefully: since $\Delta W = B \cdot A$, the adapter path must be $x \cdot B \cdot A$. Writing $x \cdot A \cdot B$ does not even typecheck — $x$ is $(1, d)$ while $A$ is $(r, k)$.
19.3 Training data format: instruction tuning
To teach an LLM to follow instructions, the data has to be prepared as tightly structured example pairs. The most common format is JSON Lines (JSONL):
{"instruction": "Hãy viết email xin nghỉ phép bằng giọng điệu lịch sự.", "input": "Lý do: đi khám bệnh ngày 15/7", "output": "Kính gửi Ban Giám đốc, tôi viết email này để xin phép được nghỉ làm vào ngày 15/7 vì lý do sức khỏe cần đi khám định kỳ..."}
{"instruction": "Hãy viết email xin nghỉ phép bằng giọng điệu lịch sự.", "input": "Lý do: giải quyết việc gia đình ngày 20/7", "output": "Kính gửi anh/chị quản lý, tôi xin phép được nghỉ phép ngày 20/7 để giải quyết một số công việc gia đình đột xuất..."}19.4 Lesson 19 project: LoRA from scratch in NumPy, with all four claims checked
Section 19.2 makes four claims about LoRA. This project tests all four rather than asking you to take them on faith: the base weights do not change, the adapter contributes exactly zero at initialisation, the trainable parameter count drops by two orders of magnitude, and the adapter can be folded back into the base weights after training.
"""Lesson 19 project: LoRA from scratch in NumPy, with its claims checked.
Run: python3 lora_simulation.py
Needs: pip install numpy
The lesson makes four claims about LoRA. This file tests all four rather than
asserting them:
1. the base weights W0 never change during training,
2. at initialisation the adapter contributes exactly zero, so the model
starts out identical to the base model,
3. the number of trainable parameters collapses by two orders of magnitude,
4. after training the adapter can be folded back into W0, so inference costs
nothing extra.
"""
import numpy as np
np.random.seed(42)
D_IN, D_OUT = 8, 8
RANK = 2
ALPHA = 4.0
LEARNING_RATE = 0.01
EPOCHS = 100
def init_lora(d_in, d_out, rank):
"""A is random, B is zeros. That asymmetry is deliberate - see claim 2."""
lora_a = np.random.randn(rank, d_out) * 0.1
lora_b = np.zeros((d_in, rank))
return lora_a, lora_b
def forward(x, w0, lora_a, lora_b, rank, alpha):
"""h = x·W0 + (alpha/rank)·(x·B·A).
Note the order: B then A. Since delta_W = B·A, the adapter path must be
x·B·A. Writing x·A·B does not even typecheck - A is (rank, d_out) and x is
(1, d_in), so the shapes do not line up.
"""
base = x @ w0
adapter = (x @ lora_b) @ lora_a * (alpha / rank)
return base + adapter, base, adapter
def train(x, target, w0, lora_a, lora_b, rank, alpha, epochs, lr, verbose=True):
"""Gradient descent on A and B only. W0 is never touched below."""
scaling = alpha / rank
losses = []
for epoch in range(epochs):
h, _, _ = forward(x, w0, lora_a, lora_b, rank, alpha)
loss = np.mean((h - target) ** 2)
losses.append(loss)
d_loss_d_h = 2 * (h - target) / w0.shape[1]
grad_a = scaling * ((x @ lora_b).T @ d_loss_d_h)
grad_b = scaling * (x.T @ (d_loss_d_h @ lora_a.T))
lora_a -= lr * grad_a
lora_b -= lr * grad_b
if verbose and ((epoch + 1) % 25 == 0 or epoch == 0):
print(f" epoch {epoch + 1:3d} | loss {loss:.6f}")
return lora_a, lora_b, losses
def count_parameters(d_in, d_out, rank):
frozen = d_in * d_out
trainable = rank * d_out + d_in * rank
return frozen, trainable, frozen / trainable
def rank_experiment(rank, alpha, epochs=EPOCHS, lr=LEARNING_RATE):
"""Train one adapter from an identical starting point and return its loss."""
np.random.seed(42)
w0 = np.random.randn(D_IN, D_OUT) * 0.1
x = np.random.randn(1, D_IN)
target = np.random.randn(1, D_OUT)
lora_a, lora_b = init_lora(D_IN, D_OUT, rank)
_, _, losses = train(x, target, w0, lora_a, lora_b, rank, alpha,
epochs, lr, verbose=False)
return losses[-1]
def main():
w0 = np.random.randn(D_IN, D_OUT) * 0.1
x = np.random.randn(1, D_IN)
target = np.random.randn(1, D_OUT)
lora_a, lora_b = init_lora(D_IN, D_OUT, RANK)
# --- Claim 2: the adapter starts at exactly zero -----------------------
print("=== Claim: at initialisation the adapter changes nothing ===")
h0, base0, adapter0 = forward(x, w0, lora_a, lora_b, RANK, ALPHA)
print(f" largest value in the adapter path : {np.abs(adapter0).max():.1e}")
print(f" output identical to the base model: {np.array_equal(h0, base0)}")
assert np.array_equal(h0, base0), "the adapter should be inert at t=0"
print(" B is initialised to zeros, so delta_W = B·A is the zero matrix.")
print(" That is why attaching an untrained adapter cannot hurt a model:")
print(" it starts as an exact no-op, then learns away from there.\n")
# --- Claim 3: parameter counts ----------------------------------------
print("=== Claim: far fewer trainable parameters ===")
for d, r in ((D_IN, RANK), (4096, 8), (4096, 64)):
frozen, trainable, ratio = count_parameters(d, d, r)
print(f" d={d:<5} r={r:<3} frozen {frozen:>10,} trainable {trainable:>8,}"
f" {ratio:>6.0f}x fewer")
print(" The 4096 row is one attention projection of a 7B-class model.\n")
# --- Claim 1: W0 is frozen --------------------------------------------
print("=== Training the adapter (W0 must not move) ===")
w0_before = w0.copy()
lora_a, lora_b, losses = train(x, target, w0, lora_a, lora_b,
RANK, ALPHA, EPOCHS, LEARNING_RATE)
print(f" loss {losses[0]:.6f} -> {losses[-1]:.6f}"
f" ({losses[0] / losses[-1]:.0f}x lower)")
print(f" W0 bit-identical after training: {np.array_equal(w0, w0_before)}")
assert np.array_equal(w0, w0_before), "W0 was modified - it must be frozen"
print(f" largest value in B after training: {np.abs(lora_b).max():.4f}"
f" (was exactly 0)\n")
# --- Claim 4: the adapter can be merged into W0 ------------------------
print("=== Claim: the adapter can be folded into W0 for free inference ===")
delta_w = (lora_b @ lora_a) * (ALPHA / RANK)
w_merged = w0 + delta_w
h_adapter, _, _ = forward(x, w0, lora_a, lora_b, RANK, ALPHA)
h_merged = x @ w_merged
gap = np.abs(h_adapter - h_merged).max()
print(f" largest difference between the two paths: {gap:.2e}")
assert gap < 1e-12, "merging changed the output"
print(" Same numbers, one matrix multiply instead of three. This is why")
print(" LoRA adds no inference latency once the adapter is merged - and")
print(" why you can keep many small adapters for one shared base model.\n")
# --- What the rank actually buys, and the trap in measuring it -------
print("=== What does the rank r buy? ===")
print(" Sweep A: alpha fixed at 4.0, which is the obvious experiment")
for rank in (1, 2, 4, 8):
loss = rank_experiment(rank, alpha=ALPHA)
print(f" r={rank} alpha/r={ALPHA / rank:.2f} final loss {loss:.6f}")
print(" Higher rank looks WORSE. That result is an artefact, not a finding:")
print(" the adapter is scaled by alpha/r, so holding alpha fixed quietly")
print(" shrinks every update as r grows. The sweep measured the scaling,")
print(" not the capacity.\n")
print(" Sweep B: alpha scaled with r so alpha/r stays 2.0")
for rank in (1, 2, 4, 8):
loss = rank_experiment(rank, alpha=2.0 * rank)
print(f" r={rank} alpha={2.0 * rank:<5} alpha/r=2.00 final loss {loss:.6f}")
print(" Now rank helps - up to r=4, after which an 8x8 layer with one")
print(" training example has nothing left to gain. This is why the usual")
print(" advice is to raise alpha together with r rather than tune it alone.")
if __name__ == "__main__":
main()Claim 1: why matrix B is initialised to zeros
The detail most easily skipped in LoRA: $A$ is initialised randomly but $B$ is initialised to all zeros. The consequence is that $\Delta W = B \cdot A$ is exactly the zero matrix, so at the start a model with an adapter attached is identical to the base model:
=== Claim: at initialisation the adapter changes nothing ===
largest value in the adapter path : 0.0e+00
output identical to the base model: True
B is initialised to zeros, so delta_W = B·A is the zero matrix.
That is why attaching an untrained adapter cannot hurt a model:
it starts as an exact no-op, then learns away from there.
That 0.0e+00 is not "very small" but exactly zero, and True is an exact
equality comparison, not an approximate one. This property is what makes LoRA safe to deploy: attaching an
untrained adapter to a production model cannot break anything, because it starts as an exact no-op and
only then learns away from there.
Claim 2: the trainable parameter count
=== Claim: far fewer trainable parameters ===
d=8 r=2 frozen 64 trainable 32 2x fewer
d=4096 r=8 frozen 16,777,216 trainable 65,536 256x fewer
d=4096 r=64 frozen 16,777,216 trainable 524,288 32x fewer
The 4096 row is one attention projection of a 7B-class model.The $d = 4096$, $r = 8$ row is the real size of one attention projection in a 7B-class model: from 16.7 million parameters down to 65,536 — 256 times fewer. But look at the last row: raising $r$ to 64 cuts the ratio to just 32 times. LoRA's benefit is inversely proportional to the rank you pick, so "set $r$ high to be safe" is the fastest way to lose the reason you chose LoRA.
Claim 3: the base weights really are frozen
=== Training the adapter (W0 must not move) ===
epoch 1 | loss 1.740512
epoch 25 | loss 1.531319
epoch 50 | loss 0.619526
epoch 75 | loss 0.029707
epoch 100 | loss 0.001004
loss 1.740512 -> 0.001004 (1734x lower)
W0 bit-identical after training: True
largest value in B after training: 0.3704 (was exactly 0)
The line W0 bit-identical after training: True is a real assert in the code,
comparing the base matrix bit for bit before and after 100 epochs. If the training loop accidentally
touched $W_0$, the program would stop. Matrix $B$, meanwhile, goes from exactly 0 to $0.3704$ — it is the
entirety of what was learned.
Claim 4: merging the adapter into the base weights is free
=== Claim: the adapter can be folded into W0 for free inference ===
largest difference between the two paths: 2.22e-16
Same numbers, one matrix multiply instead of three. This is why
LoRA adds no inference latency once the adapter is merged - and
why you can keep many small adapters for one shared base model.A discrepancy of $2.22 \times 10^{-16}$ is exactly 64-bit floating point rounding error, meaning the two paths compute the same result. This matters operationally: once training is done you add $\frac{\alpha}{r} B A$ straight into $W_0$ and serve it as an ordinary model — LoRA adds not one millisecond of inference latency. And because an adapter is only tens of thousands of parameters, you can keep dozens of them for different tasks on one shared base model and swap them on demand.
What rank $r$ buys you — and the trap in measuring it
The natural question: does raising $r$ make the model learn better? The obvious experiment is to hold everything else constant and vary $r$. The result is surprising:
=== What does the rank r buy? ===
Sweep A: alpha fixed at 4.0, which is the obvious experiment
r=1 alpha/r=4.00 final loss 0.000000
r=2 alpha/r=2.00 final loss 0.001004
r=4 alpha/r=1.00 final loss 0.579510
r=8 alpha/r=0.50 final loss 1.516855
Higher rank looks WORSE. That result is an artefact, not a finding:
the adapter is scaled by alpha/r, so holding alpha fixed quietly
shrinks every update as r grows. The sweep measured the scaling,
not the capacity.
Sweep B: alpha scaled with r so alpha/r stays 2.0
r=1 alpha=2.0 alpha/r=2.00 final loss 0.002811
r=2 alpha=4.0 alpha/r=2.00 final loss 0.001004
r=4 alpha=8.0 alpha/r=2.00 final loss 0.000336
r=8 alpha=16.0 alpha/r=2.00 final loss 0.000616
Now rank helps - up to r=4, after which an 8x8 layer with one
training example has nothing left to gain. This is why the usual
advice is to raise alpha together with r rather than tune it alone.The cause is right there in the formula: the adapter is multiplied by $\frac{\alpha}{r}$. Holding $\alpha = 4$ fixed while raising $r$ from 1 to 8 silently shrinks that coefficient from $4.00$ to $0.50$. Sweep A did not measure the adapter's capacity, it measured the scaling. Sweep B holds $\frac{\alpha}{r} = 2.0$ constant, and only then does rank play its real role: loss falls steadily to $r = 4$ and then flattens — because an $8 \times 8$ layer with a single training example has nothing left to improve.
The operational lesson: $\alpha$ is not an independent dial. The common convention of setting $\alpha = 2r$ exists precisely for this reason — so that changing $r$ does not silently change how strongly the adapter acts.
How to run this project on your machine
-
pip install numpythenpython3 lora_simulation.py. Thanks tonp.random.seed(42), every number in this lesson reproduces exactly on your machine. -
Then try breaking it three ways:
-
In
init_lora, changenp.zeros((d_in, rank))tonp.random.randn(d_in, rank) * 0.1. The firstassertfails immediately: the adapter is no longer a no-op, meaning attaching it to a production model would change the output before it had learned anything. -
Add
w0 -= lr * 0.001inside thetrainloop. The frozen-$W_0$assertcatches it at once — exactly the kind of leak a hastily written LoRA implementation produces. -
Change
EPOCHSfrom 100 to 500 and rerun sweep A. The gaps between ranks narrow, confirming that the difference in sweep A was about convergence speed rather than capacity.
-
In
Lesson summary & what comes next
- Achieved: telling apart when to use RAG and when fine-tuning is required, and knowing the instruction-tuning data format.
-
Achieved: implementing LoRA in NumPy and verifying its four claims with
assert: the adapter starts at exactly 0, $W_0$ does not move one bit, parameters drop 256-fold, and merging the adapter reproduces the same output to within $10^{-16}$. - Achieved: understanding why $\alpha$ and $r$ have to move together — and watching a reasonable-looking experiment produce a completely inverted conclusion when $\frac{\alpha}{r}$ is left uncontrolled.
Bridge to the next lesson: you now have a fine-tuned model, a RAG system and agents. The final question for an AI engineer is the hardest one: how do you know the system is actually good, and how do you find out the moment it starts getting worse? Lesson 20 closes the roadmap with model serving, tracing and automated evaluation.
Download the practice code for this lesson
The Python file lora_simulation.py — LoRA implemented in NumPy with all four of its claims
checked by assertion, plus the alpha/rank experiment (run python3 lora_simulation.py, needs
numpy):
Comments