Lesson 10: Hardware Acceleration: GPU, NPU & AMX

Introduction: once bandwidth is enough, the next wall is computation

Lesson 9 dealt with getting data to the right place as fast as possible (UMA removes the PCIe copy). But having data quickly is not enough — a scalar CPU still has to COMPUTE one element at a time, in order. 3D graphics and deep learning both revolve around ONE core operation: matrix multiplication. This lesson compares 3 hardware architectures running that SAME operation at throughputs that differ by a factor of thousands.

1. SIMD against SIMT

SIMD (single instruction, multiple data) — modern CPUs have wide vector registers (AVX-256, say, handles 8 32-bit numbers AT ONCE in ONE instruction) — still fundamentally a scalar architecture, just with each instruction "widened" to cover more data. SIMT (single instruction, multiple threads) — how a GPU is organised: thousands of small, simple Shader Core running the SAME instruction but across thousands of independent data streams in parallel — a FUNDAMENTALLY different thing from SIMD's "widening" of one instruction.

2. NPUs & Apple AMX

NPU (neural processing unit) and the Tensor Core are DEDICATED hardware built to accelerate exactly ONE operation: matrix multiplication (and its convolution variants) — tens of times faster than the equivalent on an ordinary CPU, because the hardware is designed to do ONE thing rather than to be flexible like a CPU. Apple AMX (Apple matrix coprocessor) is a matrix coprocessor HIDDEN inside the Apple Silicon CPU core itself — no data has to move to the GPU (continuing Lesson 9's UMA spirit); the CPU issues AMX instructions directly when running through the Accelerate framework or CoreML.

SIMD (CPU vector, vd AVX-256):
  1 lenh --> xu ly CUNG LUC 8 so 32-bit trong 1 thanh ghi rong
  Van la 1 loi thuc thi (control flow), chi "no rong" DU LIEU

SIMT (GPU, vd CUDA/Metal):
  1 lenh --> hang nghin Shader Core doc lap chay CUNG lenh do
  MOI Shader Core co du lieu RIENG - that su nhieu LUONG thuc thi song song
  (neu cac luong trong 1 nhom RE NHANH khac huong -> phai chay TUAN TU ca 2 nhanh)

Kien truc      | Muc dich chinh              | Vi du
---------------|------------------------------|---------------------------
CPU + SIMD     | Da nang, phep toan da dang   | AVX/AVX-512 (x86), NEON (ARM)
GPU (SIMT)     | Song song HANG NGHIN luong   | Do hoa, huan luyen mo hinh lon
NPU/TensorCore | Chuyen dung nhan ma tran     | Suy luan AI tren thiet bi di dong
Apple AMX      | Nhan ma tran AN trong CPU    | Accelerate framework, CoreML
                 (khong can chuyen du lieu sang GPU - tiep noi tinh than UMA Bai 9)

3. Counting FLOPs and computing throughput

Multiplying two $N \times N$ matrices with the classic sequential algorithm: each of the $N^2$ result elements needs $N$ multiplications + $(N-1)$ additions — total floating-point operations (FLOPs):

$$\text{Total FLOPs} = N^2 \times (2N - 1) = 2N^3 - N^2$$

Verified for real: multiplying 1024×1024 matrices takes exactly 2.146.435.072 FLOPs (~2.15 billion). Compare the time for that SAME operation on 3 architectures — a scalar CPU (4 GFLOPS), a SIMD CPU (32 GFLOPS, equivalent to 8-wide AVX = 8× scalar), and a GPU (10 TFLOPS):

function matrixMultiplyFlops(n) {
  return 2 * Math.pow(n, 3) - Math.pow(n, 2);
}
function computeTimeSeconds(flops, flopsPerSecond) {
  return flops / flopsPerSecond;
}
// Verified: matrixMultiplyFlops(1024) = 2.146.435.072 FLOPs
// Verified: scalar (4 GFLOPS)  -> 0,5366 giay
// Verified: SIMD (32 GFLOPS)   -> 0,0671 giay (nhanh hon scalar DUNG 8 lan = 32/4)
// Verified: GPU (10 TFLOPS)    -> 0,000215 giay = 0,215 ms (nhanh hon scalar 2500 lan)

SIMD beats scalar by exactly 8× — EXACTLY the vector width, not an estimate. The GPU beats scalar by 2500× — that enormous gap comes from the GPU having thousands of shader cores genuinely working in parallel, quite unlike SIMD merely "widening" one instruction.

// Ma tran 4x4: chi 112 FLOPs - qua nho de GPU "dang" cong suc
const cmpSmall = compareComputeMethods(4, 4, 32, 10, 0.0001); // 0,1ms overhead GPU
// cmpSmall.scalarTimeSeconds ~ 0.000000028 s (khong overhead)
// cmpSmall.gpuTimeSeconds    ~ 0.0001 s (GAN NHU TOAN BO la overhead, tinh toan that ~0)
// Verified: cmpSmall.gpuTimeSeconds > cmpSmall.scalarTimeSeconds -> GPU CHAM HON!

4. The roofline model: 10 TFLOPS only means something if the data keeps up

There is a quiet assumption running through all of section 3 that has to be brought into the open: every time above was computed by dividing FLOPs by PEAK throughput. That silently assumes the hardware always has data on hand to compute with — infinite memory bandwidth. No hardware is like that. To compute you must first load, and for a great many problems it is the loading that decides the time.

The roofline model (Williams, 2009) weighs those two limits against each other using a single quantity: arithmetic intensity — how many operations you get per byte read from memory.

Dividing one by the other gives the ridge point — the minimum intensity at which bandwidth stops being the bottleneck. For a 10 TFLOPS GPU with 600 GB/s: $10\times10^{12} / 600\times10^{9} = \mathbf{16.67}$ FLOP/byte. Below that you are memory-bound, above it compute-bound.

Put two operations on that same GPU. An $N \times N$ matrix multiply does $2N^3 - N^2$ operations while touching only $3N^2$ elements — the work grows as $N^3$ and the data only as $N^2$, so the bigger it gets the better the deal. Vector addition is the opposite: 2 operations per element, but it must read x, read y and write y — a ratio that is fixed, and never improves however large the array grows.

const PEAK = 10e12;  // 10 TFLOPS
const BW   = 600e9;  // 600 GB/s  -> ridge point = 16.67 FLOP/byte

// Matrix multiply 1024x1024, FP32, each matrix touched once (ideal blocking)
rooflineAttainable(matmulArithmeticIntensity(1024, 4), PEAK, BW);
// AI = 170.58 -> { bound: 'compute', attainable: 10 TFLOPS, fractionOfPeak: 1.00 }

// Vector add y[i] = a*x[i] + y[i] on the SAME hardware
rooflineAttainable(vectorAddArithmeticIntensity(4), PEAK, BW);
// AI = 0.167  -> { bound: 'memory',  attainable: 100 GFLOPS, fractionOfPeak: 0.01 }

// Same GPU. 100% of peak against 1% of peak. Peak FLOPS predicts neither.

5. Warp divergence: the price of one stray thread

The pitfall in section 1 said the GPU "has to run both branches sequentially" when threads go different ways. That phenomenon has a name — warp divergence — and it is worth putting a number on, because intuition gets it wrong here.

A GPU does not schedule threads individually. It groups them — NVIDIA calls the group a warp (32 threads), AMD a wavefront (64) — and the whole group SHARES one program counter. The entire warp must execute the same instruction at any instant. So when it meets an if where threads go different ways, the hardware has no option but to run the then branch with the remaining threads DISABLED, then run the else branch with the other group disabled.

// All 32 threads take the same path
warpDivergence(new Array(32).fill(true));
// -> { passes: 1, efficiency: 1.00, wastedSlots: 0 }

// The warp splits evenly, 16 and 16
warpDivergence(Array.from({ length: 32 }, (_, i) => i < 16));
// -> { passes: 2, efficiency: 0.50, wastedSlots: 32 }

// ONE thread out of 32 goes the other way
warpDivergence(Array.from({ length: 32 }, (_, i) => i === 0));
// -> { passes: 2, efficiency: 0.50, wastedSlots: 32 }   identical to the 16/16 split

6. Hands-on: matrix multiply on scalar, SIMD and GPU/AMX

Change the matrix size and the three architectures' figures to watch the times move directly — try dropping N very low (to 4, say) to see the GPU overhead pitfall verified above for yourself:

Summary

• ✅ SIMD "widens" one CPU instruction to cover several data elements at once; SIMT (the GPU) runs thousands of genuinely independent threads — a fundamentally different way of organising parallelism.
• ✅ NPUs, tensor cores and Apple AMX are DEDICATED hardware built purely to accelerate matrix multiplication.
• ✅ Verified: a 1024×1024 matrix multiply is 2,146,435,072 FLOPs. SIMD beats scalar by exactly 8× (the vector width); the GPU beats scalar by 2500×.
• ✅ Pitfall: for a matrix that is too small (4×4), the GPU's data-loading overhead makes it SLOWER than the scalar CPU — verified with the engine.
• ✅ Pitfall: GPUs are very weak on complex sequential branching code — they cannot replace a CPU for everything.
• ✅ Roofline: attainable throughput = $\min(\text{peak FLOPS}, \text{AI} \times \text{bandwidth})$. Verified on the SAME 10 TFLOPS / 600 GB/s GPU: matrix multiply (AI 170) uses 100% of the power, vector addition (AI 0.167) just 1% — peak FLOPS is not enough to predict performance.
• ✅ Pitfall: for a memory-bound problem, buying hardware with more FLOPS gives exactly 0% improvement. Tiling to reuse the cache (Lesson 7) is what drags AI over to the compute-bound side.
• ✅ Warp divergence: verified, just ONE thread in 32 going the other way pulls the whole warp down to 50% efficiency — exactly the damage of an even 16/16 split. The cost is not proportional to the number of stray threads.

This whole lesson runs in one direction: to go faster, pile on more dedicated hardware — more cores, a separate matrix unit, wider bandwidth. But piling on means the chip grows, and the bigger a chip is the likelier a single speck of dust ruins it, with cost rising far faster than area. That is the economic wall standing behind every technical wall above — Lesson 11 puts a number on that price, and shows why the industry was forced to cut the big chip into small pieces (chiplets).

Review quiz