Lesson 25 — Ultimate: tiny neural net

Boss level: a neural network that solves XOR

A neural network stacks layers. Each neuron computes a weighted sum of its inputs plus a bias, then applies an activation function. ReLU is the most common: relu(z) = max(0, z). Computing the output from the inputs is the forward pass.

XOR ("one or the other, not both") is famous because no single straight line can separate it — a hidden layer is needed. With these hand-picked weights: h1 = relu(x1 + x2), h2 = relu(x1 + x2 − 1), output = h1 − 2 × h2.

relu(-0.5)   # 0
relu(1.5)    # 1.5

Why it matters: every deep-learning model, up to large language models, is this forward pass — just with billions of weights that were learned instead of hand-picked.