A neural net from scratch
Two layers, no libraries: forward pass, loss, backpropagation and gradient descent. Every derivative is checked against finite differences.
You will write a two-layer network (2 inputs, H hidden tanh units, 1 sigmoid output) that learns XOR and, in the preview, two spirals. The parameters are { W1, b1, W2, b2 } (initParams gives seeded random ones). Five functions:
forward(p, x): h[i] = tanh(b1[i] + Σ W1[i][j] · x[j]), y = sigmoid(b2 + Σ W2[i] · h[i]). Return { h, y }.
loss(p, X, Y): mean binary cross-entropy, -(Y · log y + (1 - Y) · log(1 - y)), with y clamped to [1e-7, 1 - 1e-7].
gradients(p, X, Y): the same shape as p. For each example, dz = (y - Y[k]) / n at the output; then dW2[i] += dz · h[i], db2 += dz, dh = dz · W2[i] · (1 - h[i]²), db1[i] += dh, dW1[i][j] += dh · x[j].
update(p, g, lr): a NEW object with every weight minus lr times its gradient. Do not modify p.
train(p, X, Y, epochs, lr): repeat gradients and update epochs times and record the loss after each epoch. Return { params, losses }.
The prelude ships numericGradients, which estimates every derivative by nudging the weight a little. One check compares yours against those: if they agree, your backpropagation is right.
Challenges 0/5
- forward: tanh in the hidden layer and a sigmoid at the output
- loss: mean cross-entropy, ln 2 when the net knows nothing
- gradients agrees with finite differences on every weight
- update returns new parameters and leaves the old ones untouched
- train learns XOR: the loss drops under 0.1 and all four cases come out right
function forward(p, x) {
// h[i] = tanh(b1[i] + sum of W1[i][j] * x[j]); y = sigmoid(b2 + sum of W2[i] * h[i])
return { h: [], y: 0.5 };
}
function loss(p, X, Y) {
// mean binary cross-entropy: -(Y log y + (1 - Y) log(1 - y)), with y clamped to [1e-7, 1 - 1e-7]
return 0;
}
function gradients(p, X, Y) {
// same shape as p; dz = (y - Y[k]) / n for the output, then back through W2, tanh (1 - h^2) and W1
return { W1: p.W1.map((row) => row.map(() => 0)), b1: p.b1.map(() => 0), W2: p.W2.map(() => 0), b2: 0 };
}
function update(p, g, lr) {
// a NEW params object: every weight minus lr times its gradient; p is left untouched
return p;
}
function train(p, X, Y, epochs, lr) {
// epochs times: gradients, update, record the loss; return { params, losses }
return { params: p, losses: [] };
}
const run = train(initParams(2, 4, 1), XOR.X, XOR.Y, 2000, 0.5);
console.log('loss', run.losses[run.losses.length - 1], XOR.X.map((x) => forward(run.params, x).y.toFixed(2)));The idea
A neural network is a function with many dials. The forward pass computes the output with the dials as they are; the loss measures how wrong it is; backpropagation works out, for each dial, which way to turn it to be slightly less wrong; and gradient descent turns it. That is all of training. Frameworks hide it behind .backward(), but it is these four lines.
XOR is the classic example because a single neuron cannot solve it: no straight line separates (0,1) and (1,0) from (0,0) and (1,1). With four hidden tanh units the network folds the plane and gets there in a few hundred epochs.
The part that hides the most mistakes is backpropagation. That is why the prelude ships numericGradients: it nudges each weight by a thousandth and watches how much the loss moves. Slow, but it cannot be wrong. If your derivatives agree with those to the sixth digit, you can trust them. Frameworks run the same check in their own test suites.
The spirals in the preview show the cost: the same network separates them, but with eight hidden units it needs about 2000 epochs instead of 300. With many more points or more dimensions this JavaScript loop runs out of steam, and that is where PyTorch and the GPU start. The idea is the same.
Run it on your machine
With Python and NumPy, the same network on this exercise's spirals (CC BY 4.0), with 32 hidden units:
curl -O https://vorluxai.com/data/labs/espirales.csv
python3 -c "
import numpy as np
d = np.loadtxt('espirales.csv', delimiter=',', skiprows=1)
X, Y = d[:, :2], d[:, 2:]
rng = np.random.default_rng(1)
W1, b1, W2, b2 = rng.normal(0, 0.8, (2, 32)), np.zeros(32), rng.normal(0, 0.8, (32, 1)), 0.0
for epoch in range(5000):
H = np.tanh(X @ W1 + b1); P = 1 / (1 + np.exp(-(H @ W2 + b2)))
dz = (P - Y) / len(X); dh = dz @ W2.T * (1 - H ** 2)
W2 -= 0.5 * H.T @ dz; b2 -= 0.5 * dz.sum(); W1 -= 0.5 * X.T @ dh; b1 -= 0.5 * dh.sum(0)
print('correct', ((P > 0.5) == Y).mean())
"Go deeper: the Google Machine Learning Crash Course reference →
This in production, with your data? Let's talk for 15 minutes →