Regression by gradient descent
A line that learns on its own: the squared error, its gradient and the loop that walks down it, checked against the closed form at the end.
FLATS holds 40 synthetic flats with square metres and price. You will fit price ≈ w · z + b, where z is the standardised size, in two ways that must agree. Five functions:
standardize(xs): { z, mean, sd }, with z = (x - mean) / sd and the deviation over n. If sd is 0, every z is 0.
mse(w, b, z, ys): the mean of (w · z[i] + b - ys[i])².
gradients(w, b, z, ys): { dw, db }. With error = w · z[i] + b - ys[i], dw is the mean of 2 · error · z[i] and db the mean of 2 · error.
descend(z, ys, lr, steps): start at w = 0, b = 0; steps times, subtract lr times each gradient and record the mse. Return { w, b, history }.
closedForm(z, ys): w = cov(z, ys) / var(z), b = mean(ys) - w · mean(z). The exact least-squares answer.
The preview draws the points (drag them), the exact line and the descent line after N steps, with its error curve. w divided by sd is the price per square metre the model learns.
Challenges 0/4
- standardize: mean 0, deviation 1, and zeros when nothing varies
- mse and gradients on a two-point example
- descend lowers the error at every step and reaches the exact answer
- closedForm: exact on a perfect line, and the price per m² in EUR
function standardize(xs) {
// z = (x - mean) / sd, with sd over n; return { z, mean, sd } (all zeros when sd is 0)
return { z: xs.map(() => 0), mean: 0, sd: 1 };
}
function mse(w, b, z, ys) {
// mean of (w * z[i] + b - ys[i])^2
return 0;
}
function gradients(w, b, z, ys) {
// dw = mean of 2 * error * z[i], db = mean of 2 * error, with error = w * z[i] + b - ys[i]
return { dw: 0, db: 0 };
}
function descend(z, ys, lr, steps) {
// from w = 0, b = 0: steps times, move against the gradient by lr and record mse; return { w, b, history }
return { w: 0, b: 0, history: [] };
}
function closedForm(z, ys) {
// w = covariance(z, ys) / variance(z), b = mean(ys) - w * mean(z)
return { w: 0, b: 0 };
}
const { z, sd } = standardize(M2);
console.log('EUR per m2:', closedForm(z, PRICE).w / sd, descend(z, PRICE, 0.1, 100).w / sd);The idea
Almost all of machine learning is this: a function with parameters, a measure of error and a loop that moves the parameters towards lower error. Linear regression is the case where you can see the whole thing on one screen. The mean squared error punishes big misses more than small ones, and its derivative tells you which way to turn w and b.
Standardising the square metres first is not a fad. If z is in metres and the price in hundreds of thousands, the gradient for b dwarfs the one for w, and a single step size either explodes or crawls. With z at mean 0 and deviation 1, lr = 0.1 always works and the error drops at every step.
For a line the exact answer exists: closedForm. So why descend at all? Because the closed form does not exist for a neural network or for almost anything else, and the loop does. You learn it here, where you can check it lands in the same place.
w over sd is the price per square metre the model has learned, in euros. It is the number you can explain in a meeting. The data is invented: do not use it to value anything.
Run it on your machine
With Python and NumPy, on this exercise's CSV (CC BY 4.0). The closed form and the descent must agree:
curl -O https://vorluxai.com/data/labs/pisos-valencia.csv
python3 -c "
import numpy as np
m2, price = np.loadtxt('pisos-valencia.csv', delimiter=',', skiprows=1).T
z = (m2 - m2.mean()) / m2.std()
w = b = 0.0
for step in range(300):
err = w * z + b - price
w -= 0.1 * (2 * err * z).mean(); b -= 0.1 * (2 * err).mean()
print('descent', w / m2.std(), 'closed form', np.polyfit(m2, price, 1)[0])
"Go deeper: the Google Machine Learning Crash Course reference →
This in production, with your data? Let's talk for 15 minutes →