Trust your classifier
A churn model is already trained. You set the threshold: every false positive is a needless call and every false negative a customer who leaves.
MODEL is a logistic regression already fitted on CUSTOMERS (80 synthetic customers: months as a customer, support tickets, monthly spend and whether they left). An accuracy percentage tells you nothing until you know which mistakes it makes. Write six functions:
sigmoid(z): 1 / (1 + e^-z).
probability(model, customer): z = b + Σ w[i] · (customer[feature] - mean[i]) / sd[i] over FEATURES, then sigmoid(z).
confusionMatrix(probs, labels, threshold): { tp, fp, fn, tn }. Churn is predicted when p >= threshold; labels carries 1 for the real churners.
metrics(cm): { precision, recall, accuracy }, 0 when a denominator is 0.
cost(cm, costFP, costFN): fp · costFP + fn · costFN.
bestThreshold(probs, labels, costFP, costFN): the threshold in THRESHOLDS with the lowest cost; the lowest one on a tie.
The preview draws the matrix and the cost curve, and lets you move the threshold and the relative price of each error.
Challenges 0/4
- sigmoid and probability: the S curve over the standardised features
- confusionMatrix counts the four cases from the threshold (p >= threshold is churn)
- metrics: precision, recall and accuracy, never dividing by zero
- cost and bestThreshold: the threshold drops when losing a customer is expensive
function sigmoid(z) {
// 1 / (1 + e^-z)
return 0.5;
}
function probability(model, customer) {
// z = b + sum of w[i] * (customer[feature] - mean[i]) / sd[i], then sigmoid
return 0.5;
}
function confusionMatrix(probs, labels, threshold) {
// a customer is predicted to leave when p >= threshold; count tp, fp, fn, tn
return { tp: 0, fp: 0, fn: 0, tn: 0 };
}
function metrics(cm) {
// precision, recall and accuracy; 0 when a denominator is 0
return { precision: 0, recall: 0, accuracy: 0 };
}
function cost(cm, costFP, costFN) {
// needless calls times costFP plus missed churners times costFN
return 0;
}
function bestThreshold(probs, labels, costFP, costFN) {
// the threshold in THRESHOLDS with the lowest cost (the lowest threshold on a tie)
return 0.5;
}
const probs = CUSTOMERS.map((c) => probability(MODEL, c));
console.log(confusionMatrix(probs, LABELS, 0.5), bestThreshold(probs, LABELS, 1, 8));The idea
A classifier does not decide: it gives a probability. The decision is the threshold, and you set the threshold with your business costs in hand. The confusion matrix separates the two possible mistakes: calling a customer who was not going to leave (a needless call) and not calling one who leaves (a lost customer). They almost never cost the same.
At a 0.5 threshold the model in this exercise gets a lot right, but it misses churners. If a missed churner costs as much as twenty calls, lowering the threshold to 0.2 is cheaper even though "accuracy" gets worse. That sum is the one that matters, and it is the one an accuracy percentage hides.
Precision tells you how many of your alarms were real; recall, how many real churners you caught. Raising one usually lowers the other, and the threshold is the lever between them.
This ties into the transparency the EU AI Act asks of systems that affect people: being able to say which error the system accepts, and why, is part of the documentation.
Run it on your machine
With Python and scikit-learn, on this exercise's CSV (CC BY 4.0). Change the threshold and watch the matrix move:
curl -O https://vorluxai.com/data/labs/churn-clientes.csv
python3 -c "
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix
df = pd.read_csv('churn-clientes.csv')
X, y = df[['meses', 'tickets', 'gasto_mensual_eur']], df.baja
p = LogisticRegression().fit(X, y).predict_proba(X)[:, 1]
for t in (0.2, 0.5, 0.8): print(t, confusion_matrix(y, p >= t).ravel())
"Go deeper: the Google Machine Learning Crash Course reference →
This in production, with your data? Let's talk for 15 minutes →