Spot the odd invoice
Two rules you can explain to your accountant before you touch a model: z-score and IQR. You will see why one typo hides another.
AMOUNTS holds 36 office-supply invoices (synthetic data, in EUR). Two are odd: a typo of 4250 for 425 and a duplicated order of 980. Write five small functions:
zScores(xs): (x - mean) / standard deviation, dividing by n. If every value is the same, return zeros.
quantile(sorted, p): takes a sorted list. Work out the position (n - 1) * p and interpolate between the two neighbours (the NumPy default).
iqrFences(xs): returns { q1, q3, iqr, low, high }, with low = q1 - 1.5 * iqr and high = q3 + 1.5 * iqr.
flagZ(xs, limit) and flagIQR(xs): the indexes of the flagged invoices, in order. The preview draws what each rule flags and lets you move the z limit.
Challenges 0/4
- z-score with the deviation over n, and zeros when nothing varies
- Interpolated quartiles and fences at 1.5 IQR
- With a limit of 3, the z-score only sees the 4250 typo
- The IQR flags the typo and the duplicated order
function zScores(xs) {
// (x - mean) / standard deviation, dividing by n; all zeros when every value is the same
return xs.map(() => 0);
}
function quantile(sorted, p) {
// position (n - 1) * p, then interpolate between the two neighbours
return sorted[0];
}
function iqrFences(xs) {
// q1, q3, iqr = q3 - q1, low = q1 - 1.5 * iqr, high = q3 + 1.5 * iqr
return { q1: 0, q3: 0, iqr: 0, low: 0, high: 0 };
}
function flagZ(xs, limit) {
// indexes whose |z| is above limit
return [];
}
function flagIQR(xs) {
// indexes below low or above high
return [];
}
console.log('z:', flagZ(AMOUNTS, 3), 'IQR:', flagIQR(AMOUNTS));The idea
Before you train anything, try a rule you can explain in one sentence. "This invoice is more than three deviations from the mean" or "it sits above the IQR fence" is easy to follow in a review with your accountant, and that is worth more than a model nobody can justify.
The z-score has a known flaw: the mean and the deviation come from all the data, the odd values included. The 4250 typo inflates the deviation so much that the duplicated 980 order ends up with a z below 1 and slips through. This is called masking.
The IQR uses quartiles, and an extreme value barely moves a quartile. That is why it flags both invoices. The isolation idea goes one step further: cut the range at random until the invoice is alone. An odd invoice is isolated in few cuts. This is the basis of Isolation Forest, and the preview computes it with 100 trees.
No rule decides for you. They say what to check first, and a person confirms.
Run it on your machine
With Python and pandas, on this exercise's CSV (CC BY 4.0):
curl -O https://vorluxai.com/data/labs/invoices-office-supplies.csv
python3 -c "
import pandas as pd
df = pd.read_csv('invoices-office-supplies.csv')
q1, q3 = df.amount.quantile([0.25, 0.75])
high = q3 + 1.5 * (q3 - q1)
print(df[df.amount > high])
"Go deeper: the scikit-learn reference →
This in production, with your data? Let's talk for 15 minutes →