Agents and perception · Medium · 20 min

Teach your browser to see

An 8.8 MB model turns each photo into 1,280 numbers and your nearest neighbour decides: orange, lemon or avocado. No training at all.

DATA.examples holds 18 photos from the conveyor belt of a fruit warehouse, each as { label, name, image, vec }, with vec the 1,280 numbers MobileNetV2 computed for it; DATA.query is one more photo, unlabelled. Write five functions:

meanVector(vectors): the mean, position by position. cosine(a, b): dot product over the product of the lengths; 0 when either length is 0.

nearest(examples, vec, k): the k examples most similar to vec as [{ label, name, sim }], highest sim first; equal sim keeps the order they came in.

vote(neighbours): the label with most neighbours; on a tie, the larger summed sim; if still tied, the first in alphabetical order.

leaveOneOut(examples, k, centre): classify each example with all the others (vote(nearest(...))) and return the fraction right. With centre, first subtract the mean of the others from their vectors and from the held-out one. The preview shows the examples, the score and the photo to classify.

Challenges 0/5

The mean of several vectors and the cosine between two
The nearest neighbours, in order, ties where they were
The vote: majority, then the summed similarity, then the alphabet
The 18 conveyor-belt photos, leaving one out: 15 raw, 17 centred
With the model's vectors (your photos) or the sample: a known class for the new photo

function meanVector(vectors) {
  // the mean of equally long arrays, position by position
  return [];
}

function cosine(a, b) {
  // dot product / (length of a * length of b); 0 when either has length 0
  return 0;
}

function nearest(examples, vec, k) {
  // the k examples most similar to vec as [{ label, name, sim }], highest sim first; ties keep the input order
  return [];
}

function vote(neighbours) {
  // the label with most neighbours; tie: the larger summed sim; then alphabetical
  return '';
}

function leaveOneOut(examples, k, centre) {
  // classify each example with all the others; if centre, first subtract the mean of the others from every vector
  // return the fraction classified right
  return 0;
}

console.log(leaveOneOut(DATA.examples, 1, false), leaveOneOut(DATA.examples, 1, true));

Console output appears here (console.log).

The idea

For a program to tell your good parts from the faulty ones you do not always need to train a network. A model already trained on millions of photos, cut before its last layer, turns each image into a vector: here, 1,280 numbers from MobileNetV2. Two similar photos give vectors that point the same way, and the cosine measures that. With a few labelled examples, the new photo takes the label of its nearest neighbours (k-NN). Teaching is adding examples; there is no training.

One detail makes the difference: the vectors of one model all look a bit alike. On our photos, the mean cosine between two fruits of the same class is 0.79 and between different classes 0.71. Subtracting the mean of the examples (centring) removes what they all share and leaves what sets each apart. With the 18 conveyor-belt photos, leaving one out each time, the nearest neighbour gets 15 right raw and 17 centred.

What we measured on 2026-10-19 in the browser on the DGX Spark (ONNX Runtime Web, WebAssembly, one thread): 18 to 19 ms per photo, with an 8.8 MB model. And two warnings. First, the background matters: the same three fruits photographed on five different backgrounds (crate, worktop, hand, market and plate) only get 8 of 15 right, because the model groups by background before fruit. Photograph your examples the way you will photograph what you want to classify. Second, six examples per class are few: our photo to classify, a lemon with a leaf, comes out right with k = 1 centred and wrong with k = 3, because no example has a leaf. The fix is not another model; it is two more photos.

Everything happens in your tab: the photos you add to teach it never leave your browser.

Run it on your machine

With Python and NumPy, on this exercise's vectors and those of the five-background photos (CC BY 4.0):

curl -O https://vorluxai.com/data/labs/fruta-vectores.json
python3 -c "
import json, base64, numpy as np
d = json.load(open('fruta-vectores.json'))
def vecs(items): return np.array([np.frombuffer(base64.b64decode(e['q']), np.uint8) / 255 * e['s'] for e in items]), [e['label'] for e in items]
def loo(X, y, centre):
    right = 0
    for i in range(len(X)):
        T, q, ty = np.delete(X, i, 0), X[i], y[:i] + y[i + 1:]
        if centre: m = T.mean(0); T, q = T - m, q - m
        sim = T @ q / (np.linalg.norm(T, axis=1) * np.linalg.norm(q))
        right += ty[int(sim.argmax())] == y[i]
    return f'{right}/{len(X)}'
for name in ('examples', 'fondos'):
    X, y = vecs(d[name])
    print(name, 'sin centrar', loo(X, y, False), 'centrando', loo(X, y, True))
"

Check what you learned

1. What does the model do in this exercise? It classifies the fruit directly It turns each photo into a vector of 1,280 numbers; your k-NN classifies with your examples It trains a new network on your photos It sends the photos to a server to compare them

2. What is centring the vectors for? To make them smaller To remove what all the photos share so the cosine looks at what sets them apart To make the model faster To normalise the labels

3. You photograph the good parts on the table and the faulty ones on the floor. What can happen? Nothing: the model only looks at the part The classifier learns to tell table from floor and fails when the background changes It gets slower The vectors come out negative

4. Your classifier mistakes lemons with a leaf for oranges. What do you try first? A model ten times bigger Add a few photos of lemons with a leaf as examples Raise k to 15 Remove the centring

Go deeper: the GitHub reference →

This in production, with your data? Let's talk for 15 minutes →

Challenges 0/5

The idea

Run it on your machine

Check what you learned

VORLUX AI