Supervised Learning

Supervised learning fits a model to examples whose correct outputs are given: images paired with the class of the object they show, the box around each object, the label of every pixel or the depth at every pixel. The fitted model is then used to predict those outputs for new images. Murphy (2022, §1.2) calls it the most common form of machine learning; in vision it is the setting of tasks from image classification and object detection to semantic segmentation and monocular depth estimation.

The general theory of learning from data, from empirical risk minimization and the Bayes classifier to the bias–variance trade-off, is covered in machine learning. This article covers what is particular to the supervised setting: what kind of output is predicted, which model output and decision rule go with it, where the labels come from, and the choice between treating a continuous quantity as a regression target or as classes.

Definition

In supervised learning, the task is to learn a mapping $f$ from inputs $\mathbf{x}$ to outputs $y$ from a training set of $N$ input–output pairs $\{(\mathbf{x}_n, y_n)\}_{n=1}^{N}$ (Murphy, 2022, §1.2). The outputs are also called labels, targets or responses. Torralba, Isola and Freeman (2024, §9.2) call it learning from examples, or fitting a model to data. Goodfellow, Bengio and Courville (2016, §5.1.3) trace the name to the view that the target is supplied by an instructor or teacher who shows the system what to do. They also stress that supervised and unsupervised learning are not formally defined terms.

Intuition

A teacher who marks every answer gives the learner a complete specification of the task on the examples marked. The learner’s job is to produce the same answers on examples that were not marked. Two things fix what is learned: what the labels say, and what the loss rewards. The same images labelled “disk” or “square”, labelled with a radius in pixels, or labelled with the radius rounded into five bins define three different problems. A model trained on any of them learns only what its labels distinguish.

Formal Definition

Probabilistically, supervised learning fits a conditional model $p(y \mid \mathbf{x})$ , whereas unsupervised learning fits $p(\mathbf{x})$ (Murphy, §1.3). A joint model $p(\mathbf{x}, y)$ also gives the conditional, as $p(y \mid \mathbf{x}) = p(\mathbf{x}, y) / \sum_{y'} p(\mathbf{x}, y')$ (Goodfellow et al., eq. 5.2), which is one reason the boundary between the two kinds of learning blurs.

For classification over $C$ classes, a model outputs unnormalized log-probabilities, the logits $\mathbf{a} = f(\mathbf{x}; \theta)$ , and the softmax turns them into probabilities, $p(y = c \mid \mathbf{x}; \theta) = \operatorname{softmax}_c(\mathbf{a})$ (Murphy, eqs. 1.8–1.9). The parameters are commonly fitted by minimizing the average negative log-likelihood of the training labels (§1.2.1.6), which for classes is the cross-entropy loss. Regression typically minimizes the squared error (§1.2.2). The loss function article gives the losses that other tasks use.

A predicted distribution still has to be turned into one answer, and Bayesian decision theory says which answer suits which loss (Murphy, §5.1):

\hat{y}_{0\text{–}1} = \arg\max_{y} p(y \mid \mathbf{x}), \qquad \hat{y}_{\ell_2} = \mathbb{E}[y \mid \mathbf{x}], \qquad \hat{y}_{\ell_1} = \operatorname{median}(y \mid \mathbf{x}).

The three rules minimize the expected 0–1, squared and absolute loss (§5.1.2.1, eq. 5.7; §5.1.5.1, eq. 5.34; §5.1.5.2); the first is the Bayes classifier of machine learning, and the loss function article contrasts the mean and the median. Training does not minimize the 0–1 loss itself: Torralba and colleagues (§9.7.3) note that doing so is a discrete, NP-hard problem, and that cross-entropy, which is continuous and differentiable, is commonly used instead.

Kinds of Output

Goodfellow and colleagues (§5.1.3) note that regression, classification and structured-output problems are traditionally the ones called supervised. Each kind of output comes with its own form of model output (its head):

Classification chooses one of $C$ mutually exclusive classes: a softmax over $C$ logits, trained with cross-entropy. Torralba and colleagues (§9.7.3) write the target as a one-hot code. Logistic regression, k-nearest neighbours, support vector machines and decision trees are classical classifiers.
Multi-label classification predicts a subset of tags, such as every object category present in an image. Murphy (§10.3) models it as $C$ independent binary outputs, $p(\mathbf{y} \mid \mathbf{x}) = \prod_c \operatorname{Ber}(y_c \mid \sigma(\mathbf{w}_c^\top \mathbf{x}))$ (eq. 10.56): one sigmoid per tag instead of one softmax.
Regression predicts real numbers: a depth, an angle, a keypoint coordinate. The head is a linear output trained with a squared or absolute error (L1 and L2 loss). Linear regression is the classical case, and bounding box regression regresses box offsets inside detectors.
Structured outputs are vectors or other structures whose elements are related to one another. Goodfellow and colleagues (§5.1.1) give pixel-wise segmentation and image captioning as examples. Semantic segmentation predicts a class label for each pixel (Murphy, §14.5.4), which amounts to a softmax at every pixel, and heatmap regression predicts a map per keypoint.

Where Labels Come From

Where people draw the labels, image annotation covers how guidelines, agreement checks and quality control decide how far they can be trusted. Murphy (§1.3) points out that people disagree on some labels, such as the moment an action starts in a video. He suggests modelling the distribution of labels a crowd of annotators produces instead of a single label (footnote 7). For wrong labels, called label noise, one remedy Murphy describes is a likelihood in which each label is, with some probability, generated uniformly at random (§10.4.1).

Several related settings relax the requirement of one exact label per example:

Weakly supervised learning: Murphy (§19.7) includes a distribution over labels for each example, a label for a whole bag of instances (multi-instance learning), and labels inferred from a database fact (distant supervision). Label smoothing deliberately replaces exact labels with soft ones.
Semi-supervised learning: adds many unlabelled images to a few labelled ones.
Self-supervised learning: computes both the inputs and the targets from unlabelled images, through pretext tasks.

Regress or Classify

Many continuous quantities in vision can be predicted either way. A regression head outputs the value directly. A classification head divides the range into bins, predicts a probability for each bin, and turns the probabilities back into a value. Published vision methods do both:

PointPillars learns a box’s heading as a softmax classification over discretized directions, alongside regressing the box (PointPillars).
SimCC classifies each keypoint coordinate into bins smaller than a pixel (heatmap regression).
Generalized Focal Loss predicts a softmax over discretized box-side distances and reads off their expectation (bounding box regression).

In our reading, two facts bear on the choice. The first is the decision rule. Decoding a bin classifier by its argmax returns a bin centre. If the true value is spread uniformly within each bin of width $w$ , the absolute error is then at least $w/4$ on average even for a perfect classifier, since the mean of $|u|$ for $u$ uniform on $[-w/2, w/2]$ is $w/4$ (our derivation). Decoding by the expectation or the median of the bin probabilities can return values between bin centres. The second fact is that narrower bins hold fewer training images each.

Evaluation in Outline

A supervised model is judged on labelled images it was not fitted to, using the arrangement described in training, validation and test sets and cross-validation. The metric matches the output: accuracy and calibration for classes, and mean squared error or absolute error for continuous targets. Overfitting, regularization and class imbalance each have their own articles.

Examples

Regressing or classifying a disk’s radius

The example predicts one continuous property, a disk’s radius, by regression and by classification into bins, and measures what the decision rule does to the error.

Data: $24 \times 24$ $24 \times 24$ images, each with one disk.
- Radius $r \sim U[3, 9]$ px, the disk placed wholly inside the image, contrast $c \sim U[0.5, 1]$ on a background of 0.
- Each pixel stores $c$ times the fraction of its area inside the disk ( $4 \times 4$ sub-samples), plus Gaussian noise of standard deviation 0.1.
- Each of seeds 0, 1 and 2 draws 2,000 training images, then 5,000 test images.
Regression: ridge regression (alpha=1.0) on standardized pixels predicts $r$ .
Classification: logistic regression (C=1.0, lbfgs, max_iter=5000; scikit-learn fits a binary model for $K = 2$ and a multinomial one above) predicts which of $K \in \{2, 4, 8, 16, 32\}$ equal-width bins of $[3, 9]$ holds $r$ . It is decoded by the argmax bin’s centre, by the expectation $\sum_k p_k c_k$ over bin centres $c_k$ , and by the median of the density that is uniform within each bin.
Measures: test mean absolute error (MAE) and root-mean-square error (RMSE) in pixels, beside the floor $w/4$ ( $w = 6/K$ ) and a perfect classifier’s MAE.

Checks (C) and hypotheses (H), written before the pilot ran (two bars after a planning script; see the process note):

C1: a label-free check that the labels can be learned. The radius from the thresholded noise-free area, $\hat{r} = \sqrt{\#\{\text{pixels} > c/2\} / \pi}$ , is within 0.4 px of $r$ .
C2–C5: the data ranges and bin counts, the perfect classifier within three standard errors of $w/4$ , identical decoders on one-hot probabilities, and convergence.
H1: argmax MAE stays above $w/4$ at every $K$ .
H2, H2b: expectation decoding is no worse than argmax, in MAE and in RMSE.
H2c: median decoding is no worse than expectation decoding in MAE.
H3: MAE stops improving once bins hold few training images, tested as MAE at $K = 32$ being no lower than at $K = 16$ , for each decoder.

Ridge has no hypothesis. The code uses NumPy 2.3.5 and scikit-learn 1.7.2 and runs in about 8 seconds.

import os
os.environ["OMP_NUM_THREADS"] = "1"           # one BLAS thread, before NumPy is imported
os.environ["OPENBLAS_NUM_THREADS"] = "1"
os.environ["MKL_NUM_THREADS"] = "1"
import sys
import numpy as np
from sklearn.linear_model import LogisticRegression, Ridge
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

S, RMIN, RMAX, NOISE, SS = 24, 3.0, 9.0, 0.1, 4   # image size, radius range (px), noise sd, supersampling
N_TRAIN, N_TEST, KS, MAX_ITER = 2000, 5000, [2, 4, 8, 16, 32], 5000
SEEDS = [int(s) for s in sys.argv[1:]] or [0, 1, 2]


def make_images(rng, n):
    """One disk per image: radius r ~ U[3, 9], centre so the disk lies inside, contrast c ~ U[0.5, 1].
    Each pixel stores c times the fraction of its area inside the disk (4 x 4 sub-samples)."""
    r = rng.uniform(RMIN, RMAX, n)
    cx, cy = rng.uniform(r, S - r), rng.uniform(r, S - r)
    c = rng.uniform(0.5, 1.0, n)
    sub = (np.arange(S * SS) + 0.5) / SS
    cover = np.empty((n, S, S))
    for i in range(0, n, 250):
        j = slice(i, i + 250)
        inside = ((sub[None, None, :] - cx[j, None, None]) ** 2
                  + (sub[None, :, None] - cy[j, None, None]) ** 2) <= r[j, None, None] ** 2
        cover[j] = inside.reshape(-1, S, SS, S, SS).mean(axis=(2, 4))
    clean = c[:, None, None] * cover
    noisy = clean + rng.normal(0.0, NOISE, clean.shape)
    margin = min((cx - r).min(), (S - cx - r).min(), (cy - r).min(), (S - cy - r).min())
    return noisy.reshape(n, -1), clean, r, c, margin


def decode(P, centres, w):
    """Point estimates of r from bin probabilities P (n x K): argmax bin centre, expectation,
    and the median of the density that is uniform within each bin."""
    arg = centres[np.argmax(P, axis=1)]
    exp = P @ centres
    cum = np.cumsum(P, axis=1)
    k = np.argmax(cum >= 0.5, axis=1)
    below = np.where(k > 0, cum[np.arange(len(k)), k - 1], 0.0)
    med = RMIN + (k + (0.5 - below) / P[np.arange(len(k)), k]) * w
    return arg, exp, med


res = {}                                   # (seed, K) -> (MAE arg, exp, med, RMSE arg, exp, med)
c1_max, c3_ok, c4_max, iters = 0.0, True, 0.0, []
for seed in SEEDS:
    rng = np.random.default_rng(seed)
    Xtr, Ctr, rtr, ctr, mtr = make_images(rng, N_TRAIN)
    Xte, Cte, rte, cte, mte = make_images(rng, N_TEST)
    for C, r, c in [(Ctr, rtr, ctr), (Cte, rte, cte)]:   # C1: radius from thresholded noise-free area
        rhat = np.sqrt((C > c[:, None, None] / 2).sum(axis=(1, 2)) / np.pi)
        c1_max = max(c1_max, np.abs(rhat - r).max())
    rr, cc = np.r_[rtr, rte], np.r_[ctr, cte]
    print(f"seed {seed}: r in [{rr.min():.4f}, {rr.max():.4f}], c in [{cc.min():.4f}, {cc.max():.4f}], "
          f"min disk-to-frame margin {min(mtr, mte):.4f} px")
    ridge = make_pipeline(StandardScaler(), Ridge(alpha=1.0)).fit(Xtr, rtr)
    pr = ridge.predict(Xte)
    print(f"  ridge regression:  MAE {np.abs(pr - rte).mean():.4f}  RMSE {np.sqrt(((pr - rte) ** 2).mean()):.4f}")
    print("   K  w/4      perfect  min/bin |  MAE argmax  expect  median | RMSE argmax  expect  median")
    for K in KS:
        w = (RMAX - RMIN) / K
        centres = RMIN + (np.arange(K) + 0.5) * w
        ytr = np.clip(((rtr - RMIN) // w).astype(int), 0, K - 1)
        yte = np.clip(((rte - RMIN) // w).astype(int), 0, K - 1)
        perfect = np.abs(centres[yte] - rte).mean()           # C3: a perfect classifier's MAE
        c3_ok &= abs(perfect - w / 4) <= 3 * w / np.sqrt(48 * N_TEST)
        a0, e0, m0 = decode(np.eye(K)[yte], centres, w)        # C4: one-hot probabilities
        c4_max = max(c4_max, np.abs(e0 - a0).max(), np.abs(m0 - a0).max())
        clf = make_pipeline(StandardScaler(), LogisticRegression(C=1.0, max_iter=MAX_ITER)).fit(Xtr, ytr)
        iters.append(clf[-1].n_iter_.max())
        P = np.zeros((N_TEST, K))
        P[:, clf[-1].classes_] = clf.predict_proba(Xte)
        est = decode(P, centres, w)
        res[seed, K] = [np.abs(v - rte).mean() for v in est] + [np.sqrt(((v - rte) ** 2).mean()) for v in est]
        m = res[seed, K]
        print(f"  {K:2d}  {w/4:.5f}  {perfect:.5f}  {np.bincount(ytr, minlength=K).min():4d}    |"
              f"  {m[0]:.4f}     {m[1]:.4f}  {m[2]:.4f} |  {m[3]:.4f}     {m[4]:.4f}  {m[5]:.4f}")

d = lambda i, j: max(res[k][i] - res[k][j] for k in res)
h1 = min(res[s, K][0] - (RMAX - RMIN) / K / 4 for s, K in res)
print(f"C1 max |r_hat - r| = {c1_max:.4f} px (tolerance 0.4)   C3 perfect within 3 SE of w/4: {c3_ok}")
print(f"C4 max one-hot decoding difference = {c4_max:.1e}   C5 max iterations {max(iters)} of {MAX_ITER}")
print(f"H1 min (argmax MAE - w/4)      = {h1:+.5f}")
print(f"H2 max (expect - argmax), MAE  = {d(1, 0):+.5f}   H2b same, RMSE = {d(4, 3):+.5f}")
print(f"H2c max (median - expect), MAE = {d(2, 1):+.5f}")
for i, name in enumerate(["argmax", "expect", "median"]):
    print(f"H3 {name}: MAE(K=32) - MAE(K=16) per seed: "
          + "  ".join(f"{res[s, 32][i] - res[s, 16][i]:+.5f}" for s in SEEDS))

seed 0: r in [3.0002, 8.9970], c in [0.5001, 1.0000], min disk-to-frame margin 0.0001 px
  ridge regression:  MAE 0.6523  RMSE 0.8123
   K  w/4      perfect  min/bin |  MAE argmax  expect  median | RMSE argmax  expect  median
   2  0.75000  0.75241   993    |  0.8701     0.8099  0.8237 |  1.0582     0.9804  0.9987
   4  0.37500  0.37842   481    |  0.6422     0.5839  0.5986 |  0.8530     0.7701  0.7940
   8  0.18750  0.18622   237    |  0.5830     0.5229  0.5477 |  0.7851     0.6994  0.7377
  16  0.09375  0.09383   110    |  0.5624     0.4873  0.5246 |  0.7555     0.6477  0.7044
  32  0.04688  0.04675    50    |  0.5704     0.4637  0.5145 |  0.7629     0.6060  0.6865
seed 1: r in [3.0002, 8.9987], c in [0.5001, 0.9999], min disk-to-frame margin 0.0014 px
  ridge regression:  MAE 0.6468  RMSE 0.8048
   K  w/4      perfect  min/bin |  MAE argmax  expect  median | RMSE argmax  expect  median
   2  0.75000  0.75376   998    |  0.8711     0.8137  0.8283 |  1.0557     0.9802  0.9982
   4  0.37500  0.37247   491    |  0.6403     0.5867  0.6016 |  0.8451     0.7662  0.7898
   8  0.18750  0.18976   232    |  0.5800     0.5216  0.5434 |  0.7882     0.7056  0.7409
  16  0.09375  0.09411   105    |  0.5732     0.4951  0.5337 |  0.7780     0.6691  0.7251
  32  0.04688  0.04714    47    |  0.5721     0.4678  0.5135 |  0.7728     0.6153  0.6898
seed 2: r in [3.0055, 8.9997], c in [0.5001, 1.0000], min disk-to-frame margin 0.0004 px
  ridge regression:  MAE 0.6465  RMSE 0.8050
   K  w/4      perfect  min/bin |  MAE argmax  expect  median | RMSE argmax  expect  median
   2  0.75000  0.75382   998    |  0.8619     0.8056  0.8195 |  1.0408     0.9644  0.9819
   4  0.37500  0.37254   472    |  0.6340     0.5804  0.5936 |  0.8490     0.7695  0.7928
   8  0.18750  0.18626   230    |  0.5800     0.5272  0.5469 |  0.7856     0.7064  0.7406
  16  0.09375  0.09402   103    |  0.5701     0.4915  0.5297 |  0.7648     0.6518  0.7098
  32  0.04688  0.04641    46    |  0.5717     0.4696  0.5180 |  0.7597     0.6093  0.6826
C1 max |r_hat - r| = 0.3219 px (tolerance 0.4)   C3 perfect within 3 SE of w/4: True
C4 max one-hot decoding difference = 0.0e+00   C5 max iterations 310 of 5000
H1 min (argmax MAE - w/4)      = +0.11194
H2 max (expect - argmax), MAE  = -0.05283   H2b same, RMSE = -0.07550
H2c max (median - expect), MAE = +0.05074
H3 argmax: MAE(K=32) - MAE(K=16) per seed: +0.00798  -0.00102  +0.00158
H3 expect: MAE(K=32) - MAE(K=16) per seed: -0.02358  -0.02723  -0.02191
H3 median: MAE(K=32) - MAE(K=16) per seed: -0.01011  -0.02016  -0.01174

All checks held on every seed:

C1: the largest error was 0.3219 px.
C2: every seed’s ranges and margin lie within their bounds, and the smallest bins held 50, 47 and 46 training images on seeds 0, 1 and 2.
C3 held at every $K$ , C4 printed 0, C5 used at most 310 iterations, and two runs printed identical output.

The hypothesis tests:

H1 held at every $K$ and seed, with argmax at least 0.11194 px above its floor. At $K = 32$ its MAE was about twelve times the floor of 0.047 on each seed. An image put in its true bin is at most $w/2 = 0.094$ px off, so most of that error came from images put in a wrong bin (our arithmetic).
H2 and H2b held at every $K$ and seed: the expectation had lower MAE and RMSE than the argmax in every row.
H2c failed at every $K$ on every seed, as it had already failed on the planning script: the median had higher MAE than the expectation in every row.

H3 is an illustration, not a test, because its threshold was chosen after planning output:

Argmax: H3 held on seed 0 (+0.00798) and seed 2 (+0.00158), and failed on seed 1 (−0.00102).
Expectation and median: H3 failed on every seed (expectation −0.02358, −0.02723, −0.02191; median −0.01011, −0.02016, −0.01174 on seeds 0, 1 and 2): their MAE was lower at $K = 32$ than at 16.

Ridge, which has no hypothesis, had a lower MAE than every classifier decoding at $K = 2$ and a higher one than every decoding from $K = 4$ on, on each seed.

The heatmap regression example decodes a known keypoint from an ideal map with no model trained; this one trains a classifier on one scalar per image.

Process note. Six departures from the batch plan were fixed before the pilot:

A median decoder was added, because the median is the decision rule for the planned absolute error (Murphy, §5.1.5.2).
RMSE was added, because the expectation is the decision rule for squared error (§5.1.5.1).
The open settings were fixed: the contrast range, background, disks inside the frame, supersampling, standardized pixels for both models, the library defaults for C and alpha, max_iter=5000, and the draw order.
The label-free check became the thresholded area.
Checks C2–C5 were added.
H3 was given its threshold.

Two bars were set after a planning script ran the pilot’s set-up on seed 0: C1’s tolerance of 0.4 px, chosen after 0.3219 printed, and H3’s threshold $K = 16$ , chosen where argmax MAE was lowest. The pilot ran seed 0 alone. H3 for argmax held there and failed on seed 1 in the run shown; every other outcome was the same. The code was not changed after the pilot.

Common Misconceptions

“Classifying a continuous quantity throws away precision.” Argmax decoding returns only bin centres, which sets the $w/4$ floor. The expectation can fall between bin centres, and in the example it had a lower MAE than argmax at every bin count and seed.
“The softmax probability is the answer.” Probabilities still have to be turned into an answer by a decision rule, and the rule depends on the loss (Murphy, §5.1), as in the Formal Definition above.
“The labels are the ground truth.” Labels made by people carry disagreement and errors (Murphy, §1.3, §10.4).

Where It Is Used

Recognition and dense prediction: the tasks named at the top and human pose estimation learn from annotated images; segmentation and depth estimation predict an output at every pixel.
Starting points: transfer learning fine-tunes, on a labelled task, a network pretrained on other data. Data augmentation enlarges a labelled set with transformed copies whose labels stay correct.