Image Classification

Image classification assigns a category label to a whole image: “cat”, “the digit 7”, “beach scene”. It is the simplest recognition task to state, since the output is a label rather than a location or a mask, and the benchmark task on which deep networks overtook hand-crafted features. It is also called object recognition or visual categorization, although Csurka et al. (2004) reserved “recognition” for identifying a particular object instance (this cup) and “categorization” for placing any cup in one class; classification in the modern sense is the second.

Problem Definition

Given a fixed set of $K$ classes and training images $\mathbf{x}_i$ with labels $y_i \in \{1, \dots, K\}$, learn a function that maps a new image to a probability distribution over the classes,

$$f(\mathbf{x}) = \mathbf{p} = (p_1, \dots, p_K), \qquad p_k \ge 0, \quad \sum_{k=1}^{K} p_k = 1,$$

and predict $\hat{y} = \arg\max_k p_k$. In practice $f$ produces $K$ logits that a softmax turns into probabilities, and training minimizes the cross-entropy loss. The standard assumptions are that the classes are fixed in advance, that training labels are correct, and that test images come from the training distribution; the last is the one most often violated.

Input and Output

The input is an $H \times W \times C$ image array, usually resized or cropped to the size the model expects (for example $224 \times 224$ for many ImageNet models) and normalized per channel. The output is a $K$-vector of scores; the label is its argmax, or a ranked list of the top few classes. The output says nothing about where the object is, which separates classification from object detection, which adds a bounding box per object, and from segmentation, which labels every pixel.

Variants

• Multi-class (single-label): exactly one of $K$ mutually exclusive classes per image, with a softmax over the classes; the default meaning of the term.
• Multi-label: any subset of the $K$ labels can be present, as in a street photo containing a car, a person and a bicycle. The target is a multi-hot vector $\mathbf{t} \in \{0, 1\}^K$; each label gets its own sigmoid and yes/no decision, trained with binary cross-entropy, so the number of predicted labels varies from image to image.
• Binary: two classes, such as defective versus good parts, often written with one sigmoid output.
• Fine-grained classification: distinguishing subordinate categories such as bird species or car models, where differences between classes are small and local while variation within a class (pose, lighting) can be larger.
• Scene classification: labeling the place depicted (kitchen, forest) from the whole layout rather than one object.

Challenges

The same category produces very different pixels under changes of viewpoint and pose, scale, occlusion and illumination; categories show large intra-class variation (“chair” covers stools and armchairs); and the object may fill only a small part of an image amid background clutter. Csurka et al. (2004) listed most of these. Other problems lie in the data:

• Dataset bias. Torralba and Efros (2011) trained a linear SVM to tell which of twelve popular recognition datasets an image came from, and reached 39% accuracy against 8% chance. Their “car” classifiers lost on average 48% of their average precision when tested on datasets other than the one they were trained on.
• Distribution shift. Recht et al. (2019) collected new CIFAR-10 and ImageNet test sets following the original procedures and found accuracy drops of 3–15 and 11–14 percentage points across a broad range of models; the best ImageNet model fell from 83% to 72% top-1 accuracy. Model rankings were almost unchanged, and the authors attributed the drop not to tuning on the test set but to the new images being slightly harder.
• Shortcut learning. Geirhos et al. (2020) describe decision rules that score well on benchmarks but fail under harder test conditions, such as recognizing cows by the grassland around them.
• Texture bias. On images in which shape and texture cues conflict, such as a cat silhouette filled with elephant skin, Geirhos et al. (2019) found ImageNet-trained CNNs strongly biased toward the texture, unlike human observers. Training ResNet-50 on a stylized ImageNet shifted it toward shape and made it more robust to many image distortions.

Approaches

Hand-crafted features and linear classifiers

Before 2012 a classifier had three separately designed stages: extract local descriptors, encode them into one fixed-length vector per image, and train a classifier on those vectors. The descriptors were gradient histograms such as SIFT (see feature descriptors) or HOG, which Dalal and Triggs (2005) introduced for human detection.

Csurka et al. (2004) carried the bag-of-words model of text categorization over to images. They detected Harris-affine regions, described each with SIFT, clustered descriptors with k-means into a vocabulary of $k = 1000$ “visual words”, and represented each image by its histogram of word counts. On their seven-class dataset of 1,776 images, a linear SVM lowered the overall error rate from 28% for Naïve Bayes to 15%, without using the geometric layout of the words.

Perronnin and Dance (2007) applied the Fisher kernel to visual vocabularies, giving what became known as the Fisher vector: model the descriptors with a Gaussian mixture and describe an image by the gradient of their log-likelihood with respect to the mixture parameters, which records how descriptors deviate from each Gaussian, not only how many fall near it. Perronnin, Sánchez and Mensink (2010) added power and L2 normalization and spatial pyramids, raising average precision on PASCAL VOC 2007 from 47.9% to 58.3% with SIFT features and linear classifiers only; linear classifiers let it scale to hundreds of thousands of training images. Fisher-vector systems were the strongest competitors that AlexNet faced on ImageNet.

Deep networks

Deep learning replaced the three stages with one network trained end to end from pixels, learning the features instead of designing them. AlexNet (Krizhevsky et al., 2012), a convolutional neural network trained on two GPUs, won ILSVRC-2012 with a top-5 test error of 15.3% against 26.2% for the second-best entry. Later gains came largely from depth: VGG (Simonyan & Zisserman, 2015) stacked uniform $3 \times 3$ convolutions to 16–19 weight layers; GoogLeNet (Szegedy et al., 2015) reached 22 layers with Inception modules at a fixed compute budget; and ResNet (He et al., 2016) used residual connections to train 152 layers, winning ILSVRC 2015 with a 3.57% top-5 test error for an ensemble.

The Vision Transformer (ViT) of Dosovitskiy et al. (2021) cut an image into patches and fed them to a standard transformer encoder. Without the locality and translation equivariance built into convolutions, it generalized worse on small training sets, but the authors found that pre-training on 14M–300M images outweighed that missing inductive bias.

Training follows the workflow for neural networks, with regularization such as data augmentation and weight decay. For small target datasets the usual practice is transfer learning: start from a network pre-trained on a large dataset such as ImageNet and fine-tune it, or train only a new final layer on its features.

Example: classifying digit images

The NumPy code trains the linear softmax classifier of the cross-entropy article on the 64 raw pixels of scikit-learn’s $8 \times 8$ digit images: each class has one 64-value weight template, and its logit is the dot product of the image with that template. It then builds a confusion matrix, whose entry $(i, j)$ counts test images of digit $i$ predicted as $j$.

import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

digits = load_digits()                                   # 1,797 grayscale 8x8 images, labels 0..9
X = digits.images.reshape(-1, 64) / 16.0                 # each image as 64 pixel values in [0, 1]
y = digits.target
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=9, stratify=y)

K = 10
W, b = np.zeros((64, K)), np.zeros(K)                    # one weight template per class
Y = np.eye(K)[y_tr]                                      # one-hot labels

def softmax(Z):
    Z = Z - Z.max(axis=1, keepdims=True)
    E = np.exp(Z)
    return E / E.sum(axis=1, keepdims=True)

lam = 1e-3                                               # small L2 penalty on W
for step in range(500):                                  # full-batch gradient descent on cross-entropy
    P = softmax(X_tr @ W + b)
    G = (P - Y) / len(X_tr)                              # gradient w.r.t. the logits
    W -= 2.0 * (X_tr.T @ G + lam * W)
    b -= 2.0 * G.sum(axis=0)

P_te = softmax(X_te @ W + b)
pred = P_te.argmax(axis=1)
print(f"linear softmax on raw pixels: test accuracy {np.mean(pred == y_te):.3f} "
      f"({np.sum(pred != y_te)} errors in {len(y_te)})")
top2 = np.argsort(-P_te, axis=1)[:, :2]
print(f"top-2 accuracy {np.mean((top2 == y_te[:, None]).any(axis=1)):.3f}")

C = np.zeros((K, K), int)                                # rows: true digit, columns: predicted digit
np.add.at(C, (y_te, pred), 1)
print("confusion matrix (rows true 0-9, columns predicted 0-9):")
print(C)
off = [(C[i, j], i, j) for i in range(K) for j in range(K) if i != j and C[i, j] > 0]
print("errors, most frequent first:", ", ".join(f"{i}->{j} x{n}" for n, i, j in sorted(off, reverse=True)))

Output:

linear softmax on raw pixels: test accuracy 0.967 (12 errors in 360)
top-2 accuracy 0.986
confusion matrix (rows true 0-9, columns predicted 0-9):
[[36  0  0  0  0  0  0  0  0  0]
 [ 0 37  0  0  0  0  0  0  0  0]
 [ 0  0 35  0  0  0  0  0  0  0]
 [ 0  0  1 35  0  1  0  0  0  0]
 [ 0  0  0  0 35  0  0  0  0  1]
 [ 0  0  0  0  0 34  0  0  0  2]
 [ 0  0  0  0  0  0 36  0  0  0]
 [ 0  0  0  0  0  0  0 35  1  0]
 [ 0  4  0  0  0  0  0  0 31  0]
 [ 0  1  0  1  0  0  0  0  0 34]]
errors, most frequent first: 8->1 x4, 5->9 x2, 9->3 x1, 9->1 x1, 7->8 x1, 4->9 x1, 3->5 x1, 3->2 x1

The linear model classifies 96.7% of the 360 test digits correctly, and the confusion matrix shows its errors are not spread evenly: a third are 8s read as 1s, followed by 5s read as 9s. Over ten train/test splits it reached 95.8–97.5%, and 8→1 was the most frequent error in five of them. In our reading, one template per class cannot cover the different ways a digit is written.

The PyTorch code (run with PyTorch 2.14.1) trains a small CNN, like the one in the CNN article, on the same split, then a multi-label version: two digits side by side form one $8 \times 16$ image labeled with the set of digits it contains (a single label when both are the same), and the network has one sigmoid per digit and a binary cross-entropy loss.

import numpy as np
import torch
from torch import nn
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

SEED = 2
torch.manual_seed(SEED)
digits = load_digits()
X = torch.tensor(digits.images / 16.0, dtype=torch.float32).unsqueeze(1)   # (1797, 1, 8, 8)
y = torch.tensor(digits.target)
i_tr, i_te = train_test_split(np.arange(len(y)), test_size=0.2, random_state=9, stratify=digits.target)
X_tr, y_tr, X_te, y_te = X[i_tr], y[i_tr], X[i_te], y[i_te]

def cnn(width):                     # input 1 x 8 x width -> 10 logits
    return nn.Sequential(
        nn.Conv2d(1, 32, 3, padding=1), nn.BatchNorm2d(32), nn.ReLU(), nn.MaxPool2d(2),
        nn.Conv2d(32, 64, 3, padding=1), nn.BatchNorm2d(64), nn.ReLU(), nn.MaxPool2d(2),
        nn.Flatten(), nn.Linear(64 * 2 * (width // 4), 10))

def train(net, X, T, loss_fn, epochs):
    opt = torch.optim.Adam(net.parameters(), lr=3e-3)
    for _ in range(epochs):
        net.train()
        for batch in torch.randperm(len(X)).split(64):
            loss = loss_fn(net(X[batch]), T[batch])
            opt.zero_grad(); loss.backward(); opt.step()
    return net.eval()

# 1. Multi-class: one digit per image, softmax over 10 classes (cross-entropy on logits).
net = train(cnn(8), X_tr, y_tr, nn.functional.cross_entropy, epochs=30)
with torch.no_grad():
    pred = net(X_te).argmax(1)
C = torch.zeros(10, 10, dtype=torch.long)
C.index_put_((y_te, pred), torch.ones_like(y_te), accumulate=True)
print(f"CNN: test accuracy {(pred == y_te).float().mean():.3f} ({(pred != y_te).sum()} errors in {len(y_te)})")
print(C.numpy())

# 2. Multi-label: two digits side by side in one 8x16 image; the target is the set of
#    digits present (multi-hot over 10 labels), one sigmoid per label, BCE loss.
def pairs(Xs, ys, n, gen):
    a = torch.randint(len(ys), (n,), generator=gen)
    b = torch.randint(len(ys), (n,), generator=gen)
    img = torch.cat([Xs[a], Xs[b]], dim=3)                  # (n, 1, 8, 16)
    T = torch.zeros(n, 10)
    T[torch.arange(n), ys[a]] = 1
    T[torch.arange(n), ys[b]] = 1                           # a repeated digit gives one label
    return img, T

gen = torch.Generator().manual_seed(SEED)
P_tr, T_tr = pairs(X_tr, y_tr, 6000, gen)                   # pairs of training digits only
P_te, T_te = pairs(X_te, y_te, 2000, gen)                   # pairs of test digits only
ml = train(cnn(16), P_tr, T_tr, nn.functional.binary_cross_entropy_with_logits, epochs=10)
with torch.no_grad():
    present = torch.sigmoid(ml(P_te)) > 0.5                 # each label decided on its own
truth = T_te.bool()
per_label = (present == truth).float().mean(0)
print("per-label accuracy:", " ".join(f"{d}:{a:.3f}" for d, a in enumerate(per_label)))
print(f"mean per-label accuracy {per_label.mean():.3f} "
      f"(predicting 'absent' everywhere: {(~truth).float().mean():.3f})")
print(f"label set exactly right {(present == truth).all(1).float().mean():.3f}; "
      f"same digit twice: {int((T_te.sum(1) == 1).sum())} images, "
      f"one label predicted for {(present[T_te.sum(1) == 1].sum(1) == 1).float().mean():.3f}")

Output:

CNN: test accuracy 0.989 (4 errors in 360)
[[36  0  0  0  0  0  0  0  0  0]
 [ 0 37  0  0  0  0  0  0  0  0]
 [ 0  0 35  0  0  0  0  0  0  0]
 [ 0  0  0 37  0  0  0  0  0  0]
 [ 0  0  0  0 36  0  0  0  0  0]
 [ 0  0  0  1  0 34  0  0  0  1]
 [ 0  0  0  0  0  0 36  0  0  0]
 [ 0  0  0  0  0  0  0 36  0  0]
 [ 0  1  0  0  0  0  0  0 34  0]
 [ 0  0  0  0  0  0  0  0  1 35]]
per-label accuracy: 0:0.997 1:0.990 2:0.998 3:0.996 4:0.993 5:0.987 6:0.999 7:0.998 8:0.974 9:0.986
mean per-label accuracy 0.992 (predicting 'absent' everywhere: 0.811)
label set exactly right 0.931; same digit twice: 210 images, one label predicted for 0.962

The CNN cuts the errors from 12 to 4 (98.9%; over ten seeds 98.1–99.2%), and only one 8→1 confusion remains. In the multi-label task, mean per-label accuracy is 99.2%, but each label is absent from about 81% of images, so answering “absent” everywhere already scores 81.1%. The stricter measure, the whole label set exactly right, is 93.1% (over ten seeds 92.1–94.7%), and the digit 8 is again the hardest label. Because each sigmoid decides on its own, the network predicts a single label for 96.2% of the images showing the same digit twice; a softmax returning its top two classes always names two.

Datasets and Benchmarks

• MNIST: handwritten digits centered in $28 \times 28$ images, 60,000 for training and 10,000 for testing (LeCun et al., 1998); now mostly a sanity check.
• CIFAR-10 and CIFAR-100: 60,000 color images of $32 \times 32$ pixels, in 10 classes, or in 100 classes grouped into 20 superclasses, with 50,000 training and 10,000 test images (Krizhevsky, 2009).
• ImageNet: web images organized by the WordNet hierarchy (Deng et al., 2009). Its challenge subset, ILSVRC, has 1,000 classes and about 1.2 million training images (Russakovsky et al., 2015).
• Places: about 10 million scene photographs labeled with scene categories, for scene classification (Zhou et al., 2018).

Evaluation Metrics

Accuracy (top-1), the fraction of test images whose predicted label is correct, is the basic measure; its complement is the error rate. Top-5 accuracy counts a prediction as correct if the true label is among the five highest-scoring classes; ILSVRC used it because each image carries only one label even when it contains several objects (Russakovsky et al., 2015), as the accuracy article explains.

With imbalanced classes, overall accuracy can hide a class that is never recognized; per-class accuracy and the confusion matrix show where errors fall. Multi-label tasks are usually evaluated per label with precision, recall and average precision, since per-label accuracy is inflated by absent labels, as in the example. When the probabilities themselves are used, calibration matters: images predicted with 80% confidence should be right about 80% of the time. The cross-entropy loss article covers the overconfidence found in modern networks and temperature scaling.

Applications

Classification serves wherever one decision per image suffices: tagging photo collections, content moderation, product recognition, visual inspection on production lines, species identification, and screening in medical imaging. It also matters as pre-training: networks trained on ImageNet classification have long supplied the backbones of detectors, segmentation models, trackers and depth estimators.

Open Problems

• Robustness to distribution shift. Accuracy on a held-out split of the same dataset overstates accuracy on new data.
• Shortcuts and spurious correlations. A high benchmark score does not show that a model relies on the object rather than its context or texture.
• Fixed label sets. A classifier can only choose among the classes it was trained on, and has no built-in way to say that an image belongs to none of them.
• Label quality. Single labels for images that contain several objects, and inconsistent labels for ambiguous categories, limit what benchmark numbers mean.