Accuracy

Accuracy is the fraction of test examples that a classifier labels correctly: the share of images in image classification, or of pixels in semantic segmentation, whose predicted class matches the ground truth. Its complement, the error rate, is the fraction labeled wrongly. It is the simplest classification metric and the usual headline number on classification benchmarks, but it weights every example equally, which misleads when classes are imbalanced, and on a finite test set its sampling noise can exceed the differences being reported.

Formula

For $N$ test images with true labels $y_i$ and predicted labels $\hat{y}_i$,

$$\text{accuracy} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}[\hat{y}_i = y_i], \qquad \text{error rate} = 1 - \text{accuracy},$$

where $\mathbf{1}[\cdot]$ is 1 when its argument is true and 0 otherwise. For a classifier that outputs a score per class, $\hat{y}_i$ is the highest-scoring class, which is why this is also called top-1 accuracy. Top-$k$ accuracy counts an image as correct when the true label is among the $k$ highest-scoring classes $T_k(\mathbf{x}_i)$:

$$\text{top-}k\text{ accuracy} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}[y_i \in T_k(\mathbf{x}_i)].$$

With $K$ classes, $N_c$ test images of class $c$, and $\text{acc}_c$ the fraction of them classified correctly, balanced accuracy (mean per-class accuracy) is

$$\text{balanced accuracy} = \frac{1}{K} \sum_{c=1}^{K} \text{acc}_c, \qquad \text{acc}_c = \frac{1}{N_c} \sum_{i:\, y_i = c} \mathbf{1}[\hat{y}_i = c],$$

while plain accuracy is the same per-class accuracies weighted by class frequency, $\sum_c (N_c / N)\, \text{acc}_c$. For two classes, balanced accuracy is the mean of sensitivity and specificity.

Intuition

Accuracy answers one question: how likely is the classifier to label a randomly drawn test image correctly? Every image counts once, and every mistake costs the same, whether a cat is called a dog or a container ship. That makes it easy to explain, and sensible when classes are about equally frequent and errors matter equally. The variants change who counts: top-$k$ forgives a correct label ranked just below the first, and balanced accuracy gives each class, not each image, an equal vote.

Interpretation

• Range: $[0, 1]$, often reported as a percentage; higher is better. Error rates, such as ImageNet’s top-5 error, are reported the other way round.
• Chance level: guessing uniformly at random gives $1/K$, but always predicting the most frequent class gives that class’s share of the test set, which can be far higher. Accuracy means little without that baseline.
• Uncertainty: the number of correct images on a test set of $N$ independent images is binomial, so the standard error of an accuracy $a$ is

$$\text{SE} = \sqrt{\frac{a(1 - a)}{N}}.$$

At 90% accuracy on 10,000 test images, the standard error is 0.3 percentage points and an approximate 95% interval ($\pm 1.96\,\text{SE}$) is about $\pm 0.6$ points. On 360 images, as in the example below, the 95% interval at 97% accuracy spans about 3.4 points. The ILSVRC organizers instead bootstrapped the test images; on the 100,000-image classification test set, the 2014 winner’s top-5 error of 6.66% had a 99.9% interval of 6.40–6.92% (Russakovsky et al., 2015). By our calculation, the binomial formula gives the same interval.

Comparing two classifiers. When both models are scored on the same images, their errors are correlated, and the useful information is in the images where they disagree. McNemar’s test counts $n_{01}$, the images only the first model gets right, and $n_{10}$, those only the second gets right; under the hypothesis of equal error rates these should be equal, and

$$\chi^2 = \frac{(|n_{01} - n_{10}| - 1)^2}{n_{01} + n_{10}}$$

is approximately $\chi^2$-distributed with one degree of freedom, with an exact binomial version for small counts. In Dietterich’s (1998) simulations it had a low rate of false detections (Type I error), and he recommended it for learning algorithms that can be run only once; he also noted that it ignores variation due to the choice of training set.

When to Use It

Use top-1 accuracy for single-label classification when classes are roughly balanced and errors have similar costs, as on the standard digit, CIFAR, and ImageNet benchmarks. Report it with the test-set size or an interval, so readers can judge whether a difference is real. When classes are imbalanced, or the rare class is the one that matters (defects on a production line, lesions in medical images), report balanced accuracy or per-class accuracy, and the confusion matrix, whose entry $(i, j)$ counts examples of class $i$ predicted as class $j$. Precision and recall summarize one class of that matrix at a time.

Failure Cases

• Class imbalance. If 90% of test images show the background class, a model that always answers “background” scores 90% accuracy while never finding the class of interest. Its balanced accuracy is 50%, the chance level for two classes. Brodersen et al. (2010) argued for balanced accuracy because plain accuracy is optimistic when a classifier biased toward the majority class is tested on an imbalanced test set, and, for two classes, derived its posterior distribution so that it can be reported with an interval.
• Accuracy is not a training objective. Accuracy changes only when a prediction crosses a decision boundary, so its gradient is zero almost everywhere, and networks are trained on a differentiable surrogate such as the cross-entropy loss (see loss functions). The two can move independently: in the example, cross-entropy falls by about a quarter as correct predictions grow more confident, while accuracy does not change. Nor does accuracy say whether the predicted probabilities are trustworthy.
• Multi-label outputs. When each image can carry several labels, per-label accuracy is inflated by the many labels that are correctly absent, as the image classification article shows; precision, recall, and average precision are used instead.

Variants

Top-$k$ accuracy and the ImageNet challenge. Each ILSVRC image has a single class label, even when it contains several objects, so an image labeled “strawberry” might also contain an apple. The challenge let algorithms return up to five labels and counted an image as correct if any of them matched the ground truth, so that naming an unlabeled object present in the image was not penalized (Russakovsky et al., 2015). The challenge also evaluated top-1 error and a hierarchical error that charged less for confusing nearby classes in the ImageNet hierarchy, such as two dog breeds; since all three ranked the entries in the same order, ILSVRC used only top-5 error from 2012 onward.

Pixel accuracy and mean class accuracy. Semantic segmentation classifies every pixel, and the same two averages apply to pixels. With $n_{ij}$ the number of pixels of class $i$ predicted as class $j$, $t_i = \sum_j n_{ij}$ the number of pixels of class $i$, and $n_{\text{cl}}$ classes, Long, Shelhamer and Darrell (2015) reported four measures that they took from existing segmentation and scene-parsing evaluations:

$$\text{pixel accuracy} = \frac{\sum_i n_{ii}}{\sum_i t_i}, \qquad \text{mean accuracy} = \frac{1}{n_{\text{cl}}} \sum_i \frac{n_{ii}}{t_i},$$ $$\text{mean IU} = \frac{1}{n_{\text{cl}}} \sum_i \frac{n_{ii}}{t_i + \sum_j n_{ji} - n_{ii}}, \qquad \text{frequency-weighted IU} = \frac{1}{\sum_k t_k} \sum_i \frac{t_i\, n_{ii}}{t_i + \sum_j n_{ji} - n_{ii}}.$$

Pixel accuracy is dominated by large classes such as sky, road, or background, so a model can miss every small object and still score highly. Mean accuracy fixes that but ignores false positives: labeling every pixel as one class gives that class a perfect score, which is why segmentation benchmarks rank by mean intersection over union, which penalizes both misses and false alarms.

Example

The first script trains a linear softmax classifier on scikit-learn’s $8 \times 8$ digit images, printing the test cross-entropy during training, and computes top-1 and top-3 accuracy, a 95% interval, and McNemar’s test against a 1-nearest-neighbor classifier on the same test images.

import numpy as np
from scipy.stats import binomtest
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

digits = load_digits()                                    # 1,797 grayscale 8x8 digit images
X = digits.images.reshape(-1, 64) / 16.0
y = digits.target
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=1, stratify=y)
n = len(y_te)

def softmax(Z):
    E = np.exp(Z - Z.max(axis=1, keepdims=True))
    return E / E.sum(axis=1, keepdims=True)

# Linear softmax classifier, trained by full-batch gradient descent on cross-entropy.
W, b = np.zeros((64, 10)), np.zeros(10)
Y = np.eye(10)[y_tr]
for step in range(1, 4001):
    G = (softmax(X_tr @ W + b) - Y) / len(X_tr)
    W -= 1.0 * X_tr.T @ G
    b -= 1.0 * G.sum(axis=0)
    if step in (250, 1000, 2000, 4000):
        P = softmax(X_te @ W + b)
        ce = -np.mean(np.log(P[np.arange(n), y_te]))
        print(f"step {step:4d}: test cross-entropy {ce:.3f}, test accuracy {np.mean(P.argmax(1) == y_te):.3f}")

# Top-1 and top-3 accuracy: is the true digit among the k highest-scoring classes?
ranked = np.argsort(-P, axis=1)
for k in (1, 3):
    print(f"top-{k} accuracy {np.mean((ranked[:, :k] == y_te[:, None]).any(axis=1)):.3f}")

# 95% confidence interval for the top-1 accuracy on n test images.
correct = ranked[:, 0] == y_te
acc = correct.mean()
se = np.sqrt(acc * (1 - acc) / n)
ci = binomtest(int(correct.sum()), n).proportion_ci(confidence_level=0.95, method="wilson")
print(f"accuracy {acc:.3f} on {n} images: standard error {se:.3f}, "
      f"normal interval [{acc - 1.96 * se:.3f}, {acc + 1.96 * se:.3f}], "
      f"Wilson interval [{ci.low:.3f}, {ci.high:.3f}]")

# McNemar's test: compare with a 1-nearest-neighbor classifier on the same test images.
d2 = ((X_te[:, None, :] - X_tr[None, :, :]) ** 2).sum(-1)
correct_b = y_tr[d2.argmin(1)] == y_te
n01 = int(np.sum(correct & ~correct_b))      # softmax right, nearest neighbor wrong
n10 = int(np.sum(~correct & correct_b))      # softmax wrong, nearest neighbor right
p = binomtest(n01, n01 + n10, 0.5).pvalue    # exact McNemar test: only disagreements count
print(f"1-NN accuracy {correct_b.mean():.3f}; "
      f"disagreements {n01} vs {n10}; exact McNemar p = {p:.2g}")

Output:

step  250: test cross-entropy 0.179, test accuracy 0.961
step 1000: test cross-entropy 0.113, test accuracy 0.972
step 2000: test cross-entropy 0.096, test accuracy 0.972
step 4000: test cross-entropy 0.087, test accuracy 0.972
top-1 accuracy 0.972
top-3 accuracy 0.997
accuracy 0.972 on 360 images: standard error 0.009, normal interval [0.955, 0.989], Wilson interval [0.950, 0.985]
1-NN accuracy 0.983; disagreements 2 vs 6; exact McNemar p = 0.29

From step 1,000 to 4,000 the test cross-entropy falls from 0.113 to 0.087 while accuracy stays at 97.2%: the model grows more confident on images it already classifies correctly. Top-3 accuracy is 99.7%, so nearly every error ranks the true digit second or third. The 95% interval spans more than three points. Wilson’s score interval, one of the methods offered by SciPy’s binomtest, is pulled toward 0.5 and so lies below the normal interval; near 0 or 1 the normal approximation is less reliable. The nearest-neighbor classifier scores 1.1 points higher, but the two models disagree on only eight images, 2 against 6, and McNemar’s test gives $p = 0.29$ (the $\chi^2$ form above gives the same): on 360 images this is not evidence that one classifier is better. Over ten train/test splits the linear model’s accuracy ranged from 96.4% to 97.8%, and the McNemar $p$-value was above 0.1 in eight of them.

The second script turns the digits into an imbalanced task, “is this a 1?”, and then scores a synthetic segmentation in which a small class is missed.

import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

def per_class_accuracy(y_true, y_pred):
    """Fraction of each class's images predicted correctly (the recall of each class)."""
    return np.array([np.mean(y_pred[y_true == c] == c) for c in np.unique(y_true)])

# 1. Imbalanced task: is this 8x8 image the digit 1?  (about 10% positives)
digits = load_digits()
X = digits.images.reshape(-1, 64) / 16.0
t = (digits.target == 1).astype(int)
X_tr, X_te, t_tr, t_te = train_test_split(X, t, test_size=0.2, random_state=1, stratify=t)

majority = np.zeros_like(t_te)                    # always answer "not a 1"
w = np.linalg.lstsq(np.c_[X_tr, np.ones(len(X_tr))], 2.0 * t_tr - 1, rcond=None)[0]
linear = (np.c_[X_te, np.ones(len(X_te))] @ w > 0).astype(int)   # least-squares linear classifier
print(f"test images: {len(t_te)}, of which {t_te.sum()} show a 1")
for name, pred in [("majority class", majority), ("linear", linear)]:
    pc = per_class_accuracy(t_te, pred)
    print(f"{name:15s} accuracy {np.mean(pred == t_te):.3f}   per class (other, 1) {np.round(pc, 3)}"
          f"   balanced accuracy {pc.mean():.3f}")

# 2. Segmentation: a 64x64 label map with sky (0), road (1) and a small sign (2).
gt = np.zeros((64, 64), int)
gt[32:, :] = 1                                    # road fills the bottom half
gt[20:26, 40:46] = 2                              # a 6x6 sign: under 1% of the pixels
pred = gt.copy()
pred[gt == 2] = 0                                 # the model paints the sign as sky
pred[30:32, :] = 1                                # and puts the road boundary 2 rows too high
n_cls = 3
N = np.zeros((n_cls, n_cls), int)                 # N[i, j]: pixels of class i predicted as j
np.add.at(N, (gt.ravel(), pred.ravel()), 1)
pixel_acc = np.trace(N) / N.sum()
class_acc = np.diag(N) / N.sum(axis=1)
iou = np.diag(N) / (N.sum(axis=1) + N.sum(axis=0) - np.diag(N))
print("class pixel counts", N.sum(axis=1), " per-class accuracy", np.round(class_acc, 3))
print(f"pixel accuracy {pixel_acc:.3f}   mean class accuracy {class_acc.mean():.3f}   "
      f"mean IoU {iou.mean():.3f}")

Output:

test images: 360, of which 36 show a 1
majority class  accuracy 0.900   per class (other, 1) [1. 0.]   balanced accuracy 0.500
linear          accuracy 0.961   per class (other, 1) [0.988 0.722]   balanced accuracy 0.855
class pixel counts [2012 2048   36]  per-class accuracy [0.936 1.    0.   ]
pixel accuracy 0.960   mean class accuracy 0.645   mean IoU 0.620

The majority-class baseline reaches 90.0% accuracy without recognizing a single 1. The linear classifier looks only six points better by accuracy, 96.1%, yet balanced accuracy, 50.0% against 85.5%, shows how much more useful it is, and the per-class numbers show that it still misses 28% of the 1s. In the segmentation, the missed sign covers 36 of 4,096 pixels, so pixel accuracy stays at 96.0%; mean class accuracy drops to 64.5% because the sign class scores 0.

Related Metrics

• Precision and recall score one class at a time and separate false alarms from misses; the per-class accuracy above is the recall of each class.
• Intersection over union replaces pixel accuracy in segmentation and scores localization in detection.
• Cross-entropy (log loss) scores the probability given to the true class, rewarding the confidence that accuracy ignores.