Confusion Matrix

A confusion matrix is a table that counts, for a classifier evaluated on labeled test data, how many examples of each true class were predicted as each class. For an image classifier with $K$ classes it is a $K \times K$ table of counts of test images; for semantic segmentation the same table counts pixels. Correct predictions sit on the diagonal and every kind of mistake has its own cell, so the matrix shows not only how often a model is wrong but which classes it mistakes for which. Accuracy, per-class precision and recall, and the intersection over union of segmentation are all computed from it. The two-class matrix holds the familiar true positives, false positives, true negatives and false negatives.

Definition

Given the true class and the predicted class of every test example, the confusion matrix has one row and one column per class. In the convention of scikit-learn, entry $(i, j)$ is the number of examples whose true class is $i$ and whose predicted class is $j$ (scikit-learn 1.7.2 User Guide, §3.4.4.6). Rows therefore correspond to the ground truth and columns to the predictions. The scikit-learn documentation itself warns that other references may arrange the axes differently, and the section on conventions below gives examples.

A matrix describes one set of hard decisions. For a classifier that outputs scores and applies a threshold, each threshold gives a different matrix; the ROC curve and the precision–recall curve trace how the two-class matrix changes as the threshold moves.

Intuition

Think of sorting a stack of test images into a grid of boxes. Each image goes into the row of the class it really shows and the column of the class the model said. If the model were perfect, every image would land on the diagonal. Reading along a row answers “where did the images of class $i$ go?”: the diagonal cell holds the ones that were recognized, and the other cells say what they were mistaken for. Reading down a column answers the reverse question, “when the model says $j$, what is really there?”.

A single accuracy number hides that structure. A digit classifier that is 89% accurate could spread its errors evenly, or it could fail almost entirely on one digit; the two cases have the same accuracy and very different matrices.

Formal Definition

Let $N$ test examples have true labels $y_n$ and predictions $\hat{y}_n$ in $\{1, \dots, K\}$. The confusion matrix $C \in \mathbb{N}^{K \times K}$ is

$$C_{ij} = \sum_{n=1}^{N} \mathbf{1}[y_n = i]\, \mathbf{1}[\hat{y}_n = j],$$

where $\mathbf{1}[\cdot]$ is 1 when its argument is true and 0 otherwise. The row sum $t_i = \sum_j C_{ij}$ is the number of examples of class $i$ (its support), the column sum $p_j = \sum_i C_{ij}$ is the number of examples predicted as $j$, and $\sum_{i,j} C_{ij} = N$. With per-example weights $w_n$, each example adds $w_n$ instead of 1.

Two classes. With one class designated positive, the four cells are the true positives (TP, positives predicted positive), false negatives (FN, positives predicted negative), false positives (FP, negatives predicted positive) and true negatives (TN, negatives predicted negative). scikit-learn sorts the labels, so with labels 0 and 1 and 1 as the positive class its matrix is

$$C = \begin{pmatrix} \mathrm{TN} & \mathrm{FP} \\ \mathrm{FN} & \mathrm{TP} \end{pmatrix},$$

and confusion_matrix(y_true, y_pred).ravel() returns the counts in the order TN, FP, FN, TP (scikit-learn 1.7.2 documentation). The precision and recall article draws the same table with the positive class first; the counts are the same, only the order of rows and columns differs.

One class against the rest. In a $K$-class matrix, each class $k$ has its own two-class counts:

$$\mathrm{TP}_k = C_{kk}, \qquad \mathrm{FN}_k = t_k - C_{kk}, \qquad \mathrm{FP}_k = p_k - C_{kk}, \qquad \mathrm{TN}_k = N - t_k - p_k + C_{kk}.$$

Normalization. Dividing the counts gives three different tables of rates:

$$\text{by row: } \frac{C_{ij}}{t_i}, \qquad \text{by column: } \frac{C_{ij}}{p_j}, \qquad \text{by total: } \frac{C_{ij}}{N}.$$

These are scikit-learn’s normalize="true", "pred" and "all" options (scikit-learn 1.7.2 User Guide, §3.4.4.6). The row-normalized entry estimates the probability of predicting $j$ for an example of class $i$; the column-normalized entry estimates the probability that an example predicted as $j$ belongs to class $i$; the total-normalized table estimates the joint distribution of true and predicted class.

Properties

Metrics read from the matrix. Each of the following is a function of $C$ alone:

• Accuracy is the trace divided by the total, $\sum_k C_{kk} / N$, which is also the trace of the total-normalized matrix.
• The recall of class $k$ is $C_{kk}/t_k$, the diagonal of the row-normalized matrix, and its precision is $C_{kk}/p_k$, the diagonal of the column-normalized matrix. Their definitions and trade-offs belong to precision and recall. Balanced accuracy, the mean of the per-class recalls, is in the accuracy article.
• The F1 score of each class combines its precision and recall, so it and its averages over classes are also functions of the matrix.
• The intersection over union of class $k$ is $C_{kk}/(t_k + p_k - C_{kk})$, its true positives divided by true positives, false positives and false negatives together. Computed on a pixel confusion matrix, this gives the per-class IoU whose mean ranks semantic segmentation methods.
• For a two-class problem, the true positive rate and false positive rate of one matrix give one point on the ROC curve.

Imbalance acts on rows and columns differently. Every off-diagonal count is an error, but its weight depends on which way it is read. If the test set contained more or fewer examples of class $i$, drawn in the same way and with the classifier unchanged, row $i$ would scale by the same factor on average. It follows that the row-normalized matrix, and with it every per-class recall, does not depend on the class frequencies of the test set, apart from sampling noise. The column-normalized matrix does: a column sums over all true classes, so the errors of a frequent class can swamp the correct predictions of a rare one. A rare class can therefore have high recall and low precision at the same time, which the example below shows. Precision measured on a balanced test set does not carry over to deployment data with a different class mix, as the precision and recall article discusses.

Asymmetry. $C$ is not symmetric in general: class $a$ may often be predicted as $b$ while $b$ is rarely predicted as $a$. Each direction is a different mistake and the two are read separately.

Implementation details in scikit-learn 1.7.2. By default the rows and columns are the labels that appear in y_true or y_pred, in sorted order; the labels argument fixes the order or selects a subset, and examples whose true or predicted label is not in labels are dropped. A class that never appears in either array is left out unless it is listed in labels. When a row or column of a normalized matrix sums to zero, the source code replaces the resulting $0/0$ by 0 rather than leaving it undefined. ConfusionMatrixDisplay.from_predictions and from_estimator compute the matrix with the same options and plot it as an image with “True label” on the vertical axis and “Predicted label” on the horizontal one (scikit-learn 1.7.2 documentation and source code).

Other conventions. TensorFlow’s tf.math.confusion_matrix also puts the true labels on rows and predictions on columns (TensorFlow 2.21.0 source code). The R package caret’s confusionMatrix does the opposite: it cross-tabulates predictions against the reference, with the predicted classes on rows and the reference (true) classes on columns (caret v7.0-1 source code). Before reading a published matrix, check which axis holds the ground truth; swapping them exchanges precision and recall.

Multi-label data. When each image can carry several labels, it has no single true class and no single predicted class, so there is no $K \times K$ matrix of the kind above. scikit-learn’s multilabel_confusion_matrix instead returns one $2 \times 2$ matrix per label, in the same TN, FP, FN, TP layout, or, for multi-label input, one per image with samplewise=True. Given single-label multi-class data, it returns the one-against-the-rest matrix of each class (scikit-learn 1.7.2 documentation).

Examples

The example classifies scikit-learn’s $8 \times 8$ digit images with a nearest-class-mean classifier: each test image is assigned the digit whose mean training image is closest in squared pixel distance. Half of the 1,797 images train it. The test half is made imbalanced by keeping only 10 of its images of the digit 9, chosen with a fixed seed. The code builds the matrix by hand, checks it against scikit-learn 1.7.2 (NumPy 2.3.5), reads off per-class recall and precision, lists the largest confusions, and then reweights the classes to equal frequency with sample_weight to show what the imbalance does to precision.

import numpy as np
import sklearn
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix, multilabel_confusion_matrix

digits = load_digits()                                   # 1,797 grayscale 8x8 digit images
X, y = digits.images.reshape(-1, 64) / 16.0, digits.target
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.5, random_state=0, stratify=y)

# Make the test set imbalanced: keep only 10 of the roughly 90 test images of digit 9.
rng = np.random.default_rng(0)
nines = np.flatnonzero(y_te == 9)
keep = np.concatenate([np.flatnonzero(y_te != 9), rng.choice(nines, 10, replace=False)])
X_te, y_te = X_te[keep], y_te[keep]

# Nearest-class-mean classifier: assign each image to the closest mean training image.
means = np.stack([X_tr[y_tr == c].mean(axis=0) for c in range(10)])
y_pred = ((X_te[:, None, :] - means[None]) ** 2).sum(-1).argmin(axis=1)

# The confusion matrix by hand: C[i, j] counts images of true class i predicted as j.
K = 10
C = np.zeros((K, K), dtype=int)
np.add.at(C, (y_te, y_pred), 1)
assert (C == confusion_matrix(y_te, y_pred)).all()
print(f"scikit-learn {sklearn.__version__}, {len(y_te)} test images")
print("rows = true digit, columns = predicted digit")
print("     " + "".join(f"{j:4d}" for j in range(K)))
for i in range(K):
    print(f"{i:4d} " + "".join(f"{v:4d}" for v in C[i]))

# Normalizations: by row (true), by column (pred), by the total (all).
recall = np.diag(C) / C.sum(axis=1)                      # diagonal of normalize="true"
precision = np.diag(C) / C.sum(axis=0)                   # diagonal of normalize="pred"
assert np.allclose(recall, np.diag(confusion_matrix(y_te, y_pred, normalize="true")))
assert np.allclose(precision, np.diag(confusion_matrix(y_te, y_pred, normalize="pred")))
print("recall    " + " ".join(f"{v:.2f}" for v in recall))
print("precision " + " ".join(f"{v:.2f}" for v in precision))
print(f"accuracy {np.trace(C) / C.sum():.3f}  "
      f"(trace of normalize='all': {np.trace(confusion_matrix(y_te, y_pred, normalize='all')):.3f}), "
      f"balanced accuracy {recall.mean():.3f}")

# The largest off-diagonal cells: which digits are confused with which.
off = C.copy()
np.fill_diagonal(off, 0)
for flat in np.argsort(-off, axis=None, kind="stable")[:4]:
    i, j = divmod(int(flat), K)
    print(f"true {i} -> predicted {j}: {off[i, j]} images ({off[i, j] / C[i].sum():.0%} of row {i})")

# One-vs-rest 2x2 matrix for digit 9, in scikit-learn's [[TN, FP], [FN, TP]] layout.
tn, fp, fn, tp = multilabel_confusion_matrix(y_te, y_pred)[9].ravel().tolist()
print(f"digit 9 vs rest: TN={tn} FP={fp} FN={fn} TP={tp}")

# Reweight each true class to the same total, as if the test set were balanced.
w = (len(y_te) / K) / np.bincount(y_te)[y_te]
Cw = confusion_matrix(y_te, y_pred, sample_weight=w)
print("balanced prevalence: recall of 9 {:.2f}, precision of 9 {:.2f}".format(
      Cw[9, 9] / Cw[9].sum(), Cw[9, 9] / Cw[:, 9].sum()))

Output:

scikit-learn 1.7.2, 819 test images
rows = true digit, columns = predicted digit
        0   1   2   3   4   5   6   7   8   9
   0   89   0   0   0   0   0   0   0   0   0
   1    0  77   2   0   0   1   0   0   1  10
   2    0   5  75   4   0   0   0   0   3   1
   3    0   0   1  76   0   1   0   5   5   4
   4    0   1   0   0  84   0   0   4   2   0
   5    0   0   0   0   1  78   1   0   0  11
   6    1   3   0   0   0   0  86   0   1   0
   7    0   0   0   0   0   2   0  87   0   0
   8    0  10   2   0   0   2   0   1  68   4
   9    0   1   0   0   0   1   0   0   0   8
recall    1.00 0.85 0.85 0.83 0.92 0.86 0.95 0.98 0.78 0.80
precision 0.99 0.79 0.94 0.95 0.99 0.92 0.99 0.90 0.85 0.21
accuracy 0.889  (trace of normalize='all': 0.889), balanced accuracy 0.881
true 5 -> predicted 9: 11 images (12% of row 5)
true 1 -> predicted 9: 10 images (11% of row 1)
true 8 -> predicted 1: 10 images (11% of row 8)
true 2 -> predicted 1: 5 images (6% of row 2)
digit 9 vs rest: TN=779 FP=30 FN=2 TP=8
balanced prevalence: recall of 9 0.80, precision of 9 0.71

The accuracy of 88.9% says little about where the errors are; the matrix locates them. Most of them fall in a few cells. The column of 9 collects 30 images of other digits, mostly 5s and 1s, and more than half of the 8s that fail are called 1. The errors are asymmetric: ten 1s are called 9, while only one of the ten 9s is called 1; ten 8s are called 1, and one 1 is called 8. Each large cell names a specific pair of classes, here pairs of digits whose mean images lie close to many images of the other class, and so tells a developer which distinctions a better feature or classifier has to learn.

The digit 9 shows the effect of imbalance. Eight of its ten test images are recognized, a recall of 0.80, comparable to the other digits. Its precision is 0.21, because the 30 images of frequent digits that land in its column outnumber the 8 real 9s there. The one-against-the-rest counts printed by multilabel_confusion_matrix are the same numbers seen as a two-class matrix. Reweighting every class to the same total leaves the recall of 9 unchanged, since each row is scaled as a whole, and raises its precision to 0.71, because the true 9s now carry as much weight as the 1s and 5s that leak into their column. Balanced accuracy, the mean of the recall row, is close to accuracy here because only one of the ten classes is rare.

Common Misconceptions

• “Rows are always the truth.” The convention varies between libraries and texts, as shown above. A transposed matrix is still a valid confusion matrix, but reading it with the wrong orientation swaps every precision with the corresponding recall.
• “A normalized matrix is enough.” Row normalization hides the support. A row of 1.00 on the diagonal from 3 test images and one from 3,000 look identical; report the counts as well, or the support of each class.
• “Detectors have a confusion matrix like classifiers.” In object detection, predictions are matched to ground-truth boxes by IoU, so true positives, false positives and false negatives are defined, but true negatives are not: there is no finite set of correctly empty boxes, as the precision and recall article explains.

Where It Is Used

• Image classification: a common way to inspect which classes a model confuses, for instance visually similar categories, and to compute per-class recall and precision when classes are imbalanced or one error matters more than another.
• Semantic segmentation: pixel accuracy, mean class accuracy and mean IoU are all functions of the confusion matrix accumulated over every labeled pixel of the test images, as the formulas in the accuracy article show.
• Two-class decisions: defect inspection, lesion screening and other present-or-absent tasks report TP, FP, TN and FN, from which precision and recall, specificity and the ROC operating point follow.
• Segment matching: panoptic quality is built from the true positive, false positive and false negative counts of matched segments, without true negatives, as in detection.