Precision and Recall

Precision and recall measure the two ways a system that says “yes, this is a positive” can go wrong. Precision is the fraction of its positive predictions that are correct; recall is the fraction of the actual positives that it finds. A binary image classifier or an object detector trades one for the other through its decision threshold, so the two are reported together, as a curve over thresholds, or combined into the F-score. The terms come from information retrieval, where they score the documents a search returns (van Rijsbergen, 1979, ch. 7).

Formula

Confusion matrix. For a binary classifier evaluated on a labeled test set, every image falls into one of four cells:

	predicted positive	predicted negative
actually positive	true positive (TP)	false negative (FN)
actually negative	false positive (FP)	true negative (TN)

From the counts,

$$\text{precision} = \frac{\mathrm{TP}}{\mathrm{TP} + \mathrm{FP}}, \qquad \text{recall} = \frac{\mathrm{TP}}{\mathrm{TP} + \mathrm{FN}}, \qquad \text{specificity} = \frac{\mathrm{TN}}{\mathrm{TN} + \mathrm{FP}}.$$

Recall is also called sensitivity or the true positive rate (TPR), and precision the positive predictive value. The false positive rate is $\mathrm{FPR} = \mathrm{FP}/(\mathrm{FP} + \mathrm{TN}) = 1 - \text{specificity}$ (Fawcett, 2006). Accuracy, the fraction of all images classified correctly, is $(\mathrm{TP} + \mathrm{TN})/N$.

F-score. The F1 score is the harmonic mean of precision $P$ and recall $R$:

$$F_1 = \frac{2PR}{P + R} = \frac{2\,\mathrm{TP}}{2\,\mathrm{TP} + \mathrm{FP} + \mathrm{FN}}, \qquad F_\beta = \frac{(1 + \beta^2)\,PR}{\beta^2 P + R}.$$

Van Rijsbergen (1979, ch. 7) derived an effectiveness measure $E = 1 - 1/\big(\alpha/P + (1 - \alpha)/R\big)$ from the theory of measurement. With $\alpha = 1/(\beta^2 + 1)$, $E$ measures effectiveness for a user who attaches $\beta$ times as much importance to recall as to precision, and $1 - E$ is the $F_\beta$ above, although the book does not call it the F-measure. $\beta = 2$ favors recall, $\beta = 0.5$ favors precision.

Detection. In object detection, a prediction is a box with a class and a score. Each predicted box is matched to at most one ground-truth box of the same class, if their intersection over union reaches a threshold, typically 0.5. Matched predictions are TPs, unmatched predictions FPs, and unmatched ground-truth boxes FNs. PASCAL VOC assigns detections in order of decreasing confidence and counts repeated detections of one object as false: five detections of one object give one TP and four FPs (Everingham et al., 2010). True negatives are undefined: a box can be placed anywhere at any size, so there is no finite set of correctly empty boxes to count. Specificity, FPR and accuracy therefore have no standard meaning for a detector; precision and recall do.

Intuition

A classifier usually outputs a score, and an image is called positive when the score reaches a threshold. Raising the threshold makes it more cautious: false positives fall, so precision usually rises, but more real positives drop below the line, so recall falls. Recall can never increase as the threshold rises; precision can dip when the images that drop out are true positives. Calling everything positive gives a recall of 1 and a precision equal to the share of positives in the data.

Sweeping the threshold over all scores traces the precision–recall (PR) curve, with recall on the horizontal axis and precision on the vertical. The receiver operating characteristic (ROC) curve plots TPR against FPR for the same thresholds. The area under the ROC curve equals the probability that the classifier scores a randomly chosen positive above a randomly chosen negative (Fawcett, 2006). Average precision, the subject of its own article, summarizes the PR curve as one number.

Interpretation

• Range. Precision, recall, specificity and $F_\beta$ all lie in $[0, 1]$; 1 is best. Precision is undefined when nothing is predicted positive ($0/0$), and implementations choose a convention.
• What “good” means depends on prevalence. Recall and FPR are each computed within one row of the confusion matrix above, among the actual positives or the actual negatives, so they do not change when the ratio of positives to negatives changes. Precision, accuracy and F1 mix both rows and do (Fawcett, 2006). A precision of 0.5 can be excellent when positives are one image in a thousand and poor when they are half the data.
• PR or ROC. Davis and Goadrich (2006) proved that, for a test set with fixed numbers of positives and negatives, one curve dominates another in ROC space if and only if it dominates in PR space. The display differs: when negatives greatly outnumber positives, a large change in the number of false positives is a small change in FPR, so a ROC curve can look nearly perfect while precision shows how many positive calls are wrong. They also showed that linear interpolation between points is incorrect in PR space, and that optimizing the area under the ROC curve does not guarantee optimizing the area under the PR curve.

When to Use It

Use precision and recall when positives are rare or the two errors have different costs, as in most detection and retrieval problems. In visual defect inspection, for example, a missed defect reaches the customer while a false alarm costs a second look, which argues for tuning to high recall and judging the precision kept there. Fix the operating point from the cost of each error, or report $F_\beta$ with a $\beta$ that reflects it.

Detection benchmarks compute both per class from IoU-thresholded matches. Besides average precision, COCO reports average recall (AR), the maximum recall given a fixed number of detections per image (1, 10 or 100), averaged over categories and over IoU thresholds from 0.50 to 0.95 in steps of 0.05 (COCO detection evaluation page). The same counts score correspondences in feature matching, boundary pixels in edge detection benchmarks (as an F-measure), and the detection part of HOTA.

Failure Cases

• One number alone. Recall alone rewards calling everything positive; precision alone rewards a single confident prediction.
• Ignoring true negatives. F1 never uses TN. For a detector this is necessary; for a classifier whose correct rejections matter, it hides how many negatives were wrongly flagged.
• Comparing across prevalence. Precision measured on a balanced test set overstates what the classifier achieves on rare positives: 0.914 against 0.492 in the example.
• Threshold chosen on the test set. A threshold tuned on test data, as in the example, gives an optimistic score; tune it on validation data.
• Matching choices in detection. The counts depend on the IoU threshold and the matching rule: in the example, the box with IoU 0.47 would become a true positive at a threshold of 0.45, removing a false positive and a false negative.

Variants

Averaging over classes. For $K$ classes, each class gets its own one-against-the-rest confusion matrix. Macro-averaging computes precision, recall or F1 per class and averages them, giving each class equal weight; micro-averaging sums TP, FP and FN over classes first and computes the measure from the pooled counts, giving each decision equal weight, so large classes dominate (Manning, Raghavan & Schütze, 2008, §13.6). In single-label multi-class classification every error is a false positive for one class and a false negative for another, so micro-averaged precision, recall and F1 all equal accuracy, as follows from the definitions. Macro-averaging is the choice when rare classes matter.

Dice coefficient. Computed over pixels, F1 is the Dice coefficient of segmentation, which intersection over union relates to IoU.

Identity measures. IDF1 is an F1 score over detections whose identity agrees with a globally matched ground-truth trajectory, with identification precision and recall as its parts.

Example

The first part scores scikit-learn’s 8×8 digit images for “3 or not 3” with a linear score: the dot product with the difference between the mean “3” and the mean other digit over 1,000 training images. On the 797 test images it builds the confusion matrix, sweeps the threshold along the PR and ROC curves, and finds the best F1.

import numpy as np
from sklearn.datasets import load_digits

X, y = load_digits(return_X_y=True)     # 1,797 images of 8x8 pixels, values 0..16
X = X / 16.0
idx = np.random.default_rng(0).permutation(len(y))
tr, te = idx[:1000], idx[1000:]
is3 = (y == 3)                           # positive class: "is it a 3?"

# A linear classifier: score = w . x, with w the difference of the class means.
w = X[tr][is3[tr]].mean(0) - X[tr][~is3[tr]].mean(0)
s, t = X[te] @ w, is3[te]                # scores and true labels on the test set


def confusion(scores, labels, thr):
    pred = scores >= thr
    tp = np.sum(pred & labels); fp = np.sum(pred & ~labels)
    fn = np.sum(~pred & labels); tn = np.sum(~pred & ~labels)
    return tp, fp, fn, tn


# 1. Confusion matrix at one threshold.
thr = 1.0
tp, fp, fn, tn = confusion(s, t, thr)
print(f"{t.sum()} positives, {(~t).sum()} negatives; threshold {thr}")
print(f"TP={tp} FP={fp} FN={fn} TN={tn}")
print(f"precision {tp/(tp+fp):.3f}  recall {tp/(tp+fn):.3f}  "
      f"specificity {tn/(tn+fp):.3f}  accuracy {(tp+tn)/len(t):.3f}")

# 2. Sweep the threshold: points of the PR curve and the ROC curve.
print("\n  thr  precision  recall    FPR")
for thr in [0.0, 0.5, 1.0, 1.5, 2.0, 2.5]:
    tp, fp, fn, tn = confusion(s, t, thr)
    print(f"{thr:5.1f}  {tp/max(tp+fp,1):9.3f}  {tp/(tp+fn):6.3f}  {fp/(fp+tn):6.3f}")

# 3. F1 at every candidate threshold (each distinct score); keep the best.
thrs = np.unique(s)
f1 = [2*c[0] / (2*c[0] + c[1] + c[2]) for c in (confusion(s, t, h) for h in thrs)]
best = thrs[np.argmax(f1)]
tp, fp, fn, tn = confusion(s, t, best)
print(f"\nbest F1 {max(f1):.3f} at threshold {best:.2f}: "
      f"precision {tp/(tp+fp):.3f}, recall {tp/(tp+fn):.3f}, FPR {fp/(fp+tn):.3f}")

# 4. Threshold 1.0 again, on a balanced test set: keep as many negatives as positives.
neg = np.flatnonzero(~t)
keep = np.concatenate([np.flatnonzero(t),
                       np.random.default_rng(8).choice(neg, t.sum(), replace=False)])
tp, fp, fn, tn = confusion(s[keep], t[keep], 1.0)
print(f"balanced 1:1 test set: precision {tp/(tp+fp):.3f}, recall {tp/(tp+fn):.3f}, "
      f"FPR {fp/(fp+tn):.3f}")

Output:

74 positives, 723 negatives; threshold 1.0
TP=64 FP=66 FN=10 TN=657
precision 0.492  recall 0.865  specificity 0.909  accuracy 0.905

  thr  precision  recall    FPR
  0.0      0.226   1.000   0.351
  0.5      0.305   0.973   0.227
  1.0      0.492   0.865   0.091
  1.5      0.826   0.770   0.017
  2.0      1.000   0.568   0.000
  2.5      1.000   0.162   0.000

best F1 0.836 at threshold 1.62: precision 0.933, recall 0.757, FPR 0.006
balanced 1:1 test set: precision 0.914, recall 0.865, FPR 0.081

At threshold 1.0 the classifier finds 86.5% of the 3s with a false positive rate of only 9.1%, a good-looking ROC point. But negatives outnumber positives almost ten to one, so those 9.1% are 66 images, more than the 64 true positives, and precision is 0.492. Accuracy, 0.905, is even below the 0.907 of always answering “not a 3”, the baseline problem the accuracy article discusses. F1 peaks at 0.836 at threshold 1.62 (0.82–0.86 over ten train/test splits). On a balanced subset, recall is unchanged and FPR similar, but precision jumps to 0.914: precision depends on the class ratio, the ROC axes do not.

The second part counts TP, FP and FN for a detector on a synthetic 64×64 image with three bright objects, matching each detection, in order of decreasing score, to the still-unmatched ground-truth box with the highest IoU, if that IoU is at least 0.5. This simplifies both benchmarks: the PASCAL VOC code matches each detection to its highest-IoU ground-truth box and counts it false if that box is taken, while COCO’s code, like this one, takes the best unmatched box but also handles crowd regions.

import numpy as np
from scipy import ndimage

# A synthetic 64x64 image with three bright objects; their boxes are the ground truth.
img = np.zeros((64, 64))
img[5:21, 6:22] = 1; img[30:42, 40:58] = 1; img[46:60, 8:20] = 1
gt = [(sl[1].start, sl[0].start, sl[1].stop, sl[0].stop)          # (x1, y1, x2, y2)
      for sl in ndimage.find_objects(ndimage.label(img)[0])]

# Detector output: box and confidence score.
dets = [((7, 6, 22, 21), 0.95),    # good box on object 1
        ((41, 29, 57, 43), 0.90),  # good box on object 2
        ((4, 3, 20, 19), 0.80),    # duplicate on object 1
        ((30, 5, 44, 17), 0.60),   # background
        ((10, 50, 22, 62), 0.40)]  # object 3, but poorly placed


def iou(a, b):
    w = max(0, min(a[2], b[2]) - max(a[0], b[0]))
    h = max(0, min(a[3], b[3]) - max(a[1], b[1]))
    inter = w * h
    area = lambda r: (r[2] - r[0]) * (r[3] - r[1])
    return inter / (area(a) + area(b) - inter)


# Greedy matching in order of decreasing score; each ground-truth box matches once.
matched, tp_flags = set(), []
for box, score in sorted(dets, key=lambda d: -d[1]):
    ious = [iou(box, g) if i not in matched else 0.0 for i, g in enumerate(gt)]
    best = int(np.argmax(ious))
    ok = ious[best] >= 0.5
    if ok:
        matched.add(best)
    tp_flags.append(ok)
    print(f"score {score:.2f}  best IoU {max(iou(box, g) for g in gt):.2f}  "
          f"{'TP' if ok else 'FP'}")

tp = np.cumsum(tp_flags); fp = np.cumsum(~np.array(tp_flags))
print(f"\nall detections: TP={tp[-1]} FP={fp[-1]} FN={len(gt) - tp[-1]}")
print("PR points as the score threshold is lowered:")
print(" ".join(f"({r:.2f}, {p:.2f})" for r, p in zip(tp / len(gt), tp / (tp + fp))))

Output:

score 0.95  best IoU 0.88  TP
score 0.90  best IoU 0.77  TP
score 0.80  best IoU 0.62  FP
score 0.60  best IoU 0.00  FP
score 0.40  best IoU 0.47  FP

all detections: TP=2 FP=3 FN=1
PR points as the score threshold is lowered:
(0.33, 1.00) (0.67, 1.00) (0.67, 0.67) (0.67, 0.50) (0.67, 0.40)

The duplicate overlaps object 1 with IoU 0.62, but object 1 is already taken, so it is a false positive. The last box reaches only IoU 0.47 with object 3, so it counts twice: as a false positive, and object 3 as a false negative. Nothing is a true negative. Lowering the score threshold one detection at a time gives the (recall, precision) points of the PR curve, the input to average precision.

Related Metrics

• Accuracy also counts TN; it suits balanced classes with equal error costs.
• Average precision summarizes the PR curve over all thresholds and is the standard detection metric.
• Intersection over union decides which detections are true positives; over pixels it is TP / (TP + FP + FN).
• MOTA combines FP and FN counts with identity switches into one tracking error rate.