Average Precision

Average precision (AP) scores an object detector on one class with one number: the area under its precision–recall curve, traced by lowering a score threshold through the detections ranked from most to least confident. A detection counts as correct when it overlaps an unclaimed ground-truth box by enough intersection over union. The mean of AP over classes is the mean average precision (mAP), the standard summary for detection benchmarks; COCO calls this mean simply “AP” and also averages it over ten IoU thresholds. The measure comes from information retrieval, where 11-point interpolated average precision was the traditional summary of ranked search results, used for instance in the first eight TREC ad hoc evaluations (Manning, Raghavan & Schütze, 2008, §8.4).

Formula

Matching. Take all detections of class $c$ over the whole test set and sort them by decreasing score. Going down the list, each detection is matched to a ground-truth box of class $c$ in the same image whose IoU with it is at least a threshold $t$ and which no higher-scoring detection has already claimed. A matched detection is a true positive (TP); an unmatched one is a false positive (FP), and so is a second detection of an already-claimed object: five detections of one object count as one TP and four FPs (Everingham et al., 2010). The reference implementations differ in a detail: the PASCAL VOC code takes the ground-truth box with the highest IoU and counts the detection false if that box is already claimed, while COCO’s cocoeval.py takes the best box among those still unclaimed. Both exclude some ground truth: VOC ignores objects flagged difficult, and COCO ignores crowd regions, which may absorb any number of detections (the bounding box article describes the flags). A detection matched to an ignored object counts as neither TP nor FP.

Precision–recall curve. With $N$ the number of non-ignored ground-truth objects of the class and $\mathrm{TP}_k$, $\mathrm{FP}_k$ the counts among the top $k$ detections,

$$R_k = \frac{\mathrm{TP}_k}{N}, \qquad P_k = \frac{\mathrm{TP}_k}{\mathrm{TP}_k + \mathrm{FP}_k}.$$

Interpolation. The measured curve zigzags: precision falls at every FP and recovers at the next TP. AP is computed on its upper envelope, the interpolated precision

$$p_{\text{interp}}(r) = \max_{k \,:\, R_k \ge r} P_k ,$$

taken as 0 for recall levels the detector never reaches. Everingham et al. (2010) define it this way to reduce the effect of these “wiggles”, which come from small changes in the ranking. The envelope does not increase with $r$.

Three summaries. PASCAL VOC 2007 averaged the envelope at eleven recall levels (Everingham et al., 2010), as did the 2008 and 2009 challenges; VOC 2010–2012 integrates it exactly, which the development kit describes as sampling at all unique recall values (Everingham & Winn, 2012); COCO samples it at 101 levels (recThrs in cocoeval.py):

$$\mathrm{AP}_{11} = \frac{1}{11} \sum_{r \in \{0, 0.1, \ldots, 1\}} p_{\text{interp}}(r), \qquad \mathrm{AP}_{\text{all}} = \sum_{k} (R_k - R_{k-1})\, p_{\text{interp}}(R_k), \qquad \mathrm{AP}_{101} = \frac{1}{101} \sum_{r \in \{0, 0.01, \ldots, 1\}} p_{\text{interp}}(r),$$

with $R_0 = 0$. The envelope is piecewise constant, so $\mathrm{AP}_{\text{all}} = \int_0^1 p_{\text{interp}}(r)\,dr$ with no approximation. The VOC 2012 development kit says the 2010 change was made to improve precision and the ability to tell apart methods with low AP.

Averaging. For $K$ classes and COCO’s ten IoU thresholds $T = \{0.50, 0.55, \ldots, 0.95\}$,

$$\mathrm{mAP} = \frac{1}{K} \sum_{c=1}^{K} \mathrm{AP}_c, \qquad \mathrm{AP}_{\text{COCO}} = \frac{1}{|T|} \sum_{t \in T} \frac{1}{K} \sum_{c=1}^{K} \mathrm{AP}_{101}(c, t).$$

Intuition

Without interpolation, AP has a simple reading: walk down the ranked list and, at each true positive, note the precision so far; AP is the sum of these precisions divided by $N$. Each object found adds $1/N$ of recall, weighted by the precision at which it was found, and each object never found adds zero. This is how information retrieval defines AP for one query, as an approximation of the area under the uninterpolated curve (Manning, Raghavan & Schütze, 2008, §8.4). The detection versions replace each precision by the envelope, the best precision available at that recall or beyond.

Two consequences follow from the formula; they are reasoning, not statements from the benchmarks’ documentation. First, $N$ is fixed by the ground truth, so AP is bounded by the highest recall reached: a detector that never finds 20% of the objects cannot exceed an all-point AP of 0.8 (the sampled summaries, which include the level $r = 0$, can slightly exceed it: up to $9/11 \approx 0.82$ for the 11-point AP). Second, appending detections of a class with scores below all its existing detections cannot lower that class’s AP. They are matched last, so the earlier points of the curve do not change, and the envelope at each recall is a maximum over a set that can only grow; at worst they add nothing, and any that find a missed object add recall. AP therefore never penalizes, and often rewards, emitting many low-confidence boxes. Gupta, Dollár and Girshick (2019, appendix of the extended arXiv version) raised the mask AP of a Mask R-CNN baseline on LVIS v0.5 from 14.8 to 20.9 only by allowing 300 detections per image instead of 100 and removing the default minimum score of 0.05.

Interpretation

AP and mAP lie in $[0, 1]$ and are usually reported as percentages. AP = 1 requires every object to be found and every true positive to outrank every false positive. Because the score only orders detections, it follows that any strictly increasing rescaling of a class’s scores leaves its AP unchanged. Values are comparable only under the same convention: the 11-point, all-point and 101-point summaries of one curve differ, and COCO AP averaged over IoU 0.50–0.95 is much lower than AP at 0.50 for the same detector, as in the example below.

COCO reports twelve detection metrics: AP over IoU 0.50:0.05:0.95 (the primary challenge metric), $\mathrm{AP}_{50}$ (described as the PASCAL VOC metric), $\mathrm{AP}_{75}$ (strict), AP for small, medium and large objects, and average recall at three detection limits and three scales (COCO detection evaluation page). Small objects have an area below $32^2$ pixels, large ones above $96^2$, and medium ones in between, where area is the number of pixels in the segmentation mask, not the box. In cocoeval.py, a ground-truth object outside the size range is ignored, and so is an unmatched detection outside it. The page states that all metrics allow at most the 100 top-scoring detections per image, across all categories; the reference code instead applies this limit to each image and category separately. Box and mask evaluation differ only in how IoU is computed.

When to Use It

Use AP to compare detectors over all operating points when the downstream threshold is not yet known, and mAP to summarize many classes. Report $\mathrm{AP}_{50}$ when coarse localization is enough, and COCO AP or $\mathrm{AP}_{75}$ when box accuracy matters: COCO’s page notes that averaging over IoU thresholds rewards detectors with better localization. Everingham et al. (2010) adopted AP for VOC 2007 in place of the area under the ROC curve, which VOC 2006 had used for classification, to make the metric more sensitive and more interpretable, to give more visibility to performance at low recall, and to use one measure for classification and detection; the image classification article mentions its use for multi-label tasks.

Failure Cases

• Calibration and thresholds. AP depends only on the ranking within each class. It says nothing about whether a score of 0.9 means 90% correct, nor which threshold to deploy, and a detector tuned on AP can flood its output with low-score boxes that a deployed system must then filter.
• Few objects per class. Each object is a recall step of $1/N$; with ten objects, one miss costs up to a tenth of the AP. LVIS returns only the overall AP from its test server partly because per-category AP is unstable for rare categories (Gupta et al., 2019).
• Class averaging. mAP gives each class equal weight, so a rare class with three test objects counts as much as a frequent one with thousands, and the average hides which classes fail.
• The IoU range sets what is measured. $\mathrm{AP}_{50}$ barely distinguishes a tight box from a loose one; at high thresholds, a pixel of annotation noise can decide a small object’s match, as the IoU article shows.
• Incomplete annotation. An object the annotators missed turns a correct detection into a false positive. VOC’s difficult flag and COCO’s crowd regions handle known cases; LVIS’s federated design handles it for a large vocabulary.

Variants

• VOC 11-point AP (2007–2009) and all-point AP (2010–2012) at IoU 0.5, as in the formula.
• COCO AP, $\mathrm{AP}_{50}$, $\mathrm{AP}_{75}$, and $\mathrm{AP}_S$, $\mathrm{AP}_M$, $\mathrm{AP}_L$ by object size. For keypoints, cocoeval.py replaces IoU with a keypoint similarity score (its computeOks function) and keeps the AP machinery.
• LVIS federated AP. LVIS annotates each category exhaustively only on a positive set of images and certifies a negative set that contains none of it; AP for a category is computed on those images only, and a detection of that category on any other image is not evaluated. False positives are also not counted on images flagged as not exhaustively annotated (Gupta et al., 2019). The extended version reports AP separately for rare (1–10 training images), common (11–100) and frequent (>100) categories, and allows 300 detections per image.

Example

The NumPy code builds four synthetic 64 × 64 images containing eight bright rectangular objects, takes their tight boxes as ground truth, and scores ten detections that include tight boxes, loose boxes, background false positives, and one object that is never found. It implements the matching and the three summaries, and then changes one thing at a time: a duplicate box, a mislocalized box, and twenty extra low-score boxes.

import numpy as np
from scipy import ndimage

# Four synthetic 64x64 images; each object is a bright rectangle, and its
# ground-truth box (x1, y1, x2, y2) is the tight box around its pixels.
objects = {0: [(8, 10, 28, 30), (36, 30, 60, 50)],
           1: [(5, 5, 25, 35), (40, 40, 56, 56)],
           2: [(10, 20, 40, 44), (48, 6, 60, 18)],
           3: [(20, 12, 44, 36), (4, 48, 14, 60)]}
gts = []                                         # (image, box)
for k, objs in objects.items():
    img = np.zeros((64, 64))
    for x1, y1, x2, y2 in objs:
        img[y1:y2, x1:x2] = 1.0
    for sl in ndimage.find_objects(ndimage.label(img)[0]):
        gts.append((k, (sl[1].start, sl[0].start, sl[1].stop, sl[0].stop)))

# Detector output for the class: (image, box, score).
dets = [(0, (9, 11, 28, 31), 0.97),    # object 1, tight
        (1, (5, 6, 25, 35), 0.93),     # object 3, tight
        (2, (11, 20, 41, 45), 0.90),   # object 5, tight
        (3, (44, 40, 60, 60), 0.86),   # background
        (0, (38, 31, 61, 51), 0.82),   # object 2
        (1, (42, 42, 58, 58), 0.75),   # object 4, loose
        (3, (22, 15, 46, 38), 0.62),   # object 7
        (2, (34, 50, 50, 62), 0.55),   # background
        (2, (49, 7, 61, 20), 0.41),    # object 6, small
        (1, (30, 4, 38, 14), 0.30)]    # background; object 8 is never detected


def iou(a, b):                            # continuous corners, no +1 (COCO convention)
    w = max(0, min(a[2], b[2]) - max(a[0], b[0]))
    h = max(0, min(a[3], b[3]) - max(a[1], b[1]))
    area = lambda r: (r[2] - r[0]) * (r[3] - r[1])
    return w * h / (area(a) + area(b) - w * h)


def match(dets, gts, t):
    """TP flags in order of decreasing score: each detection takes the
    unmatched ground-truth box of its image with the highest IoU >= t."""
    used, flags = set(), []
    for img, box, s in sorted(dets, key=lambda d: -d[2]):
        cand = [(iou(box, g), j) for j, (gi, g) in enumerate(gts)
                if gi == img and j not in used]
        o, j = max(cand, default=(0.0, None))
        flags.append(o >= t)
        if o >= t:
            used.add(j)
    return np.array(flags)


def pr_curve(flags, n_gt):
    tp = np.cumsum(flags); fp = np.cumsum(~flags)
    return tp / n_gt, tp / (tp + fp)


def envelope(prec):                       # p_interp at each rank: max precision at this or higher recall
    return np.maximum.accumulate(prec[::-1])[::-1]


def ap_11point(rec, prec):                # PASCAL VOC 2007
    return np.mean([prec[rec >= r].max() if (rec >= r).any() else 0.0
                    for r in np.linspace(0, 1, 11)])


def ap_all_points(rec, prec):             # PASCAL VOC 2010-2012
    steps = np.diff(np.concatenate([[0.0], rec]))
    return np.sum(steps * envelope(prec))


def ap_101point(rec, prec):               # COCO, at one IoU threshold
    idx = np.searchsorted(rec, np.linspace(0, 1, 101), side="left")
    env = np.append(envelope(prec), 0.0)  # recall levels never reached get precision 0
    return env[idx].mean()


def report(name, dets):
    rec, prec = pr_curve(match(dets, gts, 0.5), len(gts))
    coco = [ap_101point(*pr_curve(match(dets, gts, t), len(gts)))
            for t in np.linspace(0.5, 0.95, 10)]
    print(f"{name:22s} {ap_11point(rec, prec):6.3f} {ap_all_points(rec, prec):6.3f} "
          f"{coco[0]:6.3f} {coco[5]:6.3f} {np.mean(coco):6.3f}")


# 1. Matching at IoU 0.5 and 0.75, and the precision-recall curve at 0.5.
f50, f75 = match(dets, gts, 0.5), match(dets, gts, 0.75)
rec, prec = pr_curve(f50, len(gts))
print(f"{len(gts)} ground-truth boxes, {len(dets)} detections")
print("score  IoU   @0.5 @0.75 recall precision p_interp")
for (img, box, s), a, b, r, p, pi in zip(sorted(dets, key=lambda d: -d[2]), f50, f75,
                                         rec, prec, envelope(prec)):
    best = max(iou(box, g) for gi, g in gts if gi == img)
    print(f"{s:5.2f}  {best:.2f}  {'TP' if a else 'FP':>4} {'TP' if b else 'FP':>5} "
          f"{r:6.3f} {p:9.3f} {pi:8.3f}")

# 2. AP under each convention, and what one change does to each.
print("\n                       VOC07  VOC12  AP50   AP75   AP(COCO)")
report("as above", dets)
report("+ duplicate at 0.88", dets + [(0, (7, 9, 27, 30), 0.88)])
shifted = [(2, (17, 25, 47, 49), 0.90) if d[2] == 0.90 else d for d in dets]
report("object 5 box shifted", shifted)  # IoU 0.90 -> 0.44
xy = np.random.default_rng(0).integers(0, 48, size=(19, 2))
extra = [(i % 4, (x, y, x + 16, y + 16), 0.25 - 0.01 * i) for i, (x, y) in enumerate(xy)]
report("+ 20 boxes below 0.30", dets + extra + [(3, (4, 47, 15, 60), 0.01)])

Output:

8 ground-truth boxes, 10 detections
score  IoU   @0.5 @0.75 recall precision p_interp
 0.97  0.86    TP    TP  0.125     1.000    1.000
 0.93  0.97    TP    TP  0.250     1.000    1.000
 0.90  0.90    TP    TP  0.375     1.000    1.000
 0.86  0.00    FP    FP  0.375     0.750    0.857
 0.82  0.80    TP    TP  0.500     0.800    0.857
 0.75  0.62    TP    FP  0.625     0.833    0.857
 0.62  0.69    TP    FP  0.750     0.857    0.857
 0.55  0.00    FP    FP  0.750     0.750    0.778
 0.41  0.68    TP    FP  0.875     0.778    0.778
 0.30  0.00    FP    FP  0.875     0.700    0.700

                       VOC07  VOC12  AP50   AP75   AP(COCO)
as above                0.746  0.794  0.791  0.479  0.497
+ duplicate at 0.88     0.700  0.744  0.742  0.462  0.473
object 5 box shifted    0.593  0.601  0.605  0.329  0.366
+ 20 boxes below 0.30   0.795  0.827  0.825  0.499  0.516

The envelope lifts the background false positive at 0.86 from precision 0.750 to 0.857, the best precision reached later. Recall stops at 0.875 because object 8 is never found, which costs the all-point AP (0.794) one eighth of the recall axis but costs the 11-point AP (0.746) the two sampled levels 0.9 and 1.0, two elevenths of its weight; the 101-point AP (0.791) is close to the exact area. At IoU 0.75 the loose boxes (IoU 0.62–0.69) become false positives, and AP falls to 0.479; COCO’s average over the ten thresholds is 0.497.

A duplicate of object 1 at score 0.88 is a false positive at every threshold and costs about 0.05 at IoU 0.5 but only 0.017 at 0.75, where fewer true positives follow it in the ranking. Shifting the tight box of object 5 until its IoU drops from 0.90 to 0.44 is costlier: it turns a true positive ranked third into a false positive and leaves the object missed, so every AP falls by 0.13 to 0.19. The twenty low-score boxes do not lower any AP: nineteen of them, placed at random, match nothing, and the twentieth finds object 8, extending recall to 1.

Related Metrics

• Precision and recall are the two axes of the curve AP summarizes; that article also defines COCO’s average recall (AR), the recall reached within a fixed number of detections per image, averaged over IoU thresholds.
• Intersection over union decides which detections are true positives.
• F1 at an operating point scores the single threshold a deployed detector uses, which AP does not.
• HOTA applies the same idea of averaging over localization thresholds to tracking, and adds association.