Intersection over Union

Intersection over union (IoU) measures how well a predicted region matches a ground-truth region: the area they share divided by the area they cover together. It is 1 for a perfect match and 0 when the regions do not touch. Object detectors are judged by thresholding the IoU between predicted and ground-truth bounding boxes, semantic segmentation by the IoU of pixel sets, and trackers use it to match detections across frames.

Formula

For two regions $A$ and $B$, with $|\cdot|$ denoting area (or pixel count),

$$\mathrm{IoU}(A, B) = \frac{|A \cap B|}{|A \cup B|} = \frac{|A \cap B|}{|A| + |B| - |A \cap B|}.$$

Boxes. For axis-aligned boxes in corner form $A = (x^{A}_{1}, y^{A}_{1}, x^{A}_{2}, y^{A}_{2})$ and likewise $B$, the intersection is itself a box, and its width and height are clamped at zero so that disjoint boxes give zero overlap rather than a negative area:

$$w_{\cap} = \max\!\big(0,\ \min(x^{A}_{2}, x^{B}_{2}) - \max(x^{A}_{1}, x^{B}_{1})\big), \qquad h_{\cap} = \max\!\big(0,\ \min(y^{A}_{2}, y^{B}_{2}) - \max(y^{A}_{1}, y^{B}_{1})\big),$$ $$\mathrm{IoU} = \frac{w_{\cap} h_{\cap}}{(x^{A}_{2} - x^{A}_{1})(y^{A}_{2} - y^{A}_{1}) + (x^{B}_{2} - x^{B}_{1})(y^{B}_{2} - y^{B}_{1}) - w_{\cap} h_{\cap}}.$$

Without the clamp, boxes separated in both $x$ and $y$ would give two negative factors and a positive, wrong intersection. The PASCAL VOC evaluation code reads corners as inclusive pixel indices and adds 1 to every width and height, while COCO’s does not; the bounding box article covers these conventions, which change the IoU of small boxes noticeably.

Masks. For segmentation, $A$ and $B$ are sets of pixels, and IoU counts pixels in their intersection and union. Counting each pixel of a class as a true positive (TP), false positive (FP) or false negative (FN) gives the equivalent form

$$\mathrm{IoU}_k = \frac{\mathrm{TP}_k}{\mathrm{TP}_k + \mathrm{FP}_k + \mathrm{FN}_k}, \qquad \mathrm{mIoU} = \frac{1}{K} \sum_{k=1}^{K} \mathrm{IoU}_k .$$

Everingham et al. (2010) introduced this per-class measure for the VOC2008 segmentation challenge, replacing the per-class percentage of correctly labelled pixels, which could mislead: labelling every pixel “car” gives a perfect score on the car class. Pixels marked void are excluded. The VOC2012 development kit ranks methods by the average over the twenty classes and background, the measure now called mean IoU (mIoU).

Intuition

Dividing by the union, not by the ground-truth area alone, penalizes both kinds of error. A prediction that misses part of the object loses intersection; a prediction that spills over the object gains union. A box that covers the whole object but is twice its area scores 0.5, and so does a box that covers exactly half the object and nothing else: IoU does not say which error was made.

The same ratio on finite sets is the Jaccard index, after the botanist Paul Jaccard; Kosub (2019) traces it to Jaccard’s 1901 study of the flora of the Alps and the Jura, and Jaccard (1912) is a later English-language paper on Alpine flora.

Interpretation

IoU lies in $[0, 1]$. It is symmetric, $\mathrm{IoU}(A, B) = \mathrm{IoU}(B, A)$, and it is unchanged when both regions are scaled by the same factor, because both areas scale by the factor squared, so small and large objects are treated alike in relative terms. $1 - \mathrm{IoU}$, the Jaccard distance, satisfies the triangle inequality and is a metric, for finite sets and for measures such as area (Kosub, 2019, gives two short proofs).

What counts as “good” depends on the task. In detection, IoU is not reported directly but thresholded: a prediction is a match if its IoU with a ground-truth box reaches the threshold. Everingham et al. note that VOC’s threshold of 0.5 was set deliberately low, because ground-truth boxes for objects such as a person with limbs spread are themselves somewhat subjective. As the example below shows, 0.5 still demands substantial overlap: two equal squares offset by a quarter of their side in both directions already fall to about 0.39.

When to Use It

• Object detection. Detections are matched to ground truth by box IoU, and the matches feed average precision. The VOC development kit requires the overlap to exceed 0.5, though its evaluation code also accepts exactly 0.5. COCO’s primary metric averages AP over ten thresholds, 0.50 to 0.95 in steps of 0.05, which, according to its evaluation page, rewards detectors with better localization; AP at 0.50 is reported separately as the VOC-style metric. Non-maximum suppression uses IoU between detections to remove duplicates.
• Segmentation. Mean IoU is the standard semantic segmentation score. COCO evaluates instance masks with the same AP machinery as boxes, only with IoU computed on masks.
• Tracking. SORT associates predicted track boxes with detections by maximizing IoU with the Hungarian algorithm and rejects pairs below a minimum IoU. MOTA on MOTChallenge counts a match only at IoU of at least 0.5, IDF1 can use an IoU threshold, and HOTA averages over IoU thresholds from 0.05 to 0.95. HOTA’s detection accuracy, TP / (TP + FN + FP), is itself a Jaccard index over counts.
• 3D detection. IoU extends to volumes. The KITTI 3D object benchmark requires a 3D box overlap of 70% for cars and 50% for pedestrians and cyclists.

Failure Cases

• Small objects. Shifting an $s \times s$ square by $d$ pixels along $x$ leaves an intersection of $s(s - d)$ and a union of $s(s + d)$, so IoU is $(s - d)/(s + d)$. One pixel costs an 8-pixel box 22% of its IoU and a 128-pixel box under 2%, so annotation noise of a pixel or two can decide whether a small detection passes a 0.5 or 0.75 threshold.
• Non-rectangular objects. Box IoU ignores shape. In the example, an elongated, tilted object fills 58% of its box; shifting the prediction by 5 pixels along or across the object gives almost the same box IoU (0.74 and 0.72) but very different mask IoU (0.80 and 0.58).
• No overlap, no information. All disjoint pairs score 0, whether the prediction misses by one pixel or by the width of the image. For tracking, fast or small objects that move more than their own size between frames get zero IoU with their own track; for training, the loss $1 - \mathrm{IoU}$ has zero gradient.
• Class imbalance in pixel scores. Mean IoU weights each class equally, which is its purpose; pixel accuracy is dominated by large classes such as background. In the example, pixel accuracy is 0.958 while one class has an IoU of only 0.70. A class absent from both prediction and ground truth makes the ratio 0/0, so an implementation must decide whether to skip it or score it.

Variants

Dice coefficient (F1). The Dice coefficient $D = 2|A \cap B| / (|A| + |B|)$ is the F1 score of the pixel classification. Because $|A| + |B| = |A \cup B| + |A \cap B|$, dividing numerator and denominator by $|A \cup B|$ gives

$$D = \frac{2\,\mathrm{IoU}}{1 + \mathrm{IoU}}, \qquad \mathrm{IoU} = \frac{D}{2 - D}.$$

The map is monotonic, so for single pairs of regions the two rank predictions identically, and $D \ge \mathrm{IoU}$, with equality only at 0 and 1. Averages over images or classes can rank methods differently, since the map is not linear.

IoU as a loss and GIoU. $1 - \mathrm{IoU}$ can serve directly as a box regression loss, but it is flat for disjoint boxes. Rezatofighi et al. (2019) proposed Generalized IoU: with $C$ the smallest enclosing box (in general, convex shape) of $A$ and $B$,

$$\mathrm{GIoU} = \mathrm{IoU} - \frac{|C \setminus (A \cup B)|}{|C|}.$$

GIoU lies in $[-1, 1]$, never exceeds IoU, equals it when the boxes fill their enclosing box, and approaches $-1$ as disjoint boxes move far apart, so $1 - \mathrm{GIoU}$ has a gradient that pulls a non-overlapping prediction toward the target. The authors reported gains when it replaced the default box losses of YOLO v3, Faster R-CNN and Mask R-CNN on PASCAL VOC and COCO.

DIoU and CIoU. Zheng et al. (2020) observed that the GIoU loss degrades to the IoU loss once one box contains the other, and converges slowly. Their Distance-IoU loss adds to $1 - \mathrm{IoU}$ the squared distance between box centers divided by the squared diagonal of $C$; the Complete-IoU loss adds a further aspect-ratio term. They also used DIoU in place of IoU in non-maximum suppression.

Rotated and 3D boxes. For oriented boxes, the intersection is a convex polygon, computed by clipping one rectangle against the other, and IoU is the ratio of polygon areas. 3D IoU uses volumes; for boxes rotated only about the vertical axis, it follows from the geometry that the intersection volume is the overlap area of the bird’s-eye-view polygons times the overlap in height.

Example

The NumPy code uses a tilted ellipse on a 120 × 80 image and reproduces the numbers quoted above.

import numpy as np

H, W = 80, 120
yy, xx = np.mgrid[0:H, 0:W] + 0.5                # pixel centers; pixel (i, j) covers [j, j+1) x [i, i+1)

def ellipse(cx, cy, a, b, deg=0.0):
    t = np.deg2rad(deg)
    u = (xx - cx) * np.cos(t) + (yy - cy) * np.sin(t)
    v = -(xx - cx) * np.sin(t) + (yy - cy) * np.cos(t)
    return (u / a) ** 2 + (v / b) ** 2 <= 1.0

def box_of(m):                                    # tight box [x1, y1, x2, y2], continuous corners
    r, c = np.nonzero(m)
    return np.array([c.min(), r.min(), c.max() + 1, r.max() + 1], float)

def box_iou(a, b):
    iw = max(0.0, min(a[2], b[2]) - max(a[0], b[0]))
    ih = max(0.0, min(a[3], b[3]) - max(a[1], b[1]))
    inter = iw * ih
    area = lambda r: (r[2] - r[0]) * (r[3] - r[1])
    return inter / (area(a) + area(b) - inter)

def mask_iou(p, g):
    return (p & g).sum() / (p | g).sum()

def giou(a, b):
    area = lambda r: (r[2] - r[0]) * (r[3] - r[1])
    iw = max(0.0, min(a[2], b[2]) - max(a[0], b[0]))
    ih = max(0.0, min(a[3], b[3]) - max(a[1], b[1]))
    inter = iw * ih
    union = area(a) + area(b) - inter
    c = [min(a[0], b[0]), min(a[1], b[1]), max(a[2], b[2]), max(a[3], b[3])]
    return inter / union - (area(c) - union) / area(c)

# 1. A tilted elliptical object, and predictions shifted by 5 pixels along and across it.
gt = ellipse(60, 40, 30, 12, deg=30)
gb = box_of(gt)
fill = gt.sum() / ((gb[2] - gb[0]) * (gb[3] - gb[1]))
print("GT box", gb, f" object fills {fill:.0%} of its box")
for name, (dx, dy) in [("along", (4, 3)), ("across", (-3, 4))]:
    m = ellipse(60 + dx, 40 + dy, 30, 12, deg=30)
    print(f"shift {name:6s} box IoU {box_iou(box_of(m), gb):.3f}  mask IoU {mask_iou(m, gt):.3f}")
pred = ellipse(64, 43, 30, 12, deg=30)

# 2. Same shift, different object sizes: squares of side s moved by d pixels in x and y.
print("side  d=1 (x)  d=2 (x)  d=2 (x and y)")
for s in [8, 16, 32, 64, 128]:
    g = np.array([0, 0, s, s], float)
    r = [box_iou(g + [d, 0, d, 0], g) for d in (1, 2)] + [box_iou(g + 2, g)]
    print(f"{s:4d}  " + "  ".join(f"{v:7.3f}" for v in r))

# 3. Dice versus IoU on the masks of part 1.
J = mask_iou(pred, gt)
dice = 2 * (pred & gt).sum() / (pred.sum() + gt.sum())
print(f"Dice {dice:.3f}   2J/(1+J) {2 * J / (1 + J):.3f}")

# 4. Mean IoU for a 3-class label map: background 0, ellipse 1, rectangle 2.
label = np.zeros((H, W), int); label[gt] = 1; label[10:30, 90:112] = 2
seg = np.zeros((H, W), int); seg[pred] = 1; seg[12:30, 92:116] = 2
ious = [((seg == k) & (label == k)).sum() / ((seg == k) | (label == k)).sum() for k in range(3)]
print("per-class IoU", np.round(ious, 3), " mean IoU", round(float(np.mean(ious)), 3),
      " pixel accuracy", round(float((seg == label).mean()), 3))

# 5. Disjoint boxes: IoU is 0 at every gap, GIoU keeps falling with distance.
g = np.array([10, 10, 30, 30], float)
print("gap   IoU    GIoU   d(1-IoU)/dx  d(1-GIoU)/dx")
for gap in [2, 10, 40]:
    p = np.array([30 + gap, 10, 50 + gap, 30], float)
    shift = lambda e: p + [e, 0, e, 0]
    h = 1e-4
    dI = 0.0 - (box_iou(shift(h), g) - box_iou(shift(-h), g)) / (2 * h)
    dG = 0.0 - (giou(shift(h), g) - giou(shift(-h), g)) / (2 * h)
    print(f"{gap:3d}  {box_iou(p, g):.3f}  {giou(p, g):+.3f}  {dI:11.4f}  {dG:12.4f}")

Output:

GT box [33. 22. 87. 58.]  object fills 58% of its box
shift along  box IoU 0.737  mask IoU 0.803
shift across box IoU 0.723  mask IoU 0.583
side  d=1 (x)  d=2 (x)  d=2 (x and y)
   8    0.778    0.600    0.391
  16    0.882    0.778    0.620
  32    0.939    0.882    0.784
  64    0.969    0.939    0.884
 128    0.984    0.969    0.940
Dice 0.890   2J/(1+J) 0.890
per-class IoU [0.951 0.803 0.703]  mean IoU 0.819  pixel accuracy 0.958
gap   IoU    GIoU   d(1-IoU)/dx  d(1-GIoU)/dx
  2  0.000  -0.048       0.0000        0.0227
 10  0.000  -0.200       0.0000        0.0160
 40  0.000  -0.500       0.0000        0.0062

The table matches $(s - d)/(s + d)$: $7/9 = 0.778$ for an 8-pixel box shifted by one pixel. Shifted two pixels in both directions, the 8-pixel box (0.391) would fail VOC’s 0.5 threshold; the 64-pixel box keeps 0.884. For the disjoint boxes, IoU and its gradient are zero at every gap, while GIoU falls from −0.048 to −0.5; the positive derivative of $1 - \mathrm{GIoU}$ with respect to the prediction’s $x$ position means gradient descent moves the box back toward the target.

Related Metrics

Average precision, built on precision and recall, summarizes detection from IoU-thresholded matches; the tracking metrics MOTA, IDF1 and HOTA use IoU to decide matches. For dense motion fields, endpoint error plays the localization role that IoU plays for regions.