Bounding Box

A bounding box is a rectangle that marks where an object is in an image. It is the output of object detectors, the ground truth of detection datasets, and the state that trackers carry from frame to frame. Four numbers describe one, but datasets and libraries write them in incompatible ways, and mix-ups are a common source of bugs.

Definition

The bounding box of an object is the smallest axis-aligned rectangle, in the pixel coordinates of a digital image, that contains every part of the object. Axis-aligned means its sides are parallel to the image rows and columns. The term also covers rotated rectangles (oriented boxes) and 3D cuboids (3D boxes), both covered below.

Intuition

Picture the tightest frame with horizontal and vertical edges around a dog in a photo. It touches the dog at its leftmost, rightmost, topmost, and bottommost points and says nothing about its shape. That is the trade: a box is cheap to annotate, predict, and compare, at the price of including background whenever the object is not itself a rectangle.

Formal Definition

Let the object occupy a set of image points $S$, with $x$ increasing to the right and $y$ increasing downward from the top-left corner. Its axis-aligned bounding box is

$$B(S) = [x_{\min}, x_{\max}] \times [y_{\min}, y_{\max}], \qquad x_{\min} = \min_{(x, y) \in S} x, \quad x_{\max} = \max_{(x, y) \in S} x,$$

and likewise for $y$. Its width and height are $w = x_{\max} - x_{\min}$ and $h = y_{\max} - y_{\min}$.

Representations

Three parameterizations are common:

• Corner form $(x_{\min}, y_{\min}, x_{\max}, y_{\max})$, often called xyxy. PASCAL VOC annotations store xmin, ymin, xmax, ymax, and the VOC development kit states that the top-left pixel has coordinates (1, 1).
• Corner-and-size form $(x, y, w, h)$, called xywh, with $(x, y)$ the top-left corner. COCO annotations store bbox as [x, y, width, height], measured from the top-left image corner and 0-indexed, according to the COCO format documentation. MOTChallenge files (multi-object tracking) also use left, top, width, height.
• Center form $(c_x, c_y, w, h)$, called cxcywh. Ultralytics YOLO label files hold one line per object, class x_center y_center width height, normalized to 0–1 by dividing $x$ quantities by the image width $W$ and $y$ quantities by the image height $H$.

SORT’s Kalman state $[u, v, s, r]$, center plus area and aspect ratio, is another parameterization; that article gives its conversions.

The conversions follow from the definitions:

$$w = x_{\max} - x_{\min}, \quad h = y_{\max} - y_{\min}, \quad c_x = \tfrac{1}{2}(x_{\min} + x_{\max}), \quad c_y = \tfrac{1}{2}(y_{\min} + y_{\max}),$$ $$x_{\min} = c_x - \tfrac{w}{2}, \quad x_{\max} = c_x + \tfrac{w}{2}, \quad (\hat{c}_x, \hat{c}_y, \hat{w}, \hat{h}) = \left(\tfrac{c_x}{W}, \tfrac{c_y}{H}, \tfrac{w}{W}, \tfrac{h}{H}\right).$$

Normalized coordinates must be multiplied back by the size of the image they refer to.

Pixel-index conventions

A box can be read in two ways. If its corners are continuous coordinates, then $w = x_{\max} - x_{\min}$. This article uses the convention in which pixel column $j$ covers $[j, j+1)$, so pixel centers sit at half-integers; as digital image notes, libraries differ on this. If its corners are inclusive pixel indices, naming the first and last pixel columns inside the object, then a box with $x_{\max} = x_{\min}$ still contains one column, and $w = x_{\max} - x_{\min} + 1$. The VOC evaluation code uses the second reading and adds 1 to every width and height. Detectron’s box utilities keep this “+ 1”; a comment there traces it to integer pixel indices and says either choice seems to work if training and testing agree. COCO’s evaluation code uses no offset. Mixing conventions changes widths and heights by one pixel, which matters most for small objects.

Properties

Transforming boxes with the image

When data augmentation resizes, flips, or crops an image, its boxes must be transformed with it. Under continuous corner coordinates:

• Resize from $W \times H$ to $W' \times H'$ scales $x$ coordinates by $W'/W$ and $y$ coordinates by $H'/H$. Normalized coordinates do not change.
• Horizontal flip maps $x \mapsto W - x$, so the corners swap roles: $x'_{\min} = W - x_{\max}$ and $x'_{\max} = W - x_{\min}$. Under inclusive 0-based indices the map is $x \mapsto W - 1 - x$ instead.
• Crop to a window with top-left corner $(a, b)$ subtracts $(a, b)$ and clips the result to the window. A clipped box no longer covers the whole object; one clipped to zero width is dropped.

Any transformation other than axis-aligned scaling and translation breaks the box. Under a rotation, affine, or homography warp, the corners map to a quadrilateral, and the box around it is all the old box alone determines. Rotating a $w \times h$ box by $\theta$ about its center moves the corners $(\pm w/2, \pm h/2)$ to horizontal offsets $\pm(\tfrac{w}{2}\cos\theta \pm \tfrac{h}{2}\sin\theta)$, so the enclosing box has width $w|\cos\theta| + h|\sin\theta|$ and, likewise, height $w|\sin\theta| + h|\cos\theta|$. It includes the old box’s rotated corners, which the object never occupied; the tight box of the rotated object is generally smaller and needs the object’s shape, such as a mask.

Oriented boxes

Elongated objects at arbitrary angles, such as ships in aerial images or lines of scene text, fit poorly in axis-aligned boxes: the box is mostly background, and neighboring boxes overlap heavily. Oriented boxes follow the object’s orientation. The five-parameter form $(c_x, c_y, w, h, \theta)$ adds a rotation angle. The eight-parameter form lists the four vertices $(x_i, y_i)$, $i = 1, \dots, 4$, and can describe any quadrilateral. Xia et al. (2018) chose the quadrilateral form for the DOTA aerial-image dataset, arguing that the angle-based form cannot compactly enclose objects whose parts deform; DOTA orders the vertices clockwise and, for classes with a visible front such as planes and ships, places the object’s “head” at the first vertex. Ultralytics’ oriented-box labels likewise store four normalized corner points per object.

The geometry of rectangles makes the five-parameter form ambiguous. A rectangle is unchanged by a half-turn, so $\theta$ and $\theta + 180^\circ$ describe the same box, and $(c_x, c_y, w, h, \theta)$ and $(c_x, c_y, h, w, \theta + 90^\circ)$ describe the same box with width and height exchanged. Implementations therefore fix an angle range and a rule for which side is $w$. It follows that any such rule gives nearly identical rectangles near the range boundary very different parameters, so a loss on the raw parameters can heavily penalize an almost perfect prediction.

3D boxes

In driving and robotics, objects are described by cuboids in metric 3D space. KITTI’s object labels give each object a 2D box in the image (left, top, right, bottom, 0-based pixel indices), 3D dimensions (height, width, length in meters), a 3D location in camera coordinates, and a yaw angle rotation_y about the camera’s vertical axis in $[-\pi, \pi]$, according to the object development kit’s readme. The kit’s corner computation (computeBox3D.m) places the location at the center of the cuboid’s bottom face, not its centroid, which matters when converting between datasets.

What a box covers

Annotation guidelines decide what a box means, and datasets differ:

• Tightness. The VOC annotation guidelines ask for all visible pixels of the object, except where a few extra pixels (under 5%, such as a car aerial) would make the box excessively large.
• Visible or full extent. VOC boxes cover the visible part of the object, not its estimated full extent, and a truncated flag marks boxes that do not cover the whole object. MOTChallenge does the opposite: an amodal box covers the whole person, including occluded parts and parts outside the image, with a computed visibility ratio. Amodal boxes keep an object’s extent stable through occlusions, which helps tracking.
• Crowds and ignore regions. COCO marks a group of objects, such as a crowd, with iscrowd=1, and its evaluation code ignores these: detections matched to them count neither as true nor as false positives. VOC ignores objects flagged difficult, and KITTI’s evaluation ignores detections in DontCare regions.

A box can always be computed from an instance mask, as in the example below, but not the other way round.

Examples

The NumPy example computes the tight box of a filled ellipse on a 100 × 60 image, converts it between formats, checks predicted boxes against the mask after a flip and a resize, and compares two boxes for the ellipse rotated by 30°.

import numpy as np

H, W = 60, 100

def ellipse_mask(cx, cy, a, b, theta_deg=0.0):
    """Filled ellipse with semi-axes a, b, rotated by theta, sampled at pixel centers."""
    yy, xx = np.mgrid[0:H, 0:W] + 0.5           # pixel (i, j) covers [j, j+1) x [i, i+1)
    t = np.deg2rad(theta_deg)
    dx, dy = xx - cx, yy - cy
    u = dx * np.cos(t) + dy * np.sin(t)          # rotate into the ellipse's own frame
    v = -dx * np.sin(t) + dy * np.cos(t)
    return (u / a) ** 2 + (v / b) ** 2 <= 1.0

def box_from_mask(m):
    """Tight box [x_min, y_min, x_max, y_max]; corners on pixel edges (max is exclusive)."""
    rows, cols = np.nonzero(m)
    return np.array([cols.min(), rows.min(), cols.max() + 1, rows.max() + 1], float)

def xyxy_to_xywh(b):   return np.array([b[0], b[1], b[2] - b[0], b[3] - b[1]])
def xywh_to_xyxy(b):   return np.array([b[0], b[1], b[0] + b[2], b[1] + b[3]])
def xyxy_to_cxcywh(b): return np.array([(b[0] + b[2]) / 2, (b[1] + b[3]) / 2, b[2] - b[0], b[3] - b[1]])
def cxcywh_to_xyxy(b): return np.array([b[0] - b[2] / 2, b[1] - b[3] / 2, b[0] + b[2] / 2, b[1] + b[3] / 2])
def area(b):           return (b[2] - b[0]) * (b[3] - b[1])

img = ellipse_mask(cx=40, cy=25, a=22, b=10)
box = box_from_mask(img)
print("xyxy          ", box)
print("xywh (COCO)   ", xyxy_to_xywh(box))
cxcywh = xyxy_to_cxcywh(box)
print("cxcywh        ", cxcywh)
print("normalized    ", np.round(cxcywh / [W, H, W, H], 4))
print("inclusive idx ", box - [0, 0, 1, 1], "width", (box[2] - 1) - box[0] + 1)
print("round trips ok", np.allclose(xywh_to_xyxy(xyxy_to_xywh(box)), box),
      np.allclose(cxcywh_to_xyxy(cxcywh), box))

# Horizontal flip: x -> W - x, so the corners swap roles.
flipped = img[:, ::-1]
fbox = np.array([W - box[2], box[1], W - box[0], box[3]])
print("flip  predicted", fbox, "from mask", box_from_mask(flipped))

# Resize to 150 x 90 with nearest-neighbor sampling: x scales by sx, y by sy.
nW, nH = 150, 90
sx, sy = nW / W, nH / H
src_c = np.floor((np.arange(nW) + 0.5) / sx).astype(int)
src_r = np.floor((np.arange(nH) + 0.5) / sy).astype(int)
resized = img[src_r][:, src_c]
rbox = box * [sx, sy, sx, sy]
print("resize predicted", rbox, "from mask", box_from_mask(resized))

# Rotate the object by 30 degrees about its center.
rot = ellipse_mask(cx=40, cy=25, a=22, b=10, theta_deg=30)
t = np.deg2rad(30)
R = np.array([[np.cos(t), -np.sin(t)], [np.sin(t), np.cos(t)]])
corners = np.array([[box[0], box[1]], [box[2], box[1]], [box[2], box[3]], [box[0], box[3]]])
rc = (corners - [40, 25]) @ R.T + [40, 25]   # rotated corners of the original box
loose = np.array([rc[:, 0].min(), rc[:, 1].min(), rc[:, 0].max(), rc[:, 1].max()])
tight = box_from_mask(rot)
print("rotated box corners -> AABB", np.round(loose, 2), "area", round(area(loose), 1))
print("tight box of rotated mask  ", tight, "area", area(tight))
print("oriented box (cx, cy, w, h, theta):", cxcywh, "30 deg, area", area(box))
print("object pixels", img.sum(), rot.sum())

Output:

xyxy           [18. 15. 62. 35.]
xywh (COCO)    [18. 15. 44. 20.]
cxcywh         [40. 25. 44. 20.]
normalized     [0.4    0.4167 0.44   0.3333]
inclusive idx  [18. 15. 61. 34.] width 44.0
round trips ok True True
flip  predicted [38. 15. 82. 35.] from mask [38. 15. 82. 35.]
resize predicted [27.  22.5 93.  52.5] from mask [27. 22. 93. 52.]
rotated box corners -> AABB [15.95  5.34 64.05 44.66] area 1891.5
tight box of rotated mask   [20. 11. 60. 39.] area 1120.0
oriented box (cx, cy, w, h, theta): [40. 25. 44. 20.] 30 deg, area 880.0
object pixels 700 690

The same 44 × 20 box is [18, 15, 62, 35] in continuous corners and [18, 15, 61, 34] in inclusive indices. The flipped box matches the flipped mask exactly. After the 1.5× resize the predicted $y$ edges fall mid-pixel, at 22.5 and 52.5, and nearest-neighbor resampling puts the mask’s edges at 22 and 52, a discretization difference within half a pixel. After rotation, the box around the old box’s rotated corners has area 1891.5, about 1.7 times the 1120 of the tight box around the rotated mask, which matches the ellipse’s exact half-extents, $\sqrt{22^2\cos^2 30^\circ + 10^2\sin^2 30^\circ} \approx 19.7$ and $\sqrt{22^2\sin^2 30^\circ + 10^2\cos^2 30^\circ} = 14$, up to pixel sampling. The oriented box keeps the original area of 880 at any angle.

Common Misconceptions

• “Four numbers are four numbers.” The same values mean different rectangles as xyxy, xywh, or cxcywh, in pixels or normalized, 0- or 1-based.
• “A box shows the whole object.” Depending on the dataset, it may show only the visible part.

Where It Is Used

Boxes are the targets and outputs of object detectors, scored by the intersection over union of predicted and ground-truth boxes. In tracking-by-detection, per-frame boxes are the measurements that Kalman filters smooth and that trackers such as SORT and ByteTrack associate across frames. Boxes also select regions for later stages: DeepSORT embeds each box’s crop with a re-identification network, and two-stage detectors classify features pooled from candidate boxes. Oriented boxes are common in aerial imagery and scene text, and 3D boxes in driving perception, as in KITTI.