Image Annotation

Image annotation, also called image labelling or data labelling, is the work of attaching human judgements to images: which classes appear, where each object is, which pixels belong to it, where its joints are, how it moves through a video. Supervised object detection and image segmentation models learn from these labels, and benchmarks score models against them, so the labels set both what a model can learn and what an evaluation can measure. Annotation is a technique in its own right because the same image can be labelled well or badly, cheaply or expensively, and the difference comes from choices made before anyone draws a box: the label type, the written guidelines, how annotators are trained and checked, how many people label each item and how their answers are combined, and how much of the work a model does.

Purpose

Annotation turns images into training and test data with known answers. Its problem is that the answers are produced by people, who are slow, cost money, and disagree. Three quantities trade against one another:

• Cost: annotator time per image or per object, which ranges from about a second for a yes-or-no question to more than an hour for a densely labelled street scene.
• Quality: how close labels are to what the guidelines intend, measured by agreement between annotators or with experts.
• Coverage: whether every object of every class is labelled (exhaustive annotation) or only some, which decides what the absence of a label means.

Every protocol in this article is a way of buying more of one of these with less of another.

When to Use It

Annotation is needed whenever a model is trained or evaluated with supervision for a task whose answers are not already recorded: new classes, a new domain (medical scans, factory parts, aerial images), a new sensor, or a test set for a model that will be deployed. A test set needs annotation even when training does not: a model trained without labels, by self-supervised learning, still needs labelled examples to be measured. The train, validation and test split decides which labels play which role.

Annotation is the wrong tool, or only part of the tool, when the answer can be measured instead of judged: depth from a LiDAR or structured-light sensor, motion from synthetic rendering with known flow, or pose from a motion-capture rig. Synthetic data gives exact labels for free, at the price of images that differ from real ones (our reasoning). When labels must be judged, it often pays to collect a cheaper label type and let a model infer the rest, as the variants below show.

How It Works

Label types

Each label type answers a different question and costs a different amount of time.

• Image-level tags state which classes are present, one per image (as in ImageNet) or several. A tag says nothing about where the object is. When a vocabulary is large, a tag must also say whether a class was checked: Open Images records verified positive and verified negative labels and leaves all other classes unknown (Kuznetsova et al., 2020).
• Bounding boxes give an axis-aligned rectangle around each object. Open Images defines the perfect box as the smallest box containing all visible parts of the object. Boxes often carry flags for cases a rectangle cannot express: truncated, occluded or difficult objects in PASCAL VOC, and boxes around a group of touching objects (crowd regions in COCO, the group-of attribute in Open Images).
• Polygons outline an object with a sequence of clicked vertices. COCO’s instance outlines average 24.5 vertices per instance, and 33.4 for instances of at least 80 × 40 pixels (Benenson, Popov & Ferrari, 2019). A polygon is rasterized into a mask for training.
• Masks assign each pixel a label: a class for semantic segmentation, an instance identity for instance segmentation, or both for panoptic segmentation. Masks can be painted with brushes, drawn as polygons, or produced interactively. Guidelines usually reserve a void or ignore label for pixels the annotator cannot decide, which training and evaluation skip; Cityscapes instructed annotators to use it when unsure (Cordts et al., 2016).
• Keypoints mark named points such as body joints, each with a visibility flag. COCO’s 17 person keypoints are stored with a flag for not labelled, labelled but not visible, and visible (human pose estimation).
• Tracks link the same object across the frames of a video, usually as one box per frame with an identity that persists while the object is in view. The MOTChallenge protocol, for example, continues a track through an occlusion while the position can be inferred and starts a new identity after a long, ambiguous one (multi-object tracking).
• 3D cuboids are boxes in 3D, drawn in a point cloud or a fused sensor view. KITTI stores each object as a 3D box tracklet in the LiDAR’s coordinates, with the object’s height, width and length, its translation and its rotation in each frame; only the yaw angle is given, the other two being assumed near zero, and each box carries occlusion and truncation levels (Geiger et al., 2013).

Other forms, such as single points, scribbles, captions and relationships between pairs of boxes, sit between these. The label type is chosen from the task: a detector needs boxes, a segmentation model needs masks, and a cheaper label type can stand in when a training method can use it (Bearman et al., 2016).

Guidelines

Written guidelines turn a task name into decisions an annotator can make the same way every time. A one-line definition is never enough. Kuznetsova et al. (2020) list corner cases that their box definition did not settle, such as whether straps are part of a camera or water part of a fountain, and found an unexpected cultural difference: in some parts of the world, annotators took a human hand to include the whole arm. Their remedy was a hand-drawn perfect box on two examples for each of the 600 classes, plus the common mistakes found in pilot studies for 20% of the classes, shown to the annotator right before each session on that class.

Guidelines are refined by doing the task. For the Segment Anything data engine, the research team first completed 30 annotation tasks to find problems with the tool and agree on how to handle complex cases, then shared videos of themselves annotating and met the annotators weekly to answer questions (Kirillov et al., 2023, arXiv appendix F.2). The same appendix notes how subjective the task remained: one annotator may segment two boots as one mask and another as two.

Workflow

A typical pipeline has the same stages whatever the label type:

1. Instructions and training. Annotators read the guidelines and practise on items with known answers, with feedback on each mistake.
2. Qualification. Only annotators who pass a test move on to real work.
3. Annotation. Items are labelled in batches, often one class per batch to avoid switching tasks.
4. Quality control. Hidden test items, redundant labels and review catch errors while work continues.
5. Aggregation. Redundant labels are combined into one, and items with too much disagreement are sent back or discarded.

Large datasets divide the work into stages, each a simpler task: COCO (Lin et al., 2015) first asked which categories are present, then where each instance is, and only then had each instance outlined and the outlines verified, and LVIS added stages that check that no instance was missed and that collect images known not to contain a category. Those pipelines are described in the dataset articles; the sections below explain the mechanisms they share.

How to Apply It

Measuring agreement

Annotation quality is usually measured by having the same items labelled twice, by different annotators who do not see each other’s work, and comparing the results:

• Boxes and masks are compared with intersection over union (IoU). When an image has several instances, the two sets are matched first; LVIS finds the matching that maximizes total mask IoU, discards matches below 0.5, and reports the mean mask IoU of the matched pairs together with an F1 score for which instances were found at all (Gupta, Dollár & Girshick, 2019). A boundary F-measure catches differences that mask IoU hides on large objects: LVIS illustrates a pair of masks with a mask IoU of 0.91 but a boundary score of 0.82.
• Pixel labels are compared by the share of pixels given the same class. In Cityscapes, 30 images annotated twice agreed on 96% of pixels, and on 98% once pixels that either annotator marked void were left out (Cordts et al., 2016).
• Keypoints are compared by distance relative to object size. COCO derived its keypoint tolerances from 5,000 redundantly annotated validation images: the spread of annotators’ clicks, relative to the person’s scale, was 0.026 for the nose and 0.107 for the hips, so the official metric forgives hip errors about four times larger than nose errors (COCO keypoint evaluation page).
• Categorical labels are compared by the share of items given the same class. When one class dominates, raw agreement looks high even for careless annotators, and chance-corrected statistics such as Cohen’s kappa adjust for it.

Published agreement figures give a sense of the ceiling. Boxes in PASCAL VOC 2007 agreed with boxes fitted to the same dataset’s masks at 88% mean IoU, a level that perturbing each box side by 4 pixels also reaches (Papadopoulos et al., 2017). Open Images boxes redrawn by a second annotator agreed with the originals at 0.87 mean IoU (Kuznetsova et al., 2020). For masks, LVIS’s pipeline after the first stage, run twice on 5,000 images, reproduced its own masks at 0.85 mean IoU, and two experts outlining the same 100 objects agreed at a mean mask IoU of 0.91 to 0.95 (95% confidence interval for COCO’s objects; Gupta, Dollár & Girshick, 2019). Two people therefore rarely draw the same object identically, and a model compared against one of them cannot be measured more finely than they agree (our reasoning; the benchmarking article discusses scores that approach the level of label noise).

Quality control

Three mechanisms recur in published protocols:

• Qualification tests with feedback. In Papadopoulos et al. (2017), annotators clicked extreme points on 5 images with known masks, saw where each click should have fallen (any pixel within 10 pixels of the true extreme region was accepted), and could retry until all their clicks were inside. Open Images raised the bar to 800 practice boxes, passed only at an IoU above 0.84 with the ground truth and at most 20 seconds per box (Kuznetsova et al., 2020).
• Gold questions. Items with known answers are hidden among real work. Papadopoulos et al. (2017) placed one such image in every batch of 10; an annotator whose clicks missed it could not submit the batch. LVIS embedded gold sets in most of its stages and used them to direct work toward the more reliable annotators (Gupta, Dollár & Girshick, 2019).
• Redundancy and consensus. Several annotators answer the same question, and the answers are combined. Open Images took the majority of 7 crowd annotators for each image-level verification, and of 3 or 1 trained internal annotators depending on how difficult the class was (Kuznetsova et al., 2020). LVIS accepted a mask only when 4 of up to 5 raters judged it high quality and sent it back for redrawing as soon as 2 rejected it (Gupta, Dollár & Girshick, 2019).

Review by a separate team completes the loop. In the Segment Anything data engine, annotators trained for a day with the vendor and then on a training queue, spending about a week there before vendor and research staff spot-checked their masks and moved them to production; afterwards a QA team gave daily feedback, and the research team reviewed annotations weekly and shared common mistakes (Kirillov et al., 2023, arXiv appendix F.2).

Example: why consensus works, and when it does not

The script below simulates annotation with exactly known ground truth. Each object is a rendered ellipse with a small “handle” sticking out of it, on a $128 \times 128$ canvas. Five simulated annotators label every object twice: as a polygon of 12 to 24 vertices clicked near the boundary, and as a box built from four extreme clicks. Each annotator has a bias on each object, drawing loosely or tightly by a random number of pixels, plus independent click noise of 1.5 pixels, and with some probability reads the guideline differently and leaves the handle out. The consensus mask is a per-pixel majority vote and the consensus box is the coordinate-wise median of the five boxes. The script compares individuals, pairs of annotators, and the consensus against the truth, over 20 seeds, and then varies how many annotators misread the guideline. It runs in about 5 seconds with NumPy 2.3.5 and scikit-image 0.25.2.

import numpy as np
from skimage.draw import ellipse, polygon
from skimage.measure import find_contours

H = W = 128
K = 5  # annotators per object


def render_object(rng):
    """Ground-truth mask: an elliptical body plus a smaller 'handle' ellipse."""
    m = np.zeros((H, W), bool)
    cy, cx = rng.uniform(44, 84, 2)
    ry, rx = rng.uniform(16, 26, 2)
    m[ellipse(cy, cx, ry, rx, (H, W), rotation=rng.uniform(0, np.pi))] = True
    body = m.copy()
    ang = rng.uniform(0, 2 * np.pi)                     # handle sticks out of the body
    hy, hx = cy + 1.1 * ry * np.sin(ang), cx + 1.1 * rx * np.cos(ang)
    m[ellipse(hy, hx, 7, 4, (H, W), rotation=-ang)] = True
    return m, body


def polygon_mask(target, rng, n_vertices, sigma, bias):
    """An annotator's polygon: n_vertices clicked near the boundary of target, each moved
    outward by the annotator's bias (pixels) plus Gaussian click noise sigma."""
    c = max(find_contours(target.astype(float), 0.5), key=len)    # (row, col) points
    idx = np.linspace(0, len(c), n_vertices, endpoint=False).astype(int)
    v = c[idx]
    centre = c.mean(axis=0)
    out = (v - centre) / np.linalg.norm(v - centre, axis=1, keepdims=True)
    v = v + bias * out + rng.normal(0, sigma, v.shape)
    m = np.zeros((H, W), bool)
    m[polygon(v[:, 0], v[:, 1], (H, W))] = True
    return m


def extreme_box(target, rng, sigma, bias):
    """A box from four extreme clicks (top, bottom, left, right) with noise and bias."""
    rows, cols = np.nonzero(target)
    y0, y1, x0, x1 = rows.min(), rows.max() + 1, cols.min(), cols.max() + 1
    y0, x0 = np.array([y0, x0]) - bias + rng.normal(0, sigma, 2)
    y1, x1 = np.array([y1, x1]) + bias + rng.normal(0, sigma, 2)
    return np.array([x0, y0, x1, y1])


def mask_iou(a, b):
    return (a & b).sum() / (a | b).sum()


def box_iou(a, b):
    iw = max(0.0, min(a[2], b[2]) - max(a[0], b[0]))
    ih = max(0.0, min(a[3], b[3]) - max(a[1], b[1]))
    inter = iw * ih
    area = lambda r: (r[2] - r[0]) * (r[3] - r[1])
    return inter / (area(a) + area(b) - inter)


def annotate(gt, body, rng, k, p_skip):
    """k simulated annotators; each has a personal bias (loose or tight outlines) and,
    with probability p_skip, follows a different reading of the guideline and leaves
    the handle out."""
    masks, boxes = [], []
    for _ in range(k):
        bias = rng.normal(0, 1.0)
        target = body if rng.random() < p_skip else gt
        masks.append(polygon_mask(target, rng, n_vertices=int(rng.integers(12, 25)),
                                  sigma=1.5, bias=bias))
        boxes.append(extreme_box(target, rng, sigma=1.5, bias=bias))
    return masks, np.array(boxes)


def run(seed, n_objects=40, k=K, p_skip=0.2):
    rng = np.random.default_rng(seed)
    keys = ["m_ind", "m_pair", "m_cons", "m_best", "m_k", "handle", "b_ind", "b_pair", "b_cons"]
    r = {key: [] for key in keys}
    for _ in range(n_objects):
        gt, body = render_object(rng)
        rows, cols = np.nonzero(gt)
        gt_box = np.array([cols.min(), rows.min(), cols.max() + 1, rows.max() + 1])
        masks, boxes = annotate(gt, body, rng, k, p_skip)
        pairs = [(i, j) for i in range(k) for j in range(i + 1, k)]
        r["m_ind"].append(np.mean([mask_iou(m, gt) for m in masks]))
        r["m_pair"].append(np.mean([mask_iou(masks[i], masks[j]) for i, j in pairs]))
        vote = np.mean(masks, axis=0) > 0.5                       # per-pixel majority vote
        r["m_cons"].append(mask_iou(vote, gt))
        r["m_best"].append(max(mask_iou(m, gt) for m in masks))
        handle = gt & ~body
        r["handle"].append((vote & handle).sum() > 0.5 * handle.sum())
        r["b_ind"].append(np.mean([box_iou(b, gt_box) for b in boxes]))
        r["b_pair"].append(np.mean([box_iou(boxes[i], boxes[j]) for i, j in pairs]))
        r["b_cons"].append(box_iou(np.median(boxes, axis=0), gt_box))  # coordinate median
        # majority vote over the first 1, 3 and 5 annotators
        r["m_k"].append([mask_iou(np.mean(masks[:n], axis=0) > 0.5, gt) for n in (1, 3, 5)])
    return {key: np.array(v) for key, v in r.items()}


r = run(seed=0)
print(f"seed 0, {len(r['m_ind'])} objects, {K} annotators each (mean IoU)")
print("            individual vs truth   annotator vs annotator   consensus vs truth")
print(f"masks           {r['m_ind'].mean():.3f}                 {r['m_pair'].mean():.3f}"
      f"                   {r['m_cons'].mean():.3f}")
print(f"boxes           {r['b_ind'].mean():.3f}                 {r['b_pair'].mean():.3f}"
      f"                   {r['b_cons'].mean():.3f}")
print("objects where the consensus mask beats the average annotator: "
      f"{(r['m_cons'] > r['m_ind']).mean():.0%}, beats the best annotator: "
      f"{(r['m_cons'] > r['m_best']).mean():.0%}")

print("\nover 20 seeds                         min     median  max")
g_m, g_b, pair, votes = [], [], [], []
for seed in range(20):
    r = run(seed)
    g_m.append(r["m_cons"].mean() - r["m_ind"].mean())
    g_b.append(r["b_cons"].mean() - r["b_ind"].mean())
    pair.append(r["m_pair"].mean())
    votes.append(r["m_k"].mean(axis=0))
for name, v in [("mask gain, consensus - individual", g_m),
                ("box gain, consensus - individual", g_b),
                ("mask IoU, annotator vs annotator", pair)]:
    print(f"{name:36s} {min(v):.3f}   {np.median(v):.3f}   {max(v):.3f}")
print("mask IoU vs truth, vote of 1 / 3 / 5 annotators (median): "
      + " / ".join(f"{x:.3f}" for x in np.median(votes, axis=0)))

# Guideline ambiguity: vary the share of annotators who leave the handle out.
print("\nshare leaving the handle out        0.0    0.2    0.4    0.6    0.8")
res = [[run(s, p_skip=p) for s in range(10)] for p in (0.0, 0.2, 0.4, 0.6, 0.8)]
for name, key in [("individual vs truth", "m_ind"), ("consensus vs truth", "m_cons"),
                  ("consensus keeps the handle", "handle")]:
    print(f"{name:34s} " + "  ".join(f"{np.median([r[key].mean() for r in rs]):.3f}"
                                     for rs in res))
print("(medians over 10 seeds)")

seed 0, 40 objects, 5 annotators each (mean IoU)
            individual vs truth   annotator vs annotator   consensus vs truth
masks           0.864                 0.828                   0.911
boxes           0.858                 0.808                   0.920
objects where the consensus mask beats the average annotator: 100%, beats the best annotator: 60%

over 20 seeds                         min     median  max
mask gain, consensus - individual    0.046   0.049   0.053
box gain, consensus - individual     0.051   0.058   0.062
mask IoU, annotator vs annotator     0.821   0.833   0.838
mask IoU vs truth, vote of 1 / 3 / 5 annotators (median): 0.869 / 0.904 / 0.919

share leaving the handle out        0.0    0.2    0.4    0.6    0.8
individual vs truth                0.871  0.865  0.859  0.855  0.850
consensus vs truth                 0.920  0.916  0.908  0.899  0.893
consensus keeps the handle         0.875  0.688  0.350  0.162  0.025
(medians over 10 seeds)

Two annotators agree with each other less well than each agrees with the truth, because their independent errors add up: inter-annotator IoU understates how accurate each annotator is. The majority vote and the median box remove much of that independent noise. The consensus mask beats the average annotator on every object of seed 0 and even the best of the five on more than half of them, and the gain holds over all 20 seeds, with little spread. Most of the gain comes from the first two extra annotators; going from three to five adds less than going from one to three, consistent with the three to seven labels per item in the protocols above (our reading).

The last block shows the limit. A vote removes errors that annotators make independently, not an error most of them share. As more annotators leave the handle out, the consensus keeps it on fewer objects, and once the misreading is the majority’s, the vote adopts it. Even with no misreading, the vote drops the handle on about one object in eight, because a thin part that noisy outlines cover inconsistently falls below the majority; voting smooths thin structures. The IoU figures barely register either effect, since the handle is a small part of each object, so a guideline question has to be checked on the cases it concerns, not read off an average score (our reading of the simulation).

Variants

Faster manual labels

The cost of a label depends on the interaction, not only on the label type. Drawing a box the traditional way means clicking an imaginary corner outside the object, dragging to the opposite corner and adjusting the sides. In the protocol of Su, Deng and Fei-Fei (2012), used for ILSVRC, the median times were 25.5 seconds to draw a box, 9.0 seconds to verify it and 7.8 seconds to check whether other instances remained, as reported by Papadopoulos et al. (2017) and Kuznetsova et al. (2020). Extreme clicking instead asks for four points on the object itself, its top, bottom, left-most and right-most points, which define the box and need no adjustment. On PASCAL VOC it took 7.0 seconds per box on VOC 2007 and 7.2 on VOC 2012, the boxes agreed with the PASCAL ground truth as well as the ground truth’s two versions agree with each other, and Fast R-CNN detectors trained on them reached the same mAP (Papadopoulos et al., 2017, Table 1). Open Images drew about 90% of its boxes this way at 7.4 seconds per box (Kuznetsova et al., 2020). The four points also lie on the object’s boundary, and using them to initialize GrabCut, a graph-cut segmentation method, gave masks 2–4% mIoU better than initializing it from a box.

Verification instead of drawing

Answering yes or no is faster than drawing. In box verification, a detector proposes a box and the annotator accepts or rejects it; rejected regions are excluded and the detector is retrained (Papadopoulos et al., 2016, as described by Kuznetsova et al., 2020). Open Images trained its verifiers to accept boxes whose IoU with the perfect box exceeds 0.7; after warm-up a verification took 3.5 seconds, or 8.5 seconds per box produced once rejections are counted, and redrawn boxes agreed with the verified ones at 0.77 mean IoU, against 0.87 for extreme clicking. The same idea scales image-level labels: Open Images ran image classifiers over every image to propose candidate classes, and humans only verified the candidates (Kuznetsova et al., 2020). The design choice is that verification can remove false positives but cannot recover what the model never proposed.

Interactive segmentation

In interactive segmentation, an annotator and a segmentation model take turns: the model proposes a mask, the annotator clicks where it is wrong, and the model updates. Classic tools such as GrabCut used boxes or scribbles to seed graph cuts. Benenson, Popov and Ferrari (2019) ran the idea at scale with a deep model, starting from each object’s box and accepting up to 4 corrective clicks per round. Measured against carefully painted masks on COCO, three rounds of corrective clicks produced masks at 84% mIoU in 34 seconds per instance, while COCO’s original polygons reached 82% at an estimated 108 seconds (for instances of at least 80 × 40 pixels); the painted reference masks themselves took about 136 seconds each (138 in a later section of the paper) and agreed with a second painting at 90% mIoU. The same campaign produced masks for 2.5 million instances in Open Images.

Model-assisted annotation and data engines

When a model is retrained on the labels it helps produce, annotation becomes a loop, which Kirillov et al. (2023) call a data engine. For SAM it had three stages. In the assisted-manual stage, professional annotators clicked foreground and background points in a browser tool running SAM in real time, with brush and eraser tools for pixel-level fixes, and were encouraged to move to the next image once a mask took more than 30 seconds; as SAM was retrained six times, the average time per mask fell from 34 to 14 seconds and the masks per image rose from 20 to 44, for 4.3 million masks on 120,000 images. In the semi-automatic stage, a detector pre-filled confident masks and annotators labelled what was left: 5.9 million more masks, at 34 seconds each because the remaining objects were harder. In the fully automatic stage, SAM alone produced 1.1 billion masks on 11 million images. Professional annotators then corrected the automatic masks of 500 images (about 50,000 masks): 94% of the automatic masks had an IoU above 90% with their corrected versions.

The same loop applies to video. Annotating every frame of a track is expensive, so tools interpolate boxes between keyframes the annotator draws, commonly by linear interpolation. Kuznetsova et al. (2021) replaced it with a model that interpolates and extrapolates boxes from image content and suggests which frame to annotate next; in simulation it needed 60% fewer manually drawn boxes than linear interpolation, and in a study with human annotators it cut measured annotation time by 50%.

Model assistance carries a risk of its own: a proposal the annotator accepts without close inspection carries the model’s errors into the labels that will later train and evaluate the next model (our reasoning). Verification protocols and audits against fully manual labels, such as SAM’s comparison with corrected masks, are the checks against it.

Weaker labels

Instead of making strong labels cheaper, a method can learn from weaker ones. Bearman et al. (2016) estimated, from times reported in earlier work, about 20 seconds per PASCAL VOC image for image-level labels over 20 classes and 239.7 seconds for full segmentation masks, and, from click times they measured, 22.1 seconds when one point is also clicked on an instance of each present class, and showed that point supervision improves a segmentation network over image-level labels for little extra time. Cityscapes’ coarse annotations, under 7 minutes per image against more than 1.5 hours for its fine ones, trade boundary accuracy for coverage in the same spirit (Cordts et al., 2016).

Best Practices

• Write guidelines with pictures, and pilot them. Show a correct example and the common mistakes for each class, as Open Images did, and revise the guidelines after a pilot round by the people who designed the task.
• Train with feedback and qualify before production. Feedback on each practice item, with unlimited retries, lets annotators reach the standard; a qualification threshold keeps out those who have not.
• Keep one task per batch. Labelling one class at a time in short batches avoids task switching, which Papadopoulos et al. (2017) cite as a source of slower and less accurate responses.
• Hide gold questions in every batch and track each annotator’s accuracy on them over time, not only at qualification.
• Collect redundant labels where errors are independent, on a sample if not on everything, and use them both to build consensus labels and to measure agreement.
• Double-annotate part of the test set and report the agreement, so that model scores can be read against it.
• Record coverage explicitly: which classes were checked on which images, which instances were left out, which regions are void or crowd. LVIS’s federated design, with negative sets and not-exhaustive flags, exists to make missing labels interpretable.
• Document the process, including who annotated and how they were paid; Gebru et al. (2021) include these among the questions a datasheet for a dataset should answer.

Pitfalls

• Ambiguous classes and boundaries. Most disagreement comes from cases the guidelines did not settle, such as the camera strap or the hand that includes an arm. A shared misreading is not removed by voting, as the example shows.
• Treating missing labels as negatives. When annotation is not exhaustive, an unlabelled object is not background; training or evaluating as if it were penalizes correct predictions.
• Reading agreement as accuracy. Annotators can agree on the same wrong answer, and agreement between two annotators also understates each one’s accuracy, as the simulation shows; agreement measures consistency with respect to the guidelines.
• Label errors in the test set. Errors in training labels cost some accuracy; errors in test labels change the measurement itself (below).
• Rasterization and format conventions. Polygons, run-length encoded masks and box conventions (corner against centre coordinates, inclusive against exclusive pixel edges) differ between datasets and tools. The COCO annotation format, stored as JSON, and tools such as CVAT and LabelMe each make their own choices, and a conversion that shifts every box by a pixel lowers measured IoU everywhere.
• Biased annotation. Who annotates, what they are shown and which images are chosen shape a dataset’s view of the world; dataset bias covers these effects.

Evidence and Impact

Cost

Published times give a scale for planning, with the caveat that each was measured under its own protocol, image set and quality target:

Label	Reported time	Source
Image-level label	about 1 s per class asked (assumed from earlier work)	Bearman et al. (2016)
Point on one instance of a class	2.4 s median (first instance)	Bearman et al. (2016)
Box verification	3.5 s per verification, 8.5 s per box produced	Kuznetsova et al. (2020)
Box, extreme clicking	7.0 s (VOC 2007), 7.4 s (Open Images)	Papadopoulos et al. (2017); Kuznetsova et al. (2020)
Box, traditional drawing with verification	25.5 s + 9.0 s (medians)	Su et al. (2012), as reported by Papadopoulos et al. (2017)
Mask, SAM-assisted clicks	34 s falling to 14 s per mask	Kirillov et al. (2023)
Mask, interactive corrective clicks	34 s per instance	Benenson et al. (2019)
Mask, COCO polygon	79 s per instance, excluding verification	COCO, as reported by Papadopoulos et al. (2017)
Mask, careful free painting	about 136 s per instance	Benenson et al. (2019)
Fine pixel labels for a street scene	over 1.5 h per image, with quality control	Cordts et al. (2016)

The totals for the traditional box differ between the two papers that report Su et al.’s times: Papadopoulos et al. (2017) count 34.5 seconds (drawing and verification, for one object per class) and note that re-drawing rejected boxes raises the median to 55 seconds, while Kuznetsova et al. (2020) quote a median of 42 seconds that includes the 7.8-second check for further instances.

Label errors and their effect

Even curated benchmarks contain wrong labels. Northcutt, Athalye and Mueller (2021) flagged likely errors in the test sets of ten datasets with confident learning, an algorithm that compares a model’s predicted probabilities with the given labels, and had each flagged item checked by five Mechanical Turk workers; an item counted as an error when fewer than three of them agreed with the given label. They estimate at least 3.3% errors on average across the ten test sets, ranging from 0.15% for MNIST to 10.12% for QuickDraw, and 2,916 confirmed errors, about 6%, in the ImageNet validation set; an expert review of images the algorithm had not flagged suggested that the ImageNet figure is closer to 20%. How such errors can reorder model rankings is discussed in benchmark overfitting.

Training is more tolerant. Zhang et al. (2017) showed that standard convolutional networks can fit a completely random labelling of their training images, so a network will memorize wrong labels if given them. Yet Rolnick et al. (2018) found that, with enough data, networks still generalize when correct labels are outnumbered: on MNIST, test accuracy stayed above 90% after each clean training example was diluted with 100 randomly labelled ones, at the cost of needing more data. The asymmetry suggests where to spend the budget: training labels can tolerate some noise if there are many of them, while test labels should be the cleanest labels a project has (our reading of these studies).

Annotators, pay and ethics

Annotation is labour, and its conditions vary widely. On crowdsourcing platforms, work is split into small paid tasks. Hara et al. (2018) recorded 2,676 workers performing 3.8 million tasks on Amazon Mechanical Turk between 2014 and 2017 and found a median hourly wage of about $2, with only 4% of workers earning more than the US federal minimum wage of $7.25 per hour. The figures depend on how unpaid time is counted, such as time spent searching for tasks and on work that is returned or rejected. Requesters set the pay: Northcutt et al. (2021, appendix of the arXiv version) paid their verifiers an hourly rate of $7.20, based on an estimated 5 seconds per image.

Other projects use trained in-house or vendor teams. Open Images drew all its boxes with Google-internal annotators, finding automated training crucial, and used both an internal pool and crowd workers for verification (Kuznetsova et al., 2020). Cityscapes produced its fine annotations in-house (Cordts et al., 2016). SAM’s annotation card reports 130 annotators, all based in Kenya, working full time for a vendor that set an hourly wage and is a certified B Corporation; images were filtered for objectionable content before annotation, and there is no process by which annotators can withdraw their contributions (Kirillov et al., 2023, arXiv appendix F.2). Such disclosures are the exception rather than the rule, which is why datasheets ask for them (Gebru et al., 2021).

History in brief

Early recognition datasets were labelled by their authors or small teams: PASCAL VOC’s boxes came from an “annotation party” at which expert annotators gathered in one place (Papadopoulos et al., 2017). Web-based tools then opened annotation to many people. Russell et al.‘s LabelMe, described in a 2005 MIT technical report, offered a web-based tool whose annotations were shared with the community as soon as they were made, and its layered polygons were the model for Cityscapes’ fine labels (Cordts et al., 2016). Crowdsourcing on Amazon Mechanical Turk made datasets such as ImageNet and COCO possible at the scale of millions of labels, with the qualification, gold-question and consensus mechanisms described above. Later, faster interactions such as extreme clicking, verification of model proposals, interactive segmentation and data engines have shifted more of the work to models, with people correcting rather than drawing.