The Pascal Visual Object Classes (VOC) Challenge

“The Pascal Visual Object Classes (VOC) Challenge” is a paper by Mark Everingham (University of Leeds), Luc Van Gool (KU Leuven), Christopher K. I. Williams (University of Edinburgh), John Winn (Microsoft Research Cambridge) and Andrew Zisserman (University of Oxford), the organizers of the challenge. It was received by the International Journal of Computer Vision in July 2008, published online in September 2009 and printed in volume 88 in 2010. The paper explains what the challenge was and why it was designed as it was: how the VOC2007 images were collected and annotated, how classification and detection were scored, and what the 17 classification and 9 detection entries of 2007 were learning. This article covers the publication, read in the authors’ PDF on the VOC website with the journal’s page numbers; the dataset across all years, its protocol as the development kit computes it and its later milestones are in the PASCAL VOC article, and the metric itself is in average precision.

Problem

By 2008 recognition research had several public datasets, and the authors’ §1.1 sets VOC against each. Caltech 101 and Caltech 256 held one, mostly uncluttered object per image, which suited their forced-choice “which one of $m$ classes” evaluation but not cluttered scenes with several objects. LabelMe had general photographs with many objects, but volunteers chose which objects to label, so its images were incompletely annotated and precision and recall could not be estimated on them. TRECVID ran a yearly competition with a new dataset, but distributed it only to participants. The Lotus Hill dataset had rich hierarchical annotation, but only a small part was freely available.

The challenge had run since 2005 (§3.2). VOC2005 used existing annotated datasets and four classes: car, motorbike, bicycle and person. VOC2006 raised that to ten and included images from the Microsoft Research Cambridge database, which had been photographed to show particular classes, so its objects tended to be large, well lit and centred (§3.1). VOC2007 doubled the classes to 20 and took every image from Flickr. The paper concentrates on the 2007 challenge, the most recent when it was written, and describes the changes already decided for VOC2008.

Contribution

The paper describes a benchmark rather than a method. Its §1 names two components, a public dataset with annotation and standard evaluation software, and an annual competition with a workshop, and two aims: a “plug and play” harness for training and testing so that algorithms can be compared, and a yearly measurement of the state of the art. Its contributions, as the paper presents them, are:

• A documented collection and annotation procedure for VOC2007, with its statistics (Tables 2 and 3) and the rules for the truncated, difficult and viewpoint attributes.
• An evaluation procedure argued from first principles (§4): independent two-class tasks per class, ranked confidences, interpolated AP, and a box-overlap criterion for detection.
• A review of the 2007 entries and results (§5, §6), with a test of statistical significance, analyses by class and object size, ranked images showing what methods confuse, and a check of generalization to the VOC2006 test set.
• A discussion of criticisms and future directions (§7).

Method

The Dataset

Collection. For each class the organizers queried Flickr with words produced by “free association” (Table 1): class names, synonyms, and scenes in which the class occurs, such as “living room” or “town centre”. For each query Flickr was asked for 100,000 matches; one image was drawn at random from them, a new query was chosen at random, and the loop repeated, with no limit per class or query. In total 500,000 images were downloaded in a few hours on one machine. Exact and near duplicates, such as copies differing in compression, were removed with the method of Chum et al. (2007). Random images were then shown to annotators, who considered 44,269 of them, annotating or discarding each. The authors note that without keywords, random Flickr images were overwhelmingly party scenes with people (§3.1).

Choice of classes. The 20 classes form a notional taxonomy of vehicles, animals, household objects and people (§3.2). The set had grown in two ways: finer subclasses, to demand semantically specific output (car versus motorbike) and harder discrimination between similar classes (cat versus dog), and new branches, animals in 2006 and household objects in 2007, to cover classes current methods found hard: animals lack distinctive parts such as wheels, and chairs are defined by function and often occluded.

Annotation. Every object of the 20 classes received a class and an axis-aligned box around its visible extent; the paper calls the box a compromise, a poor fit for a person with outstretched arms or an aeroplane taking off but quick to draw (§3.3). From VOC2006, a viewpoint (front, rear, left, right, unspecified) and a truncated flag could be used in training but were not evaluated; VOC2008 added occluded. Objects judged particularly hard to recognize, because of size, lighting, image quality, uncertain class (cows that might be sheep) or the need for much context, were marked difficult and discarded in evaluation, with no penalty for detecting them. The authors preferred marking such objects to removing them, so the data would stay useful once methods could handle them (§3.3).

For consistency, all VOC2007 annotation took place at a single “annotation party” at the University of Leeds, following written guidelines (Winn and Everingham, 2007) that covered what to label, how to place boxes and label pose, occlusion, acceptable image quality, and cases such as clothing, mud, snow, transparency, mirrors and pictures; annotators were watched periodically to check that they followed them (§3.4). The paper contrasts this with LabelMe’s distributed volunteers and with the ESP game, whose players did not know they were annotating. Afterwards one organizer checked every annotation, including for missed objects, and assigned the difficult flags; at the time of writing a single error had been reported, a viewpoint marked unspecified instead of frontal. For VOC2008 checking moved into the annotation event itself, with the difficult flag still set by an organizer.

Statistics. Table 2 gives 9,963 annotated images and 24,640 objects for VOC2007, against 5,304 images and 9,507 objects for VOC2006; the PASCAL VOC article reproduces the split totals. With twice the classes, the mean number of objects per class rose only from 951 to 1,232, driven by a quadrupling of people. “Person” had 9,218 objects, against 421 (dining table) to 2,421 (car) for the other classes, a consequence of exhaustive annotation of snapshots in which most objects appear incidentally. The authors state that object size and position are representative of Flickr but class frequencies are not: toward the end of the event, annotators discarded every image without a minority class such as sheep (§3.5).

Tasters. The segmentation subset (Table 3) has 422 trainval images with 1,215 objects and 210 test images with 607 objects, labelled per object and per class, with a 5-pixel void band around each object and difficult objects masked as void (§3.6.1). The person layout subset marks head, hand and foot boxes on 439 people in 322 trainval images and 441 people in 441 test images; annotators chose people large enough for the parts to be unambiguous, with the head and at least one other part visible (§3.6.2).

Tasks and Evaluation

Classification asks for a real-valued confidence that each test image contains at least one object of a class; detection asks for boxes with confidences (§2). Participants could enter any subset of the classes. Each task had two competitions by training data: only the VOC training and validation data, with no extra manual annotation of training or test images, or any data except the VOC test images, the second aimed at pre-built systems (§2.1). The segmentation taster had only the first. How the challenge ran, with test annotations withheld, single submissions per method and the reasons for both (§4.1), is covered in benchmarking.

§4.2 derives the evaluation from three problems with multi-class image data: images hold several classes, so a forced choice among $m$ classes cannot be used; class priors are far from uniform, and sliding-window detectors see thousands of negatives per positive, so plain accuracy is unsuitable; and the measure must not depend on the algorithm, which rules out the detection error tradeoff curves used for pedestrian detectors, defined only for sliding windows over cropped test examples. Each class is therefore scored as an independent two-class task from ranked output, by the 11-point interpolated AP of Eqs. (1)–(2). The average precision article covers the formula and its later all-point form.

AP replaced the area under the ROC curve that VOC2006 had used for classification. The paper gives four reasons: many 2006 methods scored above 95% AUC, so the measure had lost sensitivity; precision and recall are easier to interpret, especially for retrieval; AP gives more visibility to performance at low recall; and one measure would serve both main tasks. Rescoring VOC2006 with AP gave a broadly similar ranking but separated the methods more. The authors add that the interpolation reduces “wiggles” from small changes in ranking, and that a high score requires precision at all recall levels, which penalizes a method that finds only, say, side views of cars with high precision.

A detection is correct when its overlap with a ground-truth box, area of intersection over area of union (Eq. 3), exceeds 50%. The threshold was set deliberately low because boxes for non-convex objects are subjective. Detections are assigned to objects in order of decreasing confidence, and repeats count as false positives: five detections of one object give one true positive and four false (§4.2). For the segmentation taster, VOC2007 used the per-class percentage of correctly labelled pixels; since labelling every pixel “car” scores perfectly on the car class, the paper announces the per-class intersection over union of Eq. (4) for VOC2008 (§4.2.1). Person layout was scored as detection with AP, a person counting as correct only if the set of visible parts and every part box (at 50% overlap) were right (§4.2.2).

Results

Participation. Sixteen institutions submitted 28 methods in 2007, against 16 institutions and 25 methods in 2006 and 9 and 13 in 2005 (§5, Table 4). Classification drew 17 entries, detection 9. Every method used only the VOC training data: nothing was entered in the “any data” competitions, which the authors call disappointing because it left open how well methods would do with unlimited data, and how to attract commercial systems (§5.1).

Methods. Most classification entries were variations on bag of visual words: local features, usually SIFT descriptors at Harris or SIFT interest points or on a dense grid, quantized into a vocabulary, with a histogram classified by a support vector machine with a $\chi^2$ or Earth Mover’s Distance kernel (§5.1). Some added spatial tilings or a spatial pyramid; XRCE used a Fisher kernel with logistic regression and ToshCam a random forest. The greatest diversity was in fusing several feature channels. Unlike 2006, no entry classified images by detecting objects. Most detection entries were sliding-window detectors or variants (§5.2): Oxford proposed windows from interest points and classified them with an SVM on a spatial pyramid; MPI_ESSOL searched all windows by branch and bound; UoCTTI combined a coarse window with movable higher-resolution parts, the model of Felzenszwalb et al. (2008) that became the deformable part model; TKK classified boxes from an automatic segmentation; and MPI_Center was a baseline returning one centred box covering 51% of the image. The paper notes that Felzenszwalb et al. report better results than the challenge entry, obtained after the test annotations were released.

Classification (§6.1, Table 5). INRIA_Genetic had the highest AP in 19 of 20 classes. INRIA_Flat, which used the same features fused with uniform weights rather than weights learned by a genetic algorithm on validation data, was within 1–2% on most classes. XRCE was first on one class (sofa) and, by Table 5, among the top three in 17, third in 13 of them (the text says “third in 17 of 20 classes”). Median AP over classes ranged from 32.1% to 57.5%, with PRIPUVA lower at 21.1% (Fig. 7). By class, maximum AP ran from 33.1% (bottle) to 85.9% (person). The text and Fig. 9 give median AP from 15.6% (potted plant) to 75.7% (person); in our reading these do not match Table 5, where the medians over the 17 rows run from 22.2% (bottle) to 78.6% (person). Chance AP for person, from random confidences, was 43.4%, because so many images contain people, and the authors call the task “benign” as retrieval, since even sheep, the rarest class, appears in about 2% of images.

Statistical significance (§6.1.2, Fig. 8). The paper asks whether differences are significant on this fixed dataset, a different question from the variation across training and test folds that Caltech 101 users measured. Following Demšar (2006), it ranks the methods on each class, applies the Friedman test, and then the Nemenyi test, which declares two methods different when their mean ranks differ by more than a critical difference, stated as 4.9 for 17 methods and 20 classes at $p = 0.05$. The six leading methods (INRIA_Genetic, INRIA_Flat, XRCE, TKK, QMUL_LSPCH, QMUL_HSLS) are reported as not significantly different, and the authors note that 20 classes give the test little power. The statistical significance article covers the method. In our recomputation from Table 5, INRIA_Genetic’s mean rank is 1.05 and QMUL_HSLS’s 6.12, a gap of 5.07, which a critical difference of 4.9 would call significant. Demšar’s formula $q_{0.05}\sqrt{k(k+1)/(6N)}$, with $q_{0.05} = 3.458$ for $k = 17$ (the studentized range quantile divided by $\sqrt{2}$) and $N = 20$, gives 5.52, which agrees with the text’s conclusion and with the bar spanning both methods in Fig. 8. The printed 4.9 equals the undivided studentized range quantile for 17 groups (4.89); that this is the source of the difference is our conjecture.

What classifiers learn (§6.1.4–6.1.5, Figs. 10–15). Ranking test images by the median rank of five dissimilar methods showed which images were easy or confusing. Top-ranked cars were single cars on roads or many cars together; cars ranked lowest were small, dark or occluded; the most car-like negatives contained buses and lorries. All five most cat-like negatives contained dogs, which the authors attribute partly to Flickr’s similar close-ups of pets. The most bicycle-like negatives were birds on branches, which the authors explain by bag-of-words representations discarding spatial layout, so branches resemble spokes and tubes. Top-ranked chair images were indoor scenes rather than close-ups of chairs, and person was confused with dining tables and motorbikes; 68.9% of images with motorbikes also contain people. To test the dependence on context, positives whose object covered less than a threshold fraction of the image (estimated from box unions) were removed. AP for car and motorbike rose up to about 15% object pixels, while most classes showed no or negative correlation, and for chair the classifiers were very poor on images that chairs dominate.

Detection (§6.2, Table 6). Because entrants chose different classes, the authors name no overall winner. Oxford entered six vehicle classes and won all of them, by 4.0–11.4% AP; UoCTTI entered all 20, won 6 and was first or second in 14; MPI_ESSOL won 5. Methods suited different classes: UoCTTI, which emphasizes shape, scored 2.3% on dog against MPI_ESSOL’s 16.2%, while on bottle UoCTTI scored 21.4% and MPI_ESSOL 0.1%. Maximum AP ranged from 9.4% (boat) to 43.2% (car), and median AP from 2.8% to 29.4%. The best car and bicycle detectors kept precision near 100% only up to 15–20% recall, suggesting they found mainly canonical views. Good results on train were attributed partly to global methods such as MPI_Center, since trains appear large, and on horse to many images from one photographer at one gymkhana, a regularity that VOC2008’s date-randomized queries were meant to reduce. Fig. 19 varies the overlap threshold for car: AP falls steeply above 50%, rises by about 7.5% at 10%, and the ranking changes for only one pair of methods between 0 and 50%, which the authors take as evidence that the measure works. They caution that entrants knew the threshold and could tune for it. Highest-ranked false positives (Figs. 22–25) included bicycles for UoCTTI’s motorbike detector, tree branches for Oxford’s bicycle detector, and, for person, boxes with the right top and centre but the wrong height.

Object size and the difficult flag (§6.2.5, Fig. 26). The paper does not measure the effect of the difficult flag itself; it uses the flag as a tool. To ask whether detectors favour large objects, every object smaller than a threshold fraction of the image was marked difficult, so that it counts neither as a positive nor, when detected, as a false positive. All methods had lower AP on objects below about 2% of the image. MPI_Center, MPI_ESSOL and Oxford improved strongly with size, explained by the fixed 51% box, by a size prior and whole-image scoring, and by interest points missing small objects; for INRIA_Normal and IRISA on car, AP fell slightly, because some of their top-ranked correct detections were small objects.

The sketch below applies the same protocol to two synthetic detectors on synthetic boxes, scoring with the VOC2007 matching rule, in which a detection whose best-overlapping object is difficult is ignored. Both detectors place boxes with the same relative jitter and add the same random false positives; the size-biased one, standing in for an interest-point method, rarely finds small objects and is more confident on large ones.

import numpy as np

W = H = 200                                    # synthetic 200x200 images

def make_scene(rng, n_img=300):
    """Ground-truth 'car' boxes: 1-3 per image, side lengths log-uniform 6..150 px."""
    gts = []
    for i in range(n_img):
        for _ in range(rng.integers(1, 4)):
            w = np.exp(rng.uniform(np.log(6), np.log(150))); h = 0.6 * w
            x, y = rng.uniform(0, W - w), rng.uniform(0, H - h)
            gts.append((i, np.array([x, y, x + w, y + h])))
    return gts

def detector(rng, gts, n_img, size_biased):
    """Each object is found with some probability; boxes jitter by 8% of their size.
    The size-biased detector, like one fed by interest points, rarely fires on small
    objects and is more confident on large ones. Both add 1.5 false positives per image."""
    dets = []
    for i, g in gts:
        w = g[2] - g[0]
        p, mu = ((min(0.9, max(0.0, (w - 10) / 40)), 0.5 + w / 100) if size_biased
                 else (0.7, 1.0))
        if rng.random() < p:
            jit = rng.normal(0, 0.08 * w, 4)
            dets.append((i, g + jit, rng.normal(mu, 0.5)))
    for _ in range(int(1.5 * n_img)):
        w = np.exp(rng.uniform(np.log(6), np.log(150))); x, y = rng.uniform(0, W - w, 2)
        dets.append((rng.integers(n_img), np.array([x, y, x + w, y + 0.6 * w]), rng.normal(0, 0.5)))
    return dets

def iou(a, b):
    iw = max(0.0, min(a[2], b[2]) - max(a[0], b[0])); ih = max(0.0, min(a[3], b[3]) - max(a[1], b[1]))
    inter = iw * ih
    return inter / ((a[2] - a[0]) * (a[3] - a[1]) + (b[2] - b[0]) * (b[3] - b[1]) - inter)

def voc07_ap(dets, gts, difficult, thr=0.5):
    """VOC2007 rule: each detection, in order of decreasing score, goes to the ground-truth
    box it overlaps most; if that box is 'difficult' the detection is ignored, otherwise it
    is a TP above the threshold (first time) and an FP otherwise. 11-point interpolated AP."""
    by_img = {}
    for j, (i, g) in enumerate(gts):
        by_img.setdefault(i, []).append(j)
    used, flags = set(), []
    for i, box, s in sorted(dets, key=lambda d: -d[2]):
        cand = by_img.get(i, [])
        if not cand:
            flags.append(False); continue
        o, j = max((iou(box, gts[j][1]), j) for j in cand)
        if o >= thr and difficult[j]:
            continue                                   # neither TP nor FP
        tp = o >= thr and j not in used
        if tp:
            used.add(j)
        flags.append(tp)
    f = np.array(flags); n_pos = int((~difficult).sum())
    tp, fp = np.cumsum(f), np.cumsum(~f)
    rec, prec = tp / n_pos, tp / (tp + fp)
    return np.mean([prec[rec >= r].max() if (rec >= r).any() else 0 for r in np.linspace(0, 1, 11)])

size_thr = [0, 1, 2, 5, 10]                    # min box area, % of image (paper, Fig. 26)
ov_thr = [0.1, 0.3, 0.5, 0.7]                  # overlap thresholds (paper, Fig. 19)
res = {k: [] for k in ("neutral", "biased")}
for seed in range(5):
    rng = np.random.default_rng(seed)
    gts = make_scene(rng)
    area = np.array([(g[2] - g[0]) * (g[3] - g[1]) / (W * H) * 100 for _, g in gts])
    for name in res:
        dets = detector(rng, gts, 300, size_biased=(name == "biased"))
        row = [voc07_ap(dets, gts, area < t) for t in size_thr]       # small objects -> difficult
        row += [voc07_ap(dets, gts, area < 0, thr=o) for o in ov_thr]
        res[name].append(row)

print(f"{len(gts)} objects in seed 4; {np.mean(area < 2):.0%} cover under 2% of the image")
print("VOC07 AP (%), min-max over seeds 0-4")
print(" " * 8 + "".join(f"{'>=' + str(t) + '%':>11}" for t in size_thr) + "".join(f"{'ov ' + str(o):>11}" for o in ov_thr))
for name, rows in res.items():
    a = 100 * np.array(rows)
    print(f"{name:8s}" + "".join(f"{lo:6.1f}-{hi:4.1f}" for lo, hi in zip(a.min(0), a.max(0))))

605 objects in seed 4; 53% cover under 2% of the image
VOC07 AP (%), min-max over seeds 0-4
               >=0%       >=1%       >=2%       >=5%      >=10%     ov 0.1     ov 0.3     ov 0.5     ov 0.7
neutral   58.2-65.6  54.2-61.8  52.5-60.1  47.0-57.1  41.3-52.3  64.5-67.6  59.6-66.9  58.2-65.6  25.7-30.3
biased    41.2-49.2  67.3-74.7  75.5-77.5  77.1-81.3  76.3-79.0  43.0-50.6  43.0-50.6  41.2-49.2  19.3-25.4

Marking small objects difficult reverses the order of the two detectors in every seed: the size-biased detector rises by about 30 points once objects under 2% of the image stop counting, because the misses that held it down disappear from the positives. The size-neutral detector loses AP as the threshold rises, as INRIA_Normal and IRISA did on car, because its correct detections of small objects are ignored while its false positives remain. The size sweep therefore separates detectors that fail on small objects from those that do not, which is what the authors used it for; it does not show which detector is better overall. The overlap columns follow the shape of Fig. 19: AP changes little below 0.5 and falls sharply above it, since boxes jittered by 8% of their size often miss 0.7.

Progress since 2006 (§6.1.6, §6.2.6). Entrants also ran their VOC2007-trained methods on the VOC2006 test set. For 9 classification methods the rankings on the two test sets were highly correlated, but every method scored lower on VOC2006, by 5.0% (INRIA_Flat) to 12.7% (Tsinghua) in median AP, and the best 2007 method reached 62.1% median AP against 74.5% for the best 2006 entry, trained on 2006 data. The authors offer two explanations, insufficient variability in VOC2007 or overfitting to its peculiarities, such as the season of collection, and suggest the MSR Cambridge images may have helped 2006 methods. For 3 detection methods (Table 7), a 2007 method beat the best 2006 result on 7 of 10 classes, by as much as 19.1% AP on bus (Oxford) and 9.8% on person (UoCTTI), which the authors read as measurable progress in detection. Table 7 differs from the text in three places: the text says UoCTTI exceeded the 2006 results on 7 of its 10 classes, while the table shows 6 (bicycle, bus, car, horse, motorbike and person); the ranking reversal the text places on “cat”, with 53.5% against 55.5%, appears in the table’s car column; and the table shows a second reversal, for person, where IRISA leads UoCTTI on the 2007 test set (22.1% against 21.3%) but trails it on 2006 (21.1% against 26.2%), so the ranking changes for two classes, not “all but one”.

Tasters (§6.3, §6.4). Detection entries were automatically converted into segmentations, and only one segmentation-specific method was submitted, from Oxford Brookes; a detection-derived entry scored best, which the authors attribute most likely to the segmentation-only entry being unfinished. The single person layout entry, from Mannheim, localized parts on some frontal poses, but its confidences ranked results so poorly that its AP was zero; this led to the VOC2008 relaxation in which test person boxes are given.

Impact

The authors report that VOC data were referenced in 15 CVPR 2007 papers and 27 at CVPR 2008, and describe the dataset as accepted as the benchmark for object detection (§7, abstract). The protocol the paper sets out, independent per-class tasks scored by AP from ranked output, a 50% overlap criterion, duplicates as false positives and a difficult flag for ignored objects, stayed VOC’s protocol until the last challenge in 2012, with only the change to all-point AP in 2010, and COCO later kept the 0.5 threshold as one of its ten. The detector the authors recommended as a baseline, UoCTTI’s publicly released model (§6.2.1), became the deformable part model, whose development, with open-source code, the retrospective lists first among the successes of 2008–2012 (Everingham et al., 2015, §7.3). The same section credits VOC with establishing the importance of benchmarks and prompting more detailed analyses of results, citing Hoiem et al. (2012), and notes that the data were used for work its creators had not planned, such as studies of dataset bias.

Limitations

• A report on one year. The paper analyses the 2007 entries only; its statements about VOC2008 describe plans, not results.
• No measure of annotation agreement. The paper reports no inter-annotator study: §7.3 notes that moving from 10 to 20 classes made annotators err more often, because they could not hold all classes in mind, and states that data on agreement between annotators, for example on box positions, had not yet been gathered.
• Internal inconsistencies. Besides the Table 7 and Fig. 9 differences above, the printed critical difference does not match its own conclusion, and the reference list gives the Pinto et al. (2008) article pages 151–156, while the publisher lists it as article e27.
• Biases the authors found in their own data. Collection in January 2007 brought an excess of Christmas and winter scenes, one gymkhana contributed many horses, minority classes were deliberately targeted, and co-occurrences such as motorbikes with people let classifiers succeed through context (§3.1, §3.5, §6.1.4, §6.2.2).
• Evaluation choices. The overlap threshold was fixed in advance and entrants could tune for it, and §6.1.2 notes that 20 classes give significance tests little power.

The authors also answer criticisms of the challenge (§7.1). To the objection that datasets stifle innovation, they reply that datasets “have a shelf life” and must be replaced as performance saturates, while staying consistent enough to measure progress, a tension the tasters were meant to resolve, and that the workshop gave time to interesting methods as well as successful ones. To the objection that VOC is too hard, they answer that saturated datasets encourage tuning rather than fundamental progress. To Pinto et al. (2008), who argued for synthetic images with controlled variation, they reply that natural images capture real-world correlations that synthetic data cannot be expected to reproduce.

What Came After

The paper’s §7.3 lists directions for the future: more classes, with flexible scoring for labels such as hatchback, car or vehicle; object parts; “stuff” classes such as grass and sky; actions and attributes beyond nouns; video; and cheaper annotation, through volunteers, commercial annotation, Mechanical Turk or weak labels, since annotating VOC2008 had taken about 700 person-hours. Several followed in VOC itself: action classification and a growing segmentation set.

The organizers’ retrospective (Everingham et al., 2015), published after Mark Everingham’s death in 2012 with him as posthumous first author, covers 2008–2012 and does not repeat the 2010 paper’s details. It records the changes: Mechanical Turk used only to find images containing rare classes, because its boxes were not good enough; annotation and checking interleaved, with checking taking about half the annotators’ time; annotators working remotely after 2008; results submitted to an evaluation server; and AP computed from all points from 2010, because 11-point sampling was too crude to separate methods at low AP (§2.3, §2.4). It names “some evidence” of tuning on the released VOC2007 test annotations as one reason for withholding them from 2008 (§7.1.2), where the 2010 paper had cited the VOC2005 release. For significance it uses bootstrap confidence intervals for AP and rank (§3.5) instead of the Friedman and Nemenyi analysis, and it adds a normalized AP for comparing classes with different proportions of positives, which shows person to be harder than its raw AP suggests, an issue the 2010 paper had raised with its chance AP (§4.1). It also adds clustering of easy and hard images, combinations of submitted classifiers, and an analysis of detection errors over time (§4–§6).