PASCAL VOC

PASCAL VOC (Visual Object Classes) was an annual challenge in object recognition run from 2005 to 2012, whose organizers the homepage lists as Mark Everingham (University of Leeds), Luc Van Gool (ETH Zurich), Christopher Williams (University of Edinburgh), John Winn (Microsoft Research Cambridge, an organizer from 2007) and Andrew Zisserman (University of Oxford). It was funded by the EU’s PASCAL Network of Excellence and, from 2009, its successor PASCAL2 (pattern analysis, statistical modelling and computational learning). Each year brought a dataset of annotated photographs, a development kit with evaluation code, a challenge and a workshop. Its datasets, especially VOC2007 and VOC2012, and its evaluation protocol became the standard benchmark for object detection until COCO took over that role.

Purpose

Everingham et al. (2010) name two aims: challenging images with high-quality annotation and a standard evaluation method, so that algorithms can be compared on equal terms, and a yearly measurement of the state of the art. The images were meant to vary widely in size, pose, lighting and occlusion, without biases such as centered, well-lit objects.

Dataset Size

From 2007 all images were consumer photographs from Flickr. For VOC2007 the organizers retrieved 500,000 images by querying Flickr with class names, synonyms and related scenes (“living room” for chairs and sofas), removed duplicates, and annotated a random selection, split roughly in half into trainval (training plus validation) and test.

Release	Images	Objects	Source
VOC2007 train	2,501	6,301	Everingham et al. (2010), Table 2
VOC2007 val	2,510	6,307
VOC2007 test	4,952	12,032
VOC2012 trainval, detection	11,540	27,450 non-difficult (31,561 in all)	Development kit; Everingham et al. (2015)
VOC2012 trainval, segmentation	2,913	6,929	Development kit

VOC2007 totals 9,963 images and 24,640 objects. From 2009 each release added new images to the previous years’ data instead of replacing it, and the 2012 detection data is identical to 2011’s. (The homepage gives 11,530 trainval images.)

Classes

VOC2005 had four classes (bicycle, car, motorbike, person), largely from existing datasets; VOC2006 had ten. VOC2007 fixed the 20 classes used ever since, which the homepage groups as follows:

• Person: person
• Animal: bird, cat, cow, dog, horse, sheep
• Vehicle: aeroplane, bicycle, boat, bus, car, motorbike, train
• Indoor: bottle, chair, dining table, potted plant, sofa, tv/monitor

Everingham et al. (2010) describe the same classes as a notional taxonomy with four branches: vehicles, animals, household objects and people. Chairs were included partly because they are defined by function rather than appearance. The person class dominates: 9,218 of the 24,640 objects in VOC2007, because most Flickr snapshots are of people.

Annotation Types

Each object of the 20 classes has a class label and an axis-aligned bounding box around its visible extent, stored with the image’s size and filename in one XML file per image. Every image is annotated exhaustively for all 20 classes. Optional attributes describe each object:

• pose: the viewpoint of the bulk of the object, Frontal, Rear, Left or Right, or unspecified.
• truncated: the box does not cover the full object, because it extends outside the image or is hidden behind something.
• occluded: a significant part of the object inside the box is occluded, added in 2008.
• difficult: the object is hard to recognize without substantial use of context, for example because it is small, blurred or ambiguous.

Segmentation images also have two PNG label maps, one by class and one by object instance; a five-pixel band along object boundaries and ambiguous regions carry a void label (255) that evaluation ignores. Person layout images add head, hand and foot boxes for some people, and action images add action labels for people.

VOC2007 was annotated at a single event at the University of Leeds following written guidelines, and an organizer checked every annotation and assigned the difficult flags (Everingham et al., 2010); from 2010, Mechanical Turk handled early stages.

Splits

Each release divides its data into train, val, trainval and test image lists. VOC2007 was the last year whose test annotations were released. From 2008 the test images were distributed without annotations, and results had to be submitted to an evaluation server that limits the number of submissions, so that the test set cannot be used for tuning. The server still runs after the end of the challenge, with leaderboards for VOC2010–2012. For comparing design choices, the homepage suggests either the fully annotated VOC2007 data or cross-validation on the latest trainval set.

Tasks

• Classification (from 2005): predict whether an image contains at least one object of each class.
• Detection (from 2005): predict a box and confidence for each object of each class.
• Segmentation (taster from 2007, a main task from 2009): label each pixel with one of the 20 classes or background.
• Person layout (taster from 2007): predict boxes for the head, hands and feet of a given person.
• Action classification (from 2010): decide which of ten actions, such as jumping, phoning or riding a horse, a given person is performing (several can apply). Introduced as a taster, it is a main task in the 2012 development kit.

In the 2012 development kit each task has two competitions: training only on the VOC trainval data, or on any data except the test set.

Evaluation

Detection is evaluated per class. A detection is a true positive if its intersection over union with a ground-truth box of the same class exceeds 0.5; a second detection of an already-matched object is a false positive, so removing duplicates, for example with non-maximum suppression, is the detector’s job. Objects marked difficult are ignored: detecting one is neither rewarded nor penalized. Ranked detections give a precision–recall curve per class, summarized by average precision, and the mean over the 20 classes is mAP. Until 2009 AP averaged interpolated precision at eleven recall levels; from 2010 it is the exact area under the interpolated curve. Segmentation is scored by IoU per class, averaged over the 20 classes and background.

The development kit states that the top-left pixel of an image has coordinates (1, 1), and its evaluation code treats box corners as inclusive pixel indices, adding 1 to every width and height. The code below parses an invented annotation in the VOC XML layout with Python’s standard XML parser and shows the effect of that convention.

import xml.etree.ElementTree as ET
import numpy as np

# An invented annotation in the VOC XML layout (not a real VOC file).
xml_text = """
<annotation>
  <folder>VOC2012</folder>
  <filename>example_0001.jpg</filename>
  <size><width>64</width><height>48</height><depth>3</depth></size>
  <segmented>0</segmented>
  <object>
    <name>dog</name><pose>Left</pose>
    <truncated>0</truncated><occluded>0</occluded><difficult>0</difficult>
    <bndbox><xmin>9</xmin><ymin>21</ymin><xmax>40</xmax><ymax>44</ymax></bndbox>
  </object>
  <object>
    <name>bottle</name><pose>Unspecified</pose>
    <truncated>0</truncated><occluded>1</occluded><difficult>1</difficult>
    <bndbox><xmin>51</xmin><ymin>5</ymin><xmax>54</xmax><ymax>14</ymax></bndbox>
  </object>
</annotation>
"""

root = ET.fromstring(xml_text)
W, H = (int(root.find(f"size/{k}").text) for k in ("width", "height"))
objects = []
for obj in root.iter("object"):
    bb = obj.find("bndbox")
    xmin, ymin, xmax, ymax = (int(bb.find(k).text) for k in ("xmin", "ymin", "xmax", "ymax"))
    objects.append(dict(name=obj.find("name").text, difficult=obj.find("difficult").text == "1",
                        voc=(xmin, ymin, xmax, ymax)))

# VOC corners are 1-based, inclusive pixel indices: the top-left pixel is (1, 1).
# Paint each box into a 0-based NumPy mask to count the pixels it covers.
for o in objects:
    xmin, ymin, xmax, ymax = o["voc"]
    mask = np.zeros((H, W), bool)
    mask[ymin - 1:ymax, xmin - 1:xmax] = True          # rows ymin..ymax, cols xmin..xmax
    w_voc, h_voc = xmax - xmin + 1, ymax - ymin + 1
    o["xyxy"] = (xmin - 1, ymin - 1, xmax, ymax)        # 0-based, exclusive end (continuous)
    print(f"{o['name']:6s} difficult={int(o['difficult'])}  VOC {o['voc']} -> 0-based xyxy {o['xyxy']}")
    print(f"       width {w_voc} x height {h_voc} = {w_voc * h_voc} px; mask has {mask.sum()} px; "
          f"naive (xmax - xmin) x (ymax - ymin) = {(xmax - xmin) * (ymax - ymin)}")

# Objects that count as positives in VOC detection evaluation: difficult ones are ignored.
print("positives for evaluation:", [o["name"] for o in objects if not o["difficult"]])

def iou_voc(a, b):                                       # VOC devkit convention: +1 on every side
    iw = min(a[2], b[2]) - max(a[0], b[0]) + 1
    ih = min(a[3], b[3]) - max(a[1], b[1]) + 1
    if iw <= 0 or ih <= 0:
        return 0.0
    inter = iw * ih
    area = lambda r: (r[2] - r[0] + 1) * (r[3] - r[1] + 1)
    return inter / (area(a) + area(b) - inter)

def iou_naive(a, b):                                     # same numbers read as continuous corners
    iw = max(0, min(a[2], b[2]) - max(a[0], b[0]))
    ih = max(0, min(a[3], b[3]) - max(a[1], b[1]))
    inter = iw * ih
    area = lambda r: (r[2] - r[0]) * (r[3] - r[1])
    return inter / (area(a) + area(b) - inter)

# A detection shifted one pixel right of each ground-truth box, in VOC coordinates.
for o in objects:
    g = o["voc"]
    d = (g[0] + 1, g[1], g[2] + 1, g[3])
    print(f"{o['name']:6s} detection {d}: IoU with +1 = {iou_voc(g, d):.3f}, without = {iou_naive(g, d):.3f}")

Output:

dog    difficult=0  VOC (9, 21, 40, 44) -> 0-based xyxy (8, 20, 40, 44)
       width 32 x height 24 = 768 px; mask has 768 px; naive (xmax - xmin) x (ymax - ymin) = 713
bottle difficult=1  VOC (51, 5, 54, 14) -> 0-based xyxy (50, 4, 54, 14)
       width 4 x height 10 = 40 px; mask has 40 px; naive (xmax - xmin) x (ymax - ymin) = 27
positives for evaluation: ['dog']
dog    detection (10, 21, 41, 44): IoU with +1 = 0.939, without = 0.938
bottle detection (52, 5, 55, 14): IoU with +1 = 0.600, without = 0.500

The dog box spans columns 9 to 40, which is 32 pixels, not $40 - 9 = 31$, as the painted mask confirms; subtracting 1 from xmin and ymin gives the 0-based, exclusive-end corners of most Python code. The difficult bottle drops out of the positives. For the large box the conventions barely differ, but for the 4-pixel-wide bottle a one-pixel shift gives IoU 0.600 with the +1 and exactly 0.500 without it, right on the threshold: the development kit’s text requires the overlap to exceed 50%, but its VOCevaldet.m accepts an overlap of exactly 0.5 (ovmax>=VOCopts.minoverlap).

License

The homepage, under “Database Rights”, states that the VOC data includes images obtained from Flickr and that their use must respect Flickr’s terms of use (as of October 2026). The challenge pages add that image sources and owners were obscured for the challenge, with contributor details to be included in the annotation of the final release. The site names no license for the annotations, so the terms for reuse are unclear; it directs queries to the organizers.

Limitations and Bias

• Few classes and small size. Twenty classes and about 11,500 trainval images are small by later standards; class frequency is uneven, with “person” far ahead and minority classes such as sheep deliberately targeted late in annotation.
• Collection biases. Flickr ranked results by recency, so VOC2007, collected in January 2007, contains an above-average number of Christmas and winter scenes; from VOC2008 the queries included a random date (Everingham et al., 2010).
• Annotation gaps. The homepage notes that the images added in 2009 received no difficult flags, an omission, and layout and action images are not exhaustively annotated.
• A lenient threshold. IoU 0.5 rewards rough localization; Everingham et al. (2010) set it low on purpose, because boxes for shapes such as outstretched limbs are themselves ambiguous.
• Saturation. Reported VOC2007 test mAP more than doubled between DPM v5 and Faster R-CNN (see below). In our reading, after years of tuning on the same public test images, small differences between detectors say little about real progress.

Common Uses

• Detection benchmark. Many detectors report mAP on the VOC2007 test set, whose annotations are public. A common training set is the union of VOC2007 and VOC2012 trainval, written “07+12”: Faster R-CNN (Ren et al., 2015), following Fast R-CNN, reported 73.2% mAP on VOC2007 test with it, and SSD (Liu et al., 2016) uses the same notation. For the VOC2012 test set, “07++12” adds the VOC2007 test images to the training data.
• Semantic segmentation. VOC2012 segmentation is a standard benchmark. Hariharan et al. (2011) annotated semantic boundaries and figure–ground segmentations of more than 20,000 objects in about 10,000 VOC2011 trainval images, a set they named the Semantic Boundaries Dataset (SBD). Long et al. (2015) trained fully convolutional networks on its labels for 8,498 images instead of the 1,112 in the VOC2011 segmentation training set.
• Annotation format. The per-image XML layout, with Annotations/ and JPEGImages/ folders, remains a common export format of labeling tools.

Historical Milestones

• 2005. First challenge, with four classes; results were presented at a PASCAL workshop in Southampton. From 2006 the workshops were held with ECCV and ICCV in alternate years.
• 2007. Twenty classes, segmentation and person layout tasters, and AP instead of ROC area for classification.
• 2008–2009. The deformable part model of Felzenszwalb and colleagues, the reference detector for 2008–2012, won detection jointly in both years (Everingham et al., 2015).
• 2010. Action classification was introduced, AP switched to the full area under the curve, and a large-scale classification challenge based on ImageNet ran alongside.
• 2012. The last challenge; its workshop at ECCV 2012 was dedicated to Mark Everingham, who died that year. A University of Amsterdam entry that proposed boxes by selective search won detection.
• 2014. R-CNN (Girshick et al., 2014), using CNN features pretrained on ImageNet and fine-tuned, reported 53.3% mAP on VOC 2011/12 test and, with bounding-box regression, 58.5% on VOC2007 test, against 33.7% for DPM v5 in the same table.

Related Datasets

ImageNet’s ILSVRC began in 2010 as a companion to VOC, with 1,000 classes for classification and later 200 for detection. COCO (2014) kept all 20 VOC classes among its 80 detection categories, chose non-iconic scenes with objects in context, added per-instance masks, and averages AP over IoU thresholds from 0.5 to 0.95; its $\mathrm{AP}_{50}$ is the VOC-style metric.

Resources

• The PASCAL VOC homepage: pages for each challenge year with data, development kits and annotation guidelines, and the history of the challenge.
• PASCAL VOC evaluation server: test-set evaluation and leaderboards for VOC2010–2012.