Berkeley Segmentation Dataset

The Berkeley Segmentation Dataset (BSDS) is a set of natural color photographs, each divided into regions by several people, together with a benchmark that compares machine output against those human segmentations. David Martin, Charless Fowlkes, Doron Tal and Jitendra Malik of the University of California, Berkeley, introduced it at ICCV in 2001. Its public release, the BSDS300 (Berkeley Segmentation Dataset and Benchmark), has 300 images with 200 for training and 100 for testing. In 2011 Pablo Arbeláez, Michael Maire, Fowlkes and Malik released the BSDS500, which keeps the 300 images for training and validation and adds 200 new test images. The boundary benchmark defined by Martin, Fowlkes and Malik (2004) scores a detector by precision and recall after matching its boundary pixels one-to-one with each person’s boundaries. Arbeláez et al. (2011) describe its wide acceptance as the benchmark for boundary detection, and Hou, Yuille and Koch (2013) call the BSDS300 the most popular boundary benchmark. The region measures that Arbeláez et al. added made the BSDS500 a common test for image segmentation as well.

Purpose

The 2001 paper starts from a contrast. Recognition could be scored by counting errors on a shared test set such as MNIST. Segmentation was usually shown on a few hand-picked images. The authors argued that segmentations can be judged directly against those of several human observers, provided that different people segment the same image consistently, and they set out to show that they do. The paper used the data in two ways: as a benchmark, scoring Normalized Cuts against the humans, and as a source of ecological statistics, measuring how proximity, similarity of brightness and convexity relate to whether two pixels share a region.

Martin, Fowlkes and Malik (2004) used the human segmentations differently. They treated the boundaries between regions as ground truth for boundary detection, which made it possible to learn a detector from data and to rank detectors by one number. Arbeláez et al. (2011) needed to evaluate hierarchical segmentations as well as contours, so they added region measures and a larger test set. The BSDS500 page states the goal of the whole effort as an empirical basis for research on image segmentation and boundary detection.

Dataset Size

Version	Images	Split	Human segmentations	Source
Full collection	1,000	none released	12,000 by 30 subjects, half on grayscale and half on color images	BSDS300 page
BSDS300	300	200 train, 100 test	3,269 by 28 subjects (1,636 grayscale, 1,633 color)	BSDS300 page and its subject list
BSDS500	500	200 train, 100 validation, 200 test	“five different subjects on average” per image	BSDS500 page

The figures come from different documents and do not all agree. The 2001 paper describes a goal of 1,000 images with at least four grayscale and four color segmentations each. At the time of writing it had 3,000 segmentations by 25 people of 800 images, and its experiments used an early version with 150 grayscale segmentations of 50 images. The 2004 paper gives 5 to 10 segmentations for each of the 1,000 images. The subject list on the BSDS300 site, generated in October 2003, lists 28 subjects with between 3 and 298 segmentations each. Hou, Yuille and Koch (2013) counted 1,667 boundary maps from 28 labelers on the BSDS300, a figure that matches neither the color nor the grayscale total of that list. Liu et al. (2019) describe each BSDS500 image as labelled by 4 to 9 annotators. None of the official pages gives the number of BSDS500 segmentations.

The BSDS300 page gives the image size as 481 × 321 pixels; images are stored in landscape or portrait orientation. The images were chosen from the Corel image database, a collection of 40,000 images, as described below.

Classes

The dataset has no classes. A region is a set of pixels that a person drew as one “thing” or part of a thing, with no name or category attached, and two segmentations of the same image need not have the same number of regions. Datasets with named regions, such as ADE20K or PASCAL VOC, serve semantic segmentation instead.

Annotation Types

Images. Martin et al. (2001) chose 1,000 Corel images of natural scenes that contain at least one discernible object. The rule excluded images unsuited to recognition, such as photographs of marble textures or of neon reflections on wet pavement.

Segmentations. Annotators used a Java application that a server supplied with images, so that anyone with Internet access could take part. They split an existing segment in two by drawing a contour around part of it and could move pixels between segments, so every pixel belongs to exactly one segment. The 2001 paper reports that a segmentation with 10 to 20 segments took under five minutes. In its description, a server assigned images at random, with a bias toward images that others had already segmented, and made sure that no subject saw an image twice, that no image was segmented by more than five people, and that no two images had exactly the same set of annotators. The later counts of 5 to 10 segmentations per image exceed that cap, and no source explains the difference. The first annotators were students in a Berkeley graduate computer vision course in fall 2000, told to segment as naive observers. Images were shown in grayscale for half of the segmentations and in color for the other half. The BSDS300 files store each segmentation as run-length rows of segment labels (the .seg format), with header fields recording whether the image was shown in grayscale, inverted or flipped.

Instructions. The instructions were vague on purpose, so that differences between people would reflect how they perceived the scene rather than how they read the task. The two papers give different versions:

• 2001 paper. Divide each image into pieces, each a distinguished thing of roughly equal importance. Something between 2 and 20 pieces should be reasonable.
• 2004 paper. Divide the image into segments that represent “things” or “parts of things” (the paper’s words). The number is up to the subject, and something between 2 and 30 is likely to be appropriate.

Boundaries. Boundary ground truth is derived from the segmentations: a pixel is a boundary pixel where the segment label changes. In the BSDS500 each ground-truth file holds, for every annotator, the segmentation and its boundary map; the BSDS300 benchmark code derives the maps from the segmentations.

Figure-ground labels. The BSDS300 page links to figure-ground labellings for a subset of the images, generated by the same group.

Consistency. To show that segmentations of the same image agree, the 2001 paper defined two measures that do not penalize refinement, where one region is split further by another person. At pixel $p_i$, with $R(S, p_i)$ the region of segmentation $S$ that contains it, the local refinement error is

$$E(S_1, S_2, p_i) = \frac{|R(S_1, p_i) \setminus R(S_2, p_i)|}{|R(S_1, p_i)|} ,$$

which is zero where $S_1$ refines $S_2$. The Global Consistency Error takes $\min\{\sum_i E(S_1,S_2,p_i), \sum_i E(S_2,S_1,p_i)\}/n$ over the $n$ pixels, so all refinement must go in one direction. The Local Consistency Error takes $\sum_i \min\{E(S_1,S_2,p_i), E(S_2,S_1,p_i)\}/n$, so the direction may change from place to place. Pairs of human segmentations of the same image had mean errors of 0.11 (GCE) and 0.07 (LCE), against about 0.39 and 0.30 for segmentations of different images (Fig. 4; the caption lists the four means “clockwise”, which would swap the last two, but the histograms and the rule that GCE is never below LCE support this reading). Both measures are zero for one segment per pixel and for one segment per image, so the paper used them only to compare segmentations with similar numbers of segments.

Splits

• BSDS300. 200 training and 100 test images. The page asks that the test images and their segmentations stay unused until an algorithm is finished.
• BSDS500. The 300 BSDS300 images become training and validation data, and 200 new images with new human segmentations form the test set. Third-party copies of the January 2013 archive divide the data into folders of 200 training, 100 validation and 200 test images. The page’s best-practice guidelines say how to report results on both versions. Train and tune only on the training and validation sets, run once on the test set with fixed parameters, and report every score that the evaluation scripts return. For comparison with BSDS300 results, also train on the training set alone and test on the validation set. Together these rules imply that the validation set is the old BSDS300 test set.

Tasks

• Boundary (contour) detection: produce a soft boundary map for each image, scored by the boundary benchmark. See edge detection for the methods.
• Generic image segmentation: produce a segmentation, or a hierarchy of segmentations, scored by the boundary benchmark on its region boundaries and by the region measures.
• Superpixel evaluation: boundary recall and undersegmentation error against the human segmentations (see superpixels).
• Natural-image statistics: the 2001 paper’s original second use.

Evaluation

The benchmark is defined by its MATLAB and C++ code more than by the papers that describe it. This section follows the code line by line. Two versions were read:

• segbench 1.2.0, the BSDS300 benchmark code, browsed on the official BSDS300 site; the archive is dated June 2008.
• The BSR benchmark from the January 2013 BSDS500 archive. The official archive bundles the dataset images, so this article read a copy of the benchmark code in an unofficial GitHub mirror of that archive kept by the Berkeley Institute for Data Science (commit a04b7c6, 2016), not by the dataset’s authors. The pixel-matching sources in that copy (correspondPixels.cc, match.cc and the CSA wrapper) are identical to segbench 1.2.0 apart from line endings.

Boundary benchmark

The input is a soft boundary map $P_b$ with values in $[0,1]$, preferably already thinned by non-maximum suppression. The code computes counts for each test image and threshold, then pools them over the whole test set.

1. Thresholds. The BSR code uses 99 thresholds, $t = 0.01, 0.02, \dots, 0.99$; segbench used 30 by default. A segmentation given as an ultrametric contour map is thresholded at the same values. A plain set of segmentations uses one “threshold” per segmentation.
2. Thinning. $B(t) = \{P_b \ge t\}$ is thinned to one-pixel width with MATLAB’s bwmorph(..., 'thin', Inf). Version 1.2.0 moved thinning to after thresholding. Its changelog notes that this raised scores, because thinning before thresholding removed more boundary pixels. The BSDS300 page warns that this thinning may not suit every algorithm, which is why detectors apply their own non-maximum suppression first.
3. Correspondence. correspondPixels matches the machine map $B$ separately with each human boundary map $G_k$. The matching is a minimum-cost bipartite assignment in which a machine pixel and a human pixel can be paired only if their Euclidean distance is at most $d_{\max} = 0.0075 \times$ the image diagonal. That is 4.34 pixels on a 481 × 321 image. A pair costs its distance, and a pixel left unmatched is assigned to an outlier node at a cost of $100\,d_{\max}$. The assignment is one-to-one, so a second detection along the same boundary finds no partner and counts as a false positive. The 2004 paper presents this as enforcing Canny’s single-response criterion. Appendix B of that paper describes the solver. Andrew Goldberg’s CSA code runs on a sparse graph that contains only pairs within $d_{\max}$, plus outlier nodes on both sides, each joined by $d = 6$ random edges, and a high-cost “safety net” matching that guarantees a perfect matching exists. Distances are multiplied by 100 and rounded to integers.
4. Combining annotators. A machine pixel is a true positive if it is matched in at least one $G_k$ (accP = accP | match1). Recall counts the matched pixels of every $G_k$. For image $i$:

$$R_i(t) = \frac{\sum_k |\mathrm{matched}(G_{ik})|}{\sum_k |G_{ik}|}, \qquad P_i(t) = \frac{|\{\text{pixels of } B_i(t) \text{ matched in some } G_{ik}\}|}{|B_i(t)|} .$$

The 2004 paper says that the hit rate is “simply averaged” over the humans. The code pools the counts, which weights each annotator by the number of boundary pixels they drew. 5. Scores. The F-measure is $F = 2PR/(P+R)$, the harmonic mean described in precision and recall. Its general weighted form is $F = PR/(\alpha R + (1-\alpha)P)$, with $\alpha = 0.5$ in the 2004 paper. collect_eval_bdry.m produces three numbers:

• ODS (optimal dataset scale). Sum the four counts over all images at each threshold, compute $P(t)$, $R(t)$ and $F(t)$, and take the maximum. The search also interpolates linearly between neighbouring thresholds at 100 points.
• OIS (optimal image scale). For each image, take the first sampled threshold with the highest per-image $F$, without interpolation. Then sum the counts at those thresholds over images and compute one $F$. OIS is therefore not the mean of the per-image F-measures.
• AP. Take the dataset curve $P(R)$ at unique recall values, interpolate it linearly at $R = 0, 0.01, \dots, 1$, set points outside the measured recall range to zero, and compute $0.01 \sum_{j=0}^{100} \tilde P(j/100)$. That is a sum of 101 samples divided by 100. Dollár and Zitnick (2015) reported an error in the BSDS evaluation that overestimates AP by 1% and kept the code unchanged for consistency. The evaluation code in Dollár’s own toolbox keeps the computation and marks it as a minor bug that should divide by 101, which explains the 1%. For how this differs from the average precision used in object detection, see average precision.

Some details differ between the papers and the code:

• The ODS threshold. Arbeláez et al. (2011) describe ODS as a fixed threshold calibrated on the training set. collect_eval_bdry.m takes the best threshold on the evaluated images themselves, which are the test images in the released script. OIS goes further and picks each test image’s best threshold as an oracle would, which the paper says.
• The tolerance. The 2004 paper used $d_{\max} = 1\%$ of the diagonal on images reduced to 240 × 160 pixels, which is 2.88 pixels, and studied the effect of tolerances from $\sqrt{2}$ to $\sqrt{10}$ pixels. The BSR default is 0.0075 of the diagonal at full resolution. The help text of correspondPixels.m gives a default of 0.01, while the compiled function’s actual default is 0.0075, the value the benchmark scripts pass.
• Randomness. The outlier edges are drawn with a random generator seeded from the system clock. Different runs can therefore build slightly different graphs, and in rare cases a run falls back on the high-cost safety net, which the code reports with a warning when it is used more than five times.
• Human maps. segbench thinned the human boundary maps before matching. The BSR code uses the stored boundaries as they are.

Human performance. The BSDS300 code (boundaryBenchHuman.m) treats each human segmentation’s boundaries as a binary detector and matches them against the remaining annotators of the same image. It then pools the counts over subjects and images. Several human figures are quoted:

• 0.80 median human F-measure in the 2004 paper (Fig. 3), on half-resolution images.
• 0.79 on the BSDS300 and 0.80 on the BSDS500, at both ODS and OIS, in Arbeláez et al. (2011), Table 1.
• 0.79 for humans in both the grayscale and the color rankings on the BSDS300 benchmark page.
• 0.803, quoted by Liu et al. (2019) without a source.

The BSR archive contains no human-evaluation script, so how the BSDS500 human figure was computed is not documented in the code.

Region benchmark

The BSDS500 adds three measures that compare segmentations as partitions. evaluation_reg_image.m computes them on segmentations obtained by thresholding an ultrametric contour map at the same 99 levels (connected regions of $\mathrm{ucm} \le t$), or on a supplied set of segmentations.

• Segmentation covering. With the overlap $O(R,R') = |R \cap R'|/|R \cup R'|$ and $N$ pixels, Arbeláez et al. define the covering of segmentation $S$ by $S'$ as

$$C(S' \to S) = \frac{1}{N} \sum_{R \in S} |R| \max_{R' \in S'} O(R, R') .$$

The paper’s tables report the covering of the ground truth by the machine segmentation. The code computes, for every region of every human segmentation, the best overlap with any machine region (match_segmentations.m, adapted from the PASCAL VOC code). It then sums area times overlap over all regions of all annotators and divides by the total area. Since each annotator covers the whole image, this equals the average over annotators of $C(S \to G_k)$.

• Probabilistic Rand Index (PRI). The Rand index of two segmentations is the fraction of pixel pairs on which they agree, either same region in both or different regions in both. match_segmentations2.m computes it from the contingency table and averages it over annotators. Arbeláez et al. note that this average equals the PRI of $S$ against the set $\{G_k\}$ when each pair’s probability of sharing a region is estimated by the fraction of annotators who put the pair in one region. Higher is better.
• Variation of Information (VI). $\mathrm{VI}(S, G) = H(S) + H(G) - 2I(S, G)$, in bits (base-2 logarithms in the code), averaged over annotators. Lower is better.

collect_eval_reg.m reports each measure at ODS and OIS. For covering, ODS pools areas over images at each threshold and takes the best threshold, without interpolation; OIS takes each image’s best threshold, the last one in case of ties. For PRI and VI, ODS averages the per-image values over images at each threshold and takes the best; OIS averages each image’s best value. Covering has a third score, Best: each human region keeps its best overlap over all thresholds, as if every region could choose its own level of the hierarchy. On the BSDS500, Arbeláez et al. (Table 2) report humans at a covering of 0.72 and a PRI of 0.88 with a VI of 1.17, against 0.59, 0.83 and 1.69 for gPb-owt-ucm at ODS. They found that covering had the largest dynamic range of the three, followed by PRI and VI. They also note that PRI has been reported to have a small dynamic range, and that the meaning of VI when there are several ground-truth segmentations is unclear.

Reimplementing the boundary protocol

The script below reproduces evaluation_bdry_image.m and collect_eval_bdry.m in NumPy and SciPy. It runs on six rendered 160 × 240 images with known regions and replaces the benchmark’s CSA solver with scipy.optimize.linear_sum_assignment. Each image has a textured disk, a box and a faint disk on a plain background, with an image-wide contrast drawn at random. Five synthetic “annotators” redraw the true regions with about 0.7 pixels of position noise. Two of them skip the faint disk, and one splits the box in two, as a person refining a segmentation would. The detector is a Gaussian gradient magnitude on one fixed scale for all images.

The matching uses the objective of the benchmark’s assignment graph. There, each unmatched pixel takes an outlier node at cost $100\,d_{\max}$, and each matched pair of real pixels leaves one outlier node free on each side, which must then be paired with each other at the same cost. Compared with leaving every pixel unmatched, a matched pair therefore changes the total by $d - 100\,d_{\max}$. A rectangular assignment with that cost for each pair within tolerance, and zero, meaning unmatched, for every other pair, has the same optimum as the benchmark’s perfect matching with the outlier nodes removed. Because only pairs within $d_{\max}$ can match, the problem splits into connected components that are solved one at a time.

import numpy as np
from scipy import ndimage
from scipy.optimize import linear_sum_assignment
from scipy.sparse import coo_matrix
from scipy.sparse.csgraph import connected_components
from scipy.spatial import cKDTree
from skimage.morphology import thin

H, W = 160, 240
MAX_DIST = 0.0075                       # tolerance as a fraction of the image diagonal
TOL = MAX_DIST * np.hypot(H, W)         # 2.16 pixels here; 4.34 on a 481 x 321 BSDS image
yy, xx = np.mgrid[:H, :W]

def render_labels(shapes, rng, jitter, drop_weak, split):
    """Label map of one segmentation: background 0 plus disks and boxes."""
    lab = np.zeros((H, W), int)
    for k, (kind, cy, cx, r, weak) in enumerate(shapes, start=1):
        if weak and drop_weak:
            continue                                   # this annotator ignores the faint object
        cy, cx, r = cy + jitter * rng.normal(), cx + jitter * rng.normal(), r + jitter * rng.normal()
        if kind == "disk":
            m = (yy - cy) ** 2 + (xx - cx) ** 2 <= r ** 2
        else:
            m = (np.abs(yy - cy) <= r) & (np.abs(xx - cx) <= 1.6 * r)
        lab[m] = k
        if split and kind == "box":
            lab[m & (xx > cx)] = 10 + k                # refinement: the box split in two
    return lab

def seg2bdry(lab):
    """Boundary pixels: label differs from the neighbour below or to the right (simplified seg2bdry)."""
    b = np.zeros_like(lab, bool)
    b[:-1, :] |= lab[:-1, :] != lab[1:, :]
    b[:, :-1] |= lab[:, :-1] != lab[:, 1:]
    return b

def make_image(seed):
    rng = np.random.default_rng(seed)
    shapes = [("disk", rng.uniform(50, 110), rng.uniform(50, 90), rng.uniform(22, 32), False),
              ("box", rng.uniform(40, 120), rng.uniform(150, 190), rng.uniform(15, 22), False),
              ("disk", rng.uniform(40, 120), rng.uniform(100, 140), rng.uniform(12, 18), True)]
    truth = render_labels(shapes, rng, 0.0, False, False)
    levels = {0: 0.3, 1: 0.75, 2: 0.65, 3: 0.38}       # object 3 is faint against the background
    img = np.vectorize(levels.get)(truth).astype(float)
    img += 0.15 * (truth == 1) * np.sin(xx / 1.5)      # texture inside the large disk
    contrast = rng.uniform(0.3, 1.0)                   # photographs differ in contrast
    img = contrast * ndimage.gaussian_filter(img, 1.0) + 0.05 * rng.normal(size=(H, W))
    humans = [seg2bdry(render_labels(shapes, rng, 0.7, drop_weak=a >= 3, split=a == 0))
              for a in range(5)]                       # 5 synthetic annotators
    gm = ndimage.gaussian_gradient_magnitude(img, 1.5)
    return np.clip(gm / 0.12, 0, 1), humans            # soft boundary map, same scale for all images

def correspond(b1, b2):
    """One-to-one matching of boundary pixels within TOL, with the objective of match.cc's graph:
    relative to no matches, each matched pair changes the cost by its distance minus 100 * TOL.
    Returns boolean maps of matched pixels in b1 and in b2."""
    p1, p2 = np.argwhere(b1), np.argwhere(b2)
    m1, m2 = np.zeros(b1.shape, bool), np.zeros(b2.shape, bool)
    if len(p1) == 0 or len(p2) == 0:
        return m1, m2
    d = cKDTree(p1).sparse_distance_matrix(cKDTree(p2), TOL, output_type="coo_matrix")
    if d.nnz == 0:
        return m1, m2
    oc = 100 * TOL
    n1 = len(p1)
    g = coo_matrix((np.ones(d.nnz), (d.row, n1 + d.col)), shape=(n1 + len(p2),) * 2)
    _, comp = connected_components(g, directed=False)
    i_all, j_all = d.row, d.col
    w_all = np.rint(d.data * 100) - np.ceil(oc * 100)         # integer costs as in match.cc
    for c in np.unique(comp[i_all]):                           # solve each component separately
        sel = comp[i_all] == c
        rows, ri = np.unique(i_all[sel], return_inverse=True)
        cols, ci = np.unique(j_all[sel], return_inverse=True)
        cost = np.zeros((len(rows), len(cols)))                # 0 = leave both unmatched
        cost[ri, ci] = w_all[sel]                              # negative = better than two outliers
        r, k = linear_sum_assignment(cost)
        ok = cost[r, k] < 0
        m1[tuple(p1[rows[r[ok]]].T)] = True
        m2[tuple(p2[cols[k[ok]]].T)] = True
    return m1, m2

def evaluate_image(pb, humans, thresholds):
    """evaluation_bdry_image.m: counts at every threshold."""
    cnt = np.zeros((len(thresholds), 4))                       # cntR sumR cntP sumP
    for t, th in enumerate(thresholds):
        bmap = thin(pb >= th)
        acc_p = np.zeros_like(bmap)
        for gt in humans:
            m1, m2 = correspond(bmap, gt)
            acc_p |= m1                                        # a detection may match any human
            cnt[t, 0] += m2.sum(); cnt[t, 1] += gt.sum()       # recall pooled over humans
        cnt[t, 2] += acc_p.sum(); cnt[t, 3] += bmap.sum()
    return cnt

def rpf(c):
    r = c[..., 0] / np.maximum(c[..., 1], 1)
    p = c[..., 2] / np.maximum(c[..., 3], 1)
    return r, p, 2 * p * r / np.where(p + r > 0, p + r, 1)

def max_f(r, p):
    """maxF in collect_eval_bdry.m: also searches 100 points between neighbouring thresholds."""
    best = 2 * p[0] * r[0] / max(p[0] + r[0], 1e-12)
    for i in range(1, len(r)):
        d = np.linspace(0, 1, 100)
        ri, pi = r[i] * d + r[i - 1] * (1 - d), p[i] * d + p[i - 1] * (1 - d)
        best = max(best, np.max(2 * pi * ri / np.where(pi + ri > 0, pi + ri, 1)))
    return best

thresholds = np.linspace(1 / 100, 99 / 100, 99)                # nthresh = 99
data = [make_image(seed) for seed in range(6)]
counts = np.array([evaluate_image(pb, hs, thresholds) for pb, hs in data])

r, p, f = rpf(counts.sum(0))                                   # pool counts over images
ods = max_f(r, p)
best_t = [np.flatnonzero(fi == fi.max())[0] for fi in rpf(counts)[2]]   # first maximum per image
ois = rpf(sum(counts[i, t] for i, t in enumerate(best_t)))[2]
ru, idx = np.unique(r, return_index=True)
p_interp = np.interp(np.arange(0, 1.0001, 0.01), ru, p[idx], left=np.nan, right=np.nan)
ap_bsr = np.nansum(p_interp) * 0.01                            # Area_PR as collect_eval_bdry.m
print(f"tolerance {TOL:.2f} px; per-image best thresholds {np.round(thresholds[best_t], 2)}")
print(f"ODS F = {ods:.3f} (threshold {thresholds[np.argmax(f)]:.2f})   OIS F = {ois:.3f}")
print(f"AP as the code computes it = {ap_bsr:.3f}   (101 samples / 101 = {ap_bsr * 100 / 101:.3f})")

# Human consistency: each annotator scored against the other four (boundaryBenchHuman.m)
h = np.zeros(4)
for _, hs in data:
    for j, hj in enumerate(hs):
        acc = np.zeros_like(hj)
        for k, hk in enumerate(hs):
            if k != j:
                m1, m2 = correspond(hj, hk)
                acc |= m1; h += [m2.sum(), hk.sum(), 0, 0]
        h += [0, 0, acc.sum(), hj.sum()]
print("human F (leave one out) = %.3f" % rpf(h)[2])

# Why one-to-one matching: a detector that fires twice along every human boundary
gt = data[0][1][1]
double = gt | np.roll(gt, 2, axis=1)                           # each boundary repeated 2 px away
m1, _ = correspond(double, gt)
near = ndimage.distance_transform_edt(~gt) <= TOL
print(f"doubled boundaries: precision with matching {m1.sum() / double.sum():.2f}, "
      f"counting any detection near a boundary {near[double].mean():.2f}")

Output (NumPy 2.3.5, SciPy 1.16.3, scikit-image 0.25.2; about 5 seconds):

tolerance 2.16 px; per-image best thresholds [0.26 0.23 0.31 0.4  0.23 0.4 ]
ODS F = 0.687 (threshold 0.26)   OIS F = 0.816
AP as the code computes it = 0.694   (101 samples / 101 = 0.687)
human F (leave one out) = 0.873
doubled boundaries: precision with matching 0.62, counting any detection near a boundary 1.00

The gap between ODS and OIS comes from the image-wide contrast. Gradient magnitude scales with contrast, so a low-contrast image needs a lower threshold than a high-contrast one, and the best thresholds per image range from 0.23 to 0.40. ODS must use one threshold for all six images and pays for it. OIS lets an oracle choose per image. The size of the gap measures how far a detector’s output is from being calibrated across images, which is why both numbers are reported. The AP line shows the 1% inflation from the 101-sample sum. The human F-measure stays below 1 even though every annotator drew the same shapes, because of the position noise, the skipped faint disk and the split box. The doubled detector shows what the one-to-one matching is for: every one of its pixels lies within tolerance of a true boundary, so counting nearby pixels gives a precision of 1.00, while matching leaves the copies without partners. Its precision is above 0.5 only because a boundary that runs horizontally lands mostly on itself when shifted sideways.

This reimplementation was checked by reading the benchmark code, not by running it, since the benchmark needs MATLAB and compiled MEX files. It differs from the benchmark in four ways:

• The solver. linear_sum_assignment solves each dense component exactly. CSA solves one sparse graph whose random outlier edges stand in for dense ones, with the safety net as a fallback. Both minimize the same integer costs, but when several matchings are optimal they may return different ones.
• Thinning. skimage.morphology.thin stands in for bwmorph(..., 'thin', Inf), and the two can differ at single pixels.
• Boundaries. seg2bdry here keeps only the label changes to the right and below, while the official function also marks corner pixels.
• Duplicate recall values. For AP, np.unique keeps the first of several points with equal recall. MATLAB’s unique returns the first or the last occurrence depending on the version.

License

As of October 2026, the BSDS300 page says users are “free to download a portion of the dataset for non-commercial research and educational purposes”. In exchange it asks them to make their results on the test set available, and asks that work based on the dataset cite the ICCV 2001 paper. The BSDS500 page names no license or terms of use. It asks users to cite Arbeláez et al. (2011) and sets out the best-practice guidelines described under Splits. Neither page says anything about the copyright of the Corel photographs or names a license for the human segmentations, so the terms for reuse, and in particular for commercial use, are unclear. The 2001 paper says only that the images were chosen from the Corel image database. In the January 2013 source archive the gPb and UCM code carries AGPL license files; the benchmark code has no overall license file, although some of its C++ support files, such as Random.hh, carry GNU GPL headers.

Limitations and Bias

• Small and old. There are 500 Corel images at 481 × 321 pixels. Deep detectors are trained on a few hundred images with heavy augmentation. Xie and Tu (2015) multiplied the training set 32-fold by rotation and flipping. When they moved 100 test images into training and evaluated on the other 100, averaged over five random splits, their ODS rose from 0.782 to 0.797 (± 0.003). That suggests the training set, not only the model, limits scores. Later detectors added outside data, such as PASCAL VOC Context images in Liu et al. (2019).
• Ambiguous ground truth. Hou, Yuille and Koch (2013) matched each pair of human maps with the benchmark’s own assignment and graded each boundary pixel by the fraction of annotators who drew it. On the BSDS300, 29.40% of boundary labels were drawn by a single annotator (“orphan labels”), and 28.99% by all annotators. On the 200 new BSDS500 test images the proportions were 30.58% and 30.15%. A psychophysical experiment led them to conclude that orphan labels are too unreliable to judge current algorithms.
• The precision bonus. Hou et al. also pointed out an asymmetry in the protocol. A missed boundary costs as many recall pixels as there are annotators who drew it. A false alarm costs one precision pixel, and only if it matches no annotator at all. They argued that this rewards detectors that report many weak candidate boundaries. When scored on strong boundaries only, those drawn by a large share of the annotators, all nine detectors they tested lost significant performance.
• A test set that also selects the threshold. As described under Evaluation, ODS and OIS choose their thresholds on the test images, for the region measures too. The official guidelines ask for a single run on the test set, but nothing in the code enforces it, and the benchmarking article discusses the general risk of tuning on a public test set. Arbeláez et al. compared methods pairwise image by image on the BSDS300 and found every difference statistically significant except the one between mean shift and normalized cuts (their Fig. 18); later papers usually report only the three summary numbers.
• What the human score means. The human F-measure scores each annotator against the others, which are fewer than the annotators an algorithm is matched against, so it is not an upper bound on machine scores. Exceeding it, as Liu et al. (2019) reported for their detector, shows agreement with the pooled human maps under this protocol. It does not show that a detector finds boundaries the way people do.
• Properties of the region measures. In our reading of the definition, covering of the ground truth by an over-segmentation is low because each human region is matched to one small machine region. Unlike GCE and LCE, covering therefore penalizes refinement. The Best score lets every region choose its own scale.
• Tooling. The official benchmark needs MATLAB and C++ MEX files compiled for Linux or macOS. Reimplementations exist, such as the evaluation in Dollár’s structured-edge toolbox, which follows the same counting rules and calls the same matching function.

Common Uses

• Benchmark for boundary detection, reporting ODS, OIS and AP on the BSDS500 test set, from Pb and gPb to structured forests and convolutional networks (edge detection).
• Benchmark for generic segmentation, including hierarchical methods and classical baselines such as normalized cuts, mean shift and Felzenszwalb–Huttenlocher graph segmentation.
• Training data for learned boundary detectors and for learning grouping cues.
• A source of natural test images for other tasks. Denoising work, for example, reuses BSDS images in the BSD68 test set (image denoising).

Historical Milestones

All scores are F-measures at ODS on the stated version, as reported by the cited authors.

• 2001. The dataset paper scored Normalized Cuts, with the number of segments set to match each human segmentation. Its mean error against humans was 22% by LCE and 28% by GCE, against 7% and 11% between humans.
• 2004, Pb. Martin, Fowlkes and Malik trained a logistic model on brightness, color and texture gradients over half discs. In their paper (Fig. 3; half-resolution test images, $d_{\max}$ = 1% of the diagonal) the color detector reached 0.67, the grayscale one 0.65, and Canny-style Gaussian-derivative detectors 0.58, against a median human 0.80. On the official BSDS300 ranking page, which evaluates at full resolution, their color and grayscale detectors score 0.65 and 0.63. Arbeláez et al. list Pb at 0.65 on the BSDS300.
• 2011, gPb and gPb-owt-ucm. Arbeláez et al. combined multiscale local cues with spectral “globalization” (gPb), then built hierarchical segmentations whose boundaries kept the same quality. They reported gPb at 0.70 (BSDS300) and 0.71 (BSDS500), and gPb-owt-ucm at 0.71 and 0.73. On the BSDS500, gPb-owt-ucm had OIS 0.76 and AP 0.73, against Canny at 0.60, 0.63 and 0.58 (Table 1). The BSDS500 itself was released with this paper.
• 2013–2015, structured forests. Dollár and Zitnick predicted local edge patches with random forests. On the BSDS500 their variants scored 0.73 to 0.75, against 0.73 for gPb-owt-ucm, and the basic model ran at 30 frames per second (TPAMI version, Table 1). They tuned parameters on the validation set only.
• 2015, HED. Xie and Tu trained a fully convolutional network with supervision at several depths. They used labels where at least three annotators agreed as positives. It reached ODS 0.782, OIS 0.804 and AP 0.833 on the BSDS500 (ICCV version, Table 4), against a human 0.80. Appendix A of the arXiv version (arXiv:1504.06375v2) reports a retrained model at ODS 0.790, and 0.788 with late merging, the figure later papers such as Liu et al. (2019) list for HED.
• 2017–2019, above the human figure. Liu et al. reported ODS 0.806 for RCF and 0.811 with multiscale testing (TPAMI 2019 version), trained on the augmented BSDS500 training and validation sets plus PASCAL VOC Context. They described these as better than the human 0.803.

Related Datasets

• Sowerby. The 2001 paper names the Sowerby image set as the closest earlier dataset: small, not public, and with one segmentation per image.
• NYU Depth V2. Indoor RGB-D images used for edge detection with a larger tolerance. Xie and Tu (2015) and Liu et al. (2019) raise $d_{\max}$ from 0.0075 to 0.011 for it.
• MSRC and PASCAL VOC 2008. Arbeláez et al. evaluated covering on the object-instance annotations of the MSRC images and of the PASCAL VOC 2008 segmentation set.
• ADE20K. A scene dataset that, unlike BSDS, names its regions.
• Figure-ground data. The Berkeley figure-ground labels for a subset of BSDS images.

Resources

• BSDS500 page: the dataset with ground truth and benchmark code (BSR_bsds500.tgz), the gPb and OWT-UCM code, precomputed results and the best-practice guidelines.
• BSDS300 page: images, segmentations, the .seg format, the browsable segbench code, and the algorithm ranking of the original benchmark.
• BIDS mirror of the January 2013 archive: a third-party copy in which the benchmark code can be read online.
• Structured Edge Detection Toolbox: Dollár’s MATLAB toolbox, whose edgesEvalDir.m and edgesEvalImg.m reproduce the boundary benchmark and add recall at 50% precision.