A Database of Human Segmented Natural Images and its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics

“A Database of Human Segmented Natural Images and its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics” is a paper by David Martin, Charless Fowlkes, Doron Tal and Jitendra Malik of the University of California, Berkeley, presented at the IEEE International Conference on Computer Vision (ICCV) in Vancouver in July 2001. It reported a growing collection of natural photographs, each divided into regions by several people, and argued that segmentation algorithms could be scored against those people. To make that possible, it defined two error measures that do not penalize one person drawing finer regions than another, showed with them that different people segment the same image consistently, and used the data for two purposes: a benchmark, in which Normalized Cuts was compared with the humans, and a set of measurements of how proximity, brightness similarity, convexity and region size relate to grouping in natural scenes. The PAMI Technical Committee awarded it the Helmholtz Prize in 2015. This article covers the 2001 publication. The dataset as released (BSDS300 and BSDS500) and the benchmarks as the code computes them are described in the Berkeley Segmentation Dataset article.

The authors’ copy of the ICCV paper (8 pages) from the Berkeley grouping site was read, together with two copies of the Berkeley technical report UCB/CSD-01-1133, both with a cover dated January 2001 and 11 pages. The PDF in the EECS technical-report archive has the ICCV text in a longer layout. A PostScript copy in the grouping site’s papers directory, typeset on 23 January 2001, has an earlier text with the differences noted below; it is called “the January 2001 report” here. Section, equation and figure numbers are those of the ICCV paper unless stated otherwise. The IEEE Xplore copy was not compared.

Problem

The paper starts from a contrast between recognition and segmentation. Recognition is classification, so its error rate can be estimated by counting mistakes on a shared test set; the authors name the MNIST digits and the FERET faces as the test sets that let algorithms be compared by the same numbers. Segmentation results, by contrast, were usually shown on a few images chosen to look good, and a reader could not tell whether they were typical or whether the method would fail on textured scenes.

The difficulty, as the authors put it, is that “what is a correct segmentation” is a subtler question than whether a digit is a 5. Borra and Sarkar (1997) had argued from this that grouping can be evaluated only inside a task such as object recognition. The paper’s thesis is that segmentations can also be evaluated as segmentations, by comparison with several human observers, provided those observers agree enough for the comparison to be reliable. Showing that they do agree was therefore the first job.

Earlier evaluations with human observers were limited. Heath et al. (1997) had rated edge detectors by how easily people recognized objects in the edge maps, a subjective scale. The closest dataset, the Sowerby images used by Huang and by Konishi et al., was small, not public, and had one segmentation per image.

The second motivation came from natural-image statistics. Most earlier studies measured first- and second-order statistics, such as power spectra and covariances of pixels or wavelet coefficients. Brunswik and Kamiya (1953) had suggested that the Gestalt grouping factors work because they reflect the statistics of natural scenes, but by 2001 measured probability distributions existed only for good continuation. Human segmentations would let the other factors be measured directly.

Contribution

• A database and a collection tool. 1,000 Corel images selected for segmentation, a Java program anyone with Internet access could use to segment them, and a server that assigned images under rules designed to control the data.
• Two refinement-tolerant error measures, the Global and Local Consistency Errors (GCE and LCE), and the empirical finding that human segmentations of the same image are consistent under them.
• A benchmark protocol that makes an algorithm produce as many segments as each human did and averages the error, demonstrated on Normalized Cuts.
• Ecological statistics of proximity, intensity similarity, convexity and region area, computed from the human segmentations.

Method

Images, tool and instructions

The authors chose 1,000 “representative” 481 × 321 RGB images from the Corel image database, which they describe as a collection of 40,000 images widely used in computer vision. The selection rule was that an image shows a natural scene with at least one discernible object; this excluded images unsuited to recognition, such as neon signs reflected on wet pavement or marble textures. The January 2001 report states the 1,000 images and the rule but not the image size.

The Java tool represents a segment simply as a set of pixels, so every segmentation is an explicit partition. A user splits a segment by drawing a contour around part of it in a “splitter” window and can move pixels between any two segments. Figure 2 shows a session in which two of the pups in a photograph are cut out of their segment and then split from each other. The authors report that a pixel-accurate segmentation with 10 to 20 segments took less than five minutes on a standard PC. Running the tool over the Internet with a server assigning images gave the authors precise control over the content of the database as it grew.

The instructions were made vague on purpose, so that differences between people would come from different perceptual organizations of the scene rather than different readings of the task. Subjects were asked to divide each image into pieces, each a distinguished thing, of roughly equal importance, and told that between 2 and 20 pieces should be reasonable for any image. Nothing restricted the cues they could use: the paper notes that using familiarity with faces to segment one of the Figure 1 images is perfectly acceptable, accepts that a method based only on low-level color and texture coherence would then find perfect performance difficult, and presents this as deliberate, a “gold standard” without prior bias toward any cue or algorithm. The first subjects were students of a graduate computer vision class, told to segment as naive observers and shown example segmentations of simple images; the acknowledgments identify them as the Fall 2000 students of UCB CS294. Images were assigned at random, biased toward images someone else had already segmented, with three constraints: no subject saw an image twice, no image was segmented by more than five people, and no two images had exactly the same set of subjects.

Status of the database

The two texts record the collection at different moments (§2.3 and §6 of the ICCV paper):

Version	Data used for the results	Collected at the time	Stated goal
January 2001 report (PostScript copy)	150 segmentations by 10 people of 50 images, 1–5 people per image, 30 images with 3 or more	the same	5 segmentations of each of 1,000 images; the conclusion says 3–5 in both grayscale and color
ICCV paper, July 2001	the same first version, described as 150 grayscale segmentations	3,000 segmentations by 25 people of 800 images	at least 4 grayscale and 4 color segmentations of each of 1,000 images

Every result in the paper, including the ecological statistics, therefore comes from 50 images segmented in grayscale. The later counts of the full collection (12,000 segmentations by 30 subjects) and of the public BSDS300 are given in the Berkeley Segmentation Dataset article.

Error measures

Section 3 starts from the observation that two people may segment an image differently either because they perceive it differently or because they stop at different levels of detail. The paper treats the first as a real inconsistency and the second as harmless: as the introduction puts it, one can think of perception as imposing a tree on the image, and two observers with the same tree may cut it at different depths. Figure 3 shows three human segmentations of one image, two of which are simple refinements of the third while refining each other in different places (“mutual refinement”). Besides tolerating refinement, a measure should be independent of how finely the image is pixelated, robust to noise along region boundaries, and able to compare segmentations with different numbers of segments, because complex images are segmented in different numbers of pieces (§3.2 reports counts for one image varying by up to a factor of 10).

Both measures are built on a per-pixel refinement error, eq. (1): the fraction of the pixel’s region in one segmentation that lies outside its region in the other. It is zero when the first region is contained in the second, and it is not symmetric. GCE, eq. (2), sums it over all pixels in each direction and keeps the smaller sum, so all refinement must go the same way; LCE, eq. (3), takes the smaller direction separately at every pixel, so mutual refinement is also free. LCE is never larger than GCE. The formulas are written out in the Berkeley Segmentation Dataset article. The paper states the price of this tolerance itself: one segment per pixel refines every segmentation, and every segmentation refines a single segment covering the image, so both trivial answers score zero, and the measures are meaningful only between segmentations with roughly equal numbers of segments.

The measures on a rendered scene

The script below computes GCE and LCE from the joint histogram of region labels and applies them to synthetic segmentations of a rendered scene: sky, ground, a house with a roof, door and window, and a tree with a trunk and crown. One “person” draws four segments (sky, ground, house, tree), one splits only the house into its parts, one only the tree, and one draws the coarse segmentation with the house traced 3 pixels to the side. A segmentation of a different scene and the two trivial segmentations complete the human comparisons. The last part imitates the protocol of §4: a clustering of the rendered image’s intensities, a deliberately weak segmenter that uses no spatial information, is run with the number of segments of each person and compared with that person, over ten noise seeds.

import numpy as np
from scipy.cluster.vq import kmeans2

H, W = 120, 180
yy, xx = np.mgrid[:H, :W]

def scene(shift=0):
    """Label map of a rendered scene: the finest parts a person might draw.
    0 sky, 1 ground, 2 wall, 3 roof, 4 door, 5 window, 6 trunk, 7 crown.
    `shift` moves the house sideways, as a person tracing a boundary slightly off would."""
    lab = np.where(yy < 70, 0, 1)
    wall = (yy >= 45) & (yy < 90) & (xx >= 30 + shift) & (xx < 90 + shift)
    roof = (yy >= 25) & (yy < 45) & (np.abs(xx - 60 - shift) <= (yy - 25) * 1.5)
    lab[wall], lab[roof] = 2, 3
    lab[(yy >= 65) & (yy < 90) & (xx >= 52 + shift) & (xx < 66 + shift)] = 4
    lab[(yy >= 52) & (yy < 62) & (xx >= 72 + shift) & (xx < 84 + shift)] = 5
    lab[(yy >= 60) & (yy < 95) & (xx >= 128) & (xx < 136)] = 6
    lab[(yy - 45) ** 2 + (xx - 132) ** 2 <= 22 ** 2] = 7
    return lab

def merge(lab, groups):
    """A coarser segmentation: each group of part labels becomes one segment."""
    out = lab.copy()
    for g in groups:
        out[np.isin(lab, g)] = g[0]
    return out

def errors(s1, s2):
    """GCE and LCE (eqs. 2 and 3) from the joint histogram of region labels.
    A pixel in region a of s1 and b of s2 has E(s1,s2) = (|a| - |a&b|) / |a|."""
    _, a = np.unique(s1, return_inverse=True)
    _, b = np.unique(s2, return_inverse=True)
    n = np.zeros((a.max() + 1, b.max() + 1))
    np.add.at(n, (a.ravel(), b.ravel()), 1)
    r1, r2 = n.sum(1, keepdims=True), n.sum(0, keepdims=True)
    e12, e21 = (r1 - n) / r1, (r2 - n) / r2          # local refinement errors per region pair
    npix = s1.size
    gce = min((n * e12).sum(), (n * e21).sum()) / npix
    lce = (n * np.minimum(e12, e21)).sum() / npix
    return gce, lce

def nseg(s):
    return len(np.unique(s))

fine = scene()
house, tree = [2, 3, 4, 5], [6, 7]
people = {
    "coarse":      merge(fine, [house, tree]),               # sky, ground, house, tree
    "house parts": merge(fine, [tree]),                      # refines the house only
    "tree parts":  merge(fine, [house]),                     # refines the tree only
    "shifted":     merge(scene(shift=3), [house, tree]),     # coarse, house traced 3 px off
}
other = np.where(xx < 90, 0, 1) + 2 * (yy > 60) + 4 * ((xx - 40) ** 2 + (yy - 90) ** 2 < 400)

def show(name, s1, s2):
    g, l = errors(s1, s2)
    print(f"{name:34s} {nseg(s1):3d} {nseg(s2):5d}   GCE {g:.3f}  LCE {l:.3f}")

print(f"{'pair':34s} segs  segs")
show("coarse vs house parts (refines)", people["coarse"], people["house parts"])
show("house parts vs tree parts (mutual)", people["house parts"], people["tree parts"])
show("coarse vs shifted (boundary noise)", people["coarse"], people["shifted"])
show("coarse vs a different scene", people["coarse"], other)
show("coarse vs one segment", people["coarse"], np.zeros_like(fine))
show("coarse vs one segment per pixel", people["coarse"], np.arange(fine.size).reshape(H, W))

# Section 4 in miniature: an intensity clustering of the rendered image, run with the number of
# segments of each human segmentation, then compared with that segmentation. Over seeds.
shade = np.array([0.80, 0.45, 0.62, 0.30, 0.20, 0.90, 0.25, 0.40])
print("\nmachine (k-means on intensity, k = segments of each person), 10 seeds:")
for name, s in people.items():
    k = nseg(s)
    res = []
    for seed in range(10):
        rng = np.random.default_rng(seed)
        img = shade[fine] + 0.04 * np.sin(xx / 2.0) * (fine == 7) + 0.06 * rng.normal(size=fine.shape)
        _, seg = kmeans2(img.reshape(-1, 1), k, seed=seed, minit="++")
        res.append(errors(s, seg.reshape(H, W)))
    res = np.array(res)
    print(f"  vs {name:12s} k={k}:  GCE {res[:, 0].min():.3f}-{res[:, 0].max():.3f}"
          f"  LCE {res[:, 1].min():.3f}-{res[:, 1].max():.3f}")

Output (NumPy 2.3.5, SciPy 1.16.3; under a second):

pair                               segs  segs
coarse vs house parts (refines)      4     7   GCE 0.000  LCE 0.000
house parts vs tree parts (mutual)   7     5   GCE 0.018  LCE 0.000
coarse vs shifted (boundary noise)   4     4   GCE 0.035  LCE 0.035
coarse vs a different scene          4     5   GCE 0.401  LCE 0.347
coarse vs one segment                4     1   GCE 0.000  LCE 0.000
coarse vs one segment per pixel      4 21600   GCE 0.000  LCE 0.000

machine (k-means on intensity, k = segments of each person), 10 seeds:
  vs coarse       k=4:  GCE 0.274-0.332  LCE 0.250-0.291
  vs house parts  k=7:  GCE 0.257-0.295  LCE 0.213-0.252
  vs tree parts   k=5:  GCE 0.269-0.313  LCE 0.240-0.274
  vs shifted      k=4:  GCE 0.292-0.344  LCE 0.267-0.303

The first three lines reproduce the behaviour that the paper describes with Figure 3. Splitting the house costs nothing under either measure. The two people who refine different objects are free under LCE but not under GCE, because GCE must pick one direction for the whole image and the tree parts then count against the house parts; the penalty is small here only because the tree is small. A boundary traced 3 pixels off costs a few percent, in proportion to the strip of disputed pixels. A segmentation of another scene scores about 0.4 and 0.35, and the two trivial segmentations score exactly zero, which is why the benchmark has to fix the number of segments. The intensity clustering, given each person’s count, falls between agreement among the people and the different-scene comparison for every seed, which is the pattern the paper reports for Normalized Cuts; it stays far from the people because it ignores position: parts with similar shades, such as the roof, door and trunk, can fall into one cluster wherever they are, and the noise scatters single pixels between clusters.

Results

Human consistency

Section 3.2 applies both measures to all pairs of segmentations in the 150-segmentation set (Figure 4). Pairs of segmentations of the same image have error concentrated near zero, with means of 0.11 (GCE) and 0.07 (LCE); pairs of different images have broad distributions with means of 0.39 and 0.30. The separation is summarized by one number: for LCE, 5.9% of pairs lie on the wrong side of a threshold of 0.12 (same-image pairs above it or different-image pairs below it), and for GCE, 5.9% lie on the wrong side of 0.16. A spike at zero among the different-image pairs comes from degenerate segmentations of one image, which every other segmentation refines; the error matrices of Figure 5, sorted by image, show it as horizontal and vertical bands across an otherwise block-diagonal pattern. Figure 6 plots GCE against LCE for every pair and shows all points on or above the diagonal. The paper concludes that the vague task produced consistent segmentations, and that the measures are validated by separating same-image from different-image pairs.

Normalized Cuts against the humans

Section 4 turns the measures into a benchmark. Because both trivial segmentations score zero, the algorithm must produce as many segments as each human segmentation: if three people drew 4, 9 and 13 segments, the algorithm is run three times with those counts and each result is compared with the corresponding person. The mean error over all images is proposed as the summary statistic for ranking algorithms. The authors argue that this control is reasonable because segmentation without a stated granularity is ill-posed, so any algorithm should offer some control over the number of segments.

The algorithm tested was Normalized Cuts (Shi and Malik, 2000) with the contour and texture cues of Malik, Belongie, Leung and Shi’s 2000 book chapter. That version stops splitting the graph when the cut value passes a threshold; the authors changed the stopping rule so that it returns a requested number of segments. Figure 7 shows its output on four of the Figure 1 images.

Mean error (Figs. 4 and 8, §4)	GCE, same image	LCE, same image	GCE, different images	LCE, different images
Human vs. human	0.11	0.07	0.39	0.30
NCuts vs. human	0.28	0.22	0.38	0.31

The paper reads the different-image errors as the performance of a random segmentation, and concludes that Normalized Cuts is worse than the humans but better than random. Figure 9 shows the GCE for each image with at least three human segmentations, as box plots of human-versus-human and NCuts-versus-human error. The human errors are mostly tightly packed near zero, NCuts is worse on virtually every image, and its error varies from image to image, which the authors suggest can identify the kinds of image an algorithm finds hardest.

Ecological statistics

Section 5 treats the Gestalt factors as prior distributions that a Bayesian grouping algorithm could use and measures them from the human segmentations. For proximity and similarity, 1,000 points were sampled from each segmentation and every pair was scored; the figures show error bars of ±1 standard deviation.

• Proximity (Figure 10). The probability that two points share a segment, as a function of their distance. To make the estimate independent of how finely an image was segmented, distances are multiplied by $\sqrt{\text{number of segments} / \text{image area}}$ and then normalized to the range 0 to 1. The probability is 1 at distance zero and decreases monotonically; in the plot it flattens out at about a third at the largest distances, with wide error bars. The January 2001 report adds that the decrease is not trivially true: in a world made of bagels floating in space, the probability would fall, rise and fall again.
• Similarity (Figure 11). The same probability as a function of the absolute difference in gray level (256 levels). If objects were uniform in intensity and each had its own intensity, the curve would start at 1 and fall quickly to 0. The text describes the measured curve as starting at 0.6 and decaying monotonically to 0.2 (read from the plot, the points run from about 0.57 to about 0.16, with small bumps), which the authors attribute to texture, shading and lighting within objects: intensity similarity is an imperfect but informative cue. They note that the analysis could be extended to color and texture.
• Convexity (Figure 12). The ratio of a region’s area to the area of its convex hull, which is 1 for a convex region. The regions carry no figure or ground labels, so the distribution covers all regions; the authors argue that since an image has many foreground regions and only a few background ones, the foreground distribution would look much the same. The histogram rises toward 1: grouped pixels commonly form convex regions.
• Region area (Figure 13). The frequency of region areas follows a power law $y = A/x^\alpha$ with $\alpha = 1.008$, fitted after discarding the sparsely populated bins of regions larger than 25% of the image. The authors relate this to Alvarez, Gousseau and Morel (1999), who found a power law for connected components of intensity level sets, and argue from the similarity result that such level sets are only a rough approximation of perceptual segments.

Impact

The database became the Berkeley Segmentation Dataset. Its public release, the BSDS300, offered 300 of the images with all their segmentations, and its page asks that work using the data cite this paper; the page also distributed the Java segmentation tool for making ground truth on other images. Martin, Fowlkes and Malik (2004) then used the same human segmentations in a different way, as boundary ground truth, and defined the precision–recall boundary benchmark, which Arbeláez et al. (2011) describe as having found wide acceptance for boundary detection; Arbeláez et al. (2011) extended it to the BSDS500 with new test images and region measures. Both benchmarks are described in full in the Berkeley Segmentation Dataset article.

The ecological-statistics part also had direct successors from the same group. Fowlkes, Martin and Malik (2003) learned pixel affinities for segmentation from the human segmentations, treating “same segment or not” as a classification problem (below). Fowlkes, Martin and Malik (2007) had people assign figure and ground to the boundaries of 200 of the 1,000 images and found that figural regions tend to be smaller, more convex and lower in the image, measuring for figure and ground what this paper could only measure for all regions.

The PAMI Technical Committee’s Helmholtz Prize, presented at each ICCV and described by the committee as recognizing ICCV papers from ten years earlier with significant impact on computer vision research, went to this paper in 2015, fourteen years after publication, according to the committee’s list of winners.

Limitations

The paper states several limitations itself:

• The measures need matched counts. Because refinement is free, GCE and LCE give zero for one segment per pixel and for one segment per image, so they mean something only for segmentations with similar numbers of segments. The benchmark handles this by giving the algorithm each human’s count.
• Cues are unrestricted. People were free to use recognition, so an algorithm based only on low-level coherence would find perfect agreement with them difficult.
• Similarity used only intensity differences, and convexity could not separate figure from ground.

Other limitations follow from the data and the design (our reading, except where a source is named):

• A small first sample. All numbers come from the first 50 images, segmented in grayscale, with 30 images having three or more segmentations; the ecological statistics are drawn from the same 150 segmentations. The power-law fit spans about one and a half decades of region area, from about $10^3$ pixels in the plot to the 25% cutoff (about $3.9 \times 10^4$ pixels on a 481 × 321 image).
• An oracle granularity. Giving the algorithm each human’s segment count supplies information an algorithm would not have in use, and the measures say nothing about choosing the count. The benchmark therefore compares algorithms at matched granularity rather than as finished segmenters.
• A coarse baseline. The “random” reference is the error between segmentations of different images, not a model of random segmentation.
• Printed inconsistencies. The captions of Figures 4 and 8 list the four means “clockwise from the top-left”, which would make the different-image LCE mean larger than the GCE mean, impossible since LCE never exceeds GCE. The January 2001 report gives the same numbers in a table (its Figure 9) that labels them by measure, matching the reading used above: the caption lists the panels row by row. The text of §4 describes the Figure 9 boxes as blue (human) and red (NCuts), while the figure and its caption use gray and white (the EECS archive copy has the same mismatch); in the January 2001 report the caption of the figure (there Figure 10) and the text both say blue and red. In Figure 6 the left panel is titled “all segmentations” while the caption says it shows segmentations of different images, and the text says GCE > LCE “in all cases” where the definition gives LCE ≤ GCE, with equality possible.
• Later counts exceed the stated cap. The paper’s assignment rule allowed at most five people per image, while Martin, Fowlkes and Malik (2004) describe 5 to 10 segmentations per image for the full collection; no source explains the change.

What Came After

Neither later benchmark used GCE or LCE. The 2004 boundary benchmark matched boundary pixels instead of comparing regions, and the BSDS500 benchmark of Arbeláez et al. (2011) scored regions with segmentation covering, the probabilistic Rand index and the variation of information. The Berkeley Segmentation Dataset article describes these measures, the human F-measure figures and the later doubts about weak labels.

The instructions changed too. The 2004 paper quotes subjects being asked to segment the image into “things” or “parts of things” (its words), with 2 to 30 segments as a likely range, against 2 to 20 in 2001. Its human-agreement figure, a median F-measure of 0.80 on boundaries, plays, in our reading, the role that the 0.11 and 0.07 consistency means played here.

Some of the paper’s ecological claims were refined by the follow-up work. In the copy of Fowlkes, Martin and Malik (2003) on the Berkeley site, labelled as submitted to CVPR 2003, a classifier trained on brightness, color and texture cues gained nothing from adding the distance between two pixels; the authors conclude that proximity is a result of grouping rather than a cue in its own right, which is, in our reading, a different interpretation of the monotone curve in Figure 10. The same paper found that regions are mostly convex, supporting the “intervening contour” cue, and estimated the ceiling for pairwise affinities from human agreement, using the color segmentations of 500 images. The Berkeley Segmentation Dataset article lists the data’s later uses, among them the evaluation of superpixel methods.