Edge Detection

Edge detection finds the pixels where image intensity, or another local property such as color or texture, changes abruptly. Edges compress an image into a sparse line drawing that keeps much of its structure: object outlines, surface creases, shadows, and markings. For decades an edge map was the standard first stage of vision systems, feeding line fitting, segmentation, shape matching, and stereo, and edge and boundary maps are still used for these purposes and as cues for learned models.

Problem Definition

Given an image, mark the locations of significant intensity discontinuities. “Significant” is the hard part: a definition must separate real changes from noise and say at what scale a change counts.

Idealized one-dimensional profiles help name what is being detected:

• Step edge: an abrupt jump between two levels, the model most detectors are designed for.
• Ramp edge: a step spread over several pixels, as produced by defocus, motion blur, or soft shadows.
• Roof edge: a change in slope without a change in level, as at the crease between two faces of an object.
• Line (ridge or valley): a thin bright or dark band, effectively two close steps of opposite sign, such as a wire, a crack, or a printed stroke.

Canny (1986) derived optimal operators for steps, ridges, and roofs, but left the roof detector out of his implementation, finding ideal roof edges relatively rare.

Physically, intensity changes have several causes. Marr and Hildreth (1980) list surface discontinuities and reflectance and illumination boundaries; in more detail, edges arise at depth discontinuities (occluding contours), at surface-orientation discontinuities (creases), at changes of material or paint (reflectance), and at shadow and highlight boundaries (illumination). A low-level detector sees only the intensity change and cannot tell these apart, which is one reason edge maps rarely match the outlines a person would draw.

Input and Output

The input is usually a grayscale image; color images are converted, processed per channel, or handled with a color gradient.

The output takes one of several forms:

• Soft edge map: an edge strength at every pixel, such as the gradient magnitude or a boundary probability.
• Binary edge map: a thin set of marked pixels, obtained by thinning and thresholding the soft map.
• Orientation: the edge direction at each marked pixel, perpendicular to the gradient.
• Contours: marked pixels linked into chains or curves, sometimes with sub-pixel positions.

A useful distinction separates edges from boundaries. Martin, Fowlkes, and Malik (2004) define a boundary as a contour where pixel ownership changes from one object or surface to another, and an edge as an abrupt change in a low-level feature such as brightness. Texture produces many edges but no boundaries, and two objects of similar brightness can share a boundary with almost no edge. Classical detectors find edges; benchmarks built from human annotations measure boundaries.

Challenges

• Noise. Differentiation amplifies high-frequency noise, so every detector smooths, and smoothing blurs and displaces edges. Canny formalized this as a tradeoff between detection and localization.
• Scale. The same scene contains sharp edges a pixel wide and diffuse shading edges tens of pixels wide. A filter tuned to one misses or distorts the other, and the right scale varies across the image.
• Texture. Grass, fabric, and foliage produce dense high-contrast edges inside regions that a person sees as uniform.
• Blur and low contrast. Defocus spreads steps into ramps, and boundaries between similar surfaces may have almost no contrast, so their detected edges break up.
• Corners and junctions. Models of a straight edge with a constant cross-section fail where edges meet, so detectors tend to round corners and leave gaps at junctions.
• Thresholds. Converting a soft map to a binary one requires thresholds, and values that work on one image rarely suit another.

Approaches

First-derivative operators

The earliest detectors convolve the image with small derivative kernels and threshold the gradient magnitude. Roberts’ 2×2 cross differences, the Prewitt operator, and the Sobel operator differ mainly in how much they smooth across the derivative direction. They are cheap and still common as a gradient stage, but on their own they produce thick, noisy edges whose quality depends on one global threshold.

Second-derivative zero crossings

Marr and Hildreth (1980) argued that intensity changes should be detected separately at several scales, using the second derivative of a Gaussian. Under simple conditions it need not depend on orientation, so edges become the zero crossings of the Laplacian of the Gaussian-smoothed image. Zero crossings are thin by construction and form closed contours, but they also occur at weak gradient minima and must be filtered by their slope, and they shift away from the true edge where the edge is curved.

The Canny detector

Canny (1986) turned edge detection into an optimization problem: good detection, good localization, and a single response per edge. The optimal step detector he derived is well approximated by the derivative of a Gaussian, which leads to the familiar pipeline of smoothing, gradient computation, non-maximum suppression along the gradient direction, and hysteresis thresholding. The Canny edge detector became the reference classical method, and its thinning and hysteresis steps are reused far beyond it.

Multi-scale and scale-space edges

Both Marr–Hildreth and Canny used several filter widths, combined by special-purpose rules. Scale-space theory treats scale as a continuous dimension: the image is smoothed by a family of Gaussians, as in the Gaussian filter article, and features are tracked across it. Lindeberg (1998) defined a scale-space edge as a connected set of points where the gradient magnitude is maximal along the gradient direction and a normalized measure of edge strength is also maximal over scale. The selected scale then varies along the edge: fine for sharp edges, coarse for diffuse ones.

Learned boundary detection

Hand-designed derivative filters respond to texture and miss low-contrast boundaries. Once human-annotated datasets existed, boundary detection became a learning problem. Martin, Fowlkes, and Malik (2004) computed brightness, color, and texture gradients by comparing histograms in the two halves of a disc at each location and orientation, and trained a classifier, logistic regression in the end, to output the posterior probability of a boundary, known as Pb. They reported that explicit treatment of texture was essential and that a simple linear model was adequate for combining cues. Arbeláez et al. (2011) added multiscale cues and global information from spectral clustering (gPb) and turned the result into hierarchical segmentations.

Dollár and Zitnick (2013, journal version 2015) observed that edges in a small patch have strong structure, such as straight lines and T-junctions, and trained random forests to predict a whole local edge mask per patch rather than one label per pixel. The resulting structured edge detector ran in real time while matching the accuracy of much slower methods. Xie and Tu (2015) replaced hand-designed features with a fully convolutional network: holistically-nested edge detection (HED) predicts an edge map for the whole image in one pass, adds supervision to side outputs at several depths of a VGG-based network, and fuses those multi-scale outputs. Many later deep detectors build on the same image-to-image formulation.

Datasets and Benchmarks

The Berkeley Segmentation Dataset (BSDS300), introduced by Martin et al. in 2001, contains natural images each segmented by several people; it has 200 training and 100 test images. The BSDS500 of Arbeláez et al. (2011) keeps BSDS300 for training and adds 200 new test images, and became the standard benchmark for boundary detection. NYU Depth and, for semantic contours, datasets with per-class boundaries are also used.

Evaluation Metrics

Benchmarks compare a detector’s thinned boundary map with the human boundaries. Martin, Fowlkes, and Malik (2004) defined the protocol: machine and human boundary pixels are put in explicit one-to-one correspondence, allowing a small localization error, so that double detections are penalized. Matched machine pixels give precision, matched human pixels give recall, and sweeping the detector’s threshold gives a precision–recall curve summarized by the F-measure, the harmonic mean of the two. Because several people annotate each image, a machine pixel counts as a false positive only if it matches no annotator.

Arbeláez et al. (2011) report three numbers that became standard: the F-measure at the single threshold that is best over the whole dataset (ODS, optimal dataset scale), the F-measure when the best threshold is chosen separately for each image (OIS, optimal image scale), and the average precision over the curve. Human annotators, compared against each other, reach an F-measure of about 0.80. In the tables of the structured-forest and HED papers on BSDS500, Canny scores an ODS of 0.60, the structured edge detector about 0.73 to 0.75 depending on the variant, and HED 0.782, as reported by the respective authors.

Applications

• Segmentation and region extraction: boundary maps are converted into regions, as in the hierarchical segmentation of Arbeláez et al.
• Geometric primitives: lines, circles, and vanishing points are fitted to edge pixels, for example with the Hough transform.
• Inspection and measurement: sub-pixel edge positions give the dimensions of manufactured parts and the positions of features such as document borders and lane markings.
• Matching and registration: chamfer and distance-transform matching align edge maps, and edges guide stereo and alignment where intensity correlation fails.
• Object proposals and image editing: boundary strength is used to rank candidate object regions and to guide edge-aware filtering and drawing tools.

Edge maps are often cleaned with mathematical morphology, and the related corner and interest-point detectors underpin feature matching.

Open Problems

• Edges versus boundaries. Separating object boundaries from texture and shading edges requires knowledge of objects, so the line between edge detection and segmentation or recognition is blurred.
• Localization of learned detectors. Deep detectors produce thick responses and still rely on non-maximum suppression for thin output; crisp, well-localized boundaries remain an active topic.
• Ambiguous ground truth. Annotators disagree about which boundaries matter, so a single F-measure hides a real ambiguity in the task.
• Generalization. Detectors trained on a few hundred natural images may transfer poorly to medical, aerial, or industrial images, where classical detectors with tuned parameters remain common.
• Junctions and topology. Producing closed, correctly connected contours with correct junctions, rather than a set of independently scored pixels, is still largely unsolved.