Bilateral Filtering for Gray and Color Images

“Bilateral Filtering for Gray and Color Images” is an eight-page paper by Carlo Tomasi, then at Stanford University, and Roberto Manduchi, then at Apple Computer, presented at the Sixth International Conference on Computer Vision (ICCV) in Bombay in 1998. It proposed smoothing an image by averaging each pixel with neighbors that are both close to it and similar to it in value, in a single pass, and showed that measuring similarity in the CIE-Lab color space lets the same filter smooth color images without creating new colors at edges. The paper gave the bilateral filter its name, and in 2013 it received the Helmholtz Prize, the PAMI Technical Committee’s award at ICCV for papers from ten years earlier that have had a significant impact on computer vision.

Problem

Gaussian smoothing rests on the assumption that nearby pixels have similar values, so averaging them removes noise while keeping the signal. At edges the assumption fails, and the edges blur. Many edge-preserving alternatives existed. The one the authors singled out, anisotropic diffusion, adapts the averaging to local image variation but does so by solving a partial differential equation, which makes it inherently iterative; Tomasi and Manduchi pointed to the stability and efficiency questions that iteration raises.

Color images added a second problem. Smoothing the red, green, and blue bands separately disturbs the balance of colors near an edge, because the bands differ in contrast, and produces colors that were not in the scene. The paper’s motivating example is a red jacket against a blue sky, where averaging across the boundary gives pink and purple fringes.

Contribution

• Range filtering as the counterpart of domain filtering. Ordinary filtering weights pixels by closeness in the image domain; an analogous filter can weight them by similarity in the image range. Alone, a range filter is of little use, because it merely remaps the gray levels.
• The combination, named bilateral. Multiplying the two weightings gives a filter that averages only nearby, similar pixels. It is non-iterative, local, and conceptually simple, and the paper emphasized that a single application already removes noise and fine texture while keeping contours sharp.
• Perceptual color filtering. Because the similarity function takes an explicit distance, the filter can operate on color vectors and use Euclidean distance in CIE-Lab, where short distances correspond to perceived color differences. Only perceptually similar colors are then averaged.
• An analysis of how the two components interact, which the authors said earlier weighting schemes lacked.

Method

The paper defines the filter in continuous form for multiband images. A closeness function $c$ weights each point by its distance from the neighborhood center, a similarity function $s$ weights it by the photometric distance between its value and the center’s value, and the output is the normalized integral of the image values times both weights. The normalization depends on the image, because the similarity weights do. A similarity function that depends only on the difference of the two values makes the filter insensitive to adding a constant to all intensities.

The main example uses Gaussians for both functions, with a geometric spread $\sigma_d$ in pixels and a photometric spread $\sigma_r$ in gray levels. The paper’s guidance is that $\sigma_d$ sets the amount of low-pass filtering and must scale with image size, while values much closer than $\sigma_r$ are mixed and values much farther apart are not, so $\sigma_r$ must scale with image contrast.

Section 3 analyzes the range filter on its own. Ignoring space, the output at a pixel depends only on its gray level: it is the mean of the image histogram weighted by the similarity function centered at that level. The authors used this to show that range filtering compresses a unimodal histogram toward its mode. Combined with a box-shaped domain window, the bilateral filter becomes a range filter applied locally, a gray-level remapping that differs from place to place; for a noisy step, each lobe of the bimodal histogram is compressed separately, which is the separate smoothing of the two sides.

For color, the photometric distance is Euclidean distance in CIE-Lab, computed on the three bands together. The authors noted that other similarity functions, including intensity-dependent ones, and oriented domain filters were possible.

Results

The evaluation is visual; the paper reports no error measures or timings. A synthetic step of 100 gray levels with Gaussian noise of standard deviation 10, filtered with $\sigma_d = 5$ pixels and $\sigma_r = 50$ gray levels, comes out with flat, smoothed sides and a crisp step. On a photograph of a kitten, whiskers much thinner than the filter window stay sharp. A grid of results over $\sigma_d$ and $\sigma_r$ (with $\sigma_r$ from 10 to 300) shows the filter acting as a Gaussian when $\sigma_r$ is large relative to the image range, and preserving edges for $\sigma_r$ of 10 or 30. With a very wide domain kernel ($\sigma_d = 10$), detail removed by smaller kernels reappears in a hazy image, which the authors explained by the histogram-compression result: the filter has become essentially a range filter. With $\sigma_d = 3$ and $\sigma_r = 50$, a photograph loses most of its fine texture while its contours stay sharp.

For color, smoothing the jacket image with a standard low-pass filter widened the pink fringe, and bilateral filtering of the RGB bands separately made it more pronounced, though not wider. Filtering in CIE-Lab shrank the fringe considerably and introduced no extraneous colors. Five iterations flattened another image into cartoon-like regions.

On cost, the authors stated that the filter is twice as expensive as a nonseparable domain filter of the same size, and that precomputing the similarity weights in a table, one entry per possible gray-level difference, reduces it considerably.

Impact

Paris, Kornprobst, Tumblin and Durand (2009) open their survey by describing the filter as ubiquitous in image processing, with uses including denoising, texture editing and relighting, tone management, demosaicking, stylization, and optical flow estimation, and attribute its success in part to its simple formulation, its two parameters, and its non-iterative use. The same survey records that the idea was older: the nonlinear Gaussian filters of Aurich and Weule (1995) and the SUSAN framework of Smith and Brady (1997) already combined spatial and intensity weights, and Yaroslavsky’s neighborhood filter (1985) averaged nearby pixels of similar value. Tomasi and Manduchi did not cite these; among the works they did cite, they named Overton and Weymouth (1979) as the closest, a scheme multiplying a geometric and a photometric term but with slowly decaying weights that forced iteration. The name and the non-iterative Gaussian formulation the field adopted are theirs.

Durand and Dorsey (2002) used the filter to split an image into a base and a detail layer for high-dynamic-range tone mapping, with a fast piecewise-linear approximation, and related it to anisotropic diffusion. Later work recast it as a linear filter in a higher-dimensional space, which led to the bilateral grid and the permutohedral lattice; the joint (cross) bilateral filter took the range weights from a second image; and fully connected conditional random fields with bilateral pairwise terms were used to refine semantic segmentation boundaries (see Bilateral Filter).

Limitations

The paper chose its examples’ parameters, by its own account, somewhat arbitrarily, and left automatic parameter selection open. It offered no quantitative comparison with other edge-preserving filters and no noise model for setting $\sigma_r$. The brute-force cost grows with the square of the window radius, which made large kernels slow until the fast approximations of the 2000s.

Later analyses identified behaviors the paper did not discuss. Buades, Coll and Morel argued in their non-local means paper that comparing the values of single pixels is not robust when those values are noisy, and showed that such neighborhood filters turn smooth ramps into staircases. When the residual detail layer is amplified, the filter can produce gradient reversal artifacts next to edges, which He, Sun and Tang (2013) attributed to edge pixels with few similar neighbors. Impulse noise differs too much from its neighbors to be averaged away, and the Paris survey recommends prefiltering for it.

What Came After

Non-local means (2005) replaced the comparison of single values with a comparison of whole patches, which made the weights robust to noise and became the basis of patch-based denoising. The guided filter of He, Sun and Tang (2013) offered similar edge-preserving behavior with an exact algorithm whose cost does not depend on the kernel size, and avoided gradient reversal. Paris and colleagues’ 2009 survey consolidated the theory, relating the filter to local mode filtering, robust statistics, partial differential equations such as anisotropic diffusion, and the Yaroslavsky filter.