Image Filtering

Image filtering computes each pixel of an output image from a small neighborhood of pixels around the same position in the input. Blurring, sharpening, finding edges, and removing noise are all filters; they differ only in the rule that turns a neighborhood into one value. The choice of filter decides which structure survives: smoothing suppresses noise but also fine detail, and differentiation finds edges but also amplifies noise.

Definition

A filter maps an input image $f$ to an output image $g$ by applying the same rule at every pixel: the output $g[m, n]$ depends only on the input values in a window $W$ (the neighborhood or support) centered on $[m, n]$ . Filters are classified by the rule:

A linear shift-invariant filter forms a fixed weighted sum of the neighborhood. It is a convolution with a kernel of weights, and it is fully described by that kernel.
A nonlinear filter uses any other rule: sorting the neighborhood and picking a value (rank or order-statistic filters), or weighting neighbors by how similar they are to the center pixel (edge-preserving filters).

Point operations, such as gamma correction or thresholding, which use only the pixel itself, are not filters in this sense; neither are geometric transformations, which move pixels rather than recompute them.

Intuition

Picture a window sliding across the image, writing one number to the output at each position. An averaging window cancels independent noise but also spreads an edge over the width of the window. A window that subtracts its left half from its right half responds only where intensity changes. A window that takes the middle value after sorting ignores a few extreme pixels entirely. Linear filters weight neighbors by position alone; nonlinear filters can also look at their values, which lets them keep an edge sharp while smoothing on either side of it.

Formal Definition

For a window $W$ of offsets $(i, j)$ , a general filter is

g[m, n] = F\big(\{\, f[m + i,\, n + j] : (i, j) \in W \,\}\big),

with the same function $F$ at every pixel. The linear case is a weighted sum with kernel $h$ , written here as a correlation (a convolution with the kernel reflected):

g[m, n] = \sum_{(i, j) \in W} h[i, j]\, f[m + i,\, n + j] .

The median filter takes $F$ to be the median of the values. An edge-preserving filter such as the bilateral filter uses weights that depend on both offset and value, normalized to sum to one:

g[m, n] = \frac{\sum_{(i, j) \in W} w_s(i, j)\, w_r\big(f[m + i, n + j] - f[m, n]\big)\, f[m + i, n + j]}{\sum_{(i, j) \in W} w_s(i, j)\, w_r\big(f[m + i, n + j] - f[m, n]\big)},

where $w_s$ decreases with distance and $w_r$ decreases with intensity difference. Because the weights depend on the image, this is not a convolution.

Filter Families

Smoothing

Smoothing (low-pass) filters average a neighborhood to reduce noise. The box filter gives every pixel in a $k \times k$ window the weight $1/k^2$ ; running sums make its cost independent of $k$ , but its frequency response has side lobes, so some fine detail leaks through. The Gaussian filter weights neighbors by a 2D Gaussian of standard deviation $\sigma$ . It is separable, rotationally symmetric, and attenuates high frequencies smoothly, which makes it the default for noise reduction and for smoothing before differentiation or subsampling (see sampling and aliasing and the image pyramid).

Derivative and edge filters

Derivative (high-pass) filters respond to change. The Sobel operator combines a central difference in one direction with smoothing in the other to estimate the two components of the image gradient. The Laplacian operator approximates the sum of second derivatives and responds to blobs and to the two sides of an edge with opposite signs. Derivative kernels sum to zero, so they return zero on constant regions, and because they amplify high frequencies they are usually combined with Gaussian smoothing, as in derivative-of-Gaussian and Laplacian-of-Gaussian filters.

Sharpening

Unsharp masking computes $f + \alpha (f - g * f)$ for a blur kernel $g$ and gain $\alpha$ ; by linearity it is a single convolution with a kernel that has a large positive center and a negative surround, as the convolution article shows. Sharpening amplifies noise along with detail, and strong gains produce halos at edges.

Rank filters

Rank or order-statistic filters sort the values in the window and output one of them: the median filter takes the middle value, and the minimum and maximum filters take the extremes (grayscale erosion and dilation with a flat structuring element, in mathematical morphology). The median removes impulse (salt-and-pepper) noise and preserves step edges, but also removes thin lines and rounds corners.

Edge-preserving smoothing

Edge-preserving filters smooth within regions but not across boundaries. The bilateral filter, named by Tomasi and Manduchi (1998), weights each neighbor by its distance from the center and by the similarity of its value to the center value, so pixels on the other side of an edge contribute little. Anisotropic diffusion, proposed by Perona and Malik (1990), evolves the image under a diffusion equation whose conductance is chosen to favor smoothing within regions over smoothing across region boundaries. The guided filter of He, Sun, and Tang (2013) models the output as a local linear function of a guidance image (the input itself or another image) and has an exact algorithm whose cost does not depend on the window size. Non-local means goes further, averaging pixels whose surrounding patches look alike anywhere in the image.

Frequency-domain filtering

By the convolution theorem, a linear shift-invariant filter multiplies the image’s spectrum by the kernel’s frequency response. Filters can therefore be designed and applied in the frequency domain with the Fourier transform: an ideal low-pass filter zeroes all frequencies above a cutoff, a notch filter removes a periodic pattern, and a band-pass filter keeps a range of scales. Sharp cutoffs in frequency produce ringing in space.

Properties

Separability. A 2D kernel that is the outer product of two 1D kernels can be applied as two 1D passes, reducing the cost per pixel from $k^2$ to $2k$ operations. Box, Gaussian, and Sobel kernels are separable; the median filter is not.
Kernel size and scale. A filter’s support sets the scale of the structure it acts on: a wider Gaussian removes more noise but merges features closer together than its width, and larger derivative filters respond to coarser edges.
Boundary handling. Near the border the window extends past the image, and the missing values must be supplied by zero padding, replication, reflection, or wrapping. The choice changes the output within half a kernel width of the border; the convolution article compares the options.

The Denoising–Detail Trade-off

Every smoothing filter trades noise reduction against detail. Averaging $N$ pixels with independent noise of standard deviation $\sigma$ reduces it to $\sigma / \sqrt{N}$ , but only if the underlying image is constant over the window; where it is not, the average mixes different structures and blurs them. A linear filter cannot separate noise from detail at the same frequencies. Nonlinear filters use pixel values to change the trade: the median rejects outliers, and edge-preserving filters avoid averaging across large intensity differences. They still lose fine texture whose contrast is comparable to the noise, the regime that dedicated image denoising methods target.

The example below filters a noisy vertical step edge with a $5 \times 5$ box filter, a Gaussian with $\sigma = 1.5$ , and a $5 \times 5$ median filter, and measures the remaining noise in a flat region, the 10–90% width of the average edge profile, and the error against the clean image.

import numpy as np
from scipy import ndimage

rng = np.random.default_rng(0)

# A 128 x 128 image with a vertical step edge (0 -> 1) plus Gaussian noise.
clean = np.zeros((128, 128))
clean[:, 64:] = 1.0
noisy = clean + rng.normal(0.0, 0.2, clean.shape)

filters = {
    "none":           noisy,
    "box 5x5":        ndimage.uniform_filter(noisy, size=5, mode="reflect"),
    "Gaussian s=1.5": ndimage.gaussian_filter(noisy, sigma=1.5, mode="reflect"),
    "median 5x5":     ndimage.median_filter(noisy, size=5, mode="reflect"),
}

def edge_width(img):
    """10%-90% rise distance (in pixels) of the column-averaged profile."""
    profile = img.mean(axis=0)
    lo, hi = profile[:48].mean(), profile[80:].mean()
    p = (profile - lo) / (hi - lo)
    x = np.arange(p.size)
    seg = slice(54, 75)                      # region around the edge
    x10 = np.interp(0.1, p[seg], x[seg])     # profile is increasing here
    x90 = np.interp(0.9, p[seg], x[seg])
    return x90 - x10

print(f"{'filter':15s} {'noise std':>9s} {'edge 10-90%':>11s} {'RMSE':>6s}")
for name, out in filters.items():
    flat = out[:, 8:48]                      # flat region away from the edge
    rmse = np.sqrt(np.mean((out - clean)[:, 8:120] ** 2))
    print(f"{name:15s} {flat.std():9.3f} {edge_width(out):9.2f} px {rmse:6.3f}")

Output:

filter          noise std edge 10-90%   RMSE
none                0.200      0.81 px  0.199
box 5x5             0.042      3.98 px  0.073
Gaussian s=1.5      0.040      3.96 px  0.066
median 5x5          0.051      2.38 px  0.059

The box and Gaussian filters reduce the noise about fivefold, close to the $1/\sqrt{25}$ expected for a 25-pixel average, but spread the edge over about 4 pixels. The median leaves somewhat more noise in flat regions, because for Gaussian noise the median is a less efficient estimate than the mean, yet keeps the edge much narrower and has the lowest overall error. (The unfiltered edge measures 0.81 pixels, not 0, because the width is interpolated between samples.) With impulse noise the median’s advantage is much larger; see the median filter article.

Learned Filters

A convolutional layer in a neural network is a bank of linear filters whose kernels are learned from data rather than designed, followed by a nonlinearity. The first-layer kernels of the ImageNet classifier of Krizhevsky, Sutskever, and Hinton (2012), for example, include frequency- and orientation-selective filters and colored blobs, close relatives of hand-designed edge and color filters. Deep denoising and restoration networks act as learned nonlinear filters with large effective neighborhoods. They can outperform designed filters on images like their training data, while classical filters remain cheap, predictable, and controlled by a few interpretable parameters.

Common Misconceptions

“Filtering means convolution.” Only linear shift-invariant filters are convolutions. The median, bilateral, and morphological filters are not, and they cannot be described by a kernel or a frequency response.
“Edge-preserving filters preserve all detail.” They preserve high-contrast edges. Low-contrast texture, comparable in amplitude to the noise, is smoothed away like noise.
“The median filter is just a better blur.” It is better for impulse noise and step edges, but for Gaussian noise a linear filter of the same size reduces noise more, and the median removes thin lines entirely.

Where It Is Used

Preprocessing and noise reduction: Gaussian and median filtering before further analysis, and edge-preserving filters for photographic denoising and smoothing.
Derivatives and features: gradient filters for edge and corner detection, and the structure tensor behind feature tracking and Lucas–Kanade flow.
Multi-scale representations: smoothing before subsampling in an image pyramid, and filter banks at several scales and orientations for texture analysis.
Matching: window aggregation of matching costs in a cost volume, where edge-preserving filters can keep aggregation from crossing depth boundaries.
Computational photography: sharpening, detail enhancement, and tone mapping, often with edge-preserving filters.
Neural networks: convolutional layers as learned filter banks.