Image Processing

Image processing, or digital image processing, is the computation of new images from images: brightening an underexposed photograph, removing noise, sharpening, resampling to another size, correcting a lens’s geometry, filling in a scratch. The same operations measure simple quantities from images and prepare them for analysis, so they sit underneath most of computer vision, including the first layers of a convolutional network. This article maps the field: the kinds of operation, sorted by how much of the input each output pixel depends on, the problems of enhancement and restoration built from them, the shift to learned methods, and the boundary with computer vision. Each family has its own article, linked where it first appears.

Definition

Gonzalez and Woods (2018, §1.1) define digital image processing as the processing of digital images by a digital computer, where a digital image is a function $f(x, y)$ whose coordinates and intensity values are finite and discrete. A common narrower definition requires that both the input and the output be images. Gonzalez and Woods call that boundary limiting and somewhat artificial: under it, computing an image’s average intensity would not count. Their book includes processes that output images and processes that extract attributes from them, up to the recognition of individual objects.

Most operations covered here map an input image $f$ to an output image $g$ by

$$g(x, y) = T[f](x, y),$$

and they are classified by what $T$ looks at to compute one output value:

• A point operation uses only the input value at the same pixel.
• A neighborhood operation uses a small window around the same position.
• A global operation uses the whole image, often through a transform such as the Fourier transform.
• A geometric operation moves values: the output at $(x, y)$ comes from the input at another position.

Gonzalez and Woods’ textbook, for example, has sections on single-pixel, neighborhood and geometric operations and on image transforms. The classification is useful because the extent of the input that affects an output pixel predicts what an operation can and cannot do, as the sections below show.

Image Processing and Computer Vision

There is no agreed line where image processing stops and image analysis and computer vision start. Gonzalez and Woods (2018, §1.1) describe a continuum of three kinds of process:

• Low-level processes, such as noise reduction, contrast enhancement and sharpening, take images and return images.
• Mid-level processes, such as segmentation, the description of regions and the classification of individual objects, take images and return attributes: edges, contours, object identities.
• High-level processes make sense of an ensemble of recognized objects and, at the far end, perform the cognitive functions associated with human vision.

On this view image processing is the low end of the continuum and overlaps image analysis in the recognition of individual regions. Computer vision aims at the high end: using images to infer what is in the scene and to act on it. Torralba, Isola and Freeman (2024) present the same signal-processing tools from the vision side, as means of building representations useful for downstream tasks rather than as ends in themselves.

In practice the two depend on each other. Vision systems run image processing before analysis (resizing, normalizing intensities, denoising), inside their features (image gradients, edge detection, feature detection) and in training (data augmentation applies geometric and point operations at random). Image processing, in turn, increasingly uses models learned for vision. The physics that produces the input, from scene light through the lens and sensor to quantized pixel values, belongs to image formation, and much of image processing is an attempt to undo or compensate for it.

Point Operations

A point operation applies one function to every pixel value,

$$g(x, y) = T\big(f(x, y)\big),$$

so it can be stored as a look-up table with one entry per intensity level. The common ones are:

• Contrast stretching maps a range $[r_\text{lo}, r_\text{hi}]$ of input values linearly onto the full output range, clipping outside it.
• Gamma (power-law) transformation $s = c\,r^\gamma$ maps a narrow range of dark input values onto a wider output range when $\gamma < 1$ and does the opposite when $\gamma > 1$ (Gonzalez & Woods, 2018, §3.2). Many capture, printing and display devices respond to intensity as a power law, and correcting for that response is called gamma correction; standard color encodings such as sRGB are gamma-encoded, as the color spaces article explains.
• Thresholding maps values above a threshold to one level and the rest to another; with a threshold chosen from the histogram, thresholding is the simplest form of segmentation.
• Histogram equalization maps each level through the image’s cumulative histogram so that output levels are used about equally. The histogram equalization mapping is computed from the whole image histogram, a global statistic, and then applied pixel by pixel; adaptive variants such as CLAHE compute it per region, which makes them neighborhood operations.

Arithmetic between registered images, such as averaging several noisy exposures of a static scene or subtracting a background, is also pixel-wise: each output pixel depends only on the values at the same position in each input.

A point operation cannot use spatial context, which limits what it can do. It changes how values are displayed or quantized, but it cannot tell a pixel’s signal from its noise. To first order, a smooth monotonic $T$ multiplies both a small local contrast and the noise around it by the same slope $T'(r)$, so their ratio is unchanged; this follows from a Taylor expansion of $T$, and the example below shows it for a linear stretch.

Neighborhood Operations

A neighborhood operation computes each output pixel from a window of input pixels around the same position, with the same rule everywhere. Image filtering is the general framework; its main families each have articles:

• Linear filters form a weighted sum of the window: a convolution with a kernel. The Gaussian filter smooths at a scale set by its standard deviation, the Sobel operator estimates image gradients, and the Laplacian operator responds to blobs and edges through second derivatives. Torralba, Isola and Freeman (2024, ch. 15) show why convolution is the natural linear operation: an operation that is linear and treats every position the same way (translation-invariant) must be a convolution.
• Nonlinear filters use the values themselves. The median filter takes the middle value of the window and removes impulse noise while keeping step edges; the bilateral filter weights neighbors by both distance and similarity of value, so it smooths within regions but not across strong edges.
• Morphological operators probe the image with a small set called a structuring element, as mathematical morphology describes: erosion and dilation take the minimum and maximum over it, and opening, closing and top-hats combine them to remove, keep or extract structures by shape and size.

The window size sets the scale at which a neighborhood operation acts. Smoothing over a larger window removes more noise and more detail; the trade-off between the two is the subject of the image filtering article. Some methods widen the neighborhood until it covers the whole image: non-local means averages pixels whose surrounding patches look alike, wherever they are.

Global and Transform-Domain Operations

A global operation lets every output pixel depend on the whole image. The most important ones work in a transform domain: transform the image, modify the coefficients, transform back.

• Fourier transform. The Fourier transform writes an image as a sum of sinusoidal gratings. By the convolution theorem, convolution in space is multiplication of spectra, $G(u, v) = H(u, v)\,F(u, v)$, so a filter can be designed by its frequency response: low-pass, high-pass, band-pass, or a notch that removes one periodic pattern. Gonzalez and Woods (2018, §4.1) credit the fast Fourier transform (FFT) with making such processing practical; they date its “discovery” to the early 1960s, and Cooley and Tukey’s paper appeared in 1965. Filtering an $M \times N$ image with an $m \times n$ kernel in the spatial domain costs on the order of $MNmn$ operations, while forward and inverse FFTs cost on the order of $2MN \log_2 MN$, so the frequency domain wins for large kernels.
• Wavelet transforms. Wavelet transforms decompose an image into band-pass components at octave-spaced scales and several orientations, localized in space as well as in frequency. Mallat (1989) showed that the difference in information between two resolutions can be computed by projecting onto a wavelet orthonormal basis, using a pyramid algorithm; unlike the overcomplete image pyramid, such a decomposition has as many coefficients as pixels. Natural images are sparse in wavelet bases, which is why thresholding wavelet coefficients removes noise (see image denoising) and why wavelets are used in compression.
• Histogram-based mappings. Mappings such as global histogram equalization depend on the whole image through its histogram, although they are applied as point operations.

Frequency-domain filtering is not a different kind of operation from convolution: for a linear translation-invariant filter the two give the same result (up to boundary handling, since the discrete Fourier transform treats the image as periodic), and the choice is one of cost and of which description of the filter is more natural. What the transform view adds is a way to see and design filters, such as a notch, that would need a large kernel in space. The sampling and aliasing article uses the same view to explain what happens when an image is sampled or resampled.

Geometric Operations

A geometric operation changes where values are, not what they are: resizing, rotating, correcting lens distortion, rectifying a plane by a homography, or warping one image onto another. Image warping computes each output pixel by mapping its position back into the input,

$$g(\mathbf{x}) = f\big(T^{-1}(\mathbf{x})\big),$$

and, because $T^{-1}(\mathbf{x})$ rarely falls on a pixel, interpolating the input there. Nearest-neighbor interpolation gives a piecewise-constant result; bilinear interpolation weights the four surrounding pixels and is itself a linear operation (Torralba, Isola & Freeman, 2024, ch. 21); longer kernels such as bicubic are smoother at a higher cost. Shrinking an image needs low-pass filtering first, or detail too fine for the new grid turns into aliasing; an image pyramid applies this repeatedly to build a multi-scale representation.

Geometric operations are linear in the pixel values, since a rotated sum of images is the sum of rotated images, but they are not translation-invariant in general, so they are not convolutions (Torralba, Isola & Freeman, 2024, ch. 15). Estimating the transform that aligns two images is image registration, a task that uses warping as its last step.

Enhancement and Restoration

The operations above are tools; enhancement and restoration are two of the problems they serve. Gonzalez and Woods (2018, §1.4) draw the distinction this way: enhancement makes an image more suitable for a particular application and is judged subjectively, while restoration is objective, because it relies on a mathematical or probabilistic model of how the image was degraded.

Enhancement uses contrast stretching, gamma correction and histogram equalization to show detail in dark or washed-out regions, unsharp masking to emphasize edges, and smoothing to hide noise. There is no general theory of it: a method suited to X-ray images may not suit infrared satellite images (Gonzalez & Woods, 2018, §1.4).

Restoration starts from a degradation model. If the degradation is a linear, position-invariant blur $h$ plus additive noise $\eta$, the observed image is

$$g(x, y) = (h * f)(x, y) + \eta(x, y), \qquad G(u, v) = H(u, v)\,F(u, v) + N(u, v)$$

(Gonzalez & Woods, 2018, §5.1), and restoration estimates $f$ from $g$ and what is known about $h$ and $\eta$. The main problems are:

• Denoising ($h$ is the identity). The physical sources and models of the noise are in the image noise article, and the image denoising article surveys the methods, from linear and nonlinear filters through total variation, wavelet shrinkage and patch-based methods such as non-local means and BM3D to learned networks.

• Deblurring (deconvolution). Dividing by $H$, the inverse filter, recovers $F$ exactly without noise, but the noise term $N/H$ grows without bound where $|H|$ is small, typically at high frequencies. The Wiener filter, the minimum mean-square-error linear estimate, instead weights each frequency by how much signal it carries relative to noise,

  $$\hat F(u, v) = \frac{H^*(u, v)}{|H(u, v)|^2 + S_\eta(u, v) / S_f(u, v)}\, G(u, v),$$

  where $S_f$ and $S_\eta$ are the power spectra of image and noise. With $H = 1$ it reduces to the Wiener denoiser that Torralba, Isola and Freeman (2024, §27.5.2) derive from a Gaussian image prior. Richardson (1972) and Lucy (1974) derived iterative restoration methods from Bayes’ theorem, treating images and point spread functions as probability distributions; Lucy showed that his iteration keeps the estimate non-negative and normalized and increases the likelihood of the observed data at each step. When the blur itself is unknown, as with camera shake, the problem is blind deconvolution, which must estimate $h$ and $f$ together.

• Inpainting fills missing or unwanted regions so the result looks natural. Small gaps can be filled by propagating structure from the surroundings; large ones need invented content, which makes image inpainting a form of conditional image generation.

• Super-resolution recovers a higher-resolution image from one or more low-resolution observations; super-resolution models the degradation as blur followed by subsampling.

Restoration problems are ill-posed: many images $f$ explain the same $g$, and a method must choose among them using a prior, an assumption about what images are likely. Torralba, Isola and Freeman (2024, ch. 27) trace how the priors have improved. A Gaussian prior, which captures only how pixel correlation falls with distance, gives the linear Wiener filter, and its denoising results are far from satisfactory. The outputs of band-pass filters on natural images have sharply peaked, heavy-tailed histograms, and a prior built on that fact turns denoising into coring: band-pass coefficients near zero are set to zero, and larger ones are shrunk. Non-parametric models that compare whole patches, as non-local means does, capture more of the structure of natural images. Learned networks are the latest step in this sequence.

Example: Three Kinds of Operation

The example builds a synthetic image with three defects: low contrast, Gaussian noise, and vertical stripes with a period of 8 pixels, a periodic interference pattern. It then applies one operation of each kind: a contrast stretch (point), a Gaussian filter with $\sigma = 1.5$ pixels (neighborhood), and a Fourier notch that zeroes the stripes’ frequency (global). For each result it measures the contrast between the rectangle and the background, the noise and the stripe amplitude in a flat region, the contrast relative to each, and the 10–90% width of the rectangle’s top edge.

import numpy as np
from scipy import ndimage

rng = np.random.default_rng(0)

# A dim, low-contrast rectangle on a gray background, plus vertical stripes
# (period 8 px, a periodic interference pattern) and Gaussian noise.
N = 128
clean = np.full((N, N), 0.30)
clean[40:88, 32:96] = 0.40
x = np.arange(N)
stripes = 0.05 * np.sin(2 * np.pi * x / 8)          # 16 cycles across the image
observed = clean + stripes[None, :] + rng.normal(0.0, 0.03, (N, N))

# 1. Point operation: a linear contrast stretch of the 1st-99th percentiles to [0, 1].
lo, hi = np.percentile(observed, [1, 99])
point = np.clip((observed - lo) / (hi - lo), 0.0, 1.0)

# 2. Neighborhood operation: Gaussian smoothing with sigma = 1.5 px.
neigh = ndimage.gaussian_filter(observed, sigma=1.5, mode="reflect")

# 3. Global operation: zero the stripe frequency in the 2D Fourier transform (a notch).
F = np.fft.fft2(observed)
F[0, 16] = F[0, -16] = 0.0                            # row 0: no vertical variation
glob = np.real(np.fft.ifft2(F))

def measure(img):
    bg = img[100:124, 8:120]                          # flat background, 14 stripe periods
    contrast = img[48:80, 40:88].mean() - bg.mean()
    cols = bg.mean(axis=0)                             # averaging rows removes most noise
    xb = x[8:120]
    stripe = 2 * abs(np.mean(cols * np.exp(-2j * np.pi * xb / 8)))  # period-8 amplitude
    noise = np.std(bg - cols[None, :])                 # what is left: the noise
    prof = img[30:50, 32:96].mean(axis=1)              # top edge, whole stripe periods
    p = (prof - prof[:5].mean()) / (prof[-5:].mean() - prof[:5].mean())
    t = np.arange(p.size)
    width = np.interp(0.9, p, t) - np.interp(0.1, p, t)
    return contrast, noise, stripe, width

print(f"{'operation':12s} {'contrast':>8s} {'noise':>6s} {'stripe':>6s}"
      f" {'C/noise':>7s} {'C/stripe':>8s} {'edge 10-90%':>11s}")
for name, img in [("observed", observed), ("point", point),
                  ("neighborhood", neigh), ("global", glob)]:
    c, n, s, w = measure(img)
    print(f"{name:12s} {c:8.3f} {n:6.3f} {s:6.3f} {c / n:7.1f} {c / s:8.1f} {w:8.2f} px")

Output (NumPy 2.3.5, SciPy 1.16.3):

operation    contrast  noise stripe C/noise C/stripe edge 10-90%
observed        0.100  0.030  0.050     3.4      2.0     0.85 px
point           0.368  0.109  0.187     3.4      2.0     0.84 px
neighborhood    0.100  0.005  0.025    18.7      4.0     4.30 px
global          0.100  0.030  0.001     3.4    140.0     0.85 px

Each operation fixes the defect its kind can reach and no other. The contrast stretch multiplies contrast, noise and stripes by the same factor of about 3.7, so both ratios are unchanged: the image looks stronger on a display, but the rectangle is no easier to separate from the noise. The Gaussian filter divides the variance of white noise by about $4\pi\sigma^2 \approx 28$, raising contrast-to-noise more than fivefold, but it spreads the edge over about 4 pixels and only halves the stripes. That is what its frequency response predicts: a Gaussian of $\sigma = 1.5$ passes a frequency of $1/8$ cycle per pixel with gain $e^{-2\pi^2\sigma^2/64} \approx 0.50$. The notch removes the stripes almost entirely, because they occupy two Fourier coefficients and the notch zeroes exactly those, and it leaves the noise and the edge as they were. No small kernel could remove a pattern 8 pixels wide without blurring structure of the same size; in the frequency domain the pattern is two isolated coefficients.

Learned Methods

From the late 2000s, restoration methods learned from examples began to match hand-designed ones and then to outperform them on standard benchmarks. Jain and Seung (2008) trained a convolutional network for natural-image denoising on samples synthesized from a noise model and reported performance comparable to, and in some cases better than, the best wavelet and Markov random field methods of the time. Dong, Loy, He and Tang (2016, after a 2014 conference version) trained SRCNN, a convolutional network of three layers in its basic form that maps low-resolution to high-resolution images end to end, and showed that the earlier sparse-coding methods for super-resolution can be viewed as such a network whose parts were optimized separately. DnCNN (Zhang et al., 2017) learned to predict the noise residual and handled Gaussian denoising at unknown noise levels, super-resolution and JPEG deblocking with a single model. Later restoration networks use transformers, and diffusion models generate images by repeated denoising.

Learning changes where the prior comes from, not the structure of the problem. A learned network still maps a degraded image to an estimate, its convolutional layers are banks of linear filters, and the degradation models of restoration often define its training data. Ulyanov, Vedaldi and Lempitsky (2018) showed that even an untrained, randomly initialized generator network acts as a strong handcrafted prior for denoising, super-resolution and inpainting, so part of what networks contribute comes from their architecture rather than from training data. Learning also widened the field from restoring images to translating them between domains: image-to-image translation maps label maps to photographs or day scenes to night ones.

Classical methods remain in use. They need no training data, they cost little, their few parameters have a clear meaning, and they rarely invent detail. Learned methods are usually better on images like their training data, but they can produce plausible structure that is not in the scene, which matters in medical and scientific imaging and in any use where the output is evidence. Evaluation reflects the difference: PSNR and SSIM measure fidelity to a reference, while image quality assessment and LPIPS aim at perceived quality.

Historical Development

Gonzalez and Woods (2018, §1.2) trace the field’s origins as follows. Pictures sent by the Bartlane submarine cable system between London and New York in the early 1920s were digital images, coded at five gray levels and at 15 from 1929, but they were not processed by computer. Computers able to do meaningful image processing appeared in the early 1960s, and the field grew from them and from the space program: in 1964 the Jet Propulsion Laboratory used a computer to correct distortions in pictures of the Moon transmitted by Ranger 7. Medical imaging, remote sensing of the Earth and astronomy followed in the late 1960s and early 1970s.

The methods developed as computation and models of images allowed:

• 1960s: Matheron and Serra founded mathematical morphology in 1964, in work on the petrography of iron ore, as the mathematical morphology article recounts. The FFT (Cooley & Tukey, 1965) made frequency-domain filtering practical on the computers of the time.
• 1970s: Model-based restoration of blurred images, such as the Bayesian iterations of Richardson (1972) and Lucy (1974), the latter developed for astronomical data.
• 1980s: Multi-scale representations such as the Laplacian pyramid (Burt & Adelson, 1983) and wavelet theory (Mallat, 1989, from a 1987 technical report) gave a way to process images scale by scale, matching the sparse, heavy-tailed statistics of natural images.
• 1990s–2000s: Edge-preserving smoothing (anisotropic diffusion, Perona & Malik, 1990; the bilateral filter, Tomasi & Manduchi, 1998), variational methods such as total variation (Rudin, Osher & Fatemi, 1992), and patch-based methods that exploit self-similarity (non-local means, Buades, Coll & Morel, 2005; BM3D, Dabov et al., 2007); the patch-based methods became the standard baselines for denoising.
• 2008 onward: Learned networks (Jain & Seung, 2008; SRCNN; DnCNN) moved the prior from the designer to the training data.

Common Misconceptions

• “Image processing ends where computer vision begins.” The boundary is a continuum, not a line (Gonzalez & Woods, 2018, §1.1). Vision models contain image processing, and image processing now uses learned vision models.
• “Contrast enhancement adds information.” A point operation cannot separate signal from noise; it changes how values are displayed or quantized, and clipping or coarse quantization can lose information. Histogram equalization and strong gamma correction also amplify noise in the ranges they stretch.
• “Frequency-domain filtering is a different operation from convolution.” For linear translation-invariant filters it is the same operation computed another way; it differs only at the boundaries and in cost.
• “Geometric transformations are free of side effects.” Resampling at positions between pixels interpolates, which blurs (bilinear and higher-order kernels) or makes detail blocky and jagged (nearest neighbor), and a reduction in size aliases unless the image is filtered first. Repeated rotations or resizes accumulate these errors.
• “Learned methods made classical image processing obsolete.” Learned networks are built from convolutions, are often trained on data produced by classical degradation models, and are compared with classical baselines. Classical operations remain the tools of choice where predictability, speed, or the absence of invented detail matters.

Where It Is Used

• Cameras and phones: demosaicking, denoising, white balance, tone mapping and sharpening turn raw sensor data into a photograph.
• Medical, scientific and remote-sensing imaging: contrast enhancement for inspection, deconvolution in microscopy and astronomy, registration of scans, and reconstruction in computed tomography.
• Preprocessing for vision: resizing and normalization for neural networks, smoothing before edge detection, and pyramids for multi-scale detection and matching.
• Feature extraction and segmentation: image gradients and histograms of oriented gradients, corner and blob detectors, thresholding and watershed segmentation.
• Training data: data augmentation by random crops, flips, rotations and color changes.
• Compression and display: transform coding, resampling for screens of different resolutions, and gamma encoding for display.