Image Denoising

Image denoising estimates the clean image underlying a noisy observation. Every camera image contains noise, and it becomes visible whenever little light reaches the sensor: at night, indoors, in fast exposures, and with the small sensors of phones. Denoising is one of the oldest problems in image processing, a standard test bed for image models, and a step in nearly every camera pipeline.

The difficulty is that noise and detail look alike. Both are small, rapid variations in intensity, and any method that removes one risks removing the other. Each family of denoising methods is, in effect, a different assumption about what clean images look like, used to tell the two apart.

Problem Definition

The observation is modeled as a clean image $y$ corrupted by noise:

$$z = y + n .$$

The goal is an estimate $\hat{y}$ that is as close as possible to $y$, usually in mean squared error, without access to $y$ itself.

In the standard research setting, $n$ is additive white Gaussian noise (AWGN): independent at every pixel, zero mean, with a known standard deviation $\sigma$. Results are reported at fixed levels, often $\sigma = 15$, $25$, and $50$ on a 0–255 scale. The setting is convenient because clean test images can be corrupted synthetically, so the ground truth is known exactly. Real camera noise differs: raw sensor noise is signal-dependent (Poisson–Gaussian), and demosaicing, tone curves, and compression make it spatially correlated and harder to model, as Image Noise explains. A denoiser tuned to AWGN can therefore disappoint on real photographs.

Input and Output

• Input: a noisy image, grayscale or color, in raw sensor data or a processed format. Many methods also take the noise level $\sigma$, or a noise model, as an input; blind methods estimate it or do without.
• Output: a denoised image of the same size. Some methods also return a per-pixel estimate of the remaining uncertainty.

Variants

• Non-blind and blind denoising, depending on whether the noise level is given.
• Raw and sRGB denoising. Raw data has a simple noise model but needs denoising before demosaicing; processed images have more complex noise.
• Burst and video denoising, which average aligned frames as well as pixels within a frame.
• Other noise types: impulse noise, best handled by rank filters, and Poisson noise in photon-limited imaging such as astronomy and fluorescence microscopy.

Challenges

• Preserving structure. Edges, fine texture, and thin lines must survive while flat regions are smoothed. Low-contrast texture, whose amplitude is close to the noise level, is the hardest case.
• Avoiding artifacts. Each method family has typical artifacts; Buades, Coll and Morel (2005) note that practitioners name them ringing, blur, staircasing, checkerboard effects, and wavelet outliers.
• Unknown and varying noise. Real noise level varies with intensity, ISO setting, and position in the image, and it must be estimated.
• Real camera noise. Spatially correlated noise from the camera pipeline is easily mistaken for texture.
• Cost. Camera pipelines need methods that run in milliseconds on large images.

Approaches

The history of denoising is a sequence of increasingly powerful image models, from local smoothness to self-similarity to models learned from data.

Linear smoothing. A Gaussian filter or other low-pass filter averages each pixel with its neighbors. Averaging $N$ pixels divides the noise standard deviation by $\sqrt{N}$, but only where the image is constant over the window; elsewhere it blurs edges and texture. Linear filters cannot separate noise from detail at the same frequencies, a limitation explained in Image Filtering and Fourier Transform.

Rank and edge-preserving filters. The median filter removes impulse noise while keeping step edges. The bilateral filter and anisotropic diffusion weight neighbors by their similarity in value as well as their distance, so they smooth within regions but not across strong edges. They compare single pixel values, which are noisy themselves, so they work only when edges are much stronger than the noise.

Variational methods. Rudin, Osher and Fatemi (1992) posed denoising as minimizing the total variation of the image, the integral of its gradient magnitude, subject to the residual having zero mean and the variance of the noise; in its common unconstrained form,

$$\hat{y} = \arg\min_u \int |\nabla u| \, d\mathbf{x} + \lambda \int (u - z)^2 \, d\mathbf{x} .$$

Total variation penalizes oscillation but allows sharp jumps, so edges are kept. Its known weaknesses are loss of fine texture and staircasing, where smooth gradients become piecewise constant.

Wavelet shrinkage. Natural images are sparse in a wavelet basis: a few large coefficients carry most of the signal, while white noise spreads evenly over all coefficients. Donoho and Johnstone (1994) proposed thresholding the empirical wavelet coefficients, with a universal threshold $\sigma\sqrt{2 \log n}$ for $n$ samples, and showed that the result comes within a logarithmic factor of an ideal procedure told by an oracle which coefficients to keep. Thresholding in an orthogonal basis produces Gibbs-like oscillations near edges, which translation-invariant variants reduce.

Non-local and patch-based methods. Non-local means (Buades, Coll and Morel, 2005) uses self-similarity: a small patch in a natural image usually has many similar patches elsewhere, and averaging their centers removes noise without assuming local smoothness. BM3D (Dabov et al., 2007) groups similar patches into a 3D stack, filters the stack jointly in a 3D transform domain, and returns the filtered patches to their positions, in two stages: hard thresholding for a basic estimate, then Wiener filtering guided by that estimate. Weighted nuclear norm minimization (WNNM, Gu and colleagues, 2014) stacks similar patches into a matrix and estimates a low-rank approximation of it, shrinking its singular values by different amounts. For years these methods were the standard baselines for new denoisers.

Learned denoisers. DnCNN (Zhang et al., 2017) trains a deep convolutional network to predict the noise residual rather than the clean image, using batch normalization; the authors reported higher average PSNR than BM3D and WNNM on the BSD68 test set at $\sigma = 15$, $25$, and $50$, and showed that a single model trained over $\sigma \in [0, 55]$ handles blind Gaussian denoising. Later restoration networks such as SwinIR and Restormer use transformer attention to model longer-range dependencies. Supervised training needs clean targets, which are hard to obtain for real noise. Noise2Noise (Lehtinen et al., 2018) showed that a network trained with an $L_2$ loss on pairs of independent noisy observations of the same scene learns, in expectation, the same mapping as one trained on clean targets, since the expected noisy target is the clean image.

Real-noise datasets. Benchmarks built from real photographs shifted attention from synthetic AWGN. Plötz and Roth (2017) created the Darmstadt Noise Dataset (DND) from pairs of high-ISO and low-ISO photographs of 50 scenes, with careful alignment and intensity correction of the low-ISO reference, and found that several methods that did well on synthetic noise were outperformed by BM3D on real photographs. The Smartphone Image Denoising Dataset (SIDD; Abdelhamed, Lin and Brown, 2018) estimates ground truth from sequences of 150 captures of each scene and contains about 30,000 noisy images of 10 scenes from five smartphone cameras; its authors reported that networks trained on it outperform those trained with proxies such as low-ISO images.

Datasets and Benchmarks

• Synthetic AWGN. Clean images corrupted with Gaussian noise of known $\sigma$: BSD68 (68 natural images from the Berkeley Segmentation Dataset) and Set12 (12 widely used test images) for grayscale, color BSD68 and the Kodak photo set for color.
• Real noise. DND and SIDD, described above, provide real noisy images with estimated ground truth.

Evaluation Metrics

• PSNR (peak signal-to-noise ratio), $10 \log_{10}(\text{peak}^2 / \text{MSE})$ in decibels, is the standard measure. It is easy to compute and compare, but rewards smooth averages and correlates only loosely with perceived quality.
• SSIM (structural similarity) compares local means, variances, and correlations of the two images and tracks visible structural damage better.
• Visual inspection and method noise. Buades, Coll and Morel argued that the difference between the input and the denoised image, the method noise, should look like pure noise; visible edges or texture in it reveal what the method removed.

Applications

• Camera pipelines: raw and burst denoising in phones and cameras, especially in low light.
• Scientific and medical imaging: fluorescence and electron microscopy, astronomy, MRI, and low-dose CT, where fewer photons or a lower dose mean more noise.
• Preprocessing: noise reduction before edge detection, feature matching, and segmentation.
• Rendering: removing Monte Carlo noise from path-traced images rendered with few samples per pixel.
• Restoration priors: denoisers can serve as building blocks for deblurring, super-resolution, and other inverse problems, and diffusion-based image generators produce images by repeated denoising.

Open Problems

• Real noise: generalizing across cameras, ISO settings, and processing pipelines without paired training data.
• Detail versus noise: recovering low-contrast texture, and avoiding hallucinated detail from learned models, which can look plausible but is not in the scene.
• Evaluation: metrics that agree with human judgment, and ground truth for real noise.
• Efficiency: high-quality denoising of high-resolution images and video on mobile hardware.