Image Formation

Image formation is the chain of physical and computational steps that turns light in a scene into the numbers stored in an image file. Light from sources is reflected by surfaces, gathered and blurred by a lens, converted to electric charge by a sensor behind a mosaic of colour filters, sampled on a grid, digitized, and finally reshaped by the camera’s own processing into a compressed colour image. Each step leaves its mark on the pixel values a vision algorithm receives, so knowing the chain explains why the same object can produce different values, where noise and false colour come from, and when pixel values may be treated as measurements of light.

The geometry of the chain, where a scene point lands in the image, is the subject of the pinhole camera model and its corrections. This article follows the photometry: how much light arrives at each pixel, and what happens to it on the way to a number.

Definition

Image formation is the mapping from the light field of a scene to a digital image. It has a geometric part, which decides which scene point each pixel sees, and a photometric part, which decides the value recorded there. Following the order of Szeliski (2022, Sections 2.2–2.3), the photometric part can be divided into five stages:

Scene radiance: light sources illuminate surfaces, which reflect part of the light towards the camera according to their reflectance and orientation.
Optics: the lens collects light through its aperture and forms an image that is sharp only at one focus distance and is further blurred by diffraction and aberrations.
Sensor: each photosite converts the photons it receives during the exposure into charge, through a colour filter that passes one band of wavelengths; the process adds noise.
Sampling and quantization: the photosites sample the image on a grid, and an analog-to-digital converter rounds each reading to an integer.
Processing: the camera’s image signal processor (ISP) fills in the missing colours, balances them, applies a tone curve and compresses the result.

Intuition

A photograph of a white sheet of paper does not record “white”. It records how much light the paper reflected towards the lens, which depends on the lamp and the angle of the paper; how much of that light the lens let through, which depends on the aperture; how many photons happened to arrive at each photosite during the exposure; which colour filter sat over that photosite; and what curve the camera chose to map the counts to display values. Under a warm lamp the raw sensor values of the paper are reddish; the camera’s white balance makes them neutral again. A sheet in shadow gives smaller values, yet a person still sees white paper.

Computer vision often wants the opposite direction: to infer the scene (shape, material, illumination) from the values. That inverse problem is only well posed for the parts of the chain that are known or calibrated, which is why camera models, calibration and raw data matter.

Formal Definition

The chain can be written as a composition of models, each standard in its field.

Reflection. With a finite set of light sources $i$ of spectral radiance $L_i(\lambda)$ arriving from directions $\hat{\mathbf{v}}_i$ , the radiance leaving a surface point with normal $\hat{\mathbf{n}}$ towards the camera direction $\hat{\mathbf{v}}_r$ is (Szeliski, 2022, Eq. 2.86)

L_r(\hat{\mathbf{v}}_r; \lambda) = \sum_i L_i(\lambda)\, f_r(\hat{\mathbf{v}}_i, \hat{\mathbf{v}}_r, \hat{\mathbf{n}}; \lambda)\, \max(0, \cos\theta_i),

where $f_r$ is the bidirectional reflectance distribution function (BRDF) and $\theta_i$ is the angle between the incoming light and the normal.

Optics. An ideal thin lens with focal length $f$ and aperture diameter $d$ , focused near infinity, turns radiance $L$ arriving from off-axis angle $\alpha$ into irradiance $E$ on the sensor (Szeliski, 2022, Eq. 2.102):

E = L\, \frac{\pi}{4} \left( \frac{d}{f} \right)^2 \cos^4 \alpha .

Blur from defocus, diffraction and aberrations is modelled by convolving the irradiance with a point spread function (PSF) $g$ , the image of a single point of light: $E_{\text{blur}} = g * E$ . Treating $g$ as the same everywhere in the image is a simplification; real PSFs change across the field.

Sensor. A photosite with area $A_{ij}$ and colour filter $c$ accumulates, during exposure time $t$ , a mean number of photoelectrons

\mu_e[i, j] = t \int_{A_{ij}} \int \eta(\lambda)\, S_c(\lambda)\, \frac{\lambda}{h c_0}\, E_{\text{blur}}(\mathbf{x}; \lambda)\, d\lambda\, d\mathbf{x},

where $\lambda / (h c_0)$ converts energy to photons ( $h$ is Planck’s constant, $c_0$ the speed of light), $S_c$ is the filter’s spectral transmittance and $\eta$ the quantum efficiency, the fraction of photons converted to charge. This combines the per-wavelength photon count of EMVA 1288 (Eq. 3) with the spectral integration of each colour channel in Szeliski (2022, Eq. 2.110). The collected charge is Poisson distributed, the readout adds roughly Gaussian dark noise, and the camera converts electrons to digital numbers with an overall gain $K$ ; EMVA 1288 (Eqs. 5, 13 and 14) models the mean and variance of the digital value $y$ as

\mu_y = K(\mu_e + \mu_d), \qquad \sigma_y^2 = K^2 (\sigma_d^2 + \mu_e) + \tfrac{1}{12},

where $\mu_d$ is the mean dark signal, $\sigma_d^2$ the dark-noise variance, and $1/12$ the variance of rounding to integer digital numbers.

Processing. The stored image is $I = \Phi(y)$ , where $\Phi$ is the ISP: offset removal, demosaicing, white balance, colour correction, tone curve and compression. $\Phi$ is camera-specific, partly nonlinear and usually undocumented.

Scene: Illumination and Reflectance

Light sources are often approximated as point sources, whose intensity falls off with the square of distance, or as area sources; the light arriving from every direction at once, as in an outdoor courtyard, can be stored as an environment map (Szeliski, 2022, Section 2.2.1).

What a surface does with incoming light is described by its BRDF, a four-dimensional function of the incoming and outgoing directions, and of wavelength. The BRDF is reciprocal: swapping the two directions gives the same value (Helmholtz reciprocity). Most practical models split it into two parts (Szeliski, 2022, Section 2.2.2):

Diffuse (Lambertian, matte) reflection scatters light equally in all directions. The reflected radiance depends only on the angle between the light and the normal, through $\cos\theta_i$ , and not on the viewing direction. This is shading: the smooth darkening of a matte surface as it turns away from the light. Diffuse reflection usually carries the body colour of the material.
Specular reflection concentrates light around the mirror direction, producing highlights that move when the viewer moves. In models such as Phong’s, its sharpness is set by an exponent; for dielectrics such as plastics its colour is usually that of the light, not the material.

Phong’s model adds an ambient term for light that arrives indirectly, from inter-reflections or the sky. Real scenes also contain shadows and multiple bounces, which ray tracing and radiosity simulate (Szeliski, 2022, Section 2.2.2). Shafer’s dichromatic model writes the colour of a uniform material lit by a single source as the sum of an interface (specular) and a body (diffuse) term, each a fixed spectrum scaled by a geometric factor.

The consequence for vision is that a pixel value is a product of things an algorithm usually wants to separate: the surface’s reflectance (what the object is made of), the shading (its shape and orientation relative to the light), and the illumination’s strength and colour. Separating them from one image is called intrinsic image decomposition. Matching methods that assume a scene point keeps its value across images, the brightness constancy assumption, break down exactly where this product changes: at specular highlights that move with the viewer, and where the lighting changes.

Optics

Focus, aperture and depth of field

A lens gathers far more light than a pinhole but focuses only one distance sharply. The thin-lens law and the basic aperture trade-off are given in the pinhole camera model article. A point at another distance is imaged as a blur disc, the circle of confusion. From the thin-lens law (this is a derivation, not a quoted formula), a lens of focal length $f$ and f-number $N = f/d$ , focused at distance $z_f$ , images a point at distance $z$ as a disc of diameter

c = \frac{f^2\, |z - z_f|}{N\, z\, (z_f - f)} .

The blur grows with focal length and with the distance from the focus plane, and is inversely proportional to the f-number. The range of distances whose blur stays below an acceptable size is the depth of field; it depends on both the focus distance and the aperture (Szeliski, 2022, Section 2.2.3).

The f-number also sets exposure. By the irradiance formula above, the light reaching the sensor scales with $(d/f)^2 = 1/N^2$ . Full f-stops (f/1.4, f/2, f/2.8, …) are spaced by factors of $\sqrt{2}$ so that each stop halves the aperture area, the same change in exposure as halving the exposure time (Szeliski, 2022, Section 2.2.3). Aperture, exposure time and gain are therefore three ways of trading brightness against depth of field, motion blur and noise.

Vignetting

The irradiance formula contains $\cos^4\alpha$ : even an ideal lens delivers less light to the edges of the image than to the centre, because the scene patch, the aperture and the pixel are all foreshortened at oblique angles and the distance grows. This natural vignetting combines with mechanical vignetting, in which elements of a compound lens block rays near the periphery; the latter shrinks when the aperture is reduced (Szeliski, 2022, Section 2.2.3). Both can be calibrated from images of a uniformly lit target.

Diffraction

Light is a wave, and a beam limited by an aperture of diameter $D$ spreads. A point source imaged through a circular aperture becomes a bright disc surrounded by faint rings, the Airy pattern, with its first dark ring at angle $\theta = 1.22\,\lambda / D$ (Ling, Sanny and Moebs, 2016, Section 4.5). By the Rayleigh criterion, two points separated by this angle are just resolvable, and closer points are not. This limit holds for any lens or mirror of that diameter, not just a hole in a screen.

On the sensor, at a distance close to $f$ behind the lens, this angle corresponds to a radius of about $1.22\,\lambda f / D = 1.22\,\lambda N$ . For green light ( $\lambda = 0.55$ µm) the arithmetic gives 1.9 µm at f/2.8, 5.4 µm at f/8 and 10.7 µm at f/16: at the larger f-numbers the blur spans several pixels of a sensor with a 3 µm pitch, the pitch used in the example below. Closing the aperture therefore increases depth of field but eventually blurs the whole image by diffraction, the opposite of the pinhole intuition that a smaller hole is always sharper.

Aberrations

A real lens is not an ideal thin lens. The classic third-order (Seidel) aberrations are spherical aberration, coma, astigmatism, curvature of field and distortion (Szeliski, 2022, Section 2.2.3). Because the refractive index of glass varies with wavelength, simple lenses also show chromatic aberration: colours focus at slightly different distances and magnifications, giving coloured fringes at edges. Compound lenses combine glasses to reduce these errors. Of the aberrations, distortion is special: it moves image points without blurring them, and it is modelled geometrically, as described in lens distortion.

Point spread and transfer functions

All of the blur sources combine into the PSF. The PSF of a pixel is the lens blur convolved with the light-sensitive area of the photosite; its Fourier transform is the optical transfer function, whose magnitude, the modulation transfer function (MTF), says how much contrast survives at each spatial frequency (Szeliski, 2022, Section 2.3.1). Any contrast above the sensor’s Nyquist frequency that survives the lens is aliased.

The Sensor

An image sensor is an array of photosites, each converting light to charge during the exposure. In a CCD the charges are shifted from site to site to a shared amplifier; in a CMOS sensor each pixel converts charge to voltage itself and is read out through multiplexing. CMOS is now used in most digital cameras (Szeliski, 2022, Section 2.3). Video capture often uses a rolling shutter, which exposes and reads the rows one after another (Szeliski, 2022, Section 2.3), so fast motion skews the image.

Sensitivity. Not every photon produces an electron. EMVA 1288 defines the total quantum efficiency $\eta(\lambda)$ over the whole pixel area, so it includes the fill factor (the light-sensitive fraction of the pixel) and the effect of the microlenses placed over the pixels. Back-illuminated sensors and microlenses have largely removed the fill-factor limit of early designs (Szeliski, 2022, Section 2.3). In the linear model the mean digital value is the dark offset plus $K\eta$ times the number of photons (EMVA 1288, Eq. 6), which is why raw values are proportional to light once the offset is removed.

Range. A pixel holds a limited charge. EMVA 1288 distinguishes the saturation capacity, where the digital output clips, from the physical full-well capacity, and defines the dynamic range as the ratio of the saturation exposure to the smallest exposure that gives a signal-to-noise ratio of 1. Smaller pixels collect fewer photons and are noisier, which is one reason a smaller sampling pitch is not free (Szeliski, 2022, Section 2.3).

Noise. Photon shot noise (Poisson, variance equal to the mean), dark current, read noise, quantization, and the pixel-to-pixel nonuniformities of offset and gain all enter here. Their physics, the Poisson–Gaussian model and how they are measured are covered in image noise. Raising the gain (the ISO setting) brightens the image but amplifies the noise with it (Szeliski, 2022, Section 2.3).

Colour filter arrays

A photosite counts photons regardless of wavelength. Early colour television cameras used three tubes and later three sensor chips, but most cameras now place a colour filter array (CFA) over a single sensor, so that each photosite sees one band (Szeliski, 2022, Section 2.3.2). The most common layout is the Bayer pattern, from a patent that Bryce Bayer of Eastman Kodak filed in 1975 and that was granted in 1976. Its green filters occupy every other position in a checkerboard, and red and blue fill the remaining positions in alternate rows. The patent treats the green elements as luminance samples and gives them the densest sampling because human vision is most responsive to green detail; an alternative embodiment also samples red more often than blue, setting each colour’s sampling rate by the acuity of human vision for it (Bayer, 1976).

The filters’ spectral sensitivities in general do not match the CIE colour-matching functions or the spectra a standard colour space expects (Brooks et al., 2019, Section 3.5). A camera’s raw RGB is therefore a device-dependent colour space that processing must convert to a standard one such as sRGB (Szeliski, 2022, Section 2.3.2); color spaces defines those targets. Two surfaces with different spectra can give identical raw values under one light and different values under another.

Sampling and Quantization

Each photosite integrates the irradiance over its area and the exposure time, so the sensor delivers samples of the blurred image on a regular grid, one colour per site. Integration over the pixel area acts as a box filter that attenuates, but does not remove, detail finer than the grid. Detail above the Nyquist limit that the lens passes is aliased into false patterns, and many cameras place an optical anti-aliasing filter in front of the sensor to limit it (Szeliski, 2022, Section 2.3.1). The rules for when a grid is fine enough are covered in sampling and aliasing. The CFA makes this worse for colour. Red and blue are sampled only in every second row and column, so it follows that fine patterns the green channel still resolves can alias in red and blue and appear as false colour; the example below shows this, and Szeliski (2022, Section 10.3.1) notes that demosaicing errors often show up as spurious colours.

The analog-to-digital converter (ADC) then rounds each value to one of $2^b$ levels. Rounding with step $q$ adds an error of variance $q^2/12$ (EMVA 1288, Section 2.4). Szeliski (2022, Section 2.3) notes that the number of bits a camera quotes, such as 8 for JPEG or a nominal 16 for some raw formats, usually exceeds the number of bits that are useful given the noise. When the quantization step is well below the noise, as in most raw formats, rounding is invisible; when it is not, smooth gradients show banding.

In-Camera Processing

Raw sensor values do not look like a photograph. The camera’s ISP turns them into one through a sequence of steps that Szeliski (2022, Section 2.3, Fig. 2.23) and Brooks et al. (2019, Section 3) describe in similar order:

Offset (black level). The dark signal must sit above zero so that noise and nonuniformity in dark pixels are not clipped (EMVA 1288, Section 2.7; for its measurements the standard sets the offset so that fewer than about 0.5% of pixels read zero, Section 6.5). By the linear model, values are proportional to light only after the offset is subtracted.
Demosaicing. The two missing colours at each photosite are interpolated from neighbours. Bilinear interpolation of each channel is the simplest method and is the convention of the Darmstadt Noise Dataset (Brooks et al., 2019); it produces colour fringes and “zipper” patterns at edges. Better methods use the densely sampled green channel’s edges to guide red and blue (Szeliski, 2022, Section 10.3.1).
White balance. Per-channel gains, usually on red and blue, make neutral surfaces neutral under coloured light, undoing part of the illumination’s effect (Brooks et al., 2019, Section 3.4).
Colour correction. A $3 \times 3$ matrix maps the camera’s raw RGB to a standard space such as sRGB (Brooks et al., 2019, Section 3.5).
Denoising and sharpening, which change the noise’s spatial correlation.
Tone curve and gamma. Gamma compression spends more of the output levels on dark tones, to which vision is more sensitive, and is often followed by an S-shaped tone curve modelled on the characteristic curve of film (Brooks et al., 2019, Sections 3.6–3.7). Only after this step does an 8-bit output look right on a display.
Compression. JPEG converts to YCbCr, subsamples the chroma, transforms $8 \times 8$ blocks with the discrete cosine transform and quantizes the coefficients; the quality setting chiefly controls that quantization step, and heavier quantization gives block and “mosquito” artifacts (Szeliski, 2022, Section 2.3.3).

The combined mapping from the irradiance reaching the sensor to the stored value is the camera response function. For raw files it is close to linear; for JPEGs it is not, and it is usually calibrated empirically rather than modelled from first principles (Szeliski, 2022, Section 10.1.1). A digital camera’s raw file stores the values before most of these steps, which is why physically based methods prefer raw data.

Smartphones extend the chain with multi-frame processing (burst denoising, high dynamic range imaging, synthetic depth of field), the field of computational photography (Szeliski, 2022, Sections 10.2–10.3); these steps change pixel values in ways that depend on the scene and the device.

Examples

A synthetic scene through the chain

The example renders a scene whose true values are known exactly: grey bars at two spatial frequencies (period 8 and 3 pixels), a flat grey background and a matte red sphere shaded by a directional light, all under a warm light. It renders at four sub-samples per pixel side so that pixel integration can be simulated, then applies defocus blur sized by the thin-lens formula above, a Bayer mosaic with bilinear demosaicing, shot and read noise, a 10-bit ADC, and finally white balance and the sRGB curve. After each cumulative stage it reports the root-mean-square error against the ideal image (sharp, full colour, noiseless, in units where the brightest radiance is 0.9), the Michelson contrast of the coarse and fine bars, and the false colour on the bars (the mean spread between the three channels relative to their mean, after dividing out the light’s colour, which is zero for grey).

import numpy as np
from scipy import ndimage

rng = np.random.default_rng(0)
S = 4                                   # sub-samples per pixel side
H, W = 96, 128                          # sensor size in pixels
y, x = np.mgrid[0:H * S, 0:W * S] / S   # sub-sample coordinates, in pixels

# 1. Scene: reflectance times illumination (linear RGB, relative units)
illum = np.array([1.0, 0.8, 0.55])     # warm light, as camera RGB
refl = np.full((H * S, W * S, 3), 0.5)  # grey background
bars = lambda period: np.where((x % period) < period / 2, 0.9, 0.1)
left = x < 64
refl[left & (y < 48)] = bars(8)[left & (y < 48), None]    # coarse bars
refl[left & (y >= 48)] = bars(3)[left & (y >= 48), None]  # fine bars
cy, cx, r = 60, 96, 24                  # a matte (Lambertian) sphere
d2 = (x - cx) ** 2 + (y - cy) ** 2
on = d2 < r ** 2
n = np.stack([(x - cx) / r, (y - cy) / r,
              np.sqrt(np.clip(1 - d2 / r ** 2, 0, 1))], -1)
light = np.array([-0.5, -0.5, 0.7]) / np.linalg.norm([-0.5, -0.5, 0.7])
shade = np.clip(n @ light, 0, None)     # cos of incidence angle, >= 0
albedo = np.array([0.7, 0.3, 0.2])     # plus a small ambient term below
refl[on] = albedo * (0.1 + 0.9 * shade[on, None])
radiance = refl * illum

def pixels(img):                        # each photosite integrates its area
    return img.reshape(H, S, W, S, -1).mean(axis=(1, 3))

# 2. Optics: defocus blur from the thin-lens law
f, N, pitch = 8e-3, 2.8, 3e-6           # focal length, f-number, pixel pitch
z_focus, z_obj = 2.0, 1.5               # focused distance, object distance (m)
c = f**2 * abs(z_obj - z_focus) / (N * z_obj * (z_focus - f))
print(f"blur circle {c * 1e6:.2f} um = {c / pitch:.2f} pixels")
rad = c / pitch / 2 * S                 # disc radius in sub-samples
k = np.arange(-int(rad) - 1, int(rad) + 2)
disc = (k[:, None] ** 2 + k[None, :] ** 2 <= rad ** 2).astype(float)
disc /= disc.sum()
blurred = np.stack([ndimage.convolve(radiance[..., i], disc, mode="nearest")
                    for i in range(3)], -1)

# 3. Sensor: Bayer RGGB colour filter array, electrons, noise, ADC
full_well, read_noise, bits = 1000.0, 3.0, 10
exposure = 0.8 * full_well / radiance.max()       # electrons per unit radiance
cfa = np.zeros((H, W), int)                       # channel seen by each site
cfa[0::2, 1::2] = 1; cfa[1::2, 0::2] = 1; cfa[1::2, 1::2] = 2
def mosaic(rgb):
    return np.take_along_axis(rgb, cfa[..., None], axis=2)[..., 0]

# 4. Bilinear demosaicing: interpolate each channel from its own samples
kr = np.array([[1, 2, 1], [2, 4, 2], [1, 2, 1]]) / 4.0
kg = np.array([[0, 1, 0], [1, 4, 1], [0, 1, 0]]) / 4.0
def demosaic(raw):
    out = np.zeros((H, W, 3))
    for ch, kern in [(0, kr), (1, kg), (2, kr)]:
        out[..., ch] = ndimage.convolve(raw * (cfa == ch), kern, mode="mirror")
    return out

def measure(rgb, ideal):
    e = rgb / exposure                  # back to radiance units
    rmse = np.sqrt(np.mean((e - ideal) ** 2))
    def contrast(rows):                 # Michelson contrast of the column profile
        p = (e[rows, 4:60] / illum).mean(axis=(0, 2))
        return (p.max() - p.min()) / (p.max() + p.min())
    def false_colour(rows):             # grey bars should have R = G = B
        b = e[rows, 4:60] / illum
        return (b.max(axis=2) - b.min(axis=2)).mean() / b.mean()
    return (rmse, contrast(slice(8, 40)), contrast(slice(56, 88)),
            false_colour(slice(8, 40)), false_colour(slice(56, 88)))

ideal = pixels(radiance)                # sharp, full colour, noiseless
sharp_e = ideal * exposure
blur_e = pixels(blurred) * exposure
raw_e = mosaic(blur_e)
noisy_e = rng.poisson(raw_e) + rng.normal(0, read_noise, raw_e.shape)
gain = (2 ** bits - 1) / full_well      # digital numbers per electron
dn = np.clip(np.round(noisy_e * gain), 0, 2 ** bits - 1)
adc_e = dn / gain

print(f"{'':<26}{'':>7}{'contrast':>14}{'false colour':>14}")
print(f"{'stage':<26}{'RMSE':>7}{'coarse':>7}{'fine':>7}{'coarse':>7}{'fine':>7}")
for name, img in [("ideal sampling", sharp_e),
                  ("+ defocus blur", blur_e),
                  ("+ Bayer mosaic, demosaic", demosaic(raw_e)),
                  ("+ shot and read noise", demosaic(noisy_e)),
                  ("+ 10-bit ADC", demosaic(adc_e))]:
    print(f"{name:<26}" + "".join(f"{v:>7.3f}" for v in measure(img, ideal)))

# Noise: every raw site against the model var = signal + read^2 + q^2/12
model = np.sqrt(raw_e + read_noise ** 2 + 1 / (12 * gain ** 2))
print(f"raw noise / model std: {((adc_e - raw_e) / model).std():.3f}")
patch = (slice(4, 28), slice(72, 124))  # flat grey patch, green channel
g = (cfa == 1)[patch]
print(f"grey patch green: {raw_e[patch][g].mean():.0f} e-, std raw "
      f"{adc_e[patch][g].std():.1f} e-, demosaiced "
      f"{demosaic(adc_e)[patch][..., 1].std():.1f} e-")

# 5. Camera processing: white balance, sRGB tone curve, 8-bit output
rgb = demosaic(adc_e) / full_well
grey = rgb[patch].reshape(-1, 3).mean(axis=0)
balanced = np.clip(rgb * (grey[1] / grey), 0, 1)  # grey patch -> R = G = B
srgb = np.where(balanced <= 0.0031308, 12.92 * balanced,
                1.055 * balanced ** (1 / 2.4) - 0.055)
out = np.round(255 * srgb).astype(np.uint8)
print("grey patch linear RGB:", np.round(grey, 3))
print("after white balance:  ", np.round(balanced[patch].reshape(-1, 3).mean(0), 3))
print("8-bit sRGB codes:     ", out[patch].reshape(-1, 3).mean(0).round(1))

Output (NumPy 2.3.5, SciPy 1.16.3):

blur circle 3.82 um = 1.27 pixels
                                       contrast  false colour
stage                        RMSE coarse   fine coarse   fine
ideal sampling              0.000  0.800  0.800  0.000  0.000
+ defocus blur              0.046  0.800  0.590  0.000  0.000
+ Bayer mosaic, demosaic    0.133  0.756  0.221  0.295  0.588
+ shot and read noise       0.134  0.767  0.231  0.311  0.620
+ 10-bit ADC                0.134  0.767  0.231  0.311  0.620
raw noise / model std: 1.003
grey patch green: 356 e-, std raw 19.9 e-, demosaiced 15.8 e-
grey patch linear RGB: [0.444 0.355 0.243]
after white balance:   [0.355 0.355 0.355]
8-bit sRGB codes:      [160.6 160.6 160.6]

Each stage damages a different property of the known image. Defocus of about one and a quarter pixels leaves the coarse bars, four pixels wide, at full contrast in their centres, but removes a quarter of the contrast of the fine bars: blur is a low-pass filter, and it acts first on fine detail. The mosaic and bilinear demosaicing do the most damage. The fine bars, with a period of 3 pixels, lie beyond the Nyquist limit of the red and blue samples (a period of 4 pixels), so those channels alias and the grey bars turn strongly coloured, and their contrast falls to under a third of the ideal. Even the coarse bars gain colour fringes at every edge, where the interpolated red and blue lag behind the green. Almost all of the final error comes from this stage, not from noise.

Shot and read noise leave the error almost unchanged here, because they are smaller than the demosaicing error and errors add in quadrature; the slight rise in contrast is noise in the column averages, not recovered detail. The measured raw noise is within 0.3% of the model $\sigma^2 = \mu_e + \sigma_d^2 + q^2/12$ . Demosaicing lowers the measured standard deviation in the flat patch because half of the green pixels there are averages of four noisy neighbours, which also makes the noise spatially correlated. The 10-bit ADC changes nothing visible: its step is about one electron, far below the 20-electron noise in the patch, which is how converters are meant to be chosen. Finally, the warm light makes the grey patch reddish in raw values; white-balance gains make it neutral, and the sRGB curve maps its linear value of 0.355 to code 161 of 255, well above the midpoint, because gamma encoding gives more codes to dark tones.

Common Misconceptions

“Pixel values are proportional to scene brightness.” Raw values minus the black level are close to proportional. JPEG values have passed through white balance, a tone curve and gamma, so doubling the light does not double the value; color spaces gives the sRGB curve to undo.
“A smaller aperture always gives a sharper image.” It increases depth of field, but diffraction blur grows in proportion to the f-number and dominates at small apertures.
“The file’s bit depth is the measurement precision.” The number of bits often exceeds the useful bits; the noise floor decides how many levels carry information.
“An object has a colour, and the camera records it.” The raw value depends on the illuminant’s spectrum and the camera’s filters as much as on the surface; white balance estimates and removes the illuminant’s effect, and can get it wrong.
“Synthetic training data only needs Gaussian noise.” Raw sensor noise depends on the signal, and the ISP reshapes it. Brooks et al. (2019) attribute part of the gap between learned denoisers and real raw images to unrealistic synthetic training data, and reported lower error than earlier methods on the Darmstadt Noise Dataset of real raw images by synthesizing training data that inverts each pipeline step and adds shot and read noise.

Where It Is Used

Physically based vision methods invert parts of the chain. Photometric stereo recovers surface orientation from the shading term and, like accurate deblurring, needs measurements that are linear in the light, so the gamma and the per-image colour balance must be undone first (Szeliski, 2022, Section 2.3.2); shape from shading relies on the same shading term. High dynamic range imaging merges exposures after estimating the camera response function. Deblurring needs the PSF, and image denoising works best with a noise model that matches the sensor. Camera calibration estimates the geometric part of the chain, including lens distortion; photometric calibration estimates the response, vignetting and noise.

The chain also explains the assumptions and failure modes of other methods. Correspondence, optical flow and stereo assume brightness constancy, which shading changes, highlights and auto-exposure violate. Colour-based segmentation depends on white balance. Learned models inherit the processing of the images they were trained on, and data augmentation such as blur, noise and colour jitter can be read (this is an interpretation, not a claim of the augmentation literature) as sampling variations of the formation chain. In the other direction, rendering and neural radiance fields model scene radiance and then project it through a camera, which is image formation run forward.