Computational Photography

Computational photography makes photographs by combining capture and computation: a camera records one or more measurements, often chosen with the later computation in mind, and an algorithm turns them into an image that no single conventional exposure could produce. A phone that merges a burst of short frames into one low-noise picture, blurs a background according to an estimated depth map, or stitches a panorama is doing computational photography; so is a research camera with a patterned mask in its lens or a microlens array over its sensor.

The single camera and its image signal processor are the subject of digital camera; how light becomes a pixel value is followed in image formation.

Definition

There is no single definition, and the careful ones differ in how much of the camera they include.

• Algorithms on photographs. Szeliski (2022, Chapter 10 introduction) describes computational photography as applying image analysis and processing algorithms to one or more photographs to create images beyond the capabilities of traditional imaging systems. His chapter covers photometric calibration, HDR, flash photography, super-resolution and denoising, demosaicing, synthetic lens blur, matting and texture synthesis; panoramic stitching appears earlier as one instance (Section 8.2). He notes that the book does not cover novel computational sensors, optics and cameras extensively.
• Optics, sensors, illumination and processing. Raskar and Tumblin’s course page defines the field more widely: it combines computing, digital sensors, modern optics, actuators and controlled lighting to escape the limits of film-like cameras. They split it into four elements, generalized optics, generalized sensors, processing and generalized illumination, the first three forming a “computational camera”, and treat the camera as encoding rays that the processing later decodes. They distinguish epsilon photography, in which several images are taken with small changes of exposure, focus, aperture, viewpoint, illumination or time and combined, from coded photography, in which a single image (or a very few) records an encoded representation that may look distorted to a person but can be decoded into, for example, a light field or a deblurred image.
• Mobile photography. Delbracio, Kelly, Brown and Milanfar (2021, abstract) call it the science and engineering of making great images from small form-factor mobile cameras.

The first definition is about processing, the second adds the design of the capture itself, and the third is about one platform and one goal, a pleasing photograph. This article uses the broad sense, since the methods that reached phones began as research on the capture side as well as on algorithms.

Intuition

A single exposure is a set of compromises. The exposure article describes them: a long exposure collects more photons but smears motion, a short one is sharp but noisy, a wide aperture gathers light but narrows the depth of field, and a sensor’s well capacity clips the highlights of a bright scene. A small phone camera makes each compromise harder: Delbracio et al. (2021, Section 2) list its small sensor and fixed aperture, its limited dynamic range and depth of field, its limited zoom and the colour subsampling of its filter array.

Computational photography escapes a compromise by changing what is measured. Several short frames can replace one long one if they are aligned before they are added; frames with different exposures can be fused; a second picture with a flash can supply edges to a noisy picture without one; a microlens array can record the direction of rays so that focus is chosen after capture; a mask in the aperture or a fluttering shutter can make blur invertible. In each case the recorded data are no longer a finished picture but evidence from which a picture is computed.

Formal Definition

Most multi-frame methods can be read as inverting a forward model of capture. Szeliski (2022, Eq. 10.25) writes the $k$-th observed image as

$$o_k(\mathbf{x}) = D\{\, b(\mathbf{x}) * s(\hat{h}_k(\mathbf{x})) \,\} + n_k(\mathbf{x}),$$

where $s$ is the sharp image to be recovered, $\hat{h}_k$ warps it into frame $k$, $b$ is the blur kernel, $D$ downsamples onto the sensor grid and $n_k$ is noise. Burst denoising estimates $s$ at the sensor’s resolution, multi-frame super-resolution at a finer one, and HDR varies the exposure between frames; a light-field or coded camera changes the model itself so that the inversion is better posed.

The basic gain of merging follows from the noise. If $N$ frames are aligned exactly and their noise is independent with variance $\sigma^2$, their mean has variance $\sigma^2 / N$; Delbracio et al. (2021, Section 3.3.3) state that averaging same-exposure frames of a static scene reduces noise by a factor of the square root of the number of frames. The work is in making the frames align, and in deciding what to do where they do not.

Methods

Burst capture: align and merge

Burst photography captures several frames for every shutter press, aligns them to one reference frame and merges them. Delbracio et al. (2021, Section 3.3, Fig. 11) describe the pipeline of most phone cameras in four steps: capture according to an exposure schedule, alignment, merge, and enhancement (colour, tone mapping and noise reduction). Phones can keep frames in a ring buffer while the camera app is open, so that a shutter press merges frames taken just before it, a mode called zero shutter lag; the merge must be robust to misalignment, because alignment fails on occlusions, non-rigid motion and lighting changes, leaving ghosts and zipper artifacts at the edges of moving objects.

Hasinoff et al. (2016) described the system marketed as HDR+. It captures a burst of frames with the same exposure rather than a bracket, because frames of different exposure are harder to align, and sets that exposure low enough to avoid clipping highlights; merging then removes the extra noise, so that the authors treat HDR imaging as denoising (Sections 1 and 3). It works on Bayer RAW frames rather than the ISP’s processed output, for more bits per pixel, linear values and a simple noise model whose variance is $A x + B$ for signal $x$ (Sections 2 and 5). The reference frame is the sharpest of the first three, a form of lucky imaging; alignment is a coarse-to-fine search over tiles of a Gaussian pyramid; and the merge is a pairwise Wiener-like filter in the frequency domain of each tile, which takes an alternate frame’s frequency content where its difference from the reference can be ascribed to noise and falls back to the reference where it cannot (Sections 4–5, Eqs. 6–7). The shipped system captured bursts of up to 8 frames on 12–13-megapixel sensors (Section 7).

Liba et al. (2019) extended this to very low light, down to about 0.3 lux, in the system launched in November 2018 as Night Sight (Section 1). Frames that also feed the viewfinder are limited to about 66 ms, so night modes capture frames after the shutter press (Delbracio et al., 2021, Section 4.1); in Night Sight, “motion metering” measures motion before the press to choose the per-frame exposure time and gain that trade noise against motion blur: up to 333 ms per frame handheld and up to 1 s when the phone is detected to be braced or on a tripod, within a total capture of up to about 6 seconds (Sections 1–2). It adds a learned white balance trained for night scenes and a tone mapping that brightens the image but keeps the darkest regions near black, so that the picture still looks like night (Sections 4–5).

High dynamic range

When a scene spans more brightness than one exposure can record, several exposures are combined into a radiance map with the camera’s response function, the method of Debevec and Malik (1997), and the result is tone-mapped for display. High dynamic range imaging covers bracketing, response recovery, merging and ghost removal, HDR sensors, single-image reconstruction and tone mapping. HDR+ instead tone-maps its merge of equal, underexposed frames with a local operator (Hasinoff et al., 2016, Section 6).

Burst denoising and multi-frame super-resolution

Merging aligned frames is a form of image denoising that uses several observations instead of a prior alone; Szeliski (2022, Section 10.3) notes that in low light a rapid burst at low gain, combined afterwards, is preferable to one frame at high gain. When the frames are offset by fractions of a pixel, they sample the scene at different points, and multi-frame super-resolution can recover detail finer than one frame’s grid.

Wronski et al. (2019) built this into a phone. Their algorithm replaces demosaicing: it merges a burst of RAW colour-filter-array frames directly onto a full RGB grid by kernel regression, with kernel shapes adapted to local gradients and a robustness model that down-weights samples where alignment is doubtful (Sections 1 and 5). The sub-pixel offsets come from natural hand tremor: over 86 bursts captured by 10 users, the gyroscope showed random angular displacements whose standard deviation corresponded to about 0.89 pixels (Section 4, Fig. 3). The authors state that the algorithm runs at 100 ms per 12-megapixel frame and underlies the Super-Res Zoom feature and the default merge of Night Sight on Google’s flagship phone at the time. Delbracio et al. (2021, Section 4.2) add that the gain from merging is modest, comparable to a dedicated telephoto lens at no more than about 2× magnification, so phone zoom pipelines combine it with single-image upscaling.

Synthetic depth of field

A phone’s small aperture keeps nearly everything in focus, so the shallow depth of field of a large camera has to be synthesized. Szeliski (2022, Section 10.3.2) counts the wish to simulate refocusable, shallow-depth-of-field cameras among the driving forces of the field and of light-field cameras. On phones the usual method is to estimate depth and blur the image accordingly: Wadhwa et al. (2018) describe a single-camera system, marketed as Portrait Mode on Google phones, that combines a person-segmentation network with depth from the sensor’s dual-pixel autofocus hardware, which they describe as a two-sample light field with a baseline of about 1 millimetre. The dual-pixel frames are first aligned and averaged as a burst, a stereo algorithm estimates disparity, and a calibrated, edge-aware upsampling gives a dense map that drives the blur (Sections 1, 4). The authors note that they deliberately depart from physically correct defocus, for example by keeping a range of depths around the focal plane sharp. Depth from such cues is part of depth estimation and stereo vision.

Panoramas and image stitching

Images from a camera that only rotates are related by a homography, so they can be aligned from matched features, warped onto a common surface and blended into a panorama wider than any single frame. Image stitching thus combines image registration, warping and blending; the image pyramid article describes the multi-band blending that hides the seams. Szeliski (2022, Chapter 10 introduction) treats stitching as one instance of computational photography.

Flash and no-flash pairs

A flash photograph in dim light has low noise and sharp detail but flat, cold lighting and hard shadows; a picture with the available light has the right mood but is noisy or blurred. Two papers of 2004 combined the two. Petschnigg et al. (2004) denoise the no-flash image with a joint bilateral filter whose range weights come from the flash image, transfer the flash image’s detail as a ratio to its own filtered version, and fall back to ordinary filtering in flash shadows and specularities; they also use the pair for white balance, adjustable flash strength and red-eye removal. Eisemann and Durand (2004) decompose both images into a large-scale layer and a detail layer with the bilateral filter, take the large scale from the available-light image and the detail from the flash image, and detect and correct flash shadows. Szeliski (2022, Section 10.2.2) presents both and notes that the first calls its filter joint bilateral, the second cross bilateral. Hasinoff et al. (2016, Section 1) judged flash/no-flash photography not sufficiently robust.

Light fields and plenoptic cameras

A conventional camera records the total light arriving at each pixel; a light-field camera also records the direction of the rays. Ng et al. (2005) built a hand-held plenoptic camera by placing a microlens array in front of the sensor, so that each microlens images the lens aperture onto the pixels beneath it, and computed photographs refocused at different depths, with extended depth of field and small changes of viewpoint, by re-sorting the recorded rays (abstract, Section 1). They describe their optical design as very similar to Adelson and Wang’s plenoptic camera of 1992 and trace its roots to Lippmann’s integral photography of 1908 (Section 2). The price is resolution: their prototype’s sensor of about 4000 × 4000 pixels gave light fields of 292 × 292 spatial samples by just under 14 × 14 directions (Section 5.1). Szeliski (2022, Section 10.3.2) notes that commercial light-field cameras were produced, but that refocusable, shallow-depth-of-field pictures became widespread only recently, with phones. Light fields as a representation for rendering new views are part of neural rendering and novel view synthesis.

Coded imaging

Coded imaging changes the optics or the timing so that information a conventional camera would lose survives in a single photograph, at the cost of an image that must be decoded.

• Coded exposure. An ordinary exposure is a box filter in time, which turns motion into a blur that destroys high spatial frequencies and makes deconvolution ill-posed. Raskar, Agrawal and Tumblin (2006) opened and closed the shutter during the exposure in a pseudo-random binary sequence, which they call a fluttered shutter, so that the blur becomes a broadband filter and the motion can be inverted; their code had 52 time slices, half of them open, and their prototype used a ferroelectric liquid-crystal shutter. The user marks the moving region and its motion by hand.
• Coded aperture. Levin, Fergus, Durand and Freeman (2007) put a patterned occluder, in their experiments a disc of card, into an ordinary lens, with a pattern designed so that defocus blur at different depths is easy to tell apart. From a single photograph and a statistical model of images they recovered an all-focus image and a layered depth map, with occasional user strokes. On planar test scenes of known depth, local evidence alone classified the depth correctly in 80% of the images with their aperture and 40% with the unmodified lens (Section 5.3), and the mask blocked about half the light (Section 6).

Historical Development

The ideas are older than the name: integral photography dates from 1908 (as traced by Ng et al., 2005), Debevec and Malik’s radiance maps from 1997, flash/no-flash fusion from 2004, the hand-held plenoptic camera from 2005, coded exposure from 2006 and the coded aperture from 2007. The same thinking reached video with Eulerian video magnification (Wu et al., 2012), which amplifies changes too small to see, such as the colour change of skin with each heartbeat, by band-pass filtering each pixel’s intensity over time in a spatial pyramid. Raskar and Tumblin’s page lists SIGGRAPH courses on computational photography in each year from 2005 to 2008, the later ones taught with Debevec, Levoy, Nayar and others, and describes the field as evolving in three phases, from epsilon to coded photography and then to what Raskar calls essence photography, which captures the perceptually important content of a scene rather than physical quantities.

The move to phones needed cameras that software could control frame by frame. Szeliski (2022, Chapter 10 introduction) starts his account of the decade of commercial impact in 2010, when the Frankencamera paper of Adams, Talvala et al. and one of the first widely used in-camera panorama apps had just appeared; Hasinoff et al. (2016, Section 2) built HDR+ on Android’s Camera2 interface, which they describe as based on the Frankencamera’s request architecture and which gives per-frame exposure control and access to RAW data. Delbracio et al. (2021, Section 1) date the pressure for better phone cameras to around 2010, when 4G networks and high-resolution screens made photo sharing on phones common.

The deployed phone systems described above are those whose makers documented them in technical papers, all from Google Research: HDR+ on the Nexus 6, 5X and 6P (Hasinoff et al., 2016, Section 8), Portrait Mode (Wadhwa et al., 2018), Night Sight from November 2018 (Liba et al., 2019) and the super-resolution merge (Wronski et al., 2019). Hasinoff et al. (2016, Section 1) note that the algorithms of commercial camera systems are generally proprietary, and Szeliski (2022, Section 10.3.2) that the technical details of the first phone portrait mode, on a dual-camera phone, were never published. By Szeliski’s account (Chapter 10 introduction), by 2020 every smartphone had built-in panoramic stitching, HDR merging and multi-image denoising and super-resolution, and the newest phones simulated shallow depth of field with several lenses or dual pixels. Delbracio et al. (2021, Section 5.1) observed that, as of 2021, the progress in mobile imaging had come mostly from methods not based on deep networks, which were then too costly for phones and prone to artifacts that are hard to diagnose.

Examples

Burst align-and-merge on a rendered scene

The code renders a scene of known radiance (a Siemens star, 3-pixel bars, a checkerboard, a disc and a small bright ball that moves 1.5 pixels to the right per frame) and photographs it with a hand-held camera whose shake is a random walk, sampled 8 times within each frame. It compares a single short frame, a single long exposure as long as the whole burst, and a burst of 8 short frames merged three ways. Each frame collects 25 electrons at radiance 1 with 3 electrons of read noise, so the scene is dim; the long exposure collects 8 times as many electrons with one read. The burst is aligned to its middle frame by phase correlation, a global sub-pixel translation estimated with scikit-image, and merged by a plain mean or by a mean that drops, at each pixel, frames whose local $3 \times 3$ average differs from the reference frame’s by more than three standard deviations of the expected noise. The ground truth is the noise-free scene at the reference frame’s mid-exposure; the error is the root-mean-square error (RMSE) in radiance, over the whole image and in a window around the ball’s path. The long exposure is registered to the ground truth first, so its error measures blur and noise rather than an offset.

import numpy as np
from scipy import ndimage
from skimage.registration import phase_cross_correlation

H = W = 128                 # output size (pixels)
SS = 4                      # supersampling per pixel side
N, SUB = 8, 8               # frames in the burst; time steps per frame
RATE, READ = 25.0, 3.0      # electrons at radiance 1 per short frame; read noise (e-)

yy, xx = (np.mgrid[0:H * SS, 0:W * SS] + 0.5) / SS      # pixel coordinates

def radiance(dx, dy, ball):
    """Static scene seen by a camera shifted by (dx, dy), plus a moving ball."""
    x, y = xx + dx, yy + dy
    r = np.hypot(x - 40, y - 40)
    star = 0.5 + 0.4 * np.sign(np.sin(12 * np.arctan2(y - 40, x - 40)))
    img = np.where(r < 30, star, 0.25)                  # Siemens star
    bars = (np.floor((x - 76) / 3) % 2 == 0) & (x > 76) & (x < 118) \
        & (y > 14) & (y < 60)
    img = np.where(bars, 0.85, img)                      # 3-px bars
    img = np.where((x > 16) & (x < 64) & (y > 84) & (y < 116),
                   0.1 + 0.8 * ((np.floor(x / 4) + np.floor(y / 4)) % 2), img)
    img = np.where(np.hypot(x - 96, y - 96) < 14, 0.6, img)          # disc
    bx, by = ball                                        # ball moves in the scene
    img = np.where(np.hypot(x - bx, y - by) < 5, 1.0, img)
    return img.reshape(H, SS, W, SS).mean(axis=(1, 3))    # area-average to pixels

def simulate(seed):
    rng = np.random.default_rng(seed)
    steps = rng.normal(0, 0.6 / np.sqrt(SUB), size=(N * SUB, 2))   # hand shake
    path = np.cumsum(steps, axis=0)
    path -= path[(N // 2) * SUB + SUB // 2]       # reference frame at the centre
    ball = lambda t: (70 + 1.5 * t / SUB, 100.0)  # 1.5 px per frame, rightwards
    sub = np.array([radiance(*path[t], ball(t)) for t in range(N * SUB)])
    ref = N // 2
    truth = radiance(*path[ref * SUB + SUB // 2], ball(ref * SUB + SUB // 2))
    clean = sub.reshape(N, SUB, H, W).mean(axis=1)       # each short frame, noise-free

    def expose(mean_rad, length):    # length in short-frame units; e- -> radiance
        e = rng.poisson(RATE * length * mean_rad) + READ * rng.standard_normal((H, W))
        return e / (RATE * length)
    frames = np.array([expose(c, 1) for c in clean])        # burst of short frames
    long_exp = expose(sub.mean(axis=0), N)                  # one exposure, N times longer

    # Align every frame to the reference by phase correlation (global shift).
    shifts = np.array([phase_cross_correlation(frames[ref], f, upsample_factor=20)[0]
                       for f in frames])
    aligned = np.array([ndimage.shift(f, s, order=1, mode="nearest")
                        for f, s in zip(frames, shifts)])
    true_shift = np.array([path[k * SUB + SUB // 2] - path[ref * SUB + SUB // 2]
                           for k in range(N)])[:, ::-1]      # (dy, dx) convention
    align_err = np.abs(shifts - true_shift).max()
    travel = np.hypot(*(path.max(0) - path.min(0)))  # extent of the shake (px)

    # Robust merge: drop pixels whose local mean differs from the reference by
    # more than 3 standard deviations of the expected noise in that difference.
    box = lambda a: ndimage.uniform_filter(a, 3, mode="nearest")
    var = (np.clip(box(frames[ref]), 0, None) * RATE + READ**2) / RATE**2
    tol = 3 * np.sqrt(2 * var / 9)
    keep = np.abs(box(aligned) - box(frames[ref])) < tol
    keep[ref] = True
    robust = (aligned * keep).sum(0) / keep.sum(0)

    # The long exposure is registered to the truth so its error is blur, not offset.
    s_long = phase_cross_correlation(truth, long_exp, upsample_factor=20)[0]
    long_reg = ndimage.shift(long_exp, s_long, order=1, mode="nearest")

    ballmask = np.zeros((H, W), bool)
    ballmask[88:112, 60:100] = True              # area around the ball's path
    rmse = lambda a, m=np.ones((H, W), bool): np.sqrt(((a - truth)[m] ** 2).mean())
    out = {"single short frame": frames[ref],
           "single long exposure": long_reg,
           "burst, mean, unaligned": frames.mean(0),
           "burst, mean, aligned": aligned.mean(0),
           "burst, aligned + robust": robust}
    rows = {k: (rmse(v), rmse(v, ballmask)) for k, v in out.items()}
    return rows, align_err, travel, 1 - keep.mean()

rows, align_err, travel, rejected = simulate(0)
print(f"{N} frames, {RATE:.0f} e- at white per frame, read noise {READ:.0f} e-")
print(f"seed 0: shake spans {travel:.1f} px, worst alignment error "
      f"{align_err:.2f} px, {100 * rejected:.1f}% of samples rejected")
print(f"{'method':26s} RMSE   RMSE near ball")
for k, (a, b) in rows.items():
    print(f"{k:26s} {a:.4f}  {b:.4f}")

print("\nRMSE over seeds 0-9 (min-max):")
runs = [simulate(s)[0] for s in range(10)]
for k in rows:
    v = np.array([r[k] for r in runs])
    print(f"{k:26s} {v[:, 0].min():.4f}-{v[:, 0].max():.4f}  "
          f"near ball {v[:, 1].min():.4f}-{v[:, 1].max():.4f}")

Output (NumPy 2.3.5, SciPy 1.16.3, scikit-image 0.25.2):

8 frames, 25 e- at white per frame, read noise 3 e-
seed 0: shake spans 3.3 px, worst alignment error 0.47 px, 0.4% of samples rejected
method                     RMSE   RMSE near ball
single short frame         0.1785  0.1818
single long exposure       0.1194  0.1321
burst, mean, unaligned     0.1407  0.1444
burst, mean, aligned       0.0828  0.1224
burst, aligned + robust    0.0799  0.0839

RMSE over seeds 0-9 (min-max):
single short frame         0.1733-0.1785  near ball 0.1786-0.1878
single long exposure       0.0787-0.1522  near ball 0.1176-0.1670
burst, mean, unaligned     0.0905-0.1630  near ball 0.1290-0.1765
burst, mean, aligned       0.0759-0.0846  near ball 0.1194-0.1273
burst, aligned + robust    0.0728-0.0818  near ball 0.0794-0.0877

The single short frame is sharp but noise-limited, and its error barely changes between seeds because it depends on the photon count, not on the shake. The long exposure has eight times the photons, and its error depends on how far the hand happened to move: over the ten seeds it ranges from about the level of the merged bursts, when the shake was small, to nearly twice it. Averaging the burst without alignment has a range of errors above the long exposure’s; it follows from the set-up that it has the same motion blur, sampled more coarsely, and eight read-noise contributions instead of one.

Alignment removes the dependence on the shake: the aligned mean’s error stays within a narrow band over all seeds, close to the long exposure’s best case. It does not reach the ideal $\sigma/\sqrt{8}$, which would put it near 0.063 given the short frame’s error. The code does not separate the causes, but the set-up suggests that the remaining error is mostly not noise: the blur within each short frame, the smoothing of linear interpolation and the moving ball’s ghost all add errors that averaging cannot remove. Phase correlation on frames this noisy is not exact either; the image registration article explains why its whitening of the spectrum can hurt at low signal-to-noise ratios.

The moving ball is where a plain mean fails: aligning the camera motion does not align an object that moves through the scene, so the mean leaves a ghost, and its error near the ball stays near 0.12, the level of the long exposure’s best seeds. The robust merge rejects only 0.4% of the samples, yet its error near the ball falls to about the level of the whole image. That is the design choice of HDR+ in miniature: fall back to the reference frame wherever the alternates disagree with it by more than noise. It follows that the merged image is noisier where frames were rejected, which Delbracio et al. (2021, Section 3.3.3) describe as spatially varying noise that later denoising must handle.

What It Means for Vision

A vision method that receives a phone photograph is usually not receiving a single exposure. Several of the assumptions that methods make about their input then fail, in ways the sources describe.

• One instant. A merged picture spans the time of its burst, up to about 6 seconds in Night Sight (Liba et al., 2019), and robust merges fall back to the reference frame per tile or per pixel, so it follows that different regions of the picture can carry information from different numbers of frames and, near moving objects, effectively from different times, and that models built on one known exposure time, such as motion-blur analysis or rolling-shutter correction, describe the captured frames rather than the merged result.
• The sensor’s noise model. The noise in RAW frames is close to independent and signal-dependent; image noise describes the model. After a partial merge, its strength varies over the image (Delbracio et al., 2021, Section 3.3.3), and the later stages of the pipeline correlate and reshape it, as digital camera explains for a single frame.
• Measured content. Synthetic depth of field is computed from an estimated depth map and is deliberately not physically correct defocus (Wadhwa et al., 2018); super-resolved detail is reconstructed from aliasing across frames (Wronski et al., 2019); phone zoom pipelines add learned single-image upscaling (Delbracio et al., 2021, Section 4.2); and night tone mapping is designed to look like a night scene rather than to measure it (Liba et al., 2019). It follows that errors in these steps are artifacts of the computation, not properties of the scene.
• Consistency between images. The exposure and the amount of dynamic range compression are chosen per scene (Hasinoff et al., 2016, Section 3), and Night Sight chooses exposure time and gain from the measured motion (Liba et al., 2019, Section 2). It follows that the mapping from light to pixel value changes from shot to shot, which breaks the brightness constancy that matching and flow methods assume.

Delbracio et al. (2021, Section 5.3) state the general point: mobile pipelines are tuned to produce aesthetically pleasing images, while scientific and other uses need informative ones, and phone cameras are not built, configured or tuned to provide such information. It follows (a practical reading, not a rule from one source) that a vision system that measures, whether depth, colour, photometry or noise, should capture RAW frames or switch the computational modes off and record the capture settings; that a model trained on phone photographs has learned the phones’ processing as well as the scenes, which matters when it meets images from another pipeline (a dataset bias); and that a portrait photo’s background blur or a night photo’s colours are not evidence about the scene.

Common Misconceptions

• “Computational photography means neural filters.” The field includes optics, sensors and illumination as much as software (Raskar and Tumblin), and as of 2021 mobile imaging had progressed mostly without deep networks (Delbracio et al., 2021, Section 5.1).
• “Merging a burst is averaging.” Plain averaging suits only a static, well-aligned scene, which Delbracio et al. (2021, Section 3.3.3) note real photography rarely offers; deployed merges reject or down-weight content that does not match the reference (Hasinoff et al., 2016, Fig. 6).
• “Burst HDR needs bracketed exposures.” HDR+ uses frames of equal, deliberately low exposure, because they align more reliably (Hasinoff et al., 2016, Section 1).
• “Multi-frame super-resolution needs a tripod or a sensor-shift mechanism.” Natural hand tremor sufficed in the bursts Wronski et al. (2019, Section 4) analysed.
• “A light-field camera gives focus for free.” It trades spatial resolution for directional samples, as Ng et al.’s (2005) prototype shows.

Where It Is Used

Delbracio et al. (2021, Section 3.3) describe multi-frame processing as built into the default photography mode of smartphone cameras, which run the methods above, with depth for synthetic depth of field stored as a depth map. Its building blocks are general image processing tools: image registration and optical flow estimation for alignment, the bilateral filter for flash/no-flash fusion and tone mapping, and the Fourier transform for frequency-domain merging. Light fields lead to novel view synthesis and neural rendering, and coded apertures to depth from defocus. Delbracio et al. (2021, Section 5.3) expect the same techniques to be useful for scientific, automotive and satellite imaging, with the caution given above.