Structured Light

Structured light is an active way of measuring depth. A projector casts a known pattern onto the scene and a camera, mounted at a fixed offset beside it, photographs the result. Because the pattern encodes the projector’s coordinates, each camera pixel can read off which projector column (or row, or pixel) lights the surface point it sees. The projector then acts as a second camera with a known image, and depth follows by triangulation, exactly as in stereo vision, but without searching for matches in natural texture. Lanman and Taubin (2009) describe many active systems as replacing one camera of a stereo pair with a controllable light source. The idea covers a family of sensors: laboratory scanners that project a sequence of stripe or fringe images, single-shot depth cameras such as the first Microsoft Kinect that project a fixed dot pattern, and laser-line profilers on production lines.

This article covers the projector–camera geometry, the main coding schemes (binary and Gray codes, phase shifting, their combination, and single-shot spatial codes), what the sensor outputs, how it is calibrated, how the patterns are decoded, and where it fails. A worked example decodes Gray code and phase-shifted fringes on a synthetic scene with exact ground truth.

How It Works

The projector as an inverse camera

Under the pinhole model, a projector maps each point of its image plane to an outgoing ray, the reverse of a camera, which maps incoming rays to image points (Lanman & Taubin, 2009, ch. 2.3 and 3.2). A single projector column sweeps out a plane of light through the projector’s centre, and a single projector pixel a ray. A camera pixel defines a ray through the camera’s centre. Once the code tells a camera pixel which projector column lit its surface point, the point is the intersection of the camera ray $\mathbf{p} = \mathbf{q}_L + \lambda\mathbf{v}$ with the column’s plane $\mathbf{n}^\top(\mathbf{p} - \mathbf{q}_P) = 0$:

$$\lambda = \frac{\mathbf{n}^\top(\mathbf{q}_P - \mathbf{q}_L)}{\mathbf{n}^\top \mathbf{v}},$$

which is Lanman and Taubin’s ray–plane triangulation (their Eq. 2.4); it fails only when the ray lies parallel to the plane. If the code identifies both the column and the row, the point can instead be found as the closest point between the camera ray and the projector ray (ray–ray triangulation), and the two reconstructions can be compared to remove artefacts (ch. 5.4). Moreno, Son and Taubin (2015) state the same model: each projector column corresponds to a plane, each camera pixel to a ray, and the correspondence is found “with no search, independently of the scene content”.

When the projector and camera are rectified like a stereo pair, with parallel optical axes, equal focal lengths $f$ (in pixels) and a baseline $b$ along the image rows, a camera pixel at column $u$ that sees projector column $x_p$ has disparity $d = (u - c_x) - (x_p - c_{x,p})$, and depth is $Z = fb/d$, the stereo relation with the projector in place of the second camera. HyperDepth’s authors write the same $Z = bf/d$ for a projector treated as a “virtual camera” that always sees the same reference image (Fanello et al., 2016, §2.1). The stereo vision article derives the consequences, among them that an error $\delta d$ in the decoded column moves depth by about $Z^2\delta d/(fb)$.

Temporal codes: binary and Gray

The simplest scheme sweeps a single bright column across the scene, which needs as many images as the projector has columns: 1,024 for the projector Lanman and Taubin used. Coded sequences reduce this to about $\log_2$ of that number. In a binary code, image $k$ shows bit $k$ of every column’s index, so a camera pixel that records the sequence bright, dark, dark, … reads off its column’s binary number. Lanman and Taubin (2009, ch. 5.1.2) credit this sequence to Posdamer and Altschuler, dating it 1981 in their text while their reference is the 1982 journal paper.

A Gray code reorders the codewords so that adjacent columns differ in exactly one bit. Lanman and Taubin attribute its use for structured light to Inokuchi, Sato and Matsuda (1984) and give the conversion in both directions: each Gray bit is the XOR of two neighbouring binary bits, and each binary bit is the XOR of the previous binary bit with the current Gray bit (their Table 5.1). In their example, projector column 546 is 1000100010 in binary and 1100110011 in Gray code. Because neighbouring columns differ in one bit, a pixel near a stripe boundary that misreads that bit lands on the adjacent column, an error of one column, not a jump across the projector; they note that the Gray sequence “tends to be more robust to decoding errors” for this reason.

Each image must be turned into bits. Lanman and Taubin describe two ways (ch. 5.2):

• A threshold. Project an all-white and an all-black image as well, and call a pixel lit when it is brighter than the mean of the two. This adapts to the surface’s albedo but, as they note, a single fixed threshold produces decoding errors, for instance where light scattered from lit points reaches unlit ones.
• Inverse patterns. Project every bit plane followed by its inverse, and call the bit high when the plane is brighter than its inverse. This doubles the number of images ($2\lceil\log_2 w\rceil$ for $w$ columns) but is less sensitive to scattered light, because, in their words, a variable per-pixel threshold can be used.

Their reference implementation projects $2\lceil\log_2 w\rceil + 2\lceil\log_2 h\rceil + 2$ images for a $w \times h$ projector: white, black, and every column and row plane with its inverse. Its decoded output is a pair of 16-bit images holding the projector column and row of each camera pixel, with zero where no correspondence could be assigned.

Phase shifting

A binary or Gray code resolves only whole projector columns, so its depth resolution is limited by the projector’s resolution (Moreno et al., 2015). Phase shifting encodes the column instead as the phase of a sinusoidal fringe. Moreno et al. write pattern $n$ of $N$ as

$$I_n(x_p, y_p) = o + a\cos(\omega x_p + \theta_n), \qquad \theta_n = 2\pi n/N,\quad \omega = 2\pi f,$$

with offset $o$, amplitude $a$ and fringe frequency $f$ (period $T = 1/f$ columns); the camera sees $R_n = o_\omega + a_\omega\cos(\omega x_p + \theta_n)$ with unknown offset, amplitude and $x_p$, so at least three shifts are needed (their Eqs. 1–3). For $N = 4$ the solution is short: $R_3 - R_1 = 2a_\omega\sin\phi$ and $R_0 - R_2 = 2a_\omega\cos\phi$ with $\phi = \omega x_p$, so

$$\phi = \operatorname{atan2}(R_3 - R_1,\; R_0 - R_2),$$

which cancels both the offset (ambient light and the surface’s albedo times the mean projector level) and the amplitude (albedo times fringe contrast). This derivation is ours, for the shifts of Moreno et al.’s Eq. 1; with shifts of the opposite sign the sine term changes sign, so a decoder must match its own patterns. Since the phase is continuous, the decoded column is not quantized to projector pixels, and phase shifting gives one measurement per camera pixel at sub-column precision (Moreno et al., 2015).

The price is ambiguity. The phase gives only the relative phase, the position within one fringe period; the absolute column is $x_p = \hat x_p + kT$ for an unknown integer $k$, unless one period spans the whole projector. Recovering $k$ is phase unwrapping, and Moreno et al. (2015, §2–3) distinguish two kinds:

• Spatial unwrapping (fringe counting) follows the phase from pixel to pixel and adds $2\pi$ wherever it wraps. It assumes a smooth surface without depth discontinuities, such as a face, because a jump of more than half a period between neighbouring pixels is indistinguishable from a smaller jump the other way. The result is also only relative: without a reference, the whole map is known up to a multiple of the period.
• Temporal unwrapping projects fringes of several frequencies and unwraps each pixel on its own, from the lowest frequency to the highest. Moreno et al. call this Multiple Phase Shifting, which needs at least $3M$ images for $M$ frequencies, and note (citing earlier work) that the method works best when the periods grow exponentially, such as 16, 256, 4096.

Phase shifting can also be combined with a Gray code instead of a second fringe frequency: the Gray code fixes the period index $k$ at each pixel, and the phase gives the position within the period. The worked example below decodes this hybrid next to its two ingredients. Middlebury’s high-accuracy stereo ground truth, as the Middlebury article describes, was made by projecting Gray-code and sine-wave patterns and matching the decoded codes between views.

High-frequency patterns matter for a second reason. A camera pixel records ambient light, direct light from the projector, and indirect light: interreflections and subsurface scattering. Moreno et al. report that Gupta et al. observed structured light to be sensitive to these global illumination effects unless the scene is Lambertian, and that the effects are minimized when all projected frequencies lie in a narrow band. Moreno et al. note that Gray codes and multiple-frequency phase shifting use a wide spectrum of frequencies, from low to high, and so are not robust to these effects (§3.4). Micro Phase Shifting uses only high frequencies close together and unwraps by a nearest-neighbour search in a lookup table, which Moreno et al. found slow and prone to errors. Their own Embedded Phase Shifting embeds low-frequency signals in the phases of high-frequency patterns, so that temporal unwrapping still works, and needs as few as five patterns. Mirdehghan, Chen and Kutulakos (2018) go further and compute pattern sequences by optimization, minimizing the expected number of correspondence errors for a given number of patterns, projector–camera arrangement and frequency content.

Single-shot spatial codes

Temporal codes need several images of a static scene. Spatial codes put the whole label into one image, so that each pixel’s neighbourhood identifies its projector position; Lanman and Taubin (2009, ch. 1.1.2) note that purely spatial codes allow dynamic scenes, while temporal codes gain redundancy and fewer reconstruction artefacts, and they refer to Salvi, Pagès and Batlle (2004) for a survey of codes. Fanello et al. (2016) draw the same line: temporal techniques decode each pixel with a simple lookup but need fast projectors and cameras and suffer from motion artefacts (they give Intel’s F200 as a temporal device), while many consumer depth cameras use a single pseudo-random pattern from a cheap diffractive optical element, “such as in Kinect”.

The first Kinect was developed with PrimeSense, as the depth estimation article notes. PrimeSense’s patent describes an illumination assembly in which a light source shines through a single transparency containing a fixed pattern of spots, uncorrelated over the range of depths mapped; a camera at a fixed position beside it captures the pattern on the object, and depth follows from the transverse shift of the spots relative to a reference image of the pattern taken at a known distance (Freedman et al., US 8,150,142 B2, filed 2007). The patent states that its embodiments differ in this respect from laser-speckle methods, in which a diffuser creates the pattern by optical interference. Fanello et al. describe the Kinect’s pattern as produced by a diffractive optical element; the patent does not say which design a product used. Fanello et al., citing that patent and a later study, describe the PrimeSense algorithm as extracting the observed dots, matching a patch around each dot against the reference image by normalized cross-correlation, about 20 patch comparisons per pixel, and filling in a dense disparity map by region growing. Their own HyperDepth replaces the matching with per-scanline random forests that classify each pixel’s projector position directly.

Active stereo and laser lines

Two neighbouring designs are often grouped with structured light. Active stereo keeps two cameras and adds a projector whose only job is to put texture on featureless surfaces; the cameras then match as ordinary stereo matching does. Keselman et al. (2017, §2.2) describe the active texture projector of Intel’s RealSense stereo cameras this way and note that such systems, going back to 1984, were originally called unstructured light: the pattern’s structure need not be known, only densely textured, photometrically consistent and free of repetition along the matching direction. Zhang et al. (2018) use the RealSense D435’s infrared pair as their active stereo example. Laser-line (sheet-of-light) scanners project a single plane of light and sweep it, mechanically or by moving the part; Lanman and Taubin (ch. 1.1.2) describe slit scanners as removing one dimension of the slower single-spot laser scan, and a digital projector as removing the mechanical sweep. The industrial visual inspection article describes laser-line profilers and fixed-pattern triangulation cameras as they appear in machine vision.

Data It Produces

The primary output of decoding is a correspondence map: for each camera pixel, the projector column (and, with a second set of patterns, the row), as an integer for binary and Gray codes or a real number for phase shifting, plus a validity mask for pixels that are in shadow, outside the projector’s field, too dark or ambiguous. Lanman and Taubin’s decoder writes zero for pixels it cannot assign. Triangulation turns the map into a depth map or a point cloud in the camera’s frame. Single-shot sensors such as the first Kinect produce a depth image per frame from a disparity against their reference pattern; the NYU Depth v2 article gives the published conversion from that camera’s raw values to metres, which is inverse-linear in the raw value as disparity is.

Two properties of the data follow from the geometry. Only points that are both seen by the camera and lit by the projector can be measured, so a single scan has holes, and a full model needs several scans merged (Lanman & Taubin, 2009, ch. 5.5); by the geometry, the holes from projector shadows lie beside depth edges, on the side of the nearer surface away from the projector. And since depth comes from disparity, depth resolution falls with distance as $Z^2/(fb)$; a white image captured with the sequence gives each point a colour or intensity.

Characteristics

• Number of images. A Gray code with inverses for both axes of a 1024 × 768 projector needs $2 \cdot 10 + 2 \cdot 10 + 2 = 42$ images by Lanman and Taubin’s count; for columns only, 22. Four-step phase shifting needs four per frequency. Single-shot codes need one.
• Resolution. Gray codes give one integer column per pixel; phase shifting gives continuous columns, so the camera, not the projector, sets the lateral resolution (Moreno et al., 2015). Lanman and Taubin recommend a camera with at least the projector’s resolution.
• Depth precision. For a decoded column error $\delta x_p$ the depth error is about $Z^2\delta x_p/(fb)$, as for stereo. A longer baseline improves precision and enlarges the shadowed regions; Lanman and Taubin also advise avoiding small angles between camera rays and projector planes.
• Speed. Temporal codes are limited by projector and camera frame rates and need a static scene for the whole sequence. Fanello et al. (2016) give 30–60 Hz VGA as typical of depth cameras of the time; their HyperDepth prototype ran at 375 Hz.
• Wavelength. Fanello et al. (2016) note that the camera and projector of structured-light depth sensors are typically both near-infrared.

Calibration

Calibration fixes the camera’s intrinsics, the projector’s intrinsics and the rigid transform between them, after which every decoded column defines a known plane. Lanman and Taubin (2009, ch. 3.2 and 5.3) model the projector as an inverse camera with the same intrinsic parameters (focal length, principal point, skew and lens distortion) and calibrate it as an extension of Zhang’s method for cameras (camera calibration). First the camera is calibrated from a printed checkerboard. Then a checkerboard is projected onto a white planar board carrying a few printed fiducials: the fiducials give the board’s plane in camera coordinates, the projected corners are reconstructed on that plane, and the pairs of projector pixels and 3D points over 10–20 board positions calibrate the projector exactly as image points and world points calibrate a camera.

Single-shot sensors are calibrated differently: depth comes from the shift against a reference image recorded at a known distance (Freedman et al., 2012), so that image is part of the calibration. Systems that match decoded codes between two calibrated cameras, as Middlebury’s ground truth did, avoid calibrating the projector at all. By our derivation, calibration errors also grow with distance: an error in a plane’s orientation acts like a column error and, by the $Z^2/(fb)$ relation, gives a depth error that grows with the square of depth, while an error in the assumed baseline scales every depth by the same factor ($Z = fb/d$), so its error grows in proportion to depth.

Processing the Data

Decoding a temporal sequence is a per-pixel operation: classify each bit, convert the Gray code to a column, compute the phase, unwrap, and triangulate. The example below does this on a synthetic projector–camera rig and measures each decoder against exact ground truth.

Example: Gray code, phase shifting and the hybrid

The rig is rectified: a 320 × 240 camera and a 256 × 192 projector with the same focal length, 300 pixels, the projector 10 cm to the right of the camera. The camera frame has X to the right, Y down and Z forward, and depth means Z, not distance along the ray. The scene is a background plane tilted so that $Z = 1 + 0.1X$ (metres) and a box whose front face lies at $Z = 0.75$ m; the box straddles both optical centres, so neither the camera nor the projector sees its sides and its outline is a true step in depth. The surface carries a smooth synthetic albedo between 0.2 and 0.9. Each image sample is the albedo times ambient light (0.1) plus the projector’s value at the point, which is zero where the box shadows the projector or the point lies outside its field; there is no fall-off with distance. Each camera pixel averages 4 × 4 samples, so pixels on the step mix two surfaces, and Gaussian noise of standard deviation σ is added to the averaged images.

The projector shows white, black, eight Gray-code planes of its column index each followed by its inverse (most significant bit first, as Lanman and Taubin order them), and four fringes $0.5 + 0.5\cos(2\pi x_p/16 + 2\pi n/4)$ with a period of 16 columns. The Gray planes are binary per projector column; the fringes are ideal continuous sinusoids, a modelling choice of ours, since a sinusoid sampled on the projector’s pixels would itself quantize the phase. Four decoders are compared:

• Gray, inverse: each bit is plane brighter than inverse; 18 images.
• Gray, global threshold: each bit is plane brighter than one threshold for the whole image, the midpoint of the mean white and mean black images over pixels that see the projector; this is our choice of global value.
• Spatial: four-step phase, unwrapped across the image by scikit-image’s unwrap_phase (with a fixed rng, since the routine relies on a random initialization), on pixels that see the projector. Spatial unwrapping leaves the whole map uncertain by a multiple of the period, so we give it the true column at one background pixel (row 200, column 290), standing in for a marker or a known point.
• Hybrid: the Gray (inverse) column picks the period, $x = \hat x + 16\,\mathrm{round}((x_\text{Gray} - \hat x)/16)$, and the phase gives $\hat x$ within it; 22 images.

Pixels are scored in two sets. Interior pixels have all 16 samples lit and on one surface, as do all eight neighbours; the edge band holds every other pixel whose centre is lit: the step, the shadow’s border and the border of the projector’s field. For each decoder and noise level the code prints the depth RMS error (mm), the count and fraction of gross errors, where the decoded column is more than two columns from the truth (a Gray bit misread next to a stripe boundary costs up to 1.5 columns, so two separates those from code failures), and, for the Gray decoders, the number of interior bit errors against the Gray code of the true column. A NaN from a masked pixel counts as gross. The checks C0a–C0f test the Gray conversion against Lanman and Taubin’s worked example, the scene, the phase formula and unwrap_phase on the smooth test surface of its own docstring.

import os, sys
for v in ("OPENBLAS_NUM_THREADS", "OMP_NUM_THREADS", "MKL_NUM_THREADS", "VECLIB_MAXIMUM_THREADS"):
    os.environ[v] = "1"
import numpy as np
from skimage.restoration import unwrap_phase

SEED = int(sys.argv[1]) if len(sys.argv) > 1 else 0

# Rectified rig: camera at the origin, projector 0.10 m to its right (+X), same
# orientation and focal length. X right, Y down, Z forward; depth means Z.
W, H, WP, HP = 320, 240, 256, 192            # camera and projector resolution
f, b = 300.0, 0.10
cx, cy, cxp, cyp = 160.0, 120.0, 128.0, 96.0
AMB, T, N, NBITS, S = 0.1, 16.0, 4, 8, 4     # ambient, fringe period (columns), shifts, Gray bits, samples/axis
BOX = (np.array([-0.10, -0.12, 0.75]), np.array([0.20, 0.12, 1.20]))   # front face at Z = 0.75 m

def box_entry(o, d):
    """Slab test: ray parameter where o + t d enters the box (inf if missed), entry axis."""
    with np.errstate(divide="ignore", invalid="ignore"):
        t1, t2 = (BOX[0] - o) / d, (BOX[1] - o) / d
    tn, tf = np.minimum(t1, t2), np.maximum(t1, t2)
    tnear, tfar = np.nanmax(tn, -1), np.nanmin(tf, -1)
    return np.where((tnear <= tfar) & (tnear > 0), tnear, np.inf), np.nanargmax(tn, -1)

def cast(u, v):
    """Camera ray through (u, v) against the plane Z = 1 + 0.1 X and the box."""
    d = np.stack([(u - cx) / f, (v - cy) / f, np.ones_like(u)], -1)
    t_plane = 1.0 / (1 - 0.1 * d[..., 0])
    t_box, axis = box_entry(np.zeros(3), d)
    on_box = t_box < t_plane
    return np.where(on_box, t_box, t_plane)[..., None] * d, on_box, axis

def projector_view(P):
    """Continuous projector column of P, and whether the projector lights P."""
    op = np.array([b, 0.0, 0.0])
    xp, yp = f * (P[..., 0] - b) / P[..., 2] + cxp, f * P[..., 1] / P[..., 2] + cyp
    shadow = box_entry(op, P - op)[0] < 1 - 1e-9     # box between projector and P
    infield = (xp >= -0.5) & (xp < WP - 0.5) & (yp >= -0.5) & (yp < HP - 0.5)
    return xp, infield & ~shadow

def albedo(P):                                       # smooth synthetic texture, 0.2 to 0.9
    return 0.55 + 0.35 * np.sin(2 * np.pi * P[..., 0] / 0.4) * np.cos(2 * np.pi * P[..., 1] / 0.3)

def gray(j):
    return j ^ (j >> 1)

def patterns(xp, lit):
    """White, black, 8 Gray bit planes (MSB first) each followed by its inverse, 4 shifted fringes."""
    g = gray(np.clip(np.floor(xp + 0.5), 0, WP - 1).astype(int))
    pats = [np.ones_like(xp), np.zeros_like(xp)]
    for k in range(NBITS):
        bit = ((g >> (NBITS - 1 - k)) & 1).astype(float)
        pats += [bit, 1 - bit]
    pats += [0.5 + 0.5 * np.cos(2 * np.pi * xp / T + 2 * np.pi * n / N) for n in range(N)]
    return np.stack(pats) * lit

# Render: each camera pixel averages S x S samples of albedo * (ambient + projector).
vv, uu = np.mgrid[0:H, 0:W].astype(float)
offs = (np.arange(S) + 0.5) / S - 0.5
clean, lit_all, on_box_s, side = 0.0, np.ones((H, W), bool), [], 0
for oy in offs:
    for ox in offs:
        P, on_box, axis = cast(uu + ox, vv + oy)
        xp, lit = projector_view(P)
        clean = clean + albedo(P) * (AMB + patterns(xp, lit)) / S**2
        lit_all &= lit; on_box_s.append(on_box); side += int((on_box & (axis != 2)).sum())
on_box_s = np.stack(on_box_s)
Pc, box_c, _ = cast(uu, vv)                         # ground truth at pixel centres
xp_true, lit_c = projector_view(Pc)
Z_true = Pc[..., 2]
pure = lit_all & (on_box_s.all(0) | (~on_box_s).all(0))
interior = pure.copy()
for dy in (-1, 0, 1):
    for dx in (-1, 0, 1):
        interior &= np.roll(np.roll(pure, dy, 0), dx, 1)
interior[[0, -1], :] = interior[:, [0, -1]] = False
edge = lit_c & ~interior
bg, bx = interior & ~box_c, interior & box_c

def depth(x):
    with np.errstate(divide="ignore", invalid="ignore"):
        return f * b / ((uu - cx) - (x - cxp))

def gray_to_column(bits):                            # bits: (NBITS, H, W) booleans, MSB first
    g = np.zeros(bits.shape[1:], int)
    for k in range(NBITS):
        g = (g << 1) | bits[k]
    col, s = g.copy(), g >> 1
    while s.any():                                   # Gray -> binary: XOR of all right shifts
        col ^= s; s >>= 1
    return g, col

def decode(img):
    white, black = img[0], img[1]
    on, off = img[2:2 + 2 * NBITS:2], img[3:3 + 2 * NBITS:2]
    I = img[2 + 2 * NBITS:]
    mask = (white - black) > 0.1                     # decodable: projector contrast seen
    t = 0.5 * (white[mask].mean() + black[mask].mean())
    g_inv, col_inv = gray_to_column(on > off)        # each bit plane against its inverse
    g_glob, col_glob = gray_to_column(on > t)        # each bit plane against one global threshold
    psi = np.arctan2(I[3] - I[1], I[0] - I[2])       # wrapped phase of 2 pi x_p / T
    psi = (psi + np.pi) % (2 * np.pi) - np.pi        # [-pi, pi) for unwrap_phase
    xhat = T * psi / (2 * np.pi) % T                 # position within the period
    hybrid = xhat + T * np.round((col_inv - xhat) / T)
    uw = unwrap_phase(np.ma.array(psi, mask=~mask), rng=0).filled(np.nan)
    spatial = T * uw / (2 * np.pi)
    ref = (200, 290)                                 # one background pixel of known column
    spatial += T * np.round((xp_true[ref] - spatial[ref]) / T)
    amp = 0.5 * np.hypot(I[0] - I[2], I[3] - I[1])
    return dict(gray_inv=(col_inv, g_inv), gray_glob=(col_glob, g_glob),
                spatial=(spatial, None), hybrid=(hybrid, None)), t, psi, amp

# Checks on the code, the scene and the library.
js = np.arange(WP)
_, back = gray_to_column(np.array([(gray(js) >> (NBITS - 1 - k)) & 1 for k in range(NBITS)]))
print(f"C0a BIN2GRAY(546) = {gray(546):010b}; Gray->column round trip exact: {(back == js).all()}; "
      f"bits changed between neighbours: {sorted({bin(gray(j) ^ gray(j + 1)).count('1') for j in range(WP - 1)})}")
print(f"C0b camera samples on a box side face: {side}")
fr = xp_true[bg] - np.round(xp_true[bg])
print(f"C0c pixels: interior {interior.sum()} (box {bx.sum()}), edge band {edge.sum()}, unlit {(~lit_c).sum()}; "
      f"frac(x_p) on background: mean {fr.mean():+.4f}, std {fr.std():.4f} (uniform 0, 0.2887)")
dec0, t0, psi0, amp0 = decode(clean)
true_psi = 2 * np.pi * xp_true / T
pc = albedo(Pc) * (AMB + patterns(xp_true, lit_c))[2 + 2 * NBITS:]
e_pt = np.angle(np.exp(1j * (np.arctan2(pc[3] - pc[1], pc[0] - pc[2]) - true_psi)))[lit_c]
e_area = np.angle(np.exp(1j * (psi0 - true_psi)))[interior]
print(f"C0d global threshold t = {t0:.4f}")
print(f"C0e phase formula error, point samples: {np.abs(e_pt).max():.2e} rad; "
      f"pixel-area samples (interior): {np.abs(e_area).max():.2e} rad")
c0, c1 = np.ogrid[-1:1:128j, -1:1:128j]
ramp = 12 * np.pi * np.exp(-(c0**2 + c1**2))
print(f"C0f unwrap_phase on its docstring ramp: std(unwrapped - truth) = "
      f"{np.std(unwrap_phase(np.angle(np.exp(1j * ramp)), rng=0) - ramp):.2e}")

gt_code = gray(np.clip(np.round(xp_true), 0, WP - 1).astype(int))
gz = Z_true**2 / (f * b)                             # depth change per column
rng = np.random.default_rng(SEED)
print(f"\nseed {SEED}; RMS in mm; gross = |column error| > 2 (count / fraction)")
print(f"{'sigma':>6s} {'decoder':10s} {'RMS int':>10s} {'gross int':>15s} {'RMS edge':>10s} "
      f"{'gross edge':>15s} {'bit errors':>10s}")
res = {}
for sigma in (0.0, 0.005, 0.02):
    img = clean + sigma * rng.standard_normal(clean.shape)
    dec, *_ = decode(img)
    for name, (x, code) in dec.items():
        err = x - xp_true
        dz = depth(x) - Z_true
        rms = lambda m: np.sqrt(np.nanmean(dz[m]**2)) * 1e3
        gi, ge = np.abs(err[interior]) > 2, np.abs(err[edge]) > 2     # NaN (unwrapped-masked) counts as gross
        gi |= np.isnan(err[interior]); ge |= np.isnan(err[edge])
        bits = "" if code is None else str(sum(int((((code ^ gt_code) >> k) & 1)[interior].sum())
                                                for k in range(NBITS)))
        res[sigma, name] = (err, rms(interior), rms(bg))
        print(f"{sigma:6.3f} {name:10s} {rms(interior):10.4f} {gi.sum():6d} / {gi.mean():.4f} "
              f"{rms(edge):10.1f} {ge.sum():6d} / {ge.mean():.4f} {bits:>10s}")

err = res[0.0, "hybrid"][0]
print(f"\nH1 hybrid, noise-free: max |column error| interior {np.abs(err[interior]).max():.2e}")
err, _, rms_bg = res[0.0, "gray_inv"]
pred = np.sqrt(np.mean(gz[bg]**2) / 12) * 1e3
print(f"H2 Gray (inverse), noise-free: max |column error| interior {np.abs(err[interior]).max():.4f}; "
      f"background RMS {rms_bg:.4f} mm vs uniform-quantization {pred:.4f} mm (ratio {rms_bg / pred:.4f})")
print(f"H3 sigma 0.005: Gray (inverse) RMS / hybrid RMS = "
      f"{res[0.005, 'gray_inv'][1] / res[0.005, 'hybrid'][1]:.2f}")
for sigma in (0.0, 0.005, 0.02):
    err = res[sigma, "spatial"][0]
    print(f"H4 spatial sigma {sigma}: background max |error| {np.nanmax(np.abs(err[bg])):.2e}; "
          f"box error min {np.nanmin(err[bx]):.4f} max {np.nanmax(err[bx]):.4f} columns")
for sigma in (0.005, 0.02):
    with np.errstate(divide="ignore"):                # amp0 = 0 on unlit pixels, not used
        pred = np.sqrt(np.mean((gz * T / (2 * np.pi) * sigma / (np.sqrt(2) * amp0))[interior]**2)) * 1e3
    print(f"H7 hybrid sigma {sigma}: RMS {res[sigma, 'hybrid'][1]:.4f} mm vs first-order "
          f"{pred:.4f} mm (ratio {res[sigma, 'hybrid'][1] / pred:.4f})")

Output:

C0a BIN2GRAY(546) = 1100110011; Gray->column round trip exact: True; bits changed between neighbours: [1]
C0b camera samples on a box side face: 0
C0c pixels: interior 45503 (box 10881), edge band 2204, unlit 29093; frac(x_p) on background: mean +0.0839, std 0.2783 (uniform 0, 0.2887)
C0d global threshold t = 0.3326
C0e phase formula error, point samples: 1.08e-14 rad; pixel-area samples (interior): 1.46e-03 rad
C0f unwrap_phase on its docstring ramp: std(unwrapped - truth) = 3.46e-15

seed 0; RMS in mm; gross = |column error| > 2 (count / fraction)
 sigma decoder       RMS int       gross int   RMS edge      gross edge bit errors
 0.000 gray_inv       9.4949      0 / 0.0000       81.4    212 / 0.0962       4633
 0.000 gray_glob    233.6029   4404 / 0.0968      294.7    625 / 0.2836      20991
 0.000 spatial      244.5082  10881 / 0.2391      282.6    856 / 0.3884           
 0.000 hybrid         0.0470      0 / 0.0000      107.4    296 / 0.1343           
 0.005 gray_inv       9.4775      0 / 0.0000       81.9    217 / 0.0985       4535
 0.005 gray_glob    296.2405   4567 / 0.1004      458.1    640 / 0.2904      21080
 0.005 spatial      244.5152  10881 / 0.2391      282.1    856 / 0.3884           
 0.005 hybrid         1.2616      0 / 0.0000      107.6    296 / 0.1343           
 0.020 gray_inv       9.4773      0 / 0.0000       85.5    232 / 0.1053       4562
 0.020 gray_glob         inf   5053 / 0.1110      365.0    660 / 0.2995      21403
 0.020 spatial      244.6038  10881 / 0.2391      281.8    856 / 0.3884           
 0.020 hybrid         5.0749      0 / 0.0000      110.3    287 / 0.1302           

H1 hybrid, noise-free: max |column error| interior 3.73e-03
H2 Gray (inverse), noise-free: max |column error| interior 0.6200; background RMS 10.8851 mm vs uniform-quantization 9.8817 mm (ratio 1.1015)
H3 sigma 0.005: Gray (inverse) RMS / hybrid RMS = 7.51
H4 spatial sigma 0.0: background max |error| 3.73e-03; box error min 15.9981 max 16.0025 columns
H4 spatial sigma 0.005: background max |error| 3.02e-01; box error min 15.7757 max 16.2325 columns
H4 spatial sigma 0.02: background max |error| 1.38e+00; box error min 15.2547 max 17.1343 columns
H7 hybrid sigma 0.005: RMS 1.2616 mm vs first-order 1.2692 mm (ratio 0.9940)
H7 hybrid sigma 0.02: RMS 5.0749 mm vs first-order 5.0768 mm (ratio 0.9996)

All six checks held: the Gray conversion reproduces Lanman and Taubin’s code for column 546 and changes one bit between neighbours, no camera sample lands on a side of the box, the pixel counts and the spread of fractional columns on the background are within the planned bounds (mean within ±0.10 of 0, standard deviation within 0.02 of the uniform 0.2887), the global threshold is 0.3326, the phase formula is exact on point samples and within the planned $5 \times 10^{-3}$ rad on area samples, and unwrap_phase recovers its docstring surface to rounding error. The whole script runs in under a second (NumPy 2.3.5, scikit-image 0.25.2).

The hybrid is the only decoder that is both absolute and fine. Without noise its interior depth RMS is a small fraction of a millimetre, far below that of the Gray code alone, which is limited to whole columns. Its residual is not zero: the largest interior column error is $3.7 \times 10^{-3}$, and the area-sampled phase is off by up to $1.5 \times 10^{-3}$ rad while the same formula on point samples is exact (check C0e). The reason is the construction: the 16 samples of a pixel carry different albedos, so the pixel averages sinusoids with unequal weights, and the phase of that weighted sum is not the phase at the pixel’s centre.

The Gray code’s error is its quantization. Every interior pixel decodes to within 0.62 columns of the truth, never a gross error, and its depth RMS on the background is 10.9 mm against 9.9 mm predicted by treating the column error as uniform over ±0.5 column ($\sqrt{\overline{(Z^2/fb)^2}/12}$, the stereo depth error of a column error with standard deviation $1/\sqrt{12}$). Two departures from that model show in the output: the fractional columns on the background are not quite uniform (check C0c), and the largest error exceeds half a column. In our reading, a pixel that straddles a stripe boundary is decoded by whichever of plane and inverse is brighter, that is by an albedo-weighted majority of its area rather than by its centre. On the box face the error is zero by construction: $fb/0.75 = 40$ pixels exactly, so every true column on the box is an integer and all 16 samples of a box pixel fall in that one projector column.

A global threshold fails on dark surfaces. The threshold is 0.3326, while a lit pixel’s value is 1.1 times its albedo, so any surface with albedo below about 0.30 reads every plane as dark, whichever stripe falls on it, and decodes as column 0. In our reading these pixels are the decoder’s gross errors, about a tenth of the interior pixels, and it makes far more bit errors than the inverse comparison at every noise level. At σ = 0.02 at least one interior pixel decodes to a column with zero disparity, so its depth, and the decoder’s RMS, is infinite; such a pixel is already counted as gross. The inverse comparison’s own bit errors, a few thousand at each noise level, never produce a gross error; in our reading they sit at stripe boundaries, where a misread bit moves a Gray code to its neighbour.

Spatial unwrapping fails on the whole box, by exactly one period. The background, connected to the reference pixel, is right. Every one of the 10,881 box interior pixels is off by +16 columns, the fringe period, so the box is reconstructed at $fb/(40 - 16) = 1.25$ m instead of 0.75 m. By the construction, the disparity jumps at the box’s outline from about 30 pixels on the background ($fb/Z$ with $Z$ near 1 m) to 40 on the box, so at a fixed camera column the projector column falls by about 10. That is more than half the 16-column period, so the wrapped phase steps by $-1.25\pi$, which the unwrapper reads as $+0.75\pi$: the step is taken the wrong way, and the error is $+16$ columns. From the wrapped phase alone the two steps look the same; only a code that labels the period, here the Gray code, resolves it, which is why the hybrid has no gross errors anywhere on the interior.

Noise moves the phase, not the code. With σ = 0.005 and 0.02 the Gray code’s RMS barely changes, in our reading because a bit decision has the full contrast between plane and inverse as its margin, while the hybrid’s RMS grows to 1.26 and 5.07 mm. Those values match the first-order prediction $\sqrt{\overline{(Z^2/fb \cdot T/2\pi \cdot \sigma/(\sqrt{2}a'))^2}}$ to within 1%, where $a'$ is each pixel’s fringe amplitude: the phase from four samples has standard deviation $\sigma/(\sqrt{2}a')$, our derivation from the atan2 formula. At σ = 0.02 the hybrid’s RMS is still below the Gray code’s; at σ = 0.005 the Gray code’s is 7.51 times the hybrid’s (H3). Darker pixels have smaller $a'$ and noisier phase, which is why the prediction averages per pixel.

The step is the hard case for every decoder. In the edge band, where pixels mix the box and the background or are partly shadowed, between a tenth and two fifths of the pixels have gross errors even without noise: a mixed pixel averages two surfaces’ patterns, so its bits and its phase describe neither surface. The Gray (inverse) and hybrid decoders treat each pixel independently, and their gross errors stay within the edge band: they have none on the interior.

The planned outcomes were as follows; every check and hypothesis held in this run. Without noise: the hybrid’s interior column error stayed below 0.01 (H1, held); the Gray code’s interior error stayed within 0.75 columns and its background RMS within 15% of the uniform-quantization value (H2, held); spatial unwrapping was right on the background to within 0.01 columns and off by 16 ± 0.01 columns on every box pixel (H4, held); and every decoder had a larger gross-error fraction on the edge band than on the interior (H8, held). At every noise level the global threshold made more bit errors and more interior gross errors than the inverse comparison (H5, held), and the Gray (inverse) and hybrid decoders had no interior gross errors (H6, held). At σ = 0.005 the Gray code’s RMS was at least five times the hybrid’s (H3, held, 7.51), and the hybrid’s RMS at σ = 0.005 and 0.02 was within 15% of the first-order prediction (H7, held). The noisy spatial rows were planned as descriptive: the box stays near one period off (15.25 to 17.13 columns at σ = 0.02).

How the example was planned and changed. The bars of H1, H2, H4, H5, H6 and H8 were set after two noise-free planning scripts had printed the values they test, so their noise-free outcomes illustrate the derivations rather than test them. Five checks as first written failed on the planning scripts’ values, and their bars were then widened. The plan’s own record (its section R) says that H1 “as first written (< 1e-9) would fail on this set-up”, because of the albedo weighting within a pixel described above, and the bar became 0.01; for the same reason C0e, first written as an error below $10^{-9}$ rad on the pixel-area phase, failed and was split into the point-sample check and an area-sample bar of $5 \times 10^{-3}$ rad; and H4, first written with tolerances of $10^{-9}$ and $10^{-6}$ columns, failed and was given 0.01. H2 and the fractional-column check C0c were moved to background pixels, because the box’s columns are integers; C0c, first written as a mean within ±0.02 and a standard deviation within ±0.01 of 0.2887, failed on all interior pixels and on the background, and was widened to ±0.10 and ±0.02; H2’s RMS bar, first 10%, failed on the background and was widened to 15%. The first planning script’s reference pixel (row 220) lay outside the projector’s field, so it was moved to row 200, column 290. The noisy outcomes (H3, H5 and H6 with noise, H7) were not computed by any planning script, although the first-order predictions that H3’s bar and H7 use were printed by one. The example departs from the batch plan in seven ways: a slanted rather than fronto-parallel background (so that columns are not integers); pixel-area sampling (so that the step mixes surfaces); a reference pixel for spatial unwrapping; gross errors defined as more than two columns; the edge-band split and the H7 and H8 measures; one image-wide global threshold from the mean white and black images; and ideal continuous fringes. After a pilot run with seed 1, the only change was to silence a divide-by-zero warning in the H7 prediction, which changes no printed value; the run shown uses seed 0. Every check and hypothesis held in both runs. The infinite RMS of the global threshold appeared at σ = 0.005 in the pilot and at σ = 0.02 here; the noisy numbers differ slightly between the two seeds (H3’s ratio was 7.47 in the pilot, H7’s ratios 1.0002 and 1.0027, and the box’s spatial-unwrapping error at σ = 0.02 ranged from 15.23 to 17.24 columns).

Strengths

• No search in natural texture. The code labels every lit pixel, so textureless surfaces, where passive stereo fails, are measured as easily as textured ones. Fanello et al. (2016) attribute the popularity of structured-light sensors to their robustness to untextured scenes and to multipath.
• Dense and precise for static scenes. Phase shifting gives one sub-column measurement per camera pixel (Moreno et al., 2015), and Lanman and Taubin (2009) note that temporal codes benefit from redundancy, which reduces reconstruction artefacts. Benchmarks use structured light for reference geometry: Middlebury’s stereo ground truth and DTU’s reference scans, as the Middlebury and DTU articles describe.
• Simple, cheap hardware. A projector and a camera, with no moving parts (Moreno et al., 2015); single-shot sensors need only a fixed pattern generator.
• Little computation for temporal codes. Decoding is a per-pixel lookup (Fanello et al., 2016), as the example’s decoders show.

Limitations

• Occlusion and shadows. A point must be visible to the camera and lit by the projector, and the separation between them, which depth precision needs, leaves occluded regions unmeasured (Lanman & Taubin, 2009, ch. 1.1.2 and 5.5).
• Motion. Temporal codes assume the scene stays still for the whole sequence; Fanello et al. (2016) note that even fast hardware shows artefacts under fast motion, and Lanman and Taubin call both slit scanners and structured lighting ill-suited to dynamic scenes.
• Difficult materials and global illumination. Lanman and Taubin note that strongly reflective or translucent objects often violate the assumptions of active scanners. Interreflections and subsurface scattering add light that corrupts both bits and phases, which motivates the high-frequency codes described by Moreno et al. (2015).
• Ambient light and other sensors. Zhang et al. (2018) describe structured light as vulnerable to ambient illumination and to interference between several devices.
• Depth discontinuities. Spatial phase unwrapping assumes a smooth surface (Moreno et al., 2015); the example shows the failure, a whole surface off by one period. Pixels on edges mix two surfaces in every scheme.
• Range and precision. Depth error grows as $Z^2/(fb)$; projected patterns also fade with distance, which ActiveStereoNet’s authors compensate for in active stereo (Zhang et al., 2018). Fanello et al. (2016) note that temporal systems are range-limited by the precision of the coding scheme. Projector and camera defocus are among the decoding error sources Lanman and Taubin list.

Typical Applications

• 3D scanning of objects for models, measurement and fabrication, the setting of Lanman and Taubin’s course and of Moreno et al. (2015), who mention 3D printing as a motivation.
• Reference geometry for benchmarks such as Middlebury and DTU, against which stereo matching and multi-view stereo are scored.
• Consumer depth cameras. The first Kinect’s structured-light depth fed datasets such as NYU Depth v2 and real-time reconstruction such as KinectFusion. Fanello et al. (2016) present a 375 Hz structured-light prototype for tracking problems.
• Industrial inspection, where laser-line profilers and fixed-pattern cameras measure height on production lines (industrial visual inspection).

Comparison with Other Sensors

Sensor	Correspondence	Main strengths	Main weaknesses
Structured light	Decoded from a projected code	Dense, works on textureless surfaces, sub-column precision with phase	Shadows beside edges, motion (temporal codes), ambient light, interreflections
Passive stereo	Matched in natural texture	No emitter; metric depth from a known baseline	Fails on textureless or repetitive surfaces
Active stereo	Matched in projected texture	Pattern need not be known (Keselman et al., 2017)	Still a matching problem; pattern fades with distance
LiDAR	None: timing of the echo	Range error roughly independent of distance	Sparse points, cost

Fanello et al. (2016) contrast structured light with time-of-flight cameras, which suffer from multipath errors, and with passive stereo, which can fail in textureless regions. In 2009 Lanman and Taubin judged the depth resolution and accuracy of time-of-flight cameras, for static scenes, to be below those of slit scanners and structured lighting. The depth estimation article places structured light among the other active and passive sources of depth.