Stereo Camera

A stereo camera is a depth sensor built from two imagers, sometimes more, mounted rigidly side by side, exposed together and calibrated as a pair, so that the depth of each matched pixel follows by triangulation. The geometry it relies on, depth from disparity $Z = fB/d$ and an error that grows with the square of depth, is the subject of stereo vision, and finding the disparity at every pixel is stereo matching. This article is about the device that makes those two steps work in practice: the choice and mounting of the imagers, their synchronization and factory calibration, the matching hardware that many depth cameras carry, the data they output, and the device-level errors that a user meets. Products range from synchronized camera pairs that leave matching to the host, such as the pair used to record EuRoC MAV, to self-contained stereo depth cameras, also called active stereo cameras when they add a pattern projector, such as Intel’s RealSense R200 and D400 families.

How It Works

A stereo depth camera chains the steps that stereo vision describes, each fixed in the device:

Two matched imagers. The two cameras are identical so that the same scene point looks alike in both. In the RealSense R200, described by Keselman, Iselin Woodfill, Grunnet-Jepsen and Bhowmik (2017), the left and right cameras are 10-bit monochrome global-shutter VGA (640 × 480) CMOS sensors, nominally 70 mm apart, with a nominal field of view of 60° × 45° (horizontal × vertical) and a filter that blocks visible light below about 840 nm, so that they see the infrared projector (§3.3.1). In our reading, the choice suits matching: a monochrome sensor has no colour filter array to interpolate, and a global shutter exposes all rows of an image over the same interval.
Rigid mounting and factory calibration. Each R200 unit is calibrated in the factory to a sub-pixel accurate camera model for all three of its cameras, and a stiffener with mechanical tolerance guidelines is meant to keep the calibration over the unit’s lifetime (§3, §3.3). The lens model is a modified Brown–Conrady distortion model with three radial and two tangential terms (§3.3.1).
Rectification on the device. Undistortion and rectification of the left–right pair are done in hardware, so that the matcher searches a single scan line; the R200’s colour camera is undistorted on the host instead (§3, §3.3.1).
Matching on the device. The R200’s imaging processor, called DS4, computes a 7 × 7 census transform, searches a fixed range of 64 disparities, aggregates costs with a 7 × 7 box filter, picks the best candidate, refines it to up to 5 bits of sub-pixel precision and then masks out doubtful matches; it does no smoothing or interpolation, so every returned value is a real match (§3.1–§3.2). The authors give the whole depth pipeline, projector included, about one watt, and a latency of the order of tens of scan lines (§3.1). For the successor RS400 family they describe a matcher extended with techniques that move past local-only matching, naming semi-global matching as one published example, with matching costs beyond census and a larger disparity search range, but report no measurements, since no units were commercially available at publication (§5).
Depth output. The device sends disparity or depth per pixel over USB, together with the infrared images and, in RGB-D units, a colour stream.

Texture projectors

Passive stereo has nothing to match on a blank wall. Stereo depth cameras such as the RealSense R200 and D400 therefore carry an infrared projector that paints texture on the scene. Keselman et al. call this “unstructured light”, a technique they trace to stereo systems of 1984: the cameras need no knowledge of the pattern, which must only be densely textured, photometrically consistent, and free of repetition along the matching direction within the search range (§2.2). This is the difference from structured light, which labels each pixel with a projected code and can triangulate against the projector itself, as stereo vision outlines; with an unstructured pattern the projector only adds texture, and the depth still comes from matching the two cameras’ images. The R200’s projector is a Class 1 laser projecting a fixed high-contrast random-dot pattern, which can vary between units and drift slightly during use (§3.3.2). Because the cameras block visible light, the left and right images stay dark without the projector, incandescent bulbs or sunlight, and the same design works both in 0 lux and in broad daylight (§3.3.1). The librealsense documentation for the D400 series likewise lists an on-board texture projector “for tough lighting conditions” and notes that such a projector can also serve as a light source at night or in the dark (Depth from Stereo, v2.59.1).

Data It Produces

A stereo depth camera produces some or all of:

Disparity or depth maps in the frame of the reference imager, one value per pixel, with invalid pixels marked. The R200 marks rejected pixels with zero and provides no confidence map (§4.2.7).
The rectified infrared or grey images that were matched, which are themselves useful for visual odometry.
A colour image from a third camera, in RGB-D units, which must be registered to the depth map. In the R200 the colour camera is a 1920 × 1080 Bayer-pattern, rolling-shutter sensor whose stream is synchronized with the stereo pair at start of frame (§3.3.1).
Calibration data: focal length, principal point, baseline and the extrinsics of every stream.

GenICam’s Standard Features Naming Convention (SFNC 2.8) gives machine-vision cameras a vocabulary for this output. It lists stereo cameras among area-scan 3D cameras that acquire a range map in a single shot (Chapter 21), and its DisparityC output mode delivers a disparity image whose distance unit is the pixel. The device reports the parameters needed to turn it into points, Scan3dFocalLength in pixels, Scan3dBaseline in metres and the principal point Scan3dPrincipalPointU/V, and the standard gives the reconstruction

X = \frac{(u - U)\,B}{D},\qquad Y = \frac{(v - V)\,B}{D},\qquad Z = \frac{f\,B}{D},

where $D$ is the disparity at column $u$ and row $v$ and the coordinates come out in the unit of the baseline; a disparity of 0 means a point at infinity (Section 21.2.3). SFNC also allows invalid data to be marked by a confidence image or by an out-of-range value such as 0, and discourages NaN (Chapter 21). The disparity article covers sign conventions, principal-point offsets and the encodings that benchmarks use, and the depth map article covers depth storage, scale factors and back-projection.

Disparity is the natural output of a stereo device, because its error is roughly constant: Keselman et al. measured the R200’s RMS error to be roughly constant in disparity, linear as a fraction of distance and quadratic in millimetres (their Figure 4).

Characteristics

Working range

The fixed disparity search range sets a hard minimum distance, $Z_\text{min} = fB/d_\text{max}$ : anything nearer has a disparity larger than the matcher searches. For the R200 Keselman et al. give this limit as around half a metre at 480 × 360 and roughly a third of a metre at 320 × 240, since at lower resolution the fixed 64 disparities cover a larger fraction of the field of view (§4.2.1). The far end has no hard limit. In passive, well-textured conditions a disparity of one pixel corresponds to roughly 30 m at 480 × 360; indoors with the projector as the only texture the signal-to-noise ratio limits the range, and the paper’s measure, the greatest distance at which a white wall returns more than 95% of its measurements at the centre of the field of view, is roughly 6 m at 30 Hz. Higher frame rates shorten the exposure and the range, roughly with the square root of frame time in photon-limited conditions (§4.2.1, Figure 3).

Depth error

Keselman et al. write the depth error as

|\epsilon_z| = \frac{z^2}{fB}\,|\epsilon_d|, \qquad \epsilon_{\%\,\text{of}\,z} = \frac{\epsilon_\text{mm}}{z} = \epsilon_d \frac{z}{fB}

(their Equations 2 and 3), the law that stereo vision derives. The disparity error $\epsilon_d$ is treated as a constant of the system. The text puts it at about 0.1 pixel for passive systems, “from measurements in Section 4.2” (§2.1), while Table 5, the measured performance of one R200 unit at 480 × 360 and 30 Hz, prints 0.08 pixel for a static scene and 0.05 pixel for a moving one; the paper reads these numbers as evidence that sub-pixel interpolation matters (§4.2.2). In an active system, the paper notes, the projector’s light falls off with distance, so $\epsilon_d$ is only approximately constant until imaging noise overwhelms the projector (§2.1, footnote 1).

Table 5 also lists an RMS error of 2.1 mm at 1 m and 8.0 mm at 2 m, an RMS of 1% of distance at 4.1 m, a minimum distance of 0.53 m, an absolute calibration error of 5 mm at 1 m, a dynamic range of 40× and an x–y detectable size of 5 pixels or 1% of distance (read from a rendered image of page 5 of the CVF open-access copy, since the PDF’s text layer scrambles the table). The example below checks these entries against Equations 1–3 with the device’s nominal geometry.

Other device effects

Laser speckle. A laser projector’s speckle looks different from the two viewpoints, which breaks photometric consistency and can create false matches or add correlation noise; the R200 uses wavelength and bandwidth diversity to reduce it. Speckle fades when the camera or the target moves: on a target spinning at 12 rpm the RMS error fell by nearly 40% at a 33 ms exposure, and handheld measurements were better than tripod ones (§3.3.2, §4.2.6, Figure 5).
Fixed-pattern sub-pixel noise. With static input images the sub-pixel errors are also static: averaging 1,000 frames of a static scene still leaves a “bumpy” surface, which disappears when the projector moves during the averaging (§4.2.6). Performance also ripples as the true disparity moves between integer and half-pixel values (§4.2.6).
Edges. With a fixed block size the R200 merges gaps between edges narrower than about 5 pixels, about 1% of distance at 480 × 360 (§4.2.3).
Dynamic range. From the darkest to the brightest condition at which 95% of the data is returned, Keselman et al. measured roughly 40× on most R200 units (§4.2.5).
Stereo bias. Taking the reciprocal of a noisy disparity biases depth. The paper gives a planar derivation of the bias, $\epsilon = -d/(2d^2 - 1)$ for $d > \sqrt{2}$ , crediting the analysis of stereo bias to Freundlich, Zavlanos and Mordohai (2015), and finds it dominated by imaging noise below 10 m on the R200 (§4.2.4).
Range falls off with albedo and angle. For indoor scenes with little ambient infrared the paper extrapolates the 95%-fill range $r_{\rho=95\%}$ of a white wall to other targets as $r_\text{expected} \approx r_{\rho=95\%}\sqrt{\cos\theta_\text{target}\,\alpha\,\cos^7\theta_\text{FOV}}$ , where $\alpha$ is the albedo, the seventh power combines $\cos^4$ lens vignetting with a roughly $\cos^3$ projector fall-off, and the square root comes from the inverse-square law (§4.2.8; equation read from the rendered page). The paper says this matches its measurements, in which dark, tilted floors in the corners of the field of view are not visible at long distances.

Calibration

A stereo camera’s metric output depends entirely on its calibration: the intrinsics and distortion of each imager and their relative pose, estimated as in camera calibration and turned into rectifying transformations as in stereo rectification. Factory-calibrated units ship these parameters and apply them on the device. Calibration errors do not add noise; they bias every measurement the same way, frame after frame.

Keselman et al. report that imperfect fitting of the undistortion and rectification model can make an R200 return depth with a consistent offset, scale- or distance-dependent according to which part of the model is wrong, and that over several units the error was zero-mean and typically small, on the order of 5 mm at 1 m (§4.2.4). The two kinds of rotation error act differently. A relative pitch or roll moves rows apart, so true matches leave the scan line the matcher searches; the stereo rectification article measures the vertical disparity this creates and how to check for it. A relative yaw, a rotation about the vertical axis, keeps the rows aligned and shifts every disparity sideways instead, so matching still succeeds and nothing in the output reveals the error: depth is simply wrong, by an amount that grows with distance. The example below measures this.

Synchronization

Two images give a correct disparity only if they show the scene at the same instant. Synchronization therefore belongs to the device as much as calibration does:

Triggers and clocks. Machine-vision cameras combined into a rig have standard controls for this in SFNC: a trigger whose TriggerSource is “the internal signal or physical input Line” that starts acquisition, and the Precision Time Protocol (PtpEnable; PtpOffsetFromMaster reports the offset from the master clock in nanoseconds), which timestamps frames against a common clock (SFNC 2.8, feature summary tables). In our reading, a hardware trigger line wired to both cameras aligns exposures, while PTP aligns their timestamps.
Exposure alignment. With automatic exposure the two cameras may integrate for different times. The EuRoC MAV data used a stereo pair of global-shutter WVGA monochrome cameras whose mid-exposure times were aligned, so that different shutter times would not break the synchronization, as its article describes from the dataset’s former wiki page.
Triggering from another sensor. On the KITTI platform the rotating laser scanner triggers the cameras at 10 Hz, so the stereo images and scans are synchronized (as the KITTI article describes).
Mixed shutters and unsynchronized streams. The R200’s colour camera has a rolling shutter and is synchronized with the infrared pair only at the start of the frame (§3.3.1), so its rows are exposed later than the infrared images by a growing delay; the image sensor article explains rolling-shutter timing. For contrast, the Kinect colour and depth streams of TUM RGB-D were not synchronized: its article reports that depth images arrive about 20 ms after the colour images on average.

The example below derives what an offset between the two exposures does to stereo depth.

Processing the Data

Stereo depth needs cleaning, and the device and its software share the work.

Invalidation on the device. The R200 tests every match with a set of interest operators: minimum and maximum matching score, a left–right consistency threshold in 1/32 pixel, a neighbour threshold, a second-peak threshold, a texture threshold and a median threshold. A pixel failing any of them is set to zero (§4.2.7). Presets, near-Pareto-optimal threshold sets, trade density for accuracy: the paper’s Table 6 lists false-positive rates on a scene below the minimum distance of 91.3% with the “Off” preset and 0.5% with the “High” preset, and the 95%-fill range falls from 7.0 m to 4.2 m.
Filtering in disparity space. Keselman et al. report as useful removing the sparse noise of a local matcher with median or speckle (small connected component) filters, inpainting with an edge-preserving filter guided by the colour image, and applying such filters in disparity space, where errors are roughly constant, rather than in depth (§4.2.9). To flatten the “bumpy” sub-pixel noise they suggest quantizing disparity to a step $\epsilon_q$ with $\epsilon_d < \epsilon_q \le 1$ , typically $4\epsilon_d$ (§4.2.9).
librealsense filters. The scheme that librealsense recommends, and uses in its tools and demos, is a chain of decimation, conversion from depth to disparity, a spatial edge-preserving filter based on the domain transform, a temporal filter, conversion back to depth and hole filling, the two conversions applying to stereo-based cameras only. The temporal filter can smear moving objects and suits static scenes (Post-Processing Filters, v2.59.1).

For dense matching beyond the device, semi-global matching and learned matchers run on the host from the rectified images; the cost volume article describes how.

Device Errors in Numbers

The example has three parts. Part A recomputes the R200’s printed specifications from its nominal geometry: a 60° horizontal field of view, a 70 mm baseline and a 64-disparity search. Part B simulates a synchronization offset: the right image is exposed $\Delta t$ after the left while the scene moves sideways at $v = \pm 1$ m/s, and, as a second case, while the rig turns at 30°/s. Part C simulates a yaw miscalibration of the right camera. Parts B and C use an ideal rectified pair with the R200’s geometry at 480 pixels width ( $f = 415.69$ px, $B = 0.070$ m, principal point at column 240). Points lie on fronto-parallel planes at depths of 0.5–16 m, at eight columns of the left image. The image positions are exact projections, so no matching is involved, and depth is reconstructed with the SFNC formulas above.

Conventions: $x$ points right, $y$ down and $z$ forward; depth is $Z$ along the optical axis, not radial range; disparity is $d = x_L - x_R$ with the left image as reference; a yaw of $+\delta$ about $+y$ turns the optical axis to the right; a rotation $R$ gives camera coordinates $R^\top(\mathbf{P} - \mathbf{C})$ for a camera at $\mathbf{C}$ . Part A assumes that the 480 × 360 and 320 × 240 modes keep the nominal 60° field, as the paper’s explanation of the shorter minimum distance at lower resolution suggests.

Two derivations, ours, set the expectations. In Part B a point at $(X, 0, Z)$ moving sideways is seen by the right camera at $X + v\Delta t$ , so

d = \frac{f\,(B - v\Delta t)}{Z}, \qquad \frac{d}{d_\text{true}} - 1 = -\frac{v\Delta t}{B}, \qquad \frac{Z_\text{est}}{Z} - 1 = \frac{v\Delta t}{B - v\Delta t} \approx \frac{v\Delta t}{B},

exactly, for every point: the whole scene is scaled about the left camera centre by $B/(B - v\Delta t)$ , and the relative depth error does not depend on depth. The batch plan for this article stated the error as $v\Delta t/B$ ; that is only the first-order form, and the relative error of the first-order form is itself $|v\Delta t/B|$ . In Part C a point at angle $\theta$ from the right camera’s axis is seen at $f\tan(\theta - \delta)$ instead of $f\tan\theta$ , so the disparity grows by $\Delta = f[\tan\theta - \tan(\theta - \delta)] \approx f\delta(1 + \tan^2\theta)$ , which is $f\tan\delta$ on the axis, and the relative depth error is

\frac{Z_\text{est}}{Z} - 1 = -\frac{\Delta}{d_\text{true} + \Delta} \approx -\frac{Z\,\tan\delta}{B},

which grows linearly with depth (the absolute error with its square).

import os
for var in ("OPENBLAS_NUM_THREADS", "OMP_NUM_THREADS", "MKL_NUM_THREADS"):
    os.environ[var] = "1"              # single-threaded BLAS, set before importing NumPy
import numpy as np
from scipy.spatial.transform import Rotation

# ---- Part A: Intel RealSense R200 numbers from its nominal geometry (Keselman et al., 2017)
B = 0.070                  # baseline (m), "placed nominally 70mm apart"
HFOV = 60.0                # nominal horizontal field of view (degrees)
NDISP = 64                 # fixed disparity search range of the DS4 matcher

def focal_px(width):       # focal length in pixels, assuming each mode keeps the 60 deg field
    return (width / 2) / np.tan(np.radians(HFOV / 2))

def dev(x, ref):           # signed deviation from a printed value, in percent
    return 100 * (x / ref - 1)

f = focal_px(480)          # the 480 x 360 mode used for Table 5
fB = f * B
print(f"f = {focal_px(640):.3f} / {f:.3f} / {focal_px(320):.3f} px at 640 / 480 / 320 px width")
zmin480, zmin320 = fB / NDISP, focal_px(320) * B / NDISP
print(f"z_min = fB/64 at 480: {zmin480:.3f} m  ({dev(zmin480, 0.5):+.1f}% vs 1/2 m in text, "
      f"{dev(zmin480, 0.53):+.1f}% vs 0.53 m in Table 5)")
print(f"z_min = fB/64 at 320: {zmin320:.3f} m  ({dev(zmin320, 1/3):+.1f}% vs 1/3 m in text)")
print(f"z at d = 1 px (480):  {fB:.3f} m  ({dev(fB, 30.0):+.1f}% vs 30 m in text)")
eps_d = 0.08               # Table 5, disparity RMS (static)
for z, printed in [(1.0, 2.1), (2.0, 8.0)]:
    rms = eps_d * z**2 / fB * 1e3
    print(f"Eq. 2 RMS at {z:.0f} m with eps_d = 0.08: {rms:.2f} mm ({dev(rms, printed):+.1f}% vs "
          f"{printed} mm in Table 5); eps_d implied by Table 5: {printed * 1e-3 * fB / z**2:.3f} px")
print(f"Table 5 ratio RMS(2 m)/RMS(1 m) = {8.0 / 2.1:.2f} (Eq. 2: 4, {dev(8.0 / 2.1, 4.0):+.1f}%)")
r1 = 0.01 * fB / eps_d
print(f"Eq. 3 1% RMS range: {r1:.3f} m ({dev(r1, 4.1):+.1f}% vs 4.1 m in Table 5)")
print(f"5 px X-Y size = {100 * 5 / f:.2f}% of distance ({dev(5 / f, 0.01):+.1f}% vs 1%)")
dd_cal = fB * 0.005 / 1.0**2
print(f"5 mm at 1 m = {dd_cal:.4f} px of disparity = a yaw of {np.degrees(np.arctan(dd_cal / f)):.4f} deg")
r_exp = 6.0 * np.sqrt(np.cos(np.radians(60)) * 0.2 * np.cos(np.radians(15))**7)
print(f"Sec. 4.2.8 example: r_expected = {r_exp:.3f} m ({dev(r_exp, 1.4):+.1f}% vs 1.4 m in text)")

# ---- Common set-up for Parts B and C: an ideal rectified pair with the R200 geometry
W, cx = 480, 240.0         # image width and principal-point column (px)
Zs = np.array([0.5, 1, 2, 4, 8, 16.0])                  # fronto-parallel planes (m)
us = np.array([100, 150, 200, 240, 300, 350, 400, 450.0])  # left-image columns of the points
centre = us == cx
CL, CR = np.zeros(3), np.array([B, 0.0, 0.0])          # camera centres; x right, y down, z forward

def Ry(a):                 # rotation by angle a (rad) about +y: turns the optical axis toward +x
    c, s = np.cos(a), np.sin(a)
    return np.array([[c, 0, s], [0, 1, 0], [-s, 0, c]])

def project(P, R, C):      # column of points P (N x 3) in a camera at C with orientation R
    q = (P - C) @ R        # rows are R^T (P - C)
    return f * q[:, 0] / q[:, 2] + cx

def plane(Z):              # points on the plane at depth Z seen at columns us of the left image
    X = (us - cx) * Z / f
    return np.c_[X, np.zeros_like(X), np.full_like(X, Z)]

def reconstruct(xl, d):    # SFNC DisparityC: X = (u - U) B / D, Z = f B / D
    return (xl - cx) * B / d, f * B / d

for a in [0.01, 0.02, 0.03, 0.05, -0.02]:
    assert np.abs(Ry(np.radians(a)) - Rotation.from_euler("y", a, degrees=True).as_matrix()).max() < 1e-15
print("\nCheck: Ry matches scipy Rotation.from_euler('y') to 1e-15 for every angle used")
err, out = 0.0, 0
for Z in Zs:
    P = plane(Z)
    xl, xr = project(P, np.eye(3), CL), project(P, np.eye(3), CR)
    Xe, Ze = reconstruct(xl, xl - xr)
    err = max(err, np.abs(Ze / Z - 1).max(), np.abs(Xe - P[:, 0]).max() / Z)
    out += np.sum((xr < 0) | (xr >= W))
print(f"Check: synchronized, calibrated rig: max relative X/Z error {err:.1e}, points outside right image {out}")

# ---- Part B: the right image is exposed dt after the left
print("\nB. Synchronization offset, scene moving sideways at v (all 48 points)")
print(" v(m/s)  dt(ms)  d/d_true-1 (min, max)     e_Z % (min, max)        exact %   1st order %  "
      "max|e_Z-exact|  max|d/d_true-1+v dt/B|  1st/exact-1 %  outside")
for v, dt in [(1, 0.0), (1, 1e-4), (-1, 1e-4), (1, 1e-3), (-1, 1e-3), (1, 5e-3), (-1, 5e-3)]:
    ed, ez, out = [], [], 0
    for Z in Zs:
        P = plane(Z)
        xl = project(P, np.eye(3), CL)
        xr = project(P + [v * dt, 0, 0], np.eye(3), CR)   # the scene has moved by v dt
        d = xl - xr
        ed.append(d / (f * B / Z) - 1)
        ez.append(reconstruct(xl, d)[1] / Z - 1)
        out += np.sum((xr < 0) | (xr >= W))
    ed, ez = np.concatenate(ed), np.concatenate(ez)
    exact, first = v * dt / (B - v * dt), v * dt / B
    rel1 = 100 * (first / exact - 1) if dt else 0.0
    print(f"  {v:+d}   {1e3 * dt:5.1f}   {ed.min():+.6f} {ed.max():+.6f}   {100 * ez.min():+9.4f} "
          f"{100 * ez.max():+9.4f}   {100 * exact:+8.4f}   {100 * first:+8.4f}      {np.abs(ez - exact).max():.1e}"
          f"        {np.abs(ed + first).max():.1e}            {rel1:+6.2f}      {out}")

# The rig turns at 30 deg/s while the right exposure lags by 1 ms (rotation about the left centre),
# against a right camera that is permanently yawed by the same 0.03 deg (Part C).
th = np.radians(30.0) * 1e-3
print("\nB. Rig yawing at 30 deg/s, dt = 1 ms, centre column; vs a fixed 0.03 deg yaw error")
print("  Z (m)   e_Z rotating %   e_Z yawed %   rel. difference %")
for Z in Zs:
    P = plane(Z)[centre]
    xl = project(P, np.eye(3), CL)
    e_rot = reconstruct(xl, xl - project(P, Ry(th), Ry(th) @ CR))[1][0] / Z - 1
    e_yaw = reconstruct(xl, xl - project(P, Ry(th), CR))[1][0] / Z - 1
    print(f"  {Z:5.1f}   {100 * e_rot:+11.4f}   {100 * e_yaw:+10.4f}   {100 * abs(e_rot / e_yaw - 1):9.3f}")

# ---- Part C: the right camera is yawed by delta, but depth uses the factory calibration
print("\nC. Yaw miscalibration, centre column")
print(" delta(deg)  offset/(f tan delta) at Z = 0.5 ... 16 m        e_Z % at Z = 0.5 ... 16 m"
      "                          err at 1 m (mm)")
for a in [0.01, 0.02, 0.03, 0.05, -0.02]:
    dl, ratio, ez = np.radians(a), [], []
    for Z in Zs:
        P = plane(Z)[centre]
        xl = project(P, np.eye(3), CL)
        d = xl - project(P, Ry(dl), CR)
        ratio.append((d[0] - f * B / Z) / (f * np.tan(dl)))
        ez.append(reconstruct(xl, d)[1][0] / Z - 1)
    ez = np.array(ez)
    print(f"  {a:+.2f}   " + " ".join(f"{r:.4f}" for r in ratio) + "   "
          + " ".join(f"{100 * e:+8.4f}" for e in ez) + f"   {1e3 * ez[1] * Zs[1]:+7.3f}")
    print(f"         first order -Z tan(delta)/B, %: " + " ".join(f"{-100 * Z * np.tan(dl) / B:+8.4f}" for Z in Zs)
          + f"   |e_Z(16 m)/e_Z(0.5 m)| = {ez[-1] / ez[0]:.2f}")

# Across the wall at 2 m: the offset grows toward the image edges as 1 + tan^2(theta)
dl, Z = np.radians(0.02), 2.0
P = plane(Z)
xr0 = project(P, np.eye(3), CR)                     # where the point should be
off = xr0 - project(P, Ry(dl), CR)                  # disparity offset from the yaw
t = (xr0 - cx) / f                                  # tan(theta) in the right image
pred = (1 + t**2) / (1 + t[centre]**2)
xl = project(P, np.eye(3), CL)
ez = reconstruct(xl, xl - xr0 + off)[1] / Z - 1
print("\nC. Wall at 2 m, delta = +0.02 deg: offset relative to the centre column")
print("  left column      " + " ".join(f"{u:7.0f}" for u in us))
print("  offset ratio     " + " ".join(f"{r:7.4f}" for r in off / off[centre]))
print("  1 + tan^2 (norm.)" + " ".join(f"{r:7.4f}" for r in pred))
print(f"  max relative difference {100 * np.abs(off / off[centre] / pred - 1).max():.4f}%; "
      f"e_Z across the wall {100 * ez.min():+.4f}% to {100 * ez.max():+.4f}%")

Output:

f = 554.256 / 415.692 / 277.128 px at 640 / 480 / 320 px width
z_min = fB/64 at 480: 0.455 m  (-9.1% vs 1/2 m in text, -14.2% vs 0.53 m in Table 5)
z_min = fB/64 at 320: 0.303 m  (-9.1% vs 1/3 m in text)
z at d = 1 px (480):  29.098 m  (-3.0% vs 30 m in text)
Eq. 2 RMS at 1 m with eps_d = 0.08: 2.75 mm (+30.9% vs 2.1 mm in Table 5); eps_d implied by Table 5: 0.061 px
Eq. 2 RMS at 2 m with eps_d = 0.08: 11.00 mm (+37.5% vs 8.0 mm in Table 5); eps_d implied by Table 5: 0.058 px
Table 5 ratio RMS(2 m)/RMS(1 m) = 3.81 (Eq. 2: 4, -4.8%)
Eq. 3 1% RMS range: 3.637 m (-11.3% vs 4.1 m in Table 5)
5 px X-Y size = 1.20% of distance (+20.3% vs 1%)
5 mm at 1 m = 0.1455 px of disparity = a yaw of 0.0201 deg
Sec. 4.2.8 example: r_expected = 1.681 m (+20.0% vs 1.4 m in text)

Check: Ry matches scipy Rotation.from_euler('y') to 1e-15 for every angle used
Check: synchronized, calibrated rig: max relative X/Z error 2.2e-14, points outside right image 0

B. Synchronization offset, scene moving sideways at v (all 48 points)
 v(m/s)  dt(ms)  d/d_true-1 (min, max)     e_Z % (min, max)        exact %   1st order %  max|e_Z-exact|  max|d/d_true-1+v dt/B|  1st/exact-1 %  outside
  +1     0.0   -0.000000 +0.000000     -0.0000   +0.0000    +0.0000    +0.0000      2.2e-14        2.2e-14             +0.00      0
  +1     0.1   -0.001429 -0.001429     +0.1431   +0.1431    +0.1431    +0.1429      1.9e-14        1.9e-14             -0.14      0
  -1     0.1   +0.001429 +0.001429     -0.1427   -0.1427    -0.1427    -0.1429      3.1e-14        3.1e-14             +0.14      0
  +1     1.0   -0.014286 -0.014286     +1.4493   +1.4493    +1.4493    +1.4286      2.2e-14        2.1e-14             -1.43      0
  -1     1.0   +0.014286 +0.014286     -1.4085   -1.4085    -1.4085    -1.4286      9.2e-15        9.4e-15             +1.43      0
  +1     5.0   -0.071429 -0.071429     +7.6923   +7.6923    +7.6923    +7.1429      1.3e-14        1.1e-14             -7.14      0
  -1     5.0   +0.071429 +0.071429     -6.6667   -6.6667    -6.6667    -7.1429      2.0e-14        2.3e-14             +7.14      0

B. Rig yawing at 30 deg/s, dt = 1 ms, centre column; vs a fixed 0.03 deg yaw error
  Z (m)   e_Z rotating %   e_Z yawed %   rel. difference %
    0.5       -0.3726      -0.3799       1.919
    1.0       -0.7425      -0.7461       0.486
    2.0       -1.4740      -1.4758       0.121
    4.0       -2.9051      -2.9060       0.030
    8.0       -5.6461      -5.6466       0.007
   16.0      -10.6888     -10.6890       0.002

C. Yaw miscalibration, centre column
 delta(deg)  offset/(f tan delta) at Z = 0.5 ... 16 m        e_Z % at Z = 0.5 ... 16 m                          err at 1 m (mm)
  +0.01   1.0196 1.0049 1.0012 1.0003 1.0001 1.0000    -0.1270  -0.2499  -0.4968  -0.9878  -1.9558  -3.8364    -2.499
         first order -Z tan(delta)/B, %:  -0.1247  -0.2493  -0.4987  -0.9973  -1.9947  -3.9893   |e_Z(16 m)/e_Z(0.5 m)| = 30.22
  +0.02   1.0196 1.0049 1.0012 1.0003 1.0001 1.0000    -0.2536  -0.4986  -0.9887  -1.9563  -3.8366  -7.3892    -4.986
         first order -Z tan(delta)/B, %:  -0.2493  -0.4987  -0.9973  -1.9947  -3.9893  -7.9786   |e_Z(16 m)/e_Z(0.5 m)| = 29.14
  +0.03   1.0197 1.0049 1.0012 1.0003 1.0001 1.0000    -0.3799  -0.7461  -1.4758  -2.9060  -5.6466 -10.6890    -7.461
         first order -Z tan(delta)/B, %:  -0.3740  -0.7480  -1.4960  -2.9920  -5.9840 -11.9680   |e_Z(16 m)/e_Z(0.5 m)| = 28.14
  +0.05   1.0197 1.0050 1.0013 1.0003 1.0001 1.0000    -0.6316  -1.2373  -2.4357  -4.7513  -9.0695 -16.6299   -12.373
         first order -Z tan(delta)/B, %:  -0.6233  -1.2467  -2.4933  -4.9867  -9.9733 -19.9466   |e_Z(16 m)/e_Z(0.5 m)| = 26.33
  -0.02   1.0196 1.0049 1.0012 1.0003 1.0001 1.0000    +0.2549  +0.5036  +1.0086  +2.0359  +4.1554  +8.6706    +5.036
         first order -Z tan(delta)/B, %:  +0.2493  +0.4987  +0.9973  +1.9947  +3.9893  +7.9786   |e_Z(16 m)/e_Z(0.5 m)| = 34.02

C. Wall at 2 m, delta = +0.02 deg: offset relative to the centre column
  left column          100     150     200     240     300     350     400     450
  offset ratio      1.1370  1.0620  1.0160  1.0000  1.0107  1.0513  1.1209  1.2194
  1 + tan^2 (norm.) 1.1368  1.0620  1.0160  1.0000  1.0107  1.0514  1.1211  1.2196
  max relative difference 0.0176%; e_Z across the wall -1.2030% to -0.9887%

Part A: the R200’s numbers. The geometry reproduces the text’s ranges: $fB/64$ is 0.455 m at 480 pixels and 0.303 m at 320, both 9.1% short of “around 1/2 a meter” and “roughly 1/3 of a meter”, and one pixel of disparity sits at 29.1 m against “roughly 30 meters”. The geometry’s 0.455 m is 14.2% short of Table 5’s minimum distance of 0.53 m; with the 15% tolerance planned for these comparisons (H-A1), all four held, the last one close to its limit. Table 5 is less consistent with Equation 2. Its two RMS rows keep the quadratic law (8.0/2.1 = 3.81, 4.8% from 4: H-A2’s first part held), but with its own $\epsilon_d = 0.08$ and the nominal focal length, Equation 2 predicts 2.75 mm and 11.00 mm, 31% and 38% above the printed values; the rows imply $\epsilon_d \approx 0.06$ pixel (0.061 and 0.058). H-A2’s other two parts, that Equation 2 would miss by more than 15% and that the implied $\epsilon_d$ would lie between 0.05 and 0.07, held as well. The 1% RMS range from Equation 3 is 3.64 m, 11.3% short of Table 5’s 4.1 m (H-A3, within 15%: held), and 5 pixels at this focal length is 1.20% of distance rather than 1% (H-A5, within 30%: held). The paper’s own worked example in §4.2.8, with $r_{\rho=95\%} = 6.0$ m, $\alpha = 0.2$ , $\theta_\text{FOV} = 15°$ and $\theta_\text{target} = 60°$ , gives 1.68 m with the formula as printed, 20% above the “≈ 1.4m” that the text states (H-A4, that the two would differ by more than 15%: held). These differences are the paper’s, read from the rendered pages; the table describes “an example R200 unit”, whose actual focal length the paper does not give, so the comparison tests the printed numbers against the nominal geometry, not the device. Finally, Table 5’s 5 mm calibration error at 1 m corresponds to 0.146 pixel of disparity, which a yaw error of 0.020° would produce (within the planned 0.01–0.03°: held).

Part B: a synchronization offset. The two checks held: the rotation matrix agrees with SciPy’s, and with no offset and no miscalibration the SFNC formulas recover every point to $2 \times 10^{-14}$ , with no point outside the right image in any case. Lateral motion behaves exactly as derived (H-B1 and H-B2 held: every deviation from $-v\Delta t/B$ in disparity and from $v\Delta t/(B - v\Delta t)$ in depth is below $10^{-13}$ , against a planned bar of $10^{-12}$ , and the minimum and maximum over all 48 points coincide, so the error is the same at 0.5 m and at 16 m). An offset of 1 ms at 1 m/s, a slow walk past a 7 cm rig, scales depth by 1.45% when the object moves from the left camera toward the right and by −1.41% when it moves the other way, at every distance. The first-order $v\Delta t/B$ is within 1.43% of the exact value up to 1 ms and 7.14% off at 5 ms (H-B3 held). A rotating rig is different: turning at 30°/s during a 1 ms offset makes the right camera look 0.03° to the side, and the depth error then grows with distance, from −0.37% at 0.5 m to −10.7% at 16 m, as for a fixed 0.03° yaw error. The two agree within 1.92% at 0.5 m and more closely with depth (H-B4 held; its 2.5% tolerance was fixed after a planning script, run before the pilot with the same set-up, had computed this agreement, so this part is an illustration of the equivalence rather than an independent test). The residual difference comes from the construction: in the rotating case the right camera’s centre also moves on its arc about the left camera, which a fixed yaw about its own centre does not include. In our reading, for a handheld or vehicle-mounted rig at these rates, rotation during the offset is the harmful case at long range, and lateral motion the uniform one.

Part C: a yaw miscalibration. For points on the left camera’s optical axis (the centre column) the disparity offset equals $f\tan\delta$ to within 2% at every depth and angle (H-C1, within 3%: held), and it is the same at every depth, which is why the depth error grows with distance. A yaw of the right camera away from the left one ( $\delta > 0$ here) makes every depth too short and $\delta = -0.02°$ every depth too long, with magnitudes growing monotonically (H-C2 held); at $\delta = 0.02°$ the error at 1 m is −4.99 mm, the size of Keselman et al.’s calibration figure (within the planned 20% of 5 mm: held), and at 16 m it is 29 times the relative error at 0.5 m (H-C4, at least 20×: held). At long range the exact errors leave the first-order $-Z\tan\delta/B$ , smaller in magnitude for $\delta > 0$ and larger for $\delta < 0$ , because the offset becomes a sizeable part of the true disparity in $-\Delta/(d_\text{true} + \Delta)$ . Across a wall at 2 m the offset grows toward the image edges as $1 + \tan^2\theta$ , to 1.22 times the centre value at the outermost column, within 0.02% of the prediction (H-C3 held), so the wall is reconstructed bowed, with errors from −0.99% at the centre to −1.20% at the edge, rather than shifted.

The output was identical on two runs, and the script runs in well under a second (NumPy 2.3.5, SciPy 1.16.3). Every hypothesis and check was written in a plan before the script was first run. Part A is a consistency check, an illustration rather than a test: the plan records that “while reading the paper I did rough hand arithmetic for several of these” before writing the Part A hypotheses. The example departs from the batch plan for this article in several planned ways: it adds the Table 5 rows and the §4.2.8 example to Part A, adds the opposite motion direction, the 0.1 ms offset and the rotating rig to Part B, and adds the +0.03° and −0.02° angles and the bowing measure to Part C. It leaves out a planned optional part on the 5-bit sub-pixel quantization, which would repeat the quantization example in disparity, and an optional block-matching variant, since matching belongs to stereo matching and would add its own noise to a check of exact relations. One change was made after the pilot: the pilot printed Part B’s disparity error to six decimals only, which could not decide H-B1 and H-B2 at their planned $10^{-12}$ ; the column of maximum deviations from $-v\Delta t/B$ was added as a precision repair approved by the editor before the first run, and no outcome differs between the pilot and the first run. The closest published examples convert a random ±0.25 pixel disparity error to depth error in stereo vision and measure the vertical disparity of a 0.1° pitch error in stereo rectification; this example’s biases come from timing and yaw.

Strengths

Works in sunlight. With natural texture a stereo camera needs no emitter; the R200’s infrared cameras use sunlight as a light source and work in broad daylight as well as, with the projector, at 0 lux, and in passive, well-textured conditions its one-pixel disparity sits at roughly 30 m (§3.3.1, §4.2.1).
Dense, metric depth from cheap parts. Two image sensors, a rigid bar and a matcher give a depth value per pixel at video rate, with absolute scale set by the baseline.
Co-registered images. The depth map is computed in the frame of the reference image, which the device also outputs, so in our reading the two need no registration step (a colour camera still does).
Several units can share a scene. The librealsense documentation states that structured-light systems are fragile under interference from sunlight or another structured-light device, while depth from stereo “only benefits from multi-camera setup” (Depth from Stereo, v2.59.1; a vendor’s statement).

Limitations

Error grows with the square of distance, and the fixed search range sets a minimum distance; both are set at design time by $f$ , $B$ and the matcher.
Texture dependence. Without texture, passive stereo has nothing to match; with a projector, the range indoors depends on the projector’s signal, the albedo and the angle of the surface (§2.1, §4.2.8).
Calibration drift is silent. A yaw error biases depth without any visible symptom, as Part C shows; pitch and roll errors at least show up as lost matches.
Timing. Unsynchronized or rolling-shutter imagers mis-measure moving scenes, and rotation during an offset acts like a calibration error that grows with range (Part B).
Edges and fine structure. Block-based matchers merge gaps narrower than their support window and leave half-occluded regions without data.
Speckle and fixed-pattern sub-pixel noise with laser projectors (§4.2.6).

Typical Applications

Robots and drones, for obstacle avoidance and mapping; the drone vision article lists an Intel RealSense D435i stereo depth camera, 640 × 480 depth at 30 Hz, on the platform of Loquercio et al. (2021), and the robot perception article covers stereo and active depth cameras among robot sensors.
Driving datasets and driver assistance, with long-baseline rigs such as KITTI’s two pairs about 54 cm apart, where forward-facing stereo cameras estimate the distance of vehicles and pedestrians (stereo vision).
Visual-inertial odometry and SLAM, where a synchronized global-shutter pair gives metric scale, as in EuRoC MAV and visual odometry.
3D scanning and reconstruction: Keselman et al. show R200 data used for volumetric 3D reconstruction and box measurement, and the cameras built into robots, drones and computers (§6).
RGB-D sensing, where an active stereo engine is one of the depth technologies that depth maps come from.

Comparison with Other Sensors

The LiDAR article’s comparison table places the stereo camera beside the camera, the time-of-flight camera, LiDAR and radar: dense depth by triangulation of matched pixels, with an error that grows with the square of range, weak on textureless surfaces and in the dark. LiDAR measures range along each beam directly, and the same article reports that its range errors stay relatively constant with distance, and lists sparse sampling and cost among its weaknesses. Time-of-flight cameras measure range per pixel from the phase of modulated light and need no texture; the LiDAR article lists their short range and their sensitivity to background light outdoors. Structured-light cameras decode a known projected pattern and can triangulate against the projector, so one camera and a projector suffice, whereas an active stereo camera uses its projector only as texture for its two cameras; the librealsense documentation states that structured light is fragile under interference from sunlight or another structured-light device, while stereo only benefits from several cameras, a vendor’s comparison. RGB-D cameras, the class the depth map article lists by technology, are built on one of these three depth engines.