RGB-D Camera

An RGB-D camera is a depth camera and an ordinary colour camera built into one unit and calibrated together, so that every frame delivers a colour image and a depth map of the same scene. The depth part is an active sensor with its own infrared light: a structured-light projector and camera, a time-of-flight camera, or an actively textured stereo camera. The colour part is a conventional Bayer-pattern camera. What makes the device class distinctive is the pairing. The two images come from different lenses at different positions, and often at different resolutions, so a depth pixel and the colour pixel with the same coordinates do not see the same point. Before colour and depth can be used together, one image has to be registered to the other, which needs both cameras’ intrinsics, the rigid transform between them and the depth itself.

Keselman, Iselin Woodfill, Grunnet-Jepsen and Bhowmik (2017) date the popularity of portable consumer RGB-D systems to the Microsoft Kinect, which, by computing depth in hardware and sending it over USB, “kicked off wide use” of RGB-D sensors in computer vision, human–computer interaction and robotics. The three sibling articles explain how each depth engine measures depth. This article covers what they have in common once a colour camera is added: what the unit contains, what data it produces and in which units, how it is calibrated and synchronized, and how depth is mapped into the colour image. The example re-implements depth-to-colour registration on a synthetic two-camera unit and measures the artifacts that the choices in two libraries’ implementations produce.

How It Works

What is inside the unit

An RGB-D camera contains three parts:

A depth engine. Its own illumination, one or two infrared cameras and a processor that turns their images into depth. The depth is computed in the frame of an infrared camera.
A colour camera. In the RealSense R200, Keselman et al. (2017, §3.3.1) describe a 1920 × 1080 Bayer-patterned, rolling-shutter CMOS imager with an on-board image signal processor for demosaicing, colour correction and white balance, beside a pair of 640 × 480 monochrome global-shutter infrared cameras.
Calibration and synchronization. Each R200 is calibrated in the factory to a subpixel-accurate camera model for all three lenses (§3). The infrared pair is undistorted and rectified in hardware; the colour camera is undistorted in software on the host.

The depth engines differ, and the depth estimation article sets them side by side:

Structured light. Fanello et al. (2016) describe many consumer depth cameras as using a spatially varying pattern, typically a pseudo-random pattern from a cheap diffractive optical element, “such as in Kinect”, and model their set-up on the Kinect, with an infrared camera at a fixed baseline to the projector. The structured light article covers the patent of PrimeSense, with which the first Kinect was developed. The ScanNet article describes the Structure sensor that recorded it as similar in design to the Kinect v1.
Time of flight. Su et al. (2018) name Microsoft’s Kinect One as an amplitude-modulated continuous-wave ToF camera, and Guo et al. (2018, §3.4) report that the Kinect 2 uses three modulation frequencies with three phases each. The time-of-flight camera article covers the pixel.
Active stereo. The RealSense R200 and its successors match two infrared images textured by a projector (Keselman et al., 2017, §2.2); Zhang et al. (2018) use the rectified infrared pair of a RealSense D435, 1280 × 720 at 30 frames per second. The stereo camera article covers the device.

Su et al. (2018) note that, combined with conventional colour cameras, RGB-D data allows high-fidelity scene reconstruction and has enabled the collection of large 3D datasets.

Depth-to-colour registration

Let $Z(u, v)$ be the depth map, with intrinsics $K_d$ , and let a 3D point in the depth camera’s frame map into the colour camera’s frame as $\mathbf X_c = R(\mathbf X_d - \mathbf C)$ , where $R$ is the rotation between the cameras and $\mathbf C$ the colour camera’s centre in the depth frame (equivalently $\mathbf X_c = R\mathbf X_d + \mathbf t$ with $\mathbf t = -R\mathbf C$ ). Registration has four steps:

Back-project each valid depth pixel, $\mathbf X_d = Z(u, v)\,K_d^{-1}[u, v, 1]^\top$ (depth map).
Transform it, $\mathbf X_c = R(\mathbf X_d - \mathbf C)$ .
Project it with the colour camera’s intrinsics $K_c$ (and its lens distortion), $(u', v') = (f_c X_c/Z_c + c_x,\; f_c Y_c/Z_c + c_y)$ .
Write its depth into the colour pixel it lands in, keeping the nearest when several land together (a z-buffer).

The result is a depth image on the colour camera’s pixel grid. Two things make it imperfect. Pixels that the colour camera sees but the depth camera does not, because a nearer object hides them from the depth camera, receive nothing: these are disocclusions. For cameras displaced by a baseline $B$ perpendicular to the optical axes, a foreground edge at depth $Z_\text{fg}$ in front of a background at $Z_\text{bg}$ leaves a gap of about

w \approx f_c\, B \left(\frac{1}{Z_\text{fg}} - \frac{1}{Z_\text{bg}}\right)

colour pixels beside the edge, the difference of the two surfaces’ disparities (our derivation for parallel cameras; a small rotation changes it only slightly). On the other side of the object, background points that the depth camera saw land on colour pixels that see the foreground, and a z-buffer has to remove them. And because each depth pixel is forward-projected to a single point, a colour image with more pixels per unit angle than the depth image receives too few samples, leaving cracks. The librealsense documentation names the same two artifacts, sampling and occlusion, for its alignment: changing viewport changes the resolution, for which it says the interpolation needs to be nearest-neighbour “to avoid introducing non-existing values”, and pixels whose 3D points the original sensor did not see “may hold invalid texture values” (rs-align example, v2.59.1).

The novel view synthesis article uses the same forward warp with a z-buffer to render virtual views from depth and colour; registration is the special case in which the target is a second physical camera with its own intrinsics.

Two implementations

OpenCV. The registerDepth function of OpenCV contrib’s rgbd module (4.14.0, depth_registration.cpp) takes the depth camera’s matrix, the colour (“external”) camera’s matrix and distortion coefficients, a 4 × 4 rigid transform, the depth image and the output size. From the source:

A 16-bit input is scaled by 0.001 to metres, so it is taken to be in millimetres; float inputs are taken to be in metres. A zero depth is turned into NaN internally and skipped, and float inputs are expected to mark missing depth with NaN already.
Pixel $(i, j)$ is back-projected as $(i z, j z, z)$ through $K_d^{-1}$ , so pixel centres sit at integer coordinates.
The transform is documented as rbtRgb2Depth, “the rigid body transform between the cameras”, but the code multiplies it onto the depth camera’s points before projecting into the external camera, so it must map depth-camera coordinates into colour-camera coordinates.
Distortion is applied only for the external camera; the depth image is assumed undistorted.
The projected position is converted to an integer pixel, checked against the image bounds, and the transformed point’s $z$ , which is the colour camera’s depth, is written, rounded to the nearest integer for 16-bit output. An “occlusion check” writes only into an empty pixel or over a larger value.
An optional 2 × 2 dilation also writes the value into the pixels to the left, above, and above-left, with the comment that this “is what’s done on PrimeSense devices, but a more accurate scheme could be used”.

librealsense. The RealSense SDK’s align processing block (v2.59.1, src/proc/align.cpp) can align depth to the colour stream or colour to the depth stream. For each depth pixel with non-zero depth it maps two corners of the pixel, $(x - 0.5, y - 0.5)$ and $(x + 0.5, y + 0.5)$ , into the other image at that pixel’s depth, rounds each by adding 0.5 and truncating to an integer, skips the pixel if the rectangle leaves the image, and writes into every target pixel of the rectangle. Because the whole footprint is written, a target image with more pixels per unit angle than the depth image cracks far less than with point splatting (our reading; variant E in the example shows it). When depth is aligned to colour, a target pixel keeps the smaller of its current value and the incoming one, and the value written is the depth image’s own 16-bit value, the depth camera’s $Z$ , rather than the point’s depth in the colour camera’s frame. The example shows what that difference means.

The libfreenect2 driver for the Kinect for Windows v2 lists “registration of RGB and depth images” among its features (README).

Data It Produces

An RGB-D frame is a set of images with different geometry:

Depth. One value per depth pixel, along the optical axis (depth map), unless the device reports radial range, as the time-of-flight camera article explains for ToF devices. It is either unregistered, on the depth camera’s grid, or registered to colour.
Colour. An RGB image from the colour camera, at its own resolution and frame rate.
Infrared. The image of the infrared camera in whose frame depth is computed can share the depth’s pixel grid; a ToF camera’s intensity image is registered to its range image by construction (time-of-flight camera). The R200 delivers infrared, depth and colour images (Keselman et al., 2017, Fig. 1), and libfreenect2’s README lists IR and depth image transfer for the Kinect v2.

Units. Depth is often stored as 16-bit integers with a scale (depth map). In librealsense, the Z16 format holds “16-bit linear depth values”, the depth in metres being the depth scale times the pixel value; the scale is a device option, RS2_OPTION_DEPTH_UNITS, the “number of meters represented by a single depth unit”, and rs2_depth_frame_get_units returns it per frame (rs_sensor.h, rs_option.h, rs_frame.h). Extrinsics are a column-major 3 × 3 rotation and a translation in metres (rs_sensor.h). Datasets fix their own scales: TUM RGB-D stores 5,000 units per metre, and NYU Depth v2 converts raw Kinect values to metres with its toolbox. OpenCV’s registerDepth, as above, assumes millimetres for 16-bit input.

Invalid depth. The SFNC’s 3D chapter describes two ways to mark pixels without a valid measurement: a confidence (mask) image, or an out-of-bounds value of the range data, commonly 0 or a high value outside the valid range; it discourages NaN because of processing overhead (SFNC 2.8, Chapter 21). librealsense skips depth pixels “with the value of zero” when aligning and initializes the aligned image to zero (align.cpp). OpenCV uses 0 for 16-bit and NaN for float depth. The depth map article warns against reading 0 as “very close”.

Registered or not. The SFNC distinguishes two ways a camera can deliver several images. Components selected with ComponentSelector, such as intensity, infrared, range, confidence or disparity, come from one source region and “are defined to be registered”, each pixel of every component referring to the same position (SFNC 2.8, §4.16). A device with independent sensors, its example being one with a visible and an infrared sensor, is a multi-source device with a source selector for each sensor (Chapter 19). In our reading of these terms, an RGB-D camera’s infrared and depth images are components of one source, while its colour image comes from a second source, and registration is the step that turns the two into pixel-aligned images.

Characteristics

Device (source)	Depth engine	Depth images	Colour images	Synchronization
Kinect (first), in KinectFusion and TUM RGB-D	Structured light (Fanello et al., 2016)	640 × 480 at 30 Hz, noisy and with holes (KinectFusion article)	640 × 480 (TUM RGB-D)	Not synchronized; depth arrives about 20 ms after colour on average (TUM RGB-D)
Kinect One / Kinect 2	Continuous-wave ToF (Su et al., 2018; Guo et al., 2018)
RealSense R200 (Keselman et al., 2017)	Active stereo, infrared pair 70 mm apart	From the 640 × 480 infrared pair, which runs at 30, 60 or 90 Hz	1920 × 1080 at 30 Hz, rolling shutter	Colour synchronized with the pair at start of frame
Structure sensor on an iPad Air 2 (ScanNet)	Similar in design to the Kinect v1	640 × 480	1296 × 968	Hardware-synchronized at 30 Hz

Blank cells are left where no source read here gives the value.

Three properties follow from the pairing rather than from the depth engine:

Different viewpoints. The colour camera sits beside the depth engine, so a registered depth image has disocclusion holes beside every foreground edge, their width set by the baseline and the two depths as above. In the example, a 25 mm baseline with a box 1.0 m in front of a wall 2.5 m away gives $f_c B (1/Z_\text{fg} - 1/Z_\text{bg}) = 4.5$ colour pixels.
Different resolutions and fields of view. The R200’s colour camera has a nominal field of view of 70° × 43° against 60° × 45° for the infrared pair (§3.3.1). Where the colour camera samples more finely, point-wise registration cracks; where it sees beyond the depth camera, registered depth is missing.
Different shutters and timing. The R200’s colour camera has a rolling shutter and is synchronized only at the start of the frame, so its later rows are exposed after the depth images; rolling shutter covers the timing. TUM RGB-D’s Kinect streams were not synchronized at all. When the camera or the scene moves, registration with exact geometry still pairs colour and depth from different instants.

Calibration

An RGB-D unit needs the intrinsics of each camera, their lens distortion and the extrinsics between them (camera calibration, lens distortion). Manufacturers calibrate in the factory: the R200 is calibrated per unit to a modified Brown–Conrady model with three even-powered radial and two tangential terms (Keselman et al., 2017, §3 and §3.3.1), and librealsense exposes each stream’s intrinsics, its distortion model and the extrinsics between streams through its API (rs_types.h, rs_sensor.h). Datasets have recalibrated their sensors. The TUM RGB-D data were registered by the OpenNI driver with the Kinect’s factory calibration, and its authors found the depth of two sensors off by constant factors of 1.035 and 1.031 against checkerboard distances, which they corrected (TUM RGB-D). NYU Depth v2 ships its own Kinect calibration with the toolbox (NYU Depth v2), and ScanNet calibrated each device on its server from sequences its users recorded (ScanNet). Keselman et al. (2017, §4.2.4) measured the R200’s calibration error, a consistent depth offset from imperfect undistortion and rectification fits, as zero-mean and on the order of 5 mm at 1 m.

Because registration multiplies the extrinsics onto every depth point, an error in them misplaces depth in the colour image by an amount that depends on depth. The example measures the extreme case, extrinsics ignored.

Processing the Data

Registration, in either direction. Depth aligned to colour gives a depth value per colour pixel; colour aligned to depth gives a colour per depth pixel, which librealsense’s align also offers, fetching colour for every depth pixel’s footprint. In our reading, the first keeps the colour resolution and inherits the holes, while the second keeps every depth measurement.
Back-projection to coloured point clouds. Registered depth back-projects into a point cloud with one colour per point.
Filtering. Keselman et al. (2017, §4.2.9) found median or speckle filters efficient against the R200’s sparse noise, and colour-guided, edge-preserving smoothing and inpainting of the depth often helpful, such filters being best applied in disparity space; the stereo camera article covers librealsense’s post-processing chain.
Hole filling. NYU Depth v2 describes its toolbox’s hole filling by colorization; the example below shows where holes come from in registration.
Fusion and SLAM. Depth frames are fused into volumes (TSDF fusion, KinectFusion) and tracked in RGB-D SLAM, which TUM RGB-D benchmarks.

Example: registering depth to colour

The script builds a synthetic RGB-D unit and registers its depth image into its colour camera in seven ways. The depth camera has 320 × 240 pixels and $f_d$ = 285 px; the colour camera has the same size, $f_c$ = 300 px, and is 25 mm to the right of the depth camera ( $s = +1$ ) or to its left ( $s = -1$ ), rotated by 0.5° about the vertical axis. Neither camera has lens distortion. Both look at a wall 2.5 m away and a box whose front face is 1.0 m away. Exact ray casting gives the depth image, which is stored as 16-bit millimetres with a 20 × 20 patch of zeros for a surface that returned nothing, and the true depth seen by every colour pixel.

Conventions: $x$ right, $y$ down, $z$ forward; pixel centres at integer coordinates, as in both libraries’ code; depth is $Z$ along each camera’s optical axis, in metres; $\mathbf X_c = R(\mathbf X_d - \mathbf C)$ . Every colour pixel falls in one of four classes: S, its surface point is seen by the depth camera with a valid measurement, so it should receive a value; O, its point is hidden from the depth camera by the box (disoccluded); I, its point falls on the patch of zeros; F, outside the depth camera’s view. A registered value is wrong if it differs from the true colour-camera depth by more than 1 cm; the edge mask marks colour pixels whose 3 × 3 neighbourhood of true depth spans more than 0.5 m.

The variants are re-implemented in NumPy from the descriptions above; no library code is copied:

A round: each depth pixel centre is projected and written to the nearest colour pixel, with a z-buffer, storing the colour-frame $Z$ as OpenCV’s code does.
B trunc: as A, but the position is truncated (floor) instead of rounded. This is our variant; the example makes no claim about which conversion OpenCV uses.
C no z-buffer: as A, but the last sample written in row-major scan order wins.
D dilate: as A, plus OpenCV’s 2 × 2 dilation.
E footprint Zc: librealsense’s corner footprint, storing colour-frame $Z$ .
F footprint Zd: the footprint storing the depth camera’s own $Z$ , as librealsense’s align.cpp does.
G identity: as A, with the extrinsics ignored ( $R = I$ , $\mathbf C = 0$ ).

For each variant the script prints the counts of correct, wrong and missing values in S, wrong values off the edge mask, filled (and wrong) pixels in O and I, and a hole width. The hole width counts the empty pixels of the box’s middle colour row inside a window that covers the exact disoccluded run in that row plus 2 pixels on each side; a printed line applies the same measure to the exact model, in which a pixel is empty exactly when it is disoccluded. For A and B the script prints the mean sub-pixel offset between a sample’s projected column and the column it was written to, and for G the mean horizontal misalignment between identity and true projections, against the first-order value $f_c(sB/Z - \tan 0.5^\circ)$ (our derivation for small angles and points near the axis).

import os
os.environ["OPENBLAS_NUM_THREADS"] = "1"          # one BLAS thread, set before NumPy loads
import numpy as np
from scipy.ndimage import maximum_filter, minimum_filter

# Two pinhole cameras: x right, y down, z forward; pixel centres at integer
# coordinates, so pixel u covers [u - 0.5, u + 0.5). Depth is Z along the
# optical axis, in metres; the depth image is stored as uint16 millimetres
# with 0 = no measurement. Extrinsics map depth-camera points to the colour
# camera: X_c = R (X_d - C), with C the colour camera's centre.
W, H = 320, 240
FD, FC, CX, CY = 285.0, 300.0, 159.5, 119.5      # focal lengths (px), principal point
B, YAW = 0.025, np.deg2rad(0.5)                   # baseline (m), rotation about y
WALL = 2.5                                        # wall plane z = 2.5 m
BOX = np.array([-0.30, -0.25, 1.0]), np.array([0.10, 0.15, 1.3])
PATCH = (slice(30, 50), slice(250, 270))          # depth pixels with no return
TOL = 0.01                                        # a value is wrong beyond 1 cm

def Ry(a):
    c, s = np.cos(a), np.sin(a)
    return np.array([[c, 0, s], [0, 1, 0], [-s, 0, c]])

def raycast(C, R, f, u, v):
    """Rays through pixels (u, v) of a camera with centre C and X_cam = R (X - C).
    Returns the hit point X (depth frame), Z in that camera, the label (1 box,
    0 wall) and the box face's axis (-1 for the wall)."""
    dc = np.stack([(u - CX) / f, (v - CY) / f, np.ones_like(u, float)], -1)
    dw = dc @ R                                   # ray directions in the depth frame
    t_wall = (WALL - C[2]) / dw[..., 2]
    with np.errstate(divide="ignore", invalid="ignore"):
        t0, t1 = (BOX[0] - C) / dw, (BOX[1] - C) / dw
    tin, tout = np.minimum(t0, t1).max(-1), np.maximum(t0, t1).min(-1)
    hit = (tin <= tout) & (tin > 0) & (tin < t_wall)
    t = np.where(hit, tin, t_wall)                # dc has z = 1, so t is Z_cam
    face = np.where(hit, np.minimum(t0, t1).argmax(-1), -1)
    return C + t[..., None] * dw, t, hit.astype(int), face

def project(X, C, R, f):
    Xc = (X - C) @ R.T
    return f * Xc[..., 0] / Xc[..., 2] + CX, f * Xc[..., 1] / Xc[..., 2] + CY, Xc[..., 2]

def residual(X, lab):
    wall = np.abs(X[lab == 0][:, 2] - WALL).max()
    box = np.abs(X[lab == 1][:, 2] - BOX[0][2]).max()
    return max(wall, box)

# ---- the depth camera's image ----
u, v = np.meshgrid(np.arange(W, dtype=float), np.arange(H, dtype=float))
Xd_true, Zd_true, lab_d, face_d = raycast(np.zeros(3), np.eye(3), FD, u, v)
D = np.round(Zd_true * 1000).astype(np.uint16)
D[PATCH] = 0
valid = D > 0
print(f"D1 depth camera: plane residual {residual(Xd_true, lab_d):.1e} m, box faces hit (axis) "
      f"{sorted(set(face_d[lab_d == 1].tolist()))}")
print(f"D2 quantization: max |D/1000 - Z| {np.abs(D[valid] / 1000 - Zd_true[valid]).max():.4f} m, "
      f"invalid pixels {int((~valid).sum())}")

# ---- registration variants ----
def register(C, R, mode="round", zbuf=True, dilate=False, footprint=False, store="Zc"):
    """Depth image D -> colour image. Returns the registered depth (0 = none) and,
    for point splatting, the float and assigned columns of the samples."""
    Z = D[valid] / 1000.0
    uu, vv = u[valid], v[valid]
    def to_colour(du, dv):                         # deproject at depth Z, transform, project
        X = np.stack([(uu + du - CX) / FD * Z, (vv + dv - CY) / FD * Z, Z], -1)
        return project(X, C, R, FC)
    uc, vc, zc = to_colour(0.0, 0.0)
    val = zc if store == "Zc" else Z
    if footprint:                                  # corners at -0.5 and +0.5, int(x + 0.5)
        u0, v0, _ = to_colour(-0.5, -0.5)
        u1, v1, _ = to_colour(+0.5, +0.5)
        x0, y0 = np.floor(u0 + 0.5).astype(int), np.floor(v0 + 0.5).astype(int)
        x1, y1 = np.floor(u1 + 0.5).astype(int), np.floor(v1 + 0.5).astype(int)
        keep = (x0 >= 0) & (y0 >= 0) & (x1 < W) & (y1 < H)
        tx, ty, tv = [], [], []
        for dy in range(int((y1 - y0)[keep].max()) + 1):
            for dx in range(int((x1 - x0)[keep].max()) + 1):
                m = keep & (x0 + dx <= x1) & (y0 + dy <= y1)
                tx.append(x0[m] + dx); ty.append(y0[m] + dy); tv.append(val[m])
        tx, ty, tv = np.concatenate(tx), np.concatenate(ty), np.concatenate(tv)
        xi = None
    else:
        xi = np.floor(uc + 0.5) if mode == "round" else np.floor(uc)
        yi = np.floor(vc + 0.5) if mode == "round" else np.floor(vc)
        xi, yi = xi.astype(int), yi.astype(int)
        keep = (xi >= 0) & (xi < W) & (yi >= 0) & (yi < H)
        tx, ty, tv = xi[keep], yi[keep], val[keep]
        if dilate:                                 # OpenCV's 2 x 2: (x-1, y), (x, y-1), (x-1, y-1)
            ex, ey = [tx], [ty]
            for dx, dy in ((-1, 0), (0, -1), (-1, -1)):
                ex.append(tx + dx); ey.append(ty + dy)
            tx, ty, tv = np.concatenate(ex), np.concatenate(ey), np.tile(tv, 4)
            ok = (tx >= 0) & (ty >= 0)
            tx, ty, tv = tx[ok], ty[ok], tv[ok]
    out = np.full(H * W, np.inf)
    idx = ty * W + tx
    if zbuf:
        np.minimum.at(out, idx, tv)               # keep the nearest
    else:                                          # last write in row-major scan order wins
        last = len(idx) - 1 - np.unique(idx[::-1], return_index=True)[1]
        out[idx[last]] = tv[last]
    out = out.reshape(H, W)
    out[~np.isfinite(out)] = 0.0
    extra = (uc[keep], xi[keep]) if xi is not None else None
    return out, extra

VARIANTS = [("A round", {}), ("B trunc", {"mode": "trunc"}), ("C no z-buffer", {"zbuf": False}),
            ("D dilate", {"dilate": True}), ("E footprint Zc", {"footprint": True}),
            ("F footprint Zd", {"footprint": True, "store": "Zd"}), ("G identity", {})]

for s in (+1, -1):
    C, R = np.array([s * B, 0.0, 0.0]), Ry(YAW)
    # ---- ground truth in the colour camera ----
    Xc_true, Zc_true, lab_c, face_c = raycast(C, R, FC, u, v)
    ud, vd, zd = project(Xc_true, np.zeros(3), np.eye(3), FD)
    inside = (ud >= -0.5) & (ud < W - 0.5) & (vd >= -0.5) & (vd < H - 0.5)
    tdep = raycast(np.zeros(3), np.eye(3), FD, ud, vd)[1]
    O = inside & (tdep < zd - 1e-9)                       # hidden from the depth camera
    ni, nj = np.clip(np.floor(vd + 0.5).astype(int), 0, H - 1), np.clip(np.floor(ud + 0.5).astype(int), 0, W - 1)
    I = inside & ~O & ~valid[ni, nj]             # nearest depth pixel has no return
    S = inside & ~O & ~I                                  # should receive a depth value
    F = ~inside
    I_border = I & ~(np.roll(I, 1, 0) & np.roll(I, -1, 0) & np.roll(I, 1, 1) & np.roll(I, -1, 1))
    E = maximum_filter(Zc_true, 3, mode="nearest") - minimum_filter(Zc_true, 3, mode="nearest") > 0.5
    rows = np.where(lab_c.any(1))[0]
    vmid = int(np.round(np.median(rows)))
    cols = np.where(lab_c[vmid] == 1)[0]
    orun = np.where(O[vmid])[0]                           # exact disoccluded run in the middle row
    win = np.arange(max(orun.min() - 2, 0), min(orun.max() + 3, W))   # plus 2 px on each side
    def hole(has):                                        # empty pixels inside the window
        return int((~has[vmid, win]).sum())
    # D3: the exact depth-camera points, projected into the colour camera, land on
    # the colour camera's own surface point with the same Z (and do not if the
    # inverse transform is used by mistake).
    def d3(Cm, Rm, masked=True):
        uc, vc, zc = project(Xd_true, Cm, Rm, FC)
        Xv, Zv, _, _ = raycast(C, R, FC, uc, vc)
        m = valid & (np.linalg.norm(Xv - Xd_true, axis=-1) < 1e-9) if masked else valid
        return np.abs(zc - Zv)[m].max()
    print(f"\n===== colour camera {'right' if s > 0 else 'left'} of the depth camera (s = {s:+d}) =====")
    print(f"D1 colour camera: plane residual {residual(Xc_true, lab_c):.1e} m, box faces hit (axis) "
          f"{sorted(set(face_c[lab_c == 1].tolist()))}")
    print(f"D3 direction: max |z' - Z_c(u', v')| {d3(C, R):.1e} m; with the inverse transform {d3(-R @ C, R.T, masked=False):.3f} m")
    print(f"D4 classes: S {S.sum()}, O {O.sum()}, I {I.sum()} (border {I_border.sum()}), F {F.sum()}, "
          f"sum {S.sum() + O.sum() + I.sum() + F.sum()}; box rows {len(rows)}, middle row {vmid}")
    print(f"hole measure: window columns {win.min()}-{win.max()} of row {vmid}; on the exact model: {hole(~O)} px")
    print("variant          valid |  correct    wrong  missing  (% of S)        | wrong off-edge | "
          "O filled (wrong) | I filled (border) | hole")
    for name, kw in VARIANTS:
        Cv, Rv = (np.zeros(3), np.eye(3)) if name.startswith("G") else (C, R)
        reg, extra = register(Cv, Rv, **kw)
        has = reg > 0
        err = np.abs(reg - Zc_true)
        wrong = has & (err > TOL)
        w = hole(has)
        nS = S.sum()
        print(f"{name:15s} {has.sum():6d} | {(S & has & ~wrong).sum():8d} {(S & wrong).sum():8d} {(S & ~has).sum():8d}"
              f"  ({100 * (S & has & ~wrong).sum() / nS:6.3f} {100 * (S & wrong).sum() / nS:5.3f} "
              f"{100 * (S & ~has).sum() / nS:5.3f}) | {(S & wrong & ~E).sum():14d} | "
              f"{(O & has).sum():8d} ({(O & wrong).sum():4d}) | {(I & has).sum():8d} ({(I & has & I_border).sum():4d}) | {w:4d}")
        if name[0] in "AB":
            uf, ui = extra
            print(f"    mean sub-pixel offset u_float - u_assigned: {np.mean(uf - ui):+.4f} px")
        if name.startswith("G"):
            uc_id = project(Xd_true, np.zeros(3), np.eye(3), FC)[0]
            uc_ok = project(Xd_true, C, R, FC)[0]
            for lab, nm, Z in ((1, "box", 1.0), (0, "wall", 2.5)):
                m = valid & (lab_d == lab)
                print(f"    misalignment u_identity - u_true on {nm} samples: mean {np.mean((uc_id - uc_ok)[m]):+.4f} px; "
                      f"first order f_c (s B / Z - tan 0.5 deg) = {FC * (s * B / Z - np.tan(YAW)):+.4f} px")

Output (NumPy 2.3.5, SciPy 1.16.3; deterministic, no random numbers; the script runs in well under a second):

D1 depth camera: plane residual 0.0e+00 m, box faces hit (axis) [2]
D2 quantization: max |D/1000 - Z| 0.0000 m, invalid pixels 400

===== colour camera right of the depth camera (s = +1) =====
D1 colour camera: plane residual 4.4e-16 m, box faces hit (axis) [2]
D3 direction: max |z' - Z_c(u', v')| 4.4e-16 m; with the inverse transform 1.498 m
D4 classes: S 75759, O 600, I 441 (border 80), F 0, sum 76800; box rows 120, middle row 104
hole measure: window columns 183-191 of row 104; on the exact model: 5 px
variant          valid |  correct    wrong  missing  (% of S)        | wrong off-edge | O filled (wrong) | I filled (border) | hole
A round          68462 |    68348        0     7411  (90.218 0.000 9.782) |              0 |      114 ( 114) |        0 (   0) |    4
    mean sub-pixel offset u_float - u_assigned: +0.0006 px
B trunc          68459 |    68212      228     7319  (90.038 0.301 9.661) |            112 |        0 (   0) |       19 (  19) |    5
    mean sub-pixel offset u_float - u_assigned: +0.5001 px
C no z-buffer    68462 |    68348        0     7411  (90.218 0.000 9.782) |              0 |      114 ( 114) |        0 (   0) |    4
D dilate         76023 |    75497      242       20  (99.654 0.319 0.026) |              0 |      243 ( 120) |       41 (  41) |    3
E footprint Zc   75876 |    75487      242       30  (99.641 0.319 0.040) |              0 |      124 ( 120) |       23 (  23) |    4
F footprint Zd   75876 |    64687    11042       30  (85.385 14.575 0.040) |          10800 |      124 ( 120) |       23 (  23) |    4
G identity       68912 |    58082    10374     7303  (76.667 13.693 9.640) |          10254 |      456 ( 456) |        0 (   0) |    1
    misalignment u_identity - u_true on box samples: mean +4.8085 px; first order f_c (s B / Z - tan 0.5 deg) = +4.8819 px
    misalignment u_identity - u_true on wall samples: mean +0.0636 px; first order f_c (s B / Z - tan 0.5 deg) = +0.3819 px

===== colour camera left of the depth camera (s = -1) =====
D1 colour camera: plane residual 4.4e-16 m, box faces hit (axis) [2]
D3 direction: max |z' - Z_c(u', v')| 4.4e-16 m; with the inverse transform 1.510 m
D4 classes: S 75879, O 480, I 441 (border 80), F 0, sum 76800; box rows 120, middle row 104
hole measure: window columns 74-81 of row 104; on the exact model: 4 px
variant          valid |  correct    wrong  missing  (% of S)        | wrong off-edge | O filled (wrong) | I filled (border) | hole
A round          68465 |    68236      229     7414  (89.927 0.302 9.771) |            113 |        0 (   0) |        0 (   0) |    5
    mean sub-pixel offset u_float - u_assigned: +0.0060 px
B trunc          68460 |    68327        0     7552  (90.047 0.000 9.953) |              0 |      114 ( 114) |       19 (  19) |    4
    mean sub-pixel offset u_float - u_assigned: +0.4957 px
C no z-buffer    68465 |    68009      456     7414  (89.628 0.601 9.771) |            336 |        0 (   0) |        0 (   0) |    5
D dilate         75905 |    75499      242      138  (99.499 0.319 0.182) |              0 |      123 ( 120) |       41 (  41) |    4
E footprint Zc   75753 |    75367      242      270  (99.325 0.319 0.356) |              0 |      123 ( 120) |       21 (  21) |    3
F footprint Zd   75753 |    64807    10802      270  (85.408 14.236 0.356) |          10560 |      123 ( 120) |       21 (  21) |    3
G identity       68912 |    56822    11514     7543  (74.885 15.174 9.941) |          11384 |      456 ( 456) |      120 (  30) |    0
    misalignment u_identity - u_true on box samples: mean -10.1665 px; first order f_c (s B / Z - tan 0.5 deg) = -10.1181 px
    misalignment u_identity - u_true on wall samples: mean -5.9390 px; first order f_c (s B / Z - tan 0.5 deg) = -5.6181 px

Data checks. Both ray casts lie on their planes to rounding error, and both cameras see only the box’s front face. The 20 × 20 patch gives 400 invalid depth pixels; quantization error is zero because both planes lie at whole millimetres. The transform check projects the exact depth-camera points into the colour camera and finds the colour camera’s own surface depth there to 4.4e-16 m, while the inverse transform misses by 1.498 m and 1.510 m, so the direction of the extrinsics is the one the code states. The hole measure applied to the exact model prints the exact disoccluded widths, 5 pixels at $s = +1$ and 4 at $s = -1$ (that check held); both lie within a pixel of the formula’s 4.5 px. Every data check held.

Rounding, truncation and cracks. Rounded point splatting (A) writes no wrong value in S at $s = +1$ ; at $s = -1$ it writes 229, of which 113 lie off the edge mask, so the planned check of no wrong value off the edge mask (H1) held at $s = +1$ and failed at $s = -1$ . Truncation (B) writes each sample, on average, half a pixel to the left of where it projects: the mean offset is +0.5001 and +0.4957 px against +0.0006 and +0.0060 px for rounding, as expected when the floor discards each position’s fractional part (H3a held at both signs). The planned comparison that truncation makes more wrong values than rounding (H3b) held at $s = +1$ (228 against 0) and failed at $s = -1$ (0 against 229). Point splatting leaves 7,411 and 7,414 pixels of S without a value, 9.782% and 9.771%. The plan expected 4–6%, from a derivation that counted only the 5% of columns the finer colour grid leaves empty, and that check (H6, for A) failed at both signs. After the pilot we did the arithmetic once more counting rows as well: $1 - (285/300)^2 = 0.0975$ , close to the printed values; this is approximate, since it treats the surfaces as fronto-parallel, and it is not a test. Dilation (D) leaves 0.026% and 0.182% of S empty and librealsense’s footprint (E) 0.040% and 0.356%, below the planned 0.5% at both signs (H6 held for D and E). Both pay for it at the edges, with 242 wrong values in S at both signs, none of them off the edge mask.

The z-buffer. Without a z-buffer (C), at $s = +1$ the output is identical to A’s, 0 wrong values (H2a held). We expected this before the run: with the colour camera on the right, the wall samples that the box covers in the colour image lie to the box’s left in the depth image, so the row-major scan writes them first and the box overwrites them. At $s = -1$ C writes 456 wrong values against A’s 229. The plan predicted an excess of 540 ± 120 (120 box rows times the 4.5 px overlap); the printed excess is 227, and that check (H2b) failed. In our reading, whether last-write-wins gives the right answer depends on the side the colour camera sits on, which is no basis for an implementation.

Disocclusion holes. With the window measure, A’s hole in the middle row is 4 pixels at $s = +1$ and 5 at $s = -1$ , against exact widths of 5 and 4 (H4a held at both signs), and E’s is 4 and 3 (H4b held at both signs). Dilation leaves 3 and 4 pixels, one fewer than A at both signs (H5b held). The plan expected dilation to write wrong depth into disoccluded pixels at $s = -1$ and none at $s = +1$ ; D’s filled disoccluded pixels include 120 wrong ones at both signs, so H5a held at $s = -1$ and failed at $s = +1$ . A itself fills 114 disoccluded pixels at $s = +1$ , all wrong. By the construction, the box’s side edges fall exactly on depth pixel centres, $159.5 + 285 \times 0.1 = 188.0$ and $159.5 - 285 \times 0.3 = 74.0$ , and the box covers 114 depth rows (from $v = 48.25$ to $162.25$ ); the 114 filled pixels equal that row count.

Which $Z$ is stored. Footprint splatting that stores the colour camera’s depth (E) has no wrong value off the edge mask at either sign, as planned (held). Storing the depth camera’s own $Z$ , as librealsense’s align.cpp does (F), gives 10,800 and 10,560 wrong values off the edge mask, inside the planned band of 9,720 to 11,880 (H8 held at both signs). The two values differ by the rotation: $Z_c - Z_d = -\sin a\,(x - sB) + (\cos a - 1)\,Z_d$ for a rotation $a$ about the vertical axis, about 1.2 cm at the wall’s edges, where $|x| \approx 1.4$ m (our arithmetic). For a unit whose cameras are almost parallel and close together the difference is small, but it is not zero, and a user who back-projects aligned depth through the colour intrinsics should know which depth the library stored.

Ignoring the extrinsics. Projecting depth points straight through the colour camera’s intrinsics (G) shifts them horizontally by +4.8085 px on the box and +0.0636 px on the wall at $s = +1$ , and by −10.1665 and −5.9390 px at $s = -1$ . These equal the exact values a planning script computed (H7a held), and lie within 0.5 px of the first-order values +4.8819, +0.3819, −10.1181 and −5.6181 px. At $s = +1$ the rotation’s shift of about 2.6 px opposes the baseline’s, so the wall’s errors nearly cancel; at $s = -1$ they add. G writes 10,374 and 11,514 wrong values against A’s 0 and 229 (H7b held). With identity extrinsics the stored value is also the depth camera’s $Z$ , the convention F tests (construction).

Invalid pixels. A fills none of the 441 colour pixels whose surface lies on the patch of zeros (0 filled, 0 of them on the patch’s border) at either sign, so the planned check that it fills such pixels only on the border held at both signs (H9).

How the example was planned. The scene, cameras, variants, measures and hypotheses H1–H9 were written in a plan before any code ran. One planning script then evaluated the exact model only (the two ray casts, the pixel classes, the exact disoccluded widths, the exact projections of the depth pixels, the exact difference between the two cameras’ depths and the exact misalignment) without running any variant, and the criteria were completed afterwards. Because that script printed the values that the tolerances of H2b, H3a, H4, H7a and H8 rest on before those tolerances were fixed, those outcomes are illustrations of the construction rather than blind tests. Before any run, the script had two edits: an index typo in the line that classifies invalid-source pixels, which computed the same value, and the transform check’s visibility tolerance, changed from 1e-6 to 1e-9 m (with the inverse-transform line taken over all valid samples). The pilot used a first form of the hole measure that counted empty pixels starting from the pixel next to the box’s last pixel; it printed 0 whenever that one pixel held a value, whatever lay beyond. With the editor’s approval, as a repair of a broken measure, it was replaced before the first run by the window measure, with the added line that applies it to the exact model, and the pilot was repeated once; nothing else changed, and the first run is the repeated pilot unchanged, with identical output. The outcomes that changed between the two pilots are: H4a at $s = +1$ (A’s hole 0, failed, then 4, held); H4b at $s = +1$ (E’s hole 0, failed, then 4, held) and at $s = -1$ (0, failed, then 3, held); and H5b at $s = +1$ (D 0 against A 0, failed, then D 3 against A 4, held) and at $s = -1$ (D 0 against A 5, failed, then D 4 against A 5, held). The example departs from the batch plan’s description in eight ways:

It runs both baseline signs, because by our reasoning the no-z-buffer outcome depends on the scan direction relative to the baseline.
It adds librealsense’s corner footprint with both stored- $Z$ conventions (E, F).
Truncation (B) is our variant, not a statement about OpenCV.
It adds the crack, filled-hole, invalid-pixel and sub-pixel-offset measures and the data checks.
It adds an invalid patch.
Depth is stored as 16-bit millimetres, the convention registerDepth assumes.
The identity-extrinsics prediction includes the rotation term.
The cameras, baseline, rotation and scene are our choices, without lens distortion.

Strengths

Colour and depth in one frame. One device gives a colour image and metric depth of the same scene at video rate; Su et al. (2018) credit RGB-D data with enabling high-fidelity scene reconstruction and large 3D datasets.
Active depth works without texture. Each depth engine supplies its own light, so walls without texture still give depth. Keselman et al. (2017, §3.3.1) report that the R200’s projector and its ability to use sunlight let it work both at 0 lux and in broad daylight.
Factory calibration. Units are calibrated in the factory (Keselman et al., 2017, §3), and librealsense exposes each stream’s intrinsics and the extrinsics between streams, so registration can use the stored parameters.
Low cost and small size. Keselman et al. describe the R200 modules as about 100 × 10 × 4 mm and powered over one USB connector (§3); Fanello et al. (2016) attribute the spread of structured-light sensors partly to their affordability.

Limitations

Every depth engine’s own failure modes. Zhang et al. (2018) summarize them: time-of-flight systems suffer from motion artifacts and multipath interference, structured light is vulnerable to ambient illumination and interference between devices, and passive stereo struggles in textureless regions. The sibling articles discuss each.
Registration artifacts. Disocclusion holes beside every foreground edge, cracks where the colour camera samples more finely, and wrong values where implementations fill holes or skip the z-buffer, as the example shows; the librealsense documentation warns that aligned streams are synthetic and suffer from sampling and occlusion artifacts.
Timing. Colour and depth may come from different shutters and instants: the R200’s colour camera is a rolling shutter synchronized at start of frame, and TUM RGB-D’s Kinect streams were about 20 ms apart on average.
Missing depth. Dark, shiny, transparent, too near or too far surfaces return no depth (depth map); NYU Depth v2’s documentation attributes missing Kinect depth to emitter–camera shadows and to specular or low-albedo surfaces.
Range. For an active stereo engine, Keselman et al. (2017, Eq. 2) give a depth error that grows with the square of distance, and they analyse the R200’s range limits in §4.2.

Typical Applications

Dense reconstruction and RGB-D SLAM. KinectFusion fuses depth from a handheld depth camera alone; RGB-D SLAM systems are evaluated on TUM RGB-D.
Datasets. NYU Depth v2 recorded indoor scenes with a Kinect, and ScanNet recorded 1,513 scans of 707 indoor spaces with a Structure sensor; 3D reconstruction describes the pipelines built on such data.
Other uses named by sources read here. Su et al. (2018) list action recognition, facial and pose tracking, human–computer interaction, robotics and augmented and virtual reality; Keselman et al. (2017) mention hand-pose tracking, volumetric reconstruction and measuring object dimensions, and robots and drones that carry the R200.

Comparison with Other Sensors

Sensor	Output	Relation to an RGB-D camera
Time-of-flight camera	Range and amplitude from one viewpoint	A depth engine; with a colour camera it becomes an RGB-D unit
Structured light	Depth from a projector–camera pair	A depth engine (the first Kinect)
Stereo camera	Disparity and depth in the reference camera’s frame, plus its image	Its reference image is already registered to the depth; a separate colour camera is not
LiDAR	Sparse range along scanned beams	Registered to cameras the same way, by projecting points through calibrated extrinsics
Colour camera alone	Colour only	Depth must be estimated, for example by monocular depth estimation

In our reading, the RGB-D camera is less a sensing principle than a packaging of one: its depth comes from one of the active engines above, and what it adds is a second, calibrated viewpoint whose images are useful only after registration.