Depth Map

A depth map is an image in which each pixel stores how far away the scene is along that pixel’s line of sight, instead of how bright it is. Depth maps are what stereo matching, depth cameras, and monocular depth networks produce, and what 3D reconstruction and robot navigation consume. Combined with the camera’s intrinsic parameters, a depth map is a 3D measurement: every valid pixel can be lifted to a point in space.

Definition

A depth map is a function $Z(u, v)$ over the pixel grid of a camera that gives, for each pixel, the distance of the visible surface point from the camera. In computer vision the stored quantity is usually the depth $Z$, the coordinate of the point along the camera’s optical axis, not its straight-line distance from the camera center. A map of straight-line distances is often called a range image, though the two names are also used interchangeably, so a dataset’s documentation has to say which quantity it stores.

A depth map records only the nearest surface along each ray, as seen from one viewpoint. Hidden surfaces, the back of objects, and anything outside the field of view are absent.

Intuition

Picture a photograph in which every pixel holds a tape-measure reading instead of a color. The image keeps the camera’s perspective: it is a picture of the scene, not a 3D model of it. This is why depth maps are sometimes called 2.5D. They carry three-dimensional information, but only for the visible surfaces and only in the viewer’s own frame.

The name echoes Marr (1982), who described an intermediate stage of visual processing, the 2½-D sketch, as a viewer-centered representation of the visible surfaces: their orientation and rough distance from the viewer, and the contours where these change abruptly. He traces the idea to Marr and Nishihara (1978). It was a stage in a theory of human vision, not a data format, but a depth map is the same kind of viewer-centered surface description.

Formal Definition

Back-projection

For a pinhole camera with intrinsic matrix $K$, a pixel $\mathbf{x} = (u, v, 1)^\top$ in homogeneous coordinates defines a ray whose direction is the normalized image point $\hat{\mathbf{x}} = K^{-1}\mathbf{x} = (\hat{x}, \hat{y}, 1)^\top$ (camera parameters). Because the third coordinate of $\hat{\mathbf{x}}$ is 1, scaling it by the depth gives the point at that depth on the ray:

$$\mathbf{X} = Z(u, v)\, K^{-1} \mathbf{x}.$$

With zero skew this is

$$X = \frac{(u - c_x)\,Z}{f_x}, \qquad Y = \frac{(v - c_y)\,Z}{f_y}, \qquad Z = Z(u, v).$$

The point is in the camera frame; the camera’s extrinsics move it into a world frame. Applying this to every valid pixel turns a depth map into a set of 3D points, a point cloud, organized on the image grid. The reverse operation, projecting a point cloud through $K$ and keeping the nearest point per pixel, renders a depth map. Back-projection needs the depth: on its own, a pixel back-projects only to a ray (Hartley and Zisserman, 2004).

Depth versus range

The range $r$ of the same point is its Euclidean distance from the camera center:

$$r = \|\mathbf{X}\| = Z\,\|K^{-1}\mathbf{x}\| = Z\sqrt{1 + \hat{x}^2 + \hat{y}^2} = \frac{Z}{\cos\theta},$$

where $\theta$ is the angle between the pixel’s ray and the optical axis. The two agree at the principal point and differ increasingly toward the image corners. Converting between them needs only $K$. Reading a range image as depth, or the reverse, bends a flat wall into a curved surface.

Properties

• Sources. Depth maps come from stereo vision, where stereo matching produces a disparity map and depth follows as $Z = fB/d$; from RGB-D cameras based on structured light, active stereo, or time of flight; from LiDAR scans projected into a camera; from multi-view stereo; and from monocular depth estimation networks, whose output is often defined only up to an unknown scale, or scale and shift. A disparity map is thus inverse depth up to the rig’s constant $fB$.
• Missing values. Real depth maps have holes: surfaces that are too dark, shiny, transparent, too near, or too far; regions visible to the camera but not to the projector or second camera; and pixels a matcher rejected. Holes are marked with a sentinel, most often 0, sometimes NaN, and must be masked before back-projection or averaging. Ground truth can be sparse as well: KITTI’s stereo ground truth, from laser scans projected into the image, covers about half the pixels.
• Storage. Depth is often stored as a 16-bit integer image with a fixed scale. The TUM RGB-D benchmark (Sturm et al., 2012), recorded with a Microsoft Kinect at 640 × 480, stores depth as 16-bit PNG files scaled by 5000, so a stored value of 5000 means 1 m, and 0 means no data. NYU Depth v2 (Silberman et al., 2012), also recorded with a Kinect, distributes its 1,449 labeled frames in a MATLAB file as depth arrays in meters, both raw and with missing values filled in by a colorization-based in-painting method. The scale factor differs between datasets, and a 16-bit integer bounds both the resolution and the maximum depth.
• Depth discontinuities. Depth is piecewise smooth, with sharp jumps at object boundaries. Anything that mixes neighboring pixels across a jump, whether sensor optics, a matching window, or bilinear resizing, produces values between foreground and background that belong to neither surface. Back-projected, they become flying pixels, points strung out in empty space along the viewing rays. Depth maps should be resized with nearest-neighbor sampling or edge-aware filters, and flying pixels are commonly removed by rejecting pixels whose depth differs strongly from their neighbors’.
• Visualization. Depth spans a large range, and for stereo and many active sensors its precision falls with distance, so depth maps are often displayed by color-mapping inverse depth $1/Z$, which spreads near structure over most of the color range and compresses the far background. Invalid pixels are drawn in a separate color so that they are not mistaken for very near or very far surfaces.
• Registration. An RGB-D camera’s depth sensor and color camera sit apart, so pairing color with depth means back-projecting the depth map, moving the points into the color camera’s frame, and reprojecting them with its intrinsics.

Examples

The code synthesizes the depth map of a box in front of a wall, with a hole of missing data, back-projects it with $K$, converts depth to range, encodes it with the TUM convention, and shows how averaging across the box edge creates flying pixels.

import numpy as np

# Camera intrinsics (pixels) for a 640 x 480 image.
fx = fy = 525.0
cx, cy = 319.5, 239.5
K = np.array([[fx, 0, cx], [0, fy, cy], [0, 0, 1]])
H, W = 480, 640

# Synthetic scene: a wall 3 m away with a box whose front face is 1.5 m away.
Z = np.full((H, W), 3.0)
Z[181:301, 241:401] = 1.5
Z[0:40, 0:60] = 0.0                      # a hole: no measurement (0 = invalid)

# Back-projection: X = Z * K^-1 [u, v, 1]^T for every valid pixel.
u, v = np.meshgrid(np.arange(W), np.arange(H))
pix = np.stack([u, v, np.ones_like(u)], axis=-1).astype(float)   # H x W x 3
rays = pix @ np.linalg.inv(K).T          # normalized coordinates (x/z, y/z, 1)
valid = Z > 0
points = rays[valid] * Z[valid, None]    # N x 3 point cloud in the camera frame
print(f"valid pixels: {valid.sum()} of {H * W}, points: {points.shape}")
box = points[np.isclose(points[:, 2], 1.5)]
print("box face X range (m):", box[:, 0].min().round(4), box[:, 0].max().round(4))

# Depth (along the optical axis) versus range (along the ray).
R = Z * np.linalg.norm(rays, axis=-1)
for (r, c) in [(240, 320), (479, 639)]:
    print(f"pixel (u={c}, v={r}): depth Z = {Z[r, c]:.3f} m, range = {R[r, c]:.3f} m")
Z_back = R / np.linalg.norm(rays, axis=-1)          # range -> depth
print("max |Z - Z_back|:", np.abs(Z - Z_back).max())

# Storage: 16-bit PNG convention with 5000 units per meter (TUM RGB-D).
stored = np.round(Z * 5000).astype(np.uint16)
print("stored value at 1.5 m:", stored[240, 320], " max representable depth (m):",
      np.iinfo(np.uint16).max / 5000)

# Flying pixels: bilinear 2x downsampling averages across the box edge.
Zs = 0.25 * (Z[0::2, 0::2] + Z[1::2, 0::2] + Z[0::2, 1::2] + Z[1::2, 1::2])
Zn = Z[0::2, 0::2]                                  # nearest-neighbour subsampling
between = lambda D: int(((D > 1.5 + 1e-6) & (D < 3.0 - 1e-6)).sum())
print("pixels between box and wall: averaged =", between(Zs), ", nearest =", between(Zn))

Output:

valid pixels: 304800 of 307200, points: (304800, 3)
box face X range (m): -0.2243 0.23
pixel (u=320, v=240): depth Z = 1.500 m, range = 1.500 m
pixel (u=639, v=479): depth Z = 3.000 m, range = 3.769 m
max |Z - Z_back|: 0.0
stored value at 1.5 m: 7500  max representable depth (m): 13.107
pixels between box and wall: averaged = 280 , nearest = 0

The box face spans columns 241 to 400 and back-projects to a 45 cm wide patch, as $(u - c_x)Z/f_x$ predicts. At the image center, depth and range agree; at the bottom-right corner the ray is about 37° off the axis and the range is 26% larger than the depth. With 5000 units per meter, a 16-bit image cannot represent anything beyond about 13 m. Averaging 2 × 2 blocks creates 280 pixels around the box outline with depths of 2.25 m (edges) or 2.625 m (corners), surfaces that do not exist; nearest-neighbor subsampling creates none.

Common Misconceptions

• “Depth is the distance to the camera.” In most vision datasets it is the $Z$ coordinate along the optical axis. A fronto-parallel wall has constant depth but varying range.
• “A depth map is a 3D model.” It samples only the visible surfaces from one viewpoint, more sparsely with distance.
• “Zero means very close.” Zero usually means no measurement. Treating it as a depth pulls points to the camera center and corrupts averages.

Where It Is Used

• Stereo and multi-view reconstruction. Stereo matching and plane-sweep methods built on a cost volume output per-view depth or disparity maps, which multi-view stereo fuses into a single model.
• RGB-D SLAM and dense mapping. Per-frame depth maps from depth cameras are back-projected and aligned into maps of rooms and buildings; TUM RGB-D evaluates the estimated camera trajectories.
• Monocular depth estimation. Networks predict a depth map from a single image and are trained and evaluated on datasets such as NYU Depth v2 and KITTI.
• Robotics, driving, and augmented reality. Obstacle detection, grasping, and occlusion of virtual objects by real ones operate on depth maps or on the point clouds back-projected from them.