Depth Estimation

Depth estimation recovers, for every pixel of an image, how far away the visible surface is. The answer is a depth map, which can come from two calibrated cameras, many photographs, a sensor that emits its own light, or a single image interpreted by a learned model. This article maps those routes, the kind of depth each returns, and how results are measured.

Problem Definition

Given an image from a camera with intrinsics $K$, estimate a depth map $Z(u, v)$: the depth along the optical axis of the surface seen at each pixel, with invalid pixels marked. With $K$, the depth map back-projects to a point cloud.

A pixel defines only a ray, and scaling the scene about the camera center leaves the image unchanged (pinhole camera model). Depth therefore needs a second viewpoint with a known baseline, an active measurement, or learned priors about scenes. Which one is used decides whether the output is metric (in meters), known only up to scale, or only up to scale and shift (affine-invariant).

Input and Output

• Input: a rectified stereo pair, a set of images, a raw sensor signal, a single RGB image, or an image plus sparse depth.
• Output: a dense depth map (sometimes a disparity or inverse-depth map) at image resolution, often with a validity mask or per-pixel confidence. Sparse methods return depths only at matched feature points.

Variants

• Metric versus relative depth. Relative depth fixes the ordering and shape of the scene but not its scale, and sometimes not an offset in inverse depth.
• Depth completion. A camera image guides the densification of sparse depth, typically LiDAR points projected into the image. Uhrig et al. (2017) showed that ordinary convolutions handle such sparse inputs poorly, proposed sparsity-invariant convolutions, and built from KITTI raw data the dataset behind the KITTI depth completion and depth prediction benchmarks.
• Video depth. Predictions should also be consistent over time.

Challenges

• Scale ambiguity. A single image, or a single moving camera without other sensors, determines geometry only up to a global scale. Eigen et al. (2014) call global scale a fundamental ambiguity of single-image depth: in their experiments, substituting the true mean log depth into their predictions reduced the log RMSE from 0.28 to 0.22.
• Correspondence failures. Stereo and multi-view methods fail on textureless, repetitive, reflective, or transparent surfaces and in occluded regions (stereo matching).
• Range-dependent precision. For triangulation-based methods, depth error grows roughly with the square of depth, because a fixed disparity error is a larger depth error far away (stereo vision).
• Depth discontinuities. Most methods blur foreground into background at object boundaries (depth map).
• Ground truth. Dense, accurate depth for real scenes is expensive; laser-derived ground truth is sparse and has its own artifacts.

Approaches

Stereo

Stereo vision measures depth from two cameras with a known baseline. After stereo rectification, stereo matching finds a disparity for each pixel by optimizing a cost volume, for example with semi-global matching or a deep network, and depth follows as $Z = fB/d$. The result is metric wherever matching succeeds.

Multi-view stereo and structure from motion

With more than two images, structure from motion estimates the camera poses and a sparse set of triangulated points from feature matches, refined by bundle adjustment (see multi-view geometry). Its depths are sparse and, from images alone, known only up to a global scale. Multi-view stereo then computes a dense depth map per view by testing depth hypotheses against all the calibrated images, for example by plane sweeping, and fuses the maps; more views mean more baselines and fewer occlusion failures.

Active sensing

Active sensors emit their own light and return metric depth directly:

• Structured light projects a known pattern and triangulates between the projector and a camera. The first Microsoft Kinect (2010), developed with PrimeSense, worked this way (Sarbolandi et al., 2015). Middlebury’s high-accuracy stereo ground truth was also made with structured light (Middlebury).
• Time of flight (ToF) cameras measure how long emitted light takes to return. The second Kinect, released with the Xbox One, uses continuous-wave modulation and measures the phase shift of the returning signal, so it needs no triangulation (Sarbolandi et al., 2015).
• LiDAR scans laser beams across the scene and measures range along each beam: accurate at long range but sparse, which is why driving datasets such as KITTI have sparse ground truth.

Sarbolandi et al. (2015) list the error sources of such cameras, among them ambient background light, multi-path reflections, and translucent materials.

Monocular depth estimation

Single-image methods replace geometry with priors such as perspective, object sizes, and position in the image. Each stage below answered a limit of the previous one.

Hand-crafted features. Make3D (Saxena, Sun, and Ng, 2009) divided the image into superpixels and used a Markov random field, trained with supervised learning on image features, to infer the 3D position and orientation of each small planar patch. Eigen et al. (2014) noted that it relied on horizontally aligned images and suffered in less controlled settings.

Supervised deep regression. Eigen et al. (2014) regressed depth directly with a two-scale convolutional network (a coarse network over the whole image, then a fine network for local detail), trained on Kinect depth from NYU Depth v2 and LiDAR depth from KITTI. They added a scale-invariant log error, and their set of metrics was adopted by much later work.

Self-supervision from stereo and video. Because ground-truth depth is hard to record, later work trained on images alone, with view synthesis as the signal. Garg et al. (2016) trained a network to predict depth for one image of a stereo pair so that warping the other image with it reconstructs the first. Godard, Mac Aodha, and Brostow (2017, Monodepth) used bilinear sampling to make the reconstruction fully differentiable and added a left-right disparity consistency loss. Zhou et al. (2017, SfMLearner) needed only monocular video: a depth network and a pose network are trained jointly so that nearby frames warp onto the target frame. A moving monocular camera fixes no scale, so such depths are evaluated after median scaling; stereo-trained models learn scale from the baseline. Monodepth2 (Godard et al., 2019) improved monocular training with a per-pixel minimum reprojection loss for occlusions and auto-masking of pixels that violate the moving-camera assumption.

Relative depth from mixed data. Single datasets are biased, and they disagree on representation: metric depth, depth up to scale, or disparity up to scale and shift. Ranftl et al. (2022, MiDaS) predicted inverse depth and trained with losses invariant to scale and shift. This let them mix five training sources, including stereo frames from 3D movies, and test by zero-shot transfer to unseen datasets. The output is affine-invariant inverse depth, which must be aligned to be metric.

Foundation models and metric depth. Depth Anything (Yang et al., 2024) kept the affine-invariant formulation but scaled the data: a teacher trained on about 1.5 million labeled images pseudo-labeled about 62 million unlabeled images for self-training. To recover meters, ZoeDepth (Bhat et al., 2023) pre-trained a MiDaS-style relative model on 12 datasets and fine-tuned metric heads on NYU Depth v2 and KITTI. Metric3D (Yin et al., 2023) traced the cross-dataset metric ambiguity mainly to focal length and trained in a canonical camera space, so that a known focal length lets the model output metric depth for different cameras.

Other cues

Depth from focus sweeps the focus setting and assigns each pixel the depth at which it is sharpest; depth from defocus infers depth from the amount of blur. Shape from shading recovers surface orientation from intensity under assumed lighting and reflectance.

Comparison

Family	Input	Output scale	Typical sensors	Article
Stereo	Rectified image pair, calibration	Metric, dense	Stereo camera rig	Stereo matching
Multi-view stereo	Many images with known poses	Metric if poses are; else up to scale	One moving camera or several	Multi-view stereo
Structure from motion	Unordered images or video	Sparse, up to global scale	Single camera	Structure from motion
Active sensing	The sensor’s own signal	Metric, dense or sparse	Structured light, ToF, LiDAR	Depth map
Monocular, self-supervised	Single image (trained on stereo or video)	Metric (stereo-trained) or up to scale	Single camera	This article
Monocular, relative	Single image	Affine-invariant inverse depth	Single camera	This article
Monocular, metric	Single image, sometimes intrinsics	Metric, within the training domains	Single camera	This article
Depth completion	Image plus sparse depth	Metric, dense	Camera plus LiDAR	This article

Datasets and Benchmarks

• KITTI. Driving scenes with LiDAR. The Eigen split from Eigen et al. (2014), described by later work as 697 test images from 29 scenes (Godard et al., 2017), uses raw LiDAR points projected into the image, covering under 5% of pixels, and is usually evaluated with depth capped at 80 m. The later KITTI depth benchmark (Uhrig et al., 2017) has about 93,000 frames with semi-dense ground truth made by accumulating 11 laser scans and removing points inconsistent with a semi-global matching reconstruction.
• NYU Depth v2 (Silberman et al., 2012). 464 indoor scenes recorded with a Kinect, with a densely labeled subset of 1,449 frames; monocular methods are usually tested on the 654 test images of its official split (Ranftl et al., 2022).
• Middlebury. High-resolution indoor stereo pairs with structured-light ground truth, used to evaluate stereo accuracy.
• ETH3D (Schöps et al., 2017). A multi-view stereo benchmark of indoor and outdoor scenes with laser-scanned ground truth, also used for zero-shot tests of monocular models (Ranftl et al., 2022).

Evaluation Metrics

Stereo benchmarks score disparity (stereo matching). Monocular and completion methods are scored in depth, over the set $T$ of pixels with valid ground truth $z_i$ and prediction $\hat{z}_i$. Eigen et al. (2014) collected these metrics from earlier work and added the scale-invariant error:

$$\text{AbsRel} = \frac{1}{|T|}\sum_{i \in T} \frac{|\hat{z}_i - z_i|}{z_i}, \qquad \text{SqRel} = \frac{1}{|T|}\sum_{i \in T} \frac{(\hat{z}_i - z_i)^2}{z_i},$$ $$\text{RMSE} = \sqrt{\frac{1}{|T|}\sum_{i \in T} (\hat{z}_i - z_i)^2}, \qquad \text{RMSE}_{\log} = \sqrt{\frac{1}{|T|}\sum_{i \in T} (\log \hat{z}_i - \log z_i)^2},$$ $$\delta_k = \frac{1}{|T|}\,\Big|\Big\{ i \in T : \max\Big(\frac{\hat{z}_i}{z_i}, \frac{z_i}{\hat{z}_i}\Big) < 1.25^k \Big\}\Big|, \quad k = 1, 2, 3.$$

Lower is better for the errors; higher is better for the threshold accuracies $\delta_k$. The scale-invariant error, with $e_i = \log \hat{z}_i - \log z_i$, is $D = \frac{1}{|T|}\sum_{i} e_i^2 - \big(\frac{1}{|T|}\sum_{i} e_i\big)^2$, the variance of the log error: multiplying the prediction by any constant leaves it unchanged. KITTI’s depth prediction benchmark ranks methods by $\sqrt{D}$, reported as SILog (KITTI).

Predictions that are not metric must be aligned before these metrics mean anything:

• Median scaling (Zhou et al., 2017) multiplies each predicted map by $\text{median}(z)/\text{median}(\hat{z})$. It suits up-to-scale outputs. Monodepth2 reports monocular models this way and stereo-trained ones without scaling.
• Scale-and-shift alignment (Ranftl et al., 2022) suits affine-invariant outputs. With predicted inverse depth $d_i$, it solves $\min_{s,t} \sum_i (s\,d_i + t - 1/z_i)^2$ in closed form, then evaluates $1/(s\,d_i + t)$ as depth.

Both alignments use the ground truth, so aligned numbers measure shape, not the ability to recover scale, and should not be compared with unaligned metric results.

The code computes all the metrics for two synthetic predictions of a road scene, with ground truth sampled at 5% of pixels as from a LiDAR: prediction A is correct up to an unknown scale, and prediction B is affine-invariant inverse depth.

import numpy as np

def depth_metrics(pred, gt):
    """Standard depth metrics over valid pixels (both arrays 1D, in meters)."""
    ratio = np.maximum(pred / gt, gt / pred)
    log_diff = np.log(pred) - np.log(gt)
    return {
        "AbsRel": np.mean(np.abs(pred - gt) / gt),
        "SqRel": np.mean((pred - gt) ** 2 / gt),
        "RMSE": np.sqrt(np.mean((pred - gt) ** 2)),
        "RMSElog": np.sqrt(np.mean(log_diff ** 2)),
        "SIlog": np.sqrt(np.mean(log_diff ** 2) - np.mean(log_diff) ** 2),
        "d1": np.mean(ratio < 1.25),
        "d2": np.mean(ratio < 1.25 ** 2),
        "d3": np.mean(ratio < 1.25 ** 3),
    }

rng = np.random.default_rng(0)
H, W, f, cam_height, max_depth = 192, 640, 720.0, 1.65, 80.0
cy = H / 2

# Ground-truth depth of a road scene: a ground plane plus a box-shaped car 12 m away.
v = np.arange(H)[:, None] * np.ones((1, W))
gt = np.full((H, W), np.inf)                         # sky: no return
below = v > cy + 1
gt[below] = f * cam_height / (v[below] - cy)         # ground plane: Z = f h / (v - c_y)
gt[60:110, 250:400] = 12.0                            # vertical face of a car
lidar = rng.random((H, W)) < 0.05                     # sparse ground truth, as from a LiDAR
valid = lidar & (gt > 1e-3) & (gt < max_depth)
z = gt[valid]

# Prediction A: correct up to an unknown global scale (e.g. monocular self-supervision).
pred_a = 0.31 * z * np.exp(rng.normal(0, 0.08, z.size))
# Prediction B: affine-invariant inverse depth (e.g. a MiDaS-style relative model).
disp_b = 7.0 / z + 0.9 + rng.normal(0, 0.02, z.size)

def median_scale(pred):
    return pred * np.median(z) / np.median(pred)

def scale_shift_inverse_depth(disp):
    """Least-squares s, t with s*disp + t ~ 1/gt; return the aligned depth."""
    A = np.stack([disp, np.ones_like(disp)], axis=1)
    (s, t), *_ = np.linalg.lstsq(A, 1.0 / z, rcond=None)
    aligned = np.clip(s * disp + t, 1.0 / max_depth, None)
    return 1.0 / aligned, s, t

pred_b_ss, s, t = scale_shift_inverse_depth(disp_b)
rows = [
    ("A raw", pred_a),
    ("A median-scaled", median_scale(pred_a)),
    ("B 1/disp, median", median_scale(1.0 / disp_b)),
    ("B scale+shift", pred_b_ss),
]
print(f"valid ground-truth pixels: {valid.sum()} of {H * W} ({valid.mean():.1%})")
print(f"fitted inverse-depth alignment: s = {s:.4f}, t = {t:.4f}")
names = list(depth_metrics(z, z))
print(f"{'':18s}" + "".join(f"{n:>9s}" for n in names))
for label, p in rows:
    p = np.clip(p, 1e-3, max_depth)                   # cap predictions as in KITTI practice
    m = depth_metrics(p, z)
    print(f"{label:18s}" + "".join(f"{m[n]:9.3f}" for n in names))

Output:

valid ground-truth pixels: 3018 of 122880 (2.5%)
fitted inverse-depth alignment: s = 0.1406, t = -0.1256
                     AbsRel    SqRel     RMSE  RMSElog    SIlog       d1       d2       d3
A raw                 0.688   12.053   20.425    1.171    0.080    0.000    0.000    0.000
A median-scaled       0.064    0.157    2.269    0.080    0.080    0.996    1.000    1.000
B 1/disp, median      0.275    3.903   14.078    0.403    0.391    0.430    0.802    0.889
B scale+shift         0.056    0.252    3.490    0.080    0.080    0.979    1.000    1.000

Only 2.5% of pixels are valid, because the sky and the road beyond 80 m have no ground truth. Prediction A is off by a factor of 0.31, so every raw metric fails while the scale-invariant error (SIlog, the square root of $D$) is already 0.080; median scaling fixes the rest. Prediction B carries an offset in inverse depth, so median scaling leaves $\delta_1$ at 0.43. Least-squares alignment recovers $s \approx 1/7$ and $t \approx -0.9/7$, undoing the simulated distortion. Small noise in inverse depth still becomes large errors in meters for distant points, which dominate RMSE.

Applications

• Driving and robotics. Obstacle detection and mapping from stereo rigs and LiDAR, with monocular depth as a cheap complement.
• Visual odometry and SLAM. Depth from RGB-D cameras or networks initializes and densifies maps.
• 3D reconstruction. Multi-view stereo depth maps are fused into meshes and point clouds.
• Augmented reality and photography. Occlusion of virtual objects by real ones and synthetic depth of field.

Open Problems

• Metric depth from one image across arbitrary cameras. Metric3D shows that focal length matters; images without reliable intrinsics remain ambiguous.
• Hard surfaces. Glass, mirrors, water, and thin structures defeat stereo, active sensors, and learned priors alike.
• Temporal consistency of per-frame predictions in video.
• Evaluation. Sparse ground truth, depth caps, crops, and alignment choices make numbers from different papers hard to compare.