Monocular Depth Estimation

Monocular depth estimation predicts a depth map from one RGB image. The depth estimation hub gives the overview and history; this article covers what one image can determine, the losses that train networks anyway, and the protocols behind reported numbers.

Problem Definition

A network predicts a depth $\hat z_i$, or an inverse depth $\hat d_i = 1/\hat z_i$, for every pixel $i$. Depending on training, the output is metric depth in meters, depth up to scale ($\hat z = s z$ for an unknown $s > 0$), or affine-invariant inverse depth ($\hat d = s/z + t$ for unknown $s$ and $t$).

Why one image fixes depth only up to scale

Under the pinhole model, a point $\mathbf{X} = (X, Y, Z)$ in camera coordinates projects to $u = f_x X/Z + c_x$, $v = f_y Y/Z + c_y$. Replacing $\mathbf{X}$ by $s\mathbf{X}$ for any $s > 0$ leaves $(u, v)$ unchanged. It follows that a whole scene scaled about the camera center, with the same surface colours and light directions, produces the same image: no function of the pixels alone can tell a doll’s house from a house, and geometry fixes depth only up to one global factor.

A network breaks the ambiguity only through priors. For an object of known physical size $S$ that spans $S'$ pixels, similar triangles give

$$Z = \frac{f\,S}{S'},$$

with $f$ and $S'$ in pixels (Yin et al., 2023, Eq. 1, give it in physical units). Metric depth needs both a size prior, learned from semantics, and the focal length. If the focal length is unknown, $f$ and $Z$ trade off: a camera with twice the focal length at twice the distance images the object at the same size (Yin et al., 2023, observation O2). Training on many cameras therefore pairs similar images with conflicting labels.

Training data adds a second ambiguity. Stereo footage with unknown calibration gives disparity $d = fB/Z + c$, where the baseline $B$ is unknown and a horizontal principal-point offset adds a constant $c$. Inverse depth is then known up to scale and shift (Ranftl et al., 2022, Section 5), which is why they predict affine-invariant inverse depth.

Input and Output

The input is one RGB image, plus its focal length for some metric models. The output is a dense map at image resolution; inverse-depth outputs are often called disparity even without a stereo pair.

Variants

Relative models predict affine-invariant inverse depth; metric models predict meters, within one domain or zero-shot across cameras. Either can be supervised or self-supervised.

Challenges

• Sparse, biased ground truth. KITTI’s LiDAR covers under 5% of pixels (Godard et al., 2017); NYU Depth v2 is indoor only.
• Learned shortcuts. Ranftl et al. (2022) found that their model learned that lower image regions are closer, and estimated the depth of the scene shown in a painting or mirror instead of the surface itself.
• Violated assumptions. Moving objects and occlusions break self-supervised photometric losses.

Approaches

Supervised regression and the scale-invariant loss

Eigen, Puhrsch and Fergus (2014) trained a coarse-to-fine CNN on NYU Depth v2 and KITTI, masking pixels without ground truth. With $d_i = \log \hat z_i - \log z_i$ over $n$ valid pixels, they defined the scale-invariant error and the training loss

$$D(\hat z, z) = \frac{1}{n}\sum_i d_i^2 - \frac{1}{n^2}\Big(\sum_i d_i\Big)^2, \qquad L = \frac{1}{n}\sum_i d_i^2 - \frac{\lambda}{n^2}\Big(\sum_i d_i\Big)^2 .$$

$D$ is the variance of the log error, unchanged when every $\hat z_i$ is multiplied by one constant; an equivalent form compares log-depth differences of all pixel pairs (Eigen et al., 2014, Eqs. 1–3). With $\lambda = 0$ the loss is elementwise $\ell_2$ on log depth, and with $\lambda = 1$ it is $D$. They trained with $\lambda = 0.5$, reporting good absolute scale and slightly better qualitative output (Eq. 4). Because $\lambda < 1$, a global scale error is still penalized, so the network learns the scale of its training domain.

Scale-and-shift-invariant losses

To mix metric, up-to-scale and up-to-scale-and-shift training data, Ranftl et al. (2022, arXiv version, Eqs. 1–7) predict inverse depth $d \in \mathbb{R}^M$ and align it to the ground truth $d^*$ before comparing. The least-squares alignment is

$$(s, t) = \arg\min_{s,t} \sum_{i=1}^{M} \big(s\,d_i + t - d_i^*\big)^2, \qquad \mathcal{L}_{\text{ssi}} = \frac{1}{2M}\sum_{i=1}^{M} \rho\big(s\,d_i + t - d_i^*\big),$$

which has a closed-form solution, with $\rho(x) = x^2$ for the MSE variant. Their robust variant normalizes prediction and ground truth separately, with $t(d) = \operatorname{median}(d)$ and $s(d) = \frac{1}{M}\sum_i |d_i - t(d)|$, and uses absolute residuals while discarding the 20% largest in each image ($\mathcal{L}_{\text{ssitrim}}$). A multi-scale gradient-matching term (weight 0.5) sharpens discontinuities. Unlike Eigen’s loss, which handles only scale, this one also absorbs a shift in disparity.

Self-supervision from stereo pairs

Godard, Mac Aodha and Brostow (2017) trained on rectified stereo pairs without depth labels. From the left image alone the network predicts disparities for both views; the right image, resampled with the left disparity by differentiable bilinear sampling, must reconstruct the left image under an SSIM + L1 photometric loss:

$$C_{\text{ap}}^{l} = \frac{1}{N}\sum_{i,j} \alpha\,\frac{1 - \operatorname{SSIM}(I^l_{ij}, \tilde I^l_{ij})}{2} + (1 - \alpha)\,\big|I^l_{ij} - \tilde I^l_{ij}\big|,$$

with a simplified SSIM over $3 \times 3$ blocks and $\alpha = 0.85$ (Eq. 2). A left-right consistency loss makes the left disparity equal the right one at the matched position, $C_{\text{lr}}^l = \frac{1}{N}\sum_{i,j} |d^l_{ij} - d^r_{ij + d^l_{ij}}|$ (Eq. 4), with an edge-aware smoothness term. With baseline $b$ and focal length $f$ known, depth $fb/d$ is metric.

The example renders a textured plane 4 m from a stereo rig with a 10 cm baseline, warps the right image to the left for candidate depths, and evaluates this loss; it then scales depth and baseline together, as monocular training must.

import numpy as np
from scipy.ndimage import uniform_filter

# A textured fronto-parallel plane at Z = 4 m seen by a rectified stereo pair.
H, W, f, B, Z_true = 64, 96, 150.0, 0.10, 4.0
rng = np.random.default_rng(0)
freqs = rng.uniform(2, 12, (12, 2)) * rng.choice([-1, 1], (12, 2))
phases = rng.uniform(0, 2 * np.pi, 12)

def texture(X, Y):                                  # albedo painted on the plane (X, Y in m)
    return 0.5 + 0.04 * np.sum(np.cos(X[..., None] * freqs[:, 0]
                                      + Y[..., None] * freqs[:, 1] + phases), -1)

u, v = np.meshgrid(np.arange(W, dtype=float), np.arange(H, dtype=float))
def render(cam_x):                                  # camera at (cam_x, 0, 0), looking along +Z
    X = (u - W / 2) / f * Z_true + cam_x
    Y = (v - H / 2) / f * Z_true
    return texture(X, Y)
I_left, I_right = render(0.0), render(B)

def warp_from_right(I_r, depth, baseline):          # left pixel u sees right pixel u - f B / Z
    shift = f * baseline / depth
    return np.array([np.interp(u[0] - shift, u[0], row) for row in I_r])

def ssim(a, b, C1=0.01**2, C2=0.03**2):             # 3x3 block filter, as in Monodepth
    mu_a, mu_b = uniform_filter(a, 3), uniform_filter(b, 3)
    s_a = uniform_filter(a * a, 3) - mu_a**2
    s_b = uniform_filter(b * b, 3) - mu_b**2
    s_ab = uniform_filter(a * b, 3) - mu_a * mu_b
    return ((2 * mu_a * mu_b + C1) * (2 * s_ab + C2)
            / ((mu_a**2 + mu_b**2 + C1) * (s_a + s_b + C2)))

def pe(a, b, alpha=0.85):                           # SSIM + L1 photometric error per pixel
    return alpha / 2 * (1 - ssim(a, b)) + (1 - alpha) * np.abs(a - b)

def loss(depth, baseline=B):                        # mean over pixels whose source lies inside
    return pe(I_left, warp_from_right(I_right, depth, baseline))[2:-2, 12:-2].mean()

print(f"true disparity f*B/Z = {f * B / Z_true:.2f} px")
for d in [2.0, 3.0, 3.5, 3.9, 4.0, 4.1, 4.5, 6.0, 10.0]:
    print(f"candidate depth {d:5.1f} m  ->  photometric loss {loss(d):.5f}")

# Monocular video: the translation is predicted too, so (s * depth, s * translation)
# produces exactly the same warp for every s.
for s in [0.5, 2.0, 10.0]:
    print(f"depth x {s:4}, translation x {s:4}: loss {loss(s * Z_true, s * B):.5f}")

Output:

true disparity f*B/Z = 3.75 px
candidate depth   2.0 m  ->  photometric loss 0.11285
candidate depth   3.0 m  ->  photometric loss 0.01753
candidate depth   3.5 m  ->  photometric loss 0.00412
candidate depth   3.9 m  ->  photometric loss 0.00036
candidate depth   4.0 m  ->  photometric loss 0.00007
candidate depth   4.1 m  ->  photometric loss 0.00035
candidate depth   4.5 m  ->  photometric loss 0.00277
candidate depth   6.0 m  ->  photometric loss 0.01737
candidate depth  10.0 m  ->  photometric loss 0.04824
depth x  0.5, translation x  0.5: loss 0.00007
depth x  2.0, translation x  2.0: loss 0.00007
depth x 10.0, translation x 10.0: loss 0.00007

With the baseline known, the loss has a clear minimum at the true 4 m (the residual there is linear-interpolation error). When the translation is unknown, scaled depth and translation give the same loss.

Self-supervision from monocular video

Zhou et al. (2017) replaced the stereo partner with neighbouring video frames: a depth network predicts $\hat D_t$, a pose network predicts the motion $\hat T_{t \to s}$ to each source frame, and each target pixel $p_t$ is warped by

$$p_s \sim K\,\hat T_{t \to s}\,\hat D_t(p_t)\,K^{-1} p_t$$

(Zhou et al., 2017, Eq. 2) to sample the source image. Writing $\hat T_{t \to s} = [R \mid \mathbf{t}]$ (mapping target to source camera coordinates, $X_s = R X_t + \mathbf{t}$), the warped point is $R\,\hat D_t(p_t) K^{-1}p_t + \mathbf{t}$. Multiplying both $\hat D_t$ and $\mathbf{t}$ by $s$ multiplies this 3D point by $s$, which projection removes (our derivation). The loss is blind to scale, as the last three printed lines show.

Monodepth2 (Godard et al., 2019) kept the SSIM + L1 error $\text{pe}$ with $\alpha = 0.85$ and added two changes. The per-pixel minimum reprojection loss $L_p = \min_{t'} \text{pe}(I_t, I_{t' \to t})$ takes the best source frame per pixel instead of the average, so a pixel occluded in one frame is matched in another (Eq. 4). Auto-masking keeps a pixel only if warping lowers its error below that of the unwarped source, $\mu = [\min_{t'} \text{pe}(I_t, I_{t' \to t}) < \min_{t'} \text{pe}(I_t, I_{t'})]$ (Eq. 5), which drops static-camera frames and objects moving with the camera that would otherwise be learned as infinitely far.

Foundation models

As of October 2026, general-purpose models are large networks trained on millions of images:

• DPT (Ranftl, Bochkovskiy and Koltun, 2021) replaced the convolutional encoder with a vision transformer and trained with the MiDaS loss on about 1.4 million images (MIX 6), reporting zero-shot gains over MiDaS of up to 28% in relative performance.
• Depth Anything (Yang et al., 2024) trained a teacher on 1.5M labeled images from six datasets, used it to pseudo-label 62M unlabeled images, and trained a student on both, with a DINOv2-initialized encoder and the affine-invariant loss (Table 1).
• Depth Anything V2 (Yang et al., NeurIPS 2024) replaced the labeled real images with 595K synthetic images from five datasets, trained a DINOv2-G teacher on them, and trained students of 25M to 1.3B parameters on 62M pseudo-labeled real images (Table 7).

Metric models attack the focal-length ambiguity. Metric3D (Yin et al., 2023) assumes a known focal length and maps every training image to a canonical camera, by scaling the depth label by $f_c/f$ or resizing the image, undoing it at test time; it trained on 8 million images from 11 datasets. Depth Pro (Bochkovskii et al., 2025) predicts canonical inverse depth $C$ and converts it with $D_m = f_{\text{px}}/(wC)$, where $w$ is the image width, estimating the horizontal field of view with a separate head when no focal length is available. UniDepth (Piccinelli et al., 2024) predicts a dense camera representation with the depth and outputs metric 3D points without intrinsics.

Datasets and Benchmarks

• NYU Depth v2 (Silberman et al., 2012): Kinect video of 464 indoor scenes. Eigen et al. (2014) used the official 249/215 scene split and trained on about 120K raw frames. The standard test set has 654 images (Ranftl et al., 2022); Eigen et al. (2014) print 694.
• KITTI: sparse depth from a rotating LiDAR. Eigen et al. (2014) split 56 raw scenes into 28 for training and 28 for testing. Godard et al. (2017) describe the test set, now called the Eigen split, as 697 images from 29 scenes.

Evaluation Metrics

Eigen et al. (2014, Section 4.3) used five metrics from earlier work plus $D$: AbsRel, SqRel, RMSE, RMSE log and the threshold accuracy $\delta < 1.25^k$, defined in the hub. The protocol matters as much:

• Depth cap and crop. Godard et al. (2017) capped KITTI predictions at 80 m, about the dataset’s maximum depth, and their Table 2 caption says all results except Eigen’s use the crop of Garg et al. (2016), while their text says they used Eigen’s crop. ZoeDepth (arXiv v1, Table 7) lists the later conventions: NYU Depth v2, 654 images, 0.001–10 m, “Eigen” crop; KITTI, 697 images, 0.001–80 m, “Garg” crop.
• Median scaling. Zhou et al. (2017) multiply each prediction by $\operatorname{median}(z)/\operatorname{median}(\hat z)$; Monodepth2 does so for monocular models only. Its supplementary Section D.2 (arXiv v4) notes that per-image scaling hides inconsistent scale and reports the spread of per-image scales: a standard deviation of 0.093 for Monodepth2 and 0.210 for Zhou et al. (Table 8).
• Scale-and-shift alignment. Ranftl et al. (2022) fit $s$ and $t$ per image by least squares in inverse depth, cap depth (80 m for KITTI, 10 m for NYU), and report $\delta > 1.25$, the percentage of pixels outside the threshold. Depth Anything also aligns scale and shift for its zero-shot results, following MiDaS (supplementary material, Section 1).

The example ray-casts the depth of a room (floor, back wall at 15 m, two spheres), adds 5% log-normal noise, and builds three predictions: noise only, a global scale of 0.4, and distorted inverse depth $1/\hat z = 2/z + 0.3$. Each is scored unaligned, median-scaled, least-squares scaled ($\min_s \sum (s\hat z - z)^2$), and scale-and-shift aligned in inverse depth.

import numpy as np

# Ray-cast a depth map of a synthetic room: floor, back wall, two spheres.
H, W, f = 120, 160, 150.0
u, v = np.meshgrid(np.arange(W) + 0.5, np.arange(H) + 0.5)
x, y = (u - W / 2) / f, (v - H / 2) / f          # ray (x, y, 1): its parameter is depth Z
Z = np.full((H, W), 15.0)                         # back wall at Z = 15 m
floor = y > 0
Z[floor] = np.minimum(Z[floor], 1.5 / y[floor])   # floor 1.5 m below the camera (y points down)
for c, r in [((-1.0, 0.5, 5.0), 1.0), ((1.5, 0.0, 9.0), 1.5)]:
    b = x * c[0] + y * c[1] + c[2]
    a = x**2 + y**2 + 1
    disc = b**2 - a * (np.dot(c, c) - r**2)
    hit = disc > 0
    t = (b - np.sqrt(np.where(hit, disc, 0))) / a
    Z = np.where(hit & (t > 0), np.minimum(Z, t), Z)
z = Z.ravel()

def metrics(p, g):
    ratio = np.maximum(p / g, g / p)
    return {"AbsRel": np.mean(np.abs(p - g) / g), "SqRel": np.mean((p - g) ** 2 / g),
            "RMSE": np.sqrt(np.mean((p - g) ** 2)),
            "RMSElog": np.sqrt(np.mean((np.log(p) - np.log(g)) ** 2)),
            "d1": np.mean(ratio < 1.25)}

def align(p, g, how):
    if how == "none":
        return p
    if how == "median":                           # Zhou et al. (2017)
        return p * np.median(g) / np.median(p)
    if how == "LS scale":                         # min_s sum (s p - g)^2
        return p * (p @ g) / (p @ p)
    A = np.stack([1 / p, np.ones_like(p)], 1)     # MiDaS: fit s, t in inverse depth
    s, t = np.linalg.lstsq(A, 1 / g, rcond=None)[0]
    return 1 / np.clip(s / p + t, 1e-3, None)

def predictions(seed):
    eps = np.random.default_rng(seed).normal(0, 0.05, z.size)
    return {"noise only": z * np.exp(eps),
            "scale 0.4": 0.4 * z * np.exp(eps),
            "inv. affine": 1 / (2.0 / z * np.exp(eps) + 0.3)}   # 1/pred = 2/z + 0.3, noisy

hows = ["none", "median", "LS scale", "LS s,t on 1/z"]
print(f"depth range {z.min():.2f}-{z.max():.2f} m over {z.size} pixels")
print(f"{'prediction':12s} {'alignment':15s}  AbsRel   SqRel    RMSE RMSElog     d1")
for name, p in predictions(0).items():
    for how in hows:
        m = metrics(align(p, z, how), z)
        print(f"{name:12s} {how:15s}" + "".join(f"{m[k]:8.3f}" for k in m))

# Spread over seeds 0-19 of AbsRel after each alignment.
runs = np.array([[[metrics(align(p, z, h), z)["AbsRel"] for h in hows[1:]]
                  for p in predictions(s).values()] for s in range(20)])
print("AbsRel over seeds 0-19 (min-max):")
for i, name in enumerate(predictions(0)):
    print(f"  {name:12s}" + "".join(
        f"  {h}: {runs[:, i, j].min():.3f}-{runs[:, i, j].max():.3f}" for j, h in enumerate(hows[1:])))

# Eigen et al. (2014): scale-invariant log error (Eq. 3) and training loss (Eq. 4).
def eigen_loss(p, g, lam):
    d = np.log(p) - np.log(g)
    return np.mean(d**2) - lam * np.mean(d) ** 2

p = predictions(0)["noise only"]
for k in [0.1, 1.0, 10.0]:
    print(f"pred x {k:4}: D (lambda=1) = {eigen_loss(k * p, z, 1.0):.5f}, "
          f"lambda=0.5 loss = {eigen_loss(k * p, z, 0.5):.5f}, lambda=0 = {eigen_loss(k * p, z, 0.0):.5f}")
print(f"pred + 1 m: D (lambda=1) = {eigen_loss(p + 1.0, z, 1.0):.5f}")

Output:

depth range 3.78-15.00 m over 19200 pixels
prediction   alignment        AbsRel   SqRel    RMSE RMSElog     d1
noise only   none              0.040   0.025   0.557   0.050   1.000
noise only   median            0.041   0.027   0.573   0.052   1.000
noise only   LS scale          0.040   0.025   0.555   0.050   1.000
noise only   LS s,t on 1/z     0.040   0.025   0.549   0.050   1.000
scale 0.4    none              0.599   3.628   6.696   0.917   0.000
scale 0.4    median            0.041   0.027   0.573   0.052   1.000
scale 0.4    LS scale          0.040   0.025   0.555   0.050   1.000
scale 0.4    LS s,t on 1/z     0.040   0.025   0.549   0.050   1.000
inv. affine  none              0.783   6.701   9.286   1.600   0.000
inv. affine  median            0.277   0.829   3.181   0.305   0.224
inv. affine  LS scale          0.348   0.962   2.444   0.339   0.522
inv. affine  LS s,t on 1/z     0.040   0.024   0.548   0.050   1.000
AbsRel over seeds 0-19 (min-max):
  noise only    median: 0.040-0.043  LS scale: 0.040-0.040  LS s,t on 1/z: 0.039-0.040
  scale 0.4     median: 0.040-0.043  LS scale: 0.040-0.040  LS s,t on 1/z: 0.039-0.040
  inv. affine   median: 0.277-0.278  LS scale: 0.348-0.349  LS s,t on 1/z: 0.039-0.040
pred x  0.1: D (lambda=1) = 0.00248, lambda=0.5 loss = 2.65255, lambda=0 = 5.30262
pred x  1.0: D (lambda=1) = 0.00248, lambda=0.5 loss = 0.00248, lambda=0 = 0.00248
pred x 10.0: D (lambda=1) = 0.00248, lambda=0.5 loss = 2.65431, lambda=0 = 5.30613
pred + 1 m: D (lambda=1) = 0.00595

A wrong global scale ruins every raw metric ($\delta_1 = 0$) but costs nothing after any scale alignment, so aligned numbers measure shape only. Scale alone cannot repair the inverse-depth distortion, and the two scale fits even rank differently: least squares, which minimizes squared meters, wins on RMSE and $\delta_1$, and median scaling on AbsRel, SqRel and RMSE log. Only the scale-and-shift fit reaches the noise floor (AbsRel 0.039–0.040 over seeds 0–19). $D$ is identical for predictions scaled by 0.1, 1 and 10, while the $\lambda = 0.5$ loss is not; an additive 1 m error raises $D$.

Published numbers, as printed, each with the paper’s own protocol:

Paper	Model and training signal	Test data and protocol	AbsRel	$\delta < 1.25$
Eigen et al. (2014), Table 2	Coarse + fine, KITTI LiDAR	KITTI, their 28 test scenes	0.190	0.692
Godard et al. (2019), Table 1	Monodepth2, monocular video	KITTI Eigen split, 80 m cap, per-image median scaling	0.115	0.877
Godard et al. (2019), Table 1	Monodepth2, stereo	KITTI Eigen split, 80 m cap, no scaling	0.109	0.864
Ranftl et al. (2021), Table 3	DPT-Hybrid, fine-tuned	KITTI Eigen split	0.062	0.959
Yang et al. (2024), Table 2	Depth Anything ViT-L, zero-shot	KITTI, scale and shift aligned	0.076	0.947
Yang et al. (2024), Table 3	Depth Anything encoder, fine-tuned to metric	NYU Depth v2	0.056	0.984

The rows are not comparable with one another: they differ in alignment, ground truth and training data.

Applications

Single-image depth back-projects to a point cloud for 3D reconstruction and densifies maps in visual odometry and SLAM; the hub lists more.

Open Problems

• Metric depth without intrinsics. Depth Pro and UniDepth estimate the camera from the image, so their scale is only as good as that estimate.
• Scale consistency over time for video, which per-image median scaling hides.