Self-Supervised Depth Estimation

Self-supervised depth estimation trains a network that predicts a depth map from one image without any measured depth. During training the network sees a second view of the same scene, either the other image of a rectified stereo pair or a neighbouring frame of a video. The predicted depth, together with the known or predicted camera motion, warps the second view into the first, and the photometric difference between the warped and the real image is the loss. The 2017 papers that introduced the main variants called it unsupervised learning (Godard et al., 2017; Zhou et al., 2017), with view synthesis as the supervisory signal; Godard et al. (2019) call it self-supervised. It is also known by the names of its best-known systems: Monodepth, SfMLearner and Monodepth2.

Monocular depth estimation introduces the stereo loss of Monodepth with an example over candidate depths, shows why monocular video fixes depth only up to scale, and states Monodepth2’s minimum reprojection and auto-masking. This article covers the technique in depth: what each loss term does and how the official code computes it, the explainability and auto-masks, moving objects, scale, and the KITTI Eigen split on which these papers report their main results.

Purpose

Supervised depth networks need dense ground truth, which is expensive to record outdoors: on KITTI, LiDAR points reprojected into the image cover under 5% of the pixels (Godard et al., 2017). Self-supervision replaces the label with a geometric consistency that any camera pair or video provides for free. If the depth $D_t$ of every pixel of a target image $I_t$ and the motion to another view are right, then sampling the other view at the projected positions reproduces $I_t$, wherever the scene is static, visible in both views and of constant brightness. Godard et al. (2019) frame the problem as novel view synthesis through an intermediate variable, depth or disparity, which can then be read out of the network.

The idea comes from classical geometry. Zhou et al. (2017) point out that it resembles direct methods for structure and motion, which minimize the same kind of photometric error, with one difference: a direct method optimizes the depth and pose of one scene, while the network takes one gradient step per batch and so learns an implicit prior from a large corpus of related images.

When to Use It

Self-supervision suits domains with plenty of stereo footage or video but little or no depth ground truth. The choice of training signal decides what the network can learn:

• Stereo pairs give a known, fixed relative pose, so depth comes out in metric units once the baseline is known; the pose is a one-time calibration (Godard et al., 2019). They need a synchronized, rectified rig.
• Monocular video needs only one camera but adds a pose network, and yields depth only up to an unknown scale per sequence (see the section on scale below). It also breaks on a static camera and on moving objects.
• Both (Monodepth2’s “MS” training) combine the video frames with the stereo partner; the stereo baseline fixes the scale.

It is a poor fit where brightness constancy fails badly. Godard et al. (2019, Fig. 8) show failures on distorted, reflective and colour-saturated regions. In our reading, the same holds wherever many depths reproduce the image equally well, as on textureless areas and sky. Zhou et al. (2017) list the three assumptions behind the warp: a static scene, no occlusion between target and source views, and Lambertian surfaces.

How It Works

The warp

Write $K$ for the intrinsic matrix, $p_t$ for a target pixel in homogeneous coordinates, and $T_{t \to s} = [R \mid \mathbf{t}]$ for the transform from target to source camera coordinates, as in Monocular Depth Estimation. The projected source position is

$$p_s \sim K\,T_{t \to s}\,D_t(p_t)\,K^{-1} p_t ,$$

and the synthesized image is $I_{s \to t}(p_t) = I_s(p_s)$, read by bilinear interpolation of the four neighbours of $p_s$ (Zhou et al., 2017, Eq. 2 and Fig. 3). Both papers credit the sampler to spatial transformer networks. Bilinear sampling is differentiable with respect to $p_s$, and so with respect to depth and pose; it is the backward warp described in image warping, whose output grid is the target image. For a rectified stereo pair, $T_{t \to s}$ is a pure horizontal translation and the warp reduces to a horizontal shift by the disparity $fB/D_t$.

The gradient reaches the depth only through the image gradient at $p_s$. Zhou et al. (2017, Section 3.4) note that this gradient locality stalls training when the correct $p_s$ lies in a low-texture region or far from the current estimate, and they counter it with multi-scale prediction and a smoothness loss.

Photometric error

All the systems below compare $I_t$ with $I_{s \to t}$ pixel by pixel. SfMLearner used the plain L1 difference (Zhou et al., 2017, Eq. 1). Monodepth combined a single-scale SSIM term over $3 \times 3$ blocks with L1, and Monodepth2 kept the mixture:

$$\mathrm{pe}(I_a, I_b) = \frac{\alpha}{2}\bigl(1 - \mathrm{SSIM}(I_a, I_b)\bigr) + (1 - \alpha)\,\lVert I_a - I_b \rVert_1, \qquad \alpha = 0.85$$

(Godard et al., 2019, Section 3.1). In the official Monodepth2 code (layers.py, trainer.py), the SSIM layer reflection-pads the images by one pixel, computes means, variances and covariance with $3 \times 3$ average pooling and constants $C_1 = 0.01^2$, $C_2 = 0.03^2$, and returns $(1 - \mathrm{SSIM})/2$ clipped to $[0, 1]$; the L1 term is averaged over the colour channels. Godard et al. (2019) attribute the improvement of SSIM over plain pixel differences to Monodepth, which found that adding this local structure term “significantly” improved depth.

Edge-aware smoothness

Photometric error alone leaves depth undetermined wherever the image is flat. Monodepth adds an L1 penalty on disparity gradients, weighted down at image edges, since depth discontinuities usually coincide with image gradients (Godard et al., 2017, Eq. 3, which penalizes the disparity $d$ itself; written here in Monodepth2’s notation, Godard et al., 2019, Eq. 3):

$$L_s = \lvert \partial_x d^*_t \rvert\, e^{-\lvert \partial_x I_t \rvert} + \lvert \partial_y d^*_t \rvert\, e^{-\lvert \partial_y I_t \rvert} .$$

Monodepth2 applies it to the mean-normalized inverse depth $d^*_t = d_t / \overline{d_t}$ (Eq. 3), which it credits to DDVO (Wang et al., 2018), to discourage the estimated depth from shrinking. In the related work, Godard et al. (2019) describe the problem as a preference for smaller depth values that the commonly used smoothness term creates. In our reading, the reason is that in monocular training the photometric loss does not change if all depths and the translation are scaled together, so an unnormalized penalty on disparity gradients can always be lowered by scaling the disparities down; dividing by the mean removes that escape. The code computes the image gradient as the mean absolute difference over colour channels, weights the term by $\lambda = 10^{-3}$ (the disparity_smoothness default) and divides it by $2^k$ at decoder scale $k$.

SfMLearner instead minimized the L1 norm of the second-order gradients of the depth, with weight $\lambda_s = 0.5/l$ for downscaling factor $l$ (Zhou et al., 2017, Section 4).

Training from stereo pairs

Garg et al. (2016), a paper not read for this article, are described by Zhou et al. (2017) as learning single-view depth from the projection error to a calibrated stereo twin. Godard et al. (2017) describe their image formation model as not fully differentiable, linearized with a Taylor approximation, which they judged harder to optimize; Monodepth replaced it with bilinear sampling.

Monodepth’s design rests on a detail of backward warping (Godard et al., 2017, Section 3.2 and Fig. 3). Sampling the left image to produce the right one yields a disparity aligned with the right image, not with the input. Sampling the right image to reconstruct the left gives a left-aligned disparity, but with “texture-copy” artefacts and errors at depth discontinuities. Monodepth therefore predicts both a left and a right disparity map from the left image alone, reconstructs each image from the other, and ties the two maps with the left-right consistency term that Monocular Depth Estimation gives. The loss at each of four output scales is

$$C_s = \alpha_{\text{ap}}(C^l_{\text{ap}} + C^r_{\text{ap}}) + \alpha_{\text{ds}}(C^l_{\text{ds}} + C^r_{\text{ds}}) + \alpha_{\text{lr}}(C^l_{\text{lr}} + C^r_{\text{lr}}),$$

with $\alpha_{\text{ap}} = \alpha_{\text{lr}} = 1$ and $\alpha_{\text{ds}} = 0.1/r$ for a layer downscaled by $r$ (Eq. 1, Section 4.1). The disparity passes through a scaled sigmoid with a maximum of 0.3 times the image width. At test time Monodepth also runs the network on the mirrored image and blends the two disparities, taking the outer 5% at each side from the version whose disocclusion ramps lie elsewhere; results with this step are marked “pp” (post-processing).

Training from video

SfMLearner (Zhou et al., 2017) trains a depth network on the target frame and a pose network on the target concatenated with its neighbours. The pose network outputs three Euler angles and a translation per source frame, averaged over the image; the loss sums the L1 error over source frames and scales. Monodepth2 (Godard et al., 2019) uses three-frame snippets $t' \in \{t-1, t+1\}$, a ResNet18 pose network that takes a pair of frames and outputs one 6-DoF pose as an axis-angle rotation and a translation, both scaled by 0.01, and a U-Net-style depth network with a ResNet18 encoder pretrained on ImageNet. Its sigmoid output $\sigma$ becomes depth $D = 1/(a\sigma + b)$ with $a, b$ set so that depth lies between 0.1 and 100 units. The pose network is used only during training.

Two further Monodepth2 details matter for reproduction. Colour augmentation is applied only to the images fed to the networks, not to those used in the loss. And all scales are evaluated at the input resolution: lower-resolution disparity maps are upsampled before warping, because computing the loss on low-resolution images produced holes in low-texture regions and texture-copy artefacts (Section 3.2, “full-res multi-scale”).

Occlusions: minimum reprojection

Averaging $\mathrm{pe}$ over source frames penalizes the correct depth at any pixel that is hidden or out of view in one of them, since the colour sampled there belongs to something else. Monodepth2 takes the per-pixel minimum, $L_p = \min_{t'} \mathrm{pe}(I_t, I_{t' \to t})$ (Eq. 4), so each pixel is matched to a frame where it is visible. The authors report fewer artefacts at image borders and sharper occlusion boundaries. In mixed training the stereo partner is one more source frame, so a pixel occluded in the other stereo view can be matched in a video frame instead (their Fig. 4).

Moving objects: explainability masks and auto-masking

Explainability masks. SfMLearner trains a third head, sharing the first encoder layers of the pose network, that outputs a soft per-pixel mask $\hat E_s$ for each source frame, a two-channel softmax whose second channel weights the loss (Zhou et al., 2017, Sections 3.3 and 3.5):

$$L_{\text{vs}} = \sum_{s}\sum_{p} \hat E_s(p)\,\lvert I_t(p) - \hat I_s(p) \rvert .$$

Since $\hat E_s = 0$ would make the loss vanish, a cross-entropy term with constant label 1 pushes the mask towards one; the full objective is $\sum_l L^l_{\text{vs}} + \lambda_s L^l_{\text{smooth}} + \lambda_e \sum_s L_{\text{reg}}(\hat E^l_s)$ with $\lambda_e = 0.2$ (Eq. 4). The authors found the mask gave only a modest gain on KITTI, which they attributed to mostly static scenes and small occluded regions over three frames; their Fig. 10 shows masks on moving objects and on disappearing objects, but also on thin structures, a downside of the weighting. Godard et al. (2019) note that later versions of SfMLearner available online disabled the term and performed better.

Auto-masking. Monodepth2 replaces the learned mask with a binary one computed on the fly (Eq. 5): a pixel contributes only if warping lowers its error below that of the unwarped source,

$$\mu = \Bigl[\min_{t'} \mathrm{pe}(I_t, I_{t' \to t}) < \min_{t'} \mathrm{pe}(I_t, I_{t'})\Bigr].$$

Pixels that do not change between frames often indicate a static camera, an object moving at the camera’s velocity, or a low-texture region (Godard et al., 2019). Under the static-scene warp, an object that stays put in the image has zero parallax, which only infinite depth explains; this produces the “holes” of infinite depth on cars that Godard et al. show for earlier methods (Fig. 2). The final loss is $L = \mu L_p + \lambda L_s$, averaged over pixels, scales and the batch.

The code implements Eq. 5 differently from a multiplied mask. It appends the unwarped (“identity”) errors of the source frames, plus Gaussian noise of standard deviation $10^{-5}$ to break ties, to the warped errors and takes the minimum over all of them. Where an identity error wins, the pixel’s loss is a term that does not depend on the predicted depth or pose, so it passes no gradient, which has the effect of $\mu = 0$. The flag --disable_automasking turns this off, --avg_reprojection restores averaging, and --predictive_mask re-implements SfMLearner’s mask with weight 0.2.

Dynamic objects beyond auto-masking

Auto-masking handles only objects moving with the camera. An object moving at another speed is warped with the wrong parallax, and the loss is lowest at a wrong depth. Godard et al. (2019) review attempts that model object motion explicitly, with optical flow (GeoNet, Competitive Collaboration) or with precomputed instance segmentation masks of known categories (Casser et al.’s Struct2Depth), and show in their Fig. 2 that methods modelling motion explicitly (GeoNet, Competitive Collaboration and EPC++) can still fail on objects that were often moving in training, such as cars.

Gordon et al. (2019) predict, besides depth, a dense translation field $t(x, y) = t_0 + m(x, y)\,\delta t(x, y)$, where $t_0$ is the camera’s translation and the residual $\delta t$ may be non-zero only inside a “possibly mobile” mask $m$ (their Eq. 2). They found that $m$ can be as rough as the union of bounding boxes from an object detector. In their ablation, trained on Cityscapes and KITTI combined (Table 3), boxes instead of segmentation masks gave AbsRel 0.120 on Cityscapes and 0.125 on KITTI, against 0.121 and 0.124, while dropping object motion raised it to 0.172 and 0.130. They also handle occlusions geometrically: a point warped from the source counts in the loss only where its transformed depth $z'$ does not exceed the target’s depth at that position, $z' \le z_t$, applied symmetrically in both directions. And they predict the camera intrinsics, including lens distortion, with the pose; they show that the loss constrains the intrinsics only when the camera rotates, because for pure translation it depends on $K$ only through the product $K\mathbf{t}$. On the KITTI Eigen split with median scaling and an 80 m cap they report AbsRel 0.128 with learned intrinsics and 0.129 with given intrinsics (Table 1).

Scale and median scaling

A monocular model’s depth carries an arbitrary scale, since scaling depth and translation together leaves every warp unchanged (Monocular Depth Estimation derives this). Monodepth2’s depth range of 0.1 to 100 “units” therefore has no physical meaning for monocular models, and the scale may differ from image to image. Evaluation removes it with per-image median scaling, introduced by Zhou et al. (2017, Section 4.1); the depth estimation error metrics article gives the order of operations in the code. The Monodepth2 evaluation script also prints the median of the per-image ratios and their standard deviation after dividing by that median, a direct measure of how inconsistent the scale is.

Stereo supervision fixes the scale. The Monodepth2 code sets the stereo transform to a horizontal translation of 0.1 units (mono_dataset.py), while the KITTI rig’s baseline is 0.54 m, so stereo-trained predictions are multiplied by a fixed 5.4 at evaluation and not median-scaled (README and evaluate_depth.py). Pose has the same ambiguity: Zhou et al. (2017) evaluated their pose network on KITTI odometry sequences 09 and 10 with the absolute trajectory error over five-frame snippets, after optimizing a scaling factor for each method’s predictions.

How to Apply It

The code below shows how the losses treat a fixed, known depth, without training a network. A camera translates sideways by $b = 0.3$ m per frame past four fronto-parallel planes, textured with images bundled with scikit-image: a brick wall at 12 m, a static gravel occluder at 4 m, a grass panel at 6 m that moves with the camera, and a panel with the cat image at 5 m that moves at half the camera’s speed. Frames $t-1$ and $t+1$ are the sources and $t$ is the target. Depth is $Z$ in metres along the optical axis, with $y$ pointing down, and the camera centre of frame $t \pm 1$ is at $x = \pm b$ with no rotation. The pose is known, so the example isolates depth.

Ground-truth geometry classifies each target pixel: a static point seen in both sources, occluded in one, or out of view in one; or a point on one of the moving panels. For each class the code prints the mean error at the true depth, averaged over the two sources and as Monodepth2’s minimum, and the same at 10 km, the “infinitely far” hypothesis that a static-scene model needs for zero parallax. It also prints the share of pixels where the true depth has the lower error, the share that auto-masking removes at the true depth, and, for the half-speed panel, the error at twice its depth. The static-scene model explains that panel’s parallax with twice its depth: relative to the panel, the camera moves only $b/2$ per frame. Monodepth2’s SSIM + L1 error and SfMLearner’s L1 error are printed separately, since the two sources use different errors. The masked share is printed under two rules, because the paper and its code differ: Eq. 5 as printed, a strict inequality (the “masked” column), and the official code’s rule, which adds Gaussian noise of standard deviation $10^{-5}$ to the identity errors and takes the minimum over identity and warped errors together (the line after each table; the noise comes from a fixed seed).

After the first run, the static pixels seen in both sources were masked far more often than expected. Inspection found a rendering error: the wall’s texture coordinates ran off the left edge of the brick image, which left the left part of the wall untextured. The texture offset was corrected and the example re-run. Nothing else was changed; the output below is from the corrected script. A further planned measure, the per-pixel depth that minimizes each loss over a sweep of candidate depths, did not recover the true depth on static pixels seen in both sources for every compared error and aggregation in pilot runs before and after the correction, so it was set aside and is not part of the example. The two lines with the code’s masking rule, one per error, were added after a fact-check found that the code’s tie-breaking differs from the paper’s equation; the other printed lines did not change.

import numpy as np
from scipy.ndimage import map_coordinates, uniform_filter
from skimage import color, data

# Three frames of a camera translating sideways past textured fronto-parallel planes.
H, W, f, b = 120, 200, 200.0, 0.3                    # image size, focal length (px), step (m)
cx, cy = (W - 1) / 2, (H - 1) / 2
u, v = np.meshgrid(np.arange(W, dtype=float), np.arange(H, dtype=float))
gray = lambda im: (color.rgb2gray(im) if im.ndim == 3 else im / 255.0).astype(float)
inf = np.inf
# name, depth Z (m), X range, Y range (m, y down), speed relative to the camera, texture
planes = [("wall", 12.0, (-inf, inf), (-inf, inf), 0.0, gray(data.brick())),
          ("occluder", 4.0, (-1.2, -0.4), (-0.6, 0.9), 0.0, gray(data.gravel())),
          ("co-moving", 6.0, (0.2, 1.1), (0.1, 1.0), 1.0, gray(data.grass())),
          ("half-speed", 5.0, (1.0, 1.6), (-0.9, 0.0), 0.5, gray(data.chelsea()))]

def render(c):
    """Ray-cast the frame whose camera centre is at x = c: image, depth Z, surface id."""
    img, Z, sid = np.zeros((H, W)), np.full((H, W), inf), np.full((H, W), -1)
    for k, (_, z, (x0, x1), (y0, y1), speed, tex) in enumerate(planes):
        X = (u - cx) * z / f + c - speed * c         # point in the surface's own frame
        Y = (v - cy) * z / f
        hit = (X >= x0) & (X <= x1) & (Y >= y0) & (Y <= y1) & (z < Z)
        cols = (X - (0 if x0 == -inf else x0)) * f / z + (200 if x0 == -inf else 5)
        rows = (Y - (0 if y0 == -inf else y0)) * f / z + (200 if y0 == -inf else 5)
        val = map_coordinates(tex, [rows, cols], order=1, mode="nearest")
        img[hit], Z[hit], sid[hit] = val[hit], z, k
    return img, Z, sid

cams = {"t-1": -b, "t+1": b}                         # source frames; target t at x = 0
I_t, Z_t, S_t = render(0.0)
src = {s: render(c) for s, c in cams.items()}

def warp(img, c, Z):           # static-scene model: target pixel u reads source pixel u - f c / Z
    return map_coordinates(img, [v, u - f * c / Z], order=1, mode="nearest")  # border padding

def dssim(a, x):               # (1 - SSIM) / 2 on 3x3 boxes with reflection padding (Monodepth2)
    m = lambda z: uniform_filter(z, 3, mode="mirror")
    ma, mx = m(a), m(x)
    sa, sx, sax = m(a * a) - ma**2, m(x * x) - mx**2, m(a * x) - ma * mx
    s = (2 * ma * mx + 1e-4) * (2 * sax + 9e-4) / ((ma**2 + mx**2 + 1e-4) * (sa + sx + 9e-4))
    return np.clip((1 - s) / 2, 0, 1)

def pe(a, x, kind):            # photometric error per pixel
    l1 = np.abs(a - x)
    return l1 if kind == "L1" else 0.85 * dssim(a, x) + 0.15 * l1

# Ground-truth pixel classes of frame t: follow each point (with its own motion) into each source.
speed = np.array([p[4] for p in planes])[S_t]
seen, out = np.zeros((H, W), int), np.zeros((H, W), bool)
for s, c in cams.items():
    us = u - f * (c - speed * c) / Z_t
    inside = (us >= 0) & (us <= W - 1)
    ui, vi = np.clip(np.round(us).astype(int), 0, W - 1), v.astype(int)
    same = (src[s][2][vi, ui] == S_t) & (np.abs(src[s][1][vi, ui] - Z_t) < 0.01 * Z_t)
    seen += inside & same
    out |= ~inside
static = speed == 0
classes = {"static, seen in both": static & (seen == 2),
           "static, occluded in one": static & (seen == 1) & ~out,
           "static, out of view in one": static & out,
           "moving with camera": S_t == 2,
           "moving at half speed": S_t == 3}

rng = np.random.default_rng(0)                       # tie-breaking noise of the official code
for kind in ["SSIM+L1", "L1"]:
    at = lambda Z: np.stack([pe(I_t, warp(src[s][0], c, Z), kind) for s, c in cams.items()])
    true, far = at(Z_t), at(1e4)                     # true depth; "infinitely far"
    ident = np.stack([pe(I_t, src[s][0], kind) for s in cams])   # unwarped sources
    removed = ~(true.min(0) < ident.min(0))          # auto-mask mu = 0 at the true depth
    print(f"\npe = {kind}: mean error at true depth and at 10 km, share of pixels where the "
          f"true depth wins, auto-mask")
    print(f"{'class':27s} {'px':>5s} {'true avg':>9s} {'true min':>9s} {'far avg':>8s} "
          f"{'far min':>8s} {'wins avg':>9s} {'wins min':>9s} {'masked':>7s}")
    for name, m in classes.items():
        ta, tm, fa, fm = true.mean(0), true.min(0), far.mean(0), far.min(0)
        print(f"{name:27s} {m.sum():5d} {ta[m].mean():9.4f} {tm[m].mean():9.4f} {fa[m].mean():8.4f} "
              f"{fm[m].mean():8.4f} {(ta < fa)[m].mean():9.3f} {(tm < fm)[m].mean():9.3f} "
              f"{removed[m].mean():7.3f}")
    m = classes["moving at half speed"]
    double = at(2 * Z_t)                             # static depth that matches half the parallax
    print(f"moving at half speed, error at 2 x true depth: avg {double.mean(0)[m].mean():.4f}, "
          f"min {double.min(0)[m].mean():.4f}")
    # Monodepth2 code: min over [identity + N(0, 1e-5), warped]; an identity entry wins = masked.
    combined = np.concatenate([ident + rng.normal(0, 1e-5, ident.shape), true])
    code_removed = np.argmin(combined, 0) < len(cams)
    print("masked under the code's rule (noise on identity errors): " + ", ".join(
        f"{name} {code_removed[m].mean():.3f}" for name, m in classes.items()))

Output (NumPy 2.3.5, SciPy 1.16.3, scikit-image 0.25.2; the script runs in about a second):

pe = SSIM+L1: mean error at true depth and at 10 km, share of pixels where the true depth wins, auto-mask
class                          px  true avg  true min  far avg  far min  wins avg  wins min  masked
static, seen in both        19164    0.0050    0.0005   0.2351   0.1352     0.986     0.998   0.002
static, occluded in one      1872    0.1947    0.0085   0.2545   0.1560     0.628     0.968   0.032
static, out of view in one   1200    0.0103    0.0000   0.0950   0.0160     0.994     1.000   0.000
moving with camera            900    0.3889    0.3033   0.0109   0.0055     0.021     0.016   0.984
moving at half speed          864    0.0956    0.0100   0.1059   0.0484     0.295     0.221   0.507
moving at half speed, error at 2 x true depth: avg 0.0232, min 0.0049
masked under the code's rule (noise on identity errors): static, seen in both 0.002, static, occluded in one 0.032, static, out of view in one 0.000, moving with camera 0.984, moving at half speed 0.426

pe = L1: mean error at true depth and at 10 km, share of pixels where the true depth wins, auto-mask
class                          px  true avg  true min  far avg  far min  wins avg  wins min  masked
static, seen in both        19164    0.0000    0.0000   0.1039   0.0538     1.000     0.990   0.051
static, occluded in one      1872    0.0917    0.0000   0.1188   0.0527     0.581     0.989   0.044
static, out of view in one   1200    0.0084    0.0000   0.0382   0.0050     0.940     0.970   0.098
moving with camera            900    0.1401    0.0899   0.0004   0.0002     0.000     0.004   1.000
moving at half speed          864    0.0666    0.0210   0.0664   0.0211     0.227     0.150   0.660
moving at half speed, error at 2 x true depth: avg 0.0000, min 0.0000
masked under the code's rule (noise on identity errors): static, seen in both 0.029, static, occluded in one 0.021, static, out of view in one 0.049, moving with camera 0.999, moving at half speed 0.509

Where the scene obeys the assumptions, the true depth is nearly free under either error and either aggregation, and the L1 error is exactly zero because the wall’s and the occluder’s disparities (5 and 15 pixels) are whole pixels. The remaining SSIM + L1 error comes from $3 \times 3$ windows that straddle depth edges.

The occluded pixels show why Monodepth2 takes the minimum. Averaged over the two sources, the error at the true depth is almost as large as at 10 km, and the true depth wins at only about three in five of these pixels, so the averaged loss would push a large share of them towards a wrong depth. The minimum picks the source in which each pixel is visible: the error drops by a factor of more than twenty with SSIM + L1, and the true depth wins at 97–99% of the pixels. The out-of-view strip did not show the same problem here: averaging still prefers the true depth there. In our reading, border padding repeats the image’s edge column, and in this scene that copy is close to the missing content. Godard et al. (2019) report the border artefacts on real data, where this need not hold.

The panel moving with the camera is the case auto-masking was designed for. Its true depth loses to “infinitely far” at 98–100% of its pixels, by an error ratio of more than thirty, so a network trained on such frames learns a hole of infinite depth. Under Eq. 5 the mask removes 98% (SSIM + L1) and 100% (L1) of its pixels, against 0.2% and 5.1% of the static pixels seen in both sources; the code’s rule gives nearly the same shares for the panel (98.4% and 99.9%). The two rules part on the static pixels under L1. The 5.1% are ties: the textures have 8-bit values, so the L1 error of the unwarped source is sometimes exactly zero too, and the strict inequality of Eq. 5 then removes a pixel even though its depth is right. The code’s noise decides such ties at random instead, and the share falls to 2.9%. Either way some correctly placed pixels lose their gradient on this frame.

The half-speed panel is what auto-masking does not handle. Its true depth wins at only 15–30% of its pixels, while twice the true depth has a lower error under every combination, and no error at all under L1, where the shift is 6 whole pixels. A static-scene loss trained on such an object would place it twice as far away. At the true depth the mask removes 51–66% of its pixels under Eq. 5 and 43–51% under the code’s rule. In our reading this is because the unwarped source is off by the panel’s own 6-pixel motion, and the warp at the true depth is off by the same amount in the other direction, so the two errors are comparable and can be exactly equal. In our reading, the drop from one rule to the other under both errors means that many of these pixels have equal errors, or errors closer than the noise, which Eq. 5 counts as removed and the code’s noise splits at random. Neither rule removes the panel the way it removes the panel moving with the camera. In training the mask is computed with the network’s current depth, not the true one, and where the warped error reaches zero, Eq. 5 keeps every pixel whose unwarped error is above zero. The example measures single pixels at fixed depths; it does not show how a network trained with these losses would end up.

Variants

• Signal. Stereo (Godard et al., 2017, and, as Zhou et al. and Godard et al. describe it, Garg et al., 2016), monocular video (Zhou et al., 2017), or both (Godard et al., 2019, MS).
• Masking. None, a learned explainability mask (Zhou et al., 2017), auto-masking (Godard et al., 2019), or geometric occlusion tests and object masks (Gordon et al., 2019).
• Motion model. Rigid camera motion only, or added object motion: residual translation fields within detector masks (Gordon et al., 2019), or optical flow and instance masks in the methods that Godard et al. (2019) review.
• Camera model. Known intrinsics (all three original systems; Zhou et al., 2017, list this as a limitation), or intrinsics and lens distortion learned with the pose (Gordon et al., 2019).
• Weak supervision added. Godard et al. (2019) survey hybrids that add sparse ordinal depths, known object sizes, synthetic depth or classical structure from motion to the photometric loss, and they cite later work that adds depth hints from classical stereo algorithms to their own model.

Best Practices

• Use the per-pixel minimum over source frames and auto-masking for video training. In Monodepth2’s monocular ablation (Table 2a), removing either from the full model raised AbsRel (to 0.117 and 0.120 from 0.115), and replacing auto-masking with a re-implementation of SfMLearner’s predictive mask was worse than using no mask (0.123).
• Compute the loss at the input resolution at every decoder scale, and normalize disparity before the smoothness term.
• Augment only the network’s inputs. Colour jitter belongs on the images the networks see, not on those used in the photometric loss, which must keep the true brightness relationship between frames (Godard et al., 2019).
• Remove static frames, or let auto-masking handle them. Zhou et al. (2017) excluded sequences with mean optical flow under 1 pixel. Monodepth2 trained on the full Eigen training data without that filtering, and its full model reached AbsRel 0.116, against 0.146 for its baseline without its contributions (Table 2c).
• Report the scale treatment. State whether depth was median-scaled per image, scaled by a fixed factor from a stereo baseline, or not scaled, and keep the depth cap and crop fixed between compared methods.

Pitfalls

• Infinite-depth holes on vehicles that drove at the camera’s speed in training (Godard et al., 2019, Fig. 2).
• Wrong but finite depth on objects moving at other speeds, as on the half-speed panel above. Auto-masking does not detect them.
• Static cameras. With no camera motion every depth reproduces the image equally well; Monodepth2’s auto-mask can remove whole frames, and SfMLearner’s pre-processing removed them.
• Non-Lambertian and textureless surfaces. Reflections, transparency, saturation and sky violate the photometric model, and the predictions there come from smoothness and learned priors.
• Gradient locality. The bilinear sampler sees only four neighbours, so training can stall when the correct match is far away or in low texture (Zhou et al., 2017).
• Scale drift hidden by evaluation. Per-image median scaling hides an inconsistent scale; the spread of the ratios shows it.
• Texture copy. Image detail can leak into the depth map, which Monodepth’s left-right consistency and Monodepth2’s full-resolution losses both target.

Evidence and Impact

The KITTI Eigen split

The papers above report their main KITTI results on the Eigen split of the KITTI raw recordings. Eigen et al. (2014, Section 4.2) used 56 scenes of the “city”, “residential” and “road” categories, split 28 for training and 28 for testing, and excluded training frames where the car was stationary, judged by an acceleration threshold. Godard et al. (2017, Section 4.2) describe the test set as 697 images covering 29 scenes; the Monodepth2 test list (splits/eigen/test_files.txt) has 697 lines drawn from 28 drive sequences. Training on the split differs between methods:

• Monodepth (stereo): 22,600 training images from the remaining 32 scenes, the rest held out (Godard et al., 2017).
• SfMLearner (video): three-frame sequences from both colour cameras, treated independently, excluding all test scenes and static sequences: 44,540 sequences, 40,109 for training and 4,431 for validation (Zhou et al., 2017, Section 4.1).
• Monodepth2, “eigen_zhou” split: Zhou et al.’s pre-processing, giving 39,810 training and 4,424 validation triplets (Godard et al., 2019, Section 4.1). In the code’s split files the training set uses left and right images in about equal numbers (19,956 and 19,854). The validation frames come from 31 of the 32 training drives, so they are not scene-disjoint. Monodepth2 also uses the same intrinsics for every image, with the principal point at the image centre and the average KITTI focal length.
• Gordon et al. (2019): 39,835 KITTI training examples on the same split.

Evaluation uses the reprojected LiDAR of the 697 test frames, an 80 m cap and the crop usually named after Garg et al. (2016); the error metrics article covers the crops, caps and per-image averaging. Godard et al. (2017) note that this ground truth suffers from the rotating scanner, the vehicle’s motion and occlusions. Godard et al. (2019) also evaluate on the 93% of test frames that have the improved ground truth of the KITTI depth prediction benchmark; the code’s eigen_benchmark option lists 652 frames and, unlike the standard split, applies no crop. Both papers label their Eigen-split tables “KITTI 2015”, although the split is drawn from the raw recordings rather than the KITTI 2015 stereo benchmark.

Results as published

On the Eigen split, SfMLearner reported AbsRel 0.208 trained on KITTI and 0.198 when pretrained on Cityscapes, with median scaling (Zhou et al., 2017, Table 1; its separate 50 m rows aside, the table does not state the cap). With an 80 m cap and without scaling, Monodepth reported 0.148 (KITTI) and 0.124 (Cityscapes + KITTI), and 0.114 for its ResNet-50 variant with post-processing (Godard et al., 2017, Table 2). Monodepth2’s Table 1 lists some earlier methods with newer results from their GitHub repositories, for example 0.183 for SfMLearner instead of the paper’s 0.208, and reports 0.106 for its own model trained on video and stereo (MS), evaluated without scaling. Its monocular ablation, on the Eigen split with median scaling and the 640 × 192 resolution, isolates the two losses:

Monodepth2 variant (Godard et al., 2019, Table 2a)	AbsRel	RMSE (m)	$\delta < 1.25$
Baseline (none of the contributions)	0.140	5.512	0.852
Baseline + minimum reprojection	0.122	5.116	0.866
Baseline + auto-masking	0.124	5.010	0.858
Full model without minimum reprojection	0.117	4.846	0.870
Full model without auto-masking	0.120	5.074	0.872
Full model with SfMLearner’s mask instead	0.123	5.210	0.869
Full model	0.115	4.863	0.877

Each component helps on its own, and the full model is best on AbsRel and $\delta < 1.25$, although removing minimum reprojection gives a slightly lower RMSE. These numbers use the training-time choices above; models with stereo supervision are evaluated unscaled and are not comparable with the median-scaled rows.

Influence

Self-supervision made it possible to train depth networks on unlabeled stereo footage and video, and to use the jointly trained pose networks for visual odometry: Zhou et al. (2017) reported that their pose network, on five-frame snippets, beat ORB-SLAM run on the same snippets and a mean-odometry baseline, while falling short of ORB-SLAM run on the full sequences. Monodepth’s SSIM + L1 error and edge-aware smoothness carried over to Monodepth2, and Gordon et al. (2019) again compare frames with SSIM and L1 while extending the model to object motion and unknown cameras, and showed qualitatively that depth can be learned from YouTube videos taken with different, unknown cameras. Large relative-depth models trained on many labelled datasets, such as MiDaS, take a different route to single-image depth.