Depth Estimation Error Metrics

Depth estimation error metrics compare a predicted depth map with measured depth, pixel by pixel, and summarize how far off the prediction is. The usual set, often called “the Eigen metrics”, contains the absolute relative error (AbsRel, “abs rel”), the squared relative error (SqRel), the root mean squared error in meters (RMSE) and in log depth (RMSE log), the threshold accuracies $\delta < 1.25$, $1.25^2$ and $1.25^3$ (also written $\delta_1, \delta_2, \delta_3$ or “a1–a3”), and the scale-invariant log error (SILog); indoor work often adds the mean $\log_{10}$ error. Eigen, Puhrsch and Fergus (2014) collected most of them from earlier work for their single-image depth network, and most monocular depth estimation papers still report them. The depth estimation hub introduces the metrics and the two alignment procedures; this article covers what each metric responds to, and the evaluation conventions without which two papers’ numbers cannot be compared.

Formula

Let $T$ be the set of pixels with valid ground truth $z_i$ and prediction $\hat z_i$, both in meters, and $e_i = \log \hat z_i - \log z_i$ (natural log unless written $\log_{10}$). Eigen et al. (2014, Section 4.3) list:

$$\text{AbsRel} = \frac{1}{|T|}\sum_{i \in T} \frac{|\hat z_i - z_i|}{z_i}, \qquad \text{SqRel} = \frac{1}{|T|}\sum_{i \in T} \frac{(\hat z_i - z_i)^2}{z_i},$$ $$\text{RMSE} = \sqrt{\frac{1}{|T|}\sum_{i \in T} (\hat z_i - z_i)^2}, \qquad \text{RMSE}_{\log} = \sqrt{\frac{1}{|T|}\sum_{i \in T} e_i^2},$$ $$\delta_k = \frac{1}{|T|}\,\Big|\Big\{ i \in T : \max\Big(\frac{\hat z_i}{z_i}, \frac{z_i}{\hat z_i}\Big) < 1.25^k \Big\}\Big|, \qquad k = 1, 2, 3.$$

Their text defines the threshold metric as a percentage of pixels; their tables print it as a fraction, and so does most later work. The scale-invariant error is the variance of the log error,

$$D = \frac{1}{|T|}\sum_{i \in T} e_i^2 - \frac{1}{|T|^2}\Big(\sum_{i \in T} e_i\Big)^2 ,$$

which Eigen et al. report as its square root, “RMSE (log, scale inv.)”. The NIPS paper prints the first form of this error (Eq. 1) with a factor $\frac{1}{2n}$ and the expanded form (Eq. 3) with $\frac{1}{n}$. Section 4.3 points to Eq. 1, but their reported values fit the $\frac{1}{n}$ form: in Section 3.2, removing the mean log error lowers their NYU log RMSE from 0.28 to 0.22, the values in the RMSE log and scale-invariant rows of their Table 1 (our check). The KITTI benchmark page shows the $\frac{1}{n}$ form above. The training loss is a different quantity: Eigen et al. trained with $\frac{1}{n}\sum_i e_i^2 - \frac{\lambda}{n^2}(\sum_i e_i)^2$ with $\lambda = 0.5$, while $D$ is the case $\lambda = 1$ (the monocular depth estimation article explains why).

Two more metrics complete the common set:

$$\log_{10}\text{ error} = \frac{1}{|T|}\sum_{i \in T} \big|\log_{10} \hat z_i - \log_{10} z_i\big|, \qquad \text{iRMSE} = \sqrt{\frac{1}{|T|}\sum_{i \in T} \Big(\frac{1}{\hat z_i} - \frac{1}{z_i}\Big)^2}.$$

Make3D (Saxena, Sun and Ng, 2009) reported the mean absolute $\log_{10}$ error, averaged over all pixels of its test images, together with the average relative error. iRMSE, the RMSE of inverse depth, is one of the metrics of the KITTI depth prediction benchmark, given in units of 1/km.

Ranges and units. AbsRel, RMSE log, SILog and the $\log_{10}$ error are unitless; RMSE is in meters and iRMSE in 1/km. SqRel divides a squared error in m² by a depth in m, so it too is in meters, which is why it grows with the depth range of the dataset. All errors are 0 for a perfect prediction and unbounded above; lower is better. The $\delta_k$ lie in $[0, 1]$ and higher is better. The KITTI depth prediction benchmark reports SILog as $100\sqrt{D}$ (“log(m)·100”) and its relative errors as percentages (as of October 2026), not as the fractions of Eigen-style tables.

Intuition

The metrics differ in what kind of error they count as “the same size”:

• AbsRel and SqRel divide by the true depth, so a 1 m error costs ten times more at 5 m than at 50 m. SqRel squares the error but divides only once, so it keeps one factor of depth and still grows with range.
• RMSE counts meters. A 10% error is 0.5 m at 5 m and 5 m at 50 m, so on outdoor scenes RMSE is set by the far pixels.
• RMSE log, $\log_{10}$ and SILog count ratios: overestimating by a factor of 2 costs the same as underestimating by a factor of 2. SILog additionally removes the mean log error, so a prediction that is right up to one global factor scores 0.
• $\delta_k$ counts pixels whose ratio to the truth lies within $[1/1.25^k, 1.25^k]$, that is, $|e_i| < k \log 1.25 \approx 0.223k$. It ignores how wrong the failing pixels are and how close the passing ones are.

Interpretation

AbsRel is asymmetric. Since depths are positive, an underestimate can cost at most 1 per pixel, while an overestimate is unbounded: predicting half of every depth gives AbsRel 0.5, predicting twice every depth gives 1.0, and both give the same RMSE log ($\log 2 \approx 0.693$) and $\delta_1 = 0$ (the last lines of the example below). A model that hedges toward smaller depths is therefore favoured by AbsRel (our reasoning from the formula).

$\delta_1$ saturates. A prediction that is 20% too deep everywhere has $\delta_1 = 1$, because $1.2 < 1.25$; only errors above 25% register at all. $\delta_2$ and $\delta_3$ ($1.5625$ and about $1.95$) are passed by almost every pixel of a reasonable model, which is why they sit close to 1 in most tables.

As a scale of reference, Eigen et al. (2014, Table 2) printed, on their KITTI test scenes, AbsRel 0.412 and $\delta_1$ 0.556 for a prediction that ignores the input (the mean depth image of the training set) and AbsRel 0.190 and $\delta_1$ 0.692 for their network.

Protocols That Change the Numbers

The same predictions can produce very different values under different evaluation code. Each choice below has been made differently by widely used papers.

Valid pixels and depth caps

Only pixels with ground truth enter $T$. On the KITTI Eigen split the ground truth is LiDAR reprojected into the image, covering under 5% of pixels (Godard et al., 2017), and the protocol also bounds the depth. The Monodepth code (at commit b76bee4) keeps ground-truth pixels with $10^{-3} < z_i <$ max_depth (default 80 m) and clips predictions to the same interval; Monodepth2 (evaluate_depth.py, commit b017a81) does the same with fixed bounds of $10^{-3}$ and 80 m. Godard et al. (2017) chose 80 m as roughly the maximum depth in KITTI. Garg et al. (2016, arXiv v2) instead clamped their predictions to 1–50 m, because supervised methods had been trained on ground truth in that range, and Godard et al. (2017) add rows labelled “cap 50m” for comparison with them.

The cap changes the result substantially. Godard et al. (2017, Table 2) evaluated one model (ResNet, post-processed, trained on Cityscapes and KITTI) three ways; the “Garg crop” label follows the table caption, while their text says they used Eigen’s crop:

Protocol	AbsRel	SqRel	RMSE (m)	$\delta < 1.25$
Garg crop, 80 m cap	0.114	0.898	4.935	0.861
Garg crop, 50 m cap	0.108	0.657	3.729	0.873
Uncropped	0.130	1.197	5.222	0.843

Dropping the 50–80 m pixels lowers RMSE by a quarter and SqRel by more, while AbsRel barely moves: the far pixels carry most of the squared error, not of the relative error.

Crops

Two rectangular crops are in use on KITTI, both written as fractions of the image size in the Monodepth code. The “Garg crop” keeps rows from 40.8% to 99.2% of the image height and columns from 3.6% to 96.4% of the width; the “Eigen crop” keeps rows from 33.2% to 91.4% with the same columns, and the code comment says it was found by trial and error to reproduce the numbers of Eigen et al. (2014). Garg et al. (2016, arXiv v2) describe their own evaluation region as the same crop as Eigen’s, so in our reading the two names record what later code used rather than two published definitions. Monodepth2 and SfMLearner apply the Garg crop to every Eigen-split image. On NYU Depth v2, the corresponding conventions are the center crop where projected Kinect depth is valid and a 10 m maximum depth; that article also shows how much filled-in ground truth changes the scores.

Splits and ground truth

On KITTI, the Eigen split is 697 test images from 29 scenes with raw LiDAR ground truth (Godard et al., 2017), which suffers from the rotating scanner, the vehicle’s motion and occlusions at object boundaries. The KITTI depth prediction benchmark uses the denser ground truth of Uhrig et al. (2017), described in the depth estimation hub, with test depth maps withheld on the server. Its page states that it ranks methods by the square root of the scale-invariant log error (SILog) and also lists sqErrorRel, absErrorRel and iRMSE (as of October 2026); the page shows the formula only for SILog. Monodepth2 (Godard et al., 2019) notes that 93% of Eigen-split test frames have the improved ground truth and evaluates on those as an alternative. Ranftl et al. (2022, arXiv v3, Section 6) tested on yet another set: the 161 images in both the Eigen test split and the official validation set with improved ground truth. A “KITTI AbsRel” without the split and ground truth is therefore ambiguous.

Scale alignment

Methods that predict depth only up to scale are scored after aligning each prediction to its ground truth, which the hub and monocular depth estimation articles describe in detail:

• Median scaling. Zhou et al. (2017, Section 4.1) multiply each predicted map by $\operatorname{median}(z)/\operatorname{median}(\hat z)$. In the SfMLearner and Monodepth2 code, the medians are taken over the masked and cropped pixels, and the prediction is clipped to the cap after scaling. Monodepth2 applies median scaling to monocular models only; models trained with any stereo supervision are instead multiplied by a fixed 5.4, the ratio between KITTI’s 0.54 m baseline and the nominal baseline of 0.1 units used in training.
• Scale and shift. Ranftl et al. (2022, arXiv v3, Section 6) align every prediction to the ground truth in scale and shift by least squares in inverse depth, per image, cap depths at each dataset’s maximum (80 m for KITTI, 10 m for NYU Depth v2 and TUM, 72 m for ETH3D), and choose one metric per dataset: AbsRel for ETH3D and Sintel, the percentage of pixels with $\delta > 1.25$ for KITTI, NYU Depth v2 and TUM, and RMSE in disparity space for datasets that have only relative depth.

The idea is older than either: Make3D (Saxena et al., 2009) scaled and shifted the depth maps of the method of Hoiem et al. to match the global scale of its test images, noting that without this that method’s relative error rose to 7.533.

Averaging over images

The Monodepth, Monodepth2 and SfMLearner scripts compute every metric for each image and then average the per-image values, so the reported RMSE is the mean of per-image RMSEs, not the RMSE of all pixels pooled; images with few valid pixels count as much as dense ones. Make3D averaged its $\log_{10}$ error over all pixels of the test set instead. For linear metrics the two agree when every image has the same number of valid pixels, which sparse LiDAR never gives; for RMSE they differ even then, as Mean Squared Error shows.

Example

The code builds a road scene with known depth: a camera 1.65 m above a flat road, sky without ground truth, and four boxes (a wall at 65 m, a building at 38 m, cars at 22 m and 9 m). Four predictions have known errors: (a) every depth 20% too large, (b) 10% log-normal noise, so the error is proportional to depth, (c) a 30% overestimate only beyond 30 m, and (d) inverse depth blurred by a Gaussian of 2 pixels, which mixes foreground and background at every edge. Each is scored with an 80 m and a 50 m cap, with and without median scaling, following Monodepth2’s order of operations; SILog is printed as $100\sqrt{D}$, as on the KITTI server.

import numpy as np
from scipy.ndimage import gaussian_filter

# Ground-truth depth of a road scene seen by a camera 1.65 m above a flat road.
H, W, f, h = 128, 416, 360.0, 1.65
cy, cx = H / 2, W / 2
v, u = np.mgrid[0:H, 0:W] + 0.5
Z = np.full((H, W), np.inf)                        # sky: no ground truth
road = v > cy
Z[road] = f * h / (v[road] - cy)                    # road plane: Z = f h / (v - c_y)
# Fronto-parallel boxes standing on the road: (centre X, width, height, depth Z), metres.
for X, w, ht, z in [(4.0, 30.0, 12.0, 65.0), (-6.0, 8.0, 6.0, 38.0),
                    (2.0, 1.8, 1.5, 22.0), (-1.5, 1.8, 1.5, 9.0)]:
    rows = (v >= cy + f * (h - ht) / z) & (v <= cy + f * h / z)
    cols = np.abs(u - (cx + f * X / z)) <= f * w / 2 / z
    Z = np.where(rows & cols, np.minimum(Z, z), Z)

def metrics(p, g):
    r, d = np.maximum(p / g, g / p), np.log(p) - np.log(g)
    return {"AbsRel": np.mean(np.abs(p - g) / g), "SqRel": np.mean((p - g) ** 2 / g),
            "RMSE": np.sqrt(np.mean((p - g) ** 2)), "RMSElog": np.sqrt(np.mean(d ** 2)),
            "SILog": 100 * np.sqrt(np.mean(d ** 2) - np.mean(d) ** 2),
            "log10": np.mean(np.abs(np.log10(p) - np.log10(g))),
            "d1": np.mean(r < 1.25), "d2": np.mean(r < 1.25 ** 2), "d3": np.mean(r < 1.25 ** 3)}

def evaluate(pred, gt, cap, median):                # order of operations as in Monodepth2
    m = (gt > 1e-3) & (gt < cap)
    p, g = pred[m], gt[m]
    if median:
        p = p * np.median(g) / np.median(p)
    return metrics(np.clip(p, 1e-3, cap), g)

def predictions(seed):
    rng = np.random.default_rng(seed)
    inv = np.where(np.isfinite(Z), 1 / Z, 0.0)      # inverse depth, sky = 0
    return {"(a) scale x1.2": 1.2 * Z,
            "(b) noise ~ depth": Z * np.exp(rng.normal(0, 0.1, Z.shape)),
            "(c) +30% beyond 30 m": np.where(Z > 30, 1.3 * Z, Z),
            "(d) blurred edges": 1 / np.maximum(gaussian_filter(inv, 2.0), 1e-3)}

for cap in [80, 50]:
    m = (Z > 1e-3) & (Z < cap)
    print(f"cap {cap} m: {m.sum()} valid pixels, {np.mean(Z[m] > 30):.1%} beyond 30 m")
names = list(metrics(np.ones(1), np.ones(1)))
print(f"{'prediction':22s}{'cap':>4s}{'med':>4s}" + "".join(f"{n:>8s}" for n in names))
for name, P in predictions(0).items():
    for cap in [80, 50]:
        for med in [False, True]:
            r = evaluate(P, Z, cap, med)
            print(f"{name:22s}{cap:4d}{'y' if med else 'n':>4s}" + "".join(f"{r[n]:8.3f}" for n in names))

m80 = (Z > 1e-3) & (Z < 80)
print("median-scaling ratio, 80 m cap: " + ", ".join(
    f"{n[:3]} {np.median(Z[m80]) / np.median(P[m80]):.3f}" for n, P in predictions(0).items()))

# Spread of (b) over seeds 0-19 (cap 80 m, no scaling).
runs = np.array([[evaluate(predictions(s)["(b) noise ~ depth"], Z, 80, False)[n] for n in names]
                 for s in range(20)])
spread = [f"{n} {lo:.3f}-{hi:.3f}" for n, lo, hi in zip(names, runs.min(0), runs.max(0))]
print("(b) over seeds 0-19: " + ", ".join(spread[:5]) + ",\n                     " + ", ".join(spread[5:]))

# Which pixels carry each metric? Share of pixels and of each error sum, by true depth.
g = Z[m80]
rng = np.random.default_rng(0)
for label, p in [("noise ~ depth (10%)", g * np.exp(rng.normal(0, 0.1, g.size))),
                 ("constant noise (1 m)", np.maximum(g + rng.normal(0, 1.0, g.size), 1e-3))]:
    print(label)
    abs_rel, sq = np.abs(p - g) / g, (p - g) ** 2
    for lo, hi in [(0, 10), (10, 20), (20, 40), (40, 80)]:
        b = (g >= lo) & (g < hi)
        print(f"  {lo:2d}-{hi:2d} m: pixels {b.mean():6.1%}   AbsRel sum {abs_rel[b].sum() / abs_rel.sum():6.1%}"
              f"   squared-error sum {sq[b].sum() / sq.sum():6.1%}")

# Asymmetry: halving and doubling every depth.
for k in [0.5, 2.0]:
    r = metrics(k * g, g)
    print(f"pred = {k} x gt: AbsRel {r['AbsRel']:.3f}, RMSElog {r['RMSElog']:.3f}, d1 {r['d1']:.3f}")

Output:

cap 80 m: 35969 valid pixels, 44.9% beyond 30 m
cap 50 m: 25956 valid pixels, 23.6% beyond 30 m
prediction             cap med  AbsRel   SqRel    RMSE RMSElog   SILog   log10      d1      d2      d3
(a) scale x1.2          80   n   0.199   1.270   7.732   0.182   1.375   0.079   1.000   1.000   1.000
(a) scale x1.2          80   y   0.000   0.000   0.000   0.000   0.000   0.000   1.000   1.000   1.000
(a) scale x1.2          50   n   0.197   0.766   4.415   0.181   1.539   0.078   1.000   1.000   1.000
(a) scale x1.2          50   y   0.000   0.000   0.000   0.000   0.000   0.000   1.000   1.000   1.000
(b) noise ~ depth       80   n   0.080   0.317   3.858   0.099   9.940   0.035   0.977   1.000   1.000
(b) noise ~ depth       80   y   0.079   0.313   3.838   0.101   9.955   0.035   0.974   1.000   1.000
(b) noise ~ depth       50   n   0.080   0.197   2.245   0.100   9.972   0.035   0.974   1.000   1.000
(b) noise ~ depth       50   y   0.079   0.194   2.233   0.100   9.976   0.035   0.973   1.000   1.000
(c) +30% beyond 30 m    80   n   0.115   1.549   9.175   0.155  11.610   0.044   0.804   1.000   1.000
(c) +30% beyond 30 m    80   y   0.115   1.549   9.175   0.155  11.610   0.044   0.804   1.000   1.000
(c) +30% beyond 30 m    50   n   0.065   0.698   5.100   0.120  10.593   0.025   0.800   1.000   1.000
(c) +30% beyond 30 m    50   y   0.065   0.698   5.100   0.120  10.593   0.025   0.800   1.000   1.000
(d) blurred edges       80   n   0.020   0.227   3.301   0.081   8.149   0.009   0.975   0.991   0.997
(d) blurred edges       80   y   0.034   0.236   3.348   0.083   8.098   0.015   0.975   0.991   0.997
(d) blurred edges       50   n   0.016   0.134   2.121   0.078   7.780   0.007   0.973   0.992   0.996
(d) blurred edges       50   y   0.016   0.134   2.121   0.078   7.780   0.007   0.973   0.992   0.996
median-scaling ratio, 80 m cap: (a) 0.833, (b) 0.985, (c) 1.000, (d) 1.016
(b) over seeds 0-19: AbsRel 0.079-0.080, SqRel 0.314-0.324, RMSE 3.834-3.902, RMSElog 0.099-0.100, SILog 9.882-10.020,
                     log10 0.034-0.035, d1 0.976-0.979, d2 1.000-1.000, d3 1.000-1.000
noise ~ depth (10%)
   0-10 m: pixels  16.4%   AbsRel sum  16.5%   squared-error sum   0.9%
  10-20 m: pixels  27.7%   AbsRel sum  27.8%   squared-error sum   3.6%
  20-40 m: pixels  25.4%   AbsRel sum  25.1%   squared-error sum  16.5%
  40-80 m: pixels  30.4%   AbsRel sum  30.5%   squared-error sum  78.9%
constant noise (1 m)
   0-10 m: pixels  16.4%   AbsRel sum  34.2%   squared-error sum  16.3%
  10-20 m: pixels  27.7%   AbsRel sum  39.9%   squared-error sum  27.7%
  20-40 m: pixels  25.4%   AbsRel sum  16.5%   squared-error sum  25.2%
  40-80 m: pixels  30.4%   AbsRel sum   9.5%   squared-error sum  30.8%
pred = 0.5 x gt: AbsRel 0.500, RMSElog 0.693, d1 0.000
pred = 2.0 x gt: AbsRel 1.000, RMSElog 0.693, d1 0.000

A global scale error (a) moves every unaligned metric except the threshold accuracies: 20% stays inside the factor 1.25, so $\delta_1 = 1$ while AbsRel is 0.199. SILog is not exactly 0, because the cap clips the predictions of pixels deeper than 66.7 m and changes their ratio alone; median scaling, with ratio $0.833 = 1/1.2$, removes the error completely.

Proportional noise (b) gives nearly the same AbsRel, RMSE log, SILog and $\delta_1$ under both caps, while RMSE and SqRel drop by about 40% when the 50–80 m pixels leave. The contribution table shows why: under proportional noise, pixels beyond 40 m are 30% of the total and carry 79% of the squared error, while each depth band’s share of the AbsRel sum equals its share of pixels. With a constant 1 m error the pattern reverses: pixels nearer than 20 m are 44% of the total and carry 74% of the AbsRel sum, and the squared error follows the pixel count. Over seeds 0–19, (b)’s RMSE varies by under 0.07 m and its AbsRel by 0.001, far less than the differences between protocols.

Far-range error (c) is invisible to median scaling, since the median pixel lies in the error-free part (ratio 1.000), and very visible to the cap: AbsRel falls from 0.115 to 0.065 and RMSE from 9.2 to 5.1 m at 50 m. $\delta_1$ barely changes between caps, for a misleading reason: with the 80 m cap, a 30% overestimate exceeds 80 m for true depths beyond 61.5 m and is clipped, and beyond 64 m the clipped value lies within a factor 1.25 of the truth, so clipping turns failing pixels into passing ones.

Blurred edges (d) produce errors only next to depth discontinuities. It has the lowest AbsRel and $\log_{10}$ error but an RMSE close to (b)’s, because the boxes stand in front of surfaces tens of meters behind them, and it is the only prediction with ratios beyond $1.25^2$, so $\delta_2$ and $\delta_3$ fall below 1. Median scaling hurts it: the ratio 1.016 shifts every exact pixel by 1.6%, and AbsRel rises from 0.020 to 0.034.

When to Use It

Report the full set, not one metric: AbsRel and $\delta_1$ for relative accuracy, RMSE for the error in meters that a downstream planner or 3D reconstruction will see, and RMSE log or SILog to separate shape from scale. Use SILog when the method is not expected to know absolute scale, and the unaligned metrics when it is. Always state the dataset split, the ground truth, the depth range and cap, the crop, the alignment (none, median, or least-squares scale and shift, in which space) and whether averages are over images or pixels; the statistical significance of small differences also depends on how many images were averaged.

For stereo, where the error is roughly constant in pixels of disparity, disparity-space metrics are the natural choice: the bad-pixel rate and KITTI’s D1 outlier rate, and the average disparity error, the one-dimensional counterpart of endpoint error. Godard et al. (2017) note that measuring error in depth when the ground truth is given in disparities makes the non-thresholded depth metrics sensitive to the large depth errors caused by prediction errors at small disparities, that is, at long range. For a fixed error in meters, inverse depth and disparity change most for near pixels, so iRMSE, MiDaS’s disparity RMSE and the stereo metrics weight near pixels heavily, the reverse of RMSE in depth (our reasoning from the definitions).

Failure Cases

• A 20% bias passes $\delta_1$. Prediction (a) above.
• Clipping at the cap hides errors. Overestimates near the cap are clipped back toward the truth, and clipping also breaks the scale invariance of SILog.
• Median scaling hides the scale problem and can add error. Aligned numbers say nothing about whether a method knows the scale of the world; the spread of per-image scale factors, which Monodepth2’s supplementary material reports, shows how inconsistent it is (monocular depth estimation). When most pixels are already exact, a median that falls among the erroneous ones shifts every pixel.
• RMSE and SqRel are dominated by the far range, so a method can win on them by improving a band of distant road while getting nearby obstacles wrong, or lose to a method that clips more aggressively.
• AbsRel favours underestimation, by the asymmetry above.
• The ground truth defines what is measured. Sparse LiDAR skips the sky, glass and dark surfaces and is misaligned at object boundaries; filled-in ground truth on NYU Depth v2 rewards agreement with the filling algorithm in the holes.
• Numbers from different protocols are not comparable, even on the same dataset; the 50 m and 80 m rows of Godard et al. (2017) differ by a quarter in RMSE for one model.

Variants

• Scale-invariant metrics. SILog (the square root of $D$, times 100 on KITTI) for up-to-scale outputs; metrics after median or scale-and-shift alignment for relative-depth models.
• Inverse-depth metrics. iRMSE on the KITTI depth prediction benchmark and disparity RMSE for relative-depth datasets in MiDaS.
• Disparity outlier rates. The bad-pixel percentage and D1 for stereo, defined in stereo matching.
• Ordinal metrics for datasets with only relative depth annotations, such as the weighted human disagreement rate that Ranftl et al. (2022) use for the DIW dataset.

Related Metrics

• Mean Squared Error owns RMSE and MAE in general, and the difference between per-image and pooled averaging.
• Endpoint Error is the corresponding average error for optical flow, with outlier percentages that play the role of $\delta_k$.
• Chamfer distance compares reconstructed 3D points without a pixel correspondence, for evaluating the point clouds that depth maps back-project to.