ETH3D

ETH3D is a set of benchmarks for dense 3D reconstruction created at ETH Zürich, built on laser scans of indoor and outdoor scenes. It was introduced by Thomas Schöps, Johannes Schönberger, Silvano Galliani, Torsten Sattler, Konrad Schindler, Marc Pollefeys and Andreas Geiger at CVPR 2017 with three stereo tasks: high-resolution multi-view stereo from DSLR photographs, low-resolution “many-view” stereo on videos from a four-camera rig, and two-view stereo matching on rectified pairs from the same rig. The data was first released on 19 July 2017 and extended on 4 October 2017. In 2019 a fourth benchmark, for RGB-D, stereo and visual-inertial SLAM, was added with the BAD SLAM paper of Schöps, Sattler and Pollefeys. All four keep the ground truth of their test sets on an evaluation server.

Purpose

The authors set out to complement the multi-view benchmarks of the time in several respects (Schöps et al., 2017, Section 1, which numbers three; the third covers both scenes and viewpoints):

• Resolution. The DSLR images have 24 megapixels, against 6 for the Strecha et al. (2008) benchmark, 0.5 for KITTI and 0.3 for the Middlebury multi-view benchmark.
• Mobile capture. Hand-held video from synchronized low-resolution cameras with auto-exposure, the setting of real-time stereo on phones and robots.
• Unstaged scenes. The scenes are offices, courtyards, facades and natural scenes with vegetation, not laboratory set-ups.
• Unconstrained viewpoints. The cameras move with six degrees of freedom instead of along a fixed trajectory.

Their comparison table lists ETH3D as the only benchmark with all six properties compared: online evaluation, six-degree-of-freedom camera motion, multi-view stereo, two-view stereo, video and varying fields of view (Table 1). It was developed at the same time as Tanks and Temples (Knapitsch et al., 2017), which also uses laser-scanned ground truth but evaluates complete structure-from-motion and MVS pipelines on high-quality video; the ETH3D authors describe the two as complementary, and Knapitsch et al. say the same.

Dataset Size

Counts as stated on the official overview and datasets pages, as of October 2026:

Benchmark	Training	Test	Images
High-res multi-view	13 scenes, 454 DSLR images	12 scenes, 444 DSLR images	Nikon D3X, 6048 × 4032
Low-res many-view	5 videos, 4 × 165 to 4 × 300 frames	5 videos, 4 × 199 to 4 × 352 frames	4 cameras, 752 × 480, about 13.6 Hz
Low-res two-view	27 rectified pairs	20 rectified pairs	from the same rig
SLAM	61 sequences	35 sequences	RGB-D, stereo RGB and IMU, mostly about 27.1 Hz

The image totals of the high-res rows are the sums of the per-scene counts on the datasets page. The datasets page lists 16 training and 10 test captures for the two-view task; the result tables list each capture under one or two pairs (columns suffixed “l” and “s”), in our reading one for each stereo pair of the rig. All 10 test captures have both, while 5 of the 16 training captures (facade, forest and terrace) have only the “s” pair, which gives the 27 and 20 of the overview page. The paper explains the pairing: each two-view frame consists of the four images the rig takes at one time, forming two stereo pairs whose cameras share a field of view. The changelog states that a few pairs with questionable calibration were removed in October 2017.

The high-res training scenes are courtyard, delivery area, electro, facade, kicker, meadow, office, pipes, playground, relief, relief 2, terrace and terrains, with 14 to 76 images each; the test scenes have 7 (door) to 110 (bridge). The result pages group the test scenes as 9 indoor and 3 outdoor. The SLAM numbers come from the BAD SLAM paper (Section 5), which counts 61 training and 35 test datasets; the SLAM overview page splits the training set into 56 sequences with motion-capture ground truth and 5 with structure-from-motion ground truth.

Annotation Types

Laser scans. Ground-truth geometry was recorded with a Faro Focus X 330 scanner: one or more 360° scans per scene, each with up to about 28 million points and point colors from the scanner’s camera, taking about 9 minutes per scan (Schöps et al., 2017, Section 3.1). The scans were cleaned with statistical outlier removal, using a per-point threshold because point density falls with distance from the scanner, and remaining systematic errors were removed by hand. Image regions that should not be used, such as moving branches, transparent surfaces and areas where occlusion reasoning fails, were masked manually.

Image-to-scan registration. A three-stage pipeline aligns the images with the scans (Section 3.2):

1. Initial alignment. Each scan is rendered into the six faces of a cube map centered at the scanner, with missing pixels filled from the nearest neighbor, and these renderings are reconstructed together with the photographs by COLMAP’s structure from motion, which matches SIFT features between real and rendered images. Comparing SfM depths with the rendered depths gives the initial scale.
2. Scan refinement. The scan poses are refined jointly by point-to-plane ICP and then fixed.
3. Image refinement. Camera intrinsics and extrinsics are refined by dense photometric alignment, an extension of Zhou and Koltun’s color-map optimization. Because auto-exposure and outdoor lighting break brightness constancy, the cost compares intensity differences between neighboring points, $g(p, p_j)$, instead of intensities, under a Huber loss; a multi-resolution point cloud and image pyramid keep each point’s neighbors about one pixel apart in every image. A second term ties the image gradients to the laser scan’s colors, which reduces drift but makes the result depend on how well those colors are aligned with the geometry. The rig’s relative camera poses are kept rigid.

Visibility during alignment is decided by rendering a screened Poisson surface of the scan, augmented with splats (oriented discs) for thin structures such as wires. A point counts as visible if it is no more than 1 cm behind the rendered depth. The supplementary material gives the implementation details and an ablation in which the brightness-constancy variant diverges and the single-resolution variant stays near its initialization. The paper evaluates the registration only qualitatively, for lack of a more precise reference, and judges the alignments of both cameras “generally pixel accurate” from depth-edge overlays (Section 5.1).

Released ground truth. For training scenes, each scan is available in three versions (documentation page):

• raw: as recorded, outliers included;
• clean: outliers removed;
• eval: cleaned and stripped of every point assumed to be seen by at most one image, the version used for evaluation, with separate files for the DSLR and rig images of a scene.

Scan poses are given in a MeshLab project file. Training scenes also come with the occlusion data (Poisson mesh, splats and image masks), depth maps rendered from the scans since April 2018, and calibration in COLMAP’s text format with a thin-prism fisheye camera model for the original images and a pinhole model for the undistorted ones.

Two-view ground truth is the laser scan projected into the rectified left image, with occluded points dropped by the same occlusion reasoning (Section 4). It is stored as a Middlebury-style PFM disparity map with infinity for missing values, and a mask marks each pixel as without ground truth (0), seen only by the left image (128), or seen by both (255).

SLAM ground truth comes from a motion-capture system, except for the five structure-from-motion training sequences, which are kept in a separate category because they are likely less accurate (BAD SLAM, Section 5). The rig pairs synchronized global-shutter RGB cameras with active-stereo depth computed from two infrared cameras and the projector of an Asus Xtion, so color and depth are captured at the same instants.

Splits

Each benchmark has a training set with public ground truth and a test set whose ground truth is withheld; results on both are submitted through the website, with runtimes required for every submission. The October 2017 extension changed the split: it added 12 high-res, 5 low-res and 20 two-view datasets as the new test set and turned all earlier test data into training data, so that every dataset in the 2017 paper now has public ground truth (changelog). Results in the paper therefore refer to scenes that are now in the training set. Category averages, and hence category rankings, are computed only for methods with results on every dataset in the category (results pages).

Tasks

• High-resolution multi-view stereo: a point cloud of each scene from tens of calibrated DSLR images.
• Low-resolution many-view stereo: a point cloud from hundreds of video frames per camera, where the challenge is to exploit redundancy at low resolution.
• Two-view stereo: a disparity map for the left image of a rectified pair.
• SLAM: camera trajectories for monocular, stereo and RGB-D input, with or without the IMUs. The trajectory errors are described under absolute trajectory error.

Camera poses are given for the stereo tasks, so they measure dense reconstruction, not pose estimation.

Evaluation

Multi-view: accuracy, completeness and F1

ETH3D compares a submitted point cloud with the laser scan at a set of distance tolerances $t$. The results pages offer 1, 2, 5, 10, 20 and 50 cm and show 2 cm by default. The definitions below follow the official program at commit 0daa4f4 (C++, depending on PCL, Eigen and Boost), which was read line by line; it could not be built here without installing those libraries.

Free space from the scanner’s beams. Because the scan misses surfaces the scanner could not see, a reconstructed point far from every scan point is not necessarily wrong. The program models each scan point $s$ as a beam from the scanner origin, a cone whose radius at distance $r$ is

$$b(r) = b_0 + r \tan\alpha, \qquad b_0 = 1.125\ \text{mm}, \quad \alpha = 0.011^\circ,$$

(the defaults --beam_start_radius_meters and --beam_divergence_halfangle_deg). For a reconstructed point $\mathbf{p}$ in the scanner’s frame, write $\mathbf{u}_s = \mathbf{s}/\lVert \mathbf{s} \rVert$ and $\lambda = \mathbf{u}_s^\top \mathbf{p}$. The point is inside the beam of $s$ if $\lambda \ge 0$ and $\lVert \mathbf{p} - \lambda \mathbf{u}_s \rVert \le b(\lVert \mathbf{p} \rVert)$. At tolerance $t$ the point is then:

• accurate if it is inside the beam of some scan point with $\lVert \mathbf{p} - \mathbf{s} \rVert \le t$;
• inaccurate if it is not accurate but lies inside some beam in front of its scan point ($\lambda < \lVert \mathbf{s} \rVert$), in space the laser passed through;
• unobserved otherwise, and left out of the accuracy.

With several scans, a point is accurate if any scan makes it accurate, otherwise inaccurate if any scan makes it inaccurate. A point behind a scanned surface by more than $t$ is unobserved, unless another scan’s beam passes through it, as in the paper’s Figure 3, where the beam is extended past each scan point only by a sphere of radius $t$.

Completeness is the share of scan points that have a reconstructed point within $t$, found by a k-d tree search; for it, all scans are merged into one cloud. The scan here is the evaluation version, which already excludes points seen by fewer than two images.

Voxel averaging. To stop a submission from gaming either measure through point density, both are averaged over voxels rather than points (paper, Section 4). The program uses voxels of 1 cm (--voxel_size) on two grids, the second shifted by half a voxel along each axis. For completeness, every voxel that contains scan points contributes its fraction of complete scan points, while the reconstruction searched against is the whole cloud. For accuracy, every voxel that contains at least one observed reconstructed point at that tolerance contributes its fraction of accurate points. The cells of both grids are pooled and averaged with equal weight:

$$A(t) = \frac{1}{|\mathcal{V}_A(t)|}\sum_{v \in \mathcal{V}_A(t)} \frac{a_v(t)}{a_v(t) + i_v(t)}, \qquad C(t) = \frac{1}{|\mathcal{V}_C|}\sum_{v \in \mathcal{V}_C} \frac{c_v(t)}{n_v},$$

where $a_v$ and $i_v$ count accurate and inaccurate reconstructed points in voxel $v$, $c_v$ the complete scan points among its $n_v$ scan points, and $\mathcal{V}_A(t)$, $\mathcal{V}_C$ the voxels with observed reconstructed points and with scan points. The ranking measure is

$$F_1(t) = \frac{2\,A(t)\,C(t)}{A(t) + C(t)},$$

set to 0 when both are 0. ETH3D’s accuracy and completeness are the precision and recall of Tanks and Temples under other names; the free-space model and the per-voxel averaging are what differ (Tanks and Temples instead resamples both clouds to one mean point per voxel and counts points). The Chamfer distance article compares such thresholded counts with averaged distances.

Code and paper. The two agree on the definitions; the code adds what the paper leaves open:

• the beam parameters;
• the voxel size and the second, shifted grid;
• the rule for combining scans;
• that accuracy voxels are counted per tolerance, so the set of voxels averaged changes with $t$.

The two-image visibility filter that the paper states is applied when the evaluation point clouds are prepared, not in the program. The results pages do not state the voxel or beam settings the server uses, so the program’s defaults are the best available reading of the official numbers.

Example: the protocol on a synthetic scan

The script below reimplements the program in NumPy and SciPy. A scanner at the origin scans a 0.6 m wall at 1 m and a 12 cm box face at 0.4 m that shadows part of the wall, on a grid of rays spaced 1.5 mrad apart, which leaves no gaps between the default beams. Two reconstructions sample the true surfaces, including the shadowed wall that cameras could see but the scanner could not: one complete with 1.5 cm Gaussian noise on each coordinate, one with 2 mm noise and the right 45% of the wall missing. A second part adds clusters of points behind the wall and in free space, and repeats the attack the paper describes: a fog that fills the volume, plus 200,000 copies of one accurate point.

import numpy as np
from scipy.spatial import cKDTree

TOL = np.array([0.01, 0.02, 0.05, 0.10, 0.20, 0.50])  # metres, as on the ETH3D site
VOXEL = 0.01                                         # --voxel_size default
BEAM0 = 0.5 * 0.00225                                # --beam_start_radius_meters
TAN_DIV = np.tan(np.radians(0.011))                  # --beam_divergence_halfangle_deg

# Scene in the scanner frame (scanner at the origin, x forward, z up):
# a 0.6 m x 0.6 m wall at x = 1.0 and a 0.12 m box face at x = 0.4 that shadows it.
WALL_X, HALF_W, BOX_X, BOX_H = 1.0, 0.3, 0.4, 0.06

def laser_scan(step=0.0015):
    """Cast rays on an azimuth/inclination grid; keep the first hit."""
    a = np.arange(-0.30, 0.30, step)
    az, el = np.meshgrid(a, a)
    d = np.stack([np.cos(el) * np.cos(az), np.cos(el) * np.sin(az), np.sin(el)], -1)
    d = d.reshape(-1, 3)
    pts = []
    for x0, h in [(BOX_X, BOX_H), (WALL_X, HALF_W)]:
        t = x0 / d[:, 0]
        p = d * t[:, None]
        pts.append(np.where((np.abs(p[:, 1:]) <= h).all(1)[:, None], p, np.nan))
    box, wall = pts
    hit = np.where(np.isnan(box[:, :1]), wall, box)
    return hit[~np.isnan(hit[:, 0])]

def sample_surface(n, rng, y_max=np.inf):
    """Uniform samples of the wall (also behind the box) and the box face;
    wall samples with y >= y_max are dropped."""
    a_wall, a_box = (2 * HALF_W) ** 2, (2 * BOX_H) ** 2
    n_wall = int(n * a_wall / (a_wall + a_box))
    y, z = rng.uniform(-HALF_W, HALF_W, (2, n_wall))
    keep = y < y_max
    wall = np.c_[np.full(keep.sum(), WALL_X), y[keep], z[keep]]
    y, z = rng.uniform(-BOX_H, BOX_H, (2, n - n_wall))
    return np.vstack([wall, np.c_[np.full(n - n_wall, BOX_X), y, z]])

def voxel_mean(points, num, den):
    """Average num/den over the cells of two voxel grids shifted by half a voxel.
    Cells with den == 0 are skipped (accuracy: only observed points count)."""
    total, cells = np.zeros(num.shape[1]), np.zeros(num.shape[1])
    for shift in (0.0, 0.5):
        key = np.floor(points / VOXEL + shift).astype(np.int64)
        _, cell = np.unique(key, axis=0, return_inverse=True)
        n_cells = cell.max() + 1
        for k in range(num.shape[1]):
            nc = np.bincount(cell, num[:, k], n_cells)
            dc = np.bincount(cell, den[:, k], n_cells)
            ok = dc > 0
            total[k] += (nc[ok] / dc[ok]).sum()
            cells[k] += ok.sum()
    return total / np.maximum(cells, 1)

def completeness(scan, recon):
    d, _ = cKDTree(recon).query(scan, distance_upper_bound=TOL[-1])
    complete = (d[:, None] <= TOL).astype(float)
    return voxel_mean(scan, complete, np.ones_like(complete)), complete.mean(0)

def accuracy(scan, scan_dir_tree, recon):
    """Per point: accurate if inside a scan point's beam and within t of it;
    inaccurate if inside a beam in front of its scan point (free space);
    otherwise unobserved and ignored."""
    r = np.linalg.norm(recon, axis=1)
    beam = BEAM0 + r * TAN_DIV
    u = recon / r[:, None]
    lists = scan_dir_tree.query_ball_point(u, beam / r * 1.01)
    lens = np.fromiter(map(len, lists), int, len(lists))
    pi = np.repeat(np.arange(len(recon)), lens)
    si = np.fromiter((j for l in lists for j in l), int, lens.sum())
    s, p = scan[si], recon[pi]
    s_r = np.linalg.norm(s, axis=1)
    along = (s / s_r[:, None] * p).sum(1)
    in_beam = (along >= 0) & ((p * p).sum(1) - along**2 <= beam[pi] ** 2)
    dist = np.linalg.norm(s - p, axis=1)
    acc = np.zeros((len(recon), len(TOL)), bool)
    for k, t in enumerate(TOL):
        np.logical_or.at(acc[:, k], pi[in_beam & (dist <= t)], True)
    front = np.zeros(len(recon), bool)
    np.logical_or.at(front, pi[in_beam & (along < s_r)], True)
    inacc = ~acc & front[:, None]
    observed = (acc | inacc).astype(float)
    per_point = acc.sum(0) / np.maximum(observed.sum(0), 1)
    return voxel_mean(recon, acc.astype(float), observed), per_point, observed.mean(0)

def f1(p, r):
    return np.where(p + r > 0, 2 * p * r / np.maximum(p + r, 1e-12), 0.0)

scan = laser_scan()
dir_tree = cKDTree(scan / np.linalg.norm(scan, axis=1)[:, None])
print(f"laser scan: {len(scan)} points")

def evaluate(recon):
    acc, acc_pt, obs = accuracy(scan, dir_tree, recon)
    comp, _ = completeness(scan, recon)
    return acc, comp, f1(acc, comp), acc_pt, obs

def noisy_complete(rng):    # every surface, 1.5 cm noise on each coordinate
    return sample_surface(60000, rng) + rng.normal(0, 0.015, (60000, 3))

def precise_partial(rng):   # 2 mm noise, but the right 45% of the wall is missing
    x = sample_surface(60000, rng, y_max=0.03)
    return x + rng.normal(0, 0.002, x.shape)

rng = np.random.default_rng(0)
A, B = noisy_complete(rng), precise_partial(rng)
ra, rb = evaluate(A), evaluate(B)
print("\ntol [cm]   noisy+complete acc/comp/F1   precise+partial acc/comp/F1   higher F1")
for k, t in enumerate(TOL):
    print(f"{100 * t:6.0f}      {ra[0][k]:.3f} {ra[1][k]:.3f} {ra[2][k]:.3f}"
          f"             {rb[0][k]:.3f} {rb[1][k]:.3f} {rb[2][k]:.3f}"
          f"         {'noisy' if ra[2][k] > rb[2][k] else 'precise'}")
print(f"share of noisy points observed at 1 cm: {ra[4][0]:.3f}")

print("\nF1 at 1 cm and 2 cm over seeds 0-4 (noisy, precise):")
for seed in range(5):
    g = np.random.default_rng(seed)
    a, b = evaluate(noisy_complete(g)), evaluate(precise_partial(g))
    print(f"  seed {seed}: 1 cm {a[2][0]:.3f} {b[2][0]:.3f}   2 cm {a[2][1]:.3f} {b[2][1]:.3f}")

def nn_accuracy(recon, t=0.02):  # no beams, no voxels: share of points within t of the scan
    return (cKDTree(scan).query(recon, distance_upper_bound=t)[0] <= t).mean()

# Points behind the wall, points in free space, a fog, and copies of one good point.
behind = rng.uniform([1.10, -0.2, -0.2], [1.30, 0.2, 0.2], (6000, 3))
front = rng.uniform([0.70, -0.2, -0.2], [0.90, 0.2, 0.2], (6000, 3))
fog = rng.uniform([0.3, -0.35, -0.35], [1.05, 0.35, 0.35], (60000, 3))
copies = np.repeat([[WALL_X, -0.25, -0.25]], 200000, axis=0)
cases = [("noisy+complete", A), ("precise+partial", B),
         ("  + 6k points behind wall", np.vstack([B, behind])),
         ("  + 6k points in free space", np.vstack([B, front])),
         ("fog of 60k points", fog), ("  + 200k copies of 1 point", np.vstack([fog, copies]))]
print("\nat 2 cm                     voxel acc  point acc  NN acc  voxel comp  F1 (voxel)")
for name, rec in cases:
    acc, comp, f, acc_pt, _ = evaluate(rec)
    print(f"{name:27s} {acc[1]:9.3f} {acc_pt[1]:10.3f} {nn_accuracy(rec):7.3f}"
          f" {comp[1]:11.3f} {f[1]:11.3f}")

Output (NumPy 2.3.5, SciPy 1.16.3; about 8 s):

laser scan: 149173 points

tol [cm]   noisy+complete acc/comp/F1   precise+partial acc/comp/F1   higher F1
     1      0.528 1.000 0.691             0.996 0.561 0.718         precise
     2      0.789 1.000 0.882             0.996 0.573 0.727         noisy
     5      0.980 1.000 0.990             0.996 0.605 0.753         noisy
    10      0.981 1.000 0.990             0.996 0.659 0.793         noisy
    20      0.981 1.000 0.990             0.996 0.855 0.920         noisy
    50      0.981 1.000 0.990             0.996 1.000 0.998         precise
share of noisy points observed at 1 cm: 0.534

F1 at 1 cm and 2 cm over seeds 0-4 (noisy, precise):
  seed 0: 1 cm 0.691 0.718   2 cm 0.882 0.727
  seed 1: 1 cm 0.690 0.717   2 cm 0.882 0.726
  seed 2: 1 cm 0.691 0.718   2 cm 0.883 0.727
  seed 3: 1 cm 0.692 0.718   2 cm 0.882 0.727
  seed 4: 1 cm 0.692 0.717   2 cm 0.884 0.726

at 2 cm                     voxel acc  point acc  NN acc  voxel comp  F1 (voxel)
noisy+complete                  0.789      0.884   0.650       1.000       0.882
precise+partial                 0.996      0.999   0.804       0.573       0.727
  + 6k points behind wall       0.996      0.999   0.683       0.573       0.727
  + 6k points in free space     0.434      0.869   0.683       0.573       0.494
fog of 60k points               0.117      0.116   0.037       0.994       0.209
  + 200k copies of 1 point      0.117      0.936   0.778       0.994       0.209

The ranking changes twice along the tolerance axis. At 1 cm the noisy reconstruction’s accuracy is 0.53, and only 53% of its points count at all: points more than 1 cm behind the surface and those on the shadowed wall are unobserved. The precise reconstruction wins despite missing almost half of the scanned surface. From 2 cm on, the noise fits inside the tolerance and completeness decides. At 50 cm, every part of the gap is within tolerance of the covered wall, so the precise reconstruction becomes complete as well. The noisy one stays behind: its accuracy levels off at 0.981 because box-face points that the noise pushes outside the box’s outline lie in free space in front of the wall, 60 cm from the nearest scan point, and count as wrong at every tolerance. The F1 values at 1 and 2 cm change by at most 0.002 over seeds 0–4, so the first crossing comes from the geometry, not the draw. ETH3D accordingly publishes every score at all six tolerances, and its ranking holds only for the tolerance selected.

The second table separates the parts of the protocol. A naive nearest-neighbor accuracy (NN acc) counts the precise reconstruction’s points on the shadowed wall as wrong, 0.80 instead of 0.996, and drops further when points are added behind the wall. The beam model leaves both unchanged because neither region was seen by the scanner. The same model punishes points in front of the wall, where the laser passed. Voxel averaging weighs those points by the volume they spread over: 6,000 scattered points, 15% of the cloud, lower the per-point accuracy by 0.13 but the voxel accuracy by more than half, since each sparse voxel of outliers counts as much as a voxel of surface. The noisy reconstruction shows the same effect on a smaller scale (0.79 against 0.88). Finally, the fog is nearly complete but has little accuracy, and 200,000 copies of one accurate point raise its per-point accuracy from 0.12 to 0.94 but leave the voxel score and F1 unchanged: the copies fill one voxel per grid. This is the adversary that the paper’s voxel averaging is designed against.

Two simplifications separate this from a real evaluation: there is one scan, and the two-image visibility filter has no counterpart, since every scan point is kept.

Two-view: bad pixels, errors and coverage

The two-view benchmark uses the measures of the Middlebury stereo evaluation (paper, Section 4, citing Scharstein et al., 2014; results page), computed by the official program at commit 42af334:

• bad 0.5, 1, 2 and 4: percentage of pixels whose absolute disparity error exceeds the threshold (strictly greater);
• avgerr and rms: mean absolute error and root-mean-square error, in pixels;
• A50, A90, A95 and A99: the error at the 50th, 90th, 95th and 99th percentile, taken as the element at index $\lfloor nq \rfloor$ of the $n$ sorted absolute errors;
• coverage and runtime.

Each is computed over two masks: non-occluded, the pixels with ground truth that both images see (mask value 255), and all, every pixel with finite ground truth. The detail that matters most is how missing estimates are handled. A pixel that the method leaves without a value (infinite, NaN, or larger in magnitude than the image width) is not counted as an error. It is dropped from every measure and lowers only the coverage, the fraction of ground-truth pixels with an estimate. If no pixel is estimated, the program returns 100% bad pixels and infinite errors. To keep sparse methods from being compared directly with dense ones, the results table has a coverage selector (dense, at least 90%, 80% or 50%, or unrestricted).

The script mirrors this on a synthetic map, a sloped background at 10 to 20 pixels and a square at 40 pixels, with 30% of pixels lacking ground truth as with a sparse laser scan. The “dense” method fattens the square into the 25-pixel strip the right camera cannot see. The “sparse” method removes that strip, as a left-right consistency check would.

import numpy as np

def two_view_metrics(disp, gt, mask, nocc):
    """ETH3D two-view metrics: pixels without ground truth (inf) are skipped,
    and so are pixels the method left empty (inf, NaN or |d| > width)."""
    has_gt = np.isfinite(gt) & ((mask == 255) if nocc else True)
    est = has_gt & np.isfinite(disp) & (np.abs(disp) <= gt.shape[1])
    err = np.sort(np.abs(disp[est] - gt[est]))
    n = err.size
    out = {"coverage": n / has_gt.sum()}
    for t in (0.5, 1.0, 2.0, 4.0):
        out[f"bad{t}"] = 100 * (err > t).mean()
    out["avgerr"], out["rms"] = err.mean(), np.sqrt((err**2).mean())
    for q in (0.5, 0.9):
        out[f"A{int(100 * q)}"] = err[int(np.floor(n * q))]
    return out

# Synthetic 120 x 160 left-image disparity: a sloped background (10-20 px)
# and a square at 40 px. The right camera does not see a 25 px strip left of
# the square (mask 128); the laser gives no ground truth for 30% of pixels.
H, W = 120, 160
rng = np.random.default_rng(0)
gt = np.tile(np.linspace(10, 20, H)[:, None], (1, W))
gt[40:80, 70:110] = 40.0
mask = np.full((H, W), 255, np.uint8)
mask[40:80, 70 - 25:70] = 128
no_gt = rng.random((H, W)) < 0.3
gt[no_gt], mask[no_gt] = np.inf, 0

dense = gt.copy()
dense[no_gt] = 15.0                                   # methods predict everywhere
dense += rng.normal(0, 0.3, (H, W))                   # small error on visible pixels
dense[40:80, 70 - 25:70] = 40 + rng.normal(0, 0.3, (40, 25))  # foreground fattening
sparse = dense.copy()
sparse[40:80, 70 - 25:70] = np.inf                    # left-right check removes the strip

for name, d in [("dense", dense), ("sparse", sparse)]:
    for nocc in (True, False):
        m = two_view_metrics(d, gt, mask, nocc)
        print(f"{name:6s} {'non-occluded' if nocc else 'all':12s} "
              + " ".join(f"{k} {v:.3g}" for k, v in m.items()))

Output:

dense  non-occluded coverage 1 bad0.5 9.68 bad1.0 0.132 bad2.0 0 bad4.0 0 avgerr 0.241 rms 0.303 A50 0.201 A90 0.496
dense  all          coverage 1 bad0.5 14.3 bad1.0 5.25 bad2.0 5.12 bad4.0 5.12 avgerr 1.51 rms 5.67 A50 0.214 A90 0.593
sparse non-occluded coverage 1 bad0.5 9.68 bad1.0 0.132 bad2.0 0 bad4.0 0 avgerr 0.241 rms 0.303 A50 0.201 A90 0.496
sparse all          coverage 0.949 bad0.5 9.68 bad1.0 0.132 bad2.0 0 bad4.0 0 avgerr 0.241 rms 0.303 A50 0.201 A90 0.496

The official ComputeMetrics function, compiled from the pinned commit with a small harness that reads the same arrays, printed the same values to the shown precision. On non-occluded pixels the two methods are identical, because the strip is masked out. On all pixels the dense method’s fattened strip, 5% of the pixels with ground truth, appears in bad 1 to bad 4 and multiplies the RMS error by almost 20. The sparse method instead reports its non-occluded numbers at 94.9% coverage. Under the “all” mask, leaving hard pixels empty therefore looks better than estimating them badly. This is why ETH3D publishes coverage with every result and lets the table be restricted to dense methods.

SLAM

The SLAM benchmark evaluates absolute and relative trajectory errors (SLAM documentation). BAD SLAM reports the RMSE of the absolute trajectory error after an SE(3) alignment of the estimated trajectory, matching poses by timestamp and aligning them with Umeyama’s method (Section 6). An evaluation program for training results is on GitHub (ETH3D/slam-evaluation). The metric itself is described in absolute trajectory error.

License

As of October 2026, the home page states that the data is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License, so commercial use is not covered. The statement appears once, for the site’s data as a whole, and the SLAM overview and documentation pages state no separate terms; it was not checked whether the license is repeated inside the archives. The evaluation programs and the dataset pipeline are released under the BSD license (the repositories’ LICENSE files and the changelog). The documentation notes that a few DSLR images were masked “for data protection reasons” and are released only as JPEGs; it does not say what was masked.

Limitations and Bias

• Few scenes. The high-res benchmark has 25 scenes and the many-view benchmark 10 videos, and four of the five low-res training videos were recorded in high-res training scenes. RAFT-Stereo’s authors call the two-view set too small for training and evaluate on it a model trained only on SceneFlow, fine-tuned on grey, gamma-adjusted SceneFlow images to mimic ETH3D’s (Lipson et al., 2021, Section 4.3, arXiv version).
• Ground truth only where the scanner and two images agree. The evaluation scans keep only points seen by at least two images, and the free-space model ignores reconstructed points in space the scanner did not observe. A reconstruction is therefore rewarded for surfaces the scanner saw, and never penalized, or credited, for others. In our reading of the beam model, the example also shows a second consequence: with a single scan, errors behind a surface by more than $t$ go unpenalized, while errors in front of it are counted; several scans from different positions reduce this asymmetry.
• Registration checked qualitatively. The registration’s accuracy was judged visually, from depth-edge overlays and colored scans (paper, Section 5.1). Its use of laser-scan colors as a global reference creates a dependence on how well those colors fit the geometry, which the authors list as a limitation (Section 3.2).
• Static scenes, auto-exposed rig. The stereo scenes are static, apart from masked moving objects such as branches. The rig videos use automatic exposure on purpose, and the SLAM images are not white-balanced, to keep photometric consistency (SLAM documentation).
• Tolerance and scale. All tolerances are absolute distances, so in our reading a tolerance is stricter, relative to the scene, in a small indoor scene than on a facade. A score is meaningful only with its tolerance.
• Voxel weighting. Scores are per volume, not per point. As the example shows, scattered outliers weigh far more than their share of points.

Common Uses

• Multi-view stereo evaluation. Methods report the high-res F1, for example at 2 and 10 cm on the test set (Xu and Tao, 2019). Learned methods such as PatchmatchNet are trained on DTU and evaluated on ETH3D without fine-tuning (Wang et al., 2021, Section 4).
• Generalization tests for learned stereo. Networks trained on synthetic data are evaluated on ETH3D two-view data without training on it. RAFT-Stereo’s synthetic-to-real comparison uses what it calls the ETH3D validation set (in our reading the training pairs, the only ones with public ground truth) and calls bad 1 the standard threshold for ETH3D, next to 3 pixels for KITTI 2015 and 2 pixels for Middlebury (Lipson et al., 2021, Table 1, arXiv version).
• Zero-shot monocular depth. Ranftl et al. (2022) used ETH3D’s laser-scanned ground truth as one of six test sets never seen in training, scoring all 454 images with ground truth by AbsRel (arXiv v3, Section 6); the protocol is described under absolute relative error and depth estimation. The image counts of the 13 high-res training scenes also sum to 454.
• Robust Vision Challenge. The stereo benchmark was part of the 2018 Robust Vision Challenge (overview page).
• SLAM. RGB-D, stereo and visual-inertial methods are compared on the hidden test sequences; BAD SLAM compares the benchmark with TUM RGB-D and EuRoC MAV, among others (Table 1).

Historical Milestones

• 2017: baselines. In the paper, COLMAP had the best F1 at 2 cm in every scene category. On the DSLR data it reached accuracy, completeness and F1 of 91.7, 56.2 and 69.7%, against 69.5, 41.2 and 51.8% on the rig videos (Table 3); CMPMVS failed on the videos. In the two-view task, MC-CNN, the best of five methods on Middlebury and KITTI, ranked fourth by bad 2 on non-occluded pixels (Table 2b). The authors argued that the variety of their scenes discourages overfitting, which they considered especially important for learned methods (Section 5.2).
• July and October 2017: releases. The initial release was followed by the extension that created today’s test set (changelog).
• February 2018: open pipeline. The dataset processing pipeline was released under the BSD license; pre-rendered training depth maps followed in April (changelog).
• 2019: tolerance changes the order. Xu and Tao reported test-set scores from the evaluation server for their ACMM and ACMH methods and four others (Table 3). ACMM had the highest F1 at 2 cm overall (80.78%, against 73.01% for COLMAP). On the outdoor scenes, ACMH (accuracy 83.96%, completeness 80.03%) beat LTVRE (91.82%, 74.45%) by F1 at 2 cm, 81.77 against 81.41%, while LTVRE was ahead at 10 cm, 94.19 against 93.87%: the same reversal as in the example above.
• 2019: SLAM benchmark. Added on 16 June 2019. BAD SLAM’s authors found that ORB-SLAM2, the best of the methods they compared on TUM RGB-D, was outperformed by their direct method on the new, well-synchronized data. DVO SLAM did better than ORB-SLAM2 on the training set but worse on the test set (Section 6).
• 2021: learned methods arrive. PatchmatchNet, trained on DTU, reported a test F1 of 73.12% against COLMAP’s 73.01% (Wang et al., 2021, Table 3; the table does not state the tolerance, but its COLMAP value equals the 2 cm value in Xu and Tao’s table). Its authors noted that PVSNet was the only competitive purely learned method submitted at the time. RAFT-Stereo, trained only on synthetic data, reported a bad 1 of 2.44 on the two-view test set, first among published methods at the time (Lipson et al., 2021, Section 4.3, arXiv version).

Related Datasets

• Middlebury: laboratory stereo with structured-light ground truth, whose measures ETH3D’s two-view benchmark adopts; its multi-view benchmark has two small objects.
• KITTI: driving stereo with sparse LiDAR ground truth at 0.5 megapixels.
• DTU: robot-arm captures of objects with structured-light reference scans, evaluated by mean and median accuracy and completeness.
• Tanks and Temples: contemporaneous laser-scanned benchmark of complete pipelines on video, with an F-score at a per-scene threshold.
• Strecha et al. (2008): six outdoor scenes with about 30 six-megapixel images each and laser ground truth; by 2017 its online service was no longer available (as Schöps et al. describe it).
• TUM RGB-D and EuRoC MAV: the RGB-D and visual-inertial benchmarks that the SLAM benchmark is compared with in BAD SLAM’s Table 1.

Resources

• ETH3D: overview, documentation, datasets, and the results pages for the high-res multi-view, low-res many-view and low-res two-view benchmarks and the SLAM benchmark.
• Multi-view evaluation program and two-view evaluation program, C++, with Windows binaries linked from the documentation.
• Dataset pipeline, the processing used to build the datasets, and the SLAM evaluation program.