Tanks and Temples

Tanks and Temples is a benchmark for image-based 3D reconstruction of large scenes, created by Arno Knapitsch, Jaesik Park, Qian-Yi Zhou and Vladlen Koltun at Intel Labs and published in ACM Transactions on Graphics for SIGGRAPH 2017. It pairs high-resolution video of sculptures, vehicles, buildings and large interiors with ground-truth point clouds from an industrial laser scanner, and it scores what a whole pipeline produces, from input frames through structure from motion and multi-view stereo to a dense point cloud, by precision, recall and F-score at a distance threshold set separately for each scene. Statements about the paper below come from the authors’ preprint hosted on the benchmark’s website; the ACM version could not be retrieved.

Purpose

The benchmarks before it imaged small tabletop objects in a laboratory, from regularly spaced viewpoints with known camera poses: the Middlebury multi-view benchmark of Seitz et al. (2006) and DTU (Aanæs et al., 2016). Knapitsch et al. argued that this restricts attention to one stage of the pipeline and removes the difficulties that make reconstruction fail in large real environments. The two outdoor benchmarks that did leave the lab were small: the EPFL benchmark of Strecha et al. (2008) has three building facades by Knapitsch et al.’s count (the DTU papers count twelve scenes in one version and six in the other), and the UNC dataset of Merrell et al. (2007) a single building whose reference model is a few surveyed planar facets.

Tanks and Temples was built around three decisions (§3 of the paper):

• Video as input. Video gives redundant coverage of complex surfaces, narrow and wide baselines for camera tracking, more observations of view-dependent appearance, and simpler capture. The frames can always be subsampled into an image collection; the reverse is hardly possible.
• Outdoor and indoor scenes, because high-fidelity reconstruction of interiors from images was, in the authors’ words, generally regarded as an open problem.
• Complete pipelines. No camera poses are given; methods recover them, so that camera localization and dense reconstruction can be designed together.

The capture used a high-end camera rather than a phone, so that the benchmark would not be tied to the optical defects of current mobile cameras, and frames from the videos can be fed to existing SfM and MVS systems unchanged.

Dataset Size

The test set has 14 scenes in two groups (Table 1 of the paper):

Group	Scene	Camera	Area (m²)	Height (m)	$\tau$ (mm)	Video frames	GT points (M)
Intermediate	Family	Sony	5	2.1	3	4,395	5.5
Intermediate	Francis	Sony	81	15.2	5	7,830	19.3
Intermediate	Horse	Sony	10	3.2	3	6,015	6.2
Intermediate	Lighthouse	DJI	108	11.1	10	8,322	8.2
Intermediate	M60	DJI	35	3.2	5	5,616	9.7
Intermediate	Panther	DJI	34	2.9	5	6,570	12.3
Intermediate	Playground	DJI	54	2.8	10	7,463	1.7
Intermediate	Train	Sony	35	5.6	5	12,630	21.7
Advanced	Auditorium	Sony	541	6.2	10	14,640	53.4
Advanced	Ballroom	Sony	254	3.9	10	10,800	43.9
Advanced	Courtroom	Sony	206	7.8	10	7,049	43.4
Advanced	Museum	Sony	110	21.2	10	17,115	36.5
Advanced	Palace	DJI	4,295	47.2	30	21,871	41.9
Advanced	Temple	Sony	713	20.7	15	17,475	33.4

Area and height are those of the cropped ground truth, and the point counts are after cropping and resampling. The intermediate group holds sculptures, large vehicles (the M60 and Panther tanks and a train) and house-scale buildings, filmed by walking around them; the advanced group holds four large interiors filmed from inside and two large outdoor structures, the Palace of Fine Arts and the Temple of Music in San Francisco.

The paper mentions additional training scenes whose ground truth would be made public (§5.3), without listing them. As of October 2026 the download page lists seven: Barn, Caterpillar, Church, Courthouse, Ignatius, Meeting room and Truck, each with video, a sampled image set and public ground truth, plus the individual registered scans. The evaluation script’s thresholds for them are 0.003 (Ignatius), 0.005 (Caterpillar, Truck), 0.01 (Barn, Meetingroom) and 0.025 (Church, Courthouse). The code states no unit; in our reading these are metres (3 to 25 mm), on the scale of the thresholds in Table 1.

Annotation Types

Ground truth. The scenes were scanned with a FARO Focus 3D X330 HDR laser scanner, which has a range of 330 m, works indoors and in direct sunlight, captures up to 976,000 points per second, and had a ranging noise of 0.1 mm at 10.2 m and 0.3 mm at 22.7 m when calibrated shortly before capture (§4.1). It cannot see a cone of 60° directly below itself. Small statues were scanned from 4 positions, the train from 8 to 10, and the Palace and the Temple from 14 and 17; the Palace took two days. The scans were registered with FARO’s software, whose reported overlap error the authors checked against the point spacing, then cleaned of prominent outliers and cropped to the area of interest.

Two kinds of error remain in the ground truth, as the authors describe them. People walked through the scenes during hours-long scans; most of these measurements were removed by the scanner’s filtering, and the authors estimate the remaining outliers at under a thousandth of the points. Water, glass and mirrors produce mirror-image artifacts; planar ones were removed by hand, while those from complex reflective surfaces such as polished ornaments were usually left in, judged quantitatively negligible.

Because overlapping scans and varying distances make density uneven, each cleaned cloud was resampled on a uniform voxel grid of size $\tau/2$, keeping the mean of the points in each voxel. The threshold $\tau$ itself was set per scene, by examining the data and the statistics of nearest-neighbour distances in the ground truth (§4.1): the Palace is nearly 50 m tall and scanned from the ground, while the smallest statues were scanned at close range.

Input video. Three cameras were tested. On four scenes recorded with both a global-shutter Blackmagic camera and gimbal-stabilized rolling-shutter cameras, COLMAP never performed worse on the rolling-shutter footage, so the authors used the two rolling-shutter cameras for their better low-light sensitivity: a DJI Zenmuse X5R with a 12 mm lens (84° diagonal field of view) on a DJI Osmo gimbal, and a Sony a7S II with a 21 mm lens (90°) on a Pilotfly H2 gimbal, both recording 4K video above 8 megapixels (§4.2). White balance was fixed per scene, exposure kept under 10 ms, and the lens usually focused at the hyperfocal distance; scenes with strong contrast between sun and shade were recorded with automatic ISO, aperture or shutter speed, as Table 1 records.

No camera calibration is distributed. The download page says the intrinsics are withheld to encourage methods to estimate them, and suggests a pinhole model with the principal point at the image centre and focal length $f_x = f_y = 0.7W$ for a frame of width $W$ as a starting point (as of October 2026).

Splits

• Training: seven scenes with public ground truth, an alignment of the organizers’ COLMAP reconstruction to it, and a crop file, for offline evaluation with the released script.
• Intermediate and advanced test groups: ground truth withheld, scored by the evaluation server.

Each scene is offered as the raw video and as an image set for quick starts; the download page describes the training image sets as frames sampled at 1 frame per second from 29.97 frames-per-second video. The tutorial’s Ignatius example maps 263 images to 7,845 video frames. For their own evaluation the authors sampled 150 frames for Family and Horse, 500 for Palace and 300 for every other scene (§6).

Tasks

The task is dense reconstruction of a scene from its video or frames, with nothing else given: calibration, camera poses and the dense surface are all estimated. The benchmark scores the final point cloud, so a failure of SfM to connect the scene into one model costs as much as a poor depth estimate. The paper notes that a single component can still be compared by fixing the others; its own results pair one SfM system with four MVS systems and five SfM systems with one MVS system.

Evaluation

The protocol below follows the released evaluation code at commit 2a0d1b2 (the last commit, an archiving notice of September 2024, after which the repository became read-only; the evaluation files last changed in October 2022). That script evaluates the training scenes; the tutorial states that the server for the test groups uses the same pipeline, and its code is not public. The script needs Open3D 0.9, whose builds on PyPI are for Python 2.7 and 3.5–3.7 only, so it was not run here: we read it line by line, together with the Open3D v0.9.0 functions it calls, and reimplemented it in NumPy below.

Submission. For each scene of a group, a method uploads a point cloud (<scene>.ply) and the estimated camera poses of the input frames in the Redwood .log format (<scene>.log). The upload script requires every scene of the chosen group, and the tutorial asks for an entry for every input image, accepting inaccurate poses. A method that sampled its own frames from the video interpolates poses for every video frame with the toolbox’s script.

Coarse alignment. SfM output has an arbitrary position, orientation and scale, so it is first brought into the ground truth’s frame by a similarity transformation, estimated from camera positions. In registration.py, the camera centres of the submitted trajectory are matched by frame index to those of a reference trajectory: the organizers’ own COLMAP reconstruction of the same frames, mapped into the ground-truth frame by a stored transformation (<scene>_trans.txt). Open3D’s registration_ransac_based_on_correspondence fits a similarity to random samples of six correspondences with Umeyama’s (1991) least-squares method, for up to 100,000 iterations, counting as inliers the centres within 0.2 ground-truth units (20 cm in our reading) of their reference. It keeps the sample with the most inliers, breaking ties by inlier RMSE, without refitting on all inliers. A trajectory of more than 1,600 poses is taken to cover every video frame and is reduced to the reference frames with a mapping file.

The paper describes this step differently: the reconstructed poses are registered to ground-truth camera poses estimated from the ground-truth point cloud with the method of Mastin et al. (2009), and reconstructions from pipelines that output no camera poses, such as Pix4D, were aligned by hand (§6). The released script implements only the trajectory route, with a reference trajectory from COLMAP.

Refinement. The coarse alignment initializes iterative closest point with scale: Open3D’s point-to-point registration_icp with TransformationEstimationPointToPoint(True), which solves each step with Umeyama’s method, the “extension of ICP to similarity transformations” of the paper (§6). The script runs three stages, each on the reconstruction and the ground truth cropped to the evaluation volume after applying the current transformation:

Stage	Resampling of both clouds	Maximum correspondence distance
1	voxel grid of size $\tau$	$80\tau$
2	voxel grid of size $\tau/2$	$20\tau$
3	every $k$-th point, only for clouds above 4 million points	$2\tau$

The script asks for 20 iterations per stage, but passes ICPConvergenceCriteria(1e-6, 20) by position. In Open3D 0.9 the arguments are relative_fitness, relative_rmse and max_iteration, so this sets the RMSE tolerance to 20 and leaves the iteration limit at its default of 30. Open3D stops when both the fitness (the share of source points with a correspondence) and the inlier RMSE change by less than their tolerances, so each stage runs until the fitness changes by less than $10^{-6}$, or for 30 iterations.

Cropping. Each scene has a crop file defining a polygonal prism: a polygon drawn by hand in an interactive tool, extruded along one coordinate axis between two bounds. A point is inside if its coordinate along that axis lies between the bounds and it falls inside the polygon by the even–odd rule. The paper says the reconstruction is cropped to this volume; the script crops the ground truth too.

Resampling. After the final transformation and the crop, both clouds are resampled with voxel_down_sample at $\tau/2$, keeping the mean of the points in each voxel. The paper describes the reconstruction as resampled on the same grid as the ground truth. In Open3D 0.9, however, each call anchors its grid at the cloud’s own minimum corner minus half a voxel, so the two grids are generally offset by a fraction of a voxel, and the released ground truth, which the paper describes as already resampled at $\tau/2$ (§4.1), is resampled once more after cropping. Resampling removes the advantage of a dense point cloud in well-observed regions, the weighting problem described in the Chamfer distance article.

Measures. With $\mathcal{R}$ the resampled reconstruction and $\mathcal{G}$ the resampled ground truth, every point gets its Euclidean distance to the nearest point of the other set (Open3D’s compute_point_cloud_distance, a k-d tree search): $e_{r \to \mathcal{G}} = \min_{g \in \mathcal{G}} \lVert r - g \rVert$ and $e_{g \to \mathcal{R}} = \min_{r \in \mathcal{R}} \lVert g - r \rVert$. For a threshold $d$, the paper defines

$$P(d) = \frac{100}{|\mathcal{R}|} \sum_{r \in \mathcal{R}} \big[e_{r \to \mathcal{G}} < d\big], \qquad R(d) = \frac{100}{|\mathcal{G}|} \sum_{g \in \mathcal{G}} \big[e_{g \to \mathcal{R}} < d\big], \qquad F(d) = \frac{2\,P(d)\,R(d)}{P(d) + R(d)},$$

with $[\cdot]$ the Iverson bracket. Precision measures accuracy, the share of the reconstruction near the true surface; recall measures completeness, the share of the true surface the reconstruction covers (precision and recall explains the measures in general). The authors chose the harmonic mean because it goes to zero when either term does: scattered precise landmarks give high precision and almost no recall, and filling space with points does the reverse. The main measure is $F(\tau)$ at the scene’s threshold.

The script computes the same quantities, with the same strict inequality, but as fractions in $[0, 1]$ where the paper and the leaderboard use percentages. If both are zero, its harmonic mean raises a division-by-zero error; an empty cloud is scored zero. It also writes cumulative histograms of both distance sets from 0 to $5\tau$ in steps of $\tau/100$, from which its plotting script draws precision and recall curves, and colour-coded clouds showing each point’s distance up to $3\tau$.

Aggregation. In the paper’s Table 2, each pipeline has its F-score per scene, its mean F-score over each group, and its average rank over the group’s scenes, which the authors call more robust than the mean and the primary measure of relative performance; the mean, they write, indicates absolute performance. An archived copy of the leaderboard from June 2022 shows separate F-score, precision and recall tables for each group, with each method’s average rank, mean, per-scene values and runtime in seconds per image.

Example

The script reimplements this pipeline in NumPy and SciPy, with the Open3D 0.9 behaviours above: a grid anchored per cloud, the even–odd crop, a six-point RANSAC on camera centres, and three stages of scaled ICP with Open3D’s stopping rule. The synthetic scene is a column 1.2 m tall on a 2 × 2 m floor, with $\tau = 20$ mm. The ground truth is 400,000 surface samples resampled at $\tau/2$. The reconstruction has Gaussian noise of standard deviation $\tau/4$, misses the back 60° of the column, carries 2% floaters, each 4–20 cm from a reconstructed point in a random direction and most of them inside the crop, and 5% background points 3–6 m away outside it, and lives in an SfM frame related to the ground truth by an unknown similarity. Its 60 camera centres carry 3 cm of noise, and five of them have failed. The first block registers it and scores the result, the second adds known errors to the true alignment, the third varies the threshold, the crop and the density, and the last repeats the registration for five more seeds. “err mm” is the RMS distance between registered and true positions of the reconstruction’s points inside the crop.

import numpy as np
from scipy.spatial import cKDTree
from scipy.spatial.transform import Rotation

TAU = 0.02                                          # per-scene threshold (metres)
POLY = np.array([[-0.9, -0.9], [0.9, -0.9], [0.9, 0.9], [-0.9, 0.9]])
ZMIN, ZMAX = -0.1, 1.4                              # the crop prism: POLY in x-y, extruded along z


def scene(n, rng):
    """Area-uniform samples: a column (radius 0.3 m, height 1.2 m) and its top, on a 2 x 2 m floor."""
    areas = np.array([2 * np.pi * 0.3 * 1.2, np.pi * 0.09, 4 - np.pi * 0.09])
    k = rng.multinomial(n, areas / areas.sum())
    th, z = rng.uniform(0, 2 * np.pi, k[0]), rng.uniform(0, 1.2, k[0])
    r, a = 0.3 * np.sqrt(rng.uniform(0, 1, k[1])), rng.uniform(0, 2 * np.pi, k[1])
    f = rng.uniform(-1, 1, (3 * k[2], 2))
    f = f[np.hypot(*f.T) > 0.3][:k[2]]
    return np.vstack([np.c_[0.3 * np.cos(th), 0.3 * np.sin(th), z],
                      np.c_[r * np.cos(a), r * np.sin(a), np.full(k[1], 1.2)],
                      np.c_[f, np.zeros(len(f))]])


def voxel_down(P, size):
    """As Open3D 0.9: grid anchored at the cloud's own minimum bound, mean of the points per voxel."""
    idx = np.floor((P - (P.min(0) - size / 2)) / size).astype(np.int64)
    inv = np.unique(idx, axis=0, return_inverse=True)[1].ravel()
    return np.stack([np.bincount(inv, P[:, j]) for j in range(3)], 1) / np.bincount(inv)[:, None]


def in_crop(P):
    """Points inside the polygonal prism (even-odd rule, as Open3D's SelectionPolygonVolume)."""
    x, y, z = P.T
    inside = np.zeros(len(P), bool)
    for (x1, y1), (x2, y2) in zip(POLY, np.roll(POLY, -1, 0)):
        cross = ((y1 < y) & (y2 >= y)) | ((y2 < y) & (y1 >= y))
        with np.errstate(divide="ignore", invalid="ignore"):
            inside ^= cross & (x < x1 + (y - y1) / (y2 - y1) * (x2 - x1))
    return inside & (z >= ZMIN) & (z <= ZMAX)


def crop(P):
    return P[in_crop(P)]


def apply(T, P):
    return P @ T[:3, :3].T + T[:3, 3]


def umeyama(src, dst):
    """Least-squares similarity transform (scale, rotation, translation) taking src onto dst."""
    ms, md = src.mean(0), dst.mean(0)
    U, sig, Vt = np.linalg.svd((dst - md).T @ (src - ms) / len(src))
    e = np.array([1, 1, np.sign(np.linalg.det(U @ Vt))])
    T = np.eye(4)
    T[:3, :3] = (sig * e).sum() / ((src - ms) ** 2).sum(1).mean() * (U * e) @ Vt
    T[:3, 3] = md - T[:3, :3] @ ms
    return T


def ransac_traj(src, dst, rng, iters=2000, thresh=0.2):
    """Camera-centre alignment: best 6-point Umeyama sample by inlier count, then inlier RMSE.
    run.py allows 100,000 iterations; this example uses 2,000."""
    best = (-1, np.inf, None)
    for _ in range(iters):
        k = rng.choice(len(src), 6)
        T = umeyama(src[k], dst[k])
        e = np.linalg.norm(apply(T, src) - dst, axis=1)
        inl = e < thresh
        rmse = np.sqrt((e[inl] ** 2).mean()) if inl.any() else np.inf
        if inl.sum() > best[0] or (inl.sum() == best[0] and rmse < best[1]):
            best = (inl.sum(), rmse, T)
    return best[2]


def icp_sim3(src, dst, max_dist):
    """Point-to-point ICP with scale. run.py passes ICPConvergenceCriteria(1e-6, 20), which in
    Open3D 0.9 sets relative_rmse = 20 and leaves max_iteration at 30: it stops when the
    fitness (share of matched source points) changes by less than 1e-6, or after 30 iterations."""
    tree, T, P = cKDTree(dst), np.eye(4), src.copy()
    d, j = tree.query(P, distance_upper_bound=max_dist)
    for _ in range(30):
        ok = np.isfinite(d)
        U = umeyama(P[ok], dst[j[ok]])
        T, P = U @ T, apply(U, P)
        d, j = tree.query(P, distance_upper_bound=max_dist)
        if abs(np.isfinite(d).mean() - ok.mean()) < 1e-6:
            break
    return T


def register(rec, gt, T):
    """run.py's three refinement stages: (voxel size, max distance) = (tau, 80 tau),
    (tau/2, 20 tau), then no voxel step and 2 tau. Both clouds are cropped first."""
    for voxel, dist in [(TAU, 80 * TAU), (TAU / 2, 20 * TAU), (None, 2 * TAU)]:
        s, t = crop(apply(T, rec)), crop(gt)
        if voxel:
            s, t = voxel_down(s, voxel), voxel_down(t, voxel)
        T = icp_sim3(s, t, dist) @ T
    return T


def prf(rec, gt, T, d=TAU, use_crop=True):
    """EvaluateHisto: transform, crop, resample both at tau/2, distances both ways, P, R, F at d."""
    s, t = apply(T, rec), gt
    if use_crop:
        s, t = crop(s), crop(t)
    s, t = voxel_down(s, TAU / 2), voxel_down(t, TAU / 2)
    p = np.mean(cKDTree(t).query(s)[0] < d)
    r = np.mean(cKDTree(s).query(t)[0] < d)
    return p, r, 2 * p * r / (p + r)


def make_case(seed):
    rng = np.random.default_rng(seed)
    gt = voxel_down(scene(400_000, rng), TAU / 2)            # the scan, released at tau/2
    rec = scene(150_000, rng)                                 # reconstruction, true frame
    ang = np.degrees(np.arctan2(rec[:, 1], rec[:, 0]))
    hidden = (np.abs(ang) > 150) & (np.hypot(rec[:, 0], rec[:, 1]) < 0.31) & (rec[:, 2] < 1.2)
    rec = rec[~hidden] + rng.normal(0, TAU / 4, (np.sum(~hidden), 3))   # unseen back, noise
    v = rng.normal(size=(3_000, 3))                           # 2% floaters 4-20 cm from surface points
    fl = rec[rng.choice(len(rec), 3_000)] + v / np.linalg.norm(v, axis=1, keepdims=True) \
        * rng.uniform(0.04, 0.2, (3_000, 1))
    u = rng.normal(size=(7_500, 3))                           # background 3-6 m away
    bg = u / np.linalg.norm(u, axis=1, keepdims=True) * rng.uniform(3, 6, (7_500, 1))
    rec = np.vstack([rec, fl, bg])
    # SfM frame: an unknown similarity S maps it onto the ground truth.
    S = np.eye(4)
    S[:3, :3] = rng.uniform(0.1, 10) * Rotation.random(random_state=seed).as_matrix()
    S[:3, 3] = rng.uniform(-5, 5, 3)
    Si = np.linalg.inv(S)
    # Camera centres: reference trajectory (GT frame) and submitted one (SfM frame),
    # 3 cm of pose noise and five failed poses.
    t = np.linspace(0, 2 * np.pi, 60, endpoint=False)
    cams = np.c_[2 * np.cos(t), 2 * np.sin(t), np.ones(60)]
    cams_sfm = apply(Si, cams + rng.normal(0, 0.03, cams.shape))
    cams_sfm[::12] += rng.normal(0, 2.0 * np.linalg.norm(Si[:3, 0]), (5, 3))
    return rng, gt, rec, apply(Si, rec), S, cams_sfm, cams


def err_mm(T, rec, rec_sfm):
    """RMS distance (mm) between the registered and the true reconstruction inside the crop."""
    m = in_crop(rec)
    return 1000 * np.sqrt(((apply(T, rec_sfm[m]) - rec[m]) ** 2).sum(1).mean())


def row(name, err, vals):
    print(f"{name:20s}{err:>8s}" + "".join(f"{v:7.3f}" for v in vals))


rng, gt, rec, rec_sfm, S, cams_sfm, cams = make_case(0)
T0 = ransac_traj(cams_sfm, cams, rng)
T1 = register(rec_sfm, gt, T0)
print(f"tau = {TAU * 1000:.0f} mm, ground truth {len(gt)} points, reconstruction {len(rec)} points")
print(f"{'alignment':20s}{'err mm':>8s}{'P':>7s}{'R':>7s}{'F':>7s}")
for name, T in [("cameras (RANSAC)", T0), ("+ ICP, 3 stages", T1), ("true similarity", S)]:
    row(name, f"{err_mm(T, rec, rec_sfm):.1f}", prf(rec_sfm, gt, T))

print("errors added to the true alignment")
c0 = np.array([0.0, 0.0, 0.6])                      # centre of the column


def error(M=np.eye(3), shift=0.0):
    """A similarity about the column's centre, then a shift along x."""
    E = np.eye(4)
    E[:3, :3] = M
    E[:3, 3] = c0 - M @ c0 + [shift, 0, 0]
    return E


rot = lambda deg: Rotation.from_euler("x", deg, degrees=True).as_matrix()
for name, E in [("scale +0.5%", error(1.005 * np.eye(3))), ("scale +2%", error(1.02 * np.eye(3))),
                ("tilt 0.5 deg", error(rot(0.5))), ("tilt 2 deg", error(rot(2))),
                ("shift tau/2", error(shift=TAU / 2)), ("shift tau", error(shift=TAU))]:
    row(name, f"{err_mm(E @ S, rec, rec_sfm):.1f}", prf(rec_sfm, gt, E @ S))

print("threshold, crop and density (true alignment)")
for d in [TAU / 2, TAU, 2 * TAU, 4 * TAU]:
    row(f"d = {d * 1000:.0f} mm", "", prf(rec_sfm, gt, S, d))
row("no crop", "", prf(rec_sfm, gt, S, use_crop=False))
row("10% of the points", "", prf(rec_sfm[rng.random(len(rec_sfm)) < 0.1], gt, S))

print("seed  err mm (cameras, + ICP)  F(registered) - F(true)")
for seed in range(1, 6):
    rng, gt, rec, rec_sfm, S, cams_sfm, cams = make_case(seed)
    T0 = ransac_traj(cams_sfm, cams, rng)
    T1 = register(rec_sfm, gt, T0)
    print(f"{seed:4d}{err_mm(T0, rec, rec_sfm):10.1f}{err_mm(T1, rec, rec_sfm):8.1f}"
          f"{prf(rec_sfm, gt, T1)[2] - prf(rec_sfm, gt, S)[2]:+18.4f}")

Output (NumPy 2.3.5, SciPy 1.16.3):

tau = 20 mm, ground truth 67135 points, reconstruction 151403 points
alignment             err mm      P      R      F
cameras (RANSAC)        14.9  0.952  0.943  0.947
+ ICP, 3 stages          1.9  0.974  0.943  0.958
true similarity          0.0  0.974  0.942  0.958
errors added to the true alignment
scale +0.5%              4.0  0.974  0.942  0.958
scale +2%               16.1  0.926  0.940  0.933
tilt 0.5 deg             5.9  0.973  0.942  0.957
tilt 2 deg              23.7  0.721  0.844  0.778
shift tau/2             10.0  0.970  0.939  0.955
shift tau               20.0  0.916  0.932  0.924
threshold, crop and density (true alignment)
d = 10 mm                     0.879  0.923  0.900
d = 20 mm                     0.974  0.942  0.958
d = 40 mm                     0.982  0.958  0.970
d = 80 mm                     0.992  0.984  0.988
no crop                       0.896  0.951  0.923
10% of the points             0.984  0.875  0.926
seed  err mm (cameras, + ICP)  F(registered) - F(true)
   1       5.1     1.8           -0.0000
   2       8.1     4.1           -0.0002
   3       8.9     4.4           +0.0005
   4       7.7     2.7           -0.0000
   5       6.5     2.7           -0.0001

Registration on camera centres alone leaves the reconstruction 15 mm from its true position, three quarters of $\tau$, and costs 0.011 in F-score, all of it in precision; the ICP stages bring the error to 2 mm, and the score matches the one under the true similarity. Over the six seeds, the camera alignment is off by 5–15 mm and ICP brings that to 1.8–4.4 mm; for seeds 1–5 the registered F-score lies within 0.0005 of the true one. Recall is set by the unseen back of the column, about 6% of the surface, which no alignment can recover.

The errors added to the true alignment show why registration has to be accurate to a fraction of $\tau$. A shift of $\tau/2$ barely changes the score, but a shift of $\tau$ costs 0.034. The size of the displacement is not what matters: a 2% scale error and a 2° tilt move points by 16 and 24 mm (RMS), yet the tilt drops $F$ to 0.78 and the scale only to 0.93. In our reading, scaling about the column’s centre mostly slides points along the floor and the column, which leaves their distance to the surface small, while the tilt lifts the floor’s edges out of its plane.

Halving the threshold costs more than doubling it gains, because noise of $\tau/4$ already puts some points beyond $\tau/2$; the threshold weighs as much as the method, so scores are comparable only at the same $\tau$. Without the crop, the background points count against precision, which falls from 0.97 to 0.90: the crop keeps a method from being judged on scenery the ground truth never covered. Keeping a tenth of the points does not lower precision but loses 7 points of recall, since resampling evens out density but cannot add points where the reconstruction is sparser than the threshold. Yao et al. (2019) report the same effect in practice (below).

License

As of October 2026 the license page contradicts itself. Its Copyright section states that the benchmark data is published under the Creative Commons Attribution 4.0 International (CC BY 4.0) license and restates that license’s permissions to share and adapt the material for any purpose, even commercially. Its License Grant section, below it, grants a non-exclusive, non-transferable right to install the data only for non-commercial scientific research, prohibits any other use, including commercial products, services or artefacts such as 3D models, and forbids reproducing or redistributing the data without written permission. The download page describes the dataset as freely available for non-commercial purposes, subject to the license terms. The terms are therefore unclear; check with the organizers before any commercial use or redistribution. Any work that uses the dataset must reference the benchmark. The toolbox code in the GitHub repository is under the MIT License, which its README separates from the data’s license.

Limitations and Bias

• Few scenes. Each group’s mean F-score rests on six or eight scenes, and a single failed scene moves it a lot: in the paper, OpenMVG-G + OpenMVS reached 56.50 on Family but 0.00 on Playground (Table 2).
• Ground truth the scanner could not see. The scanner has a blind cone below it, and the largest structures were scanned from the ground (§4.1). In our reading, surfaces it could not reach, such as roofs, are missing from the ground truth, and a correct reconstruction of them inside the crop volume counts against precision; the paper does not quantify this.
• One threshold per scene. $\tau$ ranges from 3 mm to 30 mm and was chosen by the authors from point spacing, so F-scores are comparable across methods on one scene, not across scenes.
• The registration is part of the score. As the example shows, an alignment error of the order of $\tau$ lowers the result even when the dense geometry is unchanged. In the released script, the coarse step also depends on the reference COLMAP trajectory and on an inlier distance fixed in absolute units for every scene.
• No training geometry for learned MVS at first. Yao et al. (2018) noted that Tanks and Temples provided neither normals nor meshes from which to render training depth maps, so they could not fine-tune MVSNet on it. As of October 2026 the download page offers the individual scans of the training scenes, for computing normals or meshes.
• No intrinsics. Methods must calibrate themselves, so self-calibration is part of what is scored.

Common Uses

• Generalization test for learned multi-view stereo. Methods trained on DTU or other datasets are evaluated on the intermediate and advanced groups without fine-tuning, as MVSNet was.
• Neural surface reconstruction, evaluated by F-score on the training scenes, whose ground truth is public. Neuralangelo used six of them, leaving out Church, and trained its baselines itself because, as Li et al. (2023) wrote, no public results were available.
• Novel view synthesis, where the videos serve as large unbounded captures and only images are scored. NeRF++ used Truck, Train, M60 and Playground, with the training and test images and SfM poses prepared by Riegler and Koltun (2020) (Zhang et al., 2020); 3D Gaussian splatting used Truck and Train (Kerbl et al., 2023). These uses ignore the laser scans.
• Comparing SfM or MVS components, by fixing the other stage, as the paper did with OpenMVG and OpenMVS.

Historical Milestones

• 2017: the initial evaluation. Knapitsch et al. assembled 15 pipelines from Bundler, COLMAP, MVE, OpenMVG, Theia and VisualSFM for SfM, CMPMVS, COLMAP, MVE, OpenMVS, PMVS and SMVS for MVS, and the commercial Pix4D, run on 150 to 500 frames per scene with each tool’s recommended settings. COLMAP had the best average rank in both groups (2.38 intermediate, 1.33 advanced) and the highest mean F-score on the advanced group (27.24); Pix4D had the highest intermediate mean (43.24, against 42.14 for COLMAP). The best score on any scene was 64.45 (Pix4D on Family); on Auditorium no pipeline reached 17. Every SfM system posed at least 80% of the images, but the authors traced the largest differences in ranking to SfM failing to join a scene into one model (§8, Table 2).
• 2018: learned MVS on the intermediate group. MVSNet, trained only on DTU and run on poses and sparse points from OpenMVG, reached a mean F-score of 43.48 on the intermediate group, which its authors report ranked first among all submissions before April 18, 2018 (Yao et al., 2018, Table 2).
• 2019: the advanced group. Yao et al. (2019) wrote that almost all earlier learned methods failed on the advanced group because of memory, and their recurrent R-MVSNet reconstructed all of its scenes. They noted that the benchmark depends strongly on point density: upsampling their quarter-resolution depth maps by two raised the F-scores from 43.48 to 48.40 on the intermediate group and from 24.91 to 29.55 on the advanced group.
• 2023: neural surfaces. On six training scenes, Neuralangelo reported a mean F1 score of 0.50, against 0.38 for NeuS, 0.21 for COLMAP and 0.15 for NeuralWarp, with NeuS and NeuralWarp trained by its authors (Li et al., 2023, Table 2).

Current results are on the official leaderboard.

Related Datasets

• DTU: laboratory objects with robot-positioned cameras and structured-light scans, scored by accuracy and completeness distances; the usual training set for learned MVS methods evaluated here.
• ETH3D: published concurrently, with laser-scanned ground truth and an F1 score at several thresholds; Knapitsch et al. describe it as complementary, scoring stereo and MVS rather than full pipelines.
• Middlebury multi-view stereo (Seitz et al., 2006): two small objects with given camera poses, the laboratory benchmark that DTU enlarged.
• EPFL (Strecha et al., 2008) and UNC (Merrell et al., 2007): earlier outdoor benchmarks, as described above.
• ScanNet: indoor RGB-D scans whose reference meshes come from the RGB-D data itself.

Resources

• Official website: downloads, tutorial, license, submission (with an account) and the leaderboard for current results.
• TanksAndTemples GitHub repository: downloader, log-file conversion and interpolation scripts, uploader, and the training-set evaluation script in python_toolbox/evaluation; archived and read-only since September 2024.
• Authors’ preprint of the paper, linked from the website.