TUM RGB-D

TUM RGB-D, officially the RGB-D SLAM Dataset and Benchmark, is a collection of indoor image sequences from handheld and robot-mounted depth cameras, each with the camera’s true trajectory from a motion-capture system, together with scripts that score an estimated trajectory against it. The Computer Vision Group of the Technical University of Munich (TUM) and the Autonomous Intelligent Systems Lab of the University of Freiburg built it; its website went online in July 2011, and Jürgen Sturm, Nikolas Engelhard, Felix Endres, Wolfram Burgard and Daniel Cremers described it at IROS 2012. The colour and depth images come from a Microsoft Kinect or, in the later sequences, an Asus Xtion, at 640 × 480 and 30 Hz; the ground truth comes from eight motion-capture cameras at 100 Hz. The benchmark’s two measures, the absolute trajectory error (ATE) and the relative pose error (RPE), are the errors that SLAM and visual odometry papers report on it, and the benchmark’s scripts define how they are computed.

Purpose

When the Kinect appeared, the first RGB-D SLAM systems had no common benchmark (Sturm et al., 2011). Sturm et al. (2012) surveyed the laser and camera datasets of the time: KITTI’s stereo odometry benchmark provided no depth maps, a Kinect dataset recorded in a motion-capture studio contained only untextured point clouds, and a semantic-mapping dataset estimated its camera poses from the Kinect’s own images, too inaccurately to serve as ground truth. They therefore describe TUM RGB-D as the first RGB-D dataset suited to evaluating visual SLAM and visual odometry. Measuring a SLAM system’s internal residuals, such as its χ² error after optimization, was no substitute: the authors point out that a system that ignores its sensor data trivially has zero error. They chose to score the output trajectory against ground truth, and to ship scripts and an online service so that everyone computes the score the same way.

An earlier version was presented at an RSS workshop in 2011 (Sturm et al., 2011): nine Kinect sequences, about 50 GB, and a single relative error; every scene had been recorded twice, and the ground truth of the second recordings was held back. The IROS paper added dynamic sequences, longer trajectories, robot-mounted recordings and the ATE.

Dataset Size

Counts depend on the version. The paper’s figures are those of 2012; the download page has grown since.

Item	Count or value	Source
Sequences	39, in an office (“fr1”, 6 × 6 m²) and an industrial hall (“fr2”, 10 × 12 m²)	Sturm et al. (2012), §I and §III
Training sequences with statistics	19 (Table I)	Sturm et al. (2012)
Sequences in the six categories, all with public ground truth	47: 9 fr1, 18 fr2, 20 fr3	Download page, as of October 2026
Validation sequences, ground truth withheld	33	Download page and online evaluation form, as of October 2026
Calibration sequences	9, three per sensor	Download page, as of October 2026
Duration of the 47 sequences	15.5 s (fr3/nostructure_texture_far) to 173.2 s (fr2/large_with_loop)	Download page
Ground-truth trajectory length	0.26 m (fr3/sitting_static) to 40.4 m (fr2/pioneer_slam)	Download page
Images	640 × 480; colour 8-bit RGB PNG, depth 16-bit PNG, 30 Hz	Paper §IV, file formats page
Ground truth	100 Hz from eight Raptor-E cameras (1280 × 1024, up to 300 Hz)	Paper §IV, main page
Accelerometer	500 Hz, from the Kinect	Paper §IV

The paper does not break the 39 down. In our count, the archived download page of July 2012 lists 19 training, 14 validation and 6 calibration sequences, which sum to 39, plus 8 object-reconstruction sequences. The 47 sequences of the six categories on today’s page last 2,929 s, about 49 minutes, and their archives total about 47.5 GB, again in our arithmetic from the page’s table.

Categories

The download page, as of October 2026, groups the sequences as follows; the paper had the first three, plus calibration.

• Testing and Debugging (4): fr1/xyz, fr1/rpy, fr2/xyz, fr2/rpy, with motion mostly along or about the camera’s principal axes. The fr2 versions are slow, to avoid motion blur and rolling-shutter effects. The page recommends starting with xyz, then desk.
• Handheld SLAM (11): office and hall sequences such as fr1/desk, fr1/room, fr2/360_kidnap, in which the sensor was covered for a few seconds to test recovery, and fr3/long_office_household, a large loop through a household and office scene. fr2/large_no_loop and fr2/large_with_loop leave the motion-capture area, so their ground truth covers only the beginning and the end, and only the latter closes a loop.
• Robot SLAM (4): a Kinect on a Pioneer 3 robot, joysticked through the hall; the bag files add the robot’s laser scans and odometry.
• Structure vs. Texture (8): fr3 recordings over a flat surface or a zig-zag structure of wooden panels, each with and without texture, at heights of half a metre to two metres; the two “near” recordings of the flat surface close a loop.
• Dynamic Objects (9): fr2/desk_with_person, in which a person moves objects on the desk, and eight fr3 sequences in which two people sit and gesticulate or walk through the scene while the sensor is held still or moved along xyz, rpy or a half sphere.
• 3D Object Reconstruction (11): recordings around single objects such as a plant, teddy bears, dishes or a metallic sphere.

The fr3 sequences, recorded with an Asus Xtion according to their descriptions, are absent from the archived page of July 2012 and present in that of November 2013; the page gives 7 August 2012 as their last modification. They match the requests the authors listed in October 2012: dynamic objects, a household sequence, scenes without texture or structure, and an Xtion sensor (Sturm et al., 2012, workshop paper).

Annotation Types

The only annotation is the camera trajectory. There are no semantic labels, object poses or reference meshes.

Ground truth. The motion-capture system tracked four reflective markers attached to each Kinect (paper §VI-C). Each line of groundtruth.txt holds timestamp tx ty tz qx qy qz qw: Unix time in seconds, the position of the colour camera’s optical centre and its orientation as a unit quaternion, scalar last, relative to the world origin of the motion-capture system (file formats page): a camera-to-world pose. The paper’s estimate of its accuracy, from rod-length and checkerboard experiments in §VI: below 1 mm and 0.5° frame to frame, below 10 mm and 0.5° absolute over the capture area. The authors warn that this suffices for evaluating trajectories but not for building or evaluating precise 3D models, since an orientation error of 0.34° to 0.49° displaces points 5 m away by 30 to 43 mm.

Images and files. Each sequence is a TGZ archive with rgb/ and depth/ folders of PNG images, rgb.txt and depth.txt listing timestamps and file names, a file of accelerometer readings (imu.txt in the paper), and groundtruth.txt; ROS bag files and MRPT rawlog files carry the same data. Depth is registered to the colour camera by the OpenNI driver using the sensor’s factory calibration, so colour and depth pixels correspond one to one. Depth values are scaled by 5,000, so 5,000 means 1 m and 0 means no measurement; depth map covers the conversion to points.

Calibration. The file formats page lists colour and infrared intrinsics for each of the three sensors. It recommends the ROS default parameters (focal length 525, principal point 319.5, 239.5, no distortion) because undistorting pre-registered depth is not straightforward. The fr3 images are already undistorted. Comparing depth with checkerboard distances showed the fr1 and fr2 sensors off by constant factors of 1.035 and 1.031, already applied to the published depth (fr3: 1.000). The paper also measured depth noise of about 1 cm up to 2 m and about 5 cm at 4 m.

Synchronization. The motion-capture poses were found to be about 20 ms earlier than the colour images, and this delay was corrected in the data. The depth images arrive on average about 20 ms after the colour images, and that offset was left in the timestamps, to be handled when associating them (paper §VI-D).

Splits

The paper calls the two parts training and testing; the website calls the second part validation. They are sequences with public ground truth, and sequences whose ground truth is withheld and which can be scored only by the online tool, the paper says, to avoid over-fitting. The validation sequences repeat scenes of the training sequences (fr1/desk_validation, fr3/walking_xyz_validation and so on). There is no prescribed subset, and in October 2012 the authors observed that every published evaluation had used only some of the sequences and that none had used the validation set (Sturm et al., 2012, workshop paper). They planned a reduced set of essential sequences and a ranking system; as of October 2026 the website has neither.

Tasks

• RGB-D visual odometry: drift, measured by RPE over a fixed interval.
• RGB-D SLAM and dense mapping, such as KinectFusion-style systems: global consistency, measured by ATE.
• Monocular and learned SLAM on the colour images alone, aligned with a similarity transform because scale is unknown.
• Robustness: dynamic scenes, kidnapping, texture-poor and structure-poor scenes.
• Object reconstruction, as raw data; the benchmark has no reconstruction metric.

Evaluation

The benchmark scores trajectories only. The website recommends ATE for visual SLAM and RPE for the drift of visual odometry. The metrics themselves, why the alignment matters and how ATE relates to other trajectory errors are covered in absolute trajectory error. This section describes what the benchmark’s code computes.

Which code. The tools page links the scripts in the group’s Subversion repository, which, as of October 2026, answers every request with an error. associate.py and evaluate_rpe.py are read here from the Internet Archive’s copies of that repository; every capture of each file, from 2019–2020 to January 2026, has the same content hash. The archive holds no copy of evaluate_ate.py. It was read in the copy that the TUM Computer Vision Group keeps in its TANDEM repository (commit f8816c7). That copy contains the same associate.py byte for byte, and its evaluate_ate.py differs from a version found unchanged in several other projects only by an added option that saves the plotted trajectories. All three scripts are written for Python 2. They were run here under Python 3 from copies changed only where Python 3 requires (print statements, sorting dictionary keys, converting key views to lists); the originals were not modified.

Association

Images, estimates and ground truth are stamped by different clocks at different rates, so they are paired by timestamp first. associate.py takes two lists of timestamps, adds an --offset (default 0) to the second, and forms every pair $(a, b)$ with $|t_a - (t_b + \delta)| <$ --max_difference (default 0.02 s). It sorts those candidates by their time gap and accepts them greedily, closest first, skipping any whose stamp is already used. The result is one-to-one, nothing is interpolated, and stamps without a partner are dropped silently. The same function pairs colour with depth images and, inside evaluate_ate.py, ground truth with estimates.

ATE: evaluate_ate.py

1. Read both files (the first is the ground truth) and associate them as above, with the same --offset and --max_difference options. Fewer than two pairs is an error.
2. Multiply the estimated positions by --scale (default 1). The scale is a fixed number given by the user; nothing estimates it.
3. Align the estimated positions $\mathbf{p}_i$ to the ground-truth positions $\mathbf{q}_i$ rigidly, without scale. The script’s docstring names Horn’s method; the computation is the SVD solution. With $\Sigma = \sum_i (\mathbf{q}_i - \bar{\mathbf{q}})(\mathbf{p}_i - \bar{\mathbf{p}})^\top = U D V^\top$,

$$R = U\, \mathrm{diag}\big(1, 1, \operatorname{sign}(\det U \det V)\big)\, V^\top, \qquad \mathbf{t} = \bar{\mathbf{q}} - R\, \bar{\mathbf{p}}, \qquad e_i = \lVert R\mathbf{p}_i + \mathbf{t} - \mathbf{q}_i \rVert .$$

4. Print the RMSE of the $e_i$, in metres. --verbose adds the number of pairs and the mean, median, standard deviation (population), minimum and maximum.

The quaternions are never read: alignment and error use positions only, which gives the same value as the paper’s pose form $F_i = Q_i^{-1} S P_i$, whose translation has length $e_i$. The plot, --save and --save_associations options do not change the number.

RPE: evaluate_rpe.py

1. Read both trajectories as 4 × 4 poses, skipping lines with NaNs or an all-zero quaternion. The script does not use associate.py: each estimated stamp, plus --offset, is matched to the nearest ground-truth stamp, and a pair of poses is skipped if either match is more than twice the median ground-truth interval away (0.02 s at 100 Hz). Several estimates may share a ground-truth pose.
2. Choose pairs of estimated poses $(i, j)$:
   • with --fixed_delta, $j$ is the pose closest to $i$ plus --delta in --delta_unit: seconds (default, with delta 1), frames, metres or radians travelled along the estimate, or degrees, an option the help text does not list. A pair is dropped when $j$ is the last pose, which also drops every $i$ near the end whose target falls beyond it.
   • without it, the default, all ordered pairs $(i, j)$ of estimated poses, including $j = i$ and $j < i$. When there are 100 or more poses, --max_pairs (default 10,000) pairs are drawn uniformly with replacement, from Python’s random generator seeded with 0, so the result is repeatable for a given Python version; Python 3.2 changed how randint draws, so Python 2 and 3 sample different pairs from the same seed. In fixed-delta mode the same limit subsamples without replacement, which for sequences at 30 Hz matters only beyond about 5.5 minutes.
3. For each pair, with relative motions $A_P = P_i^{-1} P_j$ of the estimate (translation scaled by --scale) and $A_Q = Q_{i'}^{-1} Q_{j'}$ of the ground truth, compute

$$E^{\text{script}} = A_P\, A_Q^{-1}, \qquad \text{translational error } \lVert \operatorname{trans}(E^{\text{script}}) \rVert, \qquad \text{rotational error } \arccos\!\Big(\tfrac{\operatorname{tr}(R_E) - 1}{2}\Big).$$

4. Print the mean translational error in metres. --verbose adds the RMSE, median, standard deviation, minimum and maximum of the translational and rotational errors, the latter in degrees.

Where the script and the paper differ. Sturm et al. (2012) define $E_i = A_Q^{-1} A_P$ (their Eq. 1), propose the RMSE of its translation (Eq. 2), and for SLAM the average over all intervals of the per-interval RMSE (Eq. 3). The script differs in four ways:

• Composition order. The two orders give the same rotation angle, since the matrices are conjugate, but different translations: $\lVert \mathbf{t}_P - \mathbf{t}_Q \rVert$ in the paper, $\lVert \mathbf{t}_P - R_P R_Q^\top \mathbf{t}_Q \rVert$ in the script. It follows from the triangle inequality that they differ by at most $2 \sin(\theta/2) \lVert \mathbf{t}_Q \rVert$, where $\theta$ is the rotational error: small for short intervals, larger for long ones with rotation error.
• The all-pairs mode. It pools sampled ordered pairs, including zero-length and reversed ones, into one distribution, where Eq. 3 averages one RMSE per interval.
• Mean, not RMSE. The default output is the mean, where the paper proposes the RMSE.
• Its own matching. Ground truth is found by nearest neighbour within twice the median interval, not by associate.py and its --max_difference.

The tools page also states that evaluate_ate.py outputs mean, median and standard deviation; by default it prints only the RMSE.

Online evaluation

The submission form, as of October 2026, takes an estimated trajectory file for one of 41 sequences with ground truth (the six fr2 object-reconstruction sequences are not offered) or one of the 33 validation sequences. Its options mirror the scripts: an offset in seconds added to the estimate’s stamps (default 0.00), a scale (1.00), and three modes. These are ATE, recommended for SLAM; RPE for pairs a given distance apart (default 1 second; also metres, radians or frames), recommended for odometry; and RPE over all pairs, downsampled to 10,000. The form states that it runs the evaluation script and that no data are kept on the server. It is the only way to score the validation sequences. By October 2012 the service had been used more than 4,000 times (Sturm et al., 2012, workshop paper).

Example

The script reimplements the three scripts in NumPy and applies them to synthetic data with known errors. Colour and depth stamps at 30 Hz are offset by 20 ms, as the paper reports for the Kinect. A handheld camera circles a desk at uneven speed for 40 s, with ground truth at 100 Hz. The estimate at 30 Hz has a heading drift of 0.05° per second about the vertical, 3 mm and 0.1° of noise per pose, its own world frame, and timestamps 25 ms late. Both trajectories are written in the benchmark’s file format. It runs in about 3 s.

import random
import numpy as np

rng = np.random.default_rng(1)
T0 = 1305031100.0                      # an arbitrary Unix time, as in the benchmark's stamps

def rot(axis, a):
    c, s = np.cos(a), np.sin(a)
    i, j = [(1, 2), (2, 0), (0, 1)][axis]
    R = np.eye(3); R[i, i] = R[j, j] = c; R[i, j], R[j, i] = -s, s
    return R

def true_pose(tau):
    """Camera-to-world pose of a handheld camera circling a desk (z forward, y down)."""
    w = 2 * np.pi / 20.0
    a = w * tau + 0.5 * np.sin(3 * w * tau)                   # uneven speed around the loop
    p = np.array([1.2 * np.cos(a), 0.8 * np.sin(a), 1.4 + 0.15 * np.sin(7 * w * tau)])
    z = np.array([0.0, 0.0, 0.75]) - p; z /= np.linalg.norm(z)     # look at the desk
    x = np.cross(z, [0, 0, 1.0]); x /= np.linalg.norm(x); y = np.cross(z, x)
    return np.column_stack([x, y, z]), p

def quat_xyzw(R):
    """Unit quaternion (x, y, z, w) of a rotation matrix (eigenvector of the symmetric 4 x 4 matrix K)."""
    K = np.array([[R[0, 0] - R[1, 1] - R[2, 2], R[1, 0] + R[0, 1], R[2, 0] + R[0, 2], R[2, 1] - R[1, 2]],
                  [R[1, 0] + R[0, 1], R[1, 1] - R[0, 0] - R[2, 2], R[2, 1] + R[1, 2], R[0, 2] - R[2, 0]],
                  [R[2, 0] + R[0, 2], R[2, 1] + R[1, 2], R[2, 2] - R[0, 0] - R[1, 1], R[1, 0] - R[0, 1]],
                  [R[2, 1] - R[1, 2], R[0, 2] - R[2, 0], R[1, 0] - R[0, 1], np.trace(R)]]) / 3
    return np.linalg.eigh(K)[1][:, -1]

def write_tum(path, stamps, poses):
    """One pose per line: 'timestamp tx ty tz qx qy qz qw' (camera position and orientation)."""
    with open(path, "w") as f:
        f.write("# timestamp tx ty tz qx qy qz qw\n")
        for t, (R, p) in zip(stamps, poses):
            f.write("%.6f %s\n" % (t, " ".join("%.6f" % v for v in (*p, *quat_xyzw(R)))))

def read_tum(path):
    a = np.loadtxt(path, comments="#")
    x, y, z, w = a[:, 4:8].T / np.linalg.norm(a[:, 4:8], axis=1)
    R = np.stack([[1 - 2*(y*y + z*z), 2*(x*y - z*w), 2*(x*z + y*w)],
                  [2*(x*y + z*w), 1 - 2*(x*x + z*z), 2*(y*z - x*w)],
                  [2*(x*z - y*w), 2*(y*z + x*w), 1 - 2*(x*x + y*y)]]).transpose(2, 0, 1)
    T = np.tile(np.eye(4), (len(a), 1, 1)); T[:, :3, :3] = R; T[:, :3, 3] = a[:, 1:4]
    return a[:, 0], T

def associate(first, second, offset=0.0, max_difference=0.02):
    """associate.py: all pairs closer than max_difference, taken greedily by increasing gap."""
    second = np.asarray(second) + offset
    cand = []
    for a, t in enumerate(first):
        lo, hi = np.searchsorted(second, [t - max_difference, t + max_difference])
        cand += [(abs(t - second[b]), a, b) for b in range(lo, hi) if abs(t - second[b]) < max_difference]
    used_a, used_b, matches = set(), set(), []
    for d, a, b in sorted(cand):
        if a not in used_a and b not in used_b:
            used_a.add(a); used_b.add(b); matches.append((a, b))
    return sorted(matches)

def ate(gt_file, est_file, offset=0.0, max_difference=0.02):
    """evaluate_ate.py: associate, align positions rigidly (SVD, no scale), per-pose distances."""
    (tg, G), (te, E) = read_tum(gt_file), read_tum(est_file)
    m = associate(tg, te, offset, max_difference)
    Q = G[[a for a, b in m], :3, 3]; P = E[[b for a, b in m], :3, 3]
    U, _, Vt = np.linalg.svd((Q - Q.mean(0)).T @ (P - P.mean(0)))
    S = np.diag([1, 1, np.sign(np.linalg.det(U) * np.linalg.det(Vt))])
    R = U @ S @ Vt; t = Q.mean(0) - R @ P.mean(0)
    return np.linalg.norm(P @ R.T + t - Q, axis=1)

def closest(L, t):
    i = np.clip(np.searchsorted(L, t), 1, len(L) - 1)
    return np.where(np.abs(L[i - 1] - t) <= np.abs(L[i] - t), i - 1, i)

def rpe_pairs(gt_file, est_file, pairs, offset=0.0, order="script"):
    """evaluate_rpe.py's error for estimate index pairs (i, j); order='paper' uses Eq. 1 instead."""
    (tg, G), (te, E) = read_tum(gt_file), read_tum(est_file)
    i, j = np.array(pairs).T
    gi, gj = closest(tg, te[i] + offset), closest(tg, te[j] + offset)
    gap = 2 * np.median(np.diff(tg))
    ok = (np.abs(tg[gi] - te[i] - offset) <= gap) & (np.abs(tg[gj] - te[j] - offset) <= gap)
    inv = np.linalg.inv
    A_est, A_gt = inv(E[i]) @ E[j], inv(G[gi]) @ G[gj]          # relative motions
    err = A_est @ inv(A_gt) if order == "script" else inv(A_gt) @ A_est
    trans = np.linalg.norm(err[ok, :3, 3], axis=1)
    rot = np.degrees(np.arccos(np.clip((np.trace(err[ok, :3, :3], axis1=1, axis2=2) - 1) / 2, -1, 1)))
    return trans, rot

def fixed_delta_pairs(est_file, delta_s):
    te = read_tum(est_file)[0]
    j = closest(te, te + delta_s)
    return [(a, b) for a, b in enumerate(j) if b != len(te) - 1]

# 1. Colour and depth stamps: depth frame k belongs to colour frame k but is stamped 20 ms later.
rgb = T0 + np.arange(900) / 30 + rng.normal(0, 0.001, 900)
depth = rgb + 0.020 + rng.normal(0, 0.002, 900)
for off in (0.0, -0.020):
    m = associate(rgb, depth, off)
    print(f"rgb-depth, offset {off:+.3f} s: {len(m)} pairs, {np.mean([a == b for a, b in m]):.0%} correct")

# 2. Ground truth at 100 Hz; estimate at 30 Hz with heading drift, noise, its own world frame,
#    and stamps 25 ms late.
tau_gt = np.arange(0, 40, 0.01)
write_tum("groundtruth.txt", T0 + tau_gt, [true_pose(t) for t in tau_gt])
tau = np.arange(0, 40, 1 / 30) + rng.normal(0, 0.001, 1200)
beta, p0 = np.radians(0.05), true_pose(0)[1]                 # drift: 0.05 deg/s about the vertical
G = rot(2, np.radians(50))                                    # the estimator's world frame
est = []
for t in tau:
    R, p = true_pose(t)
    D = rot(2, beta * t)
    R = D @ R @ rot(0, rng.normal(0, 0.002)) @ rot(1, rng.normal(0, 0.002))
    p = D @ (p - p0) + p0 + rng.normal(0, 0.003, 3)
    est.append((G @ R, G @ (p - p0)))
write_tum("estimate.txt", T0 + tau + 0.025, est)

# 3. ATE as evaluate_ate.py reports it, without and with the clock offset corrected.
for off in (0.0, -0.025):
    e = ate("groundtruth.txt", "estimate.txt", off)
    print(f"ATE, offset {off:+.3f} s: {len(e)} pairs, rmse {np.sqrt(np.mean(e**2)):.4f} m, "
          f"mean {e.mean():.4f}, median {np.median(e):.4f}, std {e.std():.4f}")
offs = np.arange(-0.060, 0.0101, 0.005)
rmse = [np.sqrt(np.mean(ate("groundtruth.txt", "estimate.txt", o) ** 2)) for o in offs]
print("ATE rmse over offsets -60..+10 ms (mm):", " ".join(f"{1000 * r:.1f}" for r in rmse))

# 4. RPE with --fixed_delta --delta 1 --delta_unit s, and the paper's Eq. 1 on the same pairs.
pairs, per_pair = fixed_delta_pairs("estimate.txt", 1.0), {}
for order in ("script", "paper"):
    tr, ro = rpe_pairs("groundtruth.txt", "estimate.txt", pairs, -0.025, order)
    print(f"RPE 1 s ({order}): {len(tr)} pairs, trans rmse {np.sqrt(np.mean(tr**2)):.4f} m, "
          f"mean {tr.mean():.4f} m, rot rmse {np.sqrt(np.mean(ro**2)):.3f} deg")
    per_pair[order] = tr
print(f"largest per-pair difference between the two orders: "
      f"{1000 * np.abs(per_pair['script'] - per_pair['paper']).max():.2f} mm")

# 5. RPE over all pairs: the script's default (10,000 random ordered pairs, Python's random
#    module seeded with 0, mean printed) against Eq. 3, the mean over Delta of RMSE(Delta).
random.seed(0)
n = 1200
rand_pairs = [(random.randint(0, n - 1), random.randint(0, n - 1)) for _ in range(10000)]
tr, _ = rpe_pairs("groundtruth.txt", "estimate.txt", rand_pairs, -0.025)
eq3 = np.mean([np.sqrt(np.mean(rpe_pairs("groundtruth.txt", "estimate.txt",      # Delta = 1..n-1
               [(i, i + d) for i in range(n - d)], -0.025, "paper")[0] ** 2)) for d in range(1, n)])
print(f"RPE all pairs: script mean {tr.mean():.4f} m, script rmse {np.sqrt(np.mean(tr**2)):.4f} m, "
      f"Eq. 3 {eq3:.4f} m")
print(f"  sampled pairs with j < i: {sum(j < i for i, j in rand_pairs)}, "
      f"with j = i: {sum(j == i for i, j in rand_pairs)}")

Output:

rgb-depth, offset +0.000 s: 889 pairs, 1% correct
rgb-depth, offset -0.020 s: 900 pairs, 100% correct
ATE, offset +0.000 s: 1200 pairs, rmse 0.0187 m, mean 0.0170, median 0.0154, std 0.0079
ATE, offset -0.025 s: 1200 pairs, rmse 0.0154 m, mean 0.0136, median 0.0123, std 0.0072
ATE rmse over offsets -60..+10 ms (mm): 21.5 20.2 18.8 17.7 16.7 16.0 15.5 15.4 15.5 15.9 16.6 17.5 18.7 19.9 21.4
RPE 1 s (script): 1169 pairs, trans rmse 0.0078 m, mean 0.0071 m, rot rmse 0.251 deg
RPE 1 s (paper): 1169 pairs, trans rmse 0.0078 m, mean 0.0071 m, rot rmse 0.251 deg
largest per-pair difference between the two orders: 4.82 mm
RPE all pairs: script mean 0.0187 m, script rmse 0.0231 m, Eq. 3 0.0251 m
  sampled pairs with j < i: 4998, with j = i: 11

On the same files, the Python 3 copies of the official scripts printed the same values to the shown precision. associate.py returned the same pairs for both offsets. evaluate_ate.py --verbose gave RMSE 0.018692 and 0.015385 m. evaluate_rpe.py --fixed_delta --delta 1 --delta_unit s --offset -0.025 --verbose gave a translational RMSE of 0.007819 m and a rotational RMSE of 0.250876°. In its default mode it gave a mean of 0.018675 m, the number it prints without --verbose, and an RMSE of 0.023132 m.

Association never fails loudly. With depth stamped 20 ms late, the nearest depth stamp of almost every colour image belongs to the previous frame, 13 ms away, so the script pairs nearly every image with the wrong depth map; with the offset the pairing is exact. In the trajectory, the late stamps do not cost a single match, because a 100 Hz ground truth always has a pose within 5 ms. Each estimate is simply compared with where the camera was 25 ms later. The ATE rises from 15.4 to 18.7 mm, and the sweep over offsets has its minimum at the true −25 ms. It follows from the geometry that a rigid alignment cannot absorb the shift here, because the speed varies around the loop; at constant speed on a circle a time shift is a rotation about the centre, which the alignment would remove. In our reading, a sweep like this is a reasonable way to find an unknown offset, provided the motion is uneven.

The relative errors show the drift separately: 0.05° per second accounts for only part of the 0.25° rotational RPE over 1 s, the rest is per-pose noise. Here the two composition orders agree in RMSE and mean to four decimals, although single pairs differ by up to 4.8 mm, within the bound above. The all-pairs numbers are another matter. The default output, a mean over pooled random pairs, is 19% below the RMSE of the same pairs and 26% below the paper’s Eq. 3, and half the sampled pairs run backwards. A paper that reports “RPE” from this script without naming the mode and the statistic cannot be compared with one that implements Eq. 3.

License

As of October 2026, the main page states that, unless stated otherwise, all data are licensed under a Creative Commons 4.0 Attribution License (CC BY 4.0) and the accompanying source code under the BSD-2-Clause License. The page asks users to cite the IROS 2012 paper. The paper and the 2012 workshop paper gave the licence as CC BY 3.0. The script headers carry a BSD copyright notice of 2013 by Jürgen Sturm, TUM, in the three-clause wording, which differs from the two-clause licence the page names.

The images show offices, a hall and, in the dynamic sequences, people sitting and walking at desks. The main page, the download page and the 2012 paper say nothing about the consent of the people recorded (checked October 2026).

Limitations and Bias

• Consumer sensor effects. The Kinect’s colour camera has a rolling shutter, and its exposure varies with the scene (paper §VI-D). Colour and depth are not synchronized, and the depth calibration is imperfect. Schöps et al. (2019) found that direct RGB-D methods are highly sensitive to these effects. On synthetic renderings of TUM RGB-D scenes, simulated rolling shutter and asynchronous frames both worsened every method’s ATE, and their own method improved markedly on the real sequences when it optimized the intrinsics and a depth deformation (their Tables 2 and 3). ORB-SLAM2’s authors reported a 4% scale bias in the depth maps of the fr2 sequences, which they compensated in their runs and which, they wrote, could partly explain their much better results (Mur-Artal and Tardós, 2017). The page’s correction factor of 1.031 is already applied to these maps.
• Ground truth near its limit. The stated absolute accuracy, below 10 mm and 0.5°, is of the same order as the best ATEs now reported on some sequences, such as ORB-SLAM2’s 4 mm on fr2/xyz. In our reading, differences of a few millimetres between such methods are below what the ground truth can confirm.
• Gaps in the ground truth. fr2/large_no_loop and fr2/large_with_loop have ground truth only near their ends, so ATE there measures only the parts inside the capture area.
• Small, indoor, and short. An office, a hall and the fr3 settings, with trajectories of at most about 40 m and three minutes. Drift that builds up over long runs and outdoor conditions are not tested.
• A test set in name only. Schöps et al. (2019) noted that the validation sequences show the same scenes as the training sequences and that there is no leaderboard (their Table 1), so the benchmark is used almost entirely on its public ground truth.
• Selective reporting. With no prescribed subset, papers choose sequences; Schöps et al. (2019) compare methods on fr1/desk, fr2/xyz and fr3/office as “commonly used” ones. Failures on omitted sequences are invisible, and, as the scripts show, ATE averages only over the poses a system did output.

Common Uses

RGB-D SLAM papers report ATE RMSE on a handful of sequences, such as fr1/desk, fr1/room, fr2/desk, fr2/xyz and “fr3/office” (in our reading fr3/long_office_household) in the tables of Kerl et al. (2013) and Mur-Artal and Tardós (2017), computed with the official script or with re-implementations such as the evo package. Visual odometry is scored by RPE over 1 s, reported in metres per second (Kerl et al., 2013, Tables I and II). Monocular and learned systems run on the colour images of the fr1 sequences and align with a similarity transform (Teed and Deng, 2021). The dynamic-object sequences serve their stated purpose, testing the robustness of SLAM and odometry to people moving in the scene, and the depth and colour images serve as test data for RGB-D reconstruction.

Historical Milestones

• 2011, website and RSS workshop. The site went online in July 2011, and the workshop paper described nine sequences and a single relative error (Sturm et al., 2011); the fr2 sequences followed three months later (Sturm et al., 2012, workshop paper).
• 2012, first results. RGB-D SLAM, from the same groups, reported translational and rotational RMSE from the benchmark’s tool on the nine fr1 sequences, 9.7 cm and 3.95° on average with SIFTGPU features, from 2.1 cm on fr1/xyz to 21.9 cm on fr1/room (Endres et al., 2012, Tables I and II; the text gives 20.1 cm for fr1/room). The October 2012 workshop paper identifies these as RPE.
• 2012, the benchmark paper and a reversal that did not happen. The IROS paper proposed ATE alongside RPE. A workshop paper at the same conference reported that the six peer-reviewed evaluations and one master’s thesis published so far had all used RPE, and none ATE. It announced plans to retract ATE as redundant and recommend RPE alone, and to cut the benchmark to a few sequences with a ranking system. As of October 2026 the website still offers both measures and all sequences, and the SLAM results below are ATEs.
• 2013, DVO SLAM. Kerl et al. reported ATE RMSE on eight fr1 sequences, fr2/desk and fr3/office (their Table III): 0.021 m on fr1/desk, and averages of 0.034 m for their dense method, 0.054 m for RGB-D SLAM, 0.043 m for MRSMap and 0.297 m for the PCL implementation of KinectFusion (KinFu), each over the sequences it was run on. They also scored validation sequences with the online tool (Table II).
• 2017, ORB-SLAM2. A feature-based system with bundle adjustment reported the lowest translation RMSE on most of seven sequences, 0.016 m on fr1/desk and 0.004 m on fr2/xyz, from the median of five runs, against the published results of ElasticFusion, Kintinuous, DVO SLAM and RGB-D SLAM (Mur-Artal and Tardós, 2017, Table III).
• 2019, the benchmark’s limits. BAD SLAM’s authors attributed part of ORB-SLAM2’s lead over direct methods to the sensor’s rolling shutter and unsynchronized streams, and recorded a successor benchmark with synchronized global-shutter cameras, the SLAM part of ETH3D (Schöps et al., 2019).
• 2021, learned monocular SLAM. DROID-SLAM tracked all nine fr1 sequences from colour alone, with a mean ATE of 0.038 m, where ORB-SLAM2 and ORB-SLAM3 failed on most (Teed and Deng, 2021, Table 4). The evaluation script in the authors’ repository aligns with evo in Sim(3), with scale (commit 2dfd39f).

Related Datasets

• KITTI odometry: stereo and LiDAR driving sequences with GPS/IMU ground truth, scored by relative errors per unit length; the outdoor counterpart of the same period.
• EuRoC MAV: stereo and IMU sequences from a drone, with laser-tracker and motion-capture ground truth, for visual-inertial methods.
• ETH3D SLAM: synchronized global-shutter RGB-D sequences with a hidden test set and an online leaderboard.
• ICL-NUIM: synthetic RGB-D sequences, one of the datasets Schöps et al. (2019, Table 1) compare.
• NYU Depth v2 and ScanNet: indoor RGB-D data from consumer depth sensors, for scene understanding rather than trajectory evaluation.
• TUM VI and TUM Mono: later visual-inertial and monocular datasets from TUM, also in Schöps et al.’s comparison.

Resources

• The paper: Sturm et al. (2012), on the recording setup, the ground-truth error budget and the two error measures as first defined.
• RGB-D SLAM Dataset and Benchmark: overview, licence and publications.
• Download page: the sequences by category, with durations, lengths, descriptions and preview videos.
• File formats: image, trajectory and bag formats, intrinsics and depth correction factors.
• Tools: documentation of associate.py, evaluate_ate.py, evaluate_rpe.py and the point-cloud scripts. The repository links are dead as of October 2026; archived copies of associate.py and evaluate_rpe.py exist, and the group’s TANDEM repository holds copies of associate.py (identical) and evaluate_ate.py, and a modified evaluate_rpe.py.
• Online evaluation: the only way to score the validation sequences. The benchmark has no leaderboard.