A Benchmark for the Evaluation of RGB-D SLAM Systems

“A Benchmark for the Evaluation of RGB-D SLAM Systems” by Jürgen Sturm and Daniel Cremers of the Computer Vision Group at the Technical University of Munich (TUM) and Nikolas Engelhard, Felix Endres and Wolfram Burgard of the Autonomous Intelligent Systems Lab at the University of Freiburg was presented at the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) in Vilamoura, Portugal, in October 2012. It described 39 image sequences from a Microsoft Kinect, colour and depth at 640 × 480 and 30 Hz, each paired with the camera’s trajectory from an eight-camera motion-capture system at 100 Hz, and a set of tools for scoring an estimated trajectory against it. The paper’s main work went into the ground truth: calibrating the motion-capture system, the Kinect and the transformation between them, synchronizing their clocks, and estimating how accurate the result is. It then defined two measures already in use, the relative pose error (RPE) for the drift of visual odometry and the absolute trajectory error (ATE) for the global consistency of SLAM, with reference scripts and an online evaluation service. This article covers the 2012 publication. The dataset as it is today, the scripts as they compute the two measures, the licence and later results are described in TUM RGB-D.

This article follows the authors’ copy of the paper (eight pages, typeset 23 July 2012) from the TUM Computer Vision Group’s publications list; section, equation, table and figure numbers refer to it. The IEEE Xplore version (pages 573–580, also eight pages) was not read or compared. Two workshop papers by the same group are used for the history: the two-page RSS 2011 paper that announced the first version, and an IROS 2012 workshop paper, typeset seven weeks after the main paper, that reported how the benchmark had been used.

Problem

The Kinect gave robotics a cheap sensor that delivered colour images and dense depth maps at video rate, and the first RGB-D SLAM systems and depth-fusion systems such as KinectFusion had appeared in 2010 and 2011 (the paper cites three SLAM systems and KinectFusion). There was no common way to compare them. The SLAM datasets of the time, which the paper names as the Freiburg, Intel, Rawseeds and New College datasets, were built for laser scanners and cameras. KITTI’s stereo odometry benchmark, published the same year, had ground-truth poses but no depth maps. A Kinect dataset recorded in a motion-capture studio contained only untextured point clouds, and a dataset for semantic mapping estimated its camera poses from the Kinect’s own colour images, which the authors judged not accurate enough to serve as ground truth. Hence their claim, in §II, that theirs was the first RGB-D dataset with colour, depth and ground-truth poses suitable for evaluating visual SLAM and odometry.

The second problem was the measure. A common way to judge a SLAM system was by its own residuals after optimization, such as reprojection errors or the χ² error of the pose graph. The paper rejects this in a sentence: a low residual does not imply a good map or trajectory, since a system that uses no sensor data at all has zero error. Comparing the estimated map with a floor plan was possible but often judged only by eye, looking for thin structures, kinks and ghosts such as double walls. The authors therefore advocate, like Olson and Kaess, comparing the system’s output with ground truth end to end, and since accurate ground-truth maps are hard to obtain (§VII), they compare trajectories. §II introduces both trajectory measures as frequently employed: RPE for the drift of visual odometry (citing Konolige, Agrawal and Solà) and for the accuracy at loop closures (Kümmerle et al.; Burgard et al.), useful when only sparse relative ground truth exists, and ATE for visual SLAM (Olson and Kaess; Wulf et al.), which needs absolute ground-truth poses.

The paper presents itself as an extension of a version shown at an RSS workshop in June 2011 (Sturm et al., 2011). That version had nine Kinect sequences of about 50 GB from offices, with a single error measure “similar to” that of Kümmerle et al. (2009): the sum of squared differences between true and estimated relative motions over a time interval Δ, reported together with the fraction of poses the system failed to estimate. It also announced held-back recordings: every scene had been recorded twice and the ground truth of the second recording withheld. The IROS paper adds dynamic sequences, longer trajectories and sequences from a robot, as §II says, and replaces the single measure with two.

Contribution

• A recorded dataset. 39 sequences in an office (“fr1”, 6 × 6 m²) and an industrial hall (“fr2”, 10 × 12 m²), handheld and robot-mounted, with calibration sequences, and accelerometer data from the Kinect at 500 Hz.
• Validated ground truth. Motion-capture poses of the colour camera’s optical frame, calibrated and time-aligned to the Kinect, with an error budget measured by independent experiments (§VI).
• Two evaluation measures, RPE and ATE (§VII, Eqs. 1–5), with reference implementations and an online evaluation service.
• A training and testing split following the Middlebury optical flow benchmark and KITTI: training sequences with public ground truth, testing sequences that can be scored only on the website, “to avoid over-fitting” (§II).
• Formats and tools. PNG images, timestamp lists and a ground-truth text file per sequence, also as ROS bag and MRPT rawlog files, with scripts for association, evaluation and conversion (§V).

Method

Recording

All data were recorded on a laptop running Ubuntu 10.10 and ROS Diamondback, at the Kinect’s full resolution and frame rate (§IV). Two different Kinects were used, one for the fr1 and one for the fr2 sequences. The images came through ROS’s openni_camera package, which wraps PrimeSense’s OpenNI driver. The colour and depth images come from two cameras, so the driver’s registration option was used: it projects the depth image to 3D and back into the colour camera with a Z-buffer, using the factory calibration stored in the sensor. The depth images therefore share the colour camera’s pixel grid, which is why §VII defines every pose as that of the RGB optical frame.

The ground truth came from a MotionAnalysis system with eight Raptor-E cameras (1280 × 1024 pixels, up to 300 Hz), tracking reflective markers by triangulation; four markers were fixed to each Kinect. Motion-capture cameras illuminate the scene with infrared LEDs, and the Kinect projects an infrared pattern, so the authors checked that the two did not interfere: the LEDs appear as dim lamps in the Kinect’s infrared image without affecting the depth, and the motion-capture cameras do not see the projector at all. Every recording was also filmed with an external video camera.

Sequences

Table I lists the 19 training sequences in three categories, with duration and average translational and angular velocity; in our arithmetic they total 1,589 s, about 26.5 minutes. The text names a fourth category, calibration, and the remaining sequences of the 39 are not listed in the paper; the TUM RGB-D article reconstructs the breakdown from the website.

• Testing and Debugging (4): xyz and rpy sequences in each environment, with motion mostly along or about one axis at a time.
• Handheld SLAM (11): from fr1/desk (23 s, 0.41 m/s) to fr2/large_with_loop (173 s), including fr2/360_kidnap, in which the sensor was covered by hand for a few seconds to test recovery, and fr2/desk_with_person, in which a person moved objects on the desk. fr2/large_no_loop was recorded without visual overlap to measure long-term drift, and fr2/large_with_loop with a large loop; both left the motion-capture area, so their ground truth covers only the beginning and end.
• Robot SLAM (4): a Kinect on an ActivMedia Pioneer 3 robot, joysticked through the hall, including a search-and-rescue scene of office containers, boxes and other feature-poor objects, which the authors expected to be hard for methods that rely on distinctive keypoints.
• Calibration: per Kinect, a handheld 8 × 6 checkerboard with 20 mm squares seen by a stationary Kinect in colour and depth and in infrared, and a stationary 8 × 7 checkerboard with 108 mm squares seen by a handheld Kinect.

Unless the text says otherwise, every sequence contains several loop closures. The fr1 sequences are faster: the authors state that their camera speeds are higher on average, and in Table I the seven fr1 sequences average 0.28 m/s against 0.18 m/s for the twelve fr2 sequences (our arithmetic, unweighted).

Calibration and the error budget

§VI is the core of the paper, and its structure is a chain of independent checks.

1. Motion capture. The system was calibrated with the vendor’s Cortex software by waving a wand with three markers through the area. To validate it, the authors moved a metal rod of about 2 m with a marker at each end through the area: the measured length varies only if the space is not Euclidean. Its standard deviation was 1.96 mm over the whole capture area, printed as “7 × 7” with the unit lost in typesetting (presumably metres).
2. Kinect intrinsics and depth. Focal lengths, principal points and distortion of the colour and infrared cameras were estimated with OpenCV from the calibration sequences (Table II). Depth measured at checkerboard corners was off by a constant scale, 1.035 for the fr1 sensor and 1.031 for the fr2 sensor (Fig. 4a), and this correction was applied to the published depth. The residual depth noise was about 1 cm up to 2 m and about 5 cm at 4 m (Fig. 4b).
3. Marker model. The average error between observed markers and their rigid model was 0.60 mm in fr1 and 0.86 mm in fr2, which the authors translate into an orientation error of about 0.34° and 0.49°. At 5 m from the camera this displaces points by 30 and 43 mm, and the paper concludes that the motion-capture poses cannot be used to build or evaluate highly accurate 3D models, only trajectories.
4. Extrinsics. The transformation from the Kinect’s markers to its colour optical frame was estimated with a checkerboard carrying five markers, four on its outer corners. Checkerboard corners predicted from motion capture lay on average 3.25 mm (fr1) and 4.03 mm (fr2) from those detected in the image, which the authors attribute mostly to noise in the checkerboard detector.

From these measurements §VI-C concludes that the ground truth is accurate to better than 1 mm and 0.5° from frame to frame, and to better than 10 mm and 0.5° absolutely over the capture area, so the dataset is valid for systems whose RPE and ATE lie well above those values. Two paragraphs earlier, at the end of the marker-model discussion, §VI-C speaks of “an absolute sub-millimeter and sub-degree accuracy”; in our reading the closing statement, 10 mm absolute, is the one consistent with the rod and checkerboard errors of 2 to 4 mm.

Synchronization

The time offset between motion capture and the colour camera was found the same way: the checkerboard residual was computed for a range of assumed delays, from −100 to +100 ms in Fig. 5b, and the minimum lay where the motion-capture poses were about 20 ms earlier than the images. This delay was corrected in the published data (§VI-D). A second offset turned up while the authors evaluated their own systems: depth images arrive on average about 20 ms after the colour images. That one was deliberately left in the timestamps, and the associate.py script, which pairs colour and depth by timestamp with a user-chosen offset and maximum difference, was provided instead. The paper also warns that the Kinect’s colour camera has a rolling shutter and chooses its exposure automatically, so the strength of the distortion can vary considerably within some sequences.

Two error measures

§VII states the design choice first: score the trajectory, not the map, because accurate ground-truth maps are hard to obtain, while admitting that a good trajectory does not guarantee a good map; a small map error, such as an obstacle in a doorway, could stop a robot. Both measures take an estimated sequence $P_1, \dots, P_n$ and a ground-truth sequence $Q_1, \dots, Q_n$ in $SE(3)$, poses of the RGB optical frame relative to arbitrary, possibly different, world frames. For notation the paper assumes the two are synchronized, equally sampled and of equal length, and notes that in practice they differ in rate and length and may have gaps, so an association and interpolation step is needed. The equations are given in camera pose and discussed in absolute trajectory error; the choices the paper makes around them are these.

• RPE (Eqs. 1–3) compares the estimated motion over an interval Δ with the true one, $E_i = (Q_i^{-1}Q_{i+\Delta})^{-1}(P_i^{-1}P_{i+\Delta})$, and takes the RMSE of the translational part over the $n - \Delta$ intervals. The paper mentions the mean and the median as alternatives that weigh outliers less. It treats rotational error as optional, because rotation errors appear as translation errors once the camera moves. Δ = 1 gives the drift per frame; Δ = 30 the drift per second at 30 Hz. Δ = n, comparing only the start and end, is called a common but poor choice, because it penalizes rotation errors early in the trajectory more than late ones (citing Kümmerle et al. and Kelly). For SLAM the paper averages the RMSE over all Δ (Eq. 3), notes that this is quadratic in the trajectory length, and proposes approximating it from a fixed number of sampled pairs; the evaluation script, it says, supports both.
• ATE (Eqs. 4–5) first aligns the estimate to the ground truth with the rigid transformation $S$ of Horn’s (1987) closed-form least-squares method, then takes the RMSE of the translational parts of $F_i = Q_i^{-1} S P_i$. It is presented as a measure of global consistency for SLAM, and its practical advantage is that it can be plotted (Fig. 7a).

The paper then compares the two. RPE averaged over all intervals, it argues, combines rotation and translation in one number and is therefore “always slightly larger” than ATE, or equal when there is no rotational error, as Fig. 7b is said to show; ATE captures rotation only indirectly, through the wrong positions it causes. In the authors’ experiments the two never led to substantially different conclusions, and the ranking of methods often stayed the same.

Results

The paper contains no comparison of SLAM systems on the benchmark: no results table, and no numbers for the testing sequences. Its quantitative content is the dataset statistics (Table I), the calibration (Table II) and the error budget above, plus two illustrations produced with the authors’ own systems, which the conclusion offers as validation of the dataset and measures.

• Fig. 6 plots RPE over time on fr1/desk for two visual odometry methods compared by Steinbrücker et al. (2011), the group’s dense photometric method (labelled RBM in the figure) and generalized ICP (GICP), and reads from it that RBM has lower drift and fewer outliers.
• Fig. 7 uses trajectories from the RGB-D SLAM system of Endres et al. (2012): (a) the ATE on fr1/desk2, drawn as segments between true and estimated positions; (b) a log-log scatter of RPE against ATE for various estimated trajectories, with ATEs from under 2 cm to several metres and, in our reading of the plot, every point above the diagonal, the basis of the “always slightly larger” statement.

Numbers on the benchmark appeared in the groups’ other papers. Endres et al. (2012), their ICRA paper of the same year, reported 9.7 cm and 3.95° on average over the nine fr1 sequences with SIFTGPU features, errors the October 2012 workshop paper identifies as RPE.

Impact

By the workshop paper’s count in October 2012, the website, online since July 2011, had had more than 13,000 unique visitors in twelve months, more than 290,000 file downloads totalling almost 60 TB, and more than 4,000 uses of the online evaluation; six peer-reviewed papers and one master’s thesis had used the benchmark (Sturm et al., 2012, workshop paper, §VI). The website asks users to cite the IROS paper (as of October 2026).

On the dataset, ATE became the number SLAM papers report: Kerl et al. (2013, Table III) and Schöps et al. (2019, Table 2, “as used in” this paper) compare systems by ATE RMSE on a few sequences, and Mur-Artal and Tardós (2017, Table III) by the translation RMSE of the trajectory, while RPE over one second remained the measure for odometry (Kerl et al., 2013, Table II). Later results are collected in the TUM RGB-D article’s milestones.

The two measures are also the ones defined, with this paper as the source, in the camera pose, visual odometry and absolute trajectory error articles, which follow how they were later generalized, for example to similarity alignment for monocular systems. And the benchmark became the reference point for its successors: Schöps et al. (2019) motivate a new RGB-D SLAM benchmark by the shortcomings of this one (below).

Limitations

The paper states several limitations itself:

• Trajectories, not maps. Map quality is not evaluated, and a good trajectory does not imply a good map (§VII).
• Ground truth for trajectories only. The orientation error of the motion-capture poses, up to about 0.5°, rules out using them to build or score precise 3D models (§VI-C), and the benchmark is valid only for errors well above 10 mm and 0.5° absolute.
• Consumer-sensor effects. The rolling shutter, automatic exposure and an uncorrected ~20 ms depth delay are left to the user (§VI-D).
• Partial ground truth on the two large hall tours, which leave the capture area (§III).

Some limits are visible only on a close reading of §VII. The following are our observations.

• The equations do not handle missing poses. They assume both sequences have $n$ poses at the same times. The RSS 2011 version asked authors to report the fraction of missing poses alongside the error; the IROS paper drops this, and leaves timestamp association and interpolation to a step it does not define. As the TUM RGB-D article shows, the scripts associate by nearest timestamp without interpolating, and a pose a system never outputs simply does not count.
• Eq. 3 includes Δ = n. The sum runs to Δ = n, for which $m = n - \Delta = 0$ and Eq. 2 is undefined, and its last defined term, Δ = n − 1, is the single start-to-end comparison that the preceding paragraph calls a poor choice (the paper’s “Δ = n”). Implementations must decide what to do, which is one reason RPE values over all intervals are hard to compare.
• “Always slightly larger” is not a property of the definitions. Eq. 2 uses only the translational part of $E_i$, so the argument that RPE adds rotational error to ATE’s translational error describes how rotation errors propagate, not a bound. It follows from the definitions that with independent position noise and no rotation error at all, each relative translation combines two noise terms, so the RPE is about √2 times the ATE rather than equal to it; on a log-log plot spanning three decades, such as Fig. 7b, a factor of 1.4 does look slight. In the synthetic example of the TUM RGB-D article, Eq. 3 comes out about 60% above the ATE. Fig. 7b is evidence for one system’s trajectories, which is how the text should be read.
• The scripts did not settle the definitions. The paper says the evaluation script computes Eq. 3 exactly or approximately. The evaluate_rpe.py that survives in archived copies of the group’s repository (with a 2013 copyright header; the July 2012 version could not be checked) composes the relative motions in the opposite order from Eq. 1 and, by default, prints the mean over pooled random pairs rather than Eq. 3; the TUM RGB-D article documents the differences and their size.
• The split. The testing sequences repeat the scenes of the training sequences, and there was no ranking; the consequences surfaced later.

What Came After

A reversal that did not happen. The workshop paper of October 2012 (§VI–VII) found that every one of the seven studies it surveyed had used RPE and none ATE. It concluded that ATE was redundant, being strongly correlated with RPE, and announced plans to retract it and to recommend RPE alone, to reduce the benchmark to a few essential sequences, and to add automated submission and ranking. It also noted that the testing sequences had not been used at all. As of October 2026 the website still offers both measures and all sequences, has no ranking, and still says that a joint evaluation paper is planned once enough methods have been evaluated. The community went the other way on the measures: ATE, the one the authors meant to drop, is the one SLAM papers report (above).

New sequences. The same workshop paper listed requests from users: dynamic objects, a longer household sequence, scenes without texture or structure, and an Asus Xtion sensor. The fr3 sequences that answer them appear on the website’s download page by November 2013 and not in its July 2012 version, as the TUM RGB-D article records from archived copies.

The test set in use. Kerl et al. (2013), from TUM, scored some of the withheld validation sequences through the online service (their Table II). Schöps et al. (2019, Table 1) judged the test set not well suited for benchmarking because it shows the same scenes as the training set and has no online leaderboard, and they observed that RGB-D SLAM papers commonly select a small number of training sequences, which may be cherry-picked and on which overfitting cannot be ruled out.

The sensor. The paper’s warnings about the rolling shutter and the depth delay proved consequential. The depth calibration also proved imperfect: Mur-Artal and Tardós (2017) reported a 4% scale bias in the fr2 depth maps. They compensated it in their runs and wrote that it could partly explain their much better results; in our reading, the maps they used are the published ones, with the paper’s correction factor of 1.031 already applied. Schöps et al. (2019) found that their direct method improved markedly on TUM RGB-D when it also optimized the camera intrinsics and a depth deformation, and, on synthetic renderings of TUM RGB-D scenes, that simulated rolling shutter and asynchronous colour and depth both worsened the ATE of every method tested (their Tables 2 and 3). They attributed part of ORB-SLAM2’s lead over direct methods on TUM RGB-D to these effects and recorded a new benchmark with synchronized global-shutter cameras, a hidden test set and an online leaderboard, the SLAM benchmark of ETH3D.

The infrastructure. The data licence moved from CC BY 3.0, as stated in §I, to CC BY 4.0 (website, as of October 2026). The Subversion repository that hosted the scripts no longer serves them; the TUM RGB-D article describes where copies survive.