DROID-SLAM

DROID-SLAM is a deep learning system for visual SLAM, introduced by Zachary Teed and Jia Deng at NeurIPS 2021. It estimates the camera trajectory and a dense map from monocular, stereo or RGB-D video. Its name stands for “Differentiable Recurrent Optimization-Inspired Design”. The network borrows the feature encoders, all-pairs correlation volumes and convolutional GRU of RAFT, but instead of refining one optical flow field between two frames, it repeatedly revises the flow along every edge of a graph of frames. A differentiable dense bundle adjustment (DBA) layer converts each revision into Gauss–Newton updates of the camera poses and of a per-pixel inverse depth map for every keyframe. The same weights, trained once on monocular synthetic video from TartanAir, are used without retraining for monocular, stereo and RGB-D input on four benchmarks.

Problem Addressed

Optimization-based SLAM systems, built around full bundle adjustment of poses and map, are accurate and easy to extend to new sensors; Teed and Deng name ORB-SLAM3 as an example that supports monocular, stereo, RGB-D and inertial input. What they lack, by the authors’ account, is robustness: features get lost, the optimizer diverges, and drift accumulates. Learned systems, whether they replaced hand-crafted features, added learned energy terms to classical back-ends or trained whole SLAM or visual odometry systems end to end, were sometimes more robust but, in the authors’ words, fell far short of classical accuracy on common benchmarks. The end-to-end systems also worked on two to a dozen frames and lacked loop closure and global bundle adjustment.

The closest earlier architectures, in the authors’ comparison, were DeepV2D, their own earlier system, which alternates depth and pose updates instead of bundle adjustment, and BA-Net, whose bundle adjustment layer optimizes a few coefficients of a learned depth basis and a photometric error in feature space. DeepFactors, which the paper calls the most complete earlier deep SLAM system, also optimizes the coefficients of a learned depth basis. DROID-SLAM instead optimizes a free inverse depth at every pixel, and its residual is geometric: the distance between where a pixel lands under the current poses and depth and where the network says it should land. The authors place the method between direct and indirect SLAM. Like direct methods it uses the whole image without detecting or matching features; like indirect methods it minimizes a reprojection error, which they describe as an easier optimization problem than photometric error, with no need for image pyramids.

Architecture

For each frame $t$ the system keeps two state variables, a camera pose $G_t \in \mathrm{SE}(3)$ and an inverse depth map $\mathbf{d}_t \in \mathbb{R}_+^{H \times W}$ at one-eighth of the input resolution. Covisibility between frames is a frame graph $(\mathcal{V}, \mathcal{E})$ : an edge $(i, j)$ means that images $i$ and $j$ overlap. The graph is rebuilt during training and inference as the estimates change, and long-range edges added when the camera returns to a mapped region give loop closure. Every operation below runs on the edges of this graph. The official code implements it as a class named FactorGraph: each edge contributes a block of residual terms that link two poses and one depth map, as a factor does in a factor graph.

Feature Extraction and Correlation

As in RAFT, two convolutional encoders with six residual blocks and three downsampling layers map every new image to features at one-eighth resolution: a feature network, whose outputs build the correlation volumes, and a context network, whose outputs are injected into every update. The supplementary material gives the feature encoder 128 output channels with instance normalization and the context encoder 256 channels without normalization. For each edge $(i, j)$ the network computes the all-pairs correlation volume

C^{ij}_{u_1 v_1 u_2 v_2} = \left\langle g_\theta(I_i)_{u_1 v_1},\, g_\theta(I_j)_{u_2 v_2} \right\rangle

and average-pools its last two dimensions into a four-level pyramid, exactly as RAFT does; the RAFT article describes the volume, the pyramid and the bilinear lookup. In the official code the lookup radius is 3, so each level contributes a $7 \times 7$ window and the four levels 196 channels (droid_net.py). The paper’s formula writes the window size as $(r+1)^2$ .

Update Operator

The update operator is a $3 \times 3$ convolutional GRU with hidden state $h$ , applied per edge. At the start of each iteration the current poses and depths define where each pixel of frame $i$ should appear in frame $j$ :

\mathbf{p}_{ij} = \Pi_c\!\left(G_{ij} \circ \Pi_c^{-1}(\mathbf{p}_i, \mathbf{d}_i)\right), \qquad G_{ij} = G_j \circ G_i^{-1},

where $\mathbf{p}_i$ is the pixel grid of frame $i$ , $\Pi_c^{-1}$ lifts a pixel with inverse depth $d$ to the homogeneous point $\big((u - c_x)/f_x, (v - c_y)/f_y, 1, d\big)$ , and $\Pi_c$ is the pinhole projection. The GRU receives three inputs:

correlation features looked up from $C^{ij}$ around $\mathbf{p}_{ij}$ ;
the optical flow induced by the current camera motion and depth, together with the residual left by the previous bundle adjustment step, so that the network gets feedback from the last iteration (in the code both are clamped to ±64 pixels);
context features, added to the GRU’s inputs.

Because a convolutional GRU sees only a small neighbourhood, the hidden state is also averaged over the whole image and fed back as global context. The authors motivate this with large moving objects, whose wrong correspondences the network must learn to reject, and their ablation on a TartanAir validation split shows a benefit from it (supplementary Fig. 3).

The key change from RAFT is the output. Instead of predicting a flow increment, two convolutional heads on the hidden state predict a revision $\mathbf{r}_{ij} \in \mathbb{R}^{H \times W \times 2}$ of the correspondence field and a confidence map $\mathbf{w}_{ij} \in \mathbb{R}_+^{H \times W \times 2}$ , one weight per pixel and coordinate; the code bounds the weights with a sigmoid. The corrected correspondence is $\mathbf{p}^*_{ij} = \mathbf{p}_{ij} + \mathbf{r}_{ij}$ . Pooling the hidden states of all edges that share the source frame $i$ , a further head predicts a per-pixel damping factor $\lambda$ for the depth of frame $i$ , kept positive with a softplus, and an $8 \times 8$ convex upsampling mask for its inverse depth map.

Dense Bundle Adjustment Layer

The DBA layer maps the revised correspondences of all edges to pose and depth updates by minimizing

\mathbf{E}(G', \mathbf{d}') = \sum_{(i,j) \in \mathcal{E}} \left\lVert \mathbf{p}^*_{ij} - \Pi_c\!\left(G'_{ij} \circ \Pi_c^{-1}(\mathbf{p}_i, \mathbf{d}'_i)\right) \right\rVert^2_{\Sigma_{ij}}, \qquad \Sigma_{ij} = \operatorname{diag} \mathbf{w}_{ij},

a reprojection error in which the confidence map weights each residual (the paper calls $\lVert \cdot \rVert_\Sigma$ a Mahalanobis distance). The layer linearizes the cost with a local parameterization and takes Gauss–Newton steps $(\Delta\boldsymbol{\xi}, \Delta\mathbf{d})$ ; poses are updated by retraction on the manifold and depths by addition:

G^{(k+1)} = \operatorname{Exp}\!\big(\Delta\boldsymbol{\xi}^{(k)}\big) \circ G^{(k)}, \qquad \mathbf{d}^{(k+1)} = \Delta\mathbf{d}^{(k)} + \mathbf{d}^{(k)} .

Each residual involves the depth of a single pixel, so the depth block $C$ of the normal equations is diagonal. Ordering poses before depths,

\begin{bmatrix} B & E \\ E^\top & C \end{bmatrix} \begin{bmatrix} \Delta\boldsymbol{\xi} \\ \Delta\mathbf{d} \end{bmatrix} = \begin{bmatrix} \mathbf{v} \\ \mathbf{w} \end{bmatrix}, \qquad \Delta\boldsymbol{\xi} = \left[B - E C^{-1} E^\top\right]^{-1}\left(\mathbf{v} - E C^{-1} \mathbf{w}\right), \qquad \Delta\mathbf{d} = C^{-1}\left(\mathbf{w} - E^\top \Delta\boldsymbol{\xi}\right),

with the predicted damping $\lambda$ added to the diagonal of $C$ , whose inverse is an elementwise division. This is the Schur complement of point-based bundle adjustment (the reduced camera system), with one scalar unknown per pixel instead of a $3 \times 3$ block per point. With 1/8-resolution maps of $48 \times 64$ , every keyframe adds 3,072 depth unknowns, but the system that is actually factored has only six unknowns per free pose. The supplementary material derives the Jacobians: the projection Jacobian, the Jacobian of the transformed point with respect to a left perturbation of $G_j$ , the same for $G_i$ through the adjoint of $G_{ij}$ , and the depth Jacobian, which is the projection Jacobian applied to the translation of $G_{ij}$ .

Implementation details from the official code: the confidence weights are multiplied by 0.001 inside the layer; the pose block receives a Levenberg–Marquardt-style damping of $0.1 + 10^{-4}$ times its diagonal in the frontend ( $0.01 + 10^{-5}$ times in the backend); residuals are masked out where the transformed point’s homogeneous depth coordinate, the ratio of its depth in frame $j$ to its depth in frame $i$ , is below 0.2 (0.25 in the CUDA kernel); and the network’s damping output is $0.01\,\mathrm{softplus}(\cdot)$ , scaled further by 0.2 (plus $10^{-7}$ ) when the layer runs inside the SLAM system (ba.py, chol.py, projective_ops.py, factor_graph.py). Each application of the update operator is followed by two DBA steps. During training the layer is written in PyTorch and differentiated by automatic differentiation, with the LieTorch library handling gradients in the tangent spaces of the group elements; at inference a CUDA kernel builds the block-sparse system and a sparse Cholesky factorization solves the reduced camera block. The authors found the system unstable and prone to failure when the network was trained on flow alone and bundle adjustment was applied only at test time (supplementary Fig. 3).

Training

The network is trained only on monocular video from the synthetic TartanAir dataset (Wang et al., 2020; the paper itself was not read for this article). According to its official page, TartanAir was rendered with AirSim in photo-realistic Unreal Engine environments with varying light, weather, seasons and moving objects, with ground-truth stereo images, depth, segmentation, optical flow and camera poses, and with camera motion at different levels of aggressiveness (the dataset’s folders name them Easy and Hard). The page lists 30 environments.

Each training example is a 7-frame clip. For every training video the authors precompute a matrix of the mean optical flow between all pairs of frames, set to infinity for pairs with less than 50% overlap, and sample clips as paths through this matrix whose adjacent frames have a mean flow between 8 and 96 pixels, so that examples are neither too easy nor too hard. The network is trained for 250,000 steps with batch size 4, at a resolution of $384 \times 512$ , unrolling 15 update iterations; training took one week on four RTX 3090 GPUs.

Gauge. A monocular reconstruction is determined only up to a similarity transformation. The authors found that leaving this gauge freedom in training harmed the conditioning of the linear system and the stability of the gradients, so they fix the first two poses of every clip to their ground truth: the first removes the six rigid degrees of freedom, the second the scale. In the code, the remaining poses start at the second ground-truth pose, the inverse depths start at 1, and the loader rescales each clip so that its mean ground-truth inverse depth is 1 (train.py, base.py).

Losses. The paper supervises with a pose loss and a flow loss, applied to the output of every iteration with weights that grow by a factor $1/\gamma$ per iteration, $\gamma = 0.9$ . The flow loss is the mean $\ell_2$ distance between the flow that predicted poses and depths induce between adjacent frames and the flow induced by the ground truth. The pose loss in the paper is

\mathcal{L}_{\text{pose}} = \sum_i \left\lVert \operatorname{Log}_{\mathrm{SE}(3)}\!\left(T_i^{-1} \cdot G_i\right) \right\rVert_2 ,

with ground-truth poses $T_i$ and predictions $G_i$ . The released training script differs in its details: its pose term compares relative poses along the frame-graph edges, it adds a third term on the bundle adjustment residuals, and it weights the pose, residual and flow terms by 10, 0.01 and 0.05 (losses.py, train.py). It also chooses at random, with equal probability, between a covisibility frame graph and one that links frames up to two steps apart, and restarts the unrolled optimization from its own output with a small probability.

Inference

At test time the network runs inside a SLAM system with two asynchronous threads (paper, Section 3.4; the supplementary material has no further system details).

Initialization. Frames are collected, keeping a frame only when one update iteration estimates a mean flow above 16 pixels, until there are 12. A frame graph then links keyframes up to three steps apart, and 10 update iterations initialize poses and depths. The code’s defaults differ by dataset (a 12-frame warm-up for TUM RGB-D and TartanAir, 15 for EuRoC, 8 for ETH3D) and run two rounds of eight iterations.
Frontend. Each new frame gets features and edges to its three nearest keyframes by mean flow, a pose from a motion model, and several update iterations over the local window, with the first two poses fixed and all depths free. A redundant keyframe is then removed; if none is redundant, the oldest leaves the window.
Backend. Global bundle adjustment runs over all keyframes. In each iteration the frame graph is rebuilt from an $N \times N$ matrix of mean flows between keyframes: temporally adjacent keyframes are linked first, then edges are added in order of increasing flow, each suppressing the edges within Chebyshev distance 2 of it in index space. Thousands of frames and edges make storing all correlation volumes impossible, so the backend uses RAFT’s memory-efficient correlation, which computes correlations on demand. In the code, the flow distance mixes the full induced flow with the flow from translation alone, averages both directions, and the suppression neighbourhood is a Manhattan (L1) diamond rather than a Chebyshev square, whose radius (2 or 3, set per dataset in the evaluation scripts) shrinks for edges between nearby frames (depth_video.py, factor_graph.py).
Loop closure. No separate place recognition stage appears in the paper: loops close when the rebuilt frame graph links a keyframe to an old one whose induced flow is small, which in our reading relies on the current pose and depth estimates being good enough to predict that overlap (compare loop closure by appearance).
Non-keyframes. After full bundle adjustment of the keyframes, motion-only bundle adjustment recovers the poses of the other frames from flow to neighbouring keyframes, and all frames are evaluated.

In monocular inference the backend fixes the first pose and normalizes the mean inverse depth to 1 before global optimization (depth_video.py), so trajectories have an arbitrary scale and are compared after a similarity alignment.

Stereo and RGB-D. For RGB-D input, depth stays a variable, since sensor depth is noisy and incomplete, and the cost gains a term that penalizes the squared difference between measured and predicted depth (inverse depth, in the ETH3D section of the paper). In the CUDA kernel this term has weight 0.05 and replaces the learned damping on pixels with sensor depth. For stereo, the same system runs on twice as many frames, with the relative pose of the left and right cameras fixed in the DBA layer and cross-camera edges in the graph. These changes need no retraining.

Speed and memory. With two RTX 3090 GPUs, tracking and local bundle adjustment run on one and global bundle adjustment with loop closure on the other. The authors report an average of 20 frames per second (the camera rate) on EuRoC, at $320 \times 512$ and skipping every other frame, and 30 frames per second on TUM RGB-D at $240 \times 320$ , again skipping every other frame; the benchmark results were obtained in these settings. On TartanAir, with much faster camera motion, the system averaged 8 frames per second. The frontend fits in 8 GB of GPU memory, all TUM RGB-D results can be produced on a single GTX 1080 Ti, and EuRoC, TartanAir and ETH3D, with videos of up to 5,000 frames, need a 24 GB GPU.

Performance

All results below are from Teed and Deng (2021) and come from the one model trained on TartanAir. The primary metric is the absolute trajectory error (ATE). The paper does not state the alignment for each benchmark; the evaluation scripts in the official repository compute it as noted for each benchmark, at commit 2dfd39f, a May 2025 revision later than the paper, so the scripts used for the published numbers may have differed.

TartanAir (2021). The paper uses the official test split, whose monocular track has eight Easy (ME000–ME007) and eight Hard (MH000–MH007) trajectories, according to the dataset tools that the repository includes. On the eight Hard sequences (Table 1, ATE, units not printed), DROID-SLAM averaged 0.24, with no failure; the worst sequence, MH005, had 1.31 and the others between 0.01 and 0.30. TartanVO averaged 1.92 and a DeepV2D retrained on TartanAir 5.03, which the authors call 8 and 20 times higher; ORB-SLAM failed on two sequences. The repository’s script aligns monocular runs with a similarity, through the TartanAir tools’ ATE; stereo runs are multiplied by a fixed factor of 2.5 (a comment in the script says the camera baseline is hardcoded to 0.1 m) and then aligned without scale. Against the top three submissions of the TartanAir SLAM competition (Table 2), scored by a normalized relative pose error over all sub-trajectories of 5 to 40 m, DROID-SLAM had 0.129 on the monocular and 0.047 on the stereo track, against 0.340 and 0.119 for the best submission (SuperPoint and SuperGlue with COLMAP): 62% and 60% lower. The paper places the competition at ECCV 2020; the TartanAir tools’ README calls the same test data the CVPR 2020 Visual SLAM challenge.

EuRoC MAV (2021). On the eleven sequences with monocular input (Table 3, ATE in metres), the full system averaged 0.022 m, between 0.012 m (V1_02) and 0.043 m (MH04 and MH05), and its odometry-only variant 0.186 m. Among methods that tracked all eleven, the best was DSM at 0.126 m, so the authors report an 82% reduction; DSO averaged 0.601 m. ORB-SLAM3 failed on V2_03; over the other ten sequences its mean is 0.041 m against DROID-SLAM’s 0.023 m on the same ten, the 43% reduction the paper states. With stereo input (supplementary Table 1), DROID-SLAM averaged 0.024 m and stereo ORB-SLAM3 0.084 m, a 71% reduction; ORB-SLAM3’s average includes 0.521 m on V2_03. The repository’s EuRoC script tracks every other frame, fills in the rest, and aligns with a similarity (evo, with scale), for both monocular and stereo runs. The deep baselines (DeepFactors, both DeepV2D versions, TartanVO) averaged 0.68 m or more; D3VO with DSO is listed without an average: its row has values for five sequences, while the text says it used six of the eleven for evaluation and trained, without supervision, on the others, which show the same scenes.

TUM RGB-D (2021). On all nine freiburg1 sequences with monocular colour input (Table 4, ATE in metres), DROID-SLAM tracked every sequence, with a mean of 0.038 m; its worst was fr1/360 at 0.111 m, equal to DeepTAM, which uses RGB-D input. ORB-SLAM2 failed on six sequences and ORB-SLAM3 on five. Among methods that tracked all nine, DeepFactors averaged 0.233 m and DeepV2D 0.375 m, the 83% and 90% reductions the authors report. The authors call the benchmark notoriously difficult for monocular methods because of rolling shutter, motion blur and heavy rotation. The repository’s script uses every other image, undistorts and crops it, and aligns with a similarity.

ETH3D SLAM (2021). This benchmark, recorded for BAD SLAM with synchronized global-shutter RGB cameras and active-stereo depth (Schöps et al., 2019), has 61 training and 35 test sequences, with test ground truth withheld and results submitted online. Its leaderboard ranks methods by the area under a cumulative curve that counts, for each error threshold up to a selected maximum, the datasets on which a method’s ATE RMSE is lower; the page notes that this weighs robustness heavily, since an extra successful dataset adds more area than small improvements on many. With RGB-D input and no fine-tuning, DROID-SLAM had an AUC of 340.42 on the training and 207.79 on the test split, against 280.05 and 153.47 for BAD SLAM, and ranked first on both at the time (Fig. 4). It tracked 30 of the 32 test sequences that have image data, against 19 for the next best; the remaining sequences are “dark”, with no image data, and received no submission. The paper does not state the AUC settings. As of November 2026 the leaderboard lists DROID-SLAM’s test entry at 207.79 under its default view, SE(3)-aligned ATE RMSE with a maximum error of 8 cm, and the paper’s cumulative plot also stops at 8 cm. The repository’s ETH3D script, for training sequences, aligns without scale.

Ablations (supplementary material, TartanAir validation split): global optimization improves on local bundle adjustment alone, stereo improves on monocular input, the global context pooling helps, and odometry accuracy depends on the number of keyframes in the local window, five being the value used.

Strengths

One model, trained on synthetic monocular video, gave the best average scores in the paper’s comparisons on four benchmarks (lowest mean ATE or normalized pose error, highest ETH3D AUC), three of them real, with monocular, stereo and RGB-D input.
Few catastrophic failures: no failure on TartanAir, EuRoC or TUM RGB-D, and 30 of 32 ETH3D test sequences.
The geometry is explicit. Poses and depths come from a least-squares problem, so the camera model and additional sensors (a stereo baseline, a depth term) enter the optimization rather than the network.
Dense per-pixel depth without a learned depth basis, which the authors argue helps generalization because the representation is not tied to the training data.

Limitations

Compute and memory. The authors call resource requirements the biggest limitation: real-time operation used two RTX 3090 GPUs, and the backend, which stores feature maps of all keyframes, needs 24 GB of GPU memory for long videos. DPVO’s authors later measured DROID-SLAM’s odometry mode alone at an average of 40 frames per second and 8.7 GB of GPU memory on an RTX 3090, and noted that it slows down during fast camera motion (Teed et al., 2023).
Not real-time on fast motion. On TartanAir it averaged 8 frames per second.
Monocular scale. As in every monocular system, scale is unobservable; trajectories are scored after a similarity alignment.
Failures remain. It failed on 2 of 32 ETH3D test sequences, and its largest TartanAir error, 1.31 on MH005, is far above its average.
Determinism. The repository notes that its asynchronous, multi-process mode usually gives better results but is not deterministic (README, commit 2dfd39f).

Historical Importance

DROID-SLAM (2021) extended RAFT’s recurrent, weight-tied refinement from two-frame flow to a whole frame graph, with a bundle adjustment layer inside the recurrence. In its own comparisons it had lower average errors than classical systems such as ORB-SLAM3 and DSO on their standard benchmarks while failing less often, where the earlier learned systems it compared with were far less accurate, and it showed that a network trained only on synthetic monocular video could serve stereo and RGB-D input through the optimization alone. DPVO’s authors describe it as having significantly improved the state of the art in visual SLAM and odometry (Teed et al., 2023).

Variants

Deep Patch Visual Odometry (DPVO; Teed, Lipson and Deng, NeurIPS 2023) is a monocular visual odometry system from the same group that replaces dense flow with a sparse set of image patches tracked through time, alternating a recurrent patch update with a differentiable bundle adjustment layer, and trained end to end on synthetic data. Its authors argue that dense flow is not needed for robustness, and report lower average error than DROID-SLAM’s odometry on common benchmarks with a third of the memory, running about three times faster on average.

Within DROID-SLAM itself, the stereo and RGB-D modes are configurations rather than separate models, and the 2025 revision of the repository has an asynchronous mode that runs the frontend and backend in separate processes, optionally on two GPUs.

Implementations

As of November 2026 the official PyTorch implementation, princeton-vl/DROID-SLAM, is released under the BSD 3-Clause license with a pretrained model, demos, evaluation scripts for TartanAir, EuRoC, TUM RGB-D and ETH3D, and the training script. The latest commit (May 2025) adds multi-GPU inference; the README states that it was tested up to PyTorch 2.7, that the demos need a GPU with at least 11 GB of memory and training 24 GB.

Example: one dense bundle adjustment layer

The script implements the DBA layer of Eq. (4) and (5), as ba.py and chol.py do, in NumPy, with no network and no training: the flow targets $\mathbf{p}^*_{ij}$ and confidence weights that the update operator would predict are supplied directly. The data are depth images of a synthetic room with a box, rendered at DROID-SLAM’s operating resolution for a $384 \times 512$ input: $48 \times 64$ pixels per frame, with TartanAir’s intrinsics scaled accordingly. Seven frames, the length of a training clip, move 0.05 to the right and 0.02 forward with 1° of yaw per frame, and the frame graph links every pair of frames at most two apart, in both directions: 22 edges and 21,504 inverse depths. Each case runs 30 Gauss–Newton steps from a fixed set of targets, as many DBA steps as one training unroll (15 updates, two steps each), although in DROID-SLAM the targets change between updates.

Conventions follow the paper and the code: poses $G_t$ map world to camera coordinates (camera $x$ right, $y$ down, $z$ forward), $G_{ij} = G_j G_i^{-1}$ , updates are left perturbations $\operatorname{Exp}(\boldsymbol{\xi}) G$ with translation first in $\boldsymbol{\xi}$ , as in LieTorch, and depth is inverse depth. The gauge is fixed as in training: poses 0 and 1 stay at their ground truth, which fixes the six rigid degrees of freedom and the scale, so errors are measured directly, with no alignment. The scene is rescaled so that its mean inverse depth is 1, and the remaining poses start at pose 1 with every inverse depth at 1, as in the training script. The measures are the mean rotation error and the mean camera-centre error of poses 2 to 6 (in units of the rescaled scene, times 100), and the median relative inverse-depth error on static background pixels, with the share of those pixels within 5%.

Four cases are compared. A uses the true correspondences with weight 1. B corrupts them in two ways: the box, 10% of the pixels, moves 0.04 per frame as if it were a moving object, and 10% of the pixels of every edge get target offsets of up to ±6 pixels (±48 pixels at full resolution); all weights stay 1. C gives every corrupted pixel a weight of 0.001, a confidence that identifies all of them. D down-weights only the moving box and leaves the random outliers at weight 1. The depth damping $\lambda$ is predicted per pixel by the network, so its value is open; the script uses the value for a zero pre-activation on both of the code’s paths: $0.01\,\mathrm{softplus}(0) \approx 0.0069$ as passed to the layer in training, and 0.2 times that (plus $10^{-7}$ ) as passed inside the SLAM system. The second value was added after pilot runs, before the first run: with the first value alone, 30 steps left case A short of the exact solution. After the first run, a crash in the seed summary was fixed (the loop over case names included case A, which runs only for seed 0); nothing else changed and the table printed by that run is the same as below. The validity mask keeps the code’s test on the transformed point’s homogeneous depth (above 0.2) and also drops pixels whose current inverse depth exceeds 5, a condition the official code does not apply. The script uses NumPy 2.3.5 and takes about one to three minutes on a laptop.

import numpy as np

H, W, NF = 48, 64, 7                    # 1/8 of a 384 x 512 frame; a 7-frame training clip
FX = FY = 32.0; CX, CY = 32.0, 24.0     # TartanAir intrinsics resized to 512 x 384, divided by 8
MIN_DEPTH = 0.2

def hat(w):
    return np.array([[0, -w[2], w[1]], [w[2], 0, -w[0]], [-w[1], w[0], 0]])

def se3_exp(xi):
    """4x4 exponential of xi = (rho, omega), translation first as in LieTorch."""
    rho, w = xi[:3], xi[3:]
    th, K = np.linalg.norm(w), hat(w)
    if th < 1e-10:
        R, V = np.eye(3) + K, np.eye(3) + 0.5 * K
    else:
        R = np.eye(3) + np.sin(th) / th * K + (1 - np.cos(th)) / th**2 * K @ K
        V = np.eye(3) + (1 - np.cos(th)) / th**2 * K + (th - np.sin(th)) / th**3 * K @ K
    T = np.eye(4); T[:3, :3] = R; T[:3, 3] = V @ rho
    return T

def adjoint(T):
    R, t = T[:3, :3], T[:3, 3]
    A = np.zeros((6, 6)); A[:3, :3] = R; A[:3, 3:] = hat(t) @ R; A[3:, 3:] = R
    return A

# ---- scene: a room with a box, rendered as depth images from 7 camera poses ----
ROOM = [(-1.2, 1.2), (-0.8, 0.6), (-1.0, 2.5)]              # x, y (down), z (forward)
BOX = (np.array([-0.3, 0.0, 1.2]), np.array([0.3, 0.6, 1.6]))
V_, U_ = np.mgrid[0:H, 0:W].astype(float)
RAYS = np.stack([(U_ - CX) / FX, (V_ - CY) / FY, np.ones_like(U_)], -1).reshape(-1, 3)

def raycast(c, Rcw):
    """Depth (camera z) and box mask for a camera at centre c with rotation Rcw (camera to world)."""
    d = RAYS @ Rcw.T
    with np.errstate(divide="ignore", invalid="ignore"):
        tw = [(p - c[a]) / d[:, a] for a in range(3) for p in ROOM[a]]
        t1, t2 = (BOX[0] - c) / d, (BOX[1] - c) / d
    t_room = np.min([np.where(t > 0, t, np.inf) for t in tw], 0)
    tn, tf = np.nanmax(np.minimum(t1, t2), 1), np.nanmin(np.maximum(t1, t2), 1)
    t_box = np.where((tf >= tn) & (tn > 0), tn, np.inf)
    return np.minimum(t_room, t_box).reshape(H, W), (t_box < t_room).reshape(H, W)

def make_scene():
    poses, depths, boxes = [], [], []
    for k in range(NF):                                      # 0.05 right, 0.02 forward, 1 deg yaw per frame
        a = np.deg2rad(k)
        Rcw = np.array([[np.cos(a), 0, np.sin(a)], [0, 1, 0], [-np.sin(a), 0, np.cos(a)]])
        c = np.array([0.05 * k, 0.0, 0.02 * k])
        Z, box = raycast(c, Rcw)
        G = np.eye(4); G[:3, :3] = Rcw.T; G[:3, 3] = -Rcw.T @ c   # world-to-camera pose
        poses.append(G); depths.append(Z); boxes.append(box)
    poses, disps = np.array(poses), 1.0 / np.array(depths)
    s = disps.mean()                       # normalize the mean inverse depth to 1, as DROID's loader does
    poses[:, :3, 3] *= s
    return poses, disps / s, np.array(boxes), s

# ---- reprojection p_ij = Pi(G_ij Pi^-1(p_i, d_i)) and its Jacobians (supplementary C) ----
def reproject(Gi, Gj, d, jac=False):
    Gij = Gj @ np.linalg.inv(Gi)
    R, t = Gij[:3, :3], Gij[:3, 3]
    dd = d.reshape(-1)
    Xp = RAYS @ R.T + dd[:, None] * t                        # homogeneous point (X, Y, Z, d) moved to j
    X, Y, Z = Xp.T
    valid = (Z > MIN_DEPTH) & (dd < 1 / MIN_DEPTH)
    Z = np.where(Z < 0.5 * MIN_DEPTH, 1.0, Z)
    coords = np.stack([FX * X / Z + CX, FY * Y / Z + CY], -1)
    if not jac:
        return coords, valid
    n, o = len(dd), np.zeros(len(dd))
    Jp = np.stack([FX / Z, o, -FX * X / Z**2, o, FY / Z, -FY * Y / Z**2], -1).reshape(n, 2, 3)
    Ja = np.stack([dd, o, o, o, Z, -Y, o, dd, o, -Z, o, X, o, o, dd, Y, -X, o], -1).reshape(n, 3, 6)
    Jj = Jp @ Ja                                             # d p_ij / d xi_j  (left perturbation)
    Ji = -Jj @ adjoint(Gij)                                  # d p_ij / d xi_i
    Jz = Jp @ t                                              # d p_ij / d d_i
    return coords, valid, Ji, Jj, Jz

def dba(poses, disps, edges, targets, weights, eta, iters=30, fixedp=2):
    """Gauss-Newton on Eq. 4, Schur complement on the per-pixel inverse depths (Eq. 5)."""
    poses, disps = poses.copy(), disps.copy()
    P, HW = NF - fixedp, H * W
    for _ in range(iters):
        B = np.zeros((6 * P, 6 * P)); v = np.zeros(6 * P)
        E = np.zeros((6 * P, NF * HW)); C = np.zeros(NF * HW); w = np.zeros(NF * HW)
        for (i, j), tgt, wt in zip(edges, targets, weights):
            coords, valid, Ji, Jj, Jz = reproject(poses[i], poses[j], disps[i], jac=True)
            r = tgt - coords
            wv = 0.001 * valid[:, None] * wt                 # the code scales confidences by 0.001
            px = slice(i * HW, (i + 1) * HW)
            C[px] += np.sum(wv * Jz * Jz, 1)
            w[px] += np.sum(wv * r * Jz, 1)
            for a, Ja_ in ((i, Ji), (j, Jj)):
                if a < fixedp:
                    continue                                 # first two poses fixed: gauge and scale
                ia = slice(6 * (a - fixedp), 6 * (a - fixedp) + 6)
                v[ia] += np.einsum("nk,nkc->c", wv * r, Ja_)
                E[ia, px] += np.einsum("nk,nkc->cn", wv * Jz, Ja_)
                for b, Jb_ in ((i, Ji), (j, Jj)):
                    if b >= fixedp:
                        ib = slice(6 * (b - fixedp), 6 * (b - fixedp) + 6)
                        B[ia, ib] += np.einsum("nkc,nk,nkd->cd", Ja_, wv, Jb_)
        B[np.diag_indices_from(B)] += 0.1 + 1e-4 * np.diag(B)     # pose damping as in chol.py
        Cinv = 1.0 / (C + eta + 1e-7)                        # depth block is diagonal: invert per pixel
        S = B - (E * Cinv) @ E.T                             # reduced camera system, 30 x 30
        dxi = np.linalg.solve(S, v - E @ (Cinv * w))
        dd = Cinv * (w - E.T @ dxi)                          # back-substitution, one division per pixel
        for a in range(P):
            poses[a + fixedp] = se3_exp(dxi[6 * a:6 * a + 6]) @ poses[a + fixedp]
        disps = disps + dd.reshape(NF, H, W)
        disps = np.where(disps > 10, 0.0, disps).clip(min=0.0)
    return poses, disps

def make_targets(poses, disps, boxes, s, edges, rng):
    """True correspondences, and a corrupted copy: the box moves, and 10% of pixels are outliers."""
    clean, bad, obj_flags, out_flags = [], [], [], []
    for i, j in edges:
        coords, _ = reproject(poses[i], poses[j], disps[i])
        Gi_inv = np.linalg.inv(poses[i])
        Xw = (RAYS / disps[i].reshape(-1, 1)) @ Gi_inv[:3, :3].T + Gi_inv[:3, 3]
        Xw = Xw + [0.04 * s * (j - i), 0, 0]                # the box translates 0.04 per frame
        Xc = Xw @ poses[j][:3, :3].T + poses[j][:3, 3]
        moved = np.stack([FX * Xc[:, 0] / Xc[:, 2] + CX, FY * Xc[:, 1] / Xc[:, 2] + CY], -1)
        obj = boxes[i].reshape(-1)
        out = rng.random(H * W) < 0.10
        c = np.where(obj[:, None], moved, coords) + out[:, None] * rng.uniform(-6, 6, (H * W, 2))
        clean.append(coords); bad.append(c); obj_flags.append(obj); out_flags.append(out)
    return clean, bad, obj_flags, out_flags

def errors(pe, pg, de, dg, mask, fixedp=2):
    rot, tr = [], []
    for k in range(fixedp, NF):
        dR = pe[k][:3, :3] @ pg[k][:3, :3].T
        rot.append(np.degrees(np.arccos(np.clip((np.trace(dR) - 1) / 2, -1, 1))))
        centre = lambda G: -G[:3, :3].T @ G[:3, 3]
        tr.append(np.linalg.norm(centre(pe[k]) - centre(pg[k])))
    rel = np.abs(de - dg)[mask] / dg[mask]
    return np.mean(rot), 100 * np.mean(tr), 100 * np.median(rel), 100 * np.mean(rel < 0.05)

poses_gt, disps_gt, boxes, s = make_scene()
edges = [(i, j) for i in range(NF) for j in range(NF) if 1 <= abs(i - j) <= 2]
init = poses_gt.copy(); init[2:] = poses_gt[1]               # as in training: poses 2-6 start at pose 1
d0 = np.ones_like(disps_gt)                                  # inverse depth starts at 1 everywhere
static = ~boxes                                              # depth error on static background pixels
ETA0 = 0.01 * np.log(2.0)                                    # 0.01 * softplus(0): training path
ETAS = [("training", ETA0), ("inference", 0.2 * ETA0)]       # inference path: 0.2 * eta (+1e-7)

flow = [np.linalg.norm(reproject(poses_gt[k], poses_gt[k + 1], disps_gt[k])[0]
                       - np.stack([U_, V_], -1).reshape(-1, 2), axis=1).mean() for k in range(NF - 1)]
print(f"{NF} frames of {H}x{W}, {len(edges)} edges, box on {100 * boxes.mean():.1f}% of pixels, "
      f"mean adjacent flow {np.mean(flow):.2f} px at 1/8 resolution")
print("start: rot %.3f deg, trans %.3f, inv. depth median %.2f%%, within 5%%: %.1f%%"
      % errors(init, poses_gt, d0, disps_gt, static))

def run_cases(seed, eta):
    clean, bad, obj, out = make_targets(poses_gt, disps_gt, boxes, s, edges, np.random.default_rng(seed))
    ones = [np.ones((H * W, 2)) for _ in edges]
    oracle = [np.where((o | u)[:, None], 1e-3, 1.0) * np.ones((1, 2)) for o, u in zip(obj, out)]
    partial = [np.where(o[:, None], 1e-3, 1.0) * np.ones((1, 2)) for o in obj]
    cases = [("A clean flow, w = 1", clean, ones), ("B corrupted, w = 1", bad, ones),
             ("C corrupted, w low on all bad", bad, oracle),
             ("D corrupted, w low on box only", bad, partial)]
    res = {}
    for name, tg, wt in cases:
        if seed > 0 and name.startswith("A"):
            continue                                         # clean flow has no random part
        pe, de = dba(init, d0, edges, tg, wt, eta)
        res[name] = errors(pe, poses_gt, de, disps_gt, static)
    return res

print(f"\n{'eta path':10s} {'case':31s} {'rot (deg)':>10s} {'trans x100':>11s} {'depth med %':>12s} {'<5% (%)':>8s}")
for label, eta in ETAS:
    for name, e in run_cases(0, eta).items():
        print(f"{label:10s} {name:31s} {e[0]:10.5f} {e[1]:11.5f} {e[2]:12.4f} {e[3]:8.1f}")

print("\nseeds 0-4 (outlier positions and offsets), min-max:")
for label, eta in ETAS:
    runs = [run_cases(sd, eta) for sd in range(5)]
    for name in runs[1]:
        r = np.array([x[name] for x in runs])
        print(f"{label:10s} {name:31s} rot {r[:, 0].min():.5f}-{r[:, 0].max():.5f} deg, "
              f"depth med {r[:, 2].min():.3f}-{r[:, 2].max():.3f}%")

Output:

7 frames of 48x64, 22 edges, box on 10.0% of pixels, mean adjacent flow 1.92 px at 1/8 resolution
start: rot 3.000 deg, trans 11.751, inv. depth median 24.18%, within 5%: 9.9%

eta path   case                             rot (deg)  trans x100  depth med %  <5% (%)
training   A clean flow, w = 1                0.01093     0.14698       1.1212    100.0
training   B corrupted, w = 1                 0.75998     1.52790       4.5163     55.8
training   C corrupted, w low on all bad      0.00781     0.07059       0.5449     99.5
training   D corrupted, w low on box only     0.47891     1.11635       3.3159     67.8
inference  A clean flow, w = 1                0.00008     0.00099       0.0076    100.0
inference  B corrupted, w = 1                 0.57093     1.02896       3.8836     61.6
inference  C corrupted, w low on all bad      0.00059     0.02374       0.1964     99.7
inference  D corrupted, w low on box only     0.25077     0.44339       2.8492     70.1

seeds 0-4 (outlier positions and offsets), min-max:
training   B corrupted, w = 1              rot 0.49728-0.81123 deg, depth med 3.091-5.495%
training   C corrupted, w low on all bad   rot 0.00727-0.00781 deg, depth med 0.537-0.555%
training   D corrupted, w low on box only  rot 0.44931-0.69113 deg, depth med 2.789-4.048%
inference  B corrupted, w = 1              rot 0.29868-0.72206 deg, depth med 2.013-6.118%
inference  C corrupted, w low on all bad   rot 0.00059-0.00101 deg, depth med 0.192-0.196%
inference  D corrupted, w low on box only  rot 0.20500-0.52643 deg, depth med 1.854-4.628%

Case A is the label check. Clean targets are exactly consistent with the true poses and depths, and with the gauge fixed the true state is the unique zero of the cost, so any remaining error measures how far 30 steps got. With the inference-path damping, the layer recovers rotations to about $10^{-4}$ degrees and inverse depths to 0.008%, from a start 3° and 24% away. With the training-path damping it stops at 0.011° and 1.1%: the damping added to the diagonal depth block is large here relative to the curvature that the weighted residuals give each pixel, which in our reading makes the steps behave like small gradient steps on the depths. In our reading, this is an argument for predicting the damping rather than fixing it, since a network can lower it where the geometry constrains depth well.

Case B shows what a least-squares layer does with bad correspondences at full confidence. The moving box and the random outliers pull every pose and the background depths: the rotation error grows to more than half a degree and around 40% of the background pixels end up more than 5% off. Case C, the confidences DROID-SLAM’s update operator is trained to predict, recovers nearly the clean result: on the inference path 0.0006° against 0.00008°, the difference coming from the 0.001 weight, which leaves the corrupted pixels a small pull. On the training path C even ends below A, which shows only that neither row has converged after 30 steps: the damping enters the normal equations, not the cost, so it changes the steps but not the solution, and the inference rows show that A’s solution is the truth and C’s lies close to it, far below where the training rows stopped. Case D removes the moving object but not the scattered outliers and closes only between about a fifth and a little over half of the gap between B and C, depending on the measure and the damping, so in this example confidence has to cover isolated wrong matches as well as coherent moving regions. The seed loop changes only the positions and sizes of the random outliers: C stays within a narrow band, while the errors of B and D vary by factors of up to three with the draw.

Two simplifications separate this from the real layer. The targets here are fixed, while in DROID-SLAM the update operator revises them after every two steps, using the new residuals; and the confidences here are given, while learning them from correlation features is what the network is for.