Direct Methods

Direct methods estimate camera motion, and often scene depth, by comparing the pixel intensities of images. They warp one image into another through the unknown pose and depth and minimize the photometric error, the differences of the intensities that land on top of each other. Indirect (feature-based) methods first turn each image into keypoint positions and matches, and then minimize a geometric error, usually the reprojection error of those points. In visual odometry and SLAM, the direct family includes DTAM, DVO, LSD-SLAM, SVO (which is semi-direct) and Direct Sparse Odometry (DSO).

The idea underneath is brightness constancy: a scene point keeps its intensity from frame to frame, so the right pose and depth make the warped image match the reference image. Direct alignment of a whole camera pose generalizes the Lucas–Kanade tracker from a 2D window displacement to a 6-degree-of-freedom motion through known or estimated depth. Kerl, Sturm and Cremers (2013, “Robust Odometry Estimation for RGB-D Cameras”) describe dense RGB-D alignment in these terms.

Purpose

The distinction is about which measurements enter the estimator. Engel, Koltun and Cremers (2016) frame it with maximum likelihood. Indirect methods first preprocess the images into an intermediate representation, usually matched keypoints but sometimes dense optical flow or line segments. They then treat those values as the noisy measurements and estimate geometry and motion from them. Direct methods skip the preprocessing step and use the sensor values themselves, the light recorded at each pixel, as the measurements. With a passive camera they therefore minimize a photometric error. With sensors that measure geometry directly, such as depth cameras or laser scanners, Engel et al. note that a direct formulation may minimize a geometric error. In our reading, KinectFusion’s ICP alignment of depth maps is that case.

Skipping feature extraction has three consequences that recur in the literature:

• More of the image is used. Keypoints discard edges and smooth intensity variations, such as those on mostly white walls. Direct methods can use any pixel with gradient (Engel, Schöps and Cremers, 2014; Engel et al., 2016).
• A point need not be recognizable on its own. A direct point is the place where a reference pixel’s ray meets the surface, so it has one unknown, its inverse depth. A keypoint is defined by a detector’s response maximum, so its position on the surface is unknown too, and it has three unknowns (Engel et al., 2016).
• The photometric model matters. A direct method must explain the intensities themselves, so exposure changes, the camera’s response curve, vignetting and rolling shutter enter the error directly. Feature detectors and descriptors are built to be invariant to most of them.

When to Use It

Direct methods suit:

• Scenes with few corners but some gradient, such as walls with soft shading, edges and repetitive texture. In Engel et al.’s (2016) ablation, restricting DSO to FAST corners “significantly” reduced accuracy and robustness on their dataset.
• High frame rates and small inter-frame motion, where the linearization of the image is valid and a good initial pose is always available. SVO’s authors reported 55 frames per second on a micro aerial vehicle’s onboard computer and more than 300 on a laptop (Forster, Pizzoli and Scaramuzza, 2014).
• Dense or semi-dense reconstruction alongside tracking, since the depth estimated for tracking is itself a map (Newcombe, Lovegrove and Davison, 2011; Engel et al., 2014).
• Calibrated cameras, including a photometric calibration, or cameras whose exposure times are known (Engel et al., 2016).

They are a poor fit for:

• Large motions or poor initialization, such as relocalization, wide-baseline matching and large loop closures. The photometric error is non-convex and has a small basin of convergence (Engel et al., 2014), whereas robust detectors and descriptors allow matching even at large inter-frame motion (Forster et al., 2014). LSD-SLAM still closes loops by direct alignment, and its authors document the measures needed to widen the basin (see Variants).
• Uncalibrated or distorting cameras, with automatic exposure that is not modeled, a rolling shutter, or poorly known intrinsics. Engel et al.’s (2016) noise study, described under Evidence, shows direct tracking degrading much faster than feature-based tracking under such geometric distortion.

How It Works

The photometric error

Let a reference frame $I_r$ have a pixel $\mathbf{p}$ with inverse depth $d_\mathbf{p}$. Back-projecting gives a 3D point $\pi^{-1}(\mathbf{p}, d_\mathbf{p})$; moving it by the relative pose $T = (R, \mathbf{t})$ and projecting into the target frame $I_t$ gives the warped pixel

$$\mathbf{p}' = \pi\big(R\,\pi^{-1}(\mathbf{p}, d_\mathbf{p}) + \mathbf{t}\big).$$

Direct alignment minimizes, over the pose (and, when they are unknown, the depths),

$$E(T) = \sum_{\mathbf{p} \in \Omega} \rho\Big( I_t(\mathbf{p}') - I_r(\mathbf{p}) \Big),$$

where $\Omega$ is the set of pixels used and $\rho$ a penalty: the square, a robust function, or a variance-normalized square. An indirect method instead minimizes

$$E(T) = \sum_i \big\lVert \mathbf{u}_i - \pi(R\,\mathbf{X}_i + \mathbf{t}) \big\rVert^2$$

over measured keypoint positions $\mathbf{u}_i$ and 3D points $\mathbf{X}_i$. Both are nonlinear least-squares problems over the same pose. They differ in the residual: an intensity difference in one, an image distance in the other. The noise models therefore differ too. The direct error assumes additive noise on intensities, the indirect error noise on keypoint positions (Engel et al., 2016).

Gauss–Newton on the pose

The pose is updated by a small twist $\boldsymbol{\xi} \in \mathfrak{se}(3)$, composed with the current estimate. By the chain rule, the Jacobian of one residual is

$$\frac{\partial r}{\partial \boldsymbol{\xi}} = \nabla I_t(\mathbf{p}') \, \frac{\partial \pi}{\partial \mathbf{X}'} \, \frac{\partial \mathbf{X}'}{\partial \boldsymbol{\xi}},$$

the image gradient at the warped pixel times the projection Jacobian times the motion of the 3D point. Stacking these rows gives the normal equations $J^\top W J\,\boldsymbol{\xi} = -J^\top W \mathbf{r}$, with $W$ the weights of a robust penalty solved by iteratively reweighted least squares (Engel et al., 2014; Kerl et al., 2013, ICRA). Only pixels with image gradient contribute; a pixel in a flat region has a zero row.

The image is linear in the displacement only over a short distance. As a rule of thumb, Engel et al. (2016) put the valid radius at one to two pixels, so every parameter must be initialized well enough that warped pixels land within that radius. Three tools enlarge the basin:

• Coarse-to-fine estimation on an image pyramid. LSD-SLAM starts at only 20 × 15 pixels for loop-closure alignment, which its authors found widens the convergence radius (Engel et al., 2014).
• Motion priors: the previous pose, a constant-velocity model, or an IMU (Kerl et al., 2013, ICRA; Engel et al., 2016).
• Compositional updates from the Lucas–Kanade family. Baker and Matthews (2002) classify image-alignment algorithms as additive or compositional and as forwards or inverse. In the inverse compositional form, the Jacobian is computed on the reference image and stays fixed over the iterations. SVO uses it to align small patches (Forster et al., 2014); DTAM uses a forward-compositional cost (Newcombe et al., 2011).

Weights and robust penalties

Occlusions, moving objects, specular highlights and lighting changes produce residuals far outside a Gaussian. Kerl et al. (2013, ICRA) fitted the residual histogram of a TUM RGB-D sequence and found that a Gaussian fits poorly at any variance, whereas a Student’s t-distribution fits well. They weight each residual by $w(r) = (\nu + 1)/(\nu + (r/\sigma)^2)$ with $\nu = 5$. LSD-SLAM normalizes each residual by its variance, propagated from the uncertainty of the reference pixel’s inverse depth and the image noise, and applies a Huber norm (Engel et al., 2014). DSO uses a Huber norm and down-weights high-gradient pixels (Engel et al., 2016). DTAM drops pixels whose error exceeds a threshold that shrinks as tracking converges (Newcombe et al., 2011).

Photometric models

Brightness constancy, $I_t(\mathbf{p}') = I_r(\mathbf{p})$, fails as soon as the camera changes its exposure. Direct methods relax it in two ways.

An affine brightness transfer. Each frame gets two parameters, a gain and an offset, estimated jointly with the pose. For a pair of frames this amounts to

$$I_t(\mathbf{p}') \approx e^{a}\, I_r(\mathbf{p}) + b .$$

DSO writes the transfer function as $e^{-a_i}(I_i - b_i)$ for each frame $i$. The gain is parametrized logarithmically, which keeps it positive and, according to its authors, avoids numerical problems from multiplicative drift (Engel et al., 2016). They also note that the maximum-likelihood estimate of a multiplicative factor is biased when both images are noisy. As a result, an unconstrained gain drifts; they write that this generally has little effect on the poses but may bias them in scenes with only weak intensity variation.

A photometric camera model. DSO uses an image formation model in which the observed intensity is

$$I_i(\mathbf{x}) = G\big(t_i\, V(\mathbf{x})\, B_i(\mathbf{x})\big),$$

with irradiance $B_i$, exposure time $t_i$, a vignetting map $V$ and a nonlinear response function $G$. With a calibrated $G$ and $V$, each frame is first corrected to $G^{-1}(I_i)/V$, which is proportional to irradiance. With known exposure times, a prior then pulls the affine parameters toward zero (Engel et al., 2016). The image sensor and exposure articles describe where these effects come from.

Structure: depth from the same error

When depth is unknown, as for a monocular camera, the photometric error also constrains it. The systems estimate it in three ways:

• Dense, regularized. DTAM accumulates, for every pixel of a keyframe, the average L1 photometric error over tens to hundreds of nearby video frames in a cost volume indexed by inverse depth. It then minimizes that data term plus a regularizer, a weighted Huber norm of the inverse depth map’s gradient (a smoothed total variation), on a GPU (Newcombe et al., 2011). See cost volume and multi-view stereo.
• Semi-dense, filtered. Engel, Sturm and Cremers (2013) estimate inverse depth only at pixels with non-negligible gradient. Each estimate is a Gaussian, refined by many small-baseline stereo comparisons along epipolar lines and propagated from frame to frame. LSD-SLAM builds on this (Engel et al., 2014).
• Sparse, jointly optimized. DSO keeps a fixed number of sampled points and optimizes their inverse depths jointly with poses, intrinsics and affine parameters in a window of keyframes (Engel et al., 2016).

Engel et al. (2016) separate the extent of the image region used from a more consequential difference, the geometry prior. Dense and semi-dense methods use the connectedness of the image region to add a smoothness term, which a dense world model needs to be observable from passive vision. That prior couples neighbouring depths, which makes a statistically consistent joint optimization with the poses impractical in real time. Sparse methods have no prior, so points are independent given the poses, and the Schur complement solves the system as in bundle adjustment. The two axes are independent, and Engel et al. (2016) give examples of all four combinations:

	Indirect (geometric error)	Direct (photometric error)
Sparse	keypoint-based systems such as MonoSLAM, PTAM and ORB-SLAM	DSO; an EKF-based formulation by Jin et al. (2003)
Dense or semi-dense	geometry from dense, regularized optical flow	DTAM, a precursor by Stühmer, Gumhold and Cremers (2010), LSD-SLAM

How to Apply It

The example renders a room seen from inside: five planes textured with images bundled with scikit-image. It renders two views related by a known motion of 3.76° and 183 mm. The second view’s exposure then changes: brighter with clipping at white, or darker with an offset. Each method starts at the identity pose. Each method gets the depth of the reference view, as an RGB-D camera or a converged depth map would supply it, so the example isolates pose estimation. Four direct variants align all pixels with Gauss–Newton on a three-level pyramid. They differ in brightness model (brightness constancy or the affine model above) and in penalty (squares or Huber). The sources leave the penalty open (DVO tested none, Tukey and t-distribution weights; LSD-SLAM and DSO use Huber), so both are printed. The indirect baseline detects and matches ORB features and lifts the reference keypoints to 3D with the same depth. It runs RANSAC over four-match Gauss–Newton hypotheses and then minimizes the reprojection error over the inliers. The set-up was settled in pilot runs on the unchanged pair, before any exposure change was run. In the first, wider rooms the feature baseline failed: most ORB matches fell on the near-planar back wall, and six-point linear (DLT) pose hypotheses proved unreliable. In response to those pilot results, the room was narrowed in steps and given higher-contrast textures, and the DLT hypotheses were later replaced by the Gauss–Newton fits. A fact-check then found that, with the Gauss–Newton hypotheses, the feature baseline already recovers the unchanged motion in the original room, and that the room changes the size of the effects. The script therefore runs the same comparison, with the same methods, exposures, seeds and thresholds, in both rooms: the original one (4.4 m wide, floor 1.2 m below the camera, gravel, grass and brick textures) and the narrowed one (2 m wide, floor 0.6 m below, higher-contrast textures).

import numpy as np
from scipy import ndimage
from scipy.spatial.transform import Rotation
from skimage import color, data
from skimage.feature import ORB, match_descriptors

H, W, f = 240, 320, 300.0
cx, cy = (W - 1) / 2, (H - 1) / 2

# A room seen from inside, in the reference camera frame (x right, y down, z forward).
# Each plane n . X = d carries a bundled scikit-image picture, tiled at 160 texels per metre.
gray = lambda im: color.rgb2gray(im) if im.ndim == 3 else im / 255.0
rooms = {  # normal, d, texture, the two coordinates used as texture axes (column, row)
    "original room (4.4 m wide, floor 1.2 m below the camera)": [
        ([0, 0, 1.0], 4.0, gray(data.astronaut()), 0, 1),   # back wall
        ([0, 1.0, 0], 1.2, gray(data.gravel()), 0, 2),      # floor
        ([0, 1.0, 0], -1.8, gray(data.grass()), 0, 2),      # ceiling
        ([1.0, 0, 0], 2.2, gray(data.brick()), 2, 1),       # right wall
        ([1.0, 0, 0], -2.2, gray(data.brick()), 2, 1),      # left wall
    ],
    "narrowed room (2 m wide, floor 0.6 m below the camera)": [
        ([0, 0, 1.0], 4.0, gray(data.astronaut()), 0, 1),   # back wall
        ([0, 1.0, 0], 0.6, gray(data.camera()), 0, 2),      # floor
        ([0, 1.0, 0], -0.8, gray(data.rocket()), 0, 2),     # ceiling
        ([1.0, 0, 0], 1.0, gray(data.chelsea()), 2, 1),     # right wall
        ([1.0, 0, 0], -1.0, gray(data.coffee()), 2, 1),     # left wall
    ],
}

def render(planes, R, t, ss=3):
    """Ray-cast the room into a camera with X_cam = R X_ref + t (ss x ss supersampling)."""
    v, u = np.mgrid[0:H * ss, 0:W * ss] / ss + (1 / ss - 1) / 2
    rays = np.stack([(u - cx) / f, (v - cy) / f, np.ones_like(u)], -1) @ R  # in reference frame
    c = -R.T @ t                                                          # camera centre
    depth, img = np.full(u.shape, np.inf), np.zeros(u.shape)
    for n, d, tex, a0, a1 in planes:
        with np.errstate(divide="ignore", invalid="ignore"):
            s = (d - c @ n) / (rays @ n)                 # ray parameter = depth, since ray z = 1
        s[~np.isfinite(s) | (s <= 0)] = np.inf
        X = c + s[..., None] * rays
        val = ndimage.map_coordinates(tex, [X[..., a1] * 160, X[..., a0] * 160], order=1, mode="grid-wrap")
        hit = s < depth
        img[hit], depth[hit] = val[hit], s[hit]
    return img.reshape(H, ss, W, ss).mean((1, 3)), depth.reshape(H, ss, W, ss).mean((1, 3))

R_true = Rotation.from_euler("xyz", [2.0, -3.0, 1.0], degrees=True).as_matrix()
t_true = np.array([0.10, -0.03, 0.15])                   # metres

def pose_error(R, t):
    """Rotation error in degrees and translation error in millimetres."""
    return np.degrees(Rotation.from_matrix(R @ R_true.T).magnitude()), 1000 * np.linalg.norm(t - t_true)

def twist_jacobian(Ju, Jv, Jz, X):
    """Chain a derivative w.r.t. the 3D point X' into one w.r.t. a left-multiplied twist (v, w)."""
    x, y, z = X.T
    return np.stack([Ju, Jv, Jz, -Jv * z + Jz * y, Ju * z - Jz * x, -Ju * y + Jv * x], 1)

def update(R, t, dx):
    dR = Rotation.from_rotvec(dx[3:6]).as_matrix()
    return dR @ R, dR @ t + dx[:3]

def direct_align(I_r, Z_r, I_t, affine, huber=None, levels=3, iters=50):
    """Gauss-Newton on the photometric error I_t(p') - (e^a I_r(p) + b), coarse to fine."""
    pyr = lambda I: [I] + [I := ndimage.gaussian_filter(I, 1.0)[::2, ::2] for _ in range(levels - 1)]
    Ir_p, It_p = pyr(I_r), pyr(I_t)
    R, t, a, b = np.eye(3), np.zeros(3), 0.0, 0.0
    for l in reversed(range(levels)):
        s = 2 ** l
        fl, cxl, cyl = f / s, (cx + 0.5) / s - 0.5, (cy + 0.5) / s - 0.5
        Ir, It, Zr = Ir_p[l], It_p[l], Z_r[::s, ::s]
        gy, gx = np.gradient(It)
        v, u = np.mgrid[0:Ir.shape[0], 0:Ir.shape[1]]
        X = np.stack([(u - cxl) / fl * Zr, (v - cyl) / fl * Zr, Zr], -1).reshape(-1, 3)
        for _ in range(iters):
            Xc = X @ R.T + t
            pu, pv = fl * Xc[:, 0] / Xc[:, 2] + cxl, fl * Xc[:, 1] / Xc[:, 2] + cyl
            ok = (pu > 0) & (pu < It.shape[1] - 1) & (pv > 0) & (pv < It.shape[0] - 1)
            at = [pv[ok], pu[ok]]
            Iw, Gx, Gy = (ndimage.map_coordinates(im, at, order=1) for im in (It, gx, gy))
            ref, Xo = Ir.ravel()[ok], Xc[ok]
            r = Iw - (np.exp(a) * ref + b)
            Ju, Jv = Gx * fl / Xo[:, 2], Gy * fl / Xo[:, 2]
            Jz = -(Ju * Xo[:, 0] + Jv * Xo[:, 1]) / Xo[:, 2]
            J = twist_jacobian(Ju, Jv, Jz, Xo)
            if affine:                                   # d r / d a and d r / d b
                J = np.hstack([J, -np.exp(a) * ref[:, None], -np.ones((len(r), 1))])
            w = np.ones_like(r) if huber is None else np.minimum(1.0, huber / np.maximum(np.abs(r), 1e-12))
            dx = np.linalg.solve((J * w[:, None]).T @ J, -(J * w[:, None]).T @ r)
            R, t = update(R, t, dx)
            if affine:
                a, b = a + dx[6], b + dx[7]
            if np.linalg.norm(dx) < 1e-8:
                break
    return R, t, np.exp(a), b

def project(Xc):
    return np.column_stack([f * Xc[:, 0] / Xc[:, 2] + cx, f * Xc[:, 1] / Xc[:, 2] + cy])

def reprojection_gn(X, uv, iters=20):
    """Gauss-Newton on the reprojection error, started at the identity pose."""
    R, t = np.eye(3), np.zeros(3)
    for _ in range(iters):
        Xc = X @ R.T + t
        r = (project(Xc) - uv).ravel()
        x, y, z = Xc.T
        Jx = twist_jacobian(f / z, 0 * z, -f * x / z**2, Xc)   # row of the u residual
        Jy = twist_jacobian(0 * z, f / z, -f * y / z**2, Xc)   # row of the v residual
        J = np.stack([Jx, Jy], 1).reshape(-1, 6)
        R, t = update(R, t, np.linalg.lstsq(J, -r, rcond=None)[0])
    return R, t

def feature_align(I_r, Z_r, I_t, trials=500, thresh=2.0):
    """ORB matches, 3D points from the reference depth, RANSAC, then reprojection-error refinement."""
    kp, desc = [], []
    for I in (I_r, I_t):
        orb = ORB(n_keypoints=500, fast_threshold=0.05)
        orb.detect_and_extract(I)
        kp.append(orb.keypoints); desc.append(orb.descriptors)
    m = match_descriptors(desc[0], desc[1], cross_check=True, max_ratio=0.8)
    rc, uv = kp[0][m[:, 0]], kp[1][m[:, 1]][:, ::-1]           # (row, col) and (u, v)
    z = ndimage.map_coordinates(Z_r, rc.T, order=1)
    X = np.column_stack([(rc[:, 1] - cx) / f * z, (rc[:, 0] - cy) / f * z, z])
    rs, best = np.random.default_rng(1), np.zeros(len(X), bool)
    for _ in range(trials):                                       # hypotheses from 4 matches
        idx = rs.choice(len(X), 4, replace=False)
        R, t = reprojection_gn(X[idx], uv[idx], iters=10)
        Xc = X @ R.T + t
        if np.all(Xc[:, 2] > 0):
            inliers = np.linalg.norm(project(Xc) - uv, axis=1) < thresh
            best = inliers if inliers.sum() > best.sum() else best
    R, t = reprojection_gn(X[best], uv[best])
    return R, t, len(X), best.sum()

for room, planes in rooms.items():
    I_ref, Z_ref = render(planes, np.eye(3), np.zeros(3))   # reference image and its depth
    I_lin, _ = render(planes, R_true, t_true)               # second view, same exposure
    print(f"=== {room} ===")
    rng = np.random.default_rng(0)
    sigma = 0.01                                            # image noise, intensities in [0, 1]
    I_r = I_ref + sigma * rng.standard_normal(I_ref.shape)
    noise_t = sigma * rng.standard_normal(I_ref.shape)
    print(f"true motion: {np.degrees(Rotation.from_matrix(R_true).magnitude()):.2f} deg, "
          f"{1000 * np.linalg.norm(t_true):.0f} mm; reference depth {Z_ref.min():.2f}-{Z_ref.max():.2f} m")
    for label, gain, offset in [("same exposure", 1.0, 0.0), ("brighter", 1.6, 0.0),
                                ("darker, offset", 0.6, 0.08)]:
        exposed = gain * I_lin + offset
        I_t = np.clip(exposed, 0.0, 1.0) + noise_t
        print(f"\n{label}: target = clip({gain} I + {offset}), {100 * np.mean(exposed > 1):.1f}% of pixels saturated")
        for name, kw in [("brightness constancy, L2", dict(affine=False)),
                         ("brightness constancy, Huber", dict(affine=False, huber=0.04)),
                         ("affine brightness, L2", dict(affine=True)),
                         ("affine brightness, Huber", dict(affine=True, huber=0.04))]:
            R, t, g, o = direct_align(I_r, Z_ref, I_t, **kw)
            e_rot, e_t = pose_error(R, t)
            extra = f"  gain {g:.3f}, offset {o:+.3f}" if kw["affine"] else ""
            print(f"  direct, {name:28s} {e_rot:8.3f} deg {e_t:9.1f} mm{extra}")
        R, t, n_match, n_in = feature_align(I_r, Z_ref, I_t)
        e_rot, e_t = pose_error(R, t)
        print(f"  ORB + RANSAC + reprojection error   {e_rot:8.3f} deg {e_t:9.1f} mm  "
              f"{n_in} inliers of {n_match} matches")

Output (NumPy 2.3.5, SciPy 1.16.3, scikit-image 0.25.2; the script runs in about 10 seconds):

=== original room (4.4 m wide, floor 1.2 m below the camera) ===
true motion: 3.76 deg, 183 mm; reference depth 3.01-4.00 m

same exposure: target = clip(1.0 I + 0.0), 0.0% of pixels saturated
  direct, brightness constancy, L2        0.004 deg       0.3 mm
  direct, brightness constancy, Huber     0.002 deg       0.1 mm
  direct, affine brightness, L2           0.005 deg       0.4 mm  gain 0.976, offset +0.011
  direct, affine brightness, Huber        0.003 deg       0.2 mm  gain 0.982, offset +0.008
  ORB + RANSAC + reprojection error      0.111 deg       7.1 mm  230 inliers of 253 matches

brighter: target = clip(1.6 I + 0.0), 29.9% of pixels saturated
  direct, brightness constancy, L2        0.038 deg       3.1 mm
  direct, brightness constancy, Huber     0.372 deg      27.6 mm
  direct, affine brightness, L2           0.010 deg       0.6 mm  gain 1.239, offset +0.095
  direct, affine brightness, Huber        0.004 deg       0.3 mm  gain 1.334, offset +0.062
  ORB + RANSAC + reprojection error      0.623 deg      44.4 mm  108 inliers of 125 matches

darker, offset: target = clip(0.6 I + 0.08), 0.0% of pixels saturated
  direct, brightness constancy, L2        0.137 deg      12.3 mm
  direct, brightness constancy, Huber     0.050 deg       3.4 mm
  direct, affine brightness, L2           0.005 deg       0.4 mm  gain 0.586, offset +0.087
  direct, affine brightness, Huber        0.003 deg       0.2 mm  gain 0.587, offset +0.086
  ORB + RANSAC + reprojection error      0.043 deg       1.9 mm  211 inliers of 233 matches
=== narrowed room (2 m wide, floor 0.6 m below the camera) ===
true motion: 3.76 deg, 183 mm; reference depth 1.51-4.00 m

same exposure: target = clip(1.0 I + 0.0), 0.0% of pixels saturated
  direct, brightness constancy, L2        0.006 deg       0.3 mm
  direct, brightness constancy, Huber     0.003 deg       0.1 mm
  direct, affine brightness, L2           0.007 deg       0.4 mm  gain 0.983, offset +0.006
  direct, affine brightness, Huber        0.003 deg       0.1 mm  gain 0.989, offset +0.004
  ORB + RANSAC + reprojection error      0.060 deg       3.4 mm  284 inliers of 299 matches

brighter: target = clip(1.6 I + 0.0), 18.1% of pixels saturated
  direct, brightness constancy, L2        0.144 deg       8.0 mm
  direct, brightness constancy, Huber     0.281 deg      15.2 mm
  direct, affine brightness, L2           0.019 deg       1.2 mm  gain 1.383, offset +0.054
  direct, affine brightness, Huber        0.010 deg       0.6 mm  gain 1.513, offset +0.021
  ORB + RANSAC + reprojection error      0.032 deg       2.7 mm  183 inliers of 201 matches

darker, offset: target = clip(0.6 I + 0.08), 0.0% of pixels saturated
  direct, brightness constancy, L2        0.170 deg       3.7 mm
  direct, brightness constancy, Huber     0.067 deg       5.5 mm
  direct, affine brightness, L2           0.008 deg       0.4 mm  gain 0.590, offset +0.084
  direct, affine brightness, Huber        0.004 deg       0.2 mm  gain 0.592, offset +0.083
  ORB + RANSAC + reprojection error      0.060 deg       1.2 mm  270 inliers of 287 matches

With unchanged exposure, every method recovers the motion in both rooms. The direct variants are accurate to 0.4 mm or better, roughly ten to seventy times more accurate in translation than the feature pipeline, which does better in the narrowed room. In our reading, this is because they use every pixel’s gradient, whereas the features rest on a few hundred keypoint positions, each localized to a fraction of a pixel. Under either exposure change, brightness constancy is the wrong model. Its translation errors grow by a factor of ten to almost three hundred, to several millimetres or more. The affine model keeps every direct error at 1.2 mm or below. The Huber penalty does not reliably help brightness constancy. It is worse than the squared penalty in the brighter case in both rooms, better in the darker case in the original room, and mixed in the darker case in the narrowed one. In our reading, a global gain or offset changes every residual, most of them by more than the Huber threshold, so there is no small set of outliers for the penalty to suppress.

How the feature pipeline compares depends on the room. ORB’s binary descriptors compare intensities within a patch, so in our reading a gain leaves them unchanged except where pixels saturate. In the narrowed room the features stay within 3.4 mm under both changes, better than brightness-constancy alignment. In the original room the darker case is similar, but in the brighter case, where nearly a third of the pixels saturate, the matches halve and the feature error becomes the worst of all methods, above even brightness constancy. In our reading, clipping erased texture in the bright regions, which is why the feature pipeline lost matches. The brighter case also shows the affine model’s limit. Clipping at white is not affine, so the gain estimated by least squares falls well short of the true 1.6 (1.24 in the original room, 1.38 in the narrowed one). The Huber variant treats the saturated pixels as outliers, comes closer (1.33 and 1.51) and has the smaller pose error in both rooms. In our reading, the brightness-constancy variants converge to a biased pose rather than diverge because the exposure residual, roughly proportional to the reference intensity, is only weakly aligned with the image changes caused by small pose shifts. With larger changes or weaker texture, the outcome may differ.

Variants

Dense direct

• Whole-image alignment against a model. Kerl et al. (2013, ICRA) trace dense direct pose estimation through earlier work. Koch estimated a camera’s pose from a textured 3D model by minimizing the photometric error with a synthesized image. Comport et al. applied the approach to stereo, and Steinbrücker et al. and Audras et al. to RGB-D images. DTAM (Newcombe et al., 2011) tracks a monocular camera by aligning each live frame with a view synthesized from its dense keyframe model, first estimating rotation only and then refining the full pose. It was initialized with a feature-based stereo method and ran on a GPU.
• Dense RGB-D odometry and SLAM (DVO). Kerl et al. (2013, ICRA) align consecutive RGB-D frames with every pixel of the colour image and the depth of the first. They use t-distribution weights and a constant-velocity prior, on one CPU core. In “Dense Visual SLAM for RGB-D Cameras” (2013, IROS), the same authors minimize photometric and depth errors together under a bivariate t-distribution. They select keyframes and loop closures with an entropy-based measure and optimize the resulting pose graph. The TUM RGB-D article lists that system’s trajectory errors.

Semi-dense direct

LSD-SLAM (Engel et al., 2014) tracks each new frame against the current keyframe’s semi-dense inverse depth map by $\mathrm{SE}(3)$ alignment. It then links keyframes in a pose graph by direct alignment in $\mathrm{Sim}(3)$, which includes scale. The photometric error alone does not constrain scale, so this alignment adds a residual on the difference of inverse depths. Each keyframe’s depth is normalized to a mean inverse depth of one, so scale drift between keyframes becomes explicit and loop closures can correct it. For large loop closures, where the previous pose is no good initial guess, the authors found that efficient second-order minimization (ESM) and a very coarse pyramid widened the convergence radius without changing precision. Candidate loops are kept only if alignment in both directions agrees.

Sparse direct

Direct Sparse Odometry (Engel et al., 2016) evaluates the photometric error on small patterns of pixels around evenly spread points, including edges and weakly textured regions. Inverse depths, poses, affine brightness parameters and camera intrinsics are optimized jointly in a sliding window of keyframes, and old variables are marginalized. Its authors describe this as the photometric equivalent of windowed sparse bundle adjustment.

Semi-direct

SVO (Forster et al., 2014) uses direct alignment only to get the correspondences. Sparse model-based image alignment estimates the frame-to-frame pose by minimizing the photometric error of 4 × 4 patches around the projections of known 3D points. Each patch is then aligned individually with its reference keyframe, using affine-warped 8 × 8 patches. This relaxation step violates the epipolar constraint to gain sub-pixel correspondence. Finally, pose and structure are refined on the reprojection error, as in motion-only and structure-only bundle adjustment. Features are extracted only when a keyframe is created. New points enter the map once a depth filter that models outlier measurements has converged.

Best Practices

• Initialize within the basin. Predict the pose from a motion model or an IMU, align coarse to fine, and detect failure. DSO, for example, treats a frame’s error rising to more than twice the previous frame’s as failure and retries from several small rotations (Engel et al., 2016).
• Model the camera’s photometry, or at least estimate an affine brightness transfer per frame, whenever exposure is automatic (Engel et al., 2016). The example shows the size of the effect when it is ignored.
• Use robust weights for occlusions and moving objects, as DVO, LSD-SLAM, DSO and DTAM do. Robust weights do not substitute for a correct photometric model, as the example shows.
• Use pixels with gradient. Pixels in flat regions add computation and noise but no constraint. Semi-dense and sparse methods restrict the error to gradient-rich pixels (Engel et al., 2013; Engel et al., 2016).
• Calibrate geometry and timing carefully, including lens distortion, the rolling shutter and the synchronization between colour and depth (Schöps, Sattler and Pollefeys, 2019).

Pitfalls

• Unmodeled exposure changes. Engel et al. (2016) attempted to evaluate the open-source LSD-SLAM and SVO on their datasets. They report that both consistently failed on most sequences, mainly because they assume brightness constancy while those datasets have heavy exposure variation.
• Global illumination changes. DTAM’s authors list it as a failure mode: they assume brightness constancy throughout. Their pixel rejection handles local illumination changes during tracking, but not real-world global ones (Newcombe et al., 2011).
• Geometric distortion: rolling shutter and inaccurate intrinsics. A distortion of a pixel or two moves residuals outside the region where the linearization holds, and a feature-based error is far less affected (Engel et al., 2016; see the image sensor article on rolling shutters).
• Bad initialization. Large motions, dropped frames and loop closures put the start outside the basin of convergence. LSD-SLAM’s authors list the non-convexity of direct alignment as an important limitation (Engel et al., 2014). SVO’s authors cite feature matching’s tolerance of large inter-frame motion as a reason for its success (Forster et al., 2014).
• Monocular scale. As in any monocular system, scale is unobservable and drifts. LSD-SLAM makes it a variable of the keyframe graph rather than hiding it (Engel et al., 2014).
• Assuming the residuals are Gaussian. Kerl et al. (2013, ICRA) found the residual distribution of real RGB-D data poorly fitted by a Gaussian. Unweighted least squares was then less accurate than their t-distribution weighting.

Evidence and Impact

The two approaches have long been set side by side: the proceedings volume Vision Algorithms: Theory and Practice contains both Irani and Anandan’s “About Direct Methods” and Torr and Zisserman’s “Feature Based Methods for Structure and Motion Estimation” (both 1999; neither was read for this article). Engel et al. (2016) write that feature-based methods dominated real-time SLAM and VO for a long time, and that direct and dense formulations had gained popularity in the years before their paper. The published comparisons below each come with their own set-up.

• DTAM (Newcombe et al., 2011) was compared with PTAM qualitatively and through velocity plots on a hand-held trajectory with high accelerations. The authors reported smoother motion estimates, tracking that continued where PTAM had to relocalize, and graceful degradation under motion blur and camera defocus.
• DVO odometry (Kerl et al., 2013, ICRA, Table IV): on the four TUM RGB-D “desk” sequences, with the real-time parameter set, the average RMSE of the translational drift was 0.0588 m/s without weights, 0.1491 m/s with Tukey weights, 0.0432 m/s with t-distribution weights, and 0.0428 m/s with the temporal prior added.
• LSD-SLAM (Engel et al., 2014, Fig. 9): absolute trajectory RMSE on TUM RGB-D fr2/desk and fr2/xyz was 4.52 cm and 1.47 cm. On the same sequences the authors list 13.50 cm and 3.79 cm for semi-dense monocular VO, a tracking failure and 24.28 cm for PTAM, 1.77 cm and 1.18 cm for DVO SLAM, and 9.5 cm and 2.6 cm for a keypoint-based RGB-D SLAM. The last two use the sensor’s depth; LSD-SLAM used only the first depth map, to bootstrap the scale.
• DSO against ORB-SLAM (Engel et al., 2016): with ORB-SLAM’s loop-closure detection and relocalization disabled, DSO was reported as more accurate and more robust on the photometrically calibrated TUM monoVO dataset and on the synthetic ICL-NUIM sequences. The comparison uses cumulative error plots over all runs (each sequence forwards and backwards, several times), of the alignment error on TUM monoVO and of the absolute trajectory RMSE after Sim(3) alignment on the other two. On EuRoC MAV, which has no photometric calibration, ORB-SLAM was more accurate but less robust. The authors attribute this partly to the missing calibration and partly to ORB-SLAM implicitly closing the many small loops in those flights. In their photometric-calibration ablation, the plain brightness constancy model performed worst.
• Photometric against geometric noise (Engel et al., 2016, Section 4.3). On TUM monoVO, the authors added a smooth random displacement field of up to a few pixels to each frame, simulating rolling shutter or bad intrinsics. DSO’s accuracy deteriorated quickly while ORB-SLAM’s was much less affected. Beyond 1.5 pixels, they argue, probably no state keeps all residuals within the linearization radius. Strong spatially varying blur, their model of photometric noise, affected DSO slightly less than ORB-SLAM. They summarize that the direct approach wins on well-calibrated data and is ill-suited to strong geometric noise.
• Direct RGB-D and sensor effects. Schöps et al. (2019) found the feature-based ORB-SLAM2 ahead of all direct methods on TUM RGB-D. They attribute this, as likely, to rolling shutter, unsynchronized colour and depth, and depth distortion. They report that direct RGB-D SLAM systems are highly sensitive to these effects. On their own benchmark, recorded with synchronized global-shutter cameras (the SLAM part of ETH3D), their direct bundle adjustment outperformed ORB-SLAM2.

Taken together, the sources agree on the trade-off. Direct methods extract more information from well-calibrated images and work where keypoints are scarce. In exchange, they need an accurate camera model, photometric and geometric, and a good initial guess. The semi-direct and hybrid designs, and the use of feature-based relocalization and loop detection in otherwise direct systems, follow from that trade-off.