Plane Sweep Stereo

Plane sweep stereo computes depth from several calibrated images without rectifying them. It moves a plane through the scene in small steps; at each step it maps every image onto the plane, or onto a reference view through the plane, with the homography that the plane induces, and checks where the images agree. A pixel’s depth is the position of the plane on which its neighbourhood matched best. The procedure works for any number of views in any arrangement, needs only image warping and simple per-pixel arithmetic, and produces the depth-indexed cost volume on which much of multi-view stereo, classical and learned, is built.

Robert Collins (1996) described the idea as a space-sweep for matching point-like features, such as edgels, across many images, and illustrated it on seven aerial images. Gallup et al. (2007) credit Collins with introducing plane sweeping; Szeliski (2022, §12.9) says Collins first popularized it and credits Szeliski and Golland (1999) and Saito and Kanade (1999) with generalizing it to an arbitrary projective setting. Yang and Pollefeys (2003) ran the dense, reference-view form in real time on graphics hardware; Gallup et al. (2007) swept planes in several directions to handle slanted ground and facades, and MVSNet (Yao et al., 2018) made the warp a differentiable layer of a network.

Problem

Given images with known intrinsics and poses, find for each pixel of a chosen reference image the depth of the surface it sees. Two-view stereo matching solves this by first rectifying the pair so that matches lie on the same row; with many views in general positions, no single rectification serves them all. Plane sweeping avoids rectification: instead of searching along each epipolar line in each image separately, it hypothesizes a surface position in 3D and checks all images against it at once.

Inputs and Outputs

Input: a reference image and $N$ source images with camera matrices (usually from structure from motion); a family of planes, given by a normal direction and a range of distances; a matching cost and an aggregation window.
Output: a cost volume $C(x, y, m)$ over pixels and plane hypotheses $m$ , and from it a depth map for the reference view, often with a per-pixel confidence. Collins’s original sweep output 3D feature points and their correspondences instead.

Intuition

Place a large flat sheet in front of the cameras and project every photograph onto it, as a slide projector would. Wherever the sheet passes through a real surface, all projectors paint the same pattern on the same spot, because each of them is showing the same piece of surface. Away from the surface, the projected patterns are shifted against each other and the sheet looks blurred. Moving the sheet through the scene and recording, at each point, the position where the patterns agreed gives the depth.

In practice the sheet is sampled in the reference camera’s pixels: each source image is warped into the reference view as if the scene were the plane, and compared with the reference image. A small window around each pixel is compared rather than the pixel alone, to suppress noise. That window is where the method’s main bias comes from: the comparison is exact only where the whole window lies on the swept plane, so a surface that is tilted relative to the plane matches only along a line, not over the window.

Algorithm

Dense plane sweeping with a reference view:

Choose a family of planes $\mathbf{n}^\top\mathbf{X} = d_m$ , $m = 1, \dots, M$ , in the reference camera’s frame: a normal $\mathbf{n}$ (classically the viewing direction, $\mathbf{n} = (0, 0, 1)^\top$ , giving fronto-parallel planes) and distances $d_m$ that bracket the scene.
For each plane $m$ and each source image $k$ , compute the homography $H_k(m)$ from reference pixels to source pixels and warp the source image into the reference view, resampling with bilinear interpolation.
Compute a per-pixel dissimilarity between the reference image and each warped source image, aggregate it over a window, and combine the views (sum, mean, or a robust subset) into the cost $C(x, y, m)$ .
For each pixel, select the plane of lowest cost (winner-take-all), optionally refine between neighbouring planes by fitting a parabola to the costs, or replace this step by a global optimization such as semi-global matching or graph cuts.
Convert the chosen plane at each pixel into a depth by intersecting the pixel’s ray with it.

The full volume need not be stored for winner-take-all: a running minimum per pixel, updated plane by plane, is enough, which is one reason the method suited early graphics hardware.

Collins’s (1996) feature version differs in steps 3–5. A plane $Z = z_i$ , divided into cells, is swept along the scene’s $Z$ axis; feature points (Canny edgels in his example) from every image are back-projected onto it, and every cell near a back-projected point receives a vote, so each cell counts the viewing rays that pass close to it. Cells with enough votes become 3D features, and the features that voted for a cell are its correspondences. The plane then moves on and the counts reset.

Mathematical Formulation

The homography induced by a plane

Let the reference camera be $K[I \mid \mathbf{0}]$ and source camera $k$ be $K_k[R_k \mid \mathbf{t}_k]$ , mapping a point $\mathbf{X}$ in reference coordinates to $\mathbf{X}_k = R_k\mathbf{X} + \mathbf{t}_k$ in its own frame. For a point on the plane $\mathbf{n}^\top\mathbf{X} = d$ (with $d > 0$ its distance from the reference centre), $\mathbf{n}^\top\mathbf{X}/d = 1$ , so

\mathbf{X}_k = R_k\mathbf{X} + \mathbf{t}_k\,\frac{\mathbf{n}^\top\mathbf{X}}{d} = \left(R_k + \frac{\mathbf{t}_k\,\mathbf{n}^\top}{d}\right)\mathbf{X}.

With $\mathbf{X} = Z\,K^{-1}\mathbf{x}$ for a reference pixel $\mathbf{x}$ and projection by $K_k$ , the depth $Z$ drops out up to scale and

H_k(\mathbf{n}, d) = K_k\left(R_k + \frac{\mathbf{t}_k\,\mathbf{n}^\top}{d}\right)K^{-1}

maps reference pixels to source pixels. This is the form given in multi-view stereo for $\mathbf{n} = (0, 0, 1)^\top$ ; the homography article writes the plane as $\mathbf{n}^\top\mathbf{X} + d' = 0$ , so $d' = -d$ and its minus sign becomes a plus. Papers differ here: Gallup et al. (2007) write the plane as $[\mathbf{n}_m^\top \; {-d_m}]$ with $d_m$ the distance (their Eq. 2), while their homography and depth formulas (Eqs. 3 and 7) hold, in our reading, for the plane $\mathbf{n}_m^\top\mathbf{X} + d_m = 0$ , the form $[\mathbf{n}_m^\top \; d_m]$ their Section 3.2 uses. MVSNet’s Eq. (1), as printed in the ECCV copy, ends with $R_1^\top K_1^\top$ where, in our reading, $K_1^{-1}$ is meant. Checking a formula numerically against ray-cast points, as the code below does implicitly, settles such conventions.

The depth that plane $(\mathbf{n}, d)$ assigns to reference pixel $\mathbf{x}$ follows from $\mathbf{n}^\top Z K^{-1}\mathbf{x} = d$ :

Z(\mathbf{x}) = \frac{d}{\mathbf{n}^\top K^{-1}\mathbf{x}},

which is $Z = d$ for fronto-parallel planes and varies across the image for any other normal. Hypotheses with $\mathbf{n}^\top K^{-1}\mathbf{x} \le 0$ lie behind the camera and are skipped.

For a fixed normal, the homographies of a family differ only by a rank-one term proportional to $1/d$ . Szeliski (2022, Eq. 12.3) writes the family as $\tilde H_k + \mathbf{t}_k[0\;0\;d]$ , a base homography plus a rank-one matrix in his projective disparity $d$ , and notes that its members are related by a planar homology. Collins (1996) used the same structure for speed: he maps each image’s features once onto a canonical plane $Z = z_0$ ; for a plane parallel to it at $Z = z_i$ , the map between the two plane positions is a dilation about the foot $(C_x, C_y)$ of the camera centre,

x_i = \delta x_0 + (1 - \delta)C_x, \qquad y_i = \delta y_0 + (1 - \delta)C_y, \qquad \delta = \frac{z_i - C_z}{z_0 - C_z},

an isotropic scaling that is much cheaper to apply than a general homography.

Plane spacing

Where the planes lie matters, not only how many there are. For a source displaced from the reference by $\mathbf{t}$ , the source position of a reference pixel at depth $Z$ is $K_k(R_k K^{-1}\mathbf{x} + \mathbf{t}/Z)$ in homogeneous coordinates: it moves along the epipolar line linearly in $1/Z$ (exactly, in pixels, when the source keeps the reference’s orientation and $\mathbf{t}$ is parallel to the image plane; approximately otherwise). Planes spaced uniformly in inverse depth therefore shift the warped images by roughly equal numbers of pixels, as uniform steps in disparity do in rectified stereo; planes spaced uniformly in depth take large pixel steps near the camera and tiny ones far away. Yang and Pollefeys (2003) placed the planes for a standard two-camera rig at the depths of successive disparities, $Z = -fb/d$ in their sign convention.

Gallup et al. (2007) turned the pixel-step criterion into a rule. They define the disparity change between consecutive planes as the largest displacement, over all pixels and all images, between the positions the two planes’ homographies give, and build each family so that it is at most one pixel, which matters most for surfaces with high-frequency texture. The largest displacement occurs at the image boundary, so they warp the image outline polygon, clip it, and check only its vertices. They also avoid planes that intersect the convex hull of the camera centres, which would invert the orientation of the warp. Szeliski (2022, §12.1.2) describes the choice of sampling as application dependent and set to bracket the working volume; for obstacle avoidance, for example, the disparity can be set up to measure elevation above the ground.

Cost, aggregation and depth extraction

With a matching cost $\rho$ (absolute or squared difference, NCC, census; see cost volume), a window $\mathcal{W}$ and the warped source images $\tilde I_k^{(m)}$ , a typical cost is

C(\mathbf{x}, m) = \sum_{k=1}^{N}\;\sum_{\mathbf{u} \in \mathcal{W}} \rho\!\left(I_\text{ref}(\mathbf{x} + \mathbf{u}),\; \tilde I_k^{(m)}(\mathbf{x} + \mathbf{u})\right).

Gallup et al. used absolute differences summed over a rectangular window and over the views, with a gain ratio per source image to compensate for exposure changes, which they call essential outdoors. Szeliski (2022, §12.1.2) describes the warped images as a generalized disparity space image $\tilde I(x, y, d, k)$ , from which the usual summed cost or other statistics, such as robust variance, can be computed, or occlusions reasoned about. Yang and Pollefeys (2003) summed squared differences over windows of several sizes using the graphics card’s mipmaps, which they describe as equivalent to one large pyramid-shaped weighted window.

The best plane per pixel is $\hat m(\mathbf{x}) = \arg\min_m C(\mathbf{x}, m)$ and the depth is $Z(\mathbf{x})$ for that plane. Gallup et al. refined it to sub-plane precision by fitting a parabola to the costs around the minimum, and handled occlusions with a scheme they adopted from Kang et al. (2001): compute the cost separately for the cameras on the left and on the right of the reference and keep the smaller, since a pixel’s visibility usually changes at most once along a camera path.

Parameters

Depth range and number of planes. The range must contain the scene (Gallup et al. take it from the structure-from-motion points or from heuristics, for example no ground planes above the camera centre); the number of planes should keep the pixel step between neighbouring planes at about one pixel or less. Gallup et al. reported that a scene typically needs hundreds of planes per sweeping direction.
Spacing. Uniform in inverse depth gives roughly uniform pixel steps; MVSNet samples uniformly in depth over DTU’s narrow range (see Variants).
Plane orientation. Fronto-parallel by default; additional families aligned with the expected surfaces (ground, facades) in the multiple-direction variant.
Window size and cost. Larger windows suppress noise and ambiguity but blur depth edges and amplify the fronto-parallel bias on slanted surfaces; costs with gain compensation or normalization tolerate lighting changes.
Views. More views reduce ambiguity, but each adds occlusions; view selection and robust aggregation are discussed in multi-view stereo.

Complexity

For $M$ planes, $N$ source images and $P$ reference pixels, the sweep performs $MN$ warps and cost evaluations of $P$ pixels each; with box filtering by running sums, window aggregation costs $O(1)$ per pixel independent of the window size, giving $O(MNP)$ time. The cost is linear in the number of images, one of the conditions Collins (1996) set for a “true multi-image” method. Storing the whole volume takes $O(MP)$ memory; winner-take-all needs only $O(P)$ . On a GeForce4 card, Yang and Pollefeys (2003) reported 50–70 million disparity evaluations per second, including transfer of the images and the result. Gallup et al. (2007) computed depth from 11 grayscale images of 512 × 384 pixels on a GeForce 8800 GTX; their multiple-direction sweep with best-cost selection ran at 5.26 Hz with 282 plane hypotheses.

Implementation

The script renders a street-like scene from five cameras: a gravel ground plane 1.5 m below the cameras, a brick facade 2 m to the right running away from them, and a grass-textured back wall at 14 m, with textures from images bundled with scikit-image. Coordinates are those of the reference camera, with $X$ to the right, $Y$ down and $Z$ forward; the four source cameras are translated by 0.3 m left, right, up and down without rotation, so each is $K[I \mid -\mathbf{C}_k]$ and its homography is $K(I - \mathbf{C}_k\mathbf{n}^\top/d)K^{-1}$ . Depth means $Z$ in the reference frame. Each image is rendered with 3 × 3 supersampling and a blur chosen from each pixel’s footprint, a crude mipmap that keeps distant texture from aliasing, plus Gaussian noise. The sweep uses the absolute difference averaged over a 7 × 7 window and over the source views, winner-take-all and a parabolic refinement in the spacing variable. The scene has no occlusions, so no visibility handling is needed.

It compares a fronto-parallel sweep with a sweep in three directions (fronto-parallel, the ground normal and the facade normal, each with 32 planes spaced uniformly in inverse distance, combined by lowest cost as in Gallup et al.’s best-cost selection), and two spacings of the fronto-parallel family. Because “equal planes per family” and “equal total planes” are both reasonable budgets, the fronto-parallel sweep is run with 32 and with 96 planes. Errors are relative depth errors on pixels at least 12 pixels from the image border and 4 pixels from a boundary between surfaces.

import numpy as np
from scipy import ndimage
from skimage import data, img_as_float

H, W, f = 180, 240, 180.0
K = np.array([[f, 0, W / 2], [0, f, H / 2], [0, 0, 1]])
Kinv = np.linalg.inv(K)
TEX = 0.01                                  # metres per texel
# Scene planes n.X = d in the reference frame (X right, Y down, Z forward), each
# with a bundled scikit-image texture and the two axes that index it.
PLANES = [(np.array([0., 1., 0.]), 1.5, data.gravel(), (0, 2)),   # ground  Y = 1.5
          (np.array([1., 0., 0.]), 2.0, data.brick(), (2, 1)),    # facade  X = 2
          (np.array([0., 0., 1.]), 14.0, data.grass(), (0, 1))]   # back wall Z = 14
SIGMAS = [0, 1, 2, 4, 8, 16]                # crude mipmap: blurred copies of each texture
MIP = [[ndimage.gaussian_filter(img_as_float(t), s, mode="wrap") for s in SIGMAS]
       for _, _, t, _ in PLANES]

def trace(C, u, v):
    """Ray-cast pixels (u, v) of a camera at C (no rotation): distance, surface, point."""
    r = np.stack([u, v, np.ones_like(u)], -1) @ Kinv.T
    s = np.stack([(d - n @ C) / np.where(r @ n == 0, 1e-12, r @ n) for n, d, _, _ in PLANES])
    s[s <= 0] = np.inf
    lab, s = s.argmin(0), s.min(0)
    return s, lab, C + s[..., None] * r          # s is also the depth (r_z = 1)

def render(C, rng, ss=3, noise=0.01):
    img = np.zeros((H, W))
    for dy in (np.arange(ss) + 0.5) / ss - 0.5:
        for dx in (np.arange(ss) + 0.5) / ss - 0.5:
            u, v = np.meshgrid(np.arange(W) + dx, np.arange(H) + dy)
            s, lab, P = trace(C, u, v)
            lev = np.searchsorted(SIGMAS, 0.5 * s / f / TEX, side="right") - 1
            for p, (_, _, _, ax) in enumerate(PLANES):
                for l in np.unique(lev[lab == p]):
                    m = (lab == p) & (lev == l)
                    tc = P[m][:, list(ax)] / TEX
                    img[m] += ndimage.map_coordinates(MIP[p][l], [tc[:, 1], tc[:, 0]],
                                                      order=1, mode="grid-wrap") / ss**2
    return img + noise * rng.standard_normal(img.shape)

rng = np.random.default_rng(0)
ref = render(np.zeros(3), rng)
src_C = [np.array(c) for c in [(0.3, 0, 0), (-0.3, 0, 0), (0, 0.3, 0), (0, -0.3, 0)]]
srcs = [render(C, rng) for C in src_C]
u, v = np.meshgrid(np.arange(W), np.arange(H))
gt, lab, _ = trace(np.zeros(3), u.astype(float), v.astype(float))

pix = np.stack([u.ravel(), v.ravel(), np.ones(H * W)])
rays = Kinv @ pix
win = lambda a: ndimage.uniform_filter(a, 7)        # 7 x 7 aggregation window
ZMIN, ZMAX = 2.5, 16.0

def homography(C, n, d):
    """Reference pixel -> pixel of a camera at C (no rotation), for the plane n.X = d."""
    return K @ (np.eye(3) - np.outer(C, n) / d) @ Kinv

def sweep(n, dvals):
    """SAD cost volume for the planes n.X = d; inf where a hypothesis is invalid."""
    nr = (n @ rays).reshape(H, W)
    cost = np.full((len(dvals), H, W), np.inf)
    for m, d in enumerate(dvals):
        Z = np.where(nr > 0, d / np.where(nr > 0, nr, 1), -1)   # depth of the hypothesis
        acc, cnt = np.zeros((H, W)), np.zeros((H, W))
        for C, img in zip(src_C, srcs):
            q = homography(C, n, d) @ pix
            w = ndimage.map_coordinates(img, [q[1] / q[2], q[0] / q[2]], order=1,
                                        cval=np.nan).reshape(H, W)
            ok = win(np.isfinite(w).astype(float)) > 0.999
            acc += np.where(ok, win(np.abs(ref - np.nan_to_num(w))), 0)
            cnt += ok
        valid = (Z >= ZMIN) & (Z <= ZMAX) & (cnt >= 2)
        cost[m] = np.where(valid, acc / np.maximum(cnt, 1), np.inf)
    return cost

def readout(cost, n, p, inverse):
    """Winner-take-all plus a parabola through the neighbouring costs, in the spacing variable p."""
    m = np.clip(np.argmin(cost, 0), 1, len(p) - 2)
    cm, c0, cp = (np.take_along_axis(cost, (m + k)[None], 0)[0] for k in (-1, 0, 1))
    with np.errstate(invalid="ignore", divide="ignore"):
        den = cm - 2 * c0 + cp
        delta = np.where(np.isfinite(den) & (den > 0), np.clip((cm - cp) / (2 * den), -.5, .5), 0)
        d = p[m] + delta * (p[1] - p[0])
        Z = (1 / d if inverse else d) / (n @ rays).reshape(H, W)
    return Z, cost.min(0)

def family(n, a, b, M, inverse=True):
    p = np.linspace(1 / a, 1 / b, M) if inverse else np.linspace(a, b, M)
    return readout(sweep(n, 1 / p if inverse else p), n, p, inverse)

FRONT, GROUND, FACADE = np.array([0., 0., 1.]), np.array([0., 1., 0.]), np.array([1., 0., 0.])
inner = np.zeros((H, W), bool); inner[12:-12, 12:-12] = True
boundary = ndimage.binary_dilation(ndimage.morphological_gradient(lab, size=2) > 0, iterations=4)
evalm = inner & ~boundary
names = ["ground", "facade", "back wall"]

def report(name, Z):
    rel = np.abs(Z - gt) / gt
    cells = [f"{np.median(rel[evalm & (lab == s)]) * 100:5.2f}% {np.mean(rel[evalm & (lab == s)] < .02) * 100:5.1f}%"
             for s in range(3)]
    print(f"{name:<28s}" + " | ".join(cells))

print("evaluated pixels:", ", ".join(f"{nm} {np.sum(evalm & (lab == s))}" for s, nm in enumerate(names)))
print(f"{'median error, within 2%':<28s}" + " | ".join(f"{nm:<14s}" for nm in names))
fp32 = family(FRONT, 2.5, 16.0, 32)
report("fronto-parallel, 32", fp32[0])
report("fronto-parallel, 96", family(FRONT, 2.5, 16.0, 96)[0])
fams = [family(FRONT, 2.5, 16.0, 32), family(GROUND, 1.0, 2.0, 32), family(FACADE, 1.0, 3.0, 32)]
pick = np.argmin(np.stack([c for _, c in fams]), 0)          # best cost over directions
Zmd = np.choose(pick, [Z for Z, _ in fams])
report("3 directions, 3 x 32", Zmd)
truth = np.array([1, 2, 0])[lab]                              # family that matches each surface
print("direction chosen correctly:", ", ".join(
    f"{nm} {np.mean(pick[evalm & (lab == s)] == truth[evalm & (lab == s)]) * 100:.1f}%"
    for s, nm in enumerate(names)))

print("\nplane spacing, fronto-parallel, 32 planes from 2.5 to 16 m")
bins = [(2.5, 5), (5, 9), (9, 16)]
for inverse in (False, True):
    Z = fp32[0] if inverse else family(FRONT, 2.5, 16.0, 32, inverse=False)[0]
    p = np.linspace(1 / 2.5, 1 / 16, 32) if inverse else np.linspace(2.5, 16, 32)
    dv = 1 / p if inverse else p
    q = [[homography(C, FRONT, d) @ pix for C in src_C] for d in dv]
    D = [max(np.hypot(*(a[:2] / a[2] - b[:2] / b[2])).max() for a, b in zip(q[m], q[m + 1]))
         for m in range(31)]
    rel = np.abs(Z - gt) / gt
    cells = [f"{lo:g}-{hi:g} m {np.mean(rel[evalm & (gt >= lo) & (gt < hi)] < .02) * 100:5.1f}%"
             for lo, hi in bins]
    print(f"uniform in {'1/depth' if inverse else 'depth  '}: step {min(D):.2f}-{max(D):.2f} px | "
          + " | ".join(cells))

Output (NumPy 2.3.5, SciPy 1.16.3, scikit-image 0.25.2; about 3 seconds):

evaluated pixels: ground 9687, facade 10129, back wall 13126
median error, within 2%     ground         | facade         | back wall     
fronto-parallel, 32          0.72%  87.7% |  1.29%  70.4% |  1.77%  83.6%
fronto-parallel, 96          0.84%  83.5% |  1.35%  67.0% |  0.98%  97.9%
3 directions, 3 x 32         0.20%  98.7% |  0.54%  90.7% |  1.77%  83.5%
direction chosen correctly: ground 99.8%, facade 85.2%, back wall 99.9%

plane spacing, fronto-parallel, 32 planes from 2.5 to 16 m
uniform in depth  : step 0.09-3.20 px | 2.5-5 m  73.5% | 5-9 m  80.0% | 9-16 m  90.2%
uniform in 1/depth: step 0.59-0.59 px | 2.5-5 m  86.8% | 5-9 m  79.8% | 9-16 m  77.5%

On the back wall, which is fronto-parallel, the two sweeps with 32 planes agree, as they must: the three-direction sweep almost always picks its frontal family there. On the ground and the facade, sweeping along the surface normal raises the share of pixels within 2% from 88% to 99% and from 70% to 91%, and cuts the median error by a factor of about 2.4 to 3.6. Tripling the fronto-parallel planes, the equal-total budget, does not close the gap and is slightly worse on both slanted surfaces, although it helps the back wall, whose 32-plane error is set by the plane step (about 15% of depth at 14 m with this spacing). This separates the two error sources. On a slanted surface no single fronto-parallel plane fits the 7 × 7 window, so at every plane the window’s cost mixes matched and mismatched pixels; in our reading, finer sampling then only locates the minimum of that biased cost more precisely. The direction labels are almost perfect on the ground and the back wall but wrong for about one facade pixel in seven, where another family happened to reach a lower cost; Gallup et al. saw the same kind of scattered label errors and cleaned them with a graph cut (see Variants).

The spacing test reproduces the argument above. Uniform steps in depth move the warped images by 3.2 pixels per plane near 2.5 m and by under a tenth of a pixel at 16 m, so the near range is undersampled and loses accuracy, while the far range is sampled more finely than the images can resolve and scores better on relative error. Uniform steps in inverse depth give the same pixel step at every depth, so the share of pixels within 2% falls with distance, as depth resolution does in stereo vision.

After the first run, the header line of the first table was shortened and the labels padded so that the columns align; no computation or number changed.

Properties and Behavior

No rectification, any number of views. Every source is warped directly into the reference view, so views in general positions, including configurations with the epipole inside the image, are handled alike; Yang and Pollefeys (2003) list handling the epipole inside the image as an advantage of their per-plane projective texture mapping over rectification.
Reference-view asymmetry. Dense plane sweeping singles out a reference image. Collins (1996) named exactly this choice as what his third condition for true multi-image matching rules out, because a feature missed or occluded in the reference cannot be recovered; his own sweep treats all images alike. Depth-map pipelines compute a depth map for each view in turn and fuse them, which in our reading restores some of the symmetry.
Discretization. Depth is quantized to the planes unless refined; the quantization step in depth grows with the square of the depth under inverse-depth spacing.
Bias toward the sweep direction. Window aggregation assumes the surface is parallel to the planes over the window. Gallup et al. (2007) explain the error on oblique surfaces this way (§3.2) and found that parabolic fitting did not remove it, because only part of the window has the correct depth (§5).
Parallel and regular. All pixels, planes and views are processed by the same warp-and-compare operation, which maps directly onto graphics hardware and, later, onto tensor operations in networks.

Limitations

Slanted surfaces under a fronto-parallel sweep, as above; ground planes seen by vehicle cameras and facades seen along a street are the standard cases.
Weak texture and repetition. A plane hypothesis can only be confirmed by texture; on blank walls the cost is flat, and on repeated patterns, such as brick, several planes fit. For textureless areas, Gallup et al. added depth priors from sparse points.
Occlusions and non-Lambertian surfaces violate photo-consistency in some views; robust aggregation or view selection is needed.
Plane families must be chosen in advance. Each additional orientation multiplies the work, and the multiple-direction variant needs the dominant normals from sparse structure, vanishing points or an inertial sensor.
Range and memory. A wide depth range at fine spacing needs many planes; storing the volume for global optimization or for a network costs memory proportional to the number of planes.

Variants

Real-time sweeping on graphics hardware

Yang and Pollefeys (2003) extended an earlier multi-baseline system of Yang, Welch and Bishop (2002), which compared single pixels and so needed several cameras to disambiguate matches. They projected two images onto each plane with projective texture mapping, compared them with summed squared differences over a multi-resolution, pyramid-shaped window to handle both depth discontinuities and low texture, and picked each pixel’s best plane by winner-take-all. Gallup et al. (2007) built on this formulation and refer to it for the details of the basic algorithm.

Multiple sweeping directions

Gallup et al. (2007) targeted city reconstruction from vehicle-mounted video, where the ground and facades are imaged at oblique angles. Rather than sampling all orientations, they sweep a few families aligned with the expected surfaces:

Find the directions. The gravity vector comes from an inertial sensor or from vertical vanishing points (vanishing points). Assuming the ground has no slope across the direction of travel, the ground normal follows from gravity and the camera motion by two cross products. Facades are assumed vertical and mutually perpendicular, so one rotation about gravity remains; they project the sparse structure-from-motion points along gravity, try rotations between 0° and 90°, histogram the rotated points along the two axes, and keep the rotation whose histograms have the lowest total entropy, where points on facades pile up into few bins.
Sweep each direction with planes spaced by the one-pixel rule above.
Select per pixel. The simplest choice is the lowest cost over all directions (best-cost). Optionally, a three-label graph cut adds a Potts penalty for changing normals between neighbours and an integrability term that extends each pixel’s plane to the midpoint between neighbours and penalizes depth differences there, so slanted surfaces are not penalized.

The facade histograms also serve as a prior on plane positions: the cost is combined with the log prior, which helps on textureless areas, and only the planes with the highest prior need to be tested. On an obliquely viewed brick wall, with 282 plane hypotheses for both methods, the standard deviation of a reconstructed scanline from its best-fit line was 1.31 cm for the fronto-parallel sweep and 0.61 cm for multiple directions (Figure 5). Testing only 48 high-prior planes per frame gave 20.0 Hz, and the graph cut took less than 2 seconds. The copy read for this article is the authors’ version on Pollefeys’s page, whose reference list still contains an unfilled entry, so details may differ from the proceedings.

Other sweep surfaces and local planes

Szeliski (2022, §12.1.2) notes that cylinders, especially with panoramic images, and other surfaces can be swept instead of planes. Gallup et al. cite a sweep over spheres by Zabulis et al., which keeps the correlation surface orthogonal to the viewing rays. Voxel coloring (Seitz and Dyer) sweeps planes front to back through a volume and removes the pixels of accepted voxels from later comparisons (Szeliski, 2022, §12.7.2). Local plane sweeps (Sinha, Scharstein and Szeliski, 2014) model the scene with a collection of slanted planes (Szeliski, 2022, §12.5), and PatchMatch methods give every pixel its own plane, as described in multi-view stereo and stereo matching. Urban modelling systems used sweeps aligned to dominant planes or to vertical facades (Szeliski, 2022, §13.6.1).

Learned plane sweeping

MVSNet (Yao et al., 2018) applies the sweep to features instead of pixels. A shared convolutional network produces 32-channel feature maps at a quarter of the input resolution; each source feature map is warped into fronto-parallel planes of the reference camera by the plane-induced homography, with differentiable bilinear sampling, which the authors describe as the classical plane sweep with features in place of images. The warped volumes are combined by their variance, regularized and read out as described in multi-view stereo and cost volume. On DTU training, the 256 depth hypotheses are sampled uniformly in depth from 425 to 935 mm at 2 mm steps. The range spans a factor of about 2.2 in depth, so, by our calculation from the argument above, its pixel steps differ by a factor of about 4.8 between the near and far ends; in the example’s 2.5–16 m sweep, uniform in depth, the steps range from 0.09 to 3.20 pixels. MVSNet also uses the probability volume as a confidence: the probability summed over the four hypotheses nearest the estimate, with pixels below 0.8 discarded before fusion. Later networks such as R-MVSNet and cascade cost volumes keep the differentiable sweep and change how the volume is regularized or sampled (multi-view stereo). Dense benchmarks such as DTU, Tanks and Temples and ETH3D score the fused reconstructions that such depth maps produce, not the sweeps themselves.