Stereo Vision

Stereo vision recovers the depth of a scene from two images taken at the same time by two cameras a known distance apart. A point closer to the cameras shifts more between the two views than a distant one, so measuring that shift at every pixel turns a pair of images into a depth map. It is the computational counterpart of human binocular vision and one of the oldest routes to 3D measurement in computer vision.

Definition

Stereo vision is the estimation of 3D structure from two views whose intrinsic parameters and relative pose are known. The known geometry reduces the search for each point’s match to a line, its epipolar line, and turns each match into a 3D point by triangulation. In the standard rectified configuration, the match lies on the same image row, and its horizontal offset, the disparity, is inversely proportional to depth.

Stereo vision therefore splits into two problems: the geometry, which this article covers, and the correspondence problem of finding the disparity at every pixel, covered under stereo matching.

Intuition

Hold a finger in front of your face and close each eye in turn. The finger jumps sideways against the background, and it jumps further the closer it is. The two eyes see the scene from positions a few centimeters apart, and the size of the jump encodes distance.

Two cameras side by side do the same: a distant mountain appears at nearly the same pixel in both images, a nearby car noticeably further left in the right image. Once matching points share an image row, depth estimation becomes a search along each row followed by a single division.

Formal Definition

The rectified stereo rig

Consider two identical pinhole cameras with focal length $f$ in pixels and principal point $(c_x, c_y)$, whose image planes are coplanar and whose image rows are parallel to the line joining the camera centers. Place the left camera center at the origin and the right one at $(B, 0, 0)$, where $B$ is the baseline. A point $(X, Y, Z)$ in left-camera coordinates projects to

$$x_L = f\,\frac{X}{Z} + c_x, \qquad x_R = f\,\frac{X - B}{Z} + c_x, \qquad y_L = y_R = f\,\frac{Y}{Z} + c_y .$$

The two image points share a row, so the epipolar lines are horizontal. Their horizontal offset is the disparity

$$d = x_L - x_R = \frac{fB}{Z},$$

and inverting it gives depth from disparity:

$$Z = \frac{fB}{d}.$$

The other two coordinates follow from the left image position, $X = (x_L - c_x)\,Z/f$ and $Y = (y_L - c_y)\,Z/f$. Disparity is positive for points in front of the rig, tends to zero as $Z \to \infty$, and has units of pixels, while $fB$ has units of pixels times meters.

If rectification leaves the two principal points at different columns, $c_{x,L} \neq c_{x,R}$, the relation gains an offset: $Z = fB / (d - (c_{x,L} - c_{x,R}))$. The Middlebury 2014 data, for example, ships each scene with this offset, called doffs, alongside $f$ and $B$.

Depth resolution

Differentiating $Z = fB/d$ gives

$$\frac{\partial Z}{\partial d} = -\frac{fB}{d^2} = -\frac{Z^2}{fB}, \qquad\text{so}\qquad \Delta Z \approx \frac{Z^2}{fB}\,\Delta d .$$

A fixed disparity error $\Delta d$, set by image noise and the matcher’s sub-pixel accuracy, produces a depth error that grows with the square of depth and shrinks only linearly with the baseline and the focal length. The relative error $\Delta Z / Z = Z\,\Delta d/(fB) = \Delta d / d$ grows linearly with depth. Because $Z$ is a reciprocal of $d$, the error is also asymmetric: underestimating disparity overestimates depth by more than the same overestimate of disparity underestimates it. The triangulation article treats the general, non-rectified case.

Properties

The baseline trade-off

A longer baseline increases every disparity in proportion and so reduces depth error at every range. It also has costs:

• More occlusion. More surface is visible to one camera but hidden from the other, so more pixels have no match at all.
• Harder matching. Wider viewpoints change the appearance of surfaces more (foreshortening, specular highlights, different backgrounds at object edges).
• Larger search range. The maximum disparity, set by the nearest object, grows with $B$, so the matcher has more chances to choose a wrong candidate, and near objects may leave one camera’s field of view.

Okutomi and Kanade (1993) described this as a trade-off between precision, which favors long baselines, and freedom from matching ambiguity, which favors short ones. Their multiple-baseline method expresses each pair’s matching cost as a function of inverse distance, which is shared across baselines, and sums the costs, so that short baselines rule out false matches while long ones contribute precision. Practical rigs choose $B$ for their working range: a few centimeters for handheld devices, tens of centimeters on vehicles (the KITTI rig uses about 54 cm), and much longer bases in aerial mapping.

Rectification

Real cameras are rarely mounted with perfectly coplanar, row-aligned image planes. Rectification resamples each image by a homography chosen so that conjugate epipolar lines become the same horizontal row in both images; geometrically, both epipoles are mapped to the same point at infinity in the horizontal direction. A correspondence search then becomes a scan along one row.

The rectifying homographies are not unique, and different choices distort the images differently (Hartley and Zisserman, 2004, §11.12):

• Calibrated rectification. When both cameras’ projection matrices are known, the images can be reprojected onto a common image plane parallel to the baseline by virtually rotating each camera about its center. Fusiello, Trucco, and Verri (2000) gave a compact linear algorithm that computes the two rectifying projection matrices from the original ones.
• Uncalibrated rectification. Only the epipolar geometry, in the form of the fundamental matrix estimated from point matches, is needed to align epipolar lines. Hartley (1999) derived a matched pair of projective transformations from the fundamental matrix: one sends the epipole to infinity while staying close to a rigid motion near a chosen image point, and the other minimizes the horizontal disparities of matched points. Loop and Zhang (1999) decomposed each rectifying homography into a projective, a similarity, and a shearing component and chose them to minimize distortion. Uncalibrated rectification aligns rows but does not by itself give metric depth; converting disparity to distance still needs $f$ and $B$.

Planar rectification breaks down when an epipole lies inside or near the image, as in a camera moving forward: mapping it to infinity would stretch part of the image without bound. Hartley (1999) suggests shrinking the image window in that case, and Pollefeys, Koch, and Van Gool (1999) instead resample each image in polar coordinates around its epipole, which works for any camera motion. In calibrated stereo rigs the problem does not arise, because the cameras look roughly perpendicular to the baseline. Camera calibration and the removal of lens distortion are usually done first, since curved epipolar lines cannot be straightened by a homography.

Other properties

• Depth sampling is non-uniform. Integer disparities correspond to depth planes spaced as $Z^2/(fB)$, dense near the cameras and sparse far away. This is why disparity, or inverse depth, is the natural variable for matching and for building a cost volume.
• Metric scale. Because the baseline is a physical length, stereo depth has absolute scale, unlike depth from a single moving camera with unknown motion.

Examples

Human binocular vision

Human eyes are about 6 to 7 cm apart and see slightly different images, and the differences in the horizontal positions of corresponding points, binocular disparities, are one of the cues the visual system uses for depth. Charles Wheatstone demonstrated this in 1838 with his stereoscope, which presented a different flat drawing to each eye and produced an impression of relief. Julesz (1960) used computer-generated random-dot stereograms, in which neither image alone contains any visible shape, to show that a hidden figure appears in depth when the two images are fused. The experiment showed that the correspondence problem can be solved without first recognizing objects or any cue available to one eye alone. With a baseline this short, the depth-resolution formula implies that disparity discriminates depth best at close range.

Active stereo and structured light

Passive stereo fails where surfaces have no texture to match, such as white walls. Active stereo adds a projector that casts a pattern, often an infrared dot pattern, onto the scene, and the two cameras match the projected texture with ordinary stereo algorithms. Structured light projects coded patterns, such as stripe sequences, so that each pixel can be labeled with a code; depth then follows either from triangulating against the projector or, as for the ground truth of the Middlebury benchmark, from matching the decoded labels between two cameras.

Stereo, monocular depth, and multi-view stereo

A single image does not determine depth geometrically: every point along a pixel’s ray projects to the same place. Monocular depth estimation therefore relies on learned priors about scene layout, object sizes, and shading, and its outputs are often defined only up to scale. Stereo replaces these priors with a measurement, giving metric depth wherever matching succeeds.

Multi-view stereo extends the idea to many calibrated images. It tests depth hypotheses along each reference pixel’s ray, for example by plane sweeping (see cost volume), and checks consistency across all views, gaining more baselines and fewer occlusion failures than a single pair.

Practical Example

The code computes depth from disparity for a rig with a 700-pixel focal length and a 12 cm baseline, then shows how a quarter-pixel disparity error, a typical sub-pixel accuracy, affects depth at different ranges.

import numpy as np

f = 700.0   # focal length in pixels (rectified images)
B = 0.12    # baseline in meters

def depth_from_disparity(d, f, B):
    """Z = f B / d; non-positive disparities have no finite depth."""
    d = np.asarray(d, dtype=float)
    Z = np.full_like(d, np.inf)
    valid = d > 0
    Z[valid] = f * B / d[valid]
    return Z

# Scene points at known depths and their exact disparities.
Z_true = np.array([0.5, 1.0, 2.0, 5.0, 10.0, 20.0, 40.0])
d_true = f * B / Z_true

dd = 0.25   # disparity error of a quarter pixel
Z_far = depth_from_disparity(d_true - dd, f, B)   # disparity too small -> too far
Z_near = depth_from_disparity(d_true + dd, f, B)  # disparity too large -> too near
dZ_lin = Z_true**2 / (f * B) * dd                 # first-order prediction

print(f"f = {f:.0f} px, B = {B} m, fB = {f * B:.0f} px*m, disparity error = {dd} px")
print("   Z (m)   d (px)   Z(d+0.25)   Z(d-0.25)   Z^2/(fB)*dd   rel. error")
for Z, d, zn, zf, lin in zip(Z_true, d_true, Z_near, Z_far, dZ_lin):
    print(f"{Z:8.1f} {d:8.2f} {zn:11.3f} {zf:11.3f} {lin:13.3f} {lin / Z:11.1%}")

# Wider baseline: same depths, five times the disparity, one fifth of the error.
B2 = 0.60
print(f"\nB = {B2} m: depth error at 20 m = {20.0**2 / (f * B2) * dd:.3f} m, "
      f"disparity at 0.5 m = {f * B2 / 0.5:.0f} px")

Output:

f = 700 px, B = 0.12 m, fB = 84 px*m, disparity error = 0.25 px
   Z (m)   d (px)   Z(d+0.25)   Z(d-0.25)   Z^2/(fB)*dd   rel. error
     0.5   168.00       0.499       0.501         0.001        0.1%
     1.0    84.00       0.997       1.003         0.003        0.3%
     2.0    42.00       1.988       2.012         0.012        0.6%
     5.0    16.80       4.927       5.076         0.074        1.5%
    10.0     8.40       9.711      10.307         0.298        3.0%
    20.0     4.20      18.876      21.266         1.190        6.0%
    40.0     2.10      35.745      45.405         4.762       11.9%

B = 0.6 m: depth error at 20 m = 0.238 m, disparity at 0.5 m = 840 px

The same quarter-pixel error costs about a millimeter at half a meter and several meters at 40 m: each doubling of depth quadruples the error, as $Z^2/(fB)$ predicts. At long range the error also becomes visibly asymmetric: a smaller disparity pushes the estimate out by 5.4 m while a larger one pulls it in by 4.3 m. A five-times-wider baseline cuts the error at 20 m to under 25 cm, but an object at half a meter would then have a disparity of 840 pixels, more than the width of many images, so the near range is lost.

Common Misconceptions

• “Stereo depth has uniform accuracy.” Disparity error is roughly constant in pixels, so depth error grows with the square of distance.
• “Disparity is depth.” Disparity is inversely proportional to depth and depends on the rig. Converting it needs the focal length, the baseline, and any principal-point offset left by rectification.
• “A longer baseline is always better.” It improves precision but increases occlusions, appearance changes, and the search range, and it pushes the minimum working distance outward.
• “Rectification is only a convenience.” Dense stereo algorithms and their cost volumes assume the 1D row search it creates. Without accurate calibration and rectification, true matches drift off the searched row.

Where It Is Used

• Robotics, for obstacle detection, grasping, and navigation with metric depth from passive cameras.
• Driver assistance and autonomous driving, where forward-facing stereo cameras estimate the distance of vehicles, pedestrians, and the road, as in the KITTI benchmark.
• Augmented and virtual reality, for depth sensing on headsets and for rendering a separate image for each eye.
• Aerial and satellite mapping, where overlapping images produce digital surface models.
• 3D reconstruction, through multi-view stereo, and depth cameras based on active stereo or structured light.