Epipolar Geometry

Epipolar geometry is the geometry of two views of the same scene. It depends only on the two cameras, their intrinsic parameters and their relative position and orientation, and not on what the scene contains (Hartley and Zisserman, 2004). Its central consequence is the epipolar constraint: the match of a point in one image must lie on a known line in the other image. That turns the search for correspondences from a two-dimensional problem into a one-dimensional one, and it is the basis of stereo matching, of robust verification of feature matches, and of recovering camera motion from images. This article covers the geometry; its algebraic form for uncalibrated cameras is the fundamental matrix, and for calibrated cameras the essential matrix.

Definition

Two cameras with centers $\mathbf{C}$ and $\mathbf{C}'$ observe a scene point $\mathbf{X}$ , which appears at $\mathbf{x}$ in the first image and $\mathbf{x}'$ in the second.

The baseline is the line through the two camera centers.
An epipolar plane is any plane that contains the baseline. The plane through $\mathbf{C}$ , $\mathbf{C}'$ , and $\mathbf{X}$ is the epipolar plane of $\mathbf{X}$ .
The epipoles $\mathbf{e}$ and $\mathbf{e}'$ are the points where the baseline meets the two image planes. Equivalently, $\mathbf{e}$ is the image of the second camera center in the first view, and $\mathbf{e}'$ is the image of the first camera center in the second view.
An epipolar line is the intersection of an epipolar plane with an image plane. The epipolar line $\mathbf{l}'$ of $\mathbf{x}$ is the image in the second view of the ray that back-projects from $\mathbf{x}$ .

The epipolar constraint states that $\mathbf{x}'$ lies on $\mathbf{l}'$ , and symmetrically that $\mathbf{x}$ lies on the epipolar line $\mathbf{l}$ of $\mathbf{x}'$ .

Intuition

A pixel in the first image does not say how far away its scene point is. All it fixes is a ray from the first camera center through that pixel, and the point can be anywhere along that ray. Seen from the second camera, this ray is a line in the image, starting at the image of the first camera center (the epipole) and extending toward the image of the ray’s point at infinity. Wherever the scene point actually is, its image in the second view lies somewhere on that line.

The two rays through a correct match meet at the scene point, so they lie in one plane together with the baseline. This plane is the epipolar plane. As the scene point varies, the epipolar plane rotates about the baseline like the pages of a book about its spine, and its intersections with each image sweep out a family of lines, the pencil of epipolar lines, all passing through the epipole.

Formal Definition

The notation here is shared with the fundamental matrix and essential matrix articles. Image points are homogeneous 3-vectors: $\mathbf{x}$ in the first image and $\mathbf{x}'$ in the second. The first camera defines the world frame, and the cameras are

P = K\,[\,I \mid \mathbf{0}\,], \qquad P' = K'\,[\,R \mid \mathbf{t}\,],

so that a point $\mathbf{X}$ in first-camera coordinates has coordinates $\mathbf{X}' = R\mathbf{X} + \mathbf{t}$ in the second camera’s frame (the world-to-camera convention of camera parameters). The second camera center is $\mathbf{C}' = -R^\top\mathbf{t}$ . Normalized coordinates are $\hat{\mathbf{x}} = K^{-1}\mathbf{x}$ and $\hat{\mathbf{x}}' = K'^{-1}\mathbf{x}'$ . $[\mathbf{a}]_\times$ is the skew-symmetric matrix with $[\mathbf{a}]_\times \mathbf{b} = \mathbf{a} \times \mathbf{b}$ .

The epipoles are the images of the other camera’s center:

\mathbf{e} = P \begin{bmatrix} \mathbf{C}' \\ 1 \end{bmatrix} = -K R^\top \mathbf{t}, \qquad \mathbf{e}' = P' \begin{bmatrix} \mathbf{0} \\ 1 \end{bmatrix} = K'\mathbf{t},

each defined up to scale. The epipolar line of $\mathbf{x}$ in the second image joins $\mathbf{e}'$ to the image of any other point on the back-projected ray, for example the point at infinity in direction $K^{-1}\mathbf{x}$ , whose image is $K'RK^{-1}\mathbf{x}$ :

\mathbf{l}' = \mathbf{e}' \times \left(K' R K^{-1} \mathbf{x}\right).

In homogeneous coordinates a point lies on a line when their dot product vanishes, so the epipolar constraint is

\mathbf{x}'^\top \mathbf{l}' = 0.

Since $\mathbf{l}'$ depends linearly on $\mathbf{x}$ , the constraint can be written as one bilinear equation, $\mathbf{x}'^\top F \mathbf{x} = 0$ , with a $3 \times 3$ matrix $F$ : the fundamental matrix, with $\mathbf{l}' = F\mathbf{x}$ , $\mathbf{l} = F^\top\mathbf{x}'$ , and $F = K'^{-\top} E K^{-1}$ . In normalized coordinates the same constraint reads $\hat{\mathbf{x}}'^\top E \hat{\mathbf{x}} = 0$ with the essential matrix $E = [\mathbf{t}]_\times R$ , which expresses directly that the two rays and the baseline $\mathbf{t}$ are coplanar. Longuet-Higgins (1981) introduced this algebraic form for calibrated cameras to computer vision.

Properties

All epipolar lines pass through the epipole. Every epipolar plane contains the baseline, so every epipolar line contains the baseline’s intersection with the image. Algebraically, $\mathbf{e}'^\top \mathbf{l}' = 0$ for every $\mathbf{x}$ .
Epipolar lines come in corresponding pairs. One epipolar plane cuts both images, so each epipolar line in one image has exactly one partner in the other, and all points on $\mathbf{l}$ map to points on $\mathbf{l}'$ . Hartley and Zisserman (2004, §9.2.5) show that the two pencils of lines are related by a one-dimensional projective transformation.
The constraint is necessary, not sufficient. A point anywhere on $\mathbf{l}'$ satisfies it, so it rejects many wrong matches but cannot certify a right one. It also says nothing about a point whose true match is occluded.
The epipole may be anywhere. It lies inside the image when the other camera center projects into the image, as under forward motion, outside it for typical side-by-side configurations, and at infinity when the baseline is parallel to the image plane.
No baseline, no epipolar geometry. If the two centers coincide, as for a camera rotating about its center, there is no baseline and no epipolar plane. The views are then related by a homography instead, which maps each point to a single point rather than a line.

Special configurations

Two motions give epipolar geometry a particularly simple form, both described by Hartley and Zisserman (2004, §9.1, §9.3).

Parallel cameras. When the second camera is shifted sideways along the first camera’s $x$ -axis, with no rotation and equal intrinsics, the baseline is parallel to both image planes. The epipoles are at infinity in the $x$ direction, and the epipolar lines are horizontal and identical in both images: the match of a pixel lies in the same row of the other image. This is the configuration that stereo rectification creates artificially.

Forward motion. When a camera translates without rotating and its intrinsics stay the same, the epipole has the same coordinates in both images. As the camera advances, image points move away from the epipole along radial lines, so the epipole is the focus of expansion, the point from which the motion field appears to radiate. If the motion is along the optical axis, it is at the principal point.

Relation to homographies

When all scene points lie on one plane, the plane induces a homography $H$ with $\mathbf{x}' = H\mathbf{x}$ , and the epipolar line of $\mathbf{x}$ is the line through $\mathbf{e}'$ and $H\mathbf{x}$ . Hartley and Zisserman (2004, §9.2) use this construction to derive the fundamental matrix as $F = [\mathbf{e}']_\times H$ . The reverse does not work: from points on a single plane, or from a purely rotating camera, the epipolar geometry cannot be recovered, because the correspondences satisfy a homography and are consistent with many different epipolar geometries. These are the main degenerate cases for estimating the fundamental matrix.

Examples

The NumPy code below sets up two pinhole cameras and builds epipolar lines geometrically, without any fundamental matrix: it back-projects each pixel of the first image to a ray, images two points of that ray in the second camera (the first camera center, which gives the epipole, and a point at unit depth), and joins them. It then checks that each true match lies on its line and that all lines pass through the epipole, for a general motion, a sideways shift, and a forward motion.

import numpy as np
from scipy.spatial.transform import Rotation

np.set_printoptions(precision=3, suppress=True)
rng = np.random.default_rng(0)

K = np.array([[800.0, 0, 320], [0, 800.0, 240], [0, 0, 1]])

def camera(R, t):
    """P = K [R | t]; camera 1 is P = K [I | 0]."""
    return K @ np.hstack([R, t[:, None]])

def project(P, X):
    x = np.hstack([X, np.ones((len(X), 1))]) @ P.T
    return x / x[:, 2:]

def epipolar_lines(P1, P2, x1):
    """Epipolar lines in image 2, built geometrically: back-project each x1
    to a ray, image two points of the ray (the camera-1 centre and a point
    at depth 1) in camera 2, and join them with a cross product."""
    C1 = np.array([0.0, 0, 0, 1])                       # centre of camera 1
    e2 = P2 @ C1                                        # its image: the epipole e'
    d = np.linalg.solve(K, x1.T).T                      # ray directions (depth 1)
    p2 = np.hstack([d, np.ones((len(d), 1))]) @ P2.T    # image of a ray point
    return e2, np.cross(e2, p2)                         # line through e' and it

def dist(lines, x):
    return np.abs(np.sum(lines * x, 1)) / np.hypot(lines[:, 0], lines[:, 1])

X = rng.uniform([-2, -2, 4], [2, 2, 8], size=(6, 3))   # scene points
P1 = camera(np.eye(3), np.zeros(3))

setups = {
    "general":  (Rotation.from_euler("y", -10, degrees=True).as_matrix(),
                 np.array([-1.0, 0.1, 0.2])),
    "parallel": (np.eye(3), np.array([-0.5, 0.0, 0.0])),  # sideways shift
    "forward":  (np.eye(3), np.array([0.0, 0.0, -1.0])),  # move along the axis
}
for name, (R, t) in setups.items():
    P2 = camera(R, t)
    x1, x2 = project(P1, X), project(P2, X)
    e2, l2 = epipolar_lines(P1, P2, x1)
    if abs(e2[2]) > 1e-12:
        print(f"{name}: epipole e' at pixel {e2[:2] / e2[2]}")
    else:
        print(f"{name}: epipole e' at infinity in direction {e2[:2] / np.linalg.norm(e2[:2])}")
    print("  max distance of x' to its epipolar line (px):", f"{dist(l2, x2).max():.1e}")
    print("  max |l' . e'| (lines pass through e', algebraic):", f"{np.abs(l2 @ e2).max():.1e}")
    if name == "parallel":
        print("  max |a| / |b| over lines ax + by + c = 0:", f"{np.max(np.abs(l2[:, 0] / l2[:, 1])):.1e}")

Output:

general: epipole e' at pixel [-3680.   640.]
  max distance of x' to its epipolar line (px): 7.4e-14
  max |l' . e'| (lines pass through e', algebraic): 6.7e-12
parallel: epipole e' at infinity in direction [-1.  0.]
  max distance of x' to its epipolar line (px): 9.1e-15
  max |l' . e'| (lines pass through e', algebraic): 0.0e+00
  max |a| / |b| over lines ax + by + c = 0: 0.0e+00
forward: epipole e' at pixel [320. 240.]
  max distance of x' to its epipolar line (px): 1.4e-13
  max |l' . e'| (lines pass through e', algebraic): 1.5e-11

Every match lies on its epipolar line to rounding error. In the general case the epipole is far outside the 640 × 480 image, as is typical when the cameras are side by side. The sideways shift puts the epipole at infinity and makes every epipolar line horizontal ( $a = 0$ ), and moving forward along the optical axis puts the epipole at the principal point $(320, 240)$ , the focus of expansion.

Common Misconceptions

“The epipolar line tells you where the match is.” It tells you where the match can be. Finding it along the line still requires comparing image content, which is the job of stereo matching or descriptor matching.
“Epipoles are inside the image.” Often they are not. For a nearly parallel stereo pair they are far outside or at infinity, which is why rectification algorithms must handle epipoles at infinity.
“Epipolar lines are horizontal.” Only after rectification, or for cameras that happen to be exactly parallel. In general they converge at the epipole.
“Any two views have an epipolar geometry that can be estimated.” Two views from the same center have none, and views of a single plane do not determine it, even though their point correspondences are perfectly consistent.
“The epipolar geometry depends on the scene.” It depends only on the cameras. The scene supplies the correspondences from which it is estimated, but a different scene seen by the same two cameras has the same epipoles and epipolar lines.

Where It Is Used

Stereo matching. In stereo vision, the two images are usually rectified first, warped by a homography each so that epipolar lines become corresponding image rows (Hartley and Zisserman, 2004, §11.12). Stereo matching then searches only along a row, and the offset of the match is the disparity, from which depth follows by triangulation.
Outlier rejection in feature matching. In feature matching, putative matches between two views of a 3D scene are verified by estimating the fundamental or essential matrix with RANSAC and discarding matches that lie far from their epipolar lines. Guided matching then searches for further matches in a narrow band around each epipolar line.
Relative pose and structure from motion. For calibrated cameras the essential matrix yields the rotation and the direction of translation between two views, the starting point of two-view reconstruction, visual odometry, and structure-from-motion pipelines.
Motion analysis. For a translating camera, the epipole is the focus of expansion, which gives the direction of travel and constrains the optical flow to radial lines.