Essential Matrix

The essential matrix $E$ is the form that epipolar geometry takes when both cameras are calibrated. Writing image points in normalized coordinates, with the intrinsic matrices removed, the epipolar constraint becomes $\hat{\mathbf{x}}'^\top E\,\hat{\mathbf{x}} = 0$, and $E = [\mathbf{t}]_\times R$ is built directly from the rotation $R$ and translation $\mathbf{t}$ between the two cameras. Unlike the fundamental matrix, which determines the cameras only up to a projective transformation, $E$ yields the relative pose itself: the rotation exactly and the translation up to its length. Estimating $E$ from point matches and decomposing it is the standard way to compute the motion between two images in visual odometry and structure from motion.

Definition

Notation, shared with the epipolar geometry and fundamental matrix articles: $\mathbf{x}$ and $\mathbf{x}'$ are corresponding homogeneous pixel coordinates in the first and second image, and $\hat{\mathbf{x}} = K^{-1}\mathbf{x}$, $\hat{\mathbf{x}}' = K'^{-1}\mathbf{x}'$ are the corresponding normalized coordinates, using the intrinsic matrices $K$ and $K'$. The cameras are $P = K[\,I \mid \mathbf{0}\,]$ and $P' = K'[\,R \mid \mathbf{t}\,]$, so a point $\mathbf{X}$ in first-camera coordinates has coordinates $R\mathbf{X} + \mathbf{t}$ in the second camera’s frame. $[\mathbf{t}]_\times$ is the cross-product matrix, $[\mathbf{t}]_\times\mathbf{v} = \mathbf{t} \times \mathbf{v}$, and $F$ is the fundamental matrix with $\mathbf{x}'^\top F\mathbf{x} = 0$.

The essential matrix of the pair is

$$E = [\mathbf{t}]_\times R,$$

and corresponding points satisfy $\hat{\mathbf{x}}'^\top E\,\hat{\mathbf{x}} = 0$. It is related to the fundamental matrix by

$$F = K'^{-\top} E\, K^{-1}, \qquad E = K'^\top F K$$

(Hartley and Zisserman, 2004, §9.6). The first form follows by substituting $\hat{\mathbf{x}} = K^{-1}\mathbf{x}$ into the constraint.

Intuition

A normalized image point $\hat{\mathbf{x}}$ is the direction of the ray from the camera center through the pixel, expressed in that camera’s frame. For a correct match, three vectors lie in one plane, the epipolar plane: the ray $\hat{\mathbf{x}}'$ from the second camera, the ray from the first camera rotated into the second frame, $R\hat{\mathbf{x}}$, and the baseline $\mathbf{t}$ joining the two centers. Three vectors are coplanar exactly when their triple product vanishes:

$$\hat{\mathbf{x}}' \cdot \big(\mathbf{t} \times R\,\hat{\mathbf{x}}\big) = \hat{\mathbf{x}}'^\top [\mathbf{t}]_\times R\, \hat{\mathbf{x}} = 0.$$

That is the essential matrix constraint. It involves only directions, so it cannot reveal how long the baseline is: doubling $\mathbf{t}$ and every scene depth produces exactly the same images. Two views therefore give the direction of translation but not its magnitude.

Formal Definition

Properties

• Rank 2. $\det E = 0$, since $[\mathbf{t}]_\times$ has rank 2. The null vectors $E\hat{\mathbf{e}} = \mathbf{0}$ and $E^\top \hat{\mathbf{e}}' = \mathbf{0}$ are the epipoles in normalized coordinates, $\hat{\mathbf{e}} \propto R^\top\mathbf{t}$ and $\hat{\mathbf{e}}' \propto \mathbf{t}$.
• Equal singular values. A $3 \times 3$ matrix is an essential matrix if and only if two of its singular values are equal and the third is zero (Hartley and Zisserman, 2004, Result 9.17). For $E = [\mathbf{t}]_\times R$, both nonzero singular values equal $\|\mathbf{t}\|$. Equivalently, a nonzero $E$ is essential exactly when it satisfies the cubic constraint that Nistér (2003, 2004) built his solver on:

$$E E^\top E - \tfrac{1}{2}\operatorname{tr}\!\left(E E^\top\right) E = 0 .$$

• Five degrees of freedom. $R$ and $\mathbf{t}$ have three each, minus one for the overall scale, which leaves five, against seven for the fundamental matrix. The two extra constraints are the equality of the singular values.
• Scale ambiguity. $E$ is homogeneous: $E$ and $\lambda E$ give the same constraint, so $\|\mathbf{t}\|$, and the scale of the reconstructed scene, cannot be recovered from two views. It is usually fixed by setting $\|\mathbf{t}\| = 1$ and resolved later from a known distance, another sensor, or further views. The sign of $\mathbf{t}$ is also ambiguous at this stage, and is fixed during pose recovery.

Estimation

Correspondences are first converted to normalized coordinates with the known intrinsics, which usually come from camera calibration.

• Linear eight-point. The eight-point algorithm of the fundamental matrix, applied to normalized coordinates, gives a $3 \times 3$ matrix that satisfies the constraint in a least-squares sense but is not an essential matrix. With its singular value decomposition $U\,\mathrm{diag}(\sigma_1, \sigma_2, \sigma_3)\,V^\top$, it is projected onto the essential matrices by replacing the singular values with $(1, 1, 0)$. This is the closest essential matrix in Frobenius norm, up to scale. Longuet-Higgins (1981) introduced the linear method for this calibrated case.
• Five-point. Since $E$ has five degrees of freedom, five correspondences determine it up to finitely many solutions. Nistér (2003, 2004) gave an efficient algorithm: the five linear constraints leave a four-dimensional null space, and substituting it into the cubic constraints and $\det E = 0$ leads to a polynomial of degree ten, whose roots give up to ten real solutions. Nistér noted that Kruppa had proved there are at most eleven solutions, and that later work reduced the bound to ten. He designed the solver for use in a RANSAC hypothesize-and-test loop, scoring each solution separately, and demonstrated real-time structure and motion with it.
• Robust estimation. Matches from feature matching contain outliers, so $E$ is estimated with RANSAC, using the five-point solver on minimal samples of five. The smaller sample matters: at 50% inliers and 99% confidence, RANSAC needs about 146 samples of five, against 588 of seven and 1,177 of eight (see the table in the RANSAC article). The final estimate is refined on all inliers, usually by minimizing a geometric error and, in larger systems, by bundle adjustment.

Calibration also changes the behavior on planar scenes. For uncalibrated cameras, coplanar points leave a continuum of fundamental matrices. Nistér (2003) points out that in the calibrated setting a plane causes at most a two-fold ambiguity, which a third view generally resolves, so the five-point method keeps working on planar and near-planar scenes where the fundamental matrix breaks down.

Pose Recovery

Given $E$ with singular value decomposition $E = U\,\mathrm{diag}(1, 1, 0)\,V^\top$, with $U$ and $V$ chosen so that $\det U = \det V = +1$, and

$$W = \begin{bmatrix} 0 & -1 & 0 \\ 1 & 0 & 0 \\ 0 & 0 & 1 \end{bmatrix},$$

there are exactly four candidate poses for the second camera, $P' = [\,R \mid \mathbf{t}\,]$ with

$$R \in \{\,U W V^\top,\; U W^\top V^\top\,\}, \qquad \mathbf{t} = \pm \mathbf{u}_3,$$

where $\mathbf{u}_3$ is the last column of $U$ (Hartley and Zisserman, 2004, Result 9.19). The two signs of $\mathbf{t}$ reverse the baseline. The two rotations differ by a rotation of $180°$ about the baseline, a configuration Hartley and Zisserman call a twisted pair.

Only one of the four places a reconstructed point in front of both cameras. Choosing it is the cheirality test: triangulate a correspondence with each candidate and keep the one that gives the point positive depth in both cameras. One point suffices in theory; with noise and outliers, implementations triangulate all inliers and keep the candidate that puts the most points in front of both cameras.

Examples

The NumPy code below builds $E = [\mathbf{t}]_\times R$ from a known pose, decomposes it into the four candidates, selects one by cheirality over 20 points, and reports the rotation error and the angle between the recovered and true translation directions. It then repeats the recovery from noisy normalized coordinates, equivalent to 0.5 pixels of noise at a focal length of 800 pixels, using the linear eight-point estimate projected onto the essential matrices.

import numpy as np
from scipy.spatial.transform import Rotation

np.set_printoptions(precision=3, suppress=True)
rng = np.random.default_rng(2)

def skew(v):
    return np.array([[0, -v[2], v[1]], [v[2], 0, -v[0]], [-v[1], v[0], 0]])

# Ground truth relative pose: X' = R X + t (camera 1 at the origin)
axis = np.array([0.2, 1.0, 0.1]) / np.linalg.norm([0.2, 1.0, 0.1])
R_true = Rotation.from_rotvec(np.radians(15) * axis).as_matrix()
t_true = np.array([-0.8, 0.05, 0.3])
E = skew(t_true) @ R_true                        # E = [t]x R
print("singular values of E:", np.linalg.svd(E)[1])

# Normalized image coordinates x_hat = K^-1 x of 20 points seen by both cameras
X = rng.uniform([-2, -2, 4], [2, 2, 10], size=(20, 3))
Xc2 = X @ R_true.T + t_true
x1, x2 = X / X[:, 2:], Xc2 / Xc2[:, 2:]
print("max |x2^T E x1|:", f"{np.abs(np.sum(x2 * (x1 @ E.T), 1)).max():.1e}")

def decompose(E):
    """Four (R, t) candidates from the SVD of E (Hartley & Zisserman, Result 9.19)."""
    U, _, Vt = np.linalg.svd(E)
    if np.linalg.det(U) < 0: U = -U              # keep R a proper rotation
    if np.linalg.det(Vt) < 0: Vt = -Vt
    W = np.array([[0, -1, 0], [1, 0, 0], [0, 0, 1]])
    u3 = U[:, 2]                                 # unit-norm t, sign unknown
    return [(U @ W @ Vt, u3), (U @ W @ Vt, -u3),
            (U @ W.T @ Vt, u3), (U @ W.T @ Vt, -u3)]

def triangulate(R, t, a, b):
    """Linear (DLT) triangulation of one point with P = [I|0], P' = [R|t]."""
    P1, P2 = np.hstack([np.eye(3), np.zeros((3, 1))]), np.hstack([R, t[:, None]])
    A = np.array([a[0] * P1[2] - P1[0], a[1] * P1[2] - P1[1],
                  b[0] * P2[2] - P2[0], b[1] * P2[2] - P2[1]])
    Xh = np.linalg.svd(A)[2][-1]
    return Xh[:3] / Xh[3]

def n_in_front(R, t, x1, x2):
    """Cheirality: count points with positive depth in both cameras."""
    Xs = np.array([triangulate(R, t, a, b) for a, b in zip(x1, x2)])
    return np.sum((Xs[:, 2] > 0) & ((Xs @ R.T + t)[:, 2] > 0))

def recover_pose(E, x1, x2):
    return max(decompose(E), key=lambda c: n_in_front(*c, x1, x2))

def errors(R, t):
    rot = np.degrees(np.linalg.norm(Rotation.from_matrix(R.T @ R_true).as_rotvec()))
    cos = t @ t_true / np.linalg.norm(t) / np.linalg.norm(t_true)
    return rot, np.degrees(np.arccos(np.clip(cos, -1, 1)))

for i, (R, t) in enumerate(decompose(E)):
    print(f"candidate {i}: points in front of both cameras = {n_in_front(R, t, x1, x2):2d}/20")
R, t = recover_pose(E, x1, x2)
print("exact E: rotation error %.1e deg, translation direction error %.1e deg" % errors(R, t))
print("recovered t (unit norm):", t, " true t / |t|:", t_true / np.linalg.norm(t_true))

# The same from noisy matches: linear eight-point on normalized coordinates,
# then projection onto the essential matrices (singular values 1, 1, 0)
f = 800.0                                        # 0.5 px of noise at f = 800 px
x1n = x1 + np.hstack([rng.normal(0, 0.5 / f, (20, 2)), np.zeros((20, 1))])
x2n = x2 + np.hstack([rng.normal(0, 0.5 / f, (20, 2)), np.zeros((20, 1))])
A = np.einsum("ni,nj->nij", x2n, x1n).reshape(20, 9)
U, S, Vt = np.linalg.svd(np.linalg.svd(A)[2][-1].reshape(3, 3))
print("singular values before projection:", S / S[0])
E_est = U @ np.diag([1.0, 1.0, 0.0]) @ Vt
R, t = recover_pose(E_est, x1n, x2n)
print("noisy E: rotation error %.3f deg, translation direction error %.3f deg" % errors(R, t))

Output:

singular values of E: [0.856 0.856 0.   ]
max |x2^T E x1|: 1.1e-16
candidate 0: points in front of both cameras =  0/20
candidate 1: points in front of both cameras = 20/20
candidate 2: points in front of both cameras =  0/20
candidate 3: points in front of both cameras =  0/20
exact E: rotation error 1.0e-14 deg, translation direction error 0.0e+00 deg
recovered t (unit norm): [-0.935  0.058  0.351]  true t / |t|: [-0.935  0.058  0.351]
singular values before projection: [1.    0.987 0.024]
noisy E: rotation error 1.017 deg, translation direction error 3.761 deg

The two nonzero singular values of $E$ are equal to $\|\mathbf{t}\| = 0.856$. Exactly one of the four candidates puts all 20 points in front of both cameras, and it recovers the rotation and the unit translation direction to rounding error; the length of $\mathbf{t}$ is lost. With noise, the linear estimate is not an essential matrix (its singular values are not $(1, 1, 0)$) until it is projected, and the recovered pose is off by about 1° in rotation and about 4° in translation direction. Here the translation direction is the less accurate of the two; a RANSAC estimate refined by nonlinear optimization would reduce both errors.

Common Misconceptions

• “The essential matrix gives the translation.” It gives the direction of translation only. The baseline length, and the scale of the scene, need extra information.
• “Any rank-2 matrix estimated from normalized coordinates is an essential matrix.” It must also have two equal nonzero singular values. A linear estimate must be projected, or the five-point solver used, which enforces the constraints exactly.
• “Decomposing $E$ gives the pose.” It gives four candidates, three of them wrong. Without the cheirality test, the chosen pose may reverse the baseline, so that the scene lies behind the cameras, or twist the second camera by $180°$ about the baseline.
• “Pixel coordinates can go straight into $E$.” The essential matrix applies to normalized coordinates. With pixel coordinates, use $E = K'^\top F K$ or normalize first, and correct lens distortion before either.
• “Pure rotation is easy for $E$.” With $\mathbf{t} = \mathbf{0}$, $E$ vanishes and gives no information. Small baselines make the translation direction poorly determined even when the rotation is accurate.

History

Longuet-Higgins (1981) introduced the essential matrix to computer vision in a short paper in Nature, showing that eight point correspondences between two calibrated views give a linear equation system for a matrix built from the rotation and translation, from which the relative pose and the scene structure follow. Hartley and Zisserman (2004, §9.7.1) note that an analogous matrix had appeared earlier in the photogrammetry literature, and that the extension to uncalibrated cameras, the fundamental matrix, followed in the early 1990s. The paper itself is covered in its own article. Nistér’s five-point solver (2003, 2004) made the minimal calibrated case efficient enough for real-time use.

Where It Is Used

• Visual odometry. Monocular systems estimate the motion between consecutive frames from the essential matrix of tracked or matched features, with the scale fixed by other means such as a known camera height or an inertial sensor.
• Structure from motion. Incremental pipelines initialize the reconstruction from a pair of images: estimate $E$ with RANSAC, recover the pose, and triangulate the inlier matches. Further cameras are then added by pose estimation from the reconstructed points, and everything is refined by bundle adjustment.
• SLAM initialization. Some monocular SLAM systems bootstrap their map from the relative pose of two frames, choosing between a homography and the epipolar model depending on the scene.
• Stereo rig calibration. For a calibrated stereo pair, the rotation and translation between the two cameras define the rig’s essential matrix $[\mathbf{t}]_\times R$, and the same $R$ and $\mathbf{t}$ are used to compute the rectification for stereo vision.
• Geometric verification of matches between calibrated images, with feature matching and RANSAC.