Multi-View Geometry

Multi-view geometry studies how cameras, points in the scene, and their images constrain one another when the same scene is photographed more than once. It asks where the cameras were, where the scene points are, and which image measurements can belong together, and how much of this images alone can determine. This article maps the subject by number of views and links to the articles that develop each relation. The trifocal tensor and the reconstruction ambiguity hierarchy, which have no articles of their own, are covered here in more detail.

Definition

A scene point $\mathbf{X}_j$ seen by camera $P_i$ appears at the image point $\mathbf{x}_{ij} \simeq P_i \mathbf{X}_j$, where $\simeq$ means equality up to a non-zero scale. Multi-view geometry is the study of this equation over many points and views. Depending on what is known, it covers three problems:

• Structure: the cameras are known and the points are wanted. This is triangulation.
• Motion: the points are known and a camera is wanted. This is camera resectioning, or pose estimation when the intrinsics are known.
• Structure and motion: only image correspondences are known, and both cameras and points are wanted.

It also covers the multiple-view constraints, relations among the images alone that do not depend on the scene.

Intuition

Each image point back-projects to a ray. A correspondence across views is a claim that the rays meet at a single scene point. With one view nothing constrains the depth along the ray. With two views, the requirement that two rays meet is a single condition, so a point in one image restricts its match to a line in the other: the epipolar geometry. With three views, a point matched in two of them fixes the 3D point, so its position in the third view is determined. Lines also become useful, because the three planes back-projected from three image lines must meet in a common 3D line, a condition that two planes always satisfy.

Each added view brings more measurements per point for a fixed number of new camera unknowns. Beyond four views no new multiple-view tensor appears, and the practical tool changes from closed-form relations to a large optimization over all cameras and points.

Formal Definition

The notation follows the epipolar geometry article: $P = K[\,I \mid \mathbf{0}\,]$ and $P' = K'[\,R \mid \mathbf{t}\,]$ for two views, with a third camera $P''$ and image points $\mathbf{x} \leftrightarrow \mathbf{x}' \leftrightarrow \mathbf{x}''$ when there are three. The table below shows the ladder of relations. Minimal counts are point correspondences. Those for the camera matrix, $F$, and the two tensors come from Hartley and Zisserman (2004, §7.1 and Table 17.3); the others are developed in the linked articles.

Views	Constraint	Object (entries / DOF)	Minimal correspondences	Article
1	$\mathbf{x} \simeq P\mathbf{X}$ for known $\mathbf{X}$	camera matrix $P$ (12 / 11)	5½ points (6 in practice); 3 for a calibrated pose	camera parameters, camera calibration
2	$\mathbf{x}' \simeq H\mathbf{x}$, for a plane or pure rotation	homography $H$ (9 / 8)	4	homography
2	$\mathbf{x}'^\top F\mathbf{x} = 0$	fundamental matrix $F$ (9 / 7)	7 (up to 3 solutions); 8 linear	fundamental matrix
2	$\hat{\mathbf{x}}'^\top E\hat{\mathbf{x}} = 0$	essential matrix $E$ (9 / 5)	5 (up to 10 solutions)	essential matrix, five-point algorithm
3	trilinear incidences	trifocal tensor (27 / 18)	6 (1 or 3 solutions); 7 linear	this article
4	quadrilinear incidences	quadrifocal tensor (81 / 29)	6	this article
$m$	reprojection error over all views	cameras and points jointly	$n \ge (11m - 15)/(2m - 3)$	reprojection error

One view

A camera matrix has 11 degrees of freedom and each 3D–2D correspondence gives two equations, so 5½ correspondences determine it (Hartley and Zisserman, 2004, §7.1). With known intrinsics only the six pose parameters remain, and three points give at most four solutions (Fischler and Bolles, 1981). The general calibrated case is the perspective-n-point (PnP) problem.

Two views

Views of a plane, or from a camera rotating about its center, are related point to point by a homography. Otherwise the relation is point to line, expressed by the fundamental matrix in pixels and the essential matrix in normalized coordinates $\hat{\mathbf{x}} = K^{-1}\mathbf{x}$, with $F = K'^{-\top} E K^{-1}$.

Three views: the trifocal tensor

The trifocal tensor plays the role for three views that the fundamental matrix plays for two: it captures the projective relations between the views that do not depend on the scene (Hartley and Zisserman, 2004, ch. 15). Choose the projective frame so that the cameras are $P = [\,I \mid \mathbf{0}\,]$, $P' = [\,A \mid \mathbf{a}_4\,]$, and $P'' = [\,B \mid \mathbf{b}_4\,]$, with $\mathbf{a}_i$ and $\mathbf{b}_i$ the columns of $P'$ and $P''$. The tensor is then three $3 \times 3$ matrices:

$$T_i = \mathbf{a}_i \mathbf{b}_4^\top - \mathbf{a}_4 \mathbf{b}_i^\top, \qquad i = 1, 2, 3.$$

For corresponding lines $\mathbf{l} \leftrightarrow \mathbf{l}' \leftrightarrow \mathbf{l}''$, the line in the first view follows from the other two as $l_i = \mathbf{l}'^\top T_i\, \mathbf{l}''$. If a scene point projects to $\mathbf{x}$, and $\mathbf{l}'$ and $\mathbf{l}''$ are any lines through its images $\mathbf{x}'$ and $\mathbf{x}''$, the point–line–line incidence holds:

$$\mathbf{l}'^\top \Big( \sum_{i=1}^{3} x^i\, T_i \Big)\, \mathbf{l}'' = 0,$$

where $x^i$ is the $i$-th coordinate of $\mathbf{x}$. A point correspondence across three views gives four linearly independent equations of this kind (§16.2), so seven points determine the 27 entries linearly. Six points are the minimum, with one or three real solutions (§16.1).

The 27 entries are defined only up to scale, but the tensor has just $18 = 3 \times 11 - 15$ degrees of freedom: three camera matrices minus a projective transformation of space. The entries must therefore satisfy 8 internal constraints (§17.5). Given $\mathbf{x}$ and any line $\mathbf{l}'$ through $\mathbf{x}'$, the tensor transfers the point into the third view as $x''^k = \sum_{i,j} x^i\, l'_j\, T_i^{jk}$, where $T_i^{jk}$ is entry $(j, k)$ of $T_i$ (§15.3). Transfer by intersecting two epipolar lines fails when the scene point lies on the plane through the three camera centers. Transfer with the trifocal tensor works for such points. It fails only when $\mathbf{l}'$ is the epipolar line of $\mathbf{x}$, or for points on the baseline of the first two views.

Four and more views

The quadrifocal tensor relates four views. It has 81 entries, 29 degrees of freedom, and 51 internal constraints (§17.5). Hartley and Zisserman (2004, §1.5) note that it is seldom used because it is hard to estimate a tensor that obeys those constraints, and that the tensor approach does not extend beyond four views. For many views, practice minimizes the reprojection error over all cameras and points together: bundle adjustment, the maximum-likelihood estimate under Gaussian image noise, which tolerates points missing from some views but needs a good start (§18.1). The start comes from two- and three-view relations chained along a sequence with resectioning and triangulation (§18.6) or, when every point is seen in every view, from factorization (§18.2, §18.4).

Properties

The reconstruction ambiguity hierarchy

Images determine a reconstruction only up to a transformation of space. Replacing every camera $P_i$ by $P_i H^{-1}$ and every point $\mathbf{X}_j$ by $H\mathbf{X}_j$ leaves all the images unchanged. What is known about the cameras or the scene decides which transformations $H$ remain possible. The 3D transformation groups are those of projective geometry.

Reconstruction	Remaining ambiguity (DOF)	Information that reaches this level
Projective	$4 \times 4$ homography (15)	Point correspondences alone (projective reconstruction theorem, §10.3)
Affine	affine transformation (12)	The plane at infinity: vanishing points of parallel lines, a purely translating camera, or affine cameras (§10.4.1, §17.5.1)
Metric (similarity)	rotation, translation, and scale (7)	The absolute conic: known intrinsics, or auto-calibration from constraints such as constant intrinsics (§10.4.2, ch. 19)
Euclidean	rigid motion, that is, the choice of world frame (6)	A known length or baseline fixes the scale (§10.2); ground control points also fix the world frame (§10.5)

The projective reconstruction theorem states that if two views determine $F$ uniquely, any two reconstructions from those correspondences differ by a projective transformation, except for points on the baseline (Hartley and Zisserman, 2004, Theorem 10.1). Auto-calibration starts from a projective reconstruction and finds the transformation that makes the cameras consistent with the assumed intrinsics. Given enough views and general enough motion, the result is metric (§19.1). Faugeras, Luong and Papadopoulo (2001) organize their treatment the same way, with separate chapters on stratifying two and $n \ge 3$ views and on self-calibration. The literature often says “Euclidean” when it means metric (§10.2); true scale always needs an outside measurement.

Counting degrees of freedom

$m$ uncalibrated cameras have $11m - 15$ degrees of freedom once the projective ambiguity is removed. For two cameras this is 7, the degrees of freedom of $F$, and for three it is 18, those of the trifocal tensor (§17.5). Adding $n$ points gives $11m - 15 + 3n$ unknowns against $2mn$ measured coordinates. Reconstruction therefore requires

$$n \;\ge\; \frac{11m - 15}{2m - 3} \;=\; 5 + \frac{m}{2m - 3},$$

that is, 7 points in two views and 6 in three or more. Each point adds $2m - 3$ constraints: two coordinates per view, minus three for the new point. For affine cameras the degrees of freedom drop to $8m - 12$, and four points suffice for any number of views (§17.5.1).

Factorization

When every point is visible in every view, factorization recovers all cameras and points at once. Tomasi and Kanade (1992) stacked the coordinates of $n$ points tracked through $m$ frames into a $2m \times n$ measurement matrix. After subtracting each frame’s centroid, this matrix has rank at most 3 under orthographic projection, and the singular value decomposition factors it into motion and shape. Hartley and Zisserman (2004, §18.2) present the method for general affine cameras and note that it gives the maximum-likelihood affine reconstruction under isotropic Gaussian noise. Sturm and Triggs (1996) extended the idea to perspective cameras. Rescaling each image point by its projective depth $\lambda_{ij}$, in $\lambda_{ij}\mathbf{x}_{ij} = P_i\mathbf{X}_j$, gives a matrix of rank 4 (§18.4), and they recovered the depths from fundamental matrices and epipoles.

Examples

The code below builds the trifocal tensor of three synthetic cameras from the formula above. It draws random lines through the second and third images of each point, checks the point–line–line incidence as a distance in pixels, and transfers each point into the third view. It also measures the tensor’s degrees of freedom as the rank of its Jacobian with respect to the camera entries, and checks that cameras $P_i H$ give the same tensor.

import numpy as np
from scipy.spatial.transform import Rotation

rng = np.random.default_rng(1)
K = np.array([[800.0, 0, 320], [0, 800.0, 240], [0, 0, 1]])

def camera(rotvec, t):
    R = Rotation.from_rotvec(rotvec).as_matrix()
    return K @ np.hstack([R, np.reshape(t, (3, 1))])

def canonical(Ps):
    """Change the projective frame so that the first camera is [I | 0]."""
    H = np.linalg.inv(np.vstack([Ps[0], [0, 0, 0, 1]]))
    return [P @ H for P in Ps]

def trifocal(P1, P2, P3):
    """T_i = a_i b4^T - a4 b_i^T for P1 = [I | 0], P2 = [A | a4], P3 = [B | b4]."""
    assert np.allclose(P1, np.hstack([np.eye(3), np.zeros((3, 1))]))
    return np.array([np.outer(P2[:, i], P3[:, 3]) - np.outer(P2[:, 3], P3[:, i])
                     for i in range(3)])

def project(P, X):
    x = P @ np.append(X, 1.0)
    return x / x[2]

def line_through(x):
    """A random image line through the homogeneous point x."""
    return np.cross(x, x + np.append(rng.normal(size=2), 0.0))

cams = [camera([0, 0, 0], [0, 0, 0]),
        camera([0.02, -0.15, 0.01], [-1.0, 0.1, 0.2]),
        camera([-0.03, 0.12, -0.02], [0.8, -0.2, 0.3])]
T = trifocal(*canonical(cams))
print("entries:", T.size)

for X in rng.uniform([-1, -1, 4], [1, 1, 6], size=(4, 3)):
    x, x2, x3 = (project(P, X) for P in cams)
    l2, l3 = line_through(x2), line_through(x3)
    # Line transfer l_i = l'^T T_i l'': the image in view 1 of the 3D line
    # where the planes back-projected from l' and l'' meet. Point-line-line
    # incidence says that x lies on it.
    l = np.einsum("j,ijk,k->i", l2, T, l3)
    d = abs(l @ x) / np.hypot(l[0], l[1])
    # Point transfer x''^k = x^i l'_j T_i^{jk}, for a line l' through x'
    x3t = np.einsum("i,j,ijk->k", x, l2, T)
    err = np.linalg.norm(x3t[:2] / x3t[2] - x3[:2])
    print(f"x to transferred line: {d:.1e} px   transferred x'' error: {err:.1e} px")

# Degrees of freedom: rank of the Jacobian of T with respect to the 24 entries
# of P2 and P3 (P1 = [I | 0] fixed), minus 1 for the overall scale of T.
P1, P2, P3 = canonical(cams)
p0 = np.concatenate([P2.ravel(), P3.ravel()])
T_of = lambda p: trifocal(P1, p[:12].reshape(3, 4), p[12:].reshape(3, 4)).ravel()
J = np.column_stack([(T_of(p0 + 1e-6 * e) - T_of(p0 - 1e-6 * e)) / 2e-6
                     for e in np.eye(24)])
s = np.linalg.svd(J, compute_uv=False)
rank = int(np.sum(s > 1e-8 * s[0]))
print("Jacobian rank:", rank, "-> degrees of freedom:", rank - 1)

# Projective ambiguity: cameras P_i H give the same tensor for any 4x4 H.
H = rng.normal(size=(4, 4))
a = T.ravel() / np.linalg.norm(T)
b = trifocal(*canonical([P @ H for P in cams])).ravel()
b /= np.linalg.norm(b)
print("same tensor (up to scale) after P -> PH:", np.allclose(a, np.sign(a @ b) * b))

Output:

entries: 27
x to transferred line: 1.6e-11 px   transferred x'' error: 2.4e-12 px
x to transferred line: 7.8e-12 px   transferred x'' error: 1.8e-12 px
x to transferred line: 2.1e-12 px   transferred x'' error: 1.5e-12 px
x to transferred line: 1.1e-11 px   transferred x'' error: 2.0e-13 px
Jacobian rank: 19 -> degrees of freedom: 18
same tensor (up to scale) after P -> PH: True

The incidence and the transfer hold to rounding error. The Jacobian has rank 19, which is 18 degrees of freedom plus the overall scale, although the tensor has 27 entries and is built from 24 camera entries. The camera entries overcount because some changes leave the tensor unchanged apart from its scale: rescaling $P'$ or $P''$, and any transformation $H$ that keeps the first camera in the form $[\,I \mid \mathbf{0}\,]$. The last check shows the projective ambiguity: a tensor estimated from images gives back the cameras only up to such an $H$.

Common Misconceptions

• “Three views are just three pairs of views.” Three fundamental matrices have 21 degrees of freedom against the 18 of three cameras, so they must satisfy three compatibility conditions. They also fail to fix the cameras when the three centers are collinear, a case the trifocal tensor handles (Hartley and Zisserman, 2004, §15.4). Only three or more views constrain lines.
• “More views require higher-order tensors.” The sequence stops at the quadrifocal tensor. Many-view reconstruction is an optimization, not a larger closed-form relation.
• “A projective reconstruction is useless.” Intersections and plane-induced mappings can be computed from it, and it is the usual first step toward metric reconstruction (§10.3).
• “Minimal solvers are what estimate the geometry.” Minimal sets feed RANSAC. The final estimate comes from all the inliers, usually by minimizing reprojection error with nonlinear least squares.

Where It Is Used

• Structure from motion reconstructs cameras and sparse points from image collections. It chains two-view relative pose, triangulation, and camera resectioning, and refines everything with bundle adjustment.
• Multi-view stereo takes the calibrated cameras from structure from motion and computes dense depth or surfaces. Its two-view special case is stereo vision with stereo rectification.
• Visual odometry and SLAM estimate camera motion over a video stream with the same relations: the essential matrix or PnP for each new frame, triangulation of new map points, and often bundle adjustment over recent keyframes.
• Camera calibration is resectioning with a known target, and auto-calibration is the step from projective to metric reconstruction.
• Feature matching relies on these constraints for geometric verification and guided matching.