Reprojection Error

Reprojection error measures how well a geometric model explains what the cameras saw. Take the estimated cameras and the estimated 3D points, project the points into the images, and compare with where the points were actually detected: the distance in pixels is the reprojection error. Calibration, triangulation, camera pose estimation, and bundle adjustment all minimize it, because it measures error where the noise occurs, in the image, and because under a simple noise model its minimum is the maximum-likelihood estimate.

Definition

The reprojection error of an observation is the Euclidean distance in the image between a measured point and the projection of the estimated 3D point through the estimated camera, including its lens distortion. Summed over all observations, or turned into a root-mean-square (RMS) value, it is the objective that geometric estimation minimizes and the residual that is reported afterward.

Intuition

A detected corner or keypoint is a measurement with a small error, typically a fraction of a pixel, and a model, meaning a set of cameras and points, predicts where each measurement should be. Each prediction misses by a little, and the reprojection error collects those misses. A good fit leaves misses about as large as the detector’s noise and with no pattern; misses far larger than the noise, or misses that grow toward the image borders, signal a wrong model, a bad match, or an unmodeled effect.

Because the error is measured in the image rather than in 3D, a one-pixel miss counts the same for a point 2 m away as for one 50 m away, even though their 3D uncertainties differ greatly.

Formal Definition

Camera $i$ has the matrix $P_i = K_i[R_i \mid \mathbf{t}_i]$ (see camera parameters) and distortion coefficients $\mathbf{k}_i$. Let $\pi(P_i, \mathbf{k}_i, \mathbf{X}_j)$ be the pixel at which the 3D point $\mathbf{X}_j$ projects: multiply by $[R_i \mid \mathbf{t}_i]$, divide by depth, apply distortion, then apply $K_i$. With $\mathbf{x}_{ij}$ the measured position of point $j$ in image $i$, the residual and the per-observation reprojection error are

$$\mathbf{e}_{ij} = \mathbf{x}_{ij} - \pi(P_i, \mathbf{k}_i, \mathbf{X}_j), \qquad d_{ij} = \lVert \mathbf{e}_{ij} \rVert.$$

Over the set $\mathcal{V}$ of the $N$ observations that exist (not every point is seen in every image), the cost minimized is the sum of squares, and the usual summary is its RMS:

$$E = \sum_{(i,j) \in \mathcal{V}} \lVert \mathbf{e}_{ij} \rVert^2, \qquad \mathrm{RMS} = \sqrt{E / N}.$$

Two generalizations are common. When detection noise has a known covariance $\Sigma_{ij}$, each term becomes the squared Mahalanobis distance $\mathbf{e}_{ij}^\top \Sigma_{ij}^{-1} \mathbf{e}_{ij}$. When some observations may be outliers, the square is replaced by a robust function $\rho(\lVert \mathbf{e}_{ij} \rVert^2)$ that grows more slowly for large residuals, such as the Huber or Cauchy loss.

Properties

Maximum likelihood under Gaussian noise

Suppose each measurement is the true projection plus independent, isotropic Gaussian noise, $\mathbf{x}_{ij} = \pi(\cdot) + \mathbf{n}_{ij}$ with $\mathbf{n}_{ij} \sim \mathcal{N}(\mathbf{0}, \sigma^2 I)$. The negative log-likelihood of the parameters is then

$$-\log L = \frac{1}{2\sigma^2} \sum_{(i,j)} \lVert \mathbf{e}_{ij} \rVert^2 + \text{const},$$

so minimizing $E$ gives the maximum-likelihood estimate, by the same argument as for linear least squares. Zhang (2000) refines his closed-form calibration this way, stating the assumption as independent, identically distributed noise on the image points; Triggs et al. (2000) note that the weighted sum of squared errors equals the negative log-likelihood for Gaussian noise with covariance $W^{-1}$. Each assumption matters: correlated or anisotropic noise calls for the Mahalanobis form, and heavy-tailed errors, such as those from mismatches, make the Gaussian model wrong. Triggs et al. attribute the poor robustness of least squares entirely to that unrealistic distribution and recommend long-tailed error models instead.

Expected value at the optimum

The minimized residual is smaller than the noise that produced it, because the parameters absorb part of the noise. To first order, and exactly for linear models, with $2N$ measured coordinates and $d$ free parameters,

$$\mathbb{E}[E_{\min}] \approx \sigma^2 (2N - d), \qquad \mathrm{RMS}_{\min} \approx \sigma\sqrt{2}\,\sqrt{1 - \frac{d}{2N}}.$$

Hartley and Zisserman (2004, §5.1.3) derive this result for the maximum-likelihood estimate, stated per coordinate as $\sigma\sqrt{1 - d/2N}$ in this notation, and OpenCV’s calibrateCamera divides by $2N - d$ when it estimates $\sigma^2$ for its parameter standard deviations. For a single triangulated point seen twice, $2N = 4$ and $d = 3$, which is why the triangulation example reaches an RMS of about $\sigma/\sqrt{2}$. In bundle adjustment, $d$ excludes the gauge freedoms that the images cannot determine: the seven parameters of a 3D similarity transform for a calibrated reconstruction (Triggs et al., 2000).

Geometric versus algebraic error

Linear methods minimize an algebraic error instead: the residual $\lVert A\mathbf{p} \rVert$ of a homogeneous system, such as the direct linear transform for a homography or a camera matrix, or $\mathbf{x}'^\top F \mathbf{x}$ for the fundamental matrix. These are cheap, need no initial guess, and have closed-form solutions, but they have no direct geometric meaning and are not maximum-likelihood estimates (Hartley and Zisserman, 2004, ch. 4). For camera resectioning the relationship is explicit: the DLT row for the $x$ coordinate is $x\,\mathbf{p}^{3\top}\mathbf{X} - \mathbf{p}^{1\top}\mathbf{X} = w\,(x - \hat{x})$ with $w = \mathbf{p}^{3\top}\mathbf{X}$, so the algebraic residual is the geometric residual weighted by the point’s depth (up to the arbitrary scale of $P$), and distant points count for more (Hartley and Zisserman, 2004, §7.2.1). The standard practice is to use the algebraic solution, after normalizing coordinates, to initialize an iterative minimization of the geometric error.

For two-view relations, the reprojection error of a correspondence $\mathbf{x} \leftrightarrow \mathbf{x}'$ requires corrected points $\hat{\mathbf{x}}, \hat{\mathbf{x}}'$ that satisfy the relation exactly (for the fundamental matrix, equivalently a triangulated 3D point) and sums the squared distances in both images. That is the “gold standard” cost described in the fundamental matrix article. The Sampson error is its first-order approximation: it linearizes the constraint around the measurements and gives a closed-form distance without estimating the corrected points. It approximates the reprojection error, which it matches closely when the constraint is nearly linear over the size of the residuals, and it differs from both the one-sided transfer error of a homography and the point-to-epipolar-line distance. The homography and fundamental matrix articles give the formulas.

Reading Reported Values

A reported “reprojection error” is a number with conventions attached:

• Per point or per coordinate. OpenCV’s calibrateCamera returns “the overall RMS re-projection error”; its source divides the summed squared distances by the number of points, so it is the per-point RMS of $d_{ij}$. Dividing by the number of coordinates instead gives a value $\sqrt{2}$ smaller. With per-coordinate noise $\sigma$, the per-point RMS of the true model is $\sigma\sqrt{2}$.
• Units. Errors on distorted pixels, undistorted pixels, and normalized coordinates are not interchangeable, and downsampling an image by two halves its pixel errors.
• Training residual. The value is computed on the data used for the fit. It falls as parameters are added and is not comparable across problems with different $d/N$.
• What it can show. An RMS much larger than the detector noise points to outliers, a wrong model, or unmodeled distortion. An RMS close to it shows only that the model is consistent with the data, not that the parameters are accurate; the camera calibration article shows near-identical RMS values for good and badly constrained calibrations.

Reprojection error is therefore an objective and a diagnostic, not an accuracy metric. Accuracy is judged against ground truth, on held-out views, or by the estimated parameter uncertainties.

Examples

The NumPy and SciPy code below estimates a camera matrix from 50 known 3D points with 0.5-pixel noise. It compares the true camera, the normalized DLT (algebraic error), and Levenberg–Marquardt refinement of the reprojection error over the 11 free entries of $P$. It then averages training and held-out errors over 300 data sets, and finally adds one 50-pixel mismatch and compares a plain least-squares fit with a robust Cauchy loss.

import numpy as np
from scipy.optimize import least_squares
from scipy.spatial.transform import Rotation

rng = np.random.default_rng(1)
K = np.array([[800.0, 0, 320], [0, 800, 240], [0, 0, 1]])
R = Rotation.from_euler("xyz", [10, -20, 5], degrees=True).as_matrix()
t = np.array([0.2, -0.1, 5.0])
P_true = K @ np.c_[R, t]
sigma = 0.5                                          # pixel noise per coordinate

def project(P, X):
    x = np.c_[X, np.ones(len(X))] @ P.T
    return x[:, :2] / x[:, 2:]

def reproj(P, X, x):
    """Per-point reprojection error (pixels) and the RMS over points."""
    e = np.linalg.norm(x - project(P, X), axis=1)
    return e, np.sqrt(np.mean(e ** 2))

def dlt(X, x):
    """Camera resectioning by the normalized DLT: minimizes an algebraic error."""
    def T2(a):
        c = a.mean(0); s = np.sqrt(2) / np.linalg.norm(a - c, axis=1).mean()
        return np.array([[s, 0, -s * c[0]], [0, s, -s * c[1]], [0, 0, 1]])
    def T3(a):
        c = a.mean(0); s = np.sqrt(3) / np.linalg.norm(a - c, axis=1).mean()
        return np.r_[np.c_[s * np.eye(3), -s * c], [[0, 0, 0, 1]]]
    Tx, TX = T2(x), T3(X)
    xn = np.c_[x, np.ones(len(x))] @ Tx.T
    Xn = np.c_[X, np.ones(len(X))] @ TX.T
    A = []
    for Xi, (u, v, _) in zip(Xn, xn):
        A += [np.r_[Xi, 0 * Xi, -u * Xi], np.r_[0 * Xi, Xi, -v * Xi]]
    p = np.linalg.svd(np.array(A))[2][-1]
    P = np.linalg.inv(Tx) @ p.reshape(3, 4) @ TX
    return P / P[2, 3]

def refine(P0, X, x, loss="linear"):
    """Minimize reprojection error over the 11 free entries of P (P[2,3] = 1)."""
    def res(p):
        return (project(np.r_[p, 1.0].reshape(3, 4), X) - x).ravel()
    m = "lm" if loss == "linear" else "trf"
    s = least_squares(res, P0.ravel()[:11], method=m, loss=loss, f_scale=2 * sigma)
    return np.r_[s.x, 1.0].reshape(3, 4)

def centre(P):
    return -np.linalg.solve(P[:, :3], P[:, 3])

# One data set: 50 points in a 2 x 2 x 7 m box, about 1.3 to 8.7 m from the camera
N = 50
X = rng.uniform(-1, 1, (N, 3)) * [1, 1, 3.5]
x = project(P_true, X) + sigma * rng.standard_normal((N, 2))
P_dlt = dlt(X, x)
P_ml = refine(P_dlt, X, x)
print(f"expected RMS at the optimum: {sigma * np.sqrt(2 * (1 - 11 / (2 * N))):.3f} px")
for name, P in [("true camera", P_true), ("DLT (algebraic)", P_dlt),
                ("refined (geometric)", P_ml)]:
    e, rms = reproj(P, X, x)
    print(f"{name:20s} RMS {rms:.3f} px  max {e.max():.3f} px  "
          f"centre error {1000 * np.linalg.norm(centre(P) - centre(P_true)):5.1f} mm")

# Averages over 300 data sets: training versus held-out error, and accuracy
train, test, c_dlt, c_ml = [], [], [], []
for _ in range(300):
    X = rng.uniform(-1, 1, (2 * N, 3)) * [1, 1, 3.5]
    x = project(P_true, X) + sigma * rng.standard_normal((2 * N, 2))
    P0 = dlt(X[:N], x[:N])
    P = refine(P0, X[:N], x[:N])
    train.append(reproj(P, X[:N], x[:N])[1] ** 2)
    test.append(reproj(P, X[N:], x[N:])[1] ** 2)
    c_dlt.append(np.sum((centre(P0) - centre(P_true)) ** 2))
    c_ml.append(np.sum((centre(P) - centre(P_true)) ** 2))
print(f"mean squared RMS: training {np.mean(train):.3f}, held-out {np.mean(test):.3f},"
      f" 2 sigma^2 = {2 * sigma ** 2:.3f}")
print(f"RMS centre error: DLT {1000 * np.sqrt(np.mean(c_dlt)):.1f} mm,"
      f" refined {1000 * np.sqrt(np.mean(c_ml)):.1f} mm")

# One gross outlier: least squares versus a robust (Cauchy) loss
X = rng.uniform(-1, 1, (N, 3)) * [1, 1, 3.5]
x = project(P_true, X) + sigma * rng.standard_normal((N, 2))
x[0] += [40.0, -30.0]                                # a 50-pixel mismatch
for loss in ["linear", "cauchy"]:
    P = refine(dlt(X, x), X, x, loss)
    e, rms = reproj(P, X, x)
    print(f"{loss:7s} loss: RMS {rms:6.3f} px, median {np.median(e):.3f} px, outlier "
          f"{e[0]:5.1f} px, centre error {1000 * np.linalg.norm(centre(P) - centre(P_true)):5.1f} mm")

Output:

expected RMS at the optimum: 0.667 px
true camera          RMS 0.701 px  max 1.502 px  centre error   0.0 mm
DLT (algebraic)      RMS 0.685 px  max 1.521 px  centre error   6.6 mm
refined (geometric)  RMS 0.668 px  max 1.577 px  centre error  11.2 mm
mean squared RMS: training 0.438, held-out 0.588, 2 sigma^2 = 0.500
RMS centre error: DLT 7.8 mm, refined 5.8 mm
linear  loss: RMS  6.572 px, median 1.222 px, outlier  42.8 px, centre error  32.9 mm
cauchy  loss: RMS  7.071 px, median 0.657 px, outlier  49.7 px, centre error   3.3 mm

The refined camera fits the data better than the true camera (0.668 against 0.701 pixels, as predicted), because the fit absorbs part of the noise. In this data set it is also farther from the true camera center than the DLT estimate: lower error did not mean higher accuracy. Over 300 data sets refinement is more accurate on average (5.8 against 7.8 mm), and the training error falls below the noise level $2\sigma^2 = 0.5$ while the held-out error lies above it. With one mismatch, least squares spreads the error over all points and moves the camera center by 33 mm; the Cauchy loss isolates the outlier and stays within 3.3 mm, yet has the higher RMS. The median, not the RMS, tells the two fits apart.

Common Misconceptions

• “Lower is always better.” Adding distortion coefficients or other free parameters always lowers the training error, whether or not they improve the model. An error well below $\sigma\sqrt{2}\sqrt{1 - d/2N}$ suggests that the noise is smaller than assumed or that the model has more freedom than $d$ counts.
• “The RMS reflects typical points.” One gross outlier can dominate it. Inspect the distribution of $d_{ij}$, remove outliers with RANSAC or a robust loss, and report a median or percentile alongside the RMS.
• “Small residuals mean the distortion model is right.” An inadequate distortion model leaves residuals that grow toward the image borders and point consistently in one direction. A residual plot over the image shows this; a single average hides it.

Where It Is Used

The same cost appears throughout geometric vision, with different parameters free and fixed:

Problem	Known	Estimated by minimizing reprojection error
Camera calibration	3D target points	Intrinsics, distortion, one pose per view
Triangulation	All cameras	Each 3D point, independently
Perspective-n-Point (pose estimation)	Intrinsics, 3D points	One camera pose
Two-view refinement (fundamental matrix, homography)	Matches	The relation and the corrected points
Bundle adjustment	Matches, often intrinsics	All cameras and all 3D points jointly

All of these are nonlinear least-squares problems, usually solved by Gauss–Newton or Levenberg–Marquardt iterations from a linear or minimal-solver initialization. Structure from motion and visual SLAM run bundle adjustment repeatedly as they grow their maps, and RANSAC uses reprojection or Sampson error with a pixel threshold to decide which matches are inliers before any of these refinements run.