Camera Calibration

Camera calibration determines the numbers that turn a camera into a measuring instrument: its focal lengths, principal point, and lens distortion, and, for every image used, where the camera was. With them, pixels become rays, and rays from several views can be intersected to measure the world. Calibration is a routine first step for stereo, structure from motion, SLAM, augmented reality, robotics, and photogrammetry, and is usually done by photographing a flat printed pattern from many angles.

Problem Definition

Given images of a calibration object whose geometry is known, such as the corner positions of a checkerboard, estimate the parameters of a camera model that best explain where those points appear. For the standard model these are the intrinsic matrix $K$ with focal lengths $f_x, f_y$, principal point $(c_x, c_y)$, and optionally skew; the lens distortion coefficients; and one rotation $R_i$ and translation $\mathbf{t}_i$ per view, the extrinsic camera parameters. The parameters are fitted by minimizing the reprojection error

$$\sum_{i=1}^{n}\sum_{j=1}^{m} \left\lVert \mathbf{m}_{ij} - \pi\!\left(K, \mathbf{k}, R_i, \mathbf{t}_i, \mathbf{M}_j\right) \right\rVert^2,$$

where $\mathbf{m}_{ij}$ is the detected image position of target point $\mathbf{M}_j$ in view $i$, $\mathbf{k}$ the distortion coefficients, and $\pi$ the projection through the pinhole camera model followed by distortion. Under independent Gaussian detection noise this is the maximum likelihood estimate (Zhang, 2000). The intrinsics are assumed constant across views.

Input and Output

• Input: images of the target, the target’s geometry (for example, a $9 \times 6$ grid of corners with a 30 mm pitch), and the image size.
• Output: $K$, the distortion coefficients, one pose per image, and a residual summary, typically the RMS reprojection error. Some tools also report standard deviations of the estimated parameters; OpenCV’s calibrateCamera returns them on request together with per-view RMS errors.

Variants

• Single-camera intrinsic calibration, the core problem above.
• Stereo and multi-camera calibration additionally estimates the rigid transforms between cameras, from synchronized views of the same target. OpenCV’s stereoCalibrate estimates the rotation and translation between two cameras, which stereo vision needs for rectification and depth.
• Camera–IMU calibration estimates the transform between a camera and an inertial measurement unit and the time offset between their clocks. Furgale, Rehder, and Siegwart (2013) estimate both jointly with a continuous-time trajectory representation; their method is part of the open-source Kalibr toolbox, which also calibrates multi-camera rigs whose fields of view need not all overlap.
• Hand–eye calibration finds the transform $X$ between a camera and the robot gripper it is mounted on, from pairs of robot and camera motions $A_i$ and $B_i$ satisfying $A_iX = XB_i$. Tsai and Lenz (1989) gave an early efficient solution; OpenCV’s calibrateHandEye implements it alongside several later methods.

Challenges

• Coupling between parameters. Focal length trades off against target distance, and the principal point against the rotation, unless the views vary enough in orientation. Distortion coefficients trade off against focal length, especially when no points reach the image borders.
• Degenerate views. Views of a plane that differ only by translation, or by rotation about the plane’s normal, add no new constraint on the intrinsics (Zhang, 2000).
• Detection accuracy. Errors in corner localization, from blur, defocus, low resolution, or strong foreshortening, feed directly into the estimate.
• Model choice. Too few distortion terms leave systematic errors near the borders; too many fit noise and extrapolate badly outside the region covered by the target.

Approaches

3D calibration objects and the DLT

The classical approach images a precisely made 3D object, often two or three orthogonal planes with markers. With at least six non-coplanar points, the $3 \times 4$ camera matrix $P$ can be estimated linearly by the direct linear transform, then refined by minimizing geometric error and decomposed by an RQ decomposition into $K$ and $[R \mid \mathbf{t}]$ (Hartley and Zisserman, 2004, §7.1–7.2). Tsai (1987) proposed a two-stage technique aimed at metrology with off-the-shelf cameras: it first computes most of the pose from a constraint that is unaffected by radial distortion, the radial alignment constraint, and then solves for the effective focal length, the translation along the optical axis, and a radial distortion coefficient. It works with coplanar or non-coplanar points. Accurate 3D objects are expensive to build and awkward to use, which motivated planar methods.

Zhang’s planar method

Zhang (2000) showed that a planar pattern, printed and attached to a flat surface, is enough when it is seen in at least two orientations; neither the camera nor the plane motion need be known. Placing the target on $Z = 0$, each view’s homography from target to image is

$$H = [\mathbf{h}_1\;\mathbf{h}_2\;\mathbf{h}_3] = \lambda K[\mathbf{r}_1\;\mathbf{r}_2\;\mathbf{t}].$$

Because $\mathbf{r}_1$ and $\mathbf{r}_2$ are orthonormal, every homography gives two linear constraints on the symmetric matrix $B = K^{-\top}K^{-1}$:

$$\mathbf{h}_1^\top B\,\mathbf{h}_2 = 0, \qquad \mathbf{h}_1^\top B\,\mathbf{h}_1 = \mathbf{h}_2^\top B\,\mathbf{h}_2.$$

$B$ is the image of the absolute conic from projective geometry, and the constraints say that the images of the plane’s two circular points lie on it. Stacking the constraints from $n$ views gives a homogeneous $2n \times 6$ system $V\mathbf{b} = \mathbf{0}$ in the six distinct entries of $B$. Three views determine it in general; two suffice if the skew is assumed to be zero. $K$ follows from $B$ in closed form, and each view’s pose from $K^{-1}H$, with the rotation corrected to the nearest true rotation.

This closed-form estimate minimizes an algebraic quantity, so Zhang then refines all parameters, including two radial distortion coefficients, by Levenberg–Marquardt minimization of the reprojection error above. In the technical report version of the paper (MSR-TR-98-71), Zhang’s simulations showed errors growing linearly with noise, a large improvement from two to three views, and best results with the plane tilted around 45° to the image plane. This procedure, a homography-based initialization followed by nonlinear least squares, remains the basis of most calibration software. OpenCV’s calibrateCamera follows it for planar targets, initializing the intrinsics from the target’s homographies and then minimizing the reprojection error with Levenberg–Marquardt.

Self-calibration

Self-calibration does without a target. Rigid motion of a static scene constrains the intrinsics through the absolute conic: for a camera with fixed intrinsics, matched points in three or more images can in principle determine them and give a reconstruction up to similarity (Hartley and Zisserman, 2004, ch. 19). Zhang (2000) contrasted his method with self-calibration, which needs no setup but, with many parameters to estimate, does not always give reliable results. Structure from motion pipelines instead often start from approximate intrinsics and refine them in bundle adjustment.

Targets

• Checkerboards are the most common target: their corners are saddle points that can be localized to sub-pixel accuracy, and detectors such as OpenCV’s findChessboardCorners find the grid automatically. The whole board must be visible, so corners near image borders, where distortion is strongest, are hard to cover.
• Circle grids use the centers of circles; under perspective a circle’s image center is not the image of its center, a small bias that some methods correct.
• ChArUco boards combine a checkerboard with ArUco markers in its white squares, so each corner is identified individually and partially visible or occluded boards still yield usable points.
• AprilTag grids play the same role; Kalibr recommends its Aprilgrid target because partial views are usable and the board’s pose is resolved without orientation flips.

Datasets and Benchmarks

Calibration has no landmark benchmark. Methods are evaluated on synthetic data with known parameters, as in the example below, and on real images by the consistency of estimates across subsets of views; Zhang (2000), for instance, recalibrated from every four of his five real images.

Evaluation Metrics

The usual metric is the RMS reprojection error, in pixels, which OpenCV’s calibrateCamera returns. An error near the expected corner-detection noise indicates that the model fits the data. It has important pitfalls:

• It measures how well the model fits the data used to estimate it, not how accurate the parameters are. With few views, little tilt, or poor coverage, the parameters can be far off while the error stays low, as the example below shows.
• Adding distortion terms always reduces it, even when the extra terms overfit.
• It averages over all points, hiding systematic errors near the image borders.

Better checks are the parameters’ estimated standard deviations, per-view errors, the error on held-out views, stability across subsets of images, and known geometry, such as lines that must be straight after undistortion.

Practical Guidance

• Fix focus, zoom, and image resolution before calibrating, and keep them fixed afterward; the intrinsics belong to that configuration.
• Take many views with substantial tilt in different directions, not all fronto-parallel. OpenCV’s documentation (from version 4.14) likewise recommends multiple board poses with significant tilt that span the expected range of working distances.
• Cover the whole image, including the corners, across the set of views, since distortion is largest at the borders.
• Use a flat, rigid, accurately printed target and measure its square size: a warped or mis-scaled target biases the result without raising the reprojection error.

Examples

The code below simulates views of a $9 \times 6$ planar grid with a known camera and no distortion, estimates one homography per view by the normalized DLT, recovers $K$ in closed form as in Zhang’s method, and refines all parameters by minimizing reprojection error with SciPy’s Levenberg–Marquardt solver. It repeats this with exact data, with 0.5-pixel noise, with only three views, and with ten views tilted by at most 5°.

import numpy as np
from scipy.optimize import least_squares
from scipy.spatial.transform import Rotation

rng = np.random.default_rng(0)
K_true = np.array([[1200.0, 0, 655], [0, 1190, 470], [0, 0, 1]])   # 1280 x 960 image

# Planar target: 9 x 6 inner corners, 30 mm squares, on the plane Z = 0.
gx, gy = np.meshgrid(np.arange(9) * 0.03, np.arange(6) * 0.03)
M = np.c_[gx.ravel() - 0.12, gy.ravel() - 0.075]                   # centered (meters)


def project(K, rvec, t, M):
    Xc = np.c_[M, np.zeros(len(M))] @ Rotation.from_rotvec(rvec).as_matrix().T + t
    x = Xc @ K.T
    return x[:, :2] / x[:, 2:]


def make_views(n_views, max_tilt_deg, sigma):
    views = []
    for _ in range(n_views):
        axis = np.r_[rng.normal(size=2), 0.0]; axis /= np.linalg.norm(axis)
        tilt = np.deg2rad(rng.uniform(0.5, 1.0) * max_tilt_deg)
        spin = Rotation.from_euler("z", rng.uniform(-20, 20), degrees=True)
        rvec = (Rotation.from_rotvec(tilt * axis) * spin).as_rotvec()
        t = np.r_[rng.uniform(-0.05, 0.05, 2), rng.uniform(0.5, 0.8)]
        m = project(K_true, rvec, t, M) + sigma * rng.standard_normal((len(M), 2))
        views.append(m)
    return views


def homography(p, q):
    """Normalized DLT, q ~ H p."""
    def T(a):
        c = a.mean(0); s = np.sqrt(2) / np.linalg.norm(a - c, axis=1).mean()
        return np.array([[s, 0, -s * c[0]], [0, s, -s * c[1]], [0, 0, 1]])
    T1, T2 = T(p), T(q)
    a = np.c_[p, np.ones(len(p))] @ T1.T
    b = np.c_[q, np.ones(len(q))] @ T2.T
    A = []
    for (x, y, _), (u, v, _) in zip(a, b):
        A += [[0, 0, 0, -x, -y, -1, v * x, v * y, v], [x, y, 1, 0, 0, 0, -u * x, -u * y, -u]]
    H = np.linalg.svd(np.array(A))[2][-1].reshape(3, 3)
    return np.linalg.inv(T2) @ H @ T1


def zhang_closed_form(Hs):
    """Solve V b = 0 for B ~ K^-T K^-1 (image of the absolute conic), then K."""
    def v(H, i, j):
        hi, hj = H[:, i], H[:, j]
        return np.array([hi[0] * hj[0], hi[0] * hj[1] + hi[1] * hj[0], hi[1] * hj[1],
                         hi[2] * hj[0] + hi[0] * hj[2], hi[2] * hj[1] + hi[1] * hj[2],
                         hi[2] * hj[2]])
    V = np.vstack([np.vstack([v(H, 0, 1), v(H, 0, 0) - v(H, 1, 1)]) for H in Hs])
    b = np.linalg.svd(V)[2][-1]
    B = np.array([[b[0], b[1], b[3]], [b[1], b[2], b[4]], [b[3], b[4], b[5]]])
    if B[0, 0] < 0:                       # b is known only up to sign
        B = -B
    L = np.linalg.cholesky(B)             # B = L L^T, so K^-1 is proportional to L^T
    K = np.linalg.inv(L.T)
    return K / K[2, 2]


def extrinsics(K, H):
    """r1, r2, t from H ~ K [r1 r2 t], then the nearest rotation."""
    A = np.linalg.inv(K) @ H
    lam = 1 / np.linalg.norm(A[:, 0])
    if A[2, 2] < 0:                       # keep the target in front of the camera
        lam = -lam
    r1, r2, t = lam * A[:, 0], lam * A[:, 1], lam * A[:, 2]
    U, _, Wt = np.linalg.svd(np.c_[r1, r2, np.cross(r1, r2)])
    return Rotation.from_matrix(U @ Wt).as_rotvec(), t


def refine(K0, poses, views):
    """Minimize reprojection error over fx, fy, cx, cy, s and every view's pose."""
    def unpack(x):
        K = np.array([[x[0], x[4], x[2]], [0, x[1], x[3]], [0, 0, 1]])
        return K, x[5:].reshape(-1, 6)
    def residuals(x):
        K, P = unpack(x)
        return np.concatenate([(project(K, p[:3], p[3:], M) - m).ravel()
                               for p, m in zip(P, views)])
    x0 = np.r_[K0[0, 0], K0[1, 1], K0[0, 2], K0[1, 2], K0[0, 1],
               np.concatenate([np.r_[r, t] for r, t in poses])]
    sol = least_squares(residuals, x0, method="lm")
    rms = np.sqrt(np.mean(sol.fun.reshape(-1, 2) ** 2) * 2)
    return unpack(sol.x)[0], rms


def report(name, K):
    e = K - K_true
    print(f"{name:26s} fx {K[0,0]:7.1f} ({e[0,0]:+6.1f})  fy {K[1,1]:7.1f} ({e[1,1]:+6.1f})  "
          f"cx {K[0,2]:6.1f} ({e[0,2]:+5.1f})  cy {K[1,2]:6.1f} ({e[1,2]:+5.1f})  s {K[0,1]:+5.2f}")


for n_views, tilt, sigma in [(10, 40, 0.0), (10, 40, 0.5), (3, 40, 0.5), (10, 5, 0.5)]:
    views = make_views(n_views, tilt, sigma)
    Hs = [homography(M, m) for m in views]
    K0 = zhang_closed_form(Hs)
    K1, rms = refine(K0, [extrinsics(K0, H) for H in Hs], views)
    print(f"--- {n_views} views, tilt up to {tilt} deg, noise {sigma} px")
    report("closed form", K0)
    report(f"refined (RMS {rms:.2f} px)", K1)

Output:

--- 10 views, tilt up to 40 deg, noise 0.0 px
closed form                fx  1200.0 (  -0.0)  fy  1190.0 (  -0.0)  cx  655.0 ( +0.0)  cy  470.0 ( +0.0)  s -0.00
refined (RMS 0.00 px)      fx  1200.0 (  +0.0)  fy  1190.0 (  +0.0)  cx  655.0 ( -0.0)  cy  470.0 ( +0.0)  s -0.00
--- 10 views, tilt up to 40 deg, noise 0.5 px
closed form                fx  1194.8 (  -5.2)  fy  1184.4 (  -5.6)  cx  655.6 ( +0.6)  cy  473.5 ( +3.5)  s -1.21
refined (RMS 0.71 px)      fx  1198.6 (  -1.4)  fy  1188.6 (  -1.4)  cx  653.8 ( -1.2)  cy  472.8 ( +2.8)  s +0.24
--- 3 views, tilt up to 40 deg, noise 0.5 px
closed form                fx  1240.6 ( +40.6)  fy  1220.1 ( +30.1)  cx  647.7 ( -7.3)  cy  450.8 (-19.2)  s -3.29
refined (RMS 0.67 px)      fx  1232.3 ( +32.3)  fy  1213.4 ( +23.4)  cx  648.3 ( -6.7)  cy  456.0 (-14.0)  s -3.42
--- 10 views, tilt up to 5 deg, noise 0.5 px
closed form                fx  1550.3 (+350.3)  fy  1536.7 (+346.7)  cx  674.6 (+19.6)  cy  450.2 (-19.8)  s -0.06
refined (RMS 0.68 px)      fx  1074.4 (-125.6)  fy  1065.6 (-124.4)  cx  667.0 (+12.0)  cy  470.6 ( +0.6)  s +0.04

With exact data the closed form is exact. With ten well-tilted noisy views, refinement brings the focal lengths within about 1.5 pixels of the truth, about 0.1%, and the RMS reprojection error is 0.71 pixels, close to the $0.5\sqrt{2} \approx 0.71$ pixels expected for per-coordinate noise of 0.5 pixels. With three views, the focal length is off by more than 30 pixels in this run, and with nearly fronto-parallel views by more than 120 pixels, yet the RMS reprojection error is just as small: the data fit well, but they do not constrain the parameters.

Applications

Calibrated intrinsics let 3D vision work in metric units: stereo vision and its rectification, triangulation of points, relative pose from the essential matrix, structure from motion, and visual SLAM all assume them. Undistortion maps computed from the distortion coefficients are applied to every frame by image warping. Augmented reality renders virtual content with the real camera’s $K$, robot arms rely on hand–eye calibration, vehicles calibrate rigs of cameras, LiDARs, and IMUs into one frame, and photogrammetric measurement depends on calibration for its accuracy.

Open Problems

• Calibration in the field. Temperature, vibration, and impacts change the parameters over time, so systems on vehicles and robots need online or targetless recalibration that can be trusted without a target.
• Wide-angle and unusual optics. Fisheye, catadioptric, and underwater cameras need models beyond the polynomial pinhole-plus-distortion model, and choosing and validating a model remains partly manual.
• Learned single-image calibration. Networks that predict focal length, distortion, or the horizon from a single image make calibration possible for images in the wild, but are generally less accurate than target-based methods.