Geometry & 3D Vision
Camera Calibration
Estimating a camera's intrinsic parameters, lens distortion, and pose from images of a target with known geometry, usually by a closed-form initialization followed by minimizing reprojection error.
intermediate
Camera calibration determines the numbers that turn a camera into a measuring instrument: its focal lengths, principal point, and lens distortion, and, for every image used, where the camera was. With them, pixels become rays, and rays from several views can be intersected to measure the world. Calibration is a routine first step for stereo, structure from motion, SLAM, augmented reality, robotics, and photogrammetry, and is usually done by photographing a flat printed pattern from many angles.
Problem Definition
Given images of a calibration object whose geometry is known, such as the corner positions of a checkerboard, estimate the parameters of a camera model that best explain where those points appear. For the standard model these are the intrinsic matrix with focal lengths , principal point , and optionally skew; the lens distortion coefficients; and one rotation and translation per view, the extrinsic camera parameters. The parameters are fitted by minimizing the reprojection error
where is the detected image position of target point in view , the distortion coefficients, and the projection through the pinhole camera model followed by distortion. Under independent Gaussian detection noise this is the maximum likelihood estimate (Zhang, 2000). The intrinsics are assumed constant across views.
Input and Output
- Input: images of the target, the target’s geometry (for example, a grid of corners with a 30 mm pitch), and the image size.
- Output: , the distortion coefficients, one pose per image, and a residual summary, typically the RMS reprojection error. Some tools also report standard deviations of the estimated parameters; OpenCV’s
calibrateCamerareturns them on request together with per-view RMS errors.
Variants
- Single-camera intrinsic calibration, the core problem above.
- Stereo and multi-camera calibration additionally estimates the rigid transforms between cameras, from synchronized views of the same target. OpenCV’s
stereoCalibrateestimates the rotation and translation between two cameras, which stereo vision needs for rectification and depth. - Camera–IMU calibration estimates the transform between a camera and an inertial measurement unit and the time offset between their clocks. Furgale, Rehder, and Siegwart (2013) estimate both jointly with a continuous-time trajectory representation; their method is part of the open-source Kalibr toolbox, which also calibrates multi-camera rigs whose fields of view need not all overlap.
- Hand–eye calibration finds the transform between a camera and the robot gripper it is mounted on, from pairs of robot and camera motions and satisfying . Tsai and Lenz (1989) gave an early efficient solution; OpenCV’s
calibrateHandEyeimplements it alongside several later methods. In OpenCV 5.0.0 the function is declared for C++ only, so the Python binding does not include it.
Challenges
- Coupling between parameters. Focal length trades off against target distance, and the principal point against the rotation, unless the views vary enough in orientation. Distortion coefficients trade off against focal length, especially when no points reach the image borders.
- Degenerate views. Views of a plane that differ only by translation, or by rotation about the plane’s normal, add no new constraint on the intrinsics (Zhang, 1999; proved in the technical report MSR-TR-98-71, §4).
- Detection accuracy. Errors in corner localization, from blur, defocus, low resolution, or strong foreshortening, feed directly into the estimate.
- Model choice. Too few distortion terms leave systematic errors near the borders; too many fit noise and extrapolate badly outside the region covered by the target.
Approaches
3D calibration objects and the DLT
The classical approach images a precisely made 3D object, often two or three orthogonal planes with markers. With at least six non-coplanar points, the camera matrix can be estimated linearly by the direct linear transform, then refined by minimizing geometric error and decomposed by an RQ decomposition into and (Hartley and Zisserman, 2004, §7.1–7.2; the decomposition is in §6.2.4). Tsai (1987) proposed a two-stage technique aimed at metrology with off-the-shelf cameras: it first computes most of the pose from a constraint that is unaffected by radial distortion, the radial alignment constraint, and then solves for the effective focal length, the translation along the optical axis, and a radial distortion coefficient. It works with coplanar or non-coplanar points. Accurate 3D objects are expensive to build and awkward to use, which motivated planar methods.
Zhang’s planar method
Zhang (2000) showed that a planar pattern, printed and attached to a flat surface, is enough when it is seen in at least two orientations; neither the camera nor the plane motion need be known. Placing the target on , each view’s homography from target to image is
Because and are orthonormal, every homography gives two linear constraints on the symmetric matrix :
is the image of the absolute conic from projective geometry, and the constraints say that the images of the plane’s two circular points lie on it. Stacking the constraints from views gives a homogeneous system in the six distinct entries of . Three views determine it in general; two suffice if the skew is assumed to be zero. follows from in closed form, and each view’s pose from , with the rotation corrected to the nearest true rotation.
This closed-form estimate minimizes an algebraic quantity, so Zhang then refines all parameters, including two radial distortion coefficients, by Levenberg–Marquardt minimization of the reprojection error above. In the technical report version of the paper (MSR-TR-98-71), Zhang’s simulations showed errors growing linearly with noise, a large improvement from two to three views, and best results with the plane tilted around 45° to the image plane. This procedure, a homography-based initialization followed by nonlinear least squares, remains the basis of most calibration software. OpenCV’s calibrateCamera follows it for planar targets, initializing the intrinsics from the target’s homographies and then minimizing the reprojection error with Levenberg–Marquardt.
Self-calibration
Self-calibration does without a target. Rigid motion of a static scene constrains the intrinsics through the absolute conic: for a camera with fixed intrinsics, matched points in three or more images can in principle determine them and give a reconstruction up to similarity (Hartley and Zisserman, 2004, ch. 19). Zhang (2000) contrasted his method with self-calibration, which needs no setup but, with many parameters to estimate, does not always give reliable results. Structure from motion pipelines instead often start from approximate intrinsics and refine them in bundle adjustment.
Targets
- Checkerboards are the most common target: their corners are saddle points that can be localized to sub-pixel accuracy, and detectors such as OpenCV’s
findChessboardCornersfind the grid automatically. The whole board must be visible, so corners near image borders, where distortion is strongest, are hard to cover. - Circle grids use the centers of circles; under perspective a circle’s image center is not the image of its center, a small bias that some methods correct.
- ChArUco boards combine a checkerboard with ArUco markers in its white squares, so each corner is identified individually and partially visible or occluded boards still yield usable points.
- AprilTag grids play the same role; Kalibr recommends its Aprilgrid target because partial views are usable and the board’s pose is resolved without orientation flips.
Datasets and Benchmarks
Calibration has no landmark benchmark. Methods are evaluated on synthetic data with known parameters, as in the example below, and on real images by the consistency of estimates across subsets of views; Zhang’s 1999 conference paper, for instance, recalibrated from every four of its five real images.
Evaluation Metrics
The usual metric is the RMS reprojection error, in pixels, which OpenCV’s calibrateCamera returns. An error near the expected corner-detection noise indicates that the model fits the data. It has important pitfalls:
- It measures how well the model fits the data used to estimate it, not how accurate the parameters are. With few views, little tilt, or poor coverage, the parameters can be far off while the error stays low, as the example below shows.
- Adding distortion terms always reduces it, even when the extra terms overfit.
- It averages over all points, hiding systematic errors near the image borders.
Better checks are the parameters’ estimated standard deviations, per-view errors, the error on held-out views, stability across subsets of images, and known geometry, such as lines that must be straight after undistortion.
Practical Guidance
- Fix focus, zoom, and image resolution before calibrating, and keep them fixed afterward; the intrinsics belong to that configuration.
- Take many views with substantial tilt in different directions, not all fronto-parallel. OpenCV’s documentation (from version 4.14) likewise recommends multiple board poses with significant tilt that span the expected range of working distances.
- Cover the whole image, including the corners, across the set of views, since distortion is largest at the borders.
- Use a flat, rigid, accurately printed target and measure its square size: a warped or mis-scaled target biases the result without raising the reprojection error.
Examples
The code below simulates views of a planar grid with a known camera and no distortion, estimates one homography per view by the normalized DLT, recovers in closed form as in Zhang’s method, and refines all parameters by minimizing reprojection error with SciPy’s Levenberg–Marquardt solver. It repeats this with exact data, with 0.5-pixel noise, with only three views, and with ten views tilted by at most 5°.
import numpy as np
from scipy.optimize import least_squares
from scipy.spatial.transform import Rotation
rng = np.random.default_rng(0)
K_true = np.array([[1200.0, 0, 655], [0, 1190, 470], [0, 0, 1]]) # 1280 x 960 image
# Planar target: 9 x 6 inner corners, 30 mm squares, on the plane Z = 0.
gx, gy = np.meshgrid(np.arange(9) * 0.03, np.arange(6) * 0.03)
M = np.c_[gx.ravel() - 0.12, gy.ravel() - 0.075] # centered (meters)
def project(K, rvec, t, M):
Xc = np.c_[M, np.zeros(len(M))] @ Rotation.from_rotvec(rvec).as_matrix().T + t
x = Xc @ K.T
return x[:, :2] / x[:, 2:]
def make_views(n_views, max_tilt_deg, sigma):
views = []
for _ in range(n_views):
axis = np.r_[rng.normal(size=2), 0.0]; axis /= np.linalg.norm(axis)
tilt = np.deg2rad(rng.uniform(0.5, 1.0) * max_tilt_deg)
spin = Rotation.from_euler("z", rng.uniform(-20, 20), degrees=True)
rvec = (Rotation.from_rotvec(tilt * axis) * spin).as_rotvec()
t = np.r_[rng.uniform(-0.05, 0.05, 2), rng.uniform(0.5, 0.8)]
m = project(K_true, rvec, t, M) + sigma * rng.standard_normal((len(M), 2))
views.append(m)
return views
def homography(p, q):
"""Normalized DLT, q ~ H p."""
def T(a):
c = a.mean(0); s = np.sqrt(2) / np.linalg.norm(a - c, axis=1).mean()
return np.array([[s, 0, -s * c[0]], [0, s, -s * c[1]], [0, 0, 1]])
T1, T2 = T(p), T(q)
a = np.c_[p, np.ones(len(p))] @ T1.T
b = np.c_[q, np.ones(len(q))] @ T2.T
A = []
for (x, y, _), (u, v, _) in zip(a, b):
A += [[0, 0, 0, -x, -y, -1, v * x, v * y, v], [x, y, 1, 0, 0, 0, -u * x, -u * y, -u]]
H = np.linalg.svd(np.array(A))[2][-1].reshape(3, 3)
return np.linalg.inv(T2) @ H @ T1
def zhang_closed_form(Hs):
"""Solve V b = 0 for B ~ K^-T K^-1 (image of the absolute conic), then K."""
def v(H, i, j):
hi, hj = H[:, i], H[:, j]
return np.array([hi[0] * hj[0], hi[0] * hj[1] + hi[1] * hj[0], hi[1] * hj[1],
hi[2] * hj[0] + hi[0] * hj[2], hi[2] * hj[1] + hi[1] * hj[2],
hi[2] * hj[2]])
V = np.vstack([np.vstack([v(H, 0, 1), v(H, 0, 0) - v(H, 1, 1)]) for H in Hs])
b = np.linalg.svd(V)[2][-1]
B = np.array([[b[0], b[1], b[3]], [b[1], b[2], b[4]], [b[3], b[4], b[5]]])
if B[0, 0] < 0: # b is known only up to sign
B = -B
L = np.linalg.cholesky(B) # B = L L^T, so K^-1 is proportional to L^T
K = np.linalg.inv(L.T)
return K / K[2, 2]
def extrinsics(K, H):
"""r1, r2, t from H ~ K [r1 r2 t], then the nearest rotation."""
A = np.linalg.inv(K) @ H
lam = 1 / np.linalg.norm(A[:, 0])
if A[2, 2] < 0: # keep the target in front of the camera
lam = -lam
r1, r2, t = lam * A[:, 0], lam * A[:, 1], lam * A[:, 2]
U, _, Wt = np.linalg.svd(np.c_[r1, r2, np.cross(r1, r2)])
return Rotation.from_matrix(U @ Wt).as_rotvec(), t
def refine(K0, poses, views):
"""Minimize reprojection error over fx, fy, cx, cy, s and every view's pose."""
def unpack(x):
K = np.array([[x[0], x[4], x[2]], [0, x[1], x[3]], [0, 0, 1]])
return K, x[5:].reshape(-1, 6)
def residuals(x):
K, P = unpack(x)
return np.concatenate([(project(K, p[:3], p[3:], M) - m).ravel()
for p, m in zip(P, views)])
x0 = np.r_[K0[0, 0], K0[1, 1], K0[0, 2], K0[1, 2], K0[0, 1],
np.concatenate([np.r_[r, t] for r, t in poses])]
sol = least_squares(residuals, x0, method="lm")
rms = np.sqrt(np.mean(sol.fun.reshape(-1, 2) ** 2) * 2)
return unpack(sol.x)[0], rms
def report(name, K):
e = K - K_true
print(f"{name:26s} fx {K[0,0]:7.1f} ({e[0,0]:+6.1f}) fy {K[1,1]:7.1f} ({e[1,1]:+6.1f}) "
f"cx {K[0,2]:6.1f} ({e[0,2]:+5.1f}) cy {K[1,2]:6.1f} ({e[1,2]:+5.1f}) s {K[0,1]:+5.2f}")
for n_views, tilt, sigma in [(10, 40, 0.0), (10, 40, 0.5), (3, 40, 0.5), (10, 5, 0.5)]:
views = make_views(n_views, tilt, sigma)
Hs = [homography(M, m) for m in views]
K0 = zhang_closed_form(Hs)
K1, rms = refine(K0, [extrinsics(K0, H) for H in Hs], views)
print(f"--- {n_views} views, tilt up to {tilt} deg, noise {sigma} px")
report("closed form", K0)
report(f"refined (RMS {rms:.2f} px)", K1)
Output:
--- 10 views, tilt up to 40 deg, noise 0.0 px
closed form fx 1200.0 ( -0.0) fy 1190.0 ( -0.0) cx 655.0 ( +0.0) cy 470.0 ( +0.0) s -0.00
refined (RMS 0.00 px) fx 1200.0 ( +0.0) fy 1190.0 ( +0.0) cx 655.0 ( -0.0) cy 470.0 ( +0.0) s -0.00
--- 10 views, tilt up to 40 deg, noise 0.5 px
closed form fx 1194.8 ( -5.2) fy 1184.4 ( -5.6) cx 655.6 ( +0.6) cy 473.5 ( +3.5) s -1.21
refined (RMS 0.71 px) fx 1198.6 ( -1.4) fy 1188.6 ( -1.4) cx 653.8 ( -1.2) cy 472.8 ( +2.8) s +0.24
--- 3 views, tilt up to 40 deg, noise 0.5 px
closed form fx 1240.6 ( +40.6) fy 1220.1 ( +30.1) cx 647.7 ( -7.3) cy 450.8 (-19.2) s -3.29
refined (RMS 0.67 px) fx 1232.3 ( +32.3) fy 1213.4 ( +23.4) cx 648.3 ( -6.7) cy 456.0 (-14.0) s -3.42
--- 10 views, tilt up to 5 deg, noise 0.5 px
closed form fx 1550.3 (+350.3) fy 1536.7 (+346.7) cx 674.6 (+19.6) cy 450.2 (-19.8) s -0.06
refined (RMS 0.68 px) fx 1074.4 (-125.6) fy 1065.6 (-124.4) cx 667.0 (+12.0) cy 470.6 ( +0.6) s +0.04
With exact data the closed form is exact. With ten well-tilted noisy views, refinement brings the focal lengths within about 1.5 pixels of the truth, about 0.1%, and the RMS reprojection error is 0.71 pixels, close to the pixels expected for per-coordinate noise of 0.5 pixels. With three views, the focal length is off by more than 30 pixels in this run, and with nearly fronto-parallel views by more than 120 pixels, yet the RMS reprojection error is just as small: the data fit well, but they do not constrain the parameters.
Applications
Calibrated intrinsics let 3D vision work in metric units: stereo vision and its rectification, triangulation of points, relative pose from the essential matrix, structure from motion, and visual SLAM all assume them. Undistortion maps computed from the distortion coefficients are applied to every frame by image warping. Augmented reality renders virtual content with the real camera’s , robot arms rely on hand–eye calibration, vehicles calibrate rigs of cameras, LiDARs, and IMUs into one frame, and photogrammetric measurement depends on calibration for its accuracy.
Open Problems
- Calibration in the field. Temperature, vibration, and impacts change the parameters over time, so systems on vehicles and robots need online or targetless recalibration that can be trusted without a target.
- Wide-angle and unusual optics. Fisheye, catadioptric, and underwater cameras need models beyond the polynomial pinhole-plus-distortion model, and choosing and validating a model remains partly manual.
- Learned single-image calibration. Networks that predict focal length, distortion, or the horizon from a single image make calibration possible for images in the wild, but are generally less accurate than target-based methods.
Related
- Lens Distortion
The systematic deviation of a real lens from the pinhole model's straight-line projection, which bends straight lines in the image and is modeled, estimated, and removed as part of camera calibration.
- Image Formation
How light leaving a scene becomes an array of pixel values, through illumination and reflectance, the lens, the sensor and its colour filters, sampling and quantization, and the camera's own processing.
- A Flexible New Technique for Camera Calibration
Zhengyou Zhang's 1999 ICCV paper and 2000 IEEE TPAMI paper that calibrated a camera from a few views of a printed planar pattern, using a closed-form solution from plane homographies followed by maximum-likelihood refinement with radial distortion.
- Perspective-n-Point (PnP)
Estimating the position and orientation of a calibrated camera from known 3D points and their 2D image projections, the absolute pose problem behind visual localization, augmented reality, and camera tracking.
- OpenCV
The open-source C++ library of computer vision and image processing functions, with Python bindings that work on NumPy arrays; its modules, conventions, versions from 1.0 to 5.0, contrib modules and licence.
- Industrial Machine Vision
The system engineering of vision on the factory floor, covering machine-vision cameras and their interfaces, GenICam, triggering from encoders and line-scan acquisition, strobed lighting and optics selection, integration with PLCs and robots, gauging and measurement uncertainty in production, and smart cameras against PC-based systems.
References
- Zhang, Z. (2000). A Flexible New Technique for Camera Calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11), 1330–1334.
- Zhang, Z. (1999). Flexible Camera Calibration by Viewing a Plane from Unknown Orientations. IEEE International Conference on Computer Vision (ICCV), 1, 666–673.
- Zhang, Z. (1998). A Flexible New Technique for Camera Calibration. Technical Report MSR-TR-98-71, Microsoft Research (December 1998, with later corrections).
- Tsai, R. Y. (1987). A Versatile Camera Calibration Technique for High-Accuracy 3D Machine Vision Metrology Using Off-the-Shelf TV Cameras and Lenses. IEEE Journal on Robotics and Automation, 3(4), 323–344.
- Hartley, R. & Zisserman, A. (2004). Multiple View Geometry in Computer Vision (2nd ed.). Cambridge University Press.
- Tsai, R. Y. & Lenz, R. K. (1989). A New Technique for Fully Autonomous and Efficient 3D Robotics Hand/Eye Calibration. IEEE Transactions on Robotics and Automation, 5(3), 345–358.
- Furgale, P., Rehder, J. & Siegwart, R. (2013). Unified Temporal and Spatial Calibration for Multi-Sensor Systems. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 1280–1286.
- OpenCV. Camera Calibration and 3D Reconstruction (calib3d module documentation), OpenCV 4.x.
- OpenCV (2026). Camera calibration declarations, modules/calib/include/opencv2/calib.hpp, OpenCV 5.0.0 source code.