Perspective-n-Point (PnP)

Perspective-n-Point (PnP) is the problem of finding where a camera is and which way it points, given $n$ points whose 3D positions are known and the pixels where they appear in the image. The camera’s intrinsics are assumed known. PnP is also called absolute pose estimation, exterior orientation, or, in photogrammetry, space resection; when the intrinsics are unknown as well, the problem becomes camera resectioning. It is how an image is localized against a 3D map, how augmented reality anchors virtual objects, and how a camera tracker follows a moving camera once a map exists.

The minimal case uses three points (P3P) and has been studied since 1841. Modern practice combines a fast minimal or linear solver, RANSAC to reject wrong matches, and a final minimization of reprojection error.

Problem Definition

The camera has a known intrinsic matrix $K$ (see camera parameters) and an unknown pose: a rotation $R$ and a translation $\mathbf{t}$ that map world coordinates to camera coordinates, $\mathbf{X}_c = R\mathbf{X} + \mathbf{t}$. For $n$ correspondences $\mathbf{X}_i \leftrightarrow \mathbf{x}_i$ between 3D points and pixels,

$$\lambda_i \begin{bmatrix} \mathbf{x}_i \\ 1 \end{bmatrix} = K\,(R\mathbf{X}_i + \mathbf{t}), \qquad i = 1, \dots, n,$$

with $\lambda_i > 0$ the depth of point $i$. Applying $K^{-1}$ gives the normalized coordinates $\hat{\mathbf{x}}_i = K^{-1}\mathbf{x}_i$, the direction of the ray through each pixel: PnP seeks the rigid motion that puts every 3D point on its ray. Lens distortion is removed from the measurements first.

The pose has six degrees of freedom and each correspondence gives two equations, so three points give finitely many solutions. Under Gaussian pixel noise, the maximum-likelihood pose minimizes the reprojection error,

$$(R^*, \mathbf{t}^*) = \arg\min_{R,\,\mathbf{t}} \sum_{i=1}^{n} \bigl\lVert \mathbf{x}_i - \pi\bigl(K(R\mathbf{X}_i + \mathbf{t})\bigr) \bigr\rVert^2,$$

where $\pi$ divides by the third coordinate. Note that $\mathbf{t}$ is not the camera position; the camera center is $\mathbf{C} = -R^\top\mathbf{t}$.

Input and Output

• Input: $n \ge 3$ 2D–3D correspondences, the intrinsics $K$, and usually distortion coefficients; some correspondences are typically wrong.
• Output: $R$ and $\mathbf{t}$, often with the set of inlier correspondences. Minimal solvers return several candidate poses.

Variants

• Unknown intrinsics: camera resectioning. If $K$ is unknown, the full $3 \times 4$ camera matrix $P$ is estimated. It has 11 degrees of freedom, so it needs five and a half correspondences, in practice six, and the direct linear transform (DLT) solves for it linearly (Hartley and Zisserman, 2004, §7.1), the classical route to camera calibration from a 3D target.
• Planar scenes. When all points lie on a plane, as on a marker, a calibration board, or a wall, the image of the plane is a homography $H \sim K[\mathbf{r}_1\ \mathbf{r}_2\ \mathbf{t}]$, from which the pose follows. Planar targets get dedicated solvers, discussed below.
• Other inputs. Pose from lines instead of points, from multi-camera rigs, or with an unknown focal length.

Challenges

• Wrong matches. Matching image features against a 3D map produces many outliers, and a single one can wreck a least-squares estimate.
• Multiple solutions. P3P has up to four physically valid solutions, and even four or five points can be ambiguous (Fischler and Bolles, 1981), so extra points are needed to choose among them.
• Degenerate configurations. P3P needs three points that are not collinear and image points that are not collinear either (Persson and Nordberg, 2018). A subtler failure is the danger cylinder: the circular cylinder through the three points with its axis perpendicular to their plane. As Wang et al. (2019) review, Thompson linked it in 1966 to a singular Jacobian of the constraints, making P3P unstable when the camera center lies on it; there, two solutions also merge into a double one. The DLT, which ignores $K$, fails when the camera and points lie on a twisted cubic, or when the points lie on a plane together with a line through the camera center. A distant view of a scene with little relief comes close to the planar case (Hartley and Zisserman, 2004, §7.1, §22.1).
• Planar pose ambiguity. For a tilted plane, two quite different poses can fit noisy points almost equally well; in the planar experiments of Lepetit et al. (2009), the ambiguity is absent for a fronto-parallel plane and appears for an inclined one.

Approaches

Minimal solvers: P3P

The angle between the rays to two points follows from the image, $\cos\theta_{ij} = \hat{\mathbf{x}}_i^\top\hat{\mathbf{x}}_j / (\lVert\hat{\mathbf{x}}_i\rVert\,\lVert\hat{\mathbf{x}}_j\rVert)$, and the distance $d_{ij}$ between the 3D points is known. The unknown distances $s_i$ from the camera center to the points satisfy three law-of-cosines equations,

$$d_{ij}^2 = s_i^2 + s_j^2 - 2\,s_i s_j \cos\theta_{ij}, \qquad (i,j) \in \{(1,2), (1,3), (2,3)\}.$$

Given the $s_i$, the points are known in camera coordinates, and the pose is the rigid motion aligning them with their world coordinates. The system reduces to a quartic and has at most four positive solutions.

Haralick et al. (1994) traced direct solutions to Grunert in 1841 and compared the methods that followed in photogrammetry. Fischler and Bolles (1981) independently named the perspective-$n$-point problem and embedded a closed-form P3P solution in RANSAC (see the paper article).

Kneip, Scaramuzza, and Siegwart (2011) introduced a parametrization that computes the camera position and orientation directly, in a single stage, without first computing the points in the camera frame. Persson and Nordberg (2018) went the other way: their Lambda Twist solver works with the distances, reduces the problem to intersecting two ellipses, and finds the solutions from a single real root of a cubic instead of all roots of a quartic. It never produces invalid or duplicate solutions, and the authors reported better numerical accuracy and speed than earlier solvers on synthetic data.

Linear and non-minimal solvers

The simplest non-minimal method applies the DLT to normalized coordinates and projects the result onto the nearest rotation. It needs six non-coplanar points and ignores the rotation constraint while solving; in the experiments of Lepetit et al. (2009), a DLT with the intrinsics clamped to their known values was less accurate than EPnP, especially at high noise.

EPnP (Lepetit, Moreno-Noguer and Fua, 2009) writes every 3D point as a weighted sum of four virtual control points, $\mathbf{X}_i = \sum_{j=1}^4 \alpha_{ij}\mathbf{c}_j$ with $\sum_j \alpha_{ij} = 1$. The same weights hold in camera coordinates, so the unknowns become the twelve camera-frame coordinates of the control points rather than $n$ depths. Each correspondence gives two linear equations, stacked into $M\mathbf{y} = \mathbf{0}$ with $M$ of size $2n \times 12$. The solution is a combination of the $N$ null vectors of the $12 \times 12$ matrix $M^\top M$, with $N$ between 1 and 4 (it grows as the camera nears orthographic projection), and the weights follow from requiring the distances between control points to match their known values. Forming $M^\top M$ dominates the cost, which is therefore $O(n)$. The authors choose the control points at the centroid and along the principal directions of the data, use three control points for planar scenes, and refine the weights by a short Gauss–Newton iteration whose cost does not depend on $n$.

Planar targets

For coplanar points, the pose can be recovered from the homography by decomposing $K^{-1}H$, as in camera calibration. IPPE (infinitesimal plane-based pose estimation; Collins and Bartoli, 2014) instead uses the local transformation at one point of the plane, where the homography is best estimated, and solves for pose analytically. The authors report it far faster than PnP-based methods and in most cases more accurate. Through solvePnPGeneric, OpenCV’s implementation returns two solutions, exposing the planar ambiguity rather than hiding it.

Iterative refinement

Closed-form solutions do not minimize the geometric error, so the final step minimizes the reprojection error by nonlinear least squares, updating the rotation through a small rotation vector. Refinement converges in a few iterations from a good initialization, but it cannot fix a solution caught in the wrong branch of an ambiguity.

Robust estimation

With outliers, PnP is run inside RANSAC: sample a minimal set, solve, count correspondences whose reprojection error is below a threshold, keep the largest consensus, then re-solve on the inliers and refine. Small samples pay off: with 60% inliers, a three-point sample needs about 19 trials for 99% confidence, a six-point sample about 96.

OpenCV’s solvePnP selects among solvers such as EPnP, P3P, and IPPE through flags; its default, SOLVEPNP_ITERATIVE, initializes with the DLT (non-planar points) or a homography decomposition (planar points) and then runs Levenberg–Marquardt. solvePnPRansac adds outlier rejection.

Example

The NumPy code below simulates 200 correspondences with 1-pixel noise and replaces about 40% of them (89 here) with random wrong matches. It estimates the pose with the DLT on all points, with RANSAC over six-point DLT samples, and with Gauss–Newton refinement on the inliers, then re-classifies the inliers with the refined pose and refines again.

import numpy as np
from scipy.spatial.transform import Rotation

rng = np.random.default_rng(0)
K = np.array([[800.0, 0, 320], [0, 800, 240], [0, 0, 1]])
R_true = Rotation.from_euler("xyz", [15, -25, 5], degrees=True).as_matrix()
t_true = np.array([0.2, -0.1, 6.0])                  # world origin 6 m ahead
C_true = -R_true.T @ t_true                          # camera centre in the world

def project(R, t, X):
    Xc = X @ R.T + t
    x = Xc @ K.T
    return x[:, :2] / x[:, 2:], Xc[:, 2]

def pose_dlt(X, x):
    """Linear pose from >= 6 points: DLT on normalized coordinates x^ = K^-1 x,
    then the nearest rotation. Needs non-coplanar points."""
    xh = np.c_[x, np.ones(len(x))] @ np.linalg.inv(K).T
    c = X.mean(0); s = np.sqrt(3) / np.linalg.norm(X - c, axis=1).mean()
    Xn = np.c_[(X - c) * s, np.ones(len(X))]          # conditioned 3D points
    A = []
    for Xi, (u, v, _) in zip(Xn, xh):
        A += [np.r_[Xi, 0 * Xi, -u * Xi], np.r_[0 * Xi, Xi, -v * Xi]]
    M = np.linalg.svd(np.array(A))[2][-1].reshape(3, 4)   # M ~ [R/s | R c + t]
    if np.linalg.det(M[:, :3]) < 0:
        M = -M
    U, S, Vt = np.linalg.svd(M[:, :3])
    R = U @ Vt                                        # nearest rotation
    lam = S.mean() * s                                # unknown scale of M
    return R, M[:, 3] / lam - R @ c

def refine(R, t, X, x, iters=10):
    """Gauss-Newton on reprojection error, R <- exp([w]x) R, t <- t + dt."""
    for _ in range(iters):
        Xc = X @ R.T + t
        r = (project(R, t, X)[0] - x).ravel()
        J = np.zeros((2 * len(X), 6))
        for i, (Xi, Yi, Zi) in enumerate(Xc):
            dp = np.array([[K[0, 0] / Zi, 0, -K[0, 0] * Xi / Zi**2],
                           [0, K[1, 1] / Zi, -K[1, 1] * Yi / Zi**2]])
            RX = Xc[i] - t                            # R X_i
            skew = np.array([[0, -RX[2], RX[1]], [RX[2], 0, -RX[0]], [-RX[1], RX[0], 0]])
            J[2 * i:2 * i + 2] = np.c_[-dp @ skew, dp]
        d = np.linalg.lstsq(J, -r, rcond=None)[0]
        R = Rotation.from_rotvec(d[:3]).as_matrix() @ R
        t = t + d[3:]
    return R, t

def errors(R, t):
    ang = np.degrees(np.arccos(np.clip((np.trace(R_true.T @ R) - 1) / 2, -1, 1)))
    return ang, 1000 * np.linalg.norm(-R.T @ t - C_true)   # degrees, mm

# 200 points in a 3 m cube around the origin, 1-pixel noise, 40% outliers
n, sigma = 200, 1.0
X = rng.uniform(-1.5, 1.5, (n, 3))
x, depth = project(R_true, t_true, X)
x += sigma * rng.standard_normal((n, 2))
bad = rng.random(n) < 0.4
x[bad] = rng.uniform([0, 0], [640, 480], (bad.sum(), 2))   # random wrong matches
print(f"{bad.sum()} outliers of {n}, depths {depth.min():.1f} to {depth.max():.1f} m")

for s in (3, 6):                                      # trials for 99% confidence, w = 0.6
    print(f"sample size {s}: {np.log(0.01) / np.log(1 - 0.6**s):.0f} trials")

print("DLT on all points:   rot %.2f deg, centre %7.1f mm" % errors(*pose_dlt(X, x)))

thr = np.sqrt(5.99) * sigma                           # chi-squared, 2 dof, 95%
best = np.zeros(n, bool)
for _ in range(200):
    idx = rng.choice(n, 6, replace=False)
    R, t = pose_dlt(X[idx], x[idx])
    xp, z = project(R, t, X)
    inl = (np.linalg.norm(xp - x, axis=1) < thr) & (z > 0)
    if inl.sum() > best.sum():
        best = inl
print(f"RANSAC consensus: {best.sum()} points, {(best & bad).sum()} of them outliers")
R0, t0 = pose_dlt(X[best], x[best])
print("DLT on consensus:    rot %.2f deg, centre %7.1f mm" % errors(R0, t0))
R1, t1 = refine(R0, t0, X[best], x[best])
inl = np.linalg.norm(project(R1, t1, X)[0] - x, axis=1) < thr   # re-classify
R1, t1 = refine(R1, t1, X[inl], x[inl])
print(f"final inliers: {inl.sum()} (true inliers: {(~bad).sum()}, wrongly kept: {(inl & bad).sum()})")
print("Gauss-Newton:        rot %.2f deg, centre %7.1f mm" % errors(R1, t1))
e = np.linalg.norm(project(R1, t1, X[inl])[0] - x[inl], axis=1)
print(f"reprojection RMS on inliers: {np.sqrt(np.mean(e**2)):.3f} px")

Output:

89 outliers of 200, depths 4.1 to 8.1 m
sample size 3: 19 trials
sample size 6: 96 trials
DLT on all points:   rot 61.87 deg, centre  5527.8 mm
RANSAC consensus: 65 points, 0 of them outliers
DLT on consensus:    rot 0.54 deg, centre    62.5 mm
final inliers: 102 (true inliers: 111, wrongly kept: 0)
Gauss-Newton:        rot 0.16 deg, centre    17.3 mm
reprojection RMS on inliers: 1.199 px

Least squares on all points fails completely: the pose is off by 62° and 5.5 m. RANSAC finds a clean consensus set, but a pose computed from six noisy points explains only 65 of the 111 true inliers within the 2.45-pixel threshold. Refining and re-classifying recovers 102 of them, with no outliers kept, and Gauss–Newton reduces the rotation error from 0.54° to 0.16° and the camera-center error from 62 to 17 mm. A three-point minimal solver would need about a fifth as many samples as this six-point DLT.

Datasets and Benchmarks

Beyond synthetic tests, PnP is evaluated inside localization systems: Sattler et al. (2018) introduced the Aachen Day-Night, RobotCar Seasons, and CMU Seasons datasets for localizing query images against a 3D model under changing conditions, with ground-truth poses for the queries.

Evaluation Metrics

• Rotation error: the angle of the rotation between estimate and ground truth, $\alpha = \arccos\bigl((\operatorname{tr}(R_{\text{gt}}^\top R) - 1)/2\bigr)$.
• Position error: $\lVert \mathbf{C} - \mathbf{C}_{\text{gt}} \rVert$ with $\mathbf{C} = -R^\top\mathbf{t}$. Comparing $\mathbf{t}$ directly mixes in the rotation error.
• Recall at thresholds: the fraction of images localized within a position and an angle threshold. Sattler et al. (2018) use (0.25 m, 2°), (0.5 m, 5°), and (5 m, 10°), with looser thresholds for the Aachen night queries.
• Reprojection error on the inliers shows whether the pose is consistent with the data, but not whether it is correct: a pose on the wrong side of the planar ambiguity can fit almost as well.
• Runtime, for minimal solvers that run inside RANSAC.

Applications

• Visual localization: registering a query image to a structure-from-motion model by matching features to 3D points and solving PnP with RANSAC, for navigation and augmented reality.
• Augmented reality: the pose of a square marker from its four corners, or of the camera from tracked features with known 3D positions.
• Camera tracking in SLAM, visual odometry, and incremental structure from motion: each new frame or image is posed from matches to existing map points, before triangulation adds new points.
• Robotics: the pose of a known object relative to the camera, as in model-based recognition, and hand–eye calibration (Hartley and Zisserman, 2004, §7.3).

Open Problems

Under strong appearance changes between mapping and query images, such as day and night, the difficulty lies mostly in obtaining correct 2D–3D matches; Sattler et al. (2018) concluded that long-term localization is far from solved and called for better local features. Near-degenerate configurations, such as small markers seen from far away or points close to the danger cylinder, still produce unstable or ambiguous poses, and choosing the right branch of an ambiguity from little data remains difficult.