Augmented Reality

Augmented reality (AR) shows virtual objects together with the real world so that they appear to occupy the same space: a virtual chair standing on a real floor, a label attached to a machine part, a planned incision drawn on a patient. Unlike a film effect, the virtual content has to stay in place while the viewer moves, so an AR system must know at every moment where the camera or the eye is, what the scene’s surfaces look like, and how the scene is lit, and it must draw the result before the view changes again. Most of that is computer vision: camera pose estimation, SLAM, depth and surface estimation, and lighting estimation, followed by rendering and a display that merges real and virtual light.

Problem

Azuma (1997) defined AR by three characteristics rather than by any device: it combines real and virtual, it is interactive in real time, and it is registered in 3D. The definition excludes films, whose composites are not interactive, and 2D overlays on live video, which are interactive but not tied to 3D positions in the world, while allowing head-mounted displays, monitors and handheld screens alike.

Milgram and Kishino (1994) placed AR on a reality–virtuality continuum running from an entirely real environment to an entirely virtual one. Every display between the two extremes mixes real and virtual objects, and they called the whole range mixed reality (MR). AR is the part near the real end, where a real environment is augmented with computer graphics; near the virtual end is augmented virtuality (AV), a mostly virtual world into which real content such as live video is inserted. They also argued that the distinction between real and virtual has several aspects and proposed three dimensions for classifying such displays: how much the system knows about the world it displays (extent of world knowledge), how realistically it renders it (reproduction fidelity), and how strongly the viewer feels present in it (extent of presence metaphor).

The core technical problem is registration: real and virtual objects must line up. Azuma (1997) explained why AR is far stricter than virtual reality here. In a fully virtual display, an error between where a virtual hand is drawn and where the real hand is pits vision against the sense of body position, and vision tends to win, so an error of five millimetres may go unnoticed. In AR the conflict is between two things seen at once, the real and the virtual hand, and small offsets are obvious. He estimated that the required angular accuracy is a small fraction of a degree, and noted that errors of a few pixels were already detectable on the head-mounted displays of the time.

Registration errors divide into static errors, present even when nothing moves (optical distortion, tracker error, mechanical misalignment, wrong viewing parameters such as the offset between tracker and eye), and dynamic errors, which appear only during motion and come from system latency (Azuma, 1997). For a camera-based AR system, registering a virtual point $\mathbf{X}$ means drawing it at

$$\mathbf{x} = \pi\bigl(K(R\mathbf{X} + \mathbf{t})\bigr),$$

where $K$ holds the intrinsics from camera calibration, $(R, \mathbf{t})$ is the world-to-camera pose of the current frame, and $\pi$ divides by depth. In these terms, static errors are errors in $K$, in the model and in the pose measured at rest; dynamic errors come from using a pose that is out of date by the time the frame is seen.

History

Billinghurst, Clark and Lee (2015) trace the first computer-generated AR experience to Ivan Sutherland’s head-mounted three-dimensional display (Sutherland, 1968). As they describe it, half-silvered mirrors let the user see graphics from miniature cathode-ray tubes overlaid on the room, and the head was tracked by a ceiling-mounted mechanical linkage, later replaced by an ultrasonic tracker. For the next two decades, they write, AR research continued mainly in military and government laboratories, among them the US Air Force’s Super Cockpit program of head-mounted displays for pilots.

At Boeing, Tom Caudell and David Mizell built an AR system that showed workers which real wires to bundle into wire harnesses; Billinghurst et al. (2015) describe their 1992 paper (Caudell and Mizell, 1992) as the first academic paper with the term “augmented reality” in it and note that Caudell is “often credited with coining the phrase”. By 1994, Milgram and Kishino observed that the term was appearing in the literature with increasing regularity, mostly for optical see-through head-mounted displays. Azuma’s 1997 survey collected the applications explored by then (medical, maintenance and repair, annotation, robot path planning, entertainment and military aircraft) together with the three-part definition above, on which Billinghurst et al. (2015) still build their discussion of taxonomies.

Vision-based tracking spread with printed markers. Kato and Billinghurst (1999) developed ARToolKit, a library for real-time vision tracking of square markers; Billinghurst et al. (2015) record that it was released as open source in 2000 and became one of the most widely used AR tracking libraries. Markerless tracking followed from visual SLAM: PTAM (Klein and Murray, 2007) tracked a handheld camera against a map it built of a desk-sized workspace, without markers. Low-compute six-degree-of-freedom tracking with a phone’s existing camera and inertial sensors then made AR available on ordinary smartphones (Valentin et al., 2018), through platform frameworks such as Apple’s ARKit and Google’s ARCore, both described below from their documentation; head-mounted displays with built-in tracking cameras did the same for headsets.

Visual Data

• Camera images. A colour video stream from the device’s main camera, which is both the background of a video see-through display and the input to tracking. Head-mounted displays add dedicated tracking cameras; as of October 2026, Microsoft’s HoloLens 2 hardware documentation lists four visible-light head-tracking cameras, and Apple’s Vision Pro specifications list six world-facing tracking cameras besides the two main cameras.
• Inertial measurements. An inertial measurement unit (IMU) measures angular velocity and acceleration at a higher rate than the camera, which keeps tracking alive through fast motion and blur and makes metric scale observable.
• Depth. Some devices carry a depth sensor, such as a time-of-flight camera (HoloLens 2) or a LiDAR scanner (ARKit’s scene depth on LiDAR-equipped iPhones and iPads); others estimate depth from the moving camera (Valentin et al., 2018).
• Eye and hand data. Headsets track the user’s eyes and hands for interaction; these are inputs to the interface rather than to registration, except that optical see-through displays need the eye’s position relative to the display.

Vision Tasks and Methods

Tracking and localization

Tracking estimates the six-degree-of-freedom pose of the camera (or the head) in every frame. Two families exist.

Marker-based tracking places known patterns in the scene. ARToolKit’s markers are squares with a thick black border and a pattern inside. As Billinghurst et al. (2015) summarize the method, the image is thresholded, regions bounded by four fitted line segments become marker candidates, the four corners give the marker’s pose relative to the camera, and the interior, rectified to a fronto-parallel view, is matched against known patterns to identify the marker and resolve its rotation. Four corners of a known square determine a homography $H \sim K[\mathbf{r}_1\ \mathbf{r}_2\ \mathbf{t}]$ between the marker plane and the image, from which the pose follows; this is planar perspective-n-point, and the example below implements it. Markers are cheap and accurate when visible: Azuma (1997) reported that fiducial-based registration of the time could be accurate to about one pixel. The costs are that the marker must stay visible and unoccluded, and that a small or distant marker gives an unstable pose.

Markerless tracking uses the scene’s own texture. A camera tracked against features of an unknown scene is visual odometry, and with a persistent map and relocalization it becomes SLAM. PTAM was designed for AR: it split tracking and mapping into two threads, so that tracking could run every frame while mapping optimized keyframes in the background (Klein and Murray, 2007). Current phone and headset trackers are visual-inertial. As of October 2026, Apple’s ARKit documentation describes its world tracking as visual-inertial odometry, which combines the device’s motion sensors with the analysis of features tracked across camera frames, and Google’s ARCore documentation describes a SLAM process that detects feature points and combines them with IMU measurements to estimate the camera’s pose over time. ARKit’s documentation names the weaknesses: tracking quality drops when the camera sees a blank wall or a dark scene, and when excessive motion blurs the image.

Tracking gives poses relative to where the session started. Placing content at a known real-world location needs relocalization against a stored map, the same 2D–3D matching followed by PnP that SLAM uses to recover from tracking loss. ARCore’s Cloud Anchors, for example, upload an anchor with nearby feature points so that other devices in the same environment can render the same content attached to it.

Mapping and anchoring

A virtual object is placed by giving it a pose in the world frame. Because SLAM keeps refining its map, for example after a loop closure, the world frame itself shifts slightly, and content placed by fixed world coordinates would drift relative to the real surface it was put on. AR frameworks therefore attach content to anchors: poses tied to tracked features of the map, which are updated as the map is corrected. ARCore’s documentation gives the example of a virtual figure placed on a desk that stays on the desk when the system later adjusts the pose of the plane it detected there. Its Cloud Anchors share anchors between devices; ARKit likewise defines a local coordinate system for every anchor.

Monocular tracking alone does not know the scale of the scene, which is a problem when virtual furniture must have its real size. PTAM simply assumed that the camera moved 10 cm between the two views used to initialize the map (Klein and Murray, 2007); visual-inertial systems recover metric scale from the accelerometer, and depth sensors measure it directly.

Plane detection

Many AR applications only need the surfaces on which content can rest. PTAM fitted the dominant plane of its point map with RANSAC over triples of points and refined it from the covariance of the inliers, so that virtual characters could run on a desk (Klein and Murray, 2007). ARCore looks for clusters of feature points that lie on common horizontal or vertical surfaces, reports them as planes with boundaries, and warns that textureless surfaces such as a white wall may not be detected, because the method depends on feature points. ARKit’s documentation notes that a plane’s position and extent are inaccurate when first detected and are refined over time. A tap on the screen is turned into a placement by hit testing or ray casting: the ray through the tapped pixel is intersected with the detected planes or feature points.

Occlusion and depth

A virtual object behind a real one must be hidden by it. Azuma (1997) noted that a compositor with per-pixel depth for the real scene can decide visibility pixel by pixel, letting real objects cover virtual ones and vice versa; without depth, virtual content is always drawn on top, and Azuma called occlusion one of the strongest depth cues. Valentin et al. (2018) made the same point for phones: without depth, even a virtual object correctly occluded by a real one is impossible, and requiring a depth sensor would restrict AR to few devices. Their pipeline computes low-latency dense depth maps from the phone’s single colour camera on one CPU core, and as of October 2026 ARCore’s documentation describes depth maps computed from the main RGB camera on supported devices, used for occlusion and collisions. Learned monocular depth estimation is an alternative source of per-pixel depth. Where a depth sensor exists it is used directly: ARKit provides scene depth from the LiDAR scanner, and its person segmentation with depth lets people in the camera feed occlude virtual content according to their depth, combining semantic segmentation of people with depth estimates.

Lighting estimation

A virtual object lit differently from its surroundings looks pasted on. Debevec (1998) showed how to render synthetic objects into photographs with consistent lighting: a high-dynamic-range light probe (a photographed mirrored sphere) measures the illumination arriving at the object’s location, the scene is split into a distant part assumed unaffected by the new objects, a local part near them whose reflectance is estimated so that it can receive shadows and reflected light, and the objects themselves, and a differential rendering step adds only the change the objects cause to the photograph.

AR has no light probe and usually only a low-dynamic-range camera with a narrow view. LeGendre et al. (2019) estimated that a typical phone video sees about 6% of the panoramic scene; they trained a network to predict omnidirectional high-dynamic-range lighting from a single low-dynamic-range phone image, using training videos with three spheres of different reflectance (mirrored, matte silver and diffuse grey) in view, and ran it at interactive rates on a phone. Product frameworks expose simpler estimates as well. As of October 2026, ARCore’s documentation describes the average intensity and a colour correction of each camera image, ARKit provides a per-frame light estimate, and ARKit’s environment probes build environment textures from camera imagery for image-based lighting of reflective virtual objects.

Rendering and Latency

Once the pose is known, a virtual camera with the same intrinsics and pose renders the content, and the result is composited with the real view. The time between a head or device movement and the moment the display shows an image computed for the new pose is the motion-to-photon latency. Azuma (1997) defined the closely related end-to-end system delay, from the tracker’s measurement of the viewpoint to the display of the images computed for it, and identified it as the source of dynamic registration error. While the head turns at angular velocity $\omega$, a latency $\Delta t$ displaces virtual content by the angle $\omega\,\Delta t$, which on a display with focal length $f$ in pixels is about

$$\Delta x \approx f \tan(\omega\,\Delta t).$$

Azuma’s numbers from 1997: end-to-end delays of 100 ms were typical, and with a moderate head rotation of 50° per second that gives a 5° error, almost 60 mm at a 68 cm arm’s length; keeping the error under 0.5° at that speed would require a delay of 10 ms or less. He grouped the remedies into four: reduce the system lag, reduce the apparent lag by shifting or warping the rendered image with a newer pose just before display, delay the real video to match the virtual stream (possible only in video see-through), and predict the future pose, which he reported can cut dynamic errors by up to an order of magnitude for short delays, with inertial sensors improving prediction further.

Current systems use all of these. In the OpenXR standard, the runtime gives the application a predicted display time for each frame, and the predicted head pose becomes more accurate the later in the pipeline the application asks for it (Khronos Group, OpenXR 1.1). Microsoft’s HoloLens documentation describes hardware-assisted reprojection, which re-warps each rendered frame to the latest head pose either per pixel from the application’s depth buffer or for a single stabilization plane, and lists the visible artifacts when stabilization fails: judder (uneven motion and double images at low frame rates), swim (content swaying with the head), and colour separation on its colour-sequential displays. Apple’s Vision Pro specifications, as of October 2026, list a 12-millisecond photon-to-photon latency. Read literally, photon-to-photon latency runs from light reaching the cameras to light leaving the displays, a different quantity from motion-to-photon latency: it is the delay with which a video see-through display shows the real world itself.

Display Types

How the real and virtual images are combined defines the display class.

• Video see-through. Cameras capture the real world and the system composites the virtual content into the video before showing it, on a phone or tablet screen or in a closed headset. Azuma (1997) listed its advantages: full control of compositing pixel by pixel, including occlusion, a wider field of view, matched delays for the real and virtual streams, and the video itself as a source for vision-based registration. Its costs are that the user sees the world at the camera’s resolution and latency, from the cameras’ viewpoint rather than the eyes’, and loses sight entirely if the power fails. visionOS calls this view passthrough: real-time video from the device’s external cameras (Apple, Human Interface Guidelines, as of October 2026).
• Optical see-through. Partially reflective combiners in front of the eyes let the user see the real world directly while reflecting the virtual image into view, as in an aircraft head-up display. The real world keeps full resolution and no delay, and the user can still see if the display fails, but the virtual content is a delayed overlay on an undelayed world and the combiner reduces the light from the scene (Azuma, 1997), and registration depends on calibrating the display to the user’s eyes, which can shift as the headset slips (Billinghurst et al., 2015). Because light is added rather than replaced, virtual objects look semi-transparent and cannot hide real ones without extra optics, such as the electronically controlled occlusion mask of Kiyokawa et al. described by Billinghurst et al. (2015). In the OpenXR standard this is the additive blend mode, in which black virtual pixels appear transparent. As of October 2026, Microsoft’s HoloLens 2 hardware documentation lists see-through holographic lenses (waveguides).
• Projection. Projectors draw the virtual content directly onto the surfaces of real objects or walls, so the user wears nothing. Billinghurst et al. (2015) note the limits: a projection surface is needed, so distant objects cannot be augmented; the result is sensitive to ambient light and to shadows cast by people; and occlusion between a user’s hand and a virtual object is harder than with the other two classes.

Azuma (1997) also described monitor-based AR, a video see-through system whose display is a monitor in front of the user rather than a headset; AR on a phone or tablet is video see-through on a handheld screen.

Virtual and Mixed Reality

Virtual reality (VR) replaces the real view entirely; Azuma (1997) described virtual environments as immersing the user in a synthetic world, with AR supplementing reality rather than replacing it. VR needs the same head tracking and low latency but, as described under Problem, tolerates larger registration errors. The OpenXR standard expresses the difference as an environment blend mode: opaque, with no view of the physical world, is the typical VR mode; additive is typical of optical see-through AR; and alpha blending, where each virtual pixel’s alpha sets how much of the real world shows through, is typical of AR on phones and video see-through headsets.

“Mixed reality” is used in two ways. In Milgram and Kishino’s taxonomy it is the whole continuum. In product language it often means a headset that can move along that continuum: visionOS, for example, lets an application show content blended with passthrough in a mixed immersion style or hide the surroundings in a full immersion style, and, in its more immersive styles, fades back to passthrough as the wearer’s head nears a boundary about 1.5 m from where the experience started (Apple, Human Interface Guidelines, as of October 2026).

Example

The script renders a synthetic frame: a 10 cm square marker on a table, a 6 cm virtual cube standing on it, and a camera 0.5 m away, 50° above the table. It projects the marker’s corners, adds Gaussian noise to them as a detector would, estimates the homography from the four noisy corners with the normalized direct linear transform, and decomposes it into a pose using $K$. It then refines that pose by minimizing the reprojection error of the marker points with Levenberg–Marquardt, draws the cube with the estimated pose, and compares the overlay with the true one: the error of the cube’s base vertices, which lie on the marker plane, and of its top vertices, 6 cm above it, and the share of the cube’s pixels that are drawn wrongly. Each setting runs 300 trials. The same comparison is repeated at 1 m, with 16 points (a 4 × 4 grid over the marker, standing in for the corners of an inner pattern or other features on the plane), and at three noise levels. The last lines turn the latency formula into pixels.

import numpy as np
from scipy.optimize import least_squares
from scipy.spatial import ConvexHull
from scipy.spatial.transform import Rotation
from skimage.draw import polygon

W, H = 640, 480
K = np.array([[800.0, 0, 320], [0, 800.0, 240], [0, 0, 1]])

def look_at(C, target, up=(0.0, 0.0, 1.0)):
    """World-to-camera (R, t), OpenCV axes: x right, y down, z forward."""
    z = (target - C) / np.linalg.norm(target - C)
    x = np.cross(z, up); x /= np.linalg.norm(x)
    y = np.cross(z, x)
    R = np.stack([x, y, z])
    return R, -R @ C

def camera(dist, elev=np.radians(50), azim=np.radians(30)):
    """Camera at distance dist, 50 degrees above the marker plane, aimed at its centre."""
    C = dist * np.array([np.cos(elev) * np.cos(azim), np.cos(elev) * np.sin(azim), np.sin(elev)])
    return look_at(C, np.zeros(3))

def project(R, t, X):
    x = (X @ R.T + t) @ K.T
    return x[:, :2] / x[:, 2:]

# Marker: 10 cm square on the plane z = 0. Virtual object: 6 cm cube standing on it.
s, c = 0.10, 0.06
marker = np.array([[-1, -1], [1, -1], [1, 1], [-1, 1]]) * s / 2
grid = np.stack(np.meshgrid(*2 * [np.linspace(-s / 2, s / 2, 4)]), -1).reshape(-1, 2)
cube = np.array([[x, y, z] for z in (0, c) for x in (-c / 2, c / 2) for y in (-c / 2, c / 2)])

def homography(src, dst):
    """Normalized DLT: dst ~ H src for n >= 4 point pairs."""
    def norm(p):
        m = p.mean(0)
        d = np.sqrt(2) / np.linalg.norm(p - m, axis=1).mean()
        return np.array([[d, 0, -d * m[0]], [0, d, -d * m[1]], [0, 0, 1]])
    Ts, Td = norm(src), norm(dst)
    a = np.c_[src, np.ones(len(src))] @ Ts.T
    b = np.c_[dst, np.ones(len(dst))] @ Td.T
    A = []
    for (x, y, _), (u, v, _) in zip(a, b):
        A.append([-x, -y, -1, 0, 0, 0, u * x, u * y, u])
        A.append([0, 0, 0, -x, -y, -1, v * x, v * y, v])
    Hn = np.linalg.svd(np.array(A))[2][-1].reshape(3, 3)
    return np.linalg.inv(Td) @ Hn @ Ts

def pose_from_homography(Hm):
    """H ~ K [r1 r2 t] for a plane at z = 0; returns the nearest rotation and t."""
    B = np.linalg.inv(K) @ Hm
    B /= (np.linalg.norm(B[:, 0]) + np.linalg.norm(B[:, 1])) / 2
    if B[2, 2] < 0:                      # the marker must lie in front of the camera
        B = -B
    M = np.c_[B[:, 0], B[:, 1], np.cross(B[:, 0], B[:, 1])]
    U, _, Vt = np.linalg.svd(M)
    return U @ np.diag([1, 1, np.linalg.det(U @ Vt)]) @ Vt, B[:, 2]

def refine(R, t, X, obs):
    """Minimize the reprojection error over a rotation vector and t (Levenberg-Marquardt)."""
    res = lambda p: (project(Rotation.from_rotvec(p[:3]).as_matrix(), p[3:], X) - obs).ravel()
    p = least_squares(res, np.r_[Rotation.from_matrix(R).as_rotvec(), t], method="lm").x
    return Rotation.from_rotvec(p[:3]).as_matrix(), p[3:]

def silhouette(R, t):
    """Pixel mask of the rendered cube: the convex hull of its projected vertices."""
    p = project(R, t, cube)
    h = p[ConvexHull(p).vertices]
    mask = np.zeros((H, W), bool)
    mask[polygon(h[:, 1], h[:, 0], (H, W))] = True
    return mask

def trial(R0, t0, sigma, rng, pts2d):
    """Noisy marker points -> pose -> overlay; errors against the true overlay."""
    X = np.c_[pts2d, np.zeros(len(pts2d))]
    obs = project(R0, t0, X) + rng.normal(0, sigma, (len(pts2d), 2))
    R, t = pose_from_homography(homography(pts2d, obs))
    out = []
    for R, t in ((R, t), refine(R, t, X, obs)):
        fit = np.sqrt(np.mean(np.sum((project(R, t, X) - obs) ** 2, 1)))
        err = np.linalg.norm(project(R, t, cube) - project(R0, t0, cube), axis=1)
        out += [fit, err[:4].mean(), err[4:].mean()]   # base and top vertices
    true_mask = silhouette(R0, t0)
    out.append(np.sum(true_mask ^ silhouette(R, t)) / true_mask.sum())
    return out

# Render the synthetic frame: grey background, black marker, light-grey cube.
R0, t0 = camera(0.5)
img = np.full((H, W), 0.5)
mk = project(R0, t0, np.c_[marker, np.zeros(4)])
img[polygon(mk[:, 1], mk[:, 0], (H, W))] = 0.0
img[silhouette(R0, t0)] = 0.8
print("marker corners (px):\n", np.round(mk, 1))
print(f"pixels: marker visible {np.sum(img == 0.0)}, cube {np.sum(img == 0.8)}")

R, t = pose_from_homography(homography(marker, mk))
ang = np.degrees(np.arccos(np.clip((np.trace(R.T @ R0) - 1) / 2, -1, 1)))
print(f"noise-free pose error: {ang:.1e} deg, {np.linalg.norm(t - t0):.1e} m")

# fit: RMS residual on the marker points; *_err: mean px error of the cube's base and top
# vertices; flipped: top error above 20 px; wrong_px: share of the cube's pixels misdrawn.
print("\n                     homography only    refined (medians over 300 trials)")
print("pts dist sigma |  fit  top_err |  fit  base_err top_err flipped wrong_px")
for pts, name in ((marker, "4"), (grid, "16")):
    for dist in (0.5, 1.0):
        R0, t0 = camera(dist)
        for sigma in (0.5, 1.0, 2.0):
            rng = np.random.default_rng(0)
            r = np.array([trial(R0, t0, sigma, rng, pts) for _ in range(300)])
            m = np.median(r, 0)
            print(f"{name:>3} {dist:4.1f} {sigma:5.1f} | {m[0]:4.2f} {m[2]:7.2f}  | {m[3]:4.2f} {m[4]:8.2f}"
                  f" {m[5]:7.2f} {100 * np.mean(r[:, 5] > 20):6.1f}% {100 * m[6]:6.1f}%")

# Latency: a head turning at 50 deg/s carries a world-fixed point across the image.
print("\nlatency   overlay lag at 50 deg/s, f = 800 px")
for ms in (12, 20, 50, 100):
    print(f"{ms:4d} ms  {K[0, 0] * np.tan(np.radians(50 * ms / 1000)):6.1f} px")

Output (NumPy 2.3.5, SciPy 1.16.3, scikit-image 0.25.2; two runs gave identical output):

marker corners (px):
 [[293.1 163. ]
 [208.1 263. ]
 [352.1 331.8]
 [426.8 218.1]]
pixels: marker visible 8459, cube 16849
noise-free pose error: 0.0e+00 deg, 1.2e-16 m

                     homography only    refined (medians over 300 trials)
pts dist sigma |  fit  top_err |  fit  base_err top_err flipped wrong_px
  4  0.5   0.5 | 1.42    3.89  | 0.30     0.39    0.91    0.0%    1.1%
  4  0.5   1.0 | 2.83    7.80  | 0.61     0.78    1.81    0.0%    2.1%
  4  0.5   2.0 | 5.52   15.53  | 1.21     1.57    3.60    0.0%    4.2%
  4  1.0   0.5 | 2.74    7.20  | 0.30     0.39    0.86    0.0%    2.2%
  4  1.0   1.0 | 5.28   14.42  | 0.61     0.80    1.74    3.0%    4.3%
  4  1.0   2.0 | 9.00   29.42  | 1.07     1.68    3.90   17.0%    9.4%
 16  0.5   0.5 | 0.95    2.55  | 0.62     0.24    0.57    0.0%    0.7%
 16  0.5   1.0 | 1.90    5.12  | 1.25     0.48    1.14    0.0%    1.3%
 16  0.5   2.0 | 3.77   10.30  | 2.50     0.97    2.27    0.0%    2.7%
 16  1.0   0.5 | 1.51    4.74  | 0.62     0.24    0.55    0.0%    1.4%
 16  1.0   1.0 | 3.01    9.58  | 1.25     0.48    1.10    0.3%    2.8%
 16  1.0   2.0 | 5.59   19.11  | 2.49     0.97    2.22    5.7%    5.6%

latency   overlay lag at 50 deg/s, f = 800 px
  12 ms     8.4 px
  20 ms    14.0 px
  50 ms    34.9 px
 100 ms    70.0 px

Without noise the pipeline recovers the pose to rounding error, so every error in the table comes from the corner noise. Four observations follow.

• Decomposing the homography is only an initialization. With four corners the homography fits the noisy corners exactly, but forcing its first two columns into a rotation moves the pose, and the reprojected corners then miss by about three to five and a half times the noise. Refining the pose on the reprojection error cuts the cube’s top-vertex error by a factor of about 4 to 4.5 at 0.5 m and 7.5 to 9 at 1 m. This is why trackers refine the pose rather than use the decomposition directly.
• Error grows with height above the marker. The cube’s top vertices are off by more than twice as much as its base vertices, which lie on the marker plane that the corners constrain directly. Small rotation errors, which hardly move points on the plane, swing points above it; it follows that content far from the tracked surface registers worse than content on it.
• The residual on the marker does not measure the overlay. With 16 points the fit residual doubles compared with four corners, yet the overlay error drops by about 37%. Our reading: four corners give eight measurements for six pose parameters, so the fit absorbs most of the noise, while 32 measurements leave it visible in the residual and average it out of the pose. A tracker’s reported reprojection error is a consistency check, not the registration error a viewer sees.
• A small marker can flip. At 1 m the marker is about half as large in the image, and with 1–2 px of noise the refined pose sometimes converges to the second solution of the planar pose ambiguity described under perspective-n-point: in 3% and 17% of the four-corner trials the cube is drawn tilted the wrong way, more than 20 px off. More points reduce but do not remove the flips. When it does not flip, the pixel error at 1 m is about the same as at 0.5 m, but the cube is smaller, so the share of its pixels drawn wrongly roughly doubles.

The latency lines put these numbers in proportion: at 50° per second, a 12 ms delay already moves the overlay by about 8 px on this camera, more than the error from 2 px of corner noise, and Azuma’s 100 ms gives 70 px. Prediction and late reprojection exist because no pose estimator can fix an error that comes from time.

Challenges

• Registration accuracy. Errors of a fraction of a degree are visible (Azuma, 1997). Every stage contributes: calibration of the camera, and for optical see-through of the display to the eye; tracking noise and drift; and latency.
• Tracking robustness. Textureless surfaces, darkness and fast motion degrade visual tracking (ARKit documentation). Inertial sensing bridges short visual failures but drifts on its own.
• Dynamic and deformable scenes. Tracking and mapping assume a mostly static world. People walking through the view, moved furniture, or a deforming organ in surgical AR break that assumption.
• Scale and world alignment. Monocular systems need inertial data, depth or a known object to place content at true size, and outdoor AR needs the map aligned to geographic coordinates.
• Photometric consistency. Lighting must be estimated from a narrow, low-dynamic-range view (LeGendre et al., 2019), and the virtual content must match the camera’s exposure, white balance, noise and blur to blend in.
• Display limits. Besides the occlusion and resolution limits described under Display Types, optical see-through displays look dim against a bright outdoor scene (Billinghurst et al., 2015).

Evaluation

AR has no single benchmark; a system is judged by how well its components work and how the combined result looks to users.

• Registration error, in pixels or degrees, between where a virtual feature is drawn and where its real counterpart appears, measured statically and during motion. Azuma (1997) expressed requirements in angular terms; for a camera display, pixel error converts to angle through the focal length.
• Tracking accuracy on recorded sequences with ground-truth poses, using the trajectory metrics of visual odometry and SLAM such as absolute trajectory error, for example on visual-inertial datasets such as EuRoC MAV. These measure the pose, not the overlay, and need an alignment convention.
• Latency, motion-to-photon for the virtual content and, for video see-through, photon-to-photon for the real view, measured end to end rather than per component.
• Depth and lighting quality, with the metrics of depth estimation and of lighting estimation, often judged by how renderings under the estimated lighting compare with photographs of real objects, as LeGendre et al. (2019) did.
• User studies of task performance, comfort and perceived stability, which no component metric replaces.

Failure Modes and Risks

• Drift and jumps. Content slowly slides as tracking drifts, then jumps when a loop closure or relocalization corrects the map; anchors reduce but do not hide this.
• Wrong relocalization. Where two places look alike, a tracker can localize against the wrong one and show content in an unexpected location; Microsoft’s hologram stability documentation gives the example of two identical-looking rooms.
• Lag artifacts. The swim and judder described under Rendering and Latency, when frame rate drops or reprojection is not used.
• Occlusion errors. Virtual objects drawn in front of real objects that should hide them, or cut out by noisy depth at object boundaries.
• Pose ambiguity. Small or distant planar targets can produce a pose tilted the wrong way, as in the example.
• Safety. A video see-through display that fails leaves the user unable to see (Azuma, 1997), and any AR overlay can hide a real hazard or distract the user from it; in medical and industrial uses a misregistered overlay can direct an action to the wrong place, as Azuma’s needle-biopsy example illustrates.
• Privacy. World-facing cameras continuously record the user’s surroundings, including other people, and shared anchors upload feature data from those surroundings to a server, as ARCore’s Cloud Anchors do.

Deployment Considerations

• Platform frameworks. On phones, applications usually rely on the platform’s tracking, plane detection, depth and lighting estimates (ARKit, ARCore) rather than their own, and on headsets on the runtime’s tracking and reprojection through OpenXR or the vendor’s SDK; the APIs and the devices that support depth or LiDAR change between versions, so features should be checked at run time, as ARKit’s documentation asks for scene depth.
• Frame budget. Rendering must keep a constant frame rate: Microsoft’s documentation asks HoloLens applications to hold 60 frames per second and notes that a consistently lower frame rate looks more stable than a fluctuating one.
• Calibration. Camera intrinsics, camera–IMU extrinsics and time offsets, and for optical see-through the eye-to-display geometry, set the floor on static registration error.
• Markers where they help. Where the environment can be prepared, as in factories and labs, markers remain the cheap and accurate choice.
• Content design for the error budget. Keep content close to tracked surfaces, as the example suggests; coarse annotations tolerate errors that a surgical overlay cannot.