3D Reconstruction

3D reconstruction recovers the shape of objects and scenes in three dimensions from measurements of them: photographs, video, depth maps, or laser scans. It is also called 3D scanning, 3D modeling from images, image-based modeling, or, in surveying, photogrammetry. The answer may be a cloud of points, a triangle mesh, a voxel volume, a function whose zero level set is the surface, or a field that renders new views. This article maps the main routes to 3D shape, the stages of a typical pipeline, and how results are measured, and links to the articles that cover each part in depth.

Problem Definition

Given observations of a scene, estimate the geometry of its surfaces in a common 3D frame, and often their color as well. The observations constrain only what was seen: each pixel or range sample defines a ray, and the measurement places a point on it.

When a camera with intrinsics $K$ and world-to-camera pose $[R \mid \mathbf{t}]$, so that $\mathbf{X}_c = R\mathbf{X}_w + \mathbf{t}$ (camera parameters), reports a depth $Z$ at pixel $\mathbf{x} = (u, v, 1)^\top$, the surface point in the world frame is

$$\mathbf{X}_w = R^\top\left(Z\,K^{-1}\mathbf{x} - \mathbf{t}\right),$$

the depth map’s back-projection followed by the camera-to-world transform. Much of reconstruction is about obtaining the two ingredients, the poses and the depths, and then merging the overlapping, noisy, partial results into one consistent surface.

Three facts shape the problem. Images alone fix a reconstruction only up to a similarity transformation, or a projective one with unknown intrinsics, so metric scale needs a known baseline, an active sensor, or other outside information (multi-view geometry). Surfaces that no sensor saw remain unknown unless a prior fills them in. And every measurement is noisy, so overlapping observations disagree and must be reconciled.

Input and Output

Input: one or more images with or without known cameras, video, depth maps from stereo or RGB-D cameras, or range scans from laser scanners, sometimes with inertial or GPS data.

Output: one of several representations, often several in turn, since pipelines convert between them; 3D representations compares them.

Representation	What it stores	Typical source
Sparse point cloud	Triangulated feature points with camera poses	Structure from motion
Depth maps	One depth per pixel per view	Stereo, multi-view stereo, active sensors, monocular networks
Dense point cloud	Fused samples, often with normals and colors	Depth-map fusion, LiDAR
Triangle mesh	Vertices and faces forming a surface	Meshing of points or volumes
Volume	Occupancy or truncated signed distance on a voxel grid	Range-image and RGB-D fusion
Implicit field	A function, often a network, whose level set is the surface	Learned shape models, neural surface fitting
Radiance field	Density and view-dependent color, or 3D Gaussians	Neural rendering from posed images

Variants

• Objects or scenes. A single object seen from all around, as on a turntable, or a room, building, or city that extends beyond every image.
• Offline or real-time. Photogrammetry processes a finished capture; SLAM reconstructs while the sensor moves and needs the map to localize.
• Static or dynamic. Most methods assume a rigid scene; non-rigid reconstruction of people, faces, or deforming objects must also recover the motion.
• Geometry or appearance. Some applications need only shape; others, such as virtual tours and visual effects, need surfaces that render convincingly from new viewpoints.
• Many views or one. Single-image reconstruction replaces measurement with learned priors about object categories and scene layouts.

Challenges

• Correspondence. Passive methods must match the same surface point across images, which fails on textureless, repetitive, reflective, or transparent surfaces.
• Visibility and occlusion. Which views see a point depends on the unknown geometry, and the backs of objects and occluded regions leave holes.
• Scale and drift. Monocular reconstructions have no metric scale, and errors accumulate along long sequences (SLAM).
• Noise and outliers. Depth noise grows with distance for triangulation-based sensors (stereo vision), and mixed pixels at depth discontinuities create points in empty space.
• Size. Dense volumes grow with the cube of their resolution, and scenes may span millions of images or points.
• Ground truth. Reference geometry needs a more accurate sensor, typically a laser or structured-light scanner, and even that misses parts of the scene.

Approaches

Passive multi-view reconstruction

From photographs alone, reconstruction runs in three stages, each with its own article. Structure from motion matches features across images and recovers every camera’s pose and intrinsics together with a sparse cloud of triangulated points, refined by bundle adjustment. Multi-view stereo then uses the known cameras to estimate depth for nearly every pixel, for example by plane sweeping or PatchMatch, and fuses the per-view depth maps into a dense, consistent point cloud. Finally, meshing turns points into a surface. Poisson surface reconstruction treats oriented points as samples of the gradient of the shape’s indicator function and recovers that function by solving a sparse Poisson problem (Kazhdan, Bolitho and Hoppe, 2006); the DTU paper, for example, meshed the point clouds of all three MVS methods it tested this way. Two calibrated cameras are the special case of stereo vision, and visual odometry and SLAM are the sequential, real-time counterparts of structure from motion.

Active sensing

Active sensors emit their own light and measure depth directly, so they work on textureless surfaces and give metric scale (see depth estimation for the principles). Structured light projects a known pattern and triangulates between projector and camera; the first Kinect worked this way. Time-of-flight cameras measure the travel time of modulated light; the second Kinect measures the phase shift of the returning signal and needs no triangulation (Sarbolandi et al., 2015). LiDAR scans laser beams across the scene, accurately at long range but sparsely. RGB-D cameras pair a depth sensor with a color camera.

Each scan or depth frame covers one viewpoint, so scans are aligned, for example by iterative closest point, and fused. Curless and Levoy (1996) converted each range image to a signed distance along the sensor’s lines of sight, added it into a cumulative weighted signed distance function on a voxel grid, and extracted the zero-crossing isosurface with marching cubes; under their assumptions the result is optimal in the least-squares sense. KinectFusion (Newcombe et al., 2011) made this real-time with a handheld depth camera and a GPU: each frame is tracked against the growing global model with coarse-to-fine ICP and then fused into it. The truncated signed distance function (TSDF) that both use is the standard volume for depth fusion.

Shape from X

A single viewpoint also carries shape cues, collectively called shape from X:

• Shading. Horn (1970) recovered the shape of a smooth opaque object from one image, given its reflectance, the light source position, and extra information to resolve ambiguities, by integrating a nonlinear partial differential equation along characteristic curves.
• Photometric stereo. Woodham (1980) varied the illumination direction between images taken from a fixed viewpoint; because pixels correspond trivially, the intensities at each pixel determine its surface orientation.
• Focus and defocus. Sharpness as a function of the focus setting, or the amount of blur, indicates depth (depth estimation).
• Silhouettes. Each silhouette and its camera center define a cone that must contain the object; intersecting the cones of all views gives the visual hull, a term Matusik et al. (2000) credit to Laurentini (1994). The visual hull always contains the object and fits at least as tightly as its convex hull, but concavities can never be recovered from silhouettes alone (Matusik et al., 2000). Space carving refines it with photo-consistency (multi-view stereo).

These cues need strong assumptions about reflectance, lighting, or segmentation, so they appear today mostly in controlled settings or as components of other methods.

Learned and feed-forward reconstruction

Learning entered reconstruction in several ways. Monocular depth networks predict a depth map from one image, metric or up to scale and shift (depth estimation); back-projected, it gives a single-view point cloud.

Implicit networks represent a shape as a function of 3D position. Occupancy Networks (Mescheder et al., 2019) represent the surface as the decision boundary of a classifier of inside versus outside, and DeepSDF (Park et al., 2019) regresses a signed distance, negative inside and positive outside, conditioned on a latent code so that one network represents a whole class of shapes. Neither stores a fixed-resolution grid, both can be inferred from partial or single-view input, and a mesh can be extracted from the level set afterwards. Tatarchenko et al. (2019) showed a limit of single-view object reconstruction as then practiced: on ShapeNet, simple classification and retrieval baselines matched or beat the encoder–decoder networks, which suggests that the networks largely recognized the object category rather than reconstructed the specific object.

3D foundation models regress geometry for whole image sets without known cameras. DUSt3R (Wang et al., 2024) predicts, for a pair of images, a pointmap, a 3D point for every pixel in a common frame, from which depths, correspondences and cameras can be read off, and globally aligns the pairs when there are more than two images. VGGT (Wang et al., 2025) is a feed-forward transformer that takes one to hundreds of views and predicts cameras, depth maps, point maps, and point tracks for all of them at once; its authors report reconstruction in under a second. On DTU, VGGT’s authors reported (CVPR 2025 paper, Table 2; the same values appear in arXiv v1) a Chamfer-style overall error of 0.382 without ground-truth cameras, against 1.741 for DUSt3R and 0.295 for GeoMVSNet, a learned MVS method given the true cameras.

Neural rendering

Neural radiance fields (NeRF, Mildenhall et al., 2020) fit a network that maps a 3D position and viewing direction to a volume density and an emitted color, and render images by volume rendering along camera rays; since rendering is differentiable, posed images are the only supervision. 3D Gaussian splatting (Kerbl et al., 2023) replaces the network with up to millions of anisotropic 3D Gaussians, initialized from the sparse structure-from-motion points, and reports real-time rendering, at least 30 frames per second at 1080p. Both take their cameras from structure from motion and target novel-view synthesis: they model appearance well, but their geometry is a density field rather than a surface, and extracting an accurate surface from them is a research topic of its own.

Why the field moved

In outline, and in our reading, the order of these families follows what became available. Shape from X and stereo came first, when one or two images were the data. Affordable range scanners in the 1990s shifted the problem to registering and merging scans, which volumetric fusion solved. Benchmarks with scanned ground truth (Seitz et al., 2006) and internet photo collections drove multi-view stereo and large-scale structure from motion, and cheap depth cameras with GPUs made dense reconstruction real-time. Learning came in where geometry alone fails, for single images, missing regions, and shape priors; neural rendering shifted the target from surfaces to images; and the feed-forward models replace parts of the optimization pipeline itself.

The Typical Pipeline

A photogrammetry pipeline from photographs to a textured mesh:

1. Capture with overlap between neighboring images and varied viewpoints; calibrate the camera if possible (camera calibration, lens distortion).
2. Features and matching: detect and match local features, verified geometrically with RANSAC and the essential or fundamental matrix.
3. Structure from motion: register cameras by perspective-n-point, triangulate points, and refine with bundle adjustment to minimize reprojection error; fix scale and frame with known distances or GPS.
4. Multi-view stereo: estimate a depth map per view against its neighbors through a cost volume or PatchMatch, and filter depths that the other views do not confirm.
5. Fusion: merge the depth maps into a dense point cloud, or into a TSDF volume, the subject of this batch’s TSDF fusion article.
6. Meshing: extract a mesh, by Poisson surface reconstruction from oriented points or by marching cubes from a volume, both covered by articles in this batch.
7. Texturing and cleanup: project the images onto the mesh, remove small disconnected pieces, and simplify.
8. Evaluation against reference geometry, as below.

With a depth camera, steps 2–4 collapse into tracking each frame against the model with ICP, as in KinectFusion. ScanNet shows such a pipeline at dataset scale: BundleFusion estimated the poses, the RGB-D frames were integrated into a TSDF, and marching cubes extracted meshes at 4 mm voxels, which were then cleaned and simplified (Dai et al., 2017).

Datasets and Benchmarks

Dataset	Content	Reference geometry
Middlebury MVS (Seitz et al., 2006)	Two small objects, temple and dino; 317 and 363 calibrated views in the full sets	Laser stripe scans
DTU (Aanæs et al., 2016)	80 evaluated scenes of objects, 49 or 64 robot-arm views at 1600 × 1200, seven lighting conditions	Structured-light scans
Tanks and Temples (Knapitsch et al., 2017)	8 intermediate and 6 advanced indoor and outdoor scenes, video input, extra training scenes	Industrial laser scanner
ETH3D (Schöps et al., 2017)	Indoor and outdoor scenes, high-resolution DSLR images and low-resolution multi-camera video	Laser scans
ScanNet (Dai et al., 2017)	1,513 RGB-D scans of 707 indoor spaces, 2.5 million frames, semantic labels	Fused meshes from the RGB-D data

The Middlebury multi-view benchmark is separate from the two-view Middlebury stereo and optical flow benchmarks. Its objects are at most 16 cm across, and DTU extended its evaluation protocol (Aanæs et al., 2016). DTU’s official page lists 124 scenes, 80 of them used in the paper’s evaluation, and the paper adds an observability mask so that unobserved regions do not count against a method. Tanks and Temples evaluates complete SfM and MVS pipelines on frames sampled from video, with a distance threshold set per scene, from 3 mm to 30 mm (Table 1 of the authors’ preprint). ScanNet’s reference meshes come from the same kind of fusion pipeline that RGB-D methods use, so in our reading they are better treated as pseudo ground truth than as an independent measurement. Benchmark-specific protocols are described in multi-view stereo.

Evaluation Metrics

Reconstructions $\mathcal{R}$ and reference geometry $\mathcal{G}$ are compared as point sets, sampling meshes or volumes into points first. Every point gets the distance to the other set:

$$e_{r \to \mathcal{G}} = \min_{g \in \mathcal{G}} \lVert r - g \rVert, \qquad e_{g \to \mathcal{R}} = \min_{r \in \mathcal{R}} \lVert g - r \rVert.$$

Accuracy summarizes $e_{r \to \mathcal{G}}$: how close the reconstruction is to the truth. Completeness summarizes $e_{g \to \mathcal{R}}$: how much of the truth the reconstruction covers (Seitz et al., 2006). Each alone can be gamed, since a few careful points are accurate and a dense fog of points is complete, which is why the two are always reported together. Each benchmark summarizes them differently, as a percentile, a mean and median, or a fraction within a threshold (multi-view stereo gives the details).

Precision, recall and F-score. Tanks and Temples counts, at a distance threshold $d$, the precision $P(d)$, the share of reconstructed points with $e_{r \to \mathcal{G}} < d$, and the recall $R(d)$, the share of reference points with $e_{g \to \mathcal{R}} < d$, and combines them by their harmonic mean $F(d) = 2PR/(P + R)$ (Knapitsch et al., 2017); ETH3D ranks by the same F1 score. The harmonic mean goes to zero when either term does, so only a reconstruction that is both accurate and complete scores well at a strict threshold.

Chamfer distance averages the distances instead of thresholding them. Occupancy Networks report a Chamfer-L1 distance, the mean of the two mean distances (Mescheder et al., 2019):

$$\mathrm{CD}(\mathcal{R}, \mathcal{G}) = \frac{1}{2}\left(\frac{1}{|\mathcal{R}|}\sum_{r \in \mathcal{R}} e_{r \to \mathcal{G}} + \frac{1}{|\mathcal{G}|}\sum_{g \in \mathcal{G}} e_{g \to \mathcal{R}}\right).$$

Definitions vary: Fan et al. (2017), who used it as a training loss for point-set generation, summed squared distances in both directions, so values from different papers are comparable only under the same definition. DTU’s “overall” score in papers such as VGGT’s is the same average of mean accuracy and completeness. Tatarchenko et al. (2019) argued for the F-score over Chamfer distance because the Chamfer value depends on how far wrong points lie, not only on how many there are.

Before any of these, the reconstruction must be in the reference frame and scale. Seitz et al. aligned the reference to each submission by ICP, and Tanks and Temples registers the reconstructed camera poses to the reference ones; evaluation is then restricted to a cropping volume (Tanks and Temples) or an observability mask (DTU).

Example

The script reconstructs a unit sphere from synthetic depth maps. Six cameras sit 3 radii from the center on the coordinate axes; each renders an exact depth map by ray–sphere intersection, and each map is back-projected with the formula above. Reference points are 20,000 near-uniform samples of the true sphere; accuracy distances are exact, $\lvert \lVert \mathbf{X} \rVert - 1 \rvert$, and completeness distances come from a k-d tree. Part A adds views; part B adds depth noise with standard deviation $0.004\,Z^2$, growing with depth as for a stereo sensor, and 1% random “flying pixel” outliers; part C keeps only points that another view confirms, a simple version of the consistency check in multi-view stereo fusion. Distances are in thousandths of the radius, and the F-score threshold is $d = 0.02$.

import numpy as np
from scipy.spatial import cKDTree

H, W, f = 240, 320, 280.0
K = np.array([[f, 0, W / 2], [0, f, H / 2], [0, 0, 1]])
Kinv = np.linalg.inv(K)
u, v = np.meshgrid(np.arange(W) + 0.5, np.arange(H) + 0.5)
rays = np.stack([u, v, np.ones_like(u)], -1).reshape(-1, 3) @ Kinv.T  # K^-1 x, z = 1

def look_at(C):
    """World-to-camera pose (R, t), X_c = R X_w + t, for a camera at C facing the origin."""
    z = -C / np.linalg.norm(C)
    up = np.array([0.0, 0.0, 1.0]) if abs(z[1]) > 0.9 else np.array([0.0, 1.0, 0.0])
    x = np.cross(up, z); x /= np.linalg.norm(x)
    R = np.stack([x, np.cross(z, x), z])
    return R, -R @ C

def render_depth(R, t):
    """Depth Z along the optical axis of the unit sphere at the origin (NaN = no surface)."""
    C, d = -R.T @ t, rays @ R                       # camera centre, ray directions (world)
    b, c = d @ C, C @ C - 1.0                       # |C + s d|^2 = 1, solved for s
    a = np.einsum("ij,ij->i", d, d)
    disc = b ** 2 - a * c
    s = (-b - np.sqrt(np.maximum(disc, 0))) / a     # nearest intersection
    return np.where(disc > 0, s, np.nan).reshape(H, W)   # rays have z = 1, so Z = s

def back_project(Z, R, t):
    """Valid pixels to world points: X_w = R^T (Z K^-1 x - t)."""
    m = np.isfinite(Z).ravel()
    Xc = rays[m] * Z.ravel()[m, None]
    return (Xc - t) @ R

def fibonacci_sphere(n):                            # near-uniform samples of the true surface
    i = np.arange(n) + 0.5
    phi, z = np.pi * (3 - np.sqrt(5)) * i, 1 - 2 * i / n
    r = np.sqrt(1 - z ** 2)
    return np.stack([r * np.cos(phi), r * np.sin(phi), z], 1)

GT = fibonacci_sphere(20000)

def evaluate(P, tau=0.02):
    acc = np.abs(np.linalg.norm(P, axis=1) - 1.0)   # exact distance to the analytic sphere
    comp = cKDTree(P).query(GT)[0]                  # true surface -> nearest reconstructed point
    prec, rec = np.mean(acc < tau), np.mean(comp < tau)
    F = 2 * prec * rec / (prec + rec) if prec + rec > 0 else 0.0
    return dict(n=len(P), acc=acc.mean(), comp=comp.mean(), P=prec, R=rec, F=F,
                CD=(acc.mean() + comp.mean()) / 2)

def row(name, m):
    print(f"{name:<26s}{m['n']:>7d}{m['acc']*1000:>8.1f}{m['comp']*1000:>8.1f}"
          f"{m['P']*100:>7.1f}{m['R']*100:>7.1f}{m['F']*100:>7.1f}{m['CD']*1000:>8.1f}")

centres = 3.0 * np.array([[1, 0, 0], [-1, 0, 0], [0, 0, 1], [0, 0, -1], [0, 1, 0], [0, -1, 0]], float)
cams = [look_at(C) for C in centres]
depths = [render_depth(R, t) for R, t in cams]

def noisy(Z, rng, k=0.0, outliers=0.0):
    """Depth noise with std k*Z^2 (as for stereo) and a fraction of random 'flying pixels'."""
    Zn = Z + rng.normal(0, 1, Z.shape) * k * Z ** 2
    bad = np.isfinite(Z) & (rng.random(Z.shape) < outliers)
    Zn[bad] = rng.uniform(1.0, 4.0, bad.sum())
    return Zn

def consistent(Zs, eps):
    """Keep a pixel's point only if another view sees a depth within eps at its projection."""
    pts = []
    for i, (Ri, ti) in enumerate(cams):
        X = back_project(Zs[i], Ri, ti)
        ok = np.zeros(len(X), bool)
        for j, (Rj, tj) in enumerate(cams):
            if j == i:
                continue
            Xc = X @ Rj.T + tj
            q = Xc @ K.T
            c, r = (q[:, 0] / q[:, 2]).astype(int), (q[:, 1] / q[:, 2]).astype(int)
            inside = (Xc[:, 2] > 0) & (c >= 0) & (c < W) & (r >= 0) & (r < H)
            Zj = np.full(len(X), np.nan)
            Zj[inside] = Zs[j][r[inside], c[inside]]
            ok |= np.abs(Zj - Xc[:, 2]) < eps       # NaN (no observation) compares False
        pts.append(X[ok])
    return np.vstack(pts)

print(f"{'(distances in 1e-3 radii)':<26s}{'points':>7s}{'acc':>8s}{'comp':>8s}"
      f"{'P%':>7s}{'R%':>7s}{'F%':>7s}{'CD':>8s}")
print("A. noise-free depth, more views")
for n in [1, 2, 4, 6]:
    P = np.vstack([back_project(depths[i], *cams[i]) for i in range(n)])
    row(f"  {n} view(s)", evaluate(P))

print("B. 6 views, noisy depth")
rng = np.random.default_rng(0)
noisy_Z = [noisy(Z, rng, k=0.004) for Z in depths]
rng = np.random.default_rng(0)
dirty_Z = [noisy(Z, rng, k=0.004, outliers=0.01) for Z in depths]
for name, Zs in [("  sigma = 0.004 Z^2", noisy_Z), ("  + 1% outliers", dirty_Z)]:
    row(name, evaluate(np.vstack([back_project(Z, *c) for Z, c in zip(Zs, cams)])))

print("C. noisy + outliers, keep points confirmed by another view")
for eps in [0.08, 0.04, 0.02, 0.01]:
    row(f"  |dZ| < {eps}", evaluate(consistent(dirty_Z, eps)))

print("Seeds 0-9, 6 views + outliers, eps = 0.04:")
Fs = []
for seed in range(10):
    rng = np.random.default_rng(seed)
    Fs.append(evaluate(consistent([noisy(Z, rng, 0.004, 0.01) for Z in depths], 0.04))["F"])
print(f"  F-score range {min(Fs)*100:.1f}-{max(Fs)*100:.1f}%")

Output:

(distances in 1e-3 radii)  points     acc    comp     P%     R%     F%      CD
A. noise-free depth, more views
  1 view(s)                 30776     0.0   478.7  100.0   30.9   47.3   239.4
  2 view(s)                 61552     0.0    73.3  100.0   61.9   76.4    36.6
  4 view(s)                123104     0.0    16.0  100.0   90.2   94.8     8.0
  6 view(s)                184656     0.0     3.4  100.0  100.0  100.0     1.7
B. 6 views, noisy depth
  sigma = 0.004 Z^2        184656     9.8     6.8   88.4  100.0   93.8     8.3
  + 1% outliers            184656    13.1     6.8   87.6  100.0   93.4     9.9
C. noisy + outliers, keep points confirmed by another view
  |dZ| < 0.08              101182     6.7    18.5   97.5   82.3   89.3    12.6
  |dZ| < 0.04               74994     5.4    25.1   99.2   77.3   86.9    15.3
  |dZ| < 0.02               46089     4.7    30.1   99.7   72.7   84.1    17.4
  |dZ| < 0.01               24826     4.4    36.9   99.8   64.7   78.5    20.7
Seeds 0-9, 6 views + outliers, eps = 0.04:
  F-score range 86.7-86.9%

With exact depth every point is accurate, and the views decide completeness. A camera 3 radii away sees a cap covering a third of the sphere, so one view reaches 31% recall and two opposite views 62%; the six axis views cover everything, and the remaining mean completeness distance of 0.0034 reflects the spacing of the pixel samples. Noise moves the problem to the other side: recall stays at 100%, but 12% of points now lie farther than the threshold. The outliers raise the mean accuracy distance by a third, from 0.0098 to 0.0131, while precision drops by less than one point, which is Tatarchenko et al.’s argument in miniature: averaged distances respond to how far wrong points lie, thresholded counts only to how many there are.

The consistency check trades one measure for the other. Tightening $\varepsilon$ raises precision from 88% to almost 100%, but it also deletes every point that only one camera saw, near the edges of each view’s cap, so recall falls to 65% and the F-score falls with it. Whether filtering helps depends on the threshold $d$ and on how the dense points are meshed afterwards; here, with 1% outliers, keeping everything gives the best F-score. The seed loop shows that seed 0 is typical: over seeds 0–9 the F-score at $\varepsilon = 0.04$ stays between 86.7% and 86.9%.

The same trade-off appears in published tables. On DTU, VGGT’s authors list (CVPR 2025 paper, Table 2) Gipuma with the best accuracy in the table, 0.283, but the worst completeness, 0.873.

Applications

• Cultural heritage, film, and reverse engineering: Curless and Levoy (1996) list medicine, reverse engineering, and digital filmmaking among the uses of range scanning.
• Mapping and surveying: aerial and terrestrial photogrammetry and LiDAR for terrain, buildings, and cities.
• Robotics and driving: dense maps for navigation and obstacle detection, and object models for grasping (SLAM).
• Augmented and virtual reality: room-scale models for occlusion handling and physical interaction, uses that KinectFusion’s authors anticipated (Newcombe et al., 2011), and captured scenes for virtual tours rendered with neural methods.
• Scene understanding: ScanNet’s annotated reconstructions serve as training data for 3D object classification, semantic voxel labeling, and CAD model retrieval (Dai et al., 2017).
• Measurement and inspection of manufactured parts and structures.

Open Problems

• Hard materials: textureless, specular, and transparent surfaces defeat photo-consistency and active sensors alike.
• Dynamic and non-rigid scenes, where the rigid-scene assumption behind most pipelines fails.
• Geometry from neural rendering: recovering accurate surfaces from radiance fields and Gaussian sets, whose density need not concentrate on a surface.
• Generalization of feed-forward models to scenes, cameras, and scales unlike their training data, and to thousands of images, where optimization pipelines still operate.
• Evaluation: pseudo ground truth, unobserved regions, different Chamfer conventions, and thresholds chosen per benchmark make results hard to compare across papers.