3D Foundation Models

3D foundation models are large networks that take one or more ordinary photographs, with no known camera intrinsics or poses, and regress their 3D geometry in a single feed-forward pass: a 3D point for every pixel (a pointmap), depth maps, camera parameters and, in some models, correspondences or point tracks. They replace, or initialize, the chain of feature matching, structure from motion, bundle adjustment and multi-view stereo with a transformer trained on millions of images with 3D labels. The line runs from DUSt3R (CVPR 2024), which predicts pointmaps for a pair of images, and its matching variant MASt3R (ECCV 2024), to multi-view networks such as Fast3R and VGGT, online models such as Spann3R and CUT3R, and monocular geometry models such as MoGe (2024–2025). The models are also called feed-forward 3D reconstruction or pointmap regression models; none of the DUSt3R, MASt3R, VGGT, Fast3R or CUT3R papers calls itself a foundation model, and the label is applied by analogy with the foundation models of language and 2D vision.

This article is a hub. DUSt3R carries the pointmap formulation, the confidence-aware loss and global alignment; Depth Anything and MiDaS carry monocular relative depth; and 3D reconstruction carries the task and its benchmarks. What this article adds is the family as a whole, its training data, and the evaluation conventions that decide what its reported numbers mean, with a worked example. All papers were read in the CVF or ECVA open-access copies, with their supplements where named; section, table and appendix numbers are from those copies.

Definition

A 3D foundation model, in the sense used here, is a single network $f$ that maps a set of $N \ge 1$ images $(I_i)_{i=1}^{N}$ of a static scene, with unknown cameras, to per-pixel 3D geometry expressed in one shared coordinate frame, and possibly to the cameras themselves:

$$f\big((I_i)_{i=1}^{N}\big) = (X_i, C_i, D_i, g_i)_{i=1}^{N},$$

with $X_i \in \mathbb{R}^{H \times W \times 3}$ a pointmap, $C_i$ a per-pixel confidence or uncertainty, $D_i$ a depth map and $g_i$ the camera intrinsics and extrinsics; which outputs a model has differs. Three properties together set the family apart from earlier learned geometry:

• No camera input. Earlier learned multi-view networks such as DeMoN and DeepV2D needed the ground-truth intrinsics as input (DUSt3R paper, §2). These models take only images.
• Feed-forward geometry. The 3D output comes from one network pass, without per-scene optimization, although some models add an optimization step for many views (DUSt3R’s global alignment) or as an optional refinement (VGGT with bundle adjustment).
• Broad training. One model is trained on a mixture of many datasets with 3D labels (8 for DUSt3R, 17 named for VGGT, 32 for CUT3R) and applied zero-shot to benchmarks it was not trained on.

Intuition

A classical pipeline estimates cameras first and geometry second: match keypoints, solve for poses, then triangulate. A pointmap network inverts the order. It predicts, for each pixel, where its surface point lies in the frame of the first image, and leaves the cameras implicit in that output. Where the pixels of a second image land encodes that camera’s pose; how the points of one image spread across its pixels encodes its focal length. Cameras are then read off the pointmaps, or predicted by a separate head, and the network has to learn the geometry from data rather than from a hand-built solver.

The price is that the output has no fixed scale and no fixed position in the world. A single image cannot fix metric scale, and a set of images fixes only the shape up to a similarity transform unless the training data were metric. Every reported number therefore depends on how the prediction was aligned to the ground truth before it was scored, which is why evaluation conventions take up a large part of this article.

Formal Definition

Pointmaps and frames. A pointmap assigns a 3D point to every pixel, $X_{u,v} = K^{-1} D_{u,v} [u, v, 1]^\top$ in the camera’s own frame for intrinsics $K$ and depth $D$. DUSt3R writes $X^{n,m}$ for image $n$’s pointmap in camera $m$’s frame and predicts $X^{1,1}$ and $X^{2,1}$ for a pair (the DUSt3R article gives the equations). VGGT and Fast3R keep the same convention for $N$ images: every pointmap is in the frame of the first camera, which VGGT takes as the world frame and whose extrinsics it sets to the identity (VGGT paper, §3.1 and §3.3). CUT3R outputs, for each new frame, a pointmap in its own camera frame and one in the world frame, plus the camera-to-world transform (CUT3R paper, §3.1).

Normalization and invariance. Because a scene can be rescaled or moved without changing its images, training needs a canonical choice. DUSt3R divides both prediction and ground truth by the mean distance of their valid points to the origin. MASt3R keeps the prediction unnormalized when the ground truth is metric, setting the predicted normalization to the ground truth’s, so that 10 of its 14 training datasets teach metric scale (MASt3R paper, §3.1 and §4.1). VGGT normalizes only the ground truth, by the average distance of its points to the origin, and trains the network to output that normalization; its authors found that also normalizing the predictions, as DUSt3R does, was neither necessary nor helpful and made training less stable (VGGT supplement, App. B and F). MoGe, a monocular model, predicts an affine-invariant pointmap, defined up to a scale and a 3D shift; with the principal point at the image centre the shift reduces to one along the optical axis, and focal length and that shift are recovered afterwards by a small optimization of the reprojection error (MoGe paper, §3.1).

Loss. Most of these models inherit DUSt3R’s confidence-weighted regression loss, $\sum_i C_i \ell_i - \alpha \log C_i$, with $\ell_i$ the per-pixel 3D error. VGGT adds a gradient term to its depth and pointmap losses, a Huber loss on the cameras and a tracking loss weighted 0.05 (VGGT supplement, Eq. 1). CUT3R uses MASt3R’s metric-aware version of the regression loss and an L2 loss on the pose quaternion and the normalized translation (CUT3R paper, Eqs. 7–8). MoGe aligns the prediction to the ground truth with a solver for the optimal scale and shift under a truncated, depth-weighted L1 error, its “ROE” solver, before computing the loss (MoGe paper, §3.2).

Properties

• Over-complete outputs. Depth and cameras determine the pointmaps, and pointmaps determine the cameras up to a Perspective-n-Point solve. VGGT predicts all of them anyway and reports that training on all of them improves the pointmaps (ablation in its Table 6), and that at test time pointmaps built from its depth and camera heads were more accurate than its pointmap head on ETH3D (Table 3).
• A distinguished first image. The output frame is that of the first image. VGGT is designed to be permutation equivariant for all frames but the first, which gets its own learned camera and register tokens (§3.1 and §3.3); Fast3R always embeds the first image with the first index embedding, since it supplies the frame of the global head.
• Scale. Models trained without metric normalization (DUSt3R, VGGT) predict geometry up to scale; models trained with it (MASt3R, CUT3R) predict metric pointmaps where their training data were metric. Monocular affine-invariant models (MoGe) also leave a shift open.
• Cost with many images. A pairwise model needs a network pass per image pair and an optimization to merge them; VGGT’s authors report that DUSt3R ran out of memory beyond 32 frames in their comparison (Fig. 3). One-pass models attend over all images at once. VGGT’s backbone took 0.04 s and 1.88 GB for one frame and 8.75 s and 40.63 GB for 200 frames of $336 \times 518$ pixels on one H100 GPU (VGGT supplement, Table B). Online models keep a fixed-size state and process frames one at a time.
• Confidence. Most models output a per-pixel confidence learned without direct supervision, which downstream steps use to weight or filter points; Fast3R, for example, runs RANSAC-PnP only on the 15% of points with the highest confidence (Fast3R paper, §4.2).

Examples

Pairwise: DUSt3R and MASt3R

DUSt3R encodes two images with a shared vision transformer (ViT-Large), exchanges information between two transformer decoders by cross-attention, and regresses $X^{1,1}$, $X^{2,1}$ and their confidences; it is initialized from an off-the-shelf CroCo model, pretrained by cross-view completion. Focal lengths, relative poses and pixel matches are read off the pointmaps, and many images are merged by a global alignment that optimizes one similarity transform per pair. MASt3R starts from the DUSt3R checkpoint, adds a head of dense local features trained for matching, and predicts metric pointmaps where its data allow. The DUSt3R article covers both models, their read-outs and their results.

Multi-view in one pass: Fast3R and VGGT

Fast3R (Yang et al., CVPR 2025) generalizes DUSt3R to $N$ images. A ViT-Large encoder initialized from DUSt3R encodes each image; a fusion transformer, a ViT-Large trained from scratch, attends over the patch tokens of all images together; and two DPT heads output a global pointmap, in the first image’s frame, and a local pointmap, in each image’s own frame (§3–4). Images are told apart by index embeddings drawn at random from a pool of 1,000 during training, an adaptation of position interpolation from language models, so that a model trained with 20 views can be run on many more. The paper reports processing 1,500 images in one forward pass (§6). For reconstruction it aligns each image’s local pointmap to the global one with ICP, which its ablation found more accurate than the global pointmap alone on 7-Scenes and NRGBD, with little difference on DTU (Table 5).

VGGT (Visual Geometry Grounded Transformer; Wang et al., CVPR 2025, from Oxford’s Visual Geometry Group and Meta AI) predicts, for one to hundreds of images, each camera’s intrinsics and extrinsics, a depth map, a pointmap and features for point tracking. Each image is split into patch tokens by DINOv2 (the main text says “DINO” and cites DINOv2; the supplement names DINOv2), and one camera token and four register tokens are appended per image. The backbone has 24 blocks, each with a frame-wise and a global self-attention layer of width 1024 and 16 heads, which the authors call Alternating-Attention; it has no cross-attention layers. A camera head of four self-attention layers and a linear layer predicts a rotation quaternion, a translation and the field of view, assuming the principal point at the image centre; a DPT head predicts depth, pointmaps, their uncertainties and tracking features, and a CoTracker2-style module turns the features into tracks (§3; supplement App. B). The model has about 1.2 billion parameters. In the authors’ ablation on ETH3D, Alternating-Attention beat global self-attention only and cross-attention at the same parameter count (Table 5).

Online: Spann3R and CUT3R

Online models process a stream of images and update a memory as they go, instead of seeing all images at once. Spann3R (Wang and Agapito, 2024) is described by CUT3R’s authors as extending DUSt3R to a varying number of images with an added spatial memory, operating online; its paper was not read for this article, and only its official evaluation code is used below. CUT3R (Continuous Updating Transformer for 3D Reconstruction; Wang et al., CVPR 2025) keeps a persistent state of 768 tokens. Each incoming image, encoded by a ViT-Large initialized from DUSt3R’s encoder, interacts with the state through two ViT-Base decoders that update it and read from it at once, and heads output the frame’s pointmaps in its own and the world frame and its pose. Querying the state with a virtual camera’s raymap, a 6-channel image of ray origins and directions, without updating it, makes the model predict pointmaps for views it has not seen (§3.1–3.2). It predicts metric pointmaps and handles dynamic scenes, for which part of its training data are videos of moving content.

Single image: MoGe and depth models

MoGe (Wang et al., CVPR 2025) predicts an affine-invariant pointmap from one image, with a DINOv2-pretrained ViT encoder and a convolutional decoder, plus a mask of pixels with undefined geometry such as sky (§3.2 and §4); focal length and depth follow from the pointmap. Its authors present it as a possible “foundational model for monocular geometry” (§1). The monocular relative-depth models covered in Depth Anything and MiDaS predict affine-invariant inverse depth rather than 3D points, so that a camera model is needed to turn their output into geometry; monocular depth estimation surveys them with the metric-depth models. DUSt3R and VGGT also accept a single image: DUSt3R by feeding it twice, VGGT directly (VGGT supplement, App. F).

Training Data

All of these models are trained with full 3D supervision on mixtures of datasets with depth or 3D labels, and their mixtures largely overlap:

Model (paper)	Training data as listed
DUSt3R (§4)	8 datasets: Habitat, MegaDepth, ARKitScenes, Static Scenes 3D, BlendedMVS, ScanNet++, CO3D-v2 and Waymo; 8.5M image pairs
MASt3R (§4.1)	14 datasets: those of DUSt3R plus Map-free, WildRgb, Virtual KITTI, Unreal4K, TartanAir and an internal dataset; 10 with metric ground truth
Fast3R (§4)	6 datasets from DUSt3R’s mixture: CO3D, ScanNet++, ARKitScenes, Habitat, BlendedMVS and MegaDepth
VGGT (supplement App. B)	17 named (“including”): Co3Dv2, BlendMVS, DL3DV, MegaDepth, Kubric, WildRGB, ScanNet, HyperSim, Mapillary, Habitat, Replica, MVS-Synth, PointOdyssey, Virtual KITTI, Aria Synthetic Environments, Aria Digital Twin, and a synthetic dataset of artist-created assets similar to Objaverse
CUT3R (§3.4 and supplement §1)	32 datasets, synthetic and real, static and dynamic, including some with only camera labels (RealEstate10K) or only single views (Synscapes); the supplement’s Table 1 has 33 rows, with ARKitScenes and ARKitScenes-HighRes listed separately
MoGe (§3.3)	21 public datasets, about 9 million frames

The labels come from sensors, synthetic renderers or structure from motion; VGGT’s supplement names all three sources, so part of the “ground truth” is itself the output of a classical pipeline. VGGT was trained for 160,000 iterations on 64 A100 GPUs over nine days, with 2 to 24 frames per scene and images resized to 518 pixels on the longer side; its authors describe their mixture as broadly comparable to MASt3R’s in size and diversity (supplement App. B). Mixtures this broad overlap with test benchmarks: DUSt3R notes that ScanNet’s training split is part of its Habitat data, and VGGT’s supplement notes that MegaDepth scenes used by DUSt3R and MASt3R appear in the Image Matching Challenge benchmark, and that VGGT adopted DUSt3R’s MegaDepth split there and left the corresponding ScanNet scenes out of training for its ScanNet matching test.

Evaluation Conventions

A predicted pointmap is in the first camera’s frame and, for most models, at an unknown scale. Before a 3D error, a depth error or a pose error is computed, the prediction is aligned to the ground truth, and each benchmark tradition does this differently. The conventions below are those of the papers and of the official evaluation code, as of November 2026.

Camera pose. Relative rotation and translation accuracy over all image pairs (RRA, RTA at a threshold such as 15°) and the area under the accuracy curve up to 30° use angles only, so they need no scale alignment. DUSt3R calls the latter mAA(30) and VGGT AUC@30; both define it on the smaller of the rotation and translation errors of each pair, and VGGT’s Table 1 lists DUSt3R with the same values as DUSt3R’s own Table 2 (76.7 on CO3Dv2 and 67.7 on RealEstate10K), so the two names denote one measure here. Trajectory metrics (ATE, relative pose error) are computed after a 7-degree-of-freedom similarity (Sim(3)) alignment to the ground-truth trajectory (CUT3R §4.2, following MonST3R); see absolute trajectory error.

Depth maps. Depth is read as the $z$ coordinate of the pointmap in each camera’s frame, or from a depth head.

• Per-image median scaling. DUSt3R’s multi-view depth table (Table 3, left) normalizes its predictions with the medians of the predicted and ground-truth depths “as advocated in” the Robust Multi-View Depth benchmark (Schröppel et al., 2022), and reports the absolute relative error (rel, %) and the inlier ratio $\tau$, the share of pixels with $\max(\hat d/d, d/\hat d) < 1.03$. The benchmark’s code (alignment="median") computes the ratio per depth map. CUT3R’s single-frame depth evaluation also uses per-frame median scaling, which it attributes to DUSt3R.
• Per-sequence scale. CUT3R’s video depth evaluation (Table 2) aligns all frames of a sequence with one scale, to measure consistency between frames as well as per-frame quality. In its official code (eval/video_depth, option scale) the scale starts from the ratio of means and is refined by ten Weiszfeld steps on the absolute error, an L1 fit; a scale-and-shift option and a metric option (no alignment, for models that predict metric depth) exist beside it.
• Affine alignment. Relative-depth models are scored after a scale and shift fit, which the MiDaS and monocular depth articles describe; MoGe evaluates both scale-invariant and affine-invariant pointmaps, with the relative point error $\mathrm{Rel}^p = \lVert \hat p - p \rVert_2 / \lVert p \rVert_2$ and the inlier share $\delta_1^p$ of points with $\lVert \hat p - p \rVert_2 / \min(\lVert p \rVert_2, \lVert \hat p \rVert_2) < 0.25$ (§4.1).

Point clouds. The reconstruction is compared with reference geometry by accuracy (distance from each predicted point to its nearest reference point) and completeness (the reverse), as defined in the 3D reconstruction and Chamfer distance articles. How the prediction reaches the reference frame differs:

• DTU with ground-truth cameras. DUSt3R brings its reconstruction into DTU’s frame by fixing the camera parameters in its global alignment to the ground truth (§4.5); MASt3R triangulates its matches with the ground-truth cameras, used only for that step (§4.5). Both report DTU’s official accuracy, completeness and overall in millimetres; the DTU article gives the official protocol. VGGT’s main paper reports DTU without ground-truth cameras but does not say how its predictions are brought into DTU’s frame.
• Similarity alignment. VGGT’s point-map evaluation on ETH3D samples 10 frames per scene, aligns the predicted cloud to the ground truth with the Umeyama algorithm, a closed-form least-squares similarity over corresponding points (the Kabsch algorithm with a scale), filters invalid points with the official masks, and reports accuracy, completeness and Overall, their average, which it calls the Chamfer distance (§4.3). These are distances, not the thresholded F1 of ETH3D’s own multi-view benchmark.
• Median scale and shift, then ICP. Spann3R’s and CUT3R’s official code for 7-Scenes and Neural RGB-D (NRGBD) uses the same steps (Regr3D_t_ScaleShiftInv with gt_scale=True). In the first camera’s frame it subtracts from each cloud the median of all its $z$ values; it scales the prediction by the ratio of the two clouds’ median distances to their coordinate-wise median points; it adds the ground truth’s median $z$ back to both; and it then runs point-to-point ICP in Open3D from the identity, with a maximum correspondence distance of 0.1 (100 for DTU), a rigid fit without scale. Accuracy and completeness are reported as means and medians, with normal consistency. Spann3R’s code keeps every 20th frame of 7-Scenes and every 40th of NRGBD at $224 \times 224$ pixels; CUT3R’s code keeps every 200th and every 500th, the sparse “3 to 5 frames per scene” and “2 to 4 frames” of its paper (§4.3), and at its 512-pixel setting scores only the central $224 \times 224$ pixels of each view. Fast3R uses Spann3R’s frame skips (Table 3). The same benchmark name thus covers different frame samplings. 7-Scenes was introduced by Shotton et al. (2013) for RGB-D camera relocalization and is described in visual localization; NRGBD is the scene collection of Azinović et al. (2022).

Example: what the alignment convention reports

The script below scores simulated pointmap predictions under the four conventions above. No network is trained or loaded. It ray-casts depth images ($96 \times 72$ pixels, focal length 60 pixels) of a synthetic room, 5 m by 4 m by 2.6 m with a box and a sphere, from five cameras on a 2.4 m sideways sweep 0.4 m in front of one wall, 1.4 m high and pitched 15° down, turning from 25° to −25° in yaw, so that the relative rotations to the first camera are 13° to 50° and the baselines 0.6 to 2.4 m. Cameras use OpenCV axes ($x$ right, $y$ down, $z$ forward); depth is the $z$ coordinate; poses map each camera’s frame to the first camera’s, the frame in which DUSt3R, VGGT and Fast3R predict. A prediction is a depth map and a pose per view, back-projected with the true intrinsics into the first camera’s frame and multiplied by an unknown scale of 0.4. Five predictions each contain one known error: none, 2% log-normal depth noise, a different depth scale for each view (factors 0.90 to 1.10), a yaw error growing by 1.5° per view (pose drift with exact depths), and 3% of pixels pushed to 1.5 times their depth (flying pixels).

The conventions follow the official code read for this article: per-view median scaling (robustmvd), one Weiszfeld scale per sequence (CUT3R eval/video_depth), median shift and scale followed by ICP with Open3D v0.18.0’s defaults (Spann3R and CUT3R eval/mv_recon; ICP is reimplemented in NumPy, normal consistency is left out), and Umeyama similarity alignment (VGGT’s ETH3D protocol; the paper does not say which points enter the fit, and the example uses every pixel with its ground-truth point). Distances are in centimetres. It runs with NumPy 2.3.5 and SciPy 1.16.3 in about 9 seconds on a laptop CPU, and gives identical output on repeated runs. Before the run, the exact prediction was checked to score zero under all four conventions, and the ICP and Umeyama functions to recover a known rigid and similarity offset of the exact cloud.

import sys
import numpy as np
from scipy.spatial import cKDTree

W, H, F = 96, 72, 60.0
K = np.array([[F, 0, W / 2], [0, F, H / 2], [0, 0, 1.0]])
u, v = np.meshgrid(np.arange(W) + 0.5, np.arange(H) + 0.5)
RAYS = np.stack([(u - W / 2) / F, (v - H / 2) / F, np.ones_like(u)], -1).reshape(-1, 3)  # z = 1


# ---------- scene: a room (metres, z up) with a box and a sphere; depth images by ray casting ----------
def cast(C, d):
    """Distance along rays C + lam * d (world frame) to the first surface."""
    lo, hi = np.array([-2.5, -2.0, 0.0]), np.array([2.5, 2.0, 2.6])
    d = np.where(d == 0, 1e-12, d)
    lam = np.min(np.where(d > 0, (hi - C) / d, (lo - C) / d), axis=1)            # room, from inside
    a, b = (np.array([-0.6, 0.6, 0.0]) - C) / d, (np.array([0.4, 1.4, 0.8]) - C) / d
    t0, t1 = np.max(np.minimum(a, b), 1), np.min(np.maximum(a, b), 1)
    lam = np.where((t0 < t1) & (t0 > 0) & (t0 < lam), t0, lam)                    # box
    oc = C - np.array([1.2, 0.9, 0.5])
    bq, cq, aq = np.sum(d * oc, 1), oc @ oc - 0.45 ** 2, np.sum(d * d, 1)
    disc = bq ** 2 - aq * cq
    ts = (-bq - np.sqrt(np.maximum(disc, 0))) / aq
    return np.where((disc > 0) & (ts > 0) & (ts < lam), ts, lam)                  # sphere


def rot_z(deg):
    c, s = np.cos(np.radians(deg)), np.sin(np.radians(deg))
    return np.array([[c, -s, 0], [s, c, 0], [0, 0, 1.0]])


def camera(x, yaw, pitch=15.0):
    """Camera-to-world rotation (OpenCV axes: x right, y down, z forward) and centre."""
    fwd = rot_z(yaw) @ np.array([0, np.cos(np.radians(pitch)), -np.sin(np.radians(pitch))])
    right = np.cross(fwd, [0, 0, 1.0]); right /= np.linalg.norm(right)
    down = np.cross(fwd, right)
    return np.stack([right, down, fwd], 1), np.array([x, -1.6, 1.4])


CAMS = [camera(x, yaw) for x, yaw in zip([-1.2, -0.6, 0.0, 0.6, 1.2], [25, 12, 0, -12, -25])]
NV = len(CAMS)
DEPTH = [cast(C, RAYS @ R.T) for R, C in CAMS]                    # depth images (z), one per view
R0, C0 = CAMS[0]
REL = [(R0.T @ R, R0.T @ (C - C0)) for R, C in CAMS]              # camera v -> camera 0 (true)


def pointmap(depth, Rv0, tv0):
    """Back-project a depth image with the true K and move it into camera 0's frame."""
    return (RAYS * depth[:, None]) @ Rv0.T + tv0


GT = [pointmap(DEPTH[i], *REL[i]) for i in range(NV)]


# ---------- simulated predictions: per-view depth and pose, unknown global scale ----------
def predict(cond, rng, scale=0.4):
    depth, rel = [d.copy() for d in DEPTH], [(R.copy(), t.copy()) for R, t in REL]
    if cond == "depth noise 2%":
        depth = [d * np.exp(rng.normal(0, 0.02, d.shape)) for d in depth]
    elif cond == "per-view scale":
        depth = [d * k for d, k in zip(depth, [1.00, 0.94, 1.06, 0.90, 1.10])]
    elif cond == "pose drift":
        # camera v rotated by a yaw error of 1.5 v degrees about its own centre (world vertical axis)
        rel = []
        for i, (R, C) in enumerate(CAMS):
            Rw = rot_z(1.5 * i) @ R
            rel.append((R0.T @ Rw, R0.T @ (C - C0)))
    elif cond == "flying pixels 3%":
        for d in depth:
            idx = rng.choice(d.size, int(0.03 * d.size), replace=False)
            d[idx] *= 1.5
    depth = [scale * d for d in depth]
    rel = [(R, scale * t) for R, t in rel]
    return depth, [pointmap(depth[i], *rel[i]) for i in range(NV)]


# ---------- convention 1: per-view median scaling of depth (robustmvd alignment="median") ----------
def per_view_median(pred_depth):
    rel, tau = [], []
    for p, g in zip(pred_depth, DEPTH):
        m = g > 0
        p = p * np.median(g[m]) / np.median(p[m])
        rel.append(100 * np.mean(np.abs(p[m] - g[m]) / g[m]))
        tau.append(100 * np.mean(np.maximum(g[m] / p[m], p[m] / g[m]) < 1.03))
    return np.mean(rel), np.mean(tau)


# ---------- convention 2: one scale per sequence (CUT3R eval/video_depth, align "scale") ----------
def per_sequence_scale(pred_depth):
    p, g = np.concatenate(pred_depth), np.concatenate(DEPTH)
    s = np.mean(g) / np.mean(p)
    for _ in range(10):                                              # Weiszfeld (IRLS on L1)
        w = 1.0 / (np.abs(s * p - g) + 1e-8)
        s = np.sum(w * p * g) / np.sum(w * p ** 2)
    p = s * p
    ratio = np.maximum(g / p, p / g)
    return 100 * np.mean(np.abs(p - g) / g), 100 * np.mean(ratio < 1.03), 100 * np.mean(ratio < 1.25)


# ---------- convention 3: Spann3R / CUT3R: median shift and scale, then ICP, then Acc / Comp ----------
def median_shift_scale(pred, gt):
    P, G = pred.copy(), gt.copy()
    mz_p, mz_g = np.median(P[:, 2]), np.median(G[:, 2])               # joint median depth over views
    P[:, 2] -= mz_p; G[:, 2] -= mz_g
    sc = lambda X: np.median(np.linalg.norm(X - np.median(X, 0), axis=1))
    P *= sc(G) / np.clip(sc(P), 1e-3, 1e3)
    P[:, 2] += mz_g; G[:, 2] += mz_g
    return P, G


def umeyama(P, Q, with_scale=True):
    """s, R, t minimizing sum ||s R p + t - q||^2."""
    mp, mq = P.mean(0), Q.mean(0)
    Hm = (P - mp).T @ (Q - mq) / len(P)
    U, S, Vt = np.linalg.svd(Hm)
    d = np.sign(np.linalg.det(Vt.T @ U.T))
    R = Vt.T @ np.diag([1, 1, d]) @ U.T
    s = (S[0] + S[1] + d * S[2]) / np.mean(np.sum((P - mp) ** 2, 1)) if with_scale else 1.0
    return s, R, mq - s * R @ mp


def icp_point_to_point(P, G, thr=0.1, iters=30, eps=1e-6):
    """Open3D v0.18 registration_icp, point-to-point, no scaling, default criteria."""
    tree = cKDTree(G)
    def match(X):
        dist, idx = tree.query(X, distance_upper_bound=thr)
        ok = np.isfinite(dist)
        return ok, idx, (ok.mean(), np.sqrt(np.mean(dist[ok] ** 2)) if ok.any() else 0.0)
    ok, idx, res = match(P)
    for _ in range(iters):
        _, R, t = umeyama(P[ok], G[idx[ok]], with_scale=False)
        P = P @ R.T + t
        ok, idx, new = match(P)
        if abs(new[0] - res[0]) < eps and abs(new[1] - res[1]) < eps:
            break
        res = new
    return P


def acc_comp(P, G):
    da = cKDTree(G).query(P)[0]                                       # prediction -> ground truth
    dc = cKDTree(P).query(G)[0]                                       # ground truth -> prediction
    return da, dc


def spann3r_protocol(pred, gt):
    P, G = median_shift_scale(pred, gt)
    P = icp_point_to_point(P, G)
    da, dc = acc_comp(P, G)
    return 100 * np.array([da.mean(), np.median(da), dc.mean(), np.median(dc)])


# ---------- convention 4: VGGT ETH3D: Umeyama similarity on correspondences, then Acc / Comp ----------
def umeyama_protocol(pred, gt):
    s, R, t = umeyama(pred, gt)
    P = s * pred @ R.T + t
    da, dc = acc_comp(P, gt)
    corr = np.linalg.norm(P - gt, axis=1).mean()                      # per-pixel 3D error
    return 100 * np.array([da.mean(), dc.mean(), (da.mean() + dc.mean()) / 2, corr])


def evaluate(cond, seed):
    rng = np.random.default_rng(seed)
    pred_depth, pred_pts = predict(cond, rng)
    pv = per_view_median(pred_depth)
    ps = per_sequence_scale(pred_depth)
    P, G = np.vstack(pred_pts), np.vstack(GT)
    return pv, ps, spann3r_protocol(P, G), umeyama_protocol(P, G)


if __name__ == "__main__":
    CONDS = ["exact (scale 0.4)", "depth noise 2%", "per-view scale", "pose drift", "flying pixels 3%"]
    print("depth images: %d views of %d x %d; depth %.2f-%.2f m" % (NV, W, H, min(d.min() for d in DEPTH),
                                                                    max(d.max() for d in DEPTH)))
    print("\n1-2: depth maps (%)          per-view median    per-sequence scale")
    print("condition                     rel    tau1.03    rel   tau1.03  d1.25")
    res = {c: evaluate(c, 0) for c in CONDS}
    for c in CONDS:
        pv, ps, _, _ = res[c]
        print(f"{c:26s} {pv[0]:6.2f} {pv[1]:8.1f}  {ps[0]:7.2f} {ps[1]:7.1f} {ps[2]:6.1f}")
    print("\n3: median shift + scale, ICP (cm)    4: Umeyama Sim(3) (cm)")
    print("condition                  Acc  Acc med  Comp Comp med    Acc   Comp  Overall  corr")
    for c in CONDS:
        _, _, a, b = res[c]
        print(f"{c:24s} {a[0]:6.2f} {a[1]:7.2f} {a[2]:6.2f} {a[3]:7.2f}  {b[0]:6.2f} {b[1]:6.2f} {b[2]:7.2f} {b[3]:6.2f}")
    print("\nseeds 0-9, min-max")
    for c in ["depth noise 2%", "flying pixels 3%"]:
        r = [evaluate(c, s) for s in range(10)]
        rng_ = lambda xs: f"{min(xs):.2f}-{max(xs):.2f}"
        print(f"{c:18s} per-view rel {rng_([x[0][0] for x in r])}, per-seq rel {rng_([x[1][0] for x in r])}, "
              f"ICP Acc {rng_([x[2][0] for x in r])}, Umeyama Acc {rng_([x[3][0] for x in r])}, "
              f"corr {rng_([x[3][3] for x in r])}")

Output:

depth images: 5 views of 96 x 72; depth 1.09-4.13 m

1-2: depth maps (%)          per-view median    per-sequence scale
condition                     rel    tau1.03    rel   tau1.03  d1.25
exact (scale 0.4)            0.00    100.0     0.00   100.0  100.0
depth noise 2%               1.62     85.5     1.60    86.3  100.0
per-view scale               0.00    100.0     6.40    20.0  100.0
pose drift                   0.00    100.0     0.00   100.0  100.0
flying pixels 3%             2.74     97.0     1.50    97.0   97.0

3: median shift + scale, ICP (cm)    4: Umeyama Sim(3) (cm)
condition                  Acc  Acc med  Comp Comp med    Acc   Comp  Overall  corr
exact (scale 0.4)          0.00    0.00   0.00    0.00    0.00   0.00    0.00   0.00
depth noise 2%             3.89    3.04   2.33    2.20    3.85   2.30    3.08   4.72
per-view scale            14.05   10.11   6.93    3.83   14.21   8.28   11.24  19.24
pose drift                 3.33    2.50   2.37    2.15    3.68   2.69    3.18   7.45
flying pixels 3%           5.27    1.77   2.12    1.77    8.26   5.38    6.82  10.24

seeds 0-9, min-max
depth noise 2%     per-view rel 1.60-1.64, per-seq rel 1.58-1.60, ICP Acc 3.84-3.90, Umeyama Acc 3.83-3.88, corr 4.68-4.75
flying pixels 3%   per-view rel 2.53-2.88, per-seq rel 1.50-1.50, ICP Acc 5.08-5.43, Umeyama Acc 8.04-8.52, corr 10.02-10.53

What depth metrics cannot see. Per-view median scaling gives a perfect score to two predictions that are badly wrong as 3D reconstructions: the one whose views disagree in scale by up to 20% and the one whose cameras drift by up to 6°. Each view’s depth map is correct up to its own scale, and a per-view scale removes exactly that. A single scale per sequence exposes the scale inconsistency (rel 6.4%, only a fifth of pixels within 3%) but still scores the pose drift as perfect, because the depth maps of that prediction are exact; only the 3D conventions, which put all views in one frame, see it. This is the reason CUT3R gives for its per-sequence video depth protocol, and in our reading the reason a multi-view model’s depth table and its point-cloud table can rank models differently.

ICP against correspondence alignment. Under pose drift and under flying pixels, the ICP-based protocol reports a lower accuracy than Umeyama alignment (3.33 against 3.68 cm, and 5.27 against 8.26 cm). The two steps differ in what they fit. In our reading, Umeyama minimizes squared errors over all pixel correspondences, so the 3% of points pushed 50% deeper pull its scale and translation, and every inlier is then misplaced; the median scale ignores them, and ICP, which only matches points within 0.1 m, refits the rigid pose to the surfaces it can match, absorbing part of a drift as well. With flying pixels the median accuracy of the ICP protocol (1.77 cm) is a third of its mean (5.27 cm): the few outliers set the mean, not the median. Over seeds 0–9 the flying-pixel accuracy stays between 5.08 and 5.43 cm with ICP and between 8.04 and 8.52 cm with Umeyama; the pose-drift prediction has no random part.

What nearest-neighbour distances miss. Accuracy and completeness measure distance to the nearest surface point, not to the point the pixel should have landed on. Under pose drift the per-pixel error after the same Umeyama alignment is 7.45 cm, twice the accuracy of 3.68 cm; in our reading, the rotated views slide partly along the walls they image, which moves points without moving them off the surface. DUSt3R’s training loss and MoGe’s $\mathrm{Rel}^p$ are correspondence errors of this kind; the benchmark accuracy is not. Even the noise-only prediction shows the gap (4.72 against 3.85 cm).

A surprise in the depth rows. With flying pixels the per-view median rel (2.74%) is almost twice the per-sequence rel (1.50%), which the plan for this example did not predict. In our reading, which follows from the construction, moving 3% of depths up by half shifts each view’s median depth, so the median ratio rescales every inlier slightly too small; the L1 scale of the per-sequence fit is set by the 97% inliers and leaves them exact, so its rel equals the outliers’ share times their error, 1.5%. A convention chosen for robustness, the median, is thus not the most robust scale estimator here.

The example scores one scene with one kind of error at a time; real predictions mix them, and real benchmarks add masks, frame sampling and crops that change the numbers further.

Reported Results

Numbers are comparable only within one table, since each table fixes its protocol. All results below are as the authors report them.

Benchmark and protocol	Results as printed
CO3Dv2 and RealEstate10K, 10 random frames, AUC@30 (VGGT Table 1; RealEstate10K unseen by all methods)	VGGT feed-forward 88.2 and 85.3; with bundle adjustment 91.8 and 93.5; DUSt3R with global alignment 76.7 and 67.7; MASt3R 81.8 and 76.4; CUT3R 82.8 and 75.3; Fast3R 82.5 and 72.7
DTU, accuracy / completeness / overall in mm (VGGT Table 2)	without ground-truth cameras: VGGT 0.389 / 0.374 / 0.382, DUSt3R 2.677 / 0.805 / 1.741; with ground-truth cameras: MASt3R 0.403 / 0.344 / 0.374, GeoMVSNet 0.331 / 0.259 / 0.295
ETH3D, 10 frames, Umeyama alignment, accuracy / completeness / overall (VGGT Table 3)	VGGT from depth and camera heads 0.873 / 0.482 / 0.677, from its pointmap head 0.901 / 0.518 / 0.709; MASt3R with global alignment 0.968 / 0.684 / 0.826; DUSt3R with global alignment 1.167 / 0.842 / 1.005
7-Scenes, 3–5 frames per scene, median scale and ICP, mean accuracy / completeness (CUT3R Table 4)	CUT3R 0.126 / 0.154; DUSt3R with global alignment 0.146 / 0.181; MASt3R with global alignment 0.185 / 0.180; Spann3R 0.298 / 0.205
NRGBD, 2–4 frames per scene, same protocol (CUT3R Table 4)	CUT3R 0.099 / 0.076; MASt3R with global alignment 0.085 / 0.063; DUSt3R with global alignment 0.144 / 0.154; Spann3R 0.416 / 0.417

VGGT’s Tables 2 and 3 and CUT3R’s Table 4 print no unit; DUSt3R’s Table 3 gives the same DUSt3R DTU values in millimetres. Several details matter for reading them. VGGT’s runtime figures in Table 1, on one H100 GPU, are about 0.2 s for its feed-forward pass and 1.8 s with bundle adjustment, against about 7 s for DUSt3R and 9 s for MASt3R with global alignment and 10 s for VGGSfM v2. VGGT’s Fast3R row pairs two Fast3R variants: in Fast3R’s own Table 1, 82.5 on CO3Dv2 is the mAA(30) of Fast3R-no-outdoor, trained without BlendedMVS and MegaDepth, and 72.7 on RealEstate10K that of the full model, which scores 75.0 on CO3Dv2. On NRGBD the medians in CUT3R’s Table 4 rank the methods differently from the means: DUSt3R with global alignment has the smallest median accuracy and completeness (0.019 and 0.018), with the third-best means. In our reading this pattern, accurate typical points with a costly few, is the one the flying-pixel row of the example produces. VGGT’s DTU row and DUSt3R’s are both without ground-truth cameras, but MASt3R’s and GeoMVSNet’s are with them, so the paper’s comparison crosses settings, as its table marks. CUT3R, in its own comparison, reports that it runs online at about 17 frames per second against under 1 for the methods with global alignment (Table 4), and that revisiting the images with its final, frozen state improves the reconstruction (Table 5).

Documented Limits

The papers report these limits of their own models:

• Cameras and images outside the training distribution. VGGT does not support fisheye or panoramic images, and its reconstructions degrade under extreme input rotations (supplement App. F). VGGT, DUSt3R and MoGe all assume a principal point at the image centre.
• Dynamic scenes. VGGT handles minor non-rigid motion but fails under substantial deformation (App. F). DUSt3R’s global alignment assumes a static scene; CUT3R’s authors make this point when explaining why the alignment can harm moving objects (§4.1).
• Many views and long sequences. Fast3R’s views begin to drift for very large scenes when the number of views becomes extreme, more than about 300 in their observation, especially views with low confidence (§6). CUT3R may drift over very long sequences without a global alignment (§5). Memory grows with the number of images for one-pass models (VGGT supplement, Table B); pairwise models pay a quadratic number of passes.
• Accuracy. DUSt3R’s authors attribute their gap to calibrated multi-view stereo on DTU to regression being less accurate than sub-pixel triangulation with known cameras (§4.5). VGGT’s authors report that bundle adjustment on top of the feed-forward prediction still improves its poses, and conclude that there is still room for improvement (§4.1).
• Hallucinated structure. CUT3R’s predictions for unobserved views are deterministic regressions and can be blurry, especially for viewpoints far from the inputs (§5).
• Licences. As of November 2026 the official code of MASt3R, Spann3R and CUT3R is licensed CC BY-NC-SA 4.0 (their LICENSE files), which excludes commercial use, and the VGGT repository is under a custom “VGGT License” (v1 of 29 July 2025), which states that using or distributing its research materials means accepting the agreement and its acceptable use policy. For this article only VGGT’s LICENSE file was read from that repository.

Common Misconceptions

• “One network replaces structure from motion.” In the published comparisons the feed-forward output is often refined: DUSt3R and MASt3R need a global alignment for more than two images, VGGT reports its best pose results with bundle adjustment, and Fast3R aligns its local pointmaps to the global one with ICP before evaluation.
• “Lower Chamfer error means better cameras.” Accuracy and completeness are nearest-neighbour distances after an alignment chosen by the evaluator; they can hide pose drift that a correspondence error or a trajectory error would show, as the example’s pose-drift row does.
• “Depth results measure 3D quality.” Per-image median scaling forgives any per-view scale or pose error. For multi-view models only an evaluation in a shared frame tests consistency between views.
• “Same benchmark, same protocol.” 7-Scenes and NRGBD results are reported every 20th and 40th frame in Spann3R’s code and Fast3R’s paper, but every 200th and 500th frame in CUT3R’s code and paper. DTU numbers come with or without ground-truth cameras, and ETH3D numbers with Umeyama means or with the official thresholded F1.
• “Pointmaps are metric.” Only models trained with metric normalization on metric data, such as MASt3R and CUT3R, are trained to predict metric scale; DUSt3R and VGGT predict geometry up to scale, and their reported 3D errors come after a scale alignment. CUT3R’s video depth table shows the difference: without alignment its errors are larger than with a per-sequence scale on all three datasets (Table 2).

Where It Is Used

These models are used where camera calibration is unknown or classical matching fails: reconstructing scenes from a few photographs (3D reconstruction and multi-view stereo without calibrated cameras), estimating camera pose and intrinsics for image collections, relocalization (visual localization), and dense SLAM and visual odometry, where CUT3R reports camera trajectories on dynamic videos. Their features are reused for other tasks: VGGT’s authors fine-tuned its backbone inside a point tracking model, improving it on the TAP-Vid benchmarks, and adapted it to novel view synthesis without input camera parameters (§4.6). Their predicted cameras and points are also a common starting point for 3D Gaussian splatting and neural radiance fields, which need posed images; VGGT’s supplement names Gaussian splatting as one such integration of pointmaps (App. E).