3D Human Pose Estimation

3D human pose estimation recovers the positions in space of a person’s body joints, such as hips, knees, shoulders and wrists, from images. It extends 2D human pose estimation, which finds the same joints in the image plane, by one coordinate: depth. That coordinate is what a single photograph does not record. Many 3D skeletons project to the same 2D skeleton, so a method working from one image must bring in knowledge of how bodies are built and how they move, usually learned from motion-capture data. With two or more calibrated cameras the depth can instead be measured by triangulation. The field’s main lines follow from this: lifting detected 2D joints to 3D with a learned model, predicting 3D directly from the image (as coordinates or as volumetric heatmaps), using time in video, triangulating from several views, and fitting parametric body models that give a full surface mesh rather than a skeleton. Results are reported mostly in millimetres of joint error on a few laboratory and outdoor benchmarks, whose protocols differ in ways that change the numbers.

Problem Definition

The output is a set of $J$ joint positions $\mathbf{X}_j \in \mathbb{R}^3$, $j = 1, \dots, J$, for each person, on a fixed skeleton such as the 17-joint skeleton that most Human3.6M papers adopt (Pavllo et al., 2019). Three choices define what exactly is estimated:

• Coordinate frame. Monocular methods usually predict joints in the camera’s coordinate frame. Martinez et al. (2017) argued that asking a network to output a pose in an arbitrary world frame is unrealistic, because a rotation or translation of that frame leaves the image unchanged; in their ablation, dropping the camera frame raised the error from 67.5 to 101.1 mm (Table 5).
• Root-relative or absolute. Most monocular methods predict joints relative to a root joint, usually the pelvis, and leave out the person’s distance from the camera. Human3.6M’s own evaluation places the root at the origin of the prediction frame (Ionescu et al., 2014). Multi-view systems can measure absolute positions, and Iskakov et al. (2019) argue that absolute error is the more reasonable measure for them.
• Skeleton or surface. A skeleton of joints is the classical output. Body-model methods output the parameters of a mesh, from which joints are read off (see Body Models below).

Input and Output

• Input: one RGB image, a video, or synchronized images from several cameras with known camera parameters; in two-stage systems, the 2D joint detections of a 2D pose network, optionally with their confidence.
• Output: 3D joint coordinates in millimetres, root-relative or absolute; optionally a body mesh with shape and pose parameters; in video, one pose per frame.

Variants

• Monocular image. The hardest and most studied setting, because depth is not observed.
• Video. A sequence of frames constrains motion; temporal models smooth and disambiguate.
• Multi-view. Several calibrated, synchronized cameras, as in capture studios and the datasets’ own recording rigs. Iskakov et al. (2019) call multi-view estimation arguably the best way to obtain ground truth for monocular 3D pose estimation in the wild.
• Multi-person. Several people per image add detection, association and the relative depth of people, which root-relative single-person protocols ignore; Martinez et al. (2017) list estimating each person’s depth as the step their lifter would need for multiple people.
• Human mesh recovery. Estimating the parameters of a body model such as SMPL, which gives shape as well as pose.
• Depth sensors. With a depth camera the third coordinate is measured; the Kinect body tracker is summarized in the human pose estimation article.

Challenges

• Depth ambiguity. For a skeleton with known bone lengths, each bone seen in one image can point toward or away from the camera, so every bone is a binary choice; Lee and Chen (1985), as summarized by Martinez et al. (2017), framed single-view reconstruction as a binary decision tree that joint-angle constraints prune but rarely reduce to one solution. The overall scale is ambiguous too: a larger person further away projects to the same image (see the example).
• Scarce 3D ground truth. Accurate 3D labels come from motion capture in a laboratory. Martinez et al. (2017) point to this lack of 3D ground truth for images in the wild as what makes direct prediction from colour images hard; it is one reason that two-stage systems, trained on images for the 2D stage and on motion capture for the 3D stage, became common.
• Occlusion and foreshortening. Pavlakos et al. (2017) explain the large errors of a network that lifts 2D heatmaps without image features on “Sitting” and “Sitting Down” by self-occlusions that make 2D localization fail; Martinez et al. (2017) add that in those actions the legs are often aligned with the viewing direction, a strong foreshortening.
• Generalization. Lifters trained on laboratory poses fail on poses absent from the training data, such as people upside down, and on cropped bodies (Martinez et al., 2017).
• Inconsistent protocols. The same name, such as “Protocol #2”, means different procedures in different papers (see Evaluation Metrics).

Approaches

Early work: from 2D joints to 3D

The problem of inferring 3D joints from their 2D projections goes back at least to Lee and Chen (1985). Martinez et al. (2017) summarize the later classical routes: nearest-neighbour search in large databases of 3D poses, sparse combinations of learned pose bases, and priors on joint angles, often with a separate optimization that fitted a 3D model to 2D joints found by a convolutional network. Pavlakos et al. (2017) described such two-step pipelines, a network for 2D joints followed by optimization, as the best performers before their work, ahead of the end-to-end networks that regressed 3D coordinates from the image.

Lifting 2D joints: the simple baseline

Martinez et al. (2017) separated the task into 2D pose estimation and a learned map $f: \mathbb{R}^{2J} \to \mathbb{R}^{3J}$ from 2D joint coordinates to 3D, and asked how much of the error comes from each stage. Their lifter is a multilayer perceptron: a linear layer to 1,024 units, then two residual blocks, each made of two repetitions of linear layer, batch normalization, ReLU and dropout 0.5 wrapped in a residual connection, and a final linear layer to $3J$ outputs, with 4 to 5 million parameters and a max-norm constraint on the weights. Inputs and outputs are standardized, the 3D pose is centred on the hip and expressed in camera coordinates, and the 2D input comes from a stacked hourglass network pre-trained on MPII and optionally fine-tuned on Human3.6M.

The result surprised its authors. On Human3.6M under protocol #1 (MPJPE after aligning the root joint), with one model for all actions, the lifter scored 45.5 mm from ground-truth 2D joints, 67.5 mm from off-the-shelf hourglass detections and 62.9 mm from fine-tuned detections, against 71.9 mm for the volumetric network of Pavlakos et al. (Table 2). The authors concluded that lifting is easier than had been assumed and that a large part of the error of 3D systems comes from their 2D, visual analysis. Two further findings matter beyond the paper: a network trained on ground truth does not perform well on noisy 2D input, which batch normalization and dropout improve (Section 3.1), and the camera frame is essential (Table 5). In Table 5, removing batch normalization alone costs 21.0 mm (88.5 against 67.5 mm), removing dropout 3.9 mm and removing both 8.5 mm; the text summarizes this as 3 to 8 mm.

Martinez et al. tested the lifter, trained on ground-truth 2D, with Gaussian noise added to the 2D test input, following Moreno-Noguer: under protocol #2 its error rose from 37.10 mm without noise to 70.24 mm at a standard deviation of 20 pixels (Table 1).

Direct regression and volumetric heatmaps

The alternative keeps the image in the loop. Early end-to-end networks regressed the $3J$ coordinates directly with an L2 loss, which Pavlakos et al. (2017) found hard to learn, for the same reasons that pushed 2D pose estimation from coordinate regression to heatmap regression. They discretized the space around the subject into a volume per joint: the $x$–$y$ grid covers the person’s 2D bounding box in the image and the $z$ grid covers $[-1, 1]$ m of depth relative to the root, so the network predicts image coordinates plus metric depth relative to the root. The target for joint $n$ is a 3D Gaussian with $\sigma = 2$ voxels around its true voxel, trained with mean squared error, and the voxel with the highest response is the estimate. The network is a stack of hourglass modules.

At the highest resolution, $64 \times 64 \times 64$ voxels for 16 joints, the output has more than four million values. Pavlakos et al. therefore supervised the first hourglasses with coarse depth, using targets of size $64 \times 64 \times d$ with $d$ growing through $\{1, 2, 4, \dots, 64\}$ along the stack; $d = 1$ is a 2D heatmap. In their Human3.6M component analysis, which uses the ground-truth depth of the root joint, one hourglass scored 112.41 mm with coordinate regression and 85.82 mm with $d = 64$ volumes (Table 1); four hourglasses scored 75.06 mm stacked naively and 64.76 mm with coarse-to-fine supervision (Table 2). Their comparison with other methods estimates the root depth from the average skeleton size instead, and reports 71.90 mm MPJPE (Table 4) and 51.9 mm after Procrustes alignment (Table 5); these two numbers are the ones later papers quote. A decoupled variant, which lifts 2D heatmaps to 3D volumes without image features, scored 78.10 mm against 69.77 mm for the coarse-to-fine network of the same size (Table 3); Pavlakos et al. read this as evidence for keeping image evidence, while Martinez et al. (2017) read their own results as contradicting the premise that regressing 3D points directly is inherently hard.

Volumetric heatmaps can also be decoded by an expectation instead of an argmax, which makes the readout differentiable and sub-voxel; integral regression, covered in the heatmap regression article, takes this route, and so does the volumetric triangulation of Iskakov et al. (2019) below.

Temporal models

Video gives each frame a context. Pavllo et al. (2019) lifted sequences of 2D keypoints with a fully convolutional network over time (VideoPose3D). The input layer convolves the $2J$ coordinates of each frame over a temporal window of width $W$; $B$ residual blocks follow, each a 1D convolution with kernel $W$ and dilation $W^b$ in block $b$ and a convolution with kernel 1, with batch normalization, ReLU and dropout 0.25. With $W = 3$, $B = 4$ and $C = 1024$ channels the receptive field is $3^5 = 243$ frames, and the number of parameters grows linearly with $B$ while the receptive field grows exponentially. The default uses past and future frames; a causal variant sees only the past. Because consecutive frames are correlated, which biases batch normalization, training predicts a single frame per clip drawn from different video segments, and dilated convolutions are replaced by strided ones during training to skip unused states.

On Human3.6M with 2D keypoints from a fine-tuned cascaded pyramid network, the 243-frame model scored 46.8 mm MPJPE (protocol 1) and 36.5 mm P-MPJPE (protocol 2), against 51.8 and 40.0 mm for the same network with every kernel set to width 1, and 49.0 mm for the causal model (Table 1). From ground-truth 2D the model scored 37.2 mm (Table 3). Time also smooths the output: the mean per-joint velocity error (MPJVE) of the single-frame baseline, 11.6 on average, fell to 2.8 for the temporal model, a 76% reduction (Table 2). The same paper introduced back-projection for semi-supervised training: unlabelled video is lifted to 3D, a second network regresses the global trajectory, and the result is projected back with the camera intrinsics and compared with the input 2D keypoints, with a soft constraint matching mean bone lengths to the labelled subjects.

Multi-view triangulation

With $C$ calibrated cameras, each joint detected in each view gives two linear equations in the homogeneous joint position, and the direct linear transform solves the stacked $2C \times 4$ system by singular value decomposition. The weakness is that every view counts equally, including views in which the joint is occluded and the detection is wrong. RANSAC with a robust loss rejects such views but cuts the gradient to them.

Iskakov et al. (2019) made triangulation learnable in two ways. In algebraic triangulation, a 2D backbone (a ResNet-152 “simple baselines” network) produces heatmaps, a soft-argmax with inverse temperature $\alpha = 100$ turns them into 2D positions, and a small branch predicts a confidence $w_{c,j}$ for each camera and joint that multiplies the two rows of that view in the DLT system; the SVD is differentiable, so the whole pipeline trains end to end against 3D ground truth. In volumetric triangulation, the backbone’s feature maps (32 channels) are unprojected into a $64 \times 64 \times 64$ grid of side $L = 2.5$ m around the pelvis, which the algebraic method locates: each voxel centre is projected into every view and the features are bilinearly sampled there. The per-view volumes are aggregated by summation, by confidence-weighted summation, or by a softmax over views at each voxel, then refined by a V2V-style 3D convolutional network into 3D heatmaps, read out with a 3D soft-argmax. The 3D network can learn a pose prior and discard a wrong view, which the algebraic method cannot.

On Human3.6M, evaluated on every fifth frame of S9 and S11 with all available cameras, root-relative MPJPE was 27.4 mm for the authors’ RANSAC baseline, 26.9 mm for algebraic triangulation without confidences, 22.6 mm with them and 20.8 mm for volumetric triangulation with softmax or confidence aggregation (Table 1). The table’s averages give a 24% reduction from RANSAC to the volumetric methods (27.4 to 20.8 mm); the text describes it as “about 30%”. For absolute error they excluded scenes of S9 whose ground truth is shifted in 3D, an error that subtracting the pelvis hides, and reported 22.8 mm for RANSAC and 17.7 mm for the best volumetric model (Table 2), well under the 3.9 cm voxel. The same paper is a reminder of how different the settings are: the monocular results in its Table 1 range from 46.8 to 62.9 mm.

Body models: SMPL in outline

A skeleton says nothing about the body’s surface. SMPL (Loper et al., 2015) is a skinned, vertex-based body model learned from 1,786 high-resolution scans in many poses, with shape spaces learned from the CAESAR scans by principal component analysis. Its mesh has 6,890 vertices and its skeleton 23 joints; a pose is the axis-angle rotation of each joint plus the root, $3 \times 23 + 3 = 72$ parameters. Shape blend shapes, and pose blend shapes that are a linear function of the elements of the joints’ rotation matrices, are added to a template before linear blend skinning, and joint locations are regressed from the vertices, so joints follow the body’s shape. Because it is a standard skinned model, it runs in ordinary graphics engines.

Model-based methods estimate SMPL parameters from images. HMR (Kanazawa et al., 2018) regresses the shape (the first 10 shape coefficients), the pose and a camera end to end from an image, with a reprojection loss on 2D keypoints and an adversarial prior that judges whether parameters are realistic, so it can train from images with only 2D labels; it reported 56.8 mm reconstruction error on Human3.6M in its own protocol 2 (Table 1). SPIN (Kolotouros et al., 2019) puts an optimization-based fit (SMPLify) in the training loop: the network’s estimate initializes the fit, and the improved fit supervises the network, reaching 41.1 mm reconstruction error on Human3.6M (S9 and S11; the paper’s text does not name the protocol, and its Table 3 lists the result beside HMR’s protocol-2 56.8 mm) and 59.2 mm on the 3DPW test set without training on 3DPW (Tables 3 and 1). VIBE (Kocabas et al., 2020) extends regression of body models to video. HMR’s text gives the mesh as 6,980 vertices; the SMPL paper and SPIN give 6,890.

The SMPL model files are not open data. As of November 2026 the official website asks users to register and agree to a licence before downloading. The SMPL-Model License grants a personal, non-exclusive, free right for non-commercial scientific research, education and artistic projects only, forbids redistribution and any commercial use, including training networks for commercial use, and offers commercial licences separately. The same site points to successors with articulated hands (MANO, SMPL+H) and expressive faces (SMPL-X).

Example

The example below compares one view with two on synthetic people. A 17-joint skeleton with bone lengths in millimetres is posed by a random kinematic tree (heading, spine, neck, hips, knees, shoulders and elbows with angle limits), scaled by a random factor between 0.9 and 1.1 and placed near the origin. Two calibrated pinhole cameras ($f = 1000$ pixels, $1000 \times 1000$ images) stand 4.5 m away, 90° apart, and project the joints into their images; these 2D keypoints are the data. Part A shows the depth ambiguity of one view: each end bone (ankles, wrists, head) is moved to the other point on its camera-1 ray at the same distance from its parent, and the whole pose is scaled about the camera centre. Parts B and C compare lifting from camera 1 with triangulation from both cameras on 2,000 test poses, adding Gaussian noise of 0 to 20 pixels to the 2D test keypoints. The lifters are a mean-pose baseline, a 1-nearest-neighbour lookup, ridge regression and a two-layer perceptron, all trained on 20,000 poses to map root-centred 2D joints to root-relative 3D joints in camera-1 coordinates, as Martinez et al. (2017) do. Triangulation is the homogeneous DLT per joint. It runs with NumPy 2.3.5 and scikit-learn 1.7.2 (Python 3.13) and takes about 40 seconds, most of it training the perceptrons.

Two choices that the sources leave open are printed both ways. Procrustes alignment is called rigid by Martinez et al. (2017) and HMR but includes a scale in Pavlakos et al. (2017) and in VideoPose3D’s code, so PA-MPJPE is printed with and without scale. Whether the lifter sees noisy 2D in training also changes the comparison, so each lifter is trained once on clean 2D and once on 2D with a noise level drawn per pose from the test levels. The test poses are drawn from the training distribution and, in a second set, with both arms raised beyond the training range.

import os
os.environ["OPENBLAS_NUM_THREADS"] = "4"   # fixed thread count: same sums on every run
import warnings
import numpy as np
from sklearn.linear_model import Ridge
from sklearn.neighbors import KNeighborsRegressor
from sklearn.neural_network import MLPRegressor
from sklearn.preprocessing import StandardScaler

warnings.filterwarnings("ignore")                # the MLP stops at max_iter, by design

# 17 joints: parent, rest direction (Y up, person faces +Z), length (mm)
NAMES = ["pelvis", "r_hip", "r_knee", "r_ankle", "l_hip", "l_knee", "l_ankle", "spine",
         "thorax", "neck", "head", "l_shoulder", "l_elbow", "l_wrist", "r_shoulder",
         "r_elbow", "r_wrist"]
BONES = {1: (0, (-1, 0, 0), 130), 2: (1, (0, -1, 0), 450), 3: (2, (0, -1, 0), 440),
         4: (0, (1, 0, 0), 130), 5: (4, (0, -1, 0), 450), 6: (5, (0, -1, 0), 440),
         7: (0, (0, 1, 0), 230), 8: (7, (0, 1, 0), 250), 9: (8, (0, 1, 0), 110),
         10: (9, (0, 1, 0), 120), 11: (8, (1, 0, 0), 170), 12: (11, (0, -1, 0), 280),
         13: (12, (0, -1, 0), 250), 14: (8, (-1, 0, 0), 170), 15: (14, (0, -1, 0), 280),
         16: (15, (0, -1, 0), 250)}
PARENT = {j: b[0] for j, b in BONES.items()}

def rot(axis, deg):
    a = np.radians(deg); c, s = np.cos(a), np.sin(a)
    if axis == "x": return np.array([[1, 0, 0], [0, c, -s], [0, s, c]])
    if axis == "y": return np.array([[c, 0, s], [0, 1, 0], [-s, 0, c]])
    return np.array([[c, -s, 0], [s, c, 0], [0, 0, 1]])

def ypr(rng, y, x, z):
    """Local rotation Ry @ Rx @ Rz with angles drawn uniformly from the given ranges."""
    return rot("y", rng.uniform(*y)) @ rot("x", rng.uniform(*x)) @ rot("z", rng.uniform(*z))

def sample_pose(rng, arms_raised=False):
    L = {j: np.eye(3) for j in range(17)}
    L[0] = ypr(rng, (-180, 180), (-10, 10), (-10, 10))           # global heading
    L[7] = ypr(rng, (-30, 30), (-10, 30), (-15, 15))             # spine
    L[9] = ypr(rng, (-45, 45), (-20, 30), (0, 0))                # neck
    for hip, knee in ((1, 2), (4, 5)):
        L[hip] = ypr(rng, (-20, 20), (-100, 30), (-20, 20))      # thigh flexion, abduction
        L[knee] = rot("x", rng.uniform(0, 130))                  # knee bends backward
    for sh, el, side in ((11, 12, 1), (14, 15, -1)):
        if arms_raised:                                          # outside the training range
            L[sh] = ypr(rng, (-30, 30), (-30, 30), (100, 170)) if side > 0 else \
                    ypr(rng, (-30, 30), (-30, 30), (-170, -100))
        else:
            L[sh] = ypr(rng, (-30, 30), (-70, 40), (0, 70)) if side > 0 else \
                    ypr(rng, (-30, 30), (-70, 40), (-70, 0))
        L[el] = rot("x", -rng.uniform(0, 140))                   # elbow bends forward
    scale = rng.uniform(0.9, 1.1)
    X = np.zeros((17, 3)); Rw = {0: L[0]}
    for j in range(1, 17):                                       # forward kinematics
        p, d, length = BONES[j]
        X[j] = X[p] + Rw[p] @ (scale * length * np.array(d, float))
        Rw[j] = Rw[p] @ L[j]
    X[:, [0, 2]] += rng.uniform(-300, 300, 2)                    # position on the floor
    return X

def look_at(C, target=np.zeros(3)):
    z = (target - C) / np.linalg.norm(target - C)
    x = np.cross(z, [0, 1, 0]); x /= np.linalg.norm(x)
    return np.stack([x, np.cross(z, x), z])                      # rows: camera x, y (down), z

K = np.array([[1000.0, 0, 500], [0, 1000.0, 500], [0, 0, 1]])
CAMS = []
for az in (0, 90):                                               # two views, 90 degrees apart
    C = 4500 * np.array([np.sin(np.radians(az)), 0, np.cos(np.radians(az))]) + [0, 200, 0]
    R = look_at(C)
    CAMS.append((R, C, K @ np.hstack([R, (-R @ C)[:, None]])))

def to_cam(X, cam): R, C, _ = cam; return (X - C) @ R.T
def project(X, cam):
    x = to_cam(X, cam) @ K.T
    return x[..., :2] / x[..., 2:]

def dlt(xs, Ps):
    """Homogeneous linear triangulation of one point from n views."""
    A = np.concatenate([[x[0] * P[2] - P[0], x[1] * P[2] - P[1]] for x, P in zip(xs, Ps)])
    Xh = np.linalg.svd(A)[2][-1]
    return Xh[:3] / Xh[3]

def mpjpe(P, G, root=True):
    if root: P, G = P - P[:, :1], G - G[:, :1]
    return np.linalg.norm(P - G, axis=-1).mean()

def pa_mpjpe(P, G, scale=True):
    """MPJPE after Procrustes alignment of each prediction to its ground truth."""
    errs = []
    for p, g in zip(P, G):
        p0, g0 = p - p.mean(0), g - g.mean(0)
        U, s, Vt = np.linalg.svd(p0.T @ g0)
        D = np.diag([1, 1, np.sign(np.linalg.det(U @ Vt))])     # no reflections
        Rm = U @ D @ Vt
        a = (s * np.diag(D)).sum() / (p0 ** 2).sum() if scale else 1.0
        errs.append(np.linalg.norm(a * p0 @ Rm - g0, axis=-1).mean())
    return np.mean(errs)

rng = np.random.default_rng(0)

# ---------- A. One view cannot see depth ----------
X = sample_pose(rng)
Xc = to_cam(X, CAMS[0])
flipped = Xc.copy()
print("A. flipping each end bone along its camera-1 ray (same bone length)")
for c in (3, 6, 10, 13, 16):
    p = PARENT[c]; d = Xc[c] / np.linalg.norm(Xc[c])
    t0 = np.linalg.norm(Xc[c]); t1 = 2 * d @ flipped[p] - t0  # other root of |t d - X_p| = L
    flipped[c] = t1 * d
    print(f"  {NAMES[c]:8s} moves {abs(t1 - t0):6.1f} mm along the ray")
Xf = flipped @ CAMS[0][0] + CAMS[0][1]                          # back to world coordinates
print(f"  camera 1 reprojection change: {np.abs(project(Xf, CAMS[0]) - project(X, CAMS[0])).max():.1e} px")
print(f"  camera 2 reprojection change: {np.linalg.norm(project(Xf, CAMS[1]) - project(X, CAMS[1]), axis=1)[[3, 6, 10, 13, 16]].mean():.1f} px (mean over moved joints)")
print(f"  MPJPE of flipped pose: {mpjpe(Xf[None], X[None]):.1f} mm")
Xs = (1.1 * Xc) @ CAMS[0][0] + CAMS[0][1]                         # pose scaled 1.1x about camera 1
print(f"  scaled 1.1x about camera 1: reprojection {np.abs(project(Xs, CAMS[0]) - project(X, CAMS[0])).max():.1e} px, "
      f"MPJPE {mpjpe(Xs[None], X[None]):.1f} mm, PA-MPJPE {pa_mpjpe(Xs[None], X[None]):.1f} mm "
      f"(rigid, no scale: {pa_mpjpe(Xs[None], X[None], scale=False):.1f} mm)")

# ---------- B, C. Lifting from one view against triangulation from two ----------
def dataset(n, arms_raised=False):
    W = np.stack([sample_pose(rng, arms_raised) for _ in range(n)])
    return W, [project(W, cam) for cam in CAMS]

def lift_in(x2d): return (x2d - x2d[:, :1]).reshape(len(x2d), -1)       # root-centred 2D
def lift_out(W): Xc = to_cam(W, CAMS[0]); return (Xc - Xc[:, :1]).reshape(len(W), -1)

SIGMAS = (0, 5, 10, 20)
W_tr, x_tr = dataset(20000)
sx, sy = StandardScaler().fit(lift_in(x_tr[0])), StandardScaler().fit(lift_out(W_tr))
Yout = sy.transform(lift_out(W_tr))
sig_tr = rng.choice(SIGMAS, size=(len(W_tr), 1, 1))             # per-pose noise for "noisy" training
models = {}
for train in ("clean", "noisy"):
    x2d = x_tr[0] + (rng.normal(0, 1, x_tr[0].shape) * sig_tr if train == "noisy" else 0)
    Xin = sx.transform(lift_in(x2d))
    models[f"1-NN {train}"] = KNeighborsRegressor(1).fit(Xin, Yout)
    models[f"ridge {train}"] = Ridge(1.0).fit(Xin, Yout)
    models[f"MLP {train}"] = MLPRegressor(hidden_layer_sizes=(256, 256), batch_size=256,
                                          max_iter=60, random_state=0).fit(Xin, Yout)
mean_pose = lift_out(W_tr).mean(0)

for label, raised in (("in distribution", False), ("arms raised (out of distribution)", True)):
    W_te, x_te = dataset(2000, raised)
    G = to_cam(W_te, CAMS[0])
    rows = {}
    for s in SIGMAS:
        noisy = [x + rng.normal(0, s, x.shape) for x in x_te]
        preds = {"mean pose": np.tile(mean_pose, (len(G), 1))}
        for name, m in models.items():
            preds[name] = sy.inverse_transform(m.predict(sx.transform(lift_in(noisy[0]))))
        preds = {k: v.reshape(-1, 17, 3) for k, v in preds.items()}
        Wt = np.stack([[dlt([noisy[0][i, j], noisy[1][i, j]], [c[2] for c in CAMS])
                        for j in range(17)] for i in range(len(G))])
        preds["triangulation"] = to_cam(Wt, CAMS[0])
        for k, P in preds.items():
            rows.setdefault(k, []).append((mpjpe(P, G), pa_mpjpe(P, G), pa_mpjpe(P, G, False)))
        rows.setdefault("triang. absolute", []).append((mpjpe(preds["triangulation"], G, root=False),))
    print(f"\n{label}: MPJPE | PA-MPJPE similarity | PA-MPJPE rigid (mm), sigma = {SIGMAS} px")
    for k, v in rows.items():
        cells = " | ".join(" ".join(f"{r[i]:5.1f}" for r in v) for i in range(len(v[0])))
        print(f"  {k:16s} {cells}")

A. flipping each end bone along its camera-1 ray (same bone length)
  r_ankle  moves  466.7 mm along the ray
  l_ankle  moves  746.9 mm along the ray
  head     moves   91.1 mm along the ray
  l_wrist  moves   22.3 mm along the ray
  r_wrist  moves  443.4 mm along the ray
  camera 1 reprojection change: 1.1e-13 px
  camera 2 reprojection change: 79.0 px (mean over moved joints)
  MPJPE of flipped pose: 104.1 mm
  scaled 1.1x about camera 1: reprojection 1.1e-13 px, MPJPE 40.0 mm, PA-MPJPE 0.0 mm (rigid, no scale: 38.2 mm)

in distribution: MPJPE | PA-MPJPE similarity | PA-MPJPE rigid (mm), sigma = (0, 5, 10, 20) px
  mean pose        257.9 257.9 257.9 257.9 | 231.4 231.4 231.4 231.4 | 235.0 235.0 235.0 235.0
  1-NN clean       143.9 158.8 188.1 236.8 | 119.2 126.1 139.3 158.1 | 122.4 130.0 144.5 165.6
  ridge clean       92.9 1216.5 2351.9 4792.2 |  85.5 243.7 265.0 278.4 |  87.4 998.4 2058.3 4384.4
  MLP clean         53.5 219.0 396.5 762.1 |  48.1 130.1 191.2 250.7 |  50.5 150.9 273.4 554.0
  1-NN noisy       155.5 178.4 211.2 261.9 | 126.5 139.8 156.4 175.5 | 130.1 144.2 162.1 183.1
  ridge noisy      110.2 112.9 120.4 144.4 |  91.2  93.1  98.9 116.2 |  93.5  95.5 101.2 119.1
  MLP noisy         70.9  81.4  97.7 130.8 |  60.5  66.3  77.6 101.9 |  62.4  68.4  79.8 104.7
  triangulation      0.0  43.9  89.8 192.4 |   0.0  30.2  60.7 120.2 |   0.0  30.6  62.2 128.7
  triang. absolute   0.0  32.8  66.4 137.4

arms raised (out of distribution): MPJPE | PA-MPJPE similarity | PA-MPJPE rigid (mm), sigma = (0, 5, 10, 20) px
  mean pose        298.0 298.0 298.0 298.0 | 279.3 279.3 279.3 279.3 | 278.8 278.8 278.8 278.8
  1-NN clean       202.3 211.6 228.2 265.4 | 165.3 168.7 176.8 190.5 | 169.6 173.6 182.6 197.6
  ridge clean       93.9 1211.8 2363.6 4769.2 |  85.0 259.5 289.4 305.8 |  87.5 967.7 2036.8 4330.1
  MLP clean         93.7 218.7 383.2 730.3 |  88.2 144.3 203.9 269.6 |  89.4 159.1 270.7 526.3
  1-NN noisy       212.4 227.2 250.9 297.6 | 171.4 178.0 191.9 212.6 | 176.0 183.4 198.0 219.9
  ridge noisy      122.7 125.2 131.9 154.2 | 100.0 101.9 107.3 123.4 | 101.8 103.7 109.1 126.0
  MLP noisy        111.1 116.4 126.8 155.5 |  94.5  97.8 105.4 127.2 |  96.8 100.2 108.1 130.2
  triangulation      0.0  44.1  89.2 195.3 |   0.0  30.4  60.8 121.2 |   0.0  30.8  62.0 132.1
  triang. absolute   0.0  32.9  66.1 138.6

Part A is the ambiguity in numbers. The five flipped joints move between 22 and 747 mm, yet their image in camera 1 changes by $10^{-13}$ pixels, rounding error; camera 2 sees them 79 pixels away on average. Scaling the pose by 1.1 about camera 1’s centre also leaves its image unchanged and costs 40 mm of root-relative MPJPE. Procrustes alignment with scale removes that error entirely, while alignment without scale leaves almost all of it, which is why the two variants of PA-MPJPE should not be compared with each other.

With clean 2D input, triangulation is exact up to rounding, and the best lifter, the perceptron trained on clean 2D, still errs by 53.5 mm. In our reading, that residual combines the ambiguity a learned prior cannot resolve with the model’s own error, since the perceptron stops at its iteration limit. With noise, triangulation’s error grows in proportion to the noise, about 6.6 mm of absolute MPJPE per pixel, the order of the first-order estimate $\sigma Z / f \approx 4.5$ mm per pixel for each coordinate at 4.5 m. Its root-relative error is larger than its absolute error (192.4 against 137.4 mm at 20 pixels) because subtracting a noisy pelvis adds the pelvis’s own error to every joint.

The lifters trained on clean 2D collapse under noise. Ridge regression goes from 92.9 mm to more than a metre at 5 pixels, and the perceptron from 53.5 to 219.0 mm. In our reading, noise-free synthetic 2D lets them exploit exact relations between coordinates that small perturbations destroy; it is the failure that Martinez et al. (2017) report for networks trained on ground truth and tested on noisy 2D. The plan written before the first run had expected that the lifters’ prior might make them competitive with triangulation at 10 to 20 pixels; with clean training they are worse at every noise level. Training on noisy 2D reverses this. The noise-trained perceptron is worse on clean input (70.9 mm) but degrades slowly, and at 20 pixels it beats triangulation (130.8 against 192.4 mm root-relative, 101.9 against 120.2 mm after similarity alignment). In our reading, when the 2D evidence is poor enough, a prior over bodies is worth more than a second view, but only for a lifter trained on 2D as noisy as its test input. The nearest-neighbour lookup is far behind throughout, even on clean input, presumably because 20,000 random poses cover the space of poses and headings sparsely.

Out of distribution, every lifter gets worse, the clean-trained perceptron from 53.5 to 93.7 mm on clean input, while triangulation is unchanged; at 20 pixels the noise-trained perceptron still beats triangulation in root-relative MPJPE and, narrowly, after rigid alignment, but no longer after similarity alignment (127.2 against 121.2 mm). Ridge regression degrades least on clean input (92.9 to 93.9 mm), in our reading because, being linear, it learned no pose-specific prior to be wrong about. PA-MPJPE is never above root-relative MPJPE, and the scaled alignment is at or below the rigid one except for the mean-pose baseline out of distribution (279.3 against 278.8 mm): Procrustes alignment minimizes the sum of squared distances, not the mean of the unsquared distances that MPJPE reports, so the extra freedom of scale need not lower MPJPE.

The script was changed after its first run. The first run stopped with an error after part A, because this scikit-learn version takes the loss as the perceptron’s first positional argument; the hidden-layer sizes were then passed by keyword. The second run completed in about ten minutes, far over the plan’s aim of “under about 30 s”, and showed the clean-trained lifters failing under noise, against the plan’s expectation. Three changes followed, all made after the second run’s results had been seen: the BLAS thread count was fixed at four and the perceptron’s hidden layers reduced from the planned two layers of 512 units to two of 256, to cut the runtime; the noise-trained lifters were added, because the second run showed that the comparison depends on the training noise, a choice the sources leave open and the plan had not included; and the convergence warning, which says that the perceptron stops at its iteration limit, was silenced. The noise-trained lifters take their training noise from the same random generator before the test poses are drawn, so the test poses and test noise, and with them the printed numbers for the clean-trained lifters and for triangulation, differ from the second run’s. A printed runtime line, which differed between two otherwise identical runs, was then removed, and a code comment was reworded. Two runs of the final script give identical output.

Datasets and Benchmarks

Human3.6M

Human3.6M (Ionescu et al., 2014) is the standard laboratory benchmark: 3.6 million 3D human poses with corresponding images, recorded by four calibrated 50 Hz video cameras synchronized with a marker-based motion-capture system of 10 cameras, together with time-of-flight depth, 3D laser scans of the actors, background subtraction and person bounding boxes (paper and dataset page). Eleven professional actors, 6 male and 5 female, wearing their own clothes, perform 15 scenarios such as directions, discussion, eating, phoning, sitting, sitting down, walking and walking a dog; the dataset page’s overview lists “17 scenarios”, while its own scenario list names 15, and the paper gives 15 plus a miscellaneous set used only for testing. Poses are given on a 32-joint skeleton, reduced in testing; the paper’s experiments use 17 joints.

The official split reserves 7 subjects for training and validation (5 and 2) and 4 for testing, scored by the authors’ evaluation server on the withheld test set; the images of one test subject, S10, are withheld for privacy (Ionescu et al., 2014; dataset page). Only the training and validation subjects have public 3D annotations (Iskakov et al., 2019), seven of the eleven (Pavllo et al., 2019). Published work therefore trains on S1, S5, S6, S7 and S8 and tests on S9 and S11 (Martinez et al., 2017; Pavlakos et al., 2017; Pavllo et al., 2019); Iskakov et al. (2019) describe this as testing on the validation set and call S9 a validation actor: in their terms, the usual test subjects belong to the official validation split. Within that protocol, the frames evaluated differ: Pavlakos et al. and HMR downsample the 50 fps videos to 10 fps, Iskakov et al. use every fifth frame, and some methods evaluate every 64th frame (Pavllo et al., 2019, Table 1). Results also differ by whether one model is trained per action or one for all actions; Martinez et al. found the single model for all actions consistently better.

Access is granted on request. As of November 2026 the licence page grants a free, limited, non-transferable licence for academic use only, to requests from an academic address, treats any use of the data as acceptance of its terms, requires citing two of the authors’ papers (citing the URL is not compliant), and refers commercial licensing to the authors.

MPI-INF-3DHP

MPI-INF-3DHP (Mehta et al., 2017) was recorded with a multi-camera marker-less motion-capture system. The dataset page describes more diversity in pose, appearance, clothing, occlusion and viewpoint than earlier corpora, and a test benchmark with indoor and outdoor scenes; without markers, the 3D ground truth is noisier (Kanazawa et al., 2018; Kolotouros et al., 2019). The training set covers subjects S1 to S8 (Kolotouros et al., 2019); the test set, as used by HMR, has 2,929 valid frames of 6 subjects performing 7 actions, indoors and outdoors. Results are reported as PCK, here the percentage of 3D joints within 150 mm of the ground truth, the area under the PCK curve over a range of thresholds (AUC), and MPJPE, with and without rigid alignment (Kanazawa et al., 2018, Table 3). The dataset page records corrections to the annotations of test sequences TS3 and TS4 (March 2018) and TS6 (May 2019), so results computed before and after them differ slightly. On this benchmark SPIN reported 76.4 PCK, 37.1 AUC and 105.2 mm MPJPE without alignment, and 92.5, 55.6 and 67.5 mm after rigid alignment (Kolotouros et al., 2019, Table 4).

3DPW

3D Poses in the Wild (von Marcard et al., 2018) supplies accurate 3D poses outdoors by fusing body-worn inertial measurement units with a hand-held phone camera. Its method fits SMPL to the IMU orientations and to 2D detections over a whole sequence; on the TotalCapture dataset it was accurate to 26 mm. The released dataset has 60 video sequences, more than 51,000 frames at 30 Hz, camera poses for every frame, 2D annotations and 3D models in 18 clothing variations. The standard measure is MPJPE after Procrustes alignment, called PA-MPJPE by VIBE and reconstruction error by SPIN. Reported numbers vary with the training data: Kocabas et al. (2020, Table 1) give 56.5 mm for VIBE trained without 3DPW and 51.9 mm with its training set. Even the number quoted for one method differs between papers: HMR is listed at 81.3 mm in SPIN’s 3DPW table and at 76.7 mm in VIBE’s. Two of the dataset’s authors, Pons-Moll and Black, with Kanazawa and Mir, organized a challenge at ECCV 2020 because protocols and sequences differed from paper to paper; its protocol evaluates on the entire dataset and forbids training on 3DPW (3DPW Challenge page, as of November 2026).

Older work also reports on HumanEva-I, a smaller motion-capture dataset whose results are given after rigid alignment (Martinez et al., 2017). Multi-view work also uses CMU Panoptic, which Iskakov et al. (2019) evaluated with their own protocol because no published multi-view results existed for it.

Evaluation Metrics

MPJPE. The mean per joint position error is the average Euclidean distance between predicted and true joints (Ionescu et al., 2014, Eq. 8):

$$E_{\text{MPJPE}} = \frac{1}{J} \sum_{j=1}^{J} \left\lVert \hat{\mathbf{X}}_j - \mathbf{X}_j \right\rVert_2 ,$$

averaged over all frames, in millimetres. Under protocol #1, as Martinez et al. (2017) name the standard Human3.6M evaluation, both poses are first translated so that their root joints coincide. Because it is a mean of distances rather than squared distances, one bad joint raises it in proportion.

PA-MPJPE. The Procrustes-aligned MPJPE first aligns the prediction to the ground truth by the transformation that minimizes the squared distance between them, then computes MPJPE. VideoPose3D’s reference implementation (p_mpjpe in its code) solves for a rotation by SVD of the cross-covariance of the centred point sets, rejects reflections, and computes the scale and translation in closed form; it calls this a “rigid alignment” although it includes scale, as does the caption of the paper’s Table 1(b). The alignment removes global rotation, translation and scale, so PA-MPJPE measures the shape of the skeleton, not its placement. Names vary: P-MPJPE (Pavllo et al., 2019), PA-MPJPE (Kocabas et al., 2020), reconstruction error (Pavlakos et al., 2017; Kanazawa et al., 2018; Kolotouros et al., 2019). Pavllo et al. also add N-MPJPE, which aligns only scale, as protocol 3.

What “Protocol #2” means. Martinez et al. (2017) define protocol #2 as protocol #1 followed by a rigid alignment, on all cameras; Pavllo et al. (2019) as alignment in translation, rotation and scale; Pavlakos et al. (2017), without a protocol number, define reconstruction error up to a similarity transformation; HMR’s protocol 2 uses the same subjects but evaluates only the frontal camera (camera 3) with reconstruction error, which HMR defines as MPJPE after rigid alignment by Procrustes analysis (Kanazawa et al., 2018). Pavllo et al.’s Table 1 labels protocol 1 “reconstruction error (MPJPE)”, the term other papers reserve for the aligned error. Numbers from different papers are comparable only when these details match.

Other measures. PCK at 150 mm and AUC on MPI-INF-3DHP (above). Ionescu et al. (2014) also proposed a universal MPJPE on a skeleton with standard limb lengths, a mean per joint angle error, and a mean per joint localization error with a perceptual tolerance, by which LinKDE placed 14 of 17 joints within 150 mm on average for “Eating”. For video, the velocity error MPJVE (Pavllo et al., 2019) and the acceleration error (Kocabas et al., 2020) measure smoothness. Mesh methods add a per-vertex error (PVE). Kanazawa et al. (2018) observe that MPJPE does not correlate well with the visual quality of results.

Applications

• Motion capture without markers for animation, biomechanics and sports; Iskakov et al. (2019) name sports, action recognition, computer-assisted living, human–computer interfaces, special effects and telepresence.
• Ground truth for monocular training, by triangulating from many views (Iskakov et al., 2019).
• Action recognition from 3D skeletons, as in skeleton-based action recognition.
• Virtual and augmented reality, apparel sizing and driving, which Martinez et al. (2017) list among the uses of spatial reasoning about people; body models such as SMPL also serve avatar animation in graphics engines (Loper et al., 2015).

Open Problems

• In-the-wild accuracy. Laboratory errors are near 40 to 50 mm for the best monocular methods cited here, outdoor errors on 3DPW above 50 mm after alignment, and lifters fail on poses absent from their training data.
• Absolute depth and multiple people. Root-relative protocols ignore where each person stands; recovering each person’s distance from one camera remains a separate problem, related to monocular depth estimation.
• Comparable evaluation. Protocol names, frame sampling, alignment and training data differ across papers, and annotation corrections change results; the 3DPW Challenge was organized for this reason.
• Few cameras. Iskakov et al. (2019) motivate learnable triangulation by the excessive number of views needed to triangulate ground truth in the wild; accurate 3D from two or three uncalibrated or moving cameras remains hard, and calibration itself is a prerequisite (camera calibration).
• Licences. The main benchmark and body model are licensed for non-commercial research only, which limits their use in products.