Human Pose Estimation

Human pose estimation locates the body joints of the people in an image, such as shoulders, elbows, wrists, hips, knees and ankles, and so recovers each person’s posture as a 2D skeleton. It is also called body keypoint detection, articulated pose estimation or skeleton estimation; “pose” here means the configuration of the body, not the position and orientation of a rigid object. Classical systems matched part models of the body, in the tradition of pictorial structures, to hand-crafted features; since 2014 convolutional networks predict each joint directly or, far more often, as a heatmap. Systems for crowded scenes either first find each person with an object detector and then the joints inside each box (top-down), or find all joints in the image and group them into people (bottom-up). Results are scored with joint-distance metrics normalized by a body size: PCKh on MPII Human Pose and an average precision built on object keypoint similarity on COCO. This article covers single- and multi-person 2D pose in RGB images; 3D human pose estimation, which recovers joint positions in space, is a related task.

Problem Definition

A pose model fixes a set of $K$ keypoints and, for drawing, a skeleton of edges between them. For each person in the image, the task is to predict the image coordinates $\mathbf{x}_k \in \mathbb{R}^2$ of every keypoint $k = 1, \dots, K$, usually with a confidence for each keypoint and, when several people may be present, a score for each person. Keypoints are labeled person-centrically: “left wrist” is the person’s left wrist, wherever it appears in the image, so a method must also tell the person’s left from right (Andriluka et al., 2014).

The keypoint sets are defined by the benchmarks:

• MPII Human Pose annotates 16 body joints: right and left ankles, knees, hips, wrists, elbows and shoulders, plus pelvis, thorax, upper neck and head top, each with a visibility flag, and a head rectangle for every person (dataset download page, as of October 2026). The paper also describes eye and nose locations, 3D head and torso orientation and occlusion labels for parts (Andriluka et al., 2014).
• COCO annotates 17 keypoints for people: the nose and the left and right eyes, ears, shoulders, elbows, wrists, hips, knees and ankles. Each is stored as $(x, y, v)$ with $v = 0$ for not labeled, $v = 1$ for labeled but not visible and $v = 2$ for labeled and visible, where visible means inside the person’s segment; a skeleton list of edges serves only for visualization. Keypoints are labeled for most medium and large non-crowd people (COCO data format page); people with a segment area below $32^2$ pixels have none (keypoint evaluation page).

Single-person pose estimation assumes that the person is roughly known: MPII’s single-person protocol allows the rough location and scale of each sufficiently separated person to be used at test time, while its multi-person protocol forbids the number of people and their individual positions and scales (MPII evaluation page). Multi-person pose estimation must find an unknown number of people as well; the COCO task gives no person locations at test time, so it is detection and keypoint localization at once (keypoint evaluation page).

Input and Output

The input is an RGB image or a video frame. Depth images were the input of the Kinect body tracker (below), and top-down networks work on a crop resized to a fixed size, such as $256 \times 192$ pixels for a COCO person or $256 \times 256$ for MPII (Sun et al., 2019).

The output for each person is a $K \times 3$ array of $(x, y, \text{score})$, plus a person score used to rank detections in COCO evaluation. Inside the network, two output representations compete:

• Coordinates. The network regresses the $2K$ numbers directly, as DeepPose did (Toshev and Szegedy, 2014).
• Heatmaps. The network outputs one map per keypoint, trained to match a Gaussian bump at the joint and decoded by finding its peak (Tompson et al., 2014). For a joint at $\mathbf{x}_k$ in input pixels, a map of stride $\lambda$ (input pixels per map cell) has the target

$$H_k(\mathbf{p}) = \exp\left( -\frac{\lVert \mathbf{p} - \mathbf{x}_k / \lambda \rVert^2}{2\sigma^2} \right),$$

with $\mathbf{p}$ ranging over cells and $\sigma$ a fixed spread in cells, and the loss is the mean squared error between predicted and target maps. In HRNet’s released COCO configuration the input is $256 \times 192$, the maps are $64 \times 48$ (stride 4) and $\sigma = 2$ cells; the code rounds $\mathbf{x}_k / \lambda$ to the nearest cell before drawing the target (HRNet code, JointsDataset.py and w32_256x192_adam_lr1e-3.yaml). The HRNet paper itself states a standard deviation of 1 pixel (Sun et al., 2019). OpenPose writes its confidence-map target without the factor 2, as $\exp(-\lVert \mathbf{p} - \mathbf{x} \rVert^2 / \sigma^2)$, and takes the maximum over people where peaks overlap (Cao et al., 2017).

Variants

• Single-person versus multi-person. Single-person benchmarks (LSP, FLIC, MPII single-person) give a crop around each person or its rough location and scale; multi-person benchmarks (MPII multi-person, COCO) do not.
• Whole-body pose. COCO-WholeBody adds feet, face and hand keypoints to COCO people, 133 in all: 17 for the body, 6 for the feet, 68 for the face and 42 for the hands (Jin et al., 2020). The OpenPose library detects body, foot, hand and facial keypoints, 135 in total (Cao et al., 2021).
• Pose in video. Pose tracking links poses of the same person across frames. The PoseTrack benchmark, as described by Xiao, Wu and Wei (2018), has 514 videos with 15 body keypoints, a person identity and a head box for each annotated person.
• 3D human pose. Lifting joints to 3D, from one or several cameras, is a separate task; the 2D detectors below are a common first stage.
• Depth input. Shotton et al. (2011), the method behind the Kinect body tracker, classified each pixel of a single depth image into 31 body parts with a randomized decision forest trained on synthetic depth images, then proposed 3D joint positions from the modes of each part’s pixels, at 200 frames per second on the Xbox console. The paper received the CVPR 2011 Best Paper Award and the 2021 Longuet-Higgins Prize (CVF awards page).

Challenges

• Articulation and rare poses. Limbs rotate, bend and foreshorten. Andriluka et al. (2014) collected MPII from an activity taxonomy precisely because earlier datasets were dominated by upright people with arms down along the torso.
• Occlusion and truncation. Joints are hidden by the body itself, by objects, by other people or by the image border. MPII’s paper analysed how methods degrade with part occlusion, foreshortening, body pose and viewpoint, and noted that 15% of its people have only the upper body visible (Andriluka et al., 2014).
• Left–right and person confusions. Most of the 1–2% outliers among COCO’s doubly annotated keypoints come from annotators swapping left and right, or annotating the wrong person when two people are close (keypoint evaluation page); networks make the same mistakes, which the person-centric labels count as errors.
• Scale. COCO people with keypoints have segment areas from $32^2$ pixels upward, and many are far larger. A bottom-up network sees all of them at the image’s resolution; OpenPose’s authors attributed their lower AP on medium-sized people than top-down methods to this (Cao et al., 2017).
• Crowds and unknown numbers. Overlapping people must be separated without knowing how many there are; top-down methods inherit the detector’s misses and merged boxes, bottom-up methods must group parts correctly.
• Localization precision. Strict metric thresholds require sub-pixel accuracy from low-resolution heatmaps (see Decoding Heatmaps).

Approaches

Pictorial structures and part models

Fischler and Elschlager (1973) described an object as parts held together by springs, and Felzenszwalb and Huttenlocher made this “pictorial structure” practical: the best match minimizes the sum of each part’s appearance cost and a deformation cost for each connected pair, which dynamic programming solves exactly over a tree, in time nearly linear in the number of locations per part when the deformation cost is a Mahalanobis distance (the matching is derived in deformable part models). Their person model had limbs, torso and head as parts, each located by position, orientation and foreshortening (Felzenszwalb and Huttenlocher, 2005). Their CVPR 2000 paper, “Efficient Matching of Pictorial Structures”, received the 2010 Longuet-Higgins Prize (CVF awards page). The same machinery, with HOG filters and a star-shaped graph, underlies deformable part models for object detection.

Yang and Ramanan (2011) replaced rotated limb templates with a flexible mixture of parts. Each part $i$ at pixel position $p_i$ takes one of $T$ discrete types $t_i$, which in practice correspond to orientations, and a configuration scores

$$S(I, p, t) = \sum_{i \in V} b_i^{t_i} + \sum_{ij \in E} b_{ij}^{t_i, t_j} + \sum_{i \in V} w_i^{t_i} \cdot \phi(I, p_i) + \sum_{ij \in E} w_{ij}^{t_i, t_j} \cdot \psi(p_i - p_j),$$

with type biases and co-occurrence terms $b$, a HOG template $w_i^{t_i}$ for each part type, and a spring $\psi(p_i - p_j) = [dx, dx^2, dy, dy^2]^\top$ chosen by the pair of types. A tree over parts and types is still solved exactly by dynamic programming, and all parameters are learned together with a structured SVM. Parts were $5 \times 5$ HOG cells, the model had 14 parts at the joints or 27 with midpoints added, and a picture took about a second. On the Parse test set the model reached 74.9% PCP against 66.2% for the best earlier result (Yang and Ramanan, 2011, Table 2); the abstract and introduction claim error reductions of up to 50%, while the discussion of Parse states 25%. The journal version introduced the PCK and APK measures (Yang and Ramanan, 2013).

These models scored each part locally and linked parts only along tree edges. When MPII appeared, the best pictorial-structure model scored 42.3% and Yang and Ramanan’s model 38.3% PCPm on the new benchmark (Andriluka et al., 2014, Table 2); in plain PCP the pictorial-structure model scored 42.2% on MPII against 69.2% on the sports-centred LSP.

Deep networks: regression, then heatmaps

DeepPose (Toshev and Szegedy, 2014) regressed all joint coordinates from a $220 \times 220$ image with a seven-layer convolutional network, using coordinates normalized by the person’s box and an L2 loss. A cascade of three stages refined the joints, each later stage regressing a correction from a crop around the previous estimate of each joint. On LSP it averaged 0.61 PCP over upper and lower arms and legs, against 0.58 to 0.59 for the best earlier methods (Table 1). Tompson et al. (2014) attributed DeepPose’s inaccuracy at high precision to the difficulty of regressing coordinates directly from images.

Heatmaps. Tompson et al. (2014) slid a multi-resolution network over the image to produce a heatmap per joint, trained with mean squared error against a 2D Gaussian at the joint, and added a spatial model inspired by a Markov random field, trained jointly with the part detector. Pooling made the output smaller than the input; the paper’s Figure 2 shows $90 \times 60$ maps for a $320 \times 240$ image. On the MPII test set the method scored 79.6% PCKh@0.5 (MPII results page). Heatmaps soon became the most common output representation (Zhang et al., 2020), and later work refined the network around them:

• Convolutional pose machines (Wei et al., 2016) stacked stages that each read image features and the previous stage’s belief maps, so that later stages, with receptive fields covering the whole body, could resolve confusions between parts. A loss after every stage, called intermediate supervision, kept gradients from vanishing in the deep sequence. With six stages, $368 \times 368$ inputs and LSP data added to training, the model scored 88.5% PCKh@0.5 on MPII (MPII results page).
• Stacked hourglass networks (Newell, Yang and Deng, 2016) repeated an hourglass of pooling and upsampling with skip connections, built from residual modules, and supervised every hourglass. In the arXiv version (v2), a $256 \times 256$ input gave $64 \times 64$ heatmaps, trained against Gaussians of standard deviation 1 pixel; at test time heatmaps of the image and its mirror were averaged and the peak shifted by a quarter pixel toward its higher neighbour. Eight stacks scored 90.9% PCKh@0.5 on the MPII test set (arXiv v2, Table 2; MPII results page).
• SimpleBaseline (Xiao, Wu and Wei, 2018) showed that a ResNet followed by three transposed convolutions (256 filters of $4 \times 4$, stride 2) and a $1 \times 1$ output layer matched more complex designs. As a top-down system, with ResNet-152 at $384 \times 288$ and a person detector of 60.9 box AP, it reached 73.7 AP on COCO test-dev (Table 4). In its val2017 ablations, with people from a Faster R-CNN detector, halving the input from $256 \times 192$ to $128 \times 96$ cut ResNet-50’s AP from 70.4 to 60.6 (Table 2).
• HRNet (Sun et al., 2019) kept a high-resolution branch at 1/4 of the input throughout the network and added lower-resolution branches in parallel, exchanging information between them repeatedly, instead of recovering resolution from a low-resolution feature map. HRNet-W48 at $384 \times 288$ reached 75.5 AP on COCO test-dev, and 77.0 with extra training data; HRNet-W32 scored 92.3% PCKh@0.5 on the MPII test set (Tables 2 and 3).

SimpleBaseline’s authors described MPII as saturated, PCKh@0.5 having risen in three years from about 80% to over 90%, and called COCO the more recent and challenging benchmark (Xiao, Wu and Wei, 2018).

Multi-person: top-down and bottom-up

Top-down methods run a person detector, crop each detected box and apply a single-person network. In HRNet’s released code, a box is widened or heightened to the network input’s 4:3 height-to-width ratio and then enlarged by 1.25 before cropping (HRNet code, coco.py). Each person is processed at the same resolution whatever their size in the image, but the cost grows with the number of people and a missed detection cannot be recovered. On 1,160 COCO validation images, a convolutional pose machine reached 62.7 AP on ground-truth boxes and 52.7 on boxes from an SSD detector (Cao et al., 2017, Table 4). Mask R-CNN folds the second stage into the detector, predicting each keypoint as a one-hot mask inside every detected box.

Bottom-up methods detect every keypoint in the image and then assign keypoints to people, at a cost largely independent of the number of people. The difficulty is the grouping.

• Part affinity fields (Cao et al., 2017) predict, for each limb type $c$, a 2D vector field alongside the keypoint confidence maps. The training target at point $\mathbf{p}$ is the unit vector $\mathbf{v}$ from joint $j_1$ to joint $j_2$ if $\mathbf{p}$ lies on the limb of some person (within a width $\sigma_l$ of the segment), and zero elsewhere, averaged where people overlap. Keypoint candidates come from non-maximum suppression on the confidence maps, and a candidate limb between detections $\mathbf{d}_{j_1}$ and $\mathbf{d}_{j_2}$ is scored by the line integral of the predicted field along it,

$$E = \int_{0}^{1} L_c\big(\mathbf{p}(u)\big) \cdot \frac{\mathbf{d}_{j_2} - \mathbf{d}_{j_1}}{\lVert \mathbf{d}_{j_2} - \mathbf{d}_{j_1} \rVert_2} \, du, \qquad \mathbf{p}(u) = (1 - u)\,\mathbf{d}_{j_1} + u\,\mathbf{d}_{j_2},$$

approximated by sampling. For one limb type, choosing connections is a maximum-weight bipartite matching, solvable with the Hungarian algorithm; the full multi-person problem is NP-hard, and the authors relaxed it to independent matchings along the edges of a spanning tree of the skeleton. The method won the first COCO keypoint challenge in 2016 (keypoint evaluation page), with 60.5 AP on test-challenge and 61.8 on test-dev (Table 3), and reached 75.6% mAP on the full MPII multi-person test set (Table 1). The journal version (Cao et al., 2021) describes the OpenPose library, which the authors present as the first open-source real-time system for multi-person 2D body, foot, hand and facial keypoints.

• Associative embedding (Newell, Huang and Deng, 2017) outputs, next to each keypoint heatmap, a tag map holding one real number per pixel. The grouping loss pulls the tags of one person’s joints toward their mean $\bar{h}_n$ and pushes the means of different people apart:

$$L_g = \frac{1}{NK} \sum_{n} \sum_{k} \left( \bar{h}_n - h_k(\mathbf{x}_{nk}) \right)^2 + \frac{1}{N^2} \sum_{n} \sum_{n'} \exp\left( -\frac{1}{2\sigma^2} (\bar{h}_n - \bar{h}_{n'})^2 \right),$$

where $h_k(\mathbf{x}_{nk})$ is the tag at person $n$’s joint $k$ of $K$, for $N$ people; only distances between tags matter, not their values. Four stacked hourglasses on $512 \times 512$ inputs, with $128 \times 128$ outputs, multi-scale testing and refinement by a single-person model, reached 65.5 AP on COCO test-dev (Table 3). With ground-truth keypoint locations and predicted tags, AP on 500 held-out training images rose from 59.2 to 94.0, which the authors read as evidence that detection, not grouping, was the bottleneck.

• HigherHRNet (Cheng et al., 2020) addressed small people in bottom-up methods, which usually predict heatmaps at 1/4 of the input. A transposed-convolution module on top of HRNet adds heatmaps at 1/2 resolution, trained at both resolutions and averaged at test time, with associative embedding for grouping. It reached 70.5 AP on COCO test-dev without refinement or other post-processing, and 67.6 AP on the CrowdPose test set.

Vision transformers

Earlier transformer pose models, as Xu et al. (2022) summarize them, put transformer layers on top of CNN features (TransPose, TokenPose) or built multi-resolution transformer backbones (HRFormer). ViTPose (Xu et al., 2022) used a plain, non-hierarchical vision transformer with $16 \times 16$ patches, so its features are at 1/16 of the input, pretrained as a masked autoencoder, and a light top-down decoder: two transposed-convolution blocks to 1/4 resolution, or simply 4× bilinear upsampling and a $3 \times 3$ convolution. On COCO val2017 at $256 \times 192$, ViTPose-B scored 75.8 AP and ViTPose-H 79.1 AP when trained on COCO alone (Table 9). The largest model, with a 1-billion-parameter ViTAE-G backbone, a $576 \times 432$ input, AI Challenger as extra training data and a stronger person detector, reached 80.9 AP on COCO test-dev, and an ensemble of three 81.1 (Table 10).

Decoding heatmaps

A heatmap of stride $\lambda$ gives the argmax only to the nearest cell, an error of up to $\lambda / 2$ input pixels along each axis before mapping back to the image. Newell, Yang and Deng (2016) shifted the peak by a quarter cell toward the higher neighbour; HRNet’s released code does this along each axis by the sign of the difference between the two neighbours (HRNet code, inference.py).

DARK (Zhang et al., 2020) treats the predicted map as a Gaussian. Writing $P = \ln H$ and taking its gradient $D'$ and Hessian $D''$ at the argmax $\mathbf{m}$, a second-order Taylor expansion puts the peak at

$$\boldsymbol{\mu} = \mathbf{m} - \big(D''(\mathbf{m})\big)^{-1} D'(\mathbf{m}),$$

computed with finite differences. Because predicted maps often have several bumps near the maximum, the map is first smoothed with a Gaussian of the training spread. The paper then min–max normalizes the smoothed map and scales it to the original maximum (Eq. 11); the released code only rescales it so that its maximum equals the original’s (DarkPose code). DARK also centres training targets at the unrounded joint position. With HRNet-W32 at $128 \times 96$ on COCO val2017, AP was 61.2 without shifting, 66.9 with the quarter shift and 68.4 with DARK decoding, and 70.7 with unbiased targets as well (Tables 1 and 3; the introduction attributes the 5.7-point gain of the shift to coordinate encoding, while Table 1 varies the decoding). On COCO test-dev, HRNet-W48 at $384 \times 288$ rose from 75.5 to 76.2 AP, and from 77.0 to 77.4 with extra data (Table 7).

Example: the keypoint metrics and heatmap stride on synthetic stick figures

The example renders frontal stick figures, 80 to 400 pixels tall with random limb angles, on $512 \times 512$ images, one to three per image. Each figure’s silhouette is a binary image whose pixel count is its segment area, as in COCO; its head disk gives the MPII head rectangle. In place of a trained network, it simulates one: for each joint, the “predicted” heatmap is a Gaussian of $\sigma = 2$ cells centred at the true joint plus a localization error of 0.25 cells (standard deviation), with additive noise of standard deviation 0.02. Three pipelines produce maps: top-down crops at $256 \times 192$ and $128 \times 96$ input (heatmap stride 4, boxes from the silhouettes, so detection is perfect), and a bottom-up $128 \times 128$ map for the whole image, also stride 4, where each joint is read inside a small window around its true position (perfect grouping). Peaks are decoded by argmax, by the quarter shift as in HRNet’s code, and by DARK, smoothing with the training spread as the paper prescribes and following the released code for the rescaling and the finite differences. COCO AP follows cocoeval.py (area ranges, greedy matching by score, 101-point interpolation, at most 20 detections, detection area from the keypoints’ box as in coco.py); on seed 0 it matched the official cocoeval.py, run on the same data with NumPy’s removed np.float alias replaced, to two decimals in all nine rows. PCKh follows the MPII code: head size 0.6 times the head rectangle’s diagonal, a joint correct if its normalized distance is at most the threshold, 14 joints pooled.

import numpy as np
from scipy.ndimage import gaussian_filter

# COCO person keypoints in annotation order, and the per-keypoint sigmas of
# cocoeval.py (setKpParams); OKS uses kappa = 2 * sigma.
COCO = ["nose", "l_eye", "r_eye", "l_ear", "r_ear", "l_sho", "r_sho", "l_elb", "r_elb",
        "l_wri", "r_wri", "l_hip", "r_hip", "l_knee", "r_knee", "l_ank", "r_ank"]
SIGMAS = np.array([.26, .25, .25, .35, .35, .79, .79, .72, .72, .62, .62,
                   1.07, 1.07, .87, .87, .89, .89]) / 10.0
# The 14 joints scored by the MPII PCKh code (pelvis and thorax are left out).
MPII = ["r_ank", "r_knee", "r_hip", "l_hip", "l_knee", "l_ank", "r_wri", "r_elb",
        "r_sho", "l_sho", "l_elb", "l_wri", "neck", "head_top"]
JOINTS = COCO + ["neck", "head_top"]
CANVAS, HM_SIGMA = 512, 2.0          # image size; heatmap Gaussian sigma in cells


def stick_figure(rng, H):
    """Random frontal stick figure of height ~H: joints, capsules and head box."""
    cx, feet = rng.uniform(0.3 * H, CANVAS - 0.3 * H), rng.uniform(H + 5, CANVAS - 5)
    lean = rng.normal(0, 0.12)
    up = np.array([np.sin(lean), -np.cos(lean)])            # image y points down
    side = np.array([np.cos(lean), np.sin(lean)])
    J = {"pel": np.array([cx, feet - 0.5 * H])}
    J["tho"] = J["pel"] + 0.30 * H * up
    J["neck"] = J["tho"] + 0.05 * H * up
    hc, hr = J["neck"] + 0.07 * H * up, 0.065 * H
    J["head_top"], J["nose"] = hc + hr * up, hc - 0.01 * H * up
    for s, sg in (("l", 1), ("r", -1)):
        out = sg * side
        J[s + "_eye"] = hc + 0.015 * H * up + 0.025 * H * out
        J[s + "_ear"] = hc + 0.06 * H * out
        J[s + "_sho"], J[s + "_hip"] = J["tho"] + 0.11 * H * out, J["pel"] + 0.08 * H * out
        a = rng.uniform(-0.3, 2.8)                           # upper arm, from straight down
        b = a + rng.uniform(-1.5, 0.3)
        J[s + "_elb"] = J[s + "_sho"] + 0.17 * H * (np.cos(a) * -up + np.sin(a) * out)
        J[s + "_wri"] = J[s + "_elb"] + 0.15 * H * (np.cos(b) * -up + np.sin(b) * out)
        a = rng.uniform(-0.15, 0.6)                          # thigh and shin
        b = a + rng.uniform(-0.4, 0.2)
        J[s + "_knee"] = J[s + "_hip"] + 0.25 * H * (np.cos(a) * -up + np.sin(a) * out)
        J[s + "_ank"] = J[s + "_knee"] + 0.24 * H * (np.cos(b) * -up + np.sin(b) * out)
    caps = [("pel", "neck", 0.075), ("l_sho", "r_sho", 0.03), ("l_hip", "r_hip", 0.035)]
    for s in "lr":
        caps += [(s + "_sho", s + "_elb", .028), (s + "_elb", s + "_wri", .028),
                 (s + "_hip", s + "_knee", .028), (s + "_knee", s + "_ank", .028)]
    return J, [(J[a], J[b], r * H) for a, b, r in caps] + [(hc, hc, hr)], (hc, hr)


def render(capsules):
    """Binary silhouette image: union of capsules (a disk is a zero-length capsule)."""
    img = np.zeros((CANVAS, CANVAS), bool)
    for a, b, r in capsules:
        x0, y0 = np.maximum(np.floor(np.minimum(a, b) - r), 0).astype(int)
        x1, y1 = np.minimum(np.ceil(np.maximum(a, b) + r), CANVAS).astype(int)
        yy, xx = np.mgrid[y0:y1, x0:x1] + 0.5                 # pixel centres
        P, ab = np.stack([xx, yy], -1), b - a
        t = np.clip(((P - a) @ ab) / max(ab @ ab, 1e-9), 0, 1)
        img[y0:y1, x0:x1] |= np.linalg.norm(P - (a + t[..., None] * ab), axis=-1) <= r
    return img


def gauss_map(shape, centers, rng, noise=0.02):
    """Simulated network output: unit Gaussians (max-combined) plus noise."""
    yy, xx = np.mgrid[0:shape[0], 0:shape[1]]
    h = np.zeros(shape)
    for u, v in centers:
        h = np.maximum(h, np.exp(-((xx - u) ** 2 + (yy - v) ** 2) / (2 * HM_SIGMA ** 2)))
    return h + noise * rng.standard_normal(shape)


def decode(h, method, win=None):
    """Peak location in heatmap cells: argmax, +quarter offset, or DARK."""
    if win is not None:                                      # oracle grouping window
        (u0, u1), (v0, v1) = win
        m = np.full(h.shape, -np.inf); m[v0:v1, u0:u1] = h[v0:v1, u0:u1]
    else:
        m = h
    py, px = np.unravel_index(np.argmax(m), h.shape)
    x, y = float(px), float(py)
    H, W = h.shape
    if method == "quarter" and 1 < px < W - 1 and 1 < py < H - 1:   # HRNet code
        x += 0.25 * np.sign(h[py, px + 1] - h[py, px - 1])
        y += 0.25 * np.sign(h[py + 1, px] - h[py - 1, px])
    if method == "dark" and 1 < px < W - 2 and 1 < py < H - 2:      # DarkPose code
        g = gaussian_filter(h, HM_SIGMA, mode="constant")           # modulation
        g = np.log(np.maximum(g * h.max() / g.max(), 1e-10))
        d = np.array([g[py, px + 1] - g[py, px - 1], g[py + 1, px] - g[py - 1, px]]) / 2
        dxx = (g[py, px + 2] - 2 * g[py, px] + g[py, px - 2]) / 4
        dyy = (g[py + 2, px] - 2 * g[py, px] + g[py - 2, px]) / 4
        dxy = (g[py + 1, px + 1] - g[py - 1, px + 1] - g[py + 1, px - 1] + g[py - 1, px - 1]) / 4
        Hs = np.array([[dxx, dxy], [dxy, dyy]])
        if abs(np.linalg.det(Hs)) > 0:
            x, y = np.array([x, y]) - np.linalg.solve(Hs, d)          # Taylor step
    return x, y, h[py, px]


def oks(gt, dt):
    """cocoeval.computeOks for one pair (ground truth with labeled keypoints)."""
    d2 = ((dt["kp"] - gt["kp"]) ** 2).sum(1)
    e = d2 / (2 * SIGMAS) ** 2 / (gt["area"] + np.spacing(1)) / 2
    return np.exp(-e[gt["vis"] > 0]).mean()


def coco_ap(gts, dts, rng_area, thrs=np.linspace(.5, .95, 10), rec=np.linspace(0, 1, 101)):
    """evaluateImg + accumulate of cocoeval.py, keypoints, maxDets = 20."""
    S, M, I, npig = [], [], [], 0
    for G, D in zip(gts, dts):
        gig = np.array([not (rng_area[0] <= g["area"] <= rng_area[1]) for g in G])
        G = [G[i] for i in np.argsort(gig, kind="mergesort")]; gig = np.sort(gig)
        D = [D[i] for i in np.argsort([-d["score"] for d in D], kind="mergesort")][:20]
        iou = np.array([[oks(g, d) for g in G] for d in D])
        dout = np.array([not (rng_area[0] <= d["area"] <= rng_area[1]) for d in D])
        dtm, dig = np.zeros((len(thrs), len(D)), bool), np.zeros((len(thrs), len(D)), bool)
        for t, thr in enumerate(thrs):
            used = np.zeros(len(G), bool)
            for di in range(len(D)):
                best, m = min(thr, 1 - 1e-10), -1
                for gi in range(len(G)):
                    if used[gi] or (m > -1 and not gig[m] and gig[gi]):
                        continue
                    if iou[di, gi] >= best:
                        best, m = iou[di, gi], gi
                if m > -1:
                    used[m], dtm[t, di], dig[t, di] = True, True, gig[m]
        dig |= ~dtm & dout[None, :]
        S += [d["score"] for d in D]; M.append(dtm); I.append(dig); npig += (~gig).sum()
    o = np.argsort(-np.array(S), kind="mergesort")
    M, I = np.concatenate(M, 1)[:, o], np.concatenate(I, 1)[:, o]
    ap = []
    for t in range(len(thrs)):
        tp, fp = np.cumsum(M[t] & ~I[t]), np.cumsum(~M[t] & ~I[t])
        rc, pr = tp / npig, tp / (tp + fp + np.spacing(1))
        pr = np.maximum.accumulate(pr[::-1])[::-1]
        idx = np.searchsorted(rc, rec, side="left")
        ap.append(np.where(idx < len(pr), pr[np.minimum(idx, len(pr) - 1)], 0).mean())
    return 100 * np.mean(ap)


METHODS = ("argmax", "quarter", "dark")
PIPES = {"top-down 256x192": (64, 48), "top-down 128x96": (32, 24), "bottom-up 512": None}


def run(seed, n_img=60, tau=0.25):
    rng = np.random.default_rng(seed)
    gts, preds = [], {(p, m): [] for p in PIPES for m in METHODS}
    for _ in range(n_img):
        people = [stick_figure(rng, rng.uniform(80, 400)) for _ in range(rng.integers(1, 4))]
        G = []
        for J, caps, (hc, hr) in people:
            sil = render(caps)                               # the person's segment
            ys, xs = np.nonzero(sil)
            vis = np.where(rng.random(17) < 0.9, 2, 0); vis[0] = 2
            G.append(dict(J=J, kp=np.array([J[k] for k in COCO]), vis=vis,
                          area=float(sil.sum()), box=(xs.min(), ys.min(), xs.max(), ys.max()),
                          head=0.6 * np.linalg.norm([2 * hr, 2 * hr])))   # MPII head size
        gts.append(G)
        for pipe, hw in PIPES.items():
            out = [{m: {} for m in METHODS} for _ in G]
            if hw:   # top-down: crop each (perfectly detected) person, 4:3 box x 1.25
                for i, g in enumerate(G):
                    x0, y0, x1, y1 = g["box"]
                    c, w, h = np.array([(x0 + x1) / 2, (y0 + y1) / 2]), x1 - x0, y1 - y0
                    w, h = max(w, 0.75 * h) * 1.25, max(h, w / 0.75) * 1.25
                    cell, o = h / hw[0], c - [w / 2, h / 2]  # image px per heatmap cell
                    for k in JOINTS:
                        uv = (g["J"][k] - o) / cell + tau * rng.standard_normal(2)
                        hm = gauss_map(hw, [uv], rng)
                        for m in METHODS:
                            x, y, s = decode(hm, m)
                            out[i][m][k] = (o + cell * np.array([x, y]), s)
            else:    # bottom-up: one 128 x 128 map per joint type, oracle grouping
                for k in JOINTS:
                    uv = [g["J"][k] / 4.0 for g in G]
                    hm = gauss_map((128, 128), [p + tau * rng.standard_normal(2) for p in uv], rng)
                    for i, (u, v) in enumerate(uv):
                        ui, vi = int(round(u)), int(round(v))
                        win = ((max(ui - 3, 0), ui + 4), (max(vi - 3, 0), vi + 4))
                        for m in METHODS:
                            x, y, s = decode(hm, m, win)
                            out[i][m][k] = (4.0 * np.array([x, y]), s)
            for m in METHODS:
                preds[pipe, m].append([o_[m] for o_ in out])
    res = {}
    for key, P in preds.items():
        dts, dist = [], []
        for G, Pi in zip(gts, P):
            D = []
            for g, p in zip(G, Pi):
                kp = np.array([p[k][0] for k in COCO])
                (a, b), (c_, d_) = kp.min(0), kp.max(0)
                D.append(dict(kp=kp, score=np.mean([p[k][1] for k in COCO]),
                              area=(c_ - a) * (d_ - b)))       # loadRes: keypoint box
                dist += [np.linalg.norm(p[k][0] - g["J"][k]) / g["head"] for k in MPII]
            dts.append(D)
        dist = np.array(dist)
        res[key] = [coco_ap(gts, dts, (0, 1e10)), coco_ap(gts, dts, (32**2, 96**2)),
                    coco_ap(gts, dts, (96**2, 1e10)),
                    100 * np.mean(dist <= 0.5), 100 * np.mean(dist <= 0.1)]
    return res


# 1. How far may a keypoint be off before its similarity falls to 0.5?
print("pixel error at which keypoint similarity = 0.5")
for s in (48, 128):                                          # s = sqrt(segment area)
    d = 2 * SIGMAS[[0, 9, 11]] * s * np.sqrt(2 * np.log(2))
    print(f"  s = {s:3d} px: nose {d[0]:5.1f}  wrist {d[1]:5.1f}  hip {d[2]:5.1f}")

# 2. Decoding a noiseless 64 x 48 heatmap whose peak lies at a random sub-cell position.
rng = np.random.default_rng(0)
err = {m: [] for m in METHODS}
for _ in range(1000):
    uv = rng.uniform(5, 40, 2)
    hm = gauss_map((64, 48), [uv], rng, noise=0.0)
    for m in METHODS:
        x, y, _ = decode(hm, m)
        err[m].append(np.hypot(x - uv[0], y - uv[1]))
print("\nmean decoding error on a clean heatmap (cells):",
      "  ".join(f"{m} {np.mean(e):.3f}" for m, e in err.items()))

# 3. Heatmap decoding in full pipelines: ten scenes (seeds 0-9) of 60 images each.
runs = [run(seed) for seed in range(10)]
print("\npipeline          decoder  AP (min-max)          AP_M   AP_L  PCKh@.5  PCKh@.1")
for key in runs[0]:
    r = np.array([x[key] for x in runs])
    mu = r.mean(0)
    print(f"{key[0]:17s} {key[1]:8s} {mu[0]:5.1f} ({r[:, 0].min():5.1f}-{r[:, 0].max():5.1f})"
          f"  {mu[1]:5.1f}  {mu[2]:5.1f}  {mu[3]:6.1f}  {mu[4]:6.1f}")

pixel error at which keypoint similarity = 0.5
  s =  48 px: nose   2.9  wrist   7.0  hip  12.1
  s = 128 px: nose   7.8  wrist  18.7  hip  32.3

mean decoding error on a clean heatmap (cells): argmax 0.390  quarter 0.192  dark 0.000

pipeline          decoder  AP (min-max)          AP_M   AP_L  PCKh@.5  PCKh@.1
top-down 256x192  argmax    96.3 ( 95.4- 97.7)   97.2   95.7   100.0    58.1
top-down 256x192  quarter   99.8 ( 99.5-100.0)   99.9   99.7   100.0    80.0
top-down 256x192  dark     100.0 (100.0-100.0)  100.0  100.0   100.0    90.3
top-down 128x96   argmax    74.8 ( 73.9- 76.0)   76.5   74.1    99.9    20.4
top-down 128x96   quarter   85.2 ( 83.3- 87.4)   86.9   84.4   100.0    33.8
top-down 128x96   dark      90.4 ( 89.3- 91.9)   91.6   90.1   100.0    44.7
bottom-up 512     argmax    91.3 ( 90.0- 92.4)   85.5   97.2    99.6    65.9
bottom-up 512     quarter   95.3 ( 94.1- 96.1)   92.1   98.6    99.6    78.7
bottom-up 512     dark      96.6 ( 95.3- 98.1)   94.5   98.8    99.6    84.1

The script runs in about 15 seconds. The first block shows how unequally OKS treats keypoints: on a person of scale 48 pixels, a nose 2.9 pixels off already loses half its similarity, while a hip may be 12.1 pixels off, because COCO’s annotators themselves placed hips about four times less consistently than noses; the tolerance also grows in proportion to the person’s scale. The second block isolates quantization. On a clean Gaussian the argmax misses by 0.39 cells on average, about what a uniform position within a cell gives; the quarter shift halves that; and DARK’s Taylor step is exact, since the logarithm of a Gaussian is a quadratic that the finite differences recover without error.

The pipelines show what this does to the metrics. PCKh@0.5 is at or near 100% in every row: its tolerance, half of 0.6 times the head diagonal, is about 5.5% of a figure’s height here, more than one heatmap cell in every pipeline (our arithmetic from the figures’ proportions) and above nearly all decoding errors, so it barely sees quantization. COCO AP, which averages over OKS thresholds up to 0.95, and PCKh at a strict 0.1 both can. At $128 \times 96$ input, where a heatmap cell is twice as wide in image pixels as at $256 \times 192$, AP ranges from 74.8 with plain argmax to 90.4 with DARK decoding, the same ordering as DARK’s real HRNet-W32 results at that input size, although the simulated network is otherwise far better than a real one. In the top-down rows, AP for medium and large people is similar, because every crop is resized to the same input. The bottom-up map has the same stride for every person, so its cells are large relative to small people: AP on medium people is 11.7 points below AP on large people with argmax decoding and 4.3 points with DARK, the pattern that OpenPose’s authors reported and HigherHRNet targeted with higher-resolution maps. The spread over the ten seeds is at most 4.1 AP points in any row. Real networks add errors that this simulation leaves out: misdetections, wrong grouping, confusions between left and right, and peaks far from the joint.

Datasets and Benchmarks

• Earlier single-person sets. Parse, with 305 images of full-body poses (Yang and Ramanan, 2011); Buffy Stickmen, frames from a TV series with upper-body stick figures; LSP, sports images with 14 joints, and its 10,000-image extension; and FLIC, frames from Hollywood films with 10 upper-body joints (Toshev and Szegedy, 2014; Andriluka et al., 2014, Table 1). Their small training sets were unsuitable for models with complex appearance representations and multiple components (Andriluka et al., 2014).
• MPII Human Pose. Images from 3,913 YouTube videos chosen through a taxonomy of everyday activities; 24,920 frames with 40,522 annotated people, split by video into 28,821 training and 11,701 test people, covering 491 activities (Andriluka et al., 2014); the dataset’s home page gives 410 activities (as of October 2026). Test annotations are withheld and predictions are submitted to the organizers. Each image comes with neighbouring unannotated video frames.
• COCO keypoints. More than 200,000 images and 250,000 people with keypoints, of which over 150,000 people and 1.7 million labeled keypoints in the public train and val annotations (COCO 2020 keypoint task page); test-dev results come from an evaluation server. The 2017 split and the challenge rules are described in the COCO article.
• Extensions and others. COCO-WholeBody for whole-body keypoints (Jin et al., 2020); CrowdPose for crowded scenes (used by Cheng et al., 2020); PoseTrack for video (Xiao, Wu and Wei, 2018); AI Challenger, used as extra training data by ViTPose (Xu et al., 2022).

Evaluation Metrics

PCP. The percentage of correct parts counts a limb as correct when both of its predicted endpoints lie within half the limb’s length of the ground truth. Yang and Ramanan (2013) noted three problems: the Buffy toolkit implemented a looser variant that scores the average of the endpoints, so published numbers are not comparable; the criterion is too strict for foreshortened limbs and too loose for others; and it leaves open how predictions are matched to people. Andriluka et al. (2014) added PCPm, which uses half the mean ground-truth segment length over the test set.

PCK and PCKh. Yang and Ramanan (2013) defined a predicted keypoint as correct if it lies within $\alpha \cdot \max(h, w)$ pixels of the ground truth, where $h$ and $w$ are the height and width of the tightest box around the person’s annotated keypoints; they used $\alpha = 0.1$ on Parse and $0.2$ on the half-body Buffy. DeepPose’s PDJ and FLIC’s PCK normalize by torso size instead. Andriluka et al. (2014) normalized by head size so that the threshold does not depend on articulation, setting it at 50% of the head segment length and calling the measure PCKh. The official MPII code computes the head size from the head rectangle instead and defines, for joint $k$ of person $n$,

$$\mathrm{PCKh}@\alpha = \frac{1}{|\mathcal{V}|} \sum_{(n, k) \in \mathcal{V}} \mathbf{1}\!\left[ \frac{\lVert \hat{\mathbf{x}}_{nk} - \mathbf{x}_{nk} \rVert}{\ell_n} \le \alpha \right], \qquad \ell_n = 0.6 \, \lVert (x_2, y_2) - (x_1, y_1) \rVert,$$

where $\ell_n$ is 0.6 times the diagonal of the annotated head rectangle and $\mathcal{V}$ the joints with ground truth. In detail (evaluatePCKh.m, util_get_head_size.m, computePCK.m): the single-person subset is scored on 14 joints, without pelvis and thorax; joints without ground truth are ignored, while missing predictions are set to infinity and count as wrong; the “Total” pools all joints of all people rather than averaging per-joint scores; the reported table is at $\alpha = 0.5$, and curves over $\alpha$ from 0 to 0.5 also give an area under the curve. Multi-person MPII is scored by mean average precision over joint detections instead (MPII results page).

Object keypoint similarity (OKS). COCO scores multi-person pose like detection: OKS plays the role of IoU. For a ground-truth person with scale $s$, the square root of its segment area, and per-keypoint constants $\kappa_i$,

$$\mathrm{OKS} = \frac{\sum_i \exp\!\left( -d_i^2 / 2 s^2 \kappa_i^2 \right) \delta(v_i > 0)}{\sum_i \delta(v_i > 0)},$$

where $d_i$ is the distance between the predicted and true keypoint and the sum runs over labeled keypoints only; predicted visibility flags are not used (keypoint evaluation page). The organizers measured the per-keypoint spread $\sigma_i^2 = E[d_i^2 / s^2]$ of human annotations on 5,000 doubly annotated val images and set $\kappa_i = 2\sigma_i$, so that an error of one, two or three annotator standard deviations gives a similarity of 0.88, 0.61 or 0.32. The page lists $\sigma$ = .026 for the nose, .025 for the eyes, .035 for the ears, .079 for the shoulders, .072 for the elbows, .062 for the wrists, .107 for the hips, .087 for the knees and .089 for the ankles. On the same data, human performance at OKS 0.50 is nearly perfect (95%) and the median human OKS is about 0.91.

The official code (cocoeval.py, computeOks) implements this as e = d² / (2σ)² / (area + eps) / 2 with the segment area of the ground truth in place of $s^2$, and averages $e^{-e}$ over keypoints with $v > 0$. People with no labeled keypoints, and crowds, are ignored: a detection matched to them counts neither as a true nor as a false positive. To let detections match them at all, the code measures each predicted keypoint’s distance to a region around the person’s box, one box width and height beyond it on every side (the code comment calls it double the box), instead of to keypoints; the page calls this a lenient heuristic.

COCO keypoint AP. Detections are matched greedily in score order to the unmatched person with the highest OKS above the threshold, and precision–recall is summarized as in box detection (see average precision and precision and recall). The primary metric averages AP over OKS thresholds 0.50:0.05:0.95; the others are AP at 0.50 and 0.75, AP for medium ($32^2$ to $96^2$) and large ($> 96^2$) people, and the corresponding average recalls, with at most 20 detections per image (keypoint evaluation page; setKpParams). There is no small-person AP, since small people carry no keypoints. Each person counts equally, however many keypoints are labeled. One detail affects the area-specific numbers: when keypoint results are loaded, a detection’s area is set to the area of the box around its predicted keypoints (coco.py, loadRes), and an unmatched detection whose area falls outside the range is ignored.

Applications

• Action and activity recognition. Joint sequences are the input of skeleton-based action recognition, part of video understanding; its benchmarks use skeletons from Kinect sensors or from 2D estimators such as OpenPose and HRNet.
• Interaction and gaming. Shotton et al. (2011) list gaming, human–computer interaction, security, telepresence and health care as uses of body tracking; their method is a core component of the Kinect gaming platform.
• Animation and motion capture. Sun et al. (2019) cite animation and human–computer interaction among the uses; OpenPose’s authors added foot keypoints because 3D body reconstruction needs heels and toes (Cao et al., 2021).

Open Problems

• Crowds and occlusion. Grouping and detection both suffer when people overlap. CrowdPose was built for this case, and on it HigherHRNet’s bottom-up model outperformed the top-down methods its authors compared (Cheng et al., 2020).
• Benchmark saturation. In our reading, once networks are good a loose threshold such as MPII’s PCKh@0.5 mostly measures gross errors, which the strict end of COCO’s AP still separates; the example illustrates this but does not test real models.
• Sub-pixel precision versus cost. Higher-resolution heatmaps, larger inputs and decoders such as DARK each buy accuracy; ViTPose’s best COCO result used a 1-billion-parameter backbone and a $576 \times 432$ input (Xu et al., 2022).
• From 2D to 3D and to whole bodies. Recovering 3D pose, hands, faces and feet together, in the wild and in real time, extends the task beyond COCO’s 17 body keypoints.