OpenPose

OpenPose is a bottom-up method for multi-person 2D human pose estimation, and the name of the open-source library that implements it. Zhe Cao, Tomas Simon, Shih-En Wei and Yaser Sheikh of Carnegie Mellon University’s Robotics Institute introduced the method at CVPR 2017 as “Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields”. A convolutional neural network predicts, for the whole image at once, one confidence map per body keypoint, in the manner of heatmap regression, and one 2D vector field per limb, the part affinity field (PAF), whose vectors point along the limb from one joint to the other. Keypoint candidates come from the peaks of the maps, and each possible limb between two candidates is scored by integrating the PAF along the segment that joins them; a greedy set of bipartite matchings then assembles the candidates into skeletons. The method won the inaugural COCO keypoint challenge in 2016. A journal version (IEEE TPAMI, published online in 2019) added foot keypoints and described the OpenPose library, which also detects hand and face keypoints; the library’s body-foot model refines the PAFs before the confidence maps. The approach is also called PAF-based pose estimation.

Problem Addressed

Multi-person pose estimation has to find an unknown number of people, at any position and scale, whose limbs touch, cross and occlude one another. Cao et al. (2017) named the two strategies of the time and the weakness of each. Top-down systems run a person detector and then a single-person pose estimator on each box: they commit early, so a person the detector misses, as happens when people are close together, cannot be recovered, and their runtime grows with the number of people. Bottom-up systems detect all body parts first and then assign them to people, which decouples the runtime from the number of people, but the published ones lost that advantage in the assignment step. DeepCut (Pishchulin et al., CVPR 2016) labelled part candidates and partitioned them into people with an integer linear program over a fully connected graph, which is NP-hard; Cao et al. put its processing time on the order of hours per image. DeeperCut (Insafutdinov et al., ECCV 2016) used stronger ResNet part detectors and image-dependent pairwise terms and still needed several minutes per image, with a cap on the number of part proposals.

The CVPR 2017 paper set out a representation of association that a network can regress directly and that is informative enough for a cheap, greedy parse: the part affinity field. Its abstract reports first place in the COCO 2016 keypoints challenge and a result on MPII multi-person well above the previous best, in accuracy and speed. The task, the two families and their benchmarks are described in human pose estimation; this article covers the model.

Architecture

The network maps a $w \times h$ colour image to a set of $J$ confidence maps $\mathbf{S} = (S_1, \dots, S_J)$, one per body part, with $S_j \in \mathbb{R}^{w \times h}$, and a set of $C$ vector fields $\mathbf{L} = (L_1, \dots, L_C)$, one per limb, with $L_c \in \mathbb{R}^{w \times h \times 2}$. The paper calls every connected pair of parts a “limb” for clarity, including pairs that are not anatomical limbs, such as those in the face (Cao et al., 2017, footnote 1). A parsing step outside the network turns the maps into skeletons.

Backbone

The image first passes through the first 10 layers of VGG-19, initialized from the ImageNet classifier and fine-tuned, which produce a feature map $\mathbf{F}$ (Cao et al., 2017, §2.1). In the released COCO model, $\mathbf{F}$ comes from a layer named conv4_4_CPM (OpenPose repository, pose_deploy_linevec.prototxt), and the released training configuration produces maps at stride 8 with respect to the network input (OpenPose Training, d_setLayers.py).

Two-Branch Multi-Stage Network

The CVPR 2017 network follows the sequential prediction design of convolutional pose machines (Wei et al., 2016), split into two branches. In the first stage, one branch predicts confidence maps and the other PAFs from the image features alone, $\mathbf{S}^1 = \rho^1(\mathbf{F})$ and $\mathbf{L}^1 = \phi^1(\mathbf{F})$. Every later stage takes the image features together with both predictions of the previous stage:

$$\mathbf{S}^t = \rho^t\big(\mathbf{F}, \mathbf{S}^{t-1}, \mathbf{L}^{t-1}\big), \qquad \mathbf{L}^t = \phi^t\big(\mathbf{F}, \mathbf{S}^{t-1}, \mathbf{L}^{t-1}\big), \qquad t \ge 2,$$

where $\rho^t$ and $\phi^t$ are small CNNs (Cao et al., 2017, Eqs. 1–2). In the paper’s Fig. 3, the first stage uses $3 \times 3$ and $1 \times 1$ convolutions and later stages five $7 \times 7$ convolutions followed by two $1 \times 1$ convolutions in each branch. The large kernels give later stages a large receptive field, so a prediction at one limb can draw on parts and people far from it. The paper’s Fig. 4 shows confidence maps and PAFs that confuse left and right limbs in stage 1 and resolve the confusion by stage 6. The released COCO model has six stages; its confidence branch outputs 19 channels, 18 keypoints plus a background map, and its PAF branch 38 channels, an $x$ and a $y$ component for each of 19 connections (OpenPose repository, pose_deploy_linevec.prototxt).

Confidence Maps

A confidence map holds, at each pixel, the belief that a given part is there; with several people it should have one peak per visible instance of the part. The target for person $k$ and part $j$, whose annotated position is $\mathbf{x}_{j,k}$, is

$$S^*_{j,k}(\mathbf{p}) = \exp\!\left(-\frac{\lVert \mathbf{p} - \mathbf{x}_{j,k} \rVert_2^2}{\sigma^2}\right), \qquad S^*_j(\mathbf{p}) = \max_k S^*_{j,k}(\mathbf{p}),$$

with $\sigma$ controlling the spread (Cao et al., 2017, Eqs. 6–7). The paper takes the maximum over people rather than the average so that two nearby peaks stay distinct instead of merging into one between them. The released training code writes the Gaussian with $2\sigma^2$ in the denominator, clips the map at 1 and leaves cells below 0.01 untouched (openpose_caffe_train, putGaussianMaps); the paper’s equation has no factor 2. Target design and the decoding of heatmaps in general belong to heatmap regression.

Part Affinity Fields

The central idea of OpenPose is the representation of association. A simpler alternative, which the paper uses as a baseline, is to detect an extra point at the middle of each limb and accept a pair of candidates if a midpoint detection lies between them. Cao et al. (2017, Fig. 5) argue that in crowds such midpoints support false pairs, because the representation keeps only the limb’s position, not its orientation, and shrinks its support to a single point. A PAF keeps both position and orientation over the whole area of the limb.

For limb $c$ of person $k$, joining parts $j_1$ and $j_2$ at $\mathbf{x}_{j_1,k}$ and $\mathbf{x}_{j_2,k}$, let $\mathbf{v} = (\mathbf{x}_{j_2,k} - \mathbf{x}_{j_1,k}) / \lVert \mathbf{x}_{j_2,k} - \mathbf{x}_{j_1,k} \rVert_2$ be the unit vector along the limb, $\mathbf{v}_\perp$ a unit vector perpendicular to it and $l_{c,k}$ the limb’s length. The target is

$$L^*_{c,k}(\mathbf{p}) = \begin{cases} \mathbf{v} & \text{if } 0 \le \mathbf{v} \cdot (\mathbf{p} - \mathbf{x}_{j_1,k}) \le l_{c,k} \text{ and } \lvert \mathbf{v}_\perp \cdot (\mathbf{p} - \mathbf{x}_{j_1,k}) \rvert \le \sigma_l, \\ \mathbf{0} & \text{otherwise,} \end{cases}$$

where the limb width $\sigma_l$ is a distance in pixels. Where the limbs of several people overlap, the target is the average of the non-zero vectors,

$$L^*_c(\mathbf{p}) = \frac{1}{n_c(\mathbf{p})} \sum_k L^*_{c,k}(\mathbf{p}),$$

with $n_c(\mathbf{p})$ the number of people whose limb $c$ covers $\mathbf{p}$ (Cao et al., 2017, Eqs. 8–9). The paper does not give the value of $\sigma_l$. The released training code fills every cell of the output grid whose perpendicular distance to the limb’s line is at most 1 cell, inside the segment’s bounding box widened by 1 cell on each side, and averages where people overlap (openpose_caffe_train, putVectorMaps, with threshold = 1). At stride 8 that is a half-width of 8 input pixels; the bounding-box test also lets the field run slightly past the joints, which the paper’s condition on $\mathbf{v} \cdot (\mathbf{p} - \mathbf{x}_{j_1,k})$ does not.

PAF-First Refinement in the Journal Version

The journal version changed the stages. Its text could not be read for this article (the publisher’s site and the arXiv preprint are among the sources this site does not use), so its design is described here from the authors’ own later paper and from the released model. Hidalgo et al. (2019), by the OpenPose group, review “the PAF-based method” whose architecture they reuse as their baseline, which they call that of Cao et al. (citing the journal paper’s December 2018 arXiv preprint); for a full description they refer to the CVPR paper, although its two-branch stages differ from their summary. In that summary, the first stage predicts PAFs from the features, each later stage refines the PAFs from the features and the previous PAFs, and only after $N$ PAF stages does a network $\rho$ predict the confidence maps from the features and the final PAFs:

$$\mathbf{L}^1 = \phi^1(\mathbf{F}), \qquad \mathbf{L}^t = \phi^t\big(\mathbf{F}, \mathbf{L}^{t-1}\big) \;\; (2 \le t \le N), \qquad \mathbf{S} = \rho\big(\mathbf{F}, \mathbf{L}^N\big)$$

(Hidalgo et al., 2019, Eqs. 1–4). The released BODY_25 model has this order: four PAF stages followed by two confidence-map stages, each of which also receives the last PAF prediction. Instead of $7 \times 7$ kernels, each stage repeats a block of three chained $3 \times 3$ convolutions whose three outputs are concatenated, five blocks per stage, with PReLU activations; the outputs are 52 PAF channels, 26 connections with two components each, and 26 confidence maps, 25 keypoints plus background (OpenPose repository, models/pose/body_25/pose_deploy.prototxt; the training configuration sets four PAF stages and two confidence-map stages, OpenPose Training, d_setLayers.py). The 26 connections are the 24 edges of a tree over the 25 keypoints plus two redundant ear-to-shoulder connections (OpenPose repository, poseParameters.cpp). The journal paper’s reasons for refining PAFs first, and its ablations of the change, are not reported here because they were not read.

Training

The loss is an $L_2$ distance between predicted and target maps at the end of every stage, which is intermediate supervision in the manner of convolutional pose machines. A binary mask $W(\mathbf{p})$ is zero where the annotation is missing, so that people visible in an image but not annotated are not penalized as false positives (Cao et al., 2017, §2.1):

$$f^t_{\mathbf{S}} = \sum_{j=1}^{J} \sum_{\mathbf{p}} W(\mathbf{p}) \, \big\lVert S^t_j(\mathbf{p}) - S^*_j(\mathbf{p}) \big\rVert_2^2, \qquad f^t_{\mathbf{L}} = \sum_{c=1}^{C} \sum_{\mathbf{p}} W(\mathbf{p}) \, \big\lVert L^t_c(\mathbf{p}) - L^*_c(\mathbf{p}) \big\rVert_2^2,$$ $$f = \sum_{t=1}^{T} \big( f^t_{\mathbf{S}} + f^t_{\mathbf{L}} \big)$$

(Eqs. 3–5). The paper says the periodic supervision counters vanishing gradients by replenishing the gradient at each stage. On its MPII validation set, training with masks for unlabelled people added 2.3 points of mAP at PCKh@0.5 (Fig. 7a and §3.1).

The CVPR model was trained on COCO, whose training set Cao et al. describe as over 100,000 person instances with over a million keypoints, and on MPII Human Pose for the MPII experiments. For MPII they trained the network on the edges of a fully connected graph of the 14 parts and also a second network on only the 13 edges of a tree (Table 2). The paper gives no optimizer or schedule. The released training configuration of the BODY_25 model uses $368 \times 368$ crops, stride 8, a batch of 10 images, ImageNet-pretrained VGG-19 weights and 1,000,000 iterations, with a random scale augmentation between 1/3 and 1.1 (OpenPose Training, d_setLayers.py).

Foot keypoints needed new labels, since COCO and MPII mark each foot only by the ankle. The authors labelled six foot keypoints per person, the big toe, small toe and heel of each foot, on a subset of COCO people with the Clickworker platform: 14,000 annotations from the training set and 545 from the validation set, by the journal version’s text as quoted on the dataset’s official page. The same page lists the downloadable files as about 13,500 training and about 500 validation annotations, and does not explain the difference; Hidalgo et al. (2019) describe the dataset as a subset of 15k annotations. The page’s quotation motivates the foot points by graphics applications, such as avatar retargeting and 3D body reconstruction, that need the heel and toes. The BODY_25 model was trained on the body and foot datasets together; Hidalgo et al. (2019) describe the method as drawing each batch from one dataset with a set probability and masking the loss of the channels whose keypoints that dataset does not annotate, which they credit to Cao et al.’s body-foot detector.

Inference

At test time the network runs once on the whole image. Hidalgo et al. (2019, Fig. 2d), whose method leaves the rest of OpenPose’s pipeline intact, resize the final maps to a higher resolution before parsing to increase accuracy. In the library, each peak’s position is refined to the score-weighted centroid of a $7 \times 7$ window around it (OpenPose repository, nmsBase.cpp).

Keypoint Candidates

Non-maximum suppression on each confidence map gives a set of candidates $D_J = \{\mathbf{d}^m_j : j = 1 \dots J,\ m = 1 \dots N_j\}$, where $N_j$ is the number of candidates of part $j$. A part can have several candidates because there are several people or false detections. In the library, a candidate is a strict local maximum in a $3 \times 3$ neighbourhood above a threshold of 0.05 for the models other than the MPII ones (nmsBase.cpp, poseParameters.cpp).

Scoring Candidate Limbs by Line Integrals

For two candidates $\mathbf{d}_{j_1}$ and $\mathbf{d}_{j_2}$ of the parts of limb $c$, the association score is the line integral of the predicted PAF along the segment joining them, projected on the segment’s direction:

$$E = \int_{u=0}^{u=1} L_c\big(\mathbf{p}(u)\big) \cdot \frac{\mathbf{d}_{j_2} - \mathbf{d}_{j_1}}{\lVert \mathbf{d}_{j_2} - \mathbf{d}_{j_1} \rVert_2} \, du, \qquad \mathbf{p}(u) = (1 - u)\,\mathbf{d}_{j_1} + u\,\mathbf{d}_{j_2}$$

(Cao et al., 2017, Eqs. 10–11). A correct pair lies along a field that points in its own direction, so $E$ is close to 1; a pair whose segment crosses empty space or another person’s limb at an angle scores near 0 or below. The paper approximates the integral by sampling and summing uniformly spaced values of $u$, without saying how many. The library’s version (getScoreAB in bodyPartConnectorBase.cpp) takes $n = \max(5, \min(25, \operatorname{round}(\sqrt{5\,\Delta})))$ samples, where $\Delta$ is the larger of the horizontal and vertical extents of the segment in map cells, at $u = k/n$ for $k = 0, \dots, n - 1$, reading the nearest cell. It counts the samples whose projection exceeds 0.05, and accepts the pair only if more than 95% of the samples pass, returning the mean of the passing samples; otherwise the score is 0, except that two candidates closer than a small distance get a minimal positive score so that very short limbs, where the field is empty, are not lost. With the library’s --maximize_positives flag the thresholds are looser: 0.01 per sample and 75% of samples (flags.hpp, poseParameters.cpp). The 95% rule is strict: with $n$ up to 20 samples, $19/20 = 0.95$ does not exceed 0.95, so a single failing sample rejects the pair.

Greedy Bipartite Matching

Choosing which candidates to connect for one limb type is a maximum-weight bipartite matching. With $z^{mn}_{j_1 j_2} \in \{0, 1\}$ indicating that candidates $\mathbf{d}^m_{j_1}$ and $\mathbf{d}^n_{j_2}$ are connected and $E_{mn}$ their score,

$$\max_{Z_c} E_c = \max_{Z_c} \sum_{m \in D_{j_1}} \sum_{n \in D_{j_2}} E_{mn} \, z^{mn}_{j_1 j_2} \quad \text{subject to} \quad \sum_{n} z^{mn}_{j_1 j_2} \le 1 \;\; \forall m, \qquad \sum_{m} z^{mn}_{j_1 j_2} \le 1 \;\; \forall n,$$

so that no two limbs of the same type share a part (Cao et al., 2017, Eqs. 12–14). The paper notes that the Hungarian algorithm solves this subproblem optimally. Finding whole skeletons for many people at once is a $K$-dimensional matching problem, which is NP-hard, and the paper relaxes it twice: it keeps only the edges of a spanning tree of the skeleton instead of the complete graph between parts, and it solves the bipartite matching of each tree edge independently,

$$\max_{Z} E = \sum_{c=1}^{C} \max_{Z_c} E_c$$

(Eq. 15). Limbs that share a candidate are then merged into people. The authors justify the relaxation by the network: relations between adjacent parts are modelled explicitly by the PAFs, and relations between non-adjacent parts implicitly, because the network has a large receptive field and is trained with PAFs of other limbs.

The released library does not run the Hungarian algorithm. For each limb type it sorts all candidate pairs with a positive score in decreasing order and accepts a pair when neither of its candidates has been used, stopping after $\min(N_{j_1}, N_{j_2})$ pairs. It then grows people along the tree in a fixed order of connections, adds the ear connections only where they do not conflict, and removes people with fewer than 3 parts or with a mean score below 0.4, where a person’s score sums its part confidences and connection scores (bodyPartConnectorBase.cpp and poseParameters.cpp). A later release moved this greedy connector to the GPU (release notes, version 1.5.0).

Speed

The CNN’s cost does not depend on the number of people, and the parse grows as $O(n^2)$ in the number of people $n$ (Cao et al., 2017, §3.3). On a laptop with an NVIDIA GTX 1080, with $1080 \times 1920$ video frames resized to $368 \times 654$, parsing nine people took 0.58 ms against 99.6 ms for the CNN, and a video with 19 people ran at 8.8 frames per second. A top-down baseline of a person detector plus a single-person convolutional pose machine took roughly time proportional to the number of people (Fig. 8d).

Example: Grouping by Line Integrals on Synthetic Scenes

The script below re-implements the parse (no official code is used) on synthetic two-person scenes, where the true grouping is known. Each scene has two stick figures with the 14 parts and 13 tree limbs of the paper’s MPII model, on a $96 \times 96$ grid standing in for the network’s output; the necks are 14 to 30 cells apart and the arms take random angles, so arms of the two people often overlap and cross. No network is trained. Instead, the “predicted” maps are the paper’s targets rendered from the skeletons: confidence maps with $\sigma = 1.5$ cells, PAFs with a half-width of 1 cell, and, for the baseline, one midpoint map per limb, the paper’s one-midpoint representation. Two kinds of error stand in for a network’s: the joints are jittered by 0.5 cells (standard deviation) before rendering, and spatially smoothed Gaussian noise of standard deviation $s$ is added to every map. That error model is this example’s choice and not a property of OpenPose.

Candidates are $3 \times 3$ local maxima above 0.5. Each candidate pair is scored three ways: the PAF line integral in the paper’s form (mean of 10 uniform samples, any positive score eligible), the PAF score in the library’s form (the rule above), and the midpoint baseline (the midpoint map’s value at the pair’s midpoint). Each limb type is matched greedily, as in the library, and with the Hungarian algorithm. Two choices are left open by the sources and are both run: the scoring form, and the region the PAF covers, either the paper’s segment condition (Eq. 8) or the training code’s widened bounding box. Results count, over 7,800 ground-truth limbs, the share connected correctly, connections between the two people (“cross”), connections involving a candidate that matches no true joint within 3 cells (“false”), and the share of people recovered exactly, with all 14 parts and nothing else. The output was produced with NumPy 2.3.5 and SciPy 1.16.3; the script takes about half a minute to a minute on a laptop.

import numpy as np
from scipy.ndimage import gaussian_filter, maximum_filter
from scipy.optimize import linear_sum_assignment

H = W = 96                      # output grid of the simulated network
SIGMA, LIMB_W = 1.5, 1.0        # confidence-map spread and PAF half-width, in cells
JOINTS = ["head", "neck", "r_sho", "r_elb", "r_wri", "l_sho", "l_elb", "l_wri",
          "r_hip", "r_kne", "r_ank", "l_hip", "l_kne", "l_ank"]
LIMBS = [(1, 0), (1, 2), (2, 3), (3, 4), (1, 5), (5, 6), (6, 7),
         (1, 8), (8, 9), (9, 10), (1, 11), (11, 12), (12, 13)]   # 13-edge tree
yy, xx = np.mgrid[:H, :W].astype(float)

def make_person(rng, neck):
    j = np.zeros((14, 2)); j[1] = neck; j[0] = neck + [0, -8]
    for s, (sho, elb, wri, hip, kne, ank) in zip((-1, 1), ((2, 3, 4, 8, 9, 10), (5, 6, 7, 11, 12, 13))):
        j[sho] = neck + [6 * s, 1]
        a1, a2 = rng.uniform(0, 2 * np.pi, 2)              # arms anywhere
        j[elb] = j[sho] + 12 * np.array([np.cos(a1), np.sin(a1)])
        j[wri] = j[elb] + 11 * np.array([np.cos(a2), np.sin(a2)])
        j[hip] = neck + [4 * s, 22]
        for p, c, L in ((hip, kne, 14), (kne, ank, 13)):  # legs within 35 deg of vertical
            b = np.deg2rad(rng.uniform(-35, 35))
            j[c] = j[p] + L * np.array([np.sin(b), np.cos(b)])
    return j

def make_scene(rng):
    while True:
        n1 = np.array([rng.uniform(30, 50), rng.uniform(18, 26)])
        n2 = n1 + [rng.uniform(14, 30), rng.uniform(-6, 6)]
        people = [make_person(rng, n1), make_person(rng, n2)]
        if all((p >= 1).all() and (p <= W - 2).all() for p in people):
            return people

def render(people, region):
    """Targets of Cao et al. (2017): Eqs. 6-7 (max) and 8-9 (average). region "paper":
    0 <= along <= l (Eq. 8); "code": the segment's box widened by LIMB_W, as putVectorMaps."""
    S = np.zeros((14, H, W)); M = np.zeros((13, H, W))
    L = np.zeros((13, 2, H, W)); n = np.zeros((13, H, W))
    for P in people:
        for j in range(14):
            S[j] = np.maximum(S[j], np.exp(-((xx - P[j, 0])**2 + (yy - P[j, 1])**2) / SIGMA**2))
        for c, (a, b) in enumerate(LIMBS):
            d = P[b] - P[a]; l = np.linalg.norm(d); v = d / l
            px, py = xx - P[a, 0], yy - P[a, 1]
            along, across = v[0] * px + v[1] * py, -v[1] * px + v[0] * py
            if region == "paper":
                on = (along >= 0) & (along <= l) & (np.abs(across) <= LIMB_W)
            else:
                lo, hi = np.rint(np.minimum(P[a], P[b]) - LIMB_W), np.rint(np.maximum(P[a], P[b]) + LIMB_W)
                on = (xx >= lo[0]) & (xx < hi[0]) & (yy >= lo[1]) & (yy < hi[1]) & (np.abs(across) <= LIMB_W)
            L[c, 0][on] += v[0]; L[c, 1][on] += v[1]; n[c][on] += 1
            m = (P[a] + P[b]) / 2
            M[c] = np.maximum(M[c], np.exp(-((xx - m[0])**2 + (yy - m[1])**2) / SIGMA**2))
    L /= np.maximum(n, 1)[:, None]
    return S, L, M

def noisy(rng, maps, s):
    e = gaussian_filter(rng.normal(size=maps.shape), 1.0, axes=(-2, -1))
    return maps + s * e / e.std()

def peaks(Sj, thr=0.5):
    ys, xs = np.nonzero((Sj == maximum_filter(Sj, size=3)) & (Sj > thr))
    return np.stack([xs, ys], 1).astype(float)

def cell(p):
    return np.clip(np.rint(p).astype(int), 0, W - 1)

def pair_samples(d1, d2, u):
    """Sample points p(u) = d1 + u (d2 - d1) for all candidate pairs (Eq. 11)."""
    D = d2[None, :, :] - d1[:, None, :]                       # (na, nb, 2)
    return D, cell(d1[:, None, None, :] + u[..., None] * D[:, :, None, :])

def score_paper(Lc, d1, d2, n=10):           # Eq. 10 by 10 uniform samples
    D, q = pair_samples(d1, d2, np.linspace(0, 1, n)[None, None, :])
    v = D / np.maximum(np.linalg.norm(D, axis=-1, keepdims=True), 1e-6)
    s = Lc[0][q[..., 1], q[..., 0]] * v[..., None, 0] + Lc[1][q[..., 1], q[..., 0]] * v[..., None, 1]
    return s.mean(-1)

def score_library(Lc, d1, d2):               # getScoreAB in bodyPartConnectorBase.cpp
    D = d2[None, :, :] - d1[:, None, :]; norm = np.linalg.norm(D, axis=-1)
    n = np.clip(np.rint(np.sqrt(5 * np.abs(D).max(-1))).astype(int), 5, 25)
    k = np.arange(25)
    D, q = pair_samples(d1, d2, k / n[..., None])
    s = (Lc[0][q[..., 1], q[..., 0]] * D[..., None, 0] + Lc[1][q[..., 1], q[..., 0]] * D[..., None, 1]) \
        / np.maximum(norm, 1e-6)[..., None]
    keep = (s > 0.05) & (k < n[..., None])
    cnt = keep.sum(-1)
    ok = (cnt / n > 0.95) & (norm >= 1e-6)
    return np.where(ok, np.where(keep, s, 0).sum(-1) / np.maximum(cnt, 1), 0.0)

def score_midpoint(Mc, d1, d2):              # one-midpoint baseline (Fig. 5b)
    q = cell((d1[:, None, :] + d2[None, :, :]) / 2)
    return Mc[q[..., 1], q[..., 0]]

def match(E, greedy):
    """E: score matrix, entries <= 0 not allowed. Returns list of (i, k)."""
    if greedy:
        used_a, used_b, out = set(), set(), []
        order = np.argsort(-E, axis=None, kind="stable")
        for i, k in zip(*np.unravel_index(order[:np.count_nonzero(E > 0)], E.shape)):
            if i not in used_a and k not in used_b:
                out.append((i, k)); used_a.add(i); used_b.add(k)
        return out
    r, c = linear_sum_assignment(np.where(E > 0, E, 0), maximize=True)
    return [(i, k) for i, k in zip(r, c) if E[i, k] > 0]

def parse_and_score(people, cands, conn):
    """conn[c] = list of (i, k) candidate pairs. Returns counts for evaluation."""
    owner = []                                # (person, joint) or None for each candidate
    for j, C in enumerate(cands):
        d = np.linalg.norm(C[:, None, :] - np.array([P[j] for P in people])[None], axis=-1)
        owner.append([int(np.argmin(r)) if r.min() <= 3 else None for r in d])
    correct = cross = false_c = 0
    parent = {}                               # union-find over (joint, candidate) nodes
    def find(u):
        while parent.setdefault(u, u) != u: u = parent[u]
        return u
    for c, (a, b) in enumerate(LIMBS):
        for i, k in conn[c]:
            oa, ob = owner[a][i], owner[b][k]
            if oa is None or ob is None: false_c += 1
            elif oa != ob: cross += 1
            else: correct += 1
            parent[find((a, i))] = find((b, k))
    exact = 0
    for pi in range(len(people)):
        mine = [(j, i) for j in range(14) for i, o in enumerate(owner[j]) if o == pi]
        if len(mine) != 14 or len({j for j, _ in mine}) != 14: continue
        root = find(mine[0])
        comp = [u for u in parent if find(u) == root] if mine[0] in parent else [mine[0]]
        if set(comp) == set(mine): exact += 1
    return correct, cross, false_c, exact

N_SCENES = 300
METHODS = [("PAF paper form", "paper"), ("PAF library form", "library"), ("midpoint", "mid")]
rng = np.random.default_rng(0)
scenes = [make_scene(rng) for _ in range(N_SCENES)]
print(f"{N_SCENES} two-person scenes, {2 * 13 * N_SCENES} ground-truth limbs; "
      "each cell: greedy / Hungarian")
print(f"{'region':<6} {'noise':>5} {'score':<17} {'limb recall %':>13} {'cross':>9} {'false':>13} {'exact people %':>14}")
for region in ("paper", "code"):
    for s in (0.05, 0.10, 0.20):
        noise_rng = np.random.default_rng(1)
        tallies = {(m, g): np.zeros(4) for _, m in METHODS for g in (True, False)}
        n_cand = 0
        for people in scenes:
            jittered = [P + noise_rng.normal(0, 0.5, P.shape) for P in people]
            S, L, M = render(jittered, region)
            S, L, M = noisy(noise_rng, S, s), noisy(noise_rng, L, s), noisy(noise_rng, M, s)
            cands = [peaks(S[j]) for j in range(14)]
            n_cand += sum(len(C) for C in cands)
            for _, m in METHODS:
                f = {"paper": score_paper, "library": score_library, "mid": score_midpoint}[m]
                E = [f(M[c] if m == "mid" else L[c], cands[a], cands[b]) if len(cands[a]) and len(cands[b])
                     else np.zeros((0, 0)) for c, (a, b) in enumerate(LIMBS)]
                for g in (True, False):
                    conn = [match(Ec, g) if Ec.size else [] for Ec in E]
                    tallies[(m, g)] += parse_and_score(people, cands, conn)
        for name, m in METHODS:
            (c1, x1, f1, e1), (c2, x2, f2, e2) = tallies[(m, True)], tallies[(m, False)]
            print(f"{region:<6} {s:5.2f} {name:<17} {100 * c1 / (26 * N_SCENES):6.1f}/{100 * c2 / (26 * N_SCENES):5.1f} "
                  f"{int(x1):4d}/{int(x2):<4d} {int(f1):6d}/{int(f2):<6d} "
                  f"{100 * e1 / (2 * N_SCENES):6.1f}/{100 * e2 / (2 * N_SCENES):5.1f}")
        print(f"{region:<6} {s:5.2f} mean candidates per confidence map: {n_cand / (14 * N_SCENES):.2f}")

300 two-person scenes, 7800 ground-truth limbs; each cell: greedy / Hungarian
region noise score             limb recall %     cross         false exact people %
paper   0.05 PAF paper form      99.6/ 99.9   26/2         0/0        94.8/ 98.5
paper   0.05 PAF library form    57.0/ 57.1   14/10        0/0         0.0/  0.0
paper   0.05 midpoint            99.9/ 99.9    2/2         0/0        98.5/ 98.5
paper   0.05 mean candidates per confidence map: 2.00
paper   0.10 PAF paper form      99.5/ 99.9   28/2         1/1        94.5/ 98.5
paper   0.10 PAF library form    64.7/ 64.8   15/9         0/0         0.5/  0.5
paper   0.10 midpoint            99.9/ 99.9    1/1         0/0        98.7/ 98.7
paper   0.10 mean candidates per confidence map: 2.03
paper   0.20 PAF paper form      93.4/ 92.3   26/13    91785/93868    40.5/ 33.8
paper   0.20 PAF library form    63.7/ 63.7   17/13     7636/7687      0.0/  0.0
paper   0.20 midpoint            91.3/ 89.6    6/5     93416/94487    31.2/ 26.8
paper   0.20 mean candidates per confidence map: 28.87
code    0.05 PAF paper form      99.7/ 99.8   15/3         0/0        96.3/ 98.3
code    0.05 PAF library form    96.7/ 96.9   14/4         0/0        69.8/ 72.5
code    0.05 midpoint            99.9/ 99.9    2/2         0/0        98.5/ 98.5
code    0.05 mean candidates per confidence map: 2.00
code    0.10 PAF paper form      99.6/ 99.8   19/3         1/1        96.0/ 98.3
code    0.10 PAF library form    95.5/ 95.7   18/7         0/0        60.8/ 62.8
code    0.10 midpoint            99.9/ 99.9    1/1         0/0        98.7/ 98.7
code    0.10 mean candidates per confidence map: 2.03
code    0.20 PAF paper form      94.1/ 91.8   22/11    91731/93922    44.8/ 32.8
code    0.20 PAF library form    86.6/ 86.6   25/12     7427/7487     16.5/ 17.5
code    0.20 midpoint            91.3/ 89.6    6/5     93416/94487    31.2/ 26.8
code    0.20 mean candidates per confidence map: 28.87

At the two lower noise levels each person yields one candidate per part with very few extras (2.00 and 2.03 candidates per map for two people), so the table measures grouping alone. The paper-form line integral then connects at least 99.5% of the limbs correctly under either region, which is the property the method relies on: a field that points along each true limb separates the two people even where their arms cross. Greedy matching makes more cross-person connections than the Hungarian algorithm, 15 to 28 against 2 or 3, and the share of people recovered exactly falls accordingly, since one wrong limb merges two skeletons; the limb recalls of the two matchers differ by at most 0.4 points. The library form depends on the region. With the paper’s region it keeps only 57% to 65% of the limbs and recovers almost no person whole. As reasoning: its first sample sits on the candidate itself, a candidate that lands a fraction of a cell behind its joint falls outside the paper’s segment condition, which leaves the cells behind a joint empty, and at these limb lengths the 95% rule tolerates no failed sample. With the training code’s widened region it keeps about 96% of the limbs, still below the paper form; our reading is that a field one cell wide, sampled at rounded positions, occasionally misses a sample, and one miss is enough. A trained network produces smoother fields than these hard-edged targets, which this example does not imitate.

The midpoint baseline does as well as the PAF here, at 99.9%, unlike the paper’s MPII ablation, in which PAFs beat one midpoint by 2.9 points (Cao et al., 2017, Fig. 7a). Our reading is that in rendered maps the midpoint of a wrong pair seldom lands on the true midpoint of the same limb type of the other person, so the example does not reproduce the failure of Fig. 5b, which concerns learned maps in real crowds. At $s = 0.20$ the smoothed noise produces about 29 peaks per map, and the picture changes. The paper form and the midpoint baseline accept any positive score, so they connect tens of thousands of pairs involving false candidates and recover under half of the people; the library form’s thresholds reject most of those pairs, about 7,500 instead of about 92,000, but they also reject true limbs, and its exact-person rate stays low. Greedy matching now beats the Hungarian algorithm on limb recall for the paper form and the midpoint baseline; as reasoning, maximizing the sum of scores can trade one strong correct connection for two weaker ones that involve false candidates, which the greedy order does not do. The first version of the script rendered only the paper’s region; after the library form rejected more than a third of the true limbs, a diagnostic run (not shown) looked at which samples failed, and the training code’s region was then added as a second, sourced choice, with the paper-region rows unchanged. The scoring was also vectorized and the greedy loop stopped after the positive pairs, which changed the runtime but no printed number, and the greedy and Hungarian results were merged into one row per method.

Performance

MPII multi-person. Cao et al. (2017, Table 1) report mean average precision over body parts, with joints matched under the PCKh threshold, on MPII’s multi-person benchmark. On the 288-image test subset used by DeepCut they reached 79.7% mAP against 71.2% for DeeperCut. On the full test set they reached 72.5% at one scale and 75.6% with a search over three scales (×0.7, ×1 and ×1.3), against 59.5% for DeeperCut. The table also lists 0.005 s per image for their method against 230 s (subset) and 485 s (full set) for DeeperCut; the text calls this the inference and optimization time, but since the CNN alone took about 100 ms in the runtime analysis, the column can only cover the parse (our reading). The dataset and its multi-person protocol are described in MPII Human Pose.

Graph relaxations. On 343 MPII training images held out for validation, a network trained on all 91 edges between the 14 parts scored 78.3% mAP with the full graph solved by integer linear programming (362 s per image), 77.6% with only the tree edges (43 s) and 77.4% with the greedy tree parse (0.005 s); a network trained on only the 13 tree edges reached 79.1% with the greedy parse (Table 2). The authors attribute the gain of the last model to the smaller number of association channels, which eases training. In Fig. 7a, parsing with ground-truth keypoint positions gave 88.3% mAP and the network’s keypoints with ground-truth connections 81.6%; the text puts PAF parsing at 79.4%, while Table 2 lists 79.1% for the tree-only model on the same validation set, and the paper does not reconcile the two. Accuracy rose with every stage from one to six (Fig. 7b).

COCO keypoints. On the COCO 2016 keypoint challenge, scored by average precision over object keypoint similarity thresholds, the method reached 60.5 AP on test-challenge, against 59.8 for the top-down G-RMI entry, and 61.8 AP on test-dev (Table 3). Its AP for medium-sized people, 55.1 on test-challenge, was below G-RMI’s 56.7, which the authors attribute to having to handle the whole range of scales in one pass, where top-down methods rescale each person. HigherHRNet’s comparison table later marks the 61.8 as obtained with refinement (Cheng et al., 2020, Table 1); the CVPR paper’s Table 3 does not say whether its challenge entry used the refinement of Table 4. On 1,160 randomly chosen COCO validation images (Table 4), the six-stage network scored 58.4 AP, and refining its people with a single-person convolutional pose machine raised this to 61.0; a top-down convolutional pose machine reached 62.7 AP on ground-truth boxes and 52.7 on boxes from an SSD detector, a 10-point drop that the authors read as dependence on the detector. Most false positives came from imprecise localization rather than confusion with the background (Fig. 8).

Journal version. The journal paper’s own tables could not be read. Hidalgo et al. (2019, Table 2) list, for the body-foot OpenPose model of Cao et al. (cited as the journal paper’s arXiv preprint) with multi-scale testing, 65.3 AP on body keypoints and 77.9 AP on foot keypoints on the COCO validation set, against 66.4 and 76.8 for their own deeper body-foot network. The OpenPose release notes describe the BODY_25 model of version 1.4.0 (September 2018) as about 3% more accurate and 30% faster than the original COCO model, while the repository’s list of major features announces the combined body-foot model in June 2018 as 40% faster and 5% more accurate, without saying against which model; the repository does not reconcile the two.

Crowds. On the CrowdPose test set, HigherHRNet’s comparison table lists OpenPose at 62.7, 48.7 and 32.3 AP on the easy, medium and hard subsets (Cheng et al., 2020, Table 7). The COCO metrics and leaderboards are described in COCO.

Strengths

• Runtime nearly independent of the number of people. One network pass serves the whole image, and the parse is cheap: two orders of magnitude below the CNN’s time in the paper’s measurements.
• No person detector. People that a detector would miss or merge are still parsed, and there is no early commitment to boxes.
• Association learned from images. PAFs encode position and orientation over the full support of each limb, and the greedy tree parse came close to the full-graph integer program on MPII at a small fraction of its cost.
• A released system. Code and trained models were public from the CVPR paper on, and the library grew to detect feet, hands and faces as well.

Limitations

• Small people. All people share one input scale and one map resolution, so small people get few cells; the paper’s COCO results are weaker for medium people than those of a top-down method. HigherHRNet later targeted this case with higher-resolution maps.
• Grouping errors in contact. The paper’s failure cases include parts shared by two people, wrong connections between parts of two people, and false positives on statues and animals (Fig. 9).
• Greedy, local decisions. Each limb type is matched independently, and the library accepts pairs greedily, so an early wrong connection cannot be revised by later evidence from other limbs.
• Hand and face cost in the library. The body and foot model runs once per image, but the hand and face detectors run once per detected person, so their runtime grows with the number of people (README).
• Licence. The library is restricted to noncommercial research use, which limits its deployment in products (see Implementations).

Historical Importance

OpenPose, introduced in 2017, made bottom-up multi-person pose estimation fast enough for video. Earlier bottom-up methods solved a global partitioning problem that took minutes to hours per image, and top-down methods scaled with the number of people. By learning an association field that a greedy parse could read, Cao et al. replaced that global inference with a few bipartite matchings, won the first COCO keypoint challenge, and released a real-time system. Kreiss et al. (2019) call it a significant cornerstone of pose estimation in the deep-learning era. Hidalgo et al. (2019) list re-targeting, 3D reconstruction, re-identification, tracking and action recognition among the uses of such keypoints, and skeleton-based action recognition takes 2D skeletons as its input. PifPaf kept OpenPose’s structure of detection maps plus a learned association field and changed the representation of both.

Variants

• Journal version and BODY_25 (2018–2019). The arXiv version of December 2018, published online by IEEE TPAMI in 2019 (volume 43, issue 1, 2021), refined PAFs before confidence maps, added the six foot keypoints and their dataset, and described the library; the BODY_25 model outputs the 17 COCO keypoints plus neck, mid-hip and six foot keypoints (release notes, version 1.4.0).
• Hands and face. The library’s hand detector (2 × 21 keypoints) and face detector (70 keypoints) run on face and hand boxes approximated from the body keypoints (Hidalgo et al., 2019, §2); the hand detector comes from Simon, Joo, Matthews and Sheikh’s multiview bootstrapping work (CVPR 2017), and the face detector was trained by the same procedure (README).
• Single-network whole-body estimation (Hidalgo et al., 2019). One network predicts body, foot, hand and face keypoints with their PAFs, joined through shared keypoints (ankles for feet, wrists for hands, eyes for the face), at a $480 \times 480$ input with more convolutional blocks per PAF stage and fewer stages, and runs approximately $n$ times faster than the multi-network pipeline on an image with $n$ people; its models are released in the training repository as BODY_135.
• PifPaf (Kreiss, Bertoni and Alahi, 2019). A bottom-up method with a part intensity field to localize parts and a part association field that encodes finer information about each connection, trained with a Laplace loss, aimed at low resolution and crowded scenes; it reported 66.7 AP on COCO 2017 test-dev against 61.8 for OpenPose, the test-dev figure of the CVPR paper.
• Other grouping signals. Associative embedding groups keypoints by learned tags instead of fields, and HigherHRNet combines it with high-resolution maps; both are compared in human pose estimation. The stacked hourglass network and HRNet are the backbones of many such methods, and convolutional pose machines (Wei et al., 2016) are the single-person architecture OpenPose’s stages descend from.

Implementations

The official library is CMU-Perceptual-Computing-Lab/openpose, written in C++ on a Caffe backend. As of November 2026, the newest tagged release is version 1.7.0 (November 2020) and the most recent commit on the default branch is from August 2024; the README names Ginés Hidalgo and Yaadhav Raaj as maintainers. The README lists, as main functionality, real-time 2D multi-person detection of 15, 18 or 25 body and foot keypoints with runtime invariant to the number of people, 2 × 21 hand keypoints and 70 face keypoints with runtime that depends on the number of people, single-person 3D keypoint reconstruction by triangulation from several synchronized cameras, a camera calibration toolbox, and single-person tracking. It runs on NVIDIA GPUs (CUDA), AMD GPUs (OpenCL) and CPUs, on Ubuntu, Windows and macOS, with a command-line demo and C++ and Python APIs, and a separate repository provides a Unity plugin. The README calls the system the first real-time multi-person system to detect body, hand, face and foot keypoints jointly, 135 keypoints in total; its per-part counts (25, 2 × 21 and 70) add to 137, and it does not explain the difference.

The licence, as of November 2026, is CMU’s “Academic or Non-Profit Organization Noncommercial Research Use Only” agreement: a personal, non-exclusive, non-transferable licence for noncommercial internal research, under which derivatives and modifications are owned by Carnegie Mellon University, redistribution and sale are not permitted, and the names “OpenPose” and “Carnegie Mellon” may not be used without permission. The README points to a commercial licence through CMU’s technology-transfer listing. Bundled Caffe code keeps its own BSD-style licence (LICENSE, third-party notices). The README’s summary says that OpenPose is free for non-commercial use and “may be redistributed under these conditions”, whereas the agreement grants no right to distribute. The training code, openpose_train, released in September 2019 and marked experimental, carries the same noncommercial licence; the foot keypoint annotations are licensed separately under Creative Commons Attribution 4.0 (Human Foot Keypoint Dataset page).