Keypoint Detection

Keypoint detection locates a fixed set of named points on an object, such as the eyes, paws and tail root of an animal, the fingertips and knuckles of a hand, or the headlights and wheel centres of a car, and does so for every object of the category in an image. The points are semantic: “left front paw” means the same anatomical place on a cat and on a bison, so each detected point carries a label, and the set of points describes the object’s pose, shape or viewpoint. The task is also called semantic keypoint localization, landmark detection or part localization, and for articulated categories animal pose estimation or hand pose estimation; recent work on detecting keypoints of categories never seen in training calls it category-agnostic pose estimation (CAPE). Whole-body human pose, with its own benchmarks and grouping methods, is the subject of human pose estimation; how a network turns a keypoint into a heatmap and back is the subject of heatmap regression. This article covers what changes when the object is an animal, a hand, a face, a vehicle or a piece of furniture, or any category at all, and how the results are measured.

Semantic keypoints are a different subject from the interest points of feature detection. A corner detector such as Harris finds points that are distinctive and repeatable in the image signal, with no name attached and no fixed number; a keypoint detector must find “the left headlight” whether or not it is a corner, including when it is hidden. CarFusion uses both side by side: 14 semantic car keypoints, which can be matched across cameras but are not precise enough for triangulation, and tracked Harris corners, which are precise within one camera but cannot be associated across views (Reddy, Vo and Narasimhan, 2018, and project page).

Problem Definition

A keypoint definition fixes, for a category $c$, a list of $K_c$ named points and usually a skeleton of edges between them for drawing. Given an image, the task is to find each object of the category and predict, for each keypoint $k$, its image position $\mathbf{x}_k \in \mathbb{R}^2$, a confidence, and often whether it is visible. Labels follow COCO’s annotation format in most recent datasets: each keypoint is stored as $(x, y, v)$ with a visibility flag, so the evaluation tools written for human pose can be reused (Yu et al., 2021).

Three settings are common:

• Category-specific, single instance. The object is given as a crop or a box, and one network predicts $K_c$ keypoints for one category. Most animal, hand, face and vehicle benchmarks are evaluated this way, on a crop around each object (Yu et al., 2021; Ng et al., 2022).
• Category-specific, detection setting. The object must also be found. Tulsiani and Malik (2015) evaluated keypoints for 12 rigid PASCAL VOC categories both ways: with known boxes, and on top of an object detector whose detections are each given keypoints.
• Category-agnostic. At test time the model receives one or a few support images of a new category with their keypoints marked, and must find the same keypoints on query images; the category and its keypoint definition were not seen in training (Xu et al., 2022).

Keypoint sets are a design decision of each dataset, and they are not compatible. The Animal-Pose dataset labels up to 20 points per animal, among them throat, withers and four knees (Cao et al., 2019); AP-10K labels 17, chosen to resemble COCO’s human keypoints (Yu et al., 2021); Animal Kingdom labels 23, including four mouth points and three tail points, with limb points defined for wings and fins as well (Ng et al., 2022). Face datasets use 19, 29, 68 or 98 landmarks (Wu et al., 2018). Left and right also need a convention: for vehicles and animals they refer to the object’s own sides, as for people.

Input and Output

The input is an RGB image or video frame, usually cropped around the object and resized; AP-10K’s benchmark models use $256 \times 256$ crops (Yu et al., 2021), as do the CAPE models trained on MP-100 (Xu et al., 2022). Some benchmarks provide several calibrated views, which are used to make labels rather than as test input (Simon et al., 2017; Zimmermann et al., 2019).

The output for each object is a $K_c \times 3$ array of positions and scores. For hands, the output is often 3D: InterHand2.6M and FreiHAND annotate 3D joint positions, and their metrics measure errors in millimetres or in normalized 3D units (Moon et al., 2020; Zimmermann et al., 2019). Category-agnostic models output as many points as the support image has, between 8 and 68 on MP-100 (Xu et al., 2022).

Variants

• Animals. Single-species datasets for horses, zebras, macaques, flies and tigers came first; multi-species sets (Animal-Pose with five species, AP-10K with 54, Animal Kingdom with 850) test whether one model covers many (Yu et al., 2021; Ng et al., 2022). Laboratory tools such as DeepLabCut and SLEAP train a keypoint model for each experiment from relatively few labelled frames (DeepLabCut and SLEAP repositories).
• Hands. 21 keypoints per hand (the wrist and four points on each finger) are the usual definition (Simon et al., 2017; Moon et al., 2020), in 2D or 3D, for one hand or two interacting hands.
• Faces. Facial landmark detection, also called face alignment, predicts dense outlines of the eyes, brows, nose, mouth and jaw; it has its own datasets, metrics and methods, and is outlined below only as far as it shows the task’s common structure.
• Vehicles and rigid objects. Cars (CarFusion, ApolloCar3D), the 12 rigid PASCAL VOC categories (keypoints of Bourdev et al., 2010, as used by Tulsiani and Malik, 2015) and furniture (Keypoint-5). For rigid objects, keypoints are a step towards viewpoint and 3D shape.
• Category-agnostic keypoints. StarMap (Zhou et al., 2018) for rigid objects, then the few-shot CAPE task of Xu et al. (2022) on MP-100, and prompt-based models that also accept text (Yang et al., 2024).
• Video. APT-36K annotates animals in 2,400 clips of 15 frames each, 36,000 frames and 53,006 instances in all, with track identities (Yang et al., 2022).

Challenges

• Definitions that do not transfer. A model trained with one keypoint list cannot predict another, even for the same category: Xu et al. (2022) give the 19-point and 68-point face definitions as an example. Cross-dataset comparisons are restricted to shared keypoints; AP-10K’s authors compared with Animal-Pose on the 17 keypoints the two datasets share.
• Annotation needs expertise and consistency. AP-10K’s authors note that labelling many species requires biological knowledge to tell keypoints apart, and used 13 trained annotators with three rounds of cross-checking (Yu et al., 2021). On ApolloCar3D, a newly trained group of annotators who re-labelled the validation set outscored the algorithms by about 10% on its 3D metric, yet their own results were still unsatisfying, mainly because they could not accurately remember the meaning of all 66 car keypoints (Song et al., 2019). Hand joints are hidden so often that manual labels for occluded joints are, in Simon et al.’s words, “at best an educated guess”.
• Generalization across species and instances. In AP-10K’s domain-generalization track, an HRNet-W32 trained on the bovid family scored 0.641 mAP on deer and moose (the cervid family) but 0.015 on great apes, while one trained on monkeys reached 0.446 on the apes (Yu et al., 2021, Table 9). Horse-10 holds out whole horses: models are trained on 10 horses and tested on 20 others (Mathis et al., 2021).
• Varying structure. Categories do not have a fixed number of parts: chairs have different numbers of legs, and a channel-per-keypoint output cannot represent this (Zhou et al., 2018).
• Symmetry and self-similarity. Mirror-symmetric points (left and right eyes, left and right paws) look alike; in matching-based category-agnostic models both can respond to one support keypoint, giving duplicate or swapped predictions (Shi et al., 2023).
• Occlusion and truncation. POMNet’s authors attribute part of the low cross-category scores on vehicles to their many invisible keypoints (Xu et al., 2022), and Tulsiani and Malik (2015) found accuracy on occluded objects about 13 PCK points below the default.

Approaches

Human pose models applied to other categories

Most category-specific keypoint detectors are human pose networks with a different number of output channels. AP-10K’s authors state that human pose models apply to animals with no difference in modelling except the number of keypoints, and benchmarked SimpleBaseline (ResNet-50 and -101), the stacked hourglass network and HRNet on it (Yu et al., 2021). Simon et al. (2017) built their hand detector on convolutional pose machines, with VGG-19 features up to conv4_4 and confidence maps at stride 8; ApolloCar3D extended convolutional pose machines to car keypoints, training it on about 40,000 labelled vehicles (Song et al., 2019); CarFusion detected 14 car keypoints with a stacked hourglass network trained on KITTI (Reddy, Vo and Narasimhan, 2018); Animal Kingdom’s benchmark uses HRNet and HRNet with DARK decoding (Ng et al., 2022). The output is a heatmap per keypoint, decoded as described in heatmap regression, and multi-animal scenes are handled top-down, with a detector or ground-truth boxes, or bottom-up, with grouping, as for people.

What differs is the data. Human pose benchmarks contain many thousands of individuals, while users of animal pose tools often label only a few hundred frames of the body parts they define (Mathis et al., 2021). DeepLabCut’s README describes the toolbox as working for any animal or object that can be labelled, and attributes its need for little training data to transfer learning; its original feature detectors (ResNets with readout layers) came from the DeeperCut human pose model; its later versions add multi-animal tracking and pretrained “SuperAnimal” models for quadrupeds and top-view mice (DeepLabCut README, as of November 2026). SLEAP describes itself as a framework for multi-animal pose tracking with top-down and bottom-up training and a labelling interface for active learning (SLEAP README, as of November 2026).

How much pretraining matters has been measured. On AP-10K, HRNet-W32 trained from scratch for 210 epochs scored 0.703 mAP on the validation set and 0.738 with ImageNet pretraining; at 630 epochs from scratch the gap shrank to 0.725 against 0.735, and initializing from a COCO human pose model and fine-tuning for 210 epochs gave 0.753 (Tables 3–5). Mathis et al. (2021) found that, on Horse-10, networks that perform better on ImageNet also generalize better to unseen horses when pretrained on ImageNet, and that pretraining helps most out of domain.

Labels from geometry and multiple views

Several hand and vehicle datasets are labelled by reconstructing the object in 3D and projecting it back, so that a few confident detections or clicks label every view.

• Multiview bootstrapping. Simon et al. (2017) ran an initial hand detector on all 31 HD views of the CMU Panoptic Studio, triangulated the confident detections robustly, discarded outliers, and reprojected the 3D points into the views where the hand was hard to see, producing new training labels; repeating this improved the detector. The released hand database holds 1,912 training and 846 test hands with manual labels, 14,261 synthetic hands and 14,817 hands from Panoptic Studio labelled by bootstrapping (CMU hand database page). The paper describes the synthetic set as “around 11000” examples.
• CarFusion applied the same idea to cars filmed at an intersection by uncalibrated, unsynchronized handheld phones and action cameras: car keypoints reconstructed in 3D were reprojected to fine-tune the detector, which raised PCK on manually labelled frames from 87.1 to 91.4 at $\alpha = 0.1$ (Reddy, Vo and Narasimhan, 2018, Table 2; the text refers to it as Table 1).
• InterHand2.6M used a studio of 80 to 140 cameras. Annotators labelled 94,914 images from 9,036 time instants; triangulating and reprojecting these labels gave 698,922 images with 2D keypoints, which trained a machine annotator whose triangulated outputs, filtered with RANSAC, had an error of 2.78 mm on held-out data (Moon et al., 2020).
• CAD alignment. PASCAL3D+ annotators chose the closest CAD model for each object and clicked the 2D positions of the model’s landmarks, mostly corners, and the object’s viewpoint was solved from the 2D–3D correspondences; each landmark also received a visibility state (visible, self-occluded, occluded by another object, truncated, or unknown) (Xiang, Mottaghi and Savarese, 2014). ApolloCar3D defined 66 keypoints on its car models and recovered each car’s pose with EPnP and RANSAC, a Perspective-n-Point solution, from the clicked keypoints (Song et al., 2019).

Structure and viewpoint priors for rigid objects

For a rigid object, all keypoints move together with the viewpoint. Tulsiani and Malik (2015) predicted keypoints with a multiscale convolutional heatmap model trained on the PASCAL VOC keypoints of Bourdev et al., then multiplied in a likelihood conditioned on the predicted viewpoint. On PASCAL VOC validation with known boxes, mean PCK at $\alpha = 0.1$ rose from 61.5 for the two-scale appearance model to 68.8 with the viewpoint term, against 48.5 for Long et al.’s earlier convnet features (Table 3); in the detection setting the mean APK was 33.2 (Table 4). Their error analysis found that left–right confusions were not a major source of error (72.0 when a laterally inverted keypoint also counted, against 68.8), while loosening the threshold to $\alpha = 0.2$ raised PCK to 81.7, so imprecise localization was the main one (Table 7).

Category-agnostic keypoints

StarMap (Zhou et al., 2018) dropped the one-channel-per-keypoint output altogether. A single multi-peak heatmap marks every keypoint of every category; at each peak, three further channels give the keypoint’s 3D position in a canonical object frame (CanViewFeature), which serves as its identity, and a depth channel lifts it to 3D for viewpoint estimation. Because identities come from canonical coordinates, a chair with any number of legs can be represented. On the 2,113 non-occluded, non-truncated PASCAL3D+ validation objects, StarMap reached 78.6 mean PCK at $\alpha = 0.1$ when keypoint identities were inferred from the canonical features, slightly below Pavlakos et al.’s category-specific hourglass model, which the authors evaluated themselves (82.5, a mean that leaves out tables and motorbikes), and 92.2 against 89.0 when identities were assigned by an oracle (Table 1).

Few-shot CAPE. Xu et al. (2022) defined CAPE as $K$-shot pose estimation: train on many categories, then, given $K$ support images of an unseen category with keypoints marked, localize those keypoints in a query image. Their POse Matching Network (POMNet) extracts a feature for each support keypoint by pooling a ResNet feature map under the keypoint’s Gaussian heatmap, refines the keypoint features with a transformer module that applies self-attention among keypoints and cross-attention to the query image, and predicts each keypoint’s heatmap on the query as a matching score. Keypoints are padded to a fixed length of 100 with masked dummy features, so any keypoint count fits. On MP-100 (below), POMNet reached 79.70 PCK with one support image and 80.71 with five, against 63.58 and 64.61 for fine-tuning a 68-channel network on the support images (Table 1). Table 1 gives ProtoNet’s five-shot score as 58.56, while the text quotes 58.46.

CapeFormer (Shi et al., 2023) argued that matching alone is unreliable, because similar-looking keypoints such as mirror-image eyes both respond to one support keypoint. It treats the matching peaks as proposals and refines them in a second stage, a transformer decoder whose queries are the proposals and which regresses offsets by cross-attention to image features. With the same ResNet-50 backbone, it scored 85.31 (one-shot) and 89.30 (five-shot) PCK on MP-100 (Table 1). GraphCape (Hirschorn and Avidan, 2024) adds the category’s skeleton as a graph over the support keypoints, processed by graph convolutions in the decoder, and contributes skeleton annotations for MP-100. Against CapeFormer-T, the authors’ version of CapeFormer with a SwinV2-T backbone, it improved average PCK by 0.98 points one-shot (87.23 against 86.25) and 0.26 five-shot (91.16 against 90.90); the table caption says it wins on every split, but on five-shot split 4 CapeFormer-T scores 90.63 against 90.37 (Table 2).

Prompted detection of any keypoints. X-Pose (Yang et al., 2024) detects keypoints of several objects in a whole image, prompted either by a support image with keypoints or by text naming the object and its keypoints, and is trained on UniKPT, which merges 13 datasets into 338 keypoint types across 1,237 categories and more than 400,000 instances.

Datasets and Benchmarks

Dataset	Objects	Size	Keypoints	Protocol and metric
Animal-Pose (Cao et al., 2019)	dog, cat, horse, sheep, cow	5,517 instances in over 3,000 images	up to 20	COCO-API mAP
Horse-10 (Mathis et al., 2021)	30 thoroughbred horses	8,114 frames	22	train on 10 horses, test on unseen ones; PCK at 0.3 of nose-to-eye distance
AP-10K (Yu et al., 2021)	54 species, 23 families	10,015 images, 13,028 instances	17	7:1:2 split per species; OKS-based mAP
Animal Kingdom (Ng et al., 2022)	850 species, 5 classes for pose	33,099 frames	23	three protocols; PCK at 0.05 of box side
APT-36K (Yang et al., 2022)	30 species, 15 families	36,000 frames, 2,400 clips	17	OKS-based AP; tracking
CMU Panoptic hands (Simon et al., 2017)	hands	1,912 / 846 manual, 14,817 bootstrapped	21	PCK, normalized by 0.7 × head size
FreiHAND (Zimmermann et al., 2019)	hands, 32 subjects	33k / 4k frames	21 (3D)	AUC of 3D PCK
InterHand2.6M (Moon et al., 2020)	single and interacting hands	2.6M frames	42 (3D)	MPJPE in mm
PASCAL VOC keypoints, PASCAL3D+	12 rigid categories	over 3,000 instances per category (PASCAL3D+)	per category	PCK and APK at 0.1 of box side
CarFusion (Reddy et al., 2018)	cars, buses, trucks	100,000 cars in 53,000 images	14	PCK at 0.1 and 0.2 of box side
ApolloCar3D (Song et al., 2019)	cars	5,277 images, over 60K cars	66	3D pose and shape metric
Keypoint-5 (Wu et al., 2016)	5 furniture categories	1,000–2,000 images each	per category	PCP and average error
MP-100 (Xu et al., 2022)	100 categories, 8 super-categories	over 18K images, 20K instances	8–68	five splits, unseen test categories; PCK at 0.2 of box side

Animals. AP-10K collected about 60,000 images of mammals from existing classification and detection datasets, organized them by family and species, and labelled 10,015 of them; the remaining 49,643 keep only their family and species labels, for semi- and self-supervised work (Yu et al., 2021). Its three tracks test supervised learning, transfer from human pose, and generalization within and across families; the paper’s Table 2, abstract and introduction give 54 species, while its cross-dataset section says 60. The dataset is released under CC BY 4.0 (repository, as of November 2026). Animal Kingdom defines three pose protocols: all species split 80/20, a leave-$k$-out split that holds out 12 sub-classes such as felines and turtles, and separate splits for mammals, amphibians, reptiles, birds and fishes. HRNet scored 66.06 PCK@0.05 on the first and 39.30 on the leave-out protocol, with HRNet-DARK slightly higher (66.57 and 40.28); reptiles and amphibians were hardest among the classes (Ng et al., 2022, Table 5). The Animal-Pose paper counts 5,517 instances in more than 3,000 images, while AP-10K’s comparison table lists it with 4,666 images and 6,117 instances. Birds have their own long-standing benchmark in CUB-200-2011, which labels up to 15 part locations on each image (as described by Wu et al., 2016).

Hands. The CMU hand data combine manual labels on MPII and New Zealand Sign Language images with the bootstrapped Panoptic Studio labels (Simon et al., 2017). FreiHAND fits the MANO hand model to multi-view recordings from eight cameras; its training images were taken against a green screen, so backgrounds can be replaced, and its evaluation images indoors and outdoors. In the authors’ cross-dataset test of a 3D keypoint network trained on one dataset and evaluated on others, training on FreiHAND gave the best average rank (2.2) of eight training sets (Zimmermann et al., 2019, Table 1). InterHand2.6M recorded 36 sessions; its Table 1 lists 27 subjects, while the text counts 26. Images were captured at $4096 \times 2668$ and released at $512 \times 334$ to protect fingerprint privacy (Moon et al., 2020).

Faces. Face landmark benchmarks follow the same pattern with denser point sets: 300-W uses 68 points, COFW 29, AFLW 19 and WFLW 98, the last on 10,000 faces (7,500 for training, 2,500 for testing) with attribute labels for pose, expression, occlusion, make-up, illumination and blur (Wu et al., 2018).

Vehicles and objects. CarFusion’s released data cover sedans, SUVs, buses and trucks seen from 18 moving cameras at intersections in Pittsburgh (project page, as of November 2026); the paper itself evaluates on one intersection filmed by 22 phones and action cameras (its introduction says 21). ApolloCar3D fits one of its industry-grade CAD models with absolute scale to every car in its driving images, with 11.7 cars per image on average; its Table 1 lists 79 car models, while the text counts 34 (Song et al., 2019). Keypoint-5 covers beds, chairs, sofas, swivel chairs and tables; each image was labelled by three Mechanical Turk workers and the median taken (Wu et al., 2016). The paper’s results table reports four of the five categories, without tables.

MP-100. Xu et al. (2022) assembled MP-100 from existing datasets, including COCO, 300W, AFLW, OneHand10K, DeepFashion2, AP-10K, MacaquePose, Vinegar Fly, Desert Locust, CUB-200, CarFusion, AnimalWeb and Keypoint-5, covering human hands, faces and bodies, animal faces and bodies, clothes, furniture and vehicles. Each of five splits divides the 100 categories into 70 for training, 10 for validation and 20 for testing, with no overlap, and the test sets of the five splits do not overlap either, so every category is tested once as unseen. Test sets have 100 instances per category, and evaluation samples 3,000 support–query episodes per test category. A harder leave-one-super-category-out protocol trains on all but one super-category: there POMNet scored 34.92 PCK on vehicles and 47.27 on furniture (Table 2), and CapeFormer, one-shot, 45.40 and 52.49 (Shi et al., 2023, Table 2).

Evaluation Metrics

PCK. The percentage (or probability) of correct keypoints counts a predicted keypoint as correct when it lies within $\alpha \cdot d$ of the ground truth, where $d$ is a size of the object:

$$\mathrm{PCK}@\alpha = \frac{1}{N} \sum_{i=1}^{N} \mathbb{1}\!\left( \lVert \hat{\mathbf{x}}_i - \mathbf{x}_i \rVert_2 \le \alpha\, d \right).$$

Yang and Ramanan (2013) took $d = \max(h, w)$ of the box around the object, and Tulsiani and Malik (2015) carried this to rigid categories; StarMap, CarFusion, Animal Kingdom and MP-100 use the same normalizer (Zhou et al., 2018; Reddy, Vo and Narasimhan, 2018; Ng et al., 2022; Xu et al., 2022). The threshold, however, differs by benchmark: 0.1 on PASCAL keypoints, 0.05 on Animal Kingdom, 0.2 on MP-100, so numbers are comparable only within one protocol. Other benchmarks normalize by a body part, so that the threshold does not change with articulation or truncation: Simon et al. (2017) use 0.7 times the person’s head size, about the length of an outstretched hand; Horse-10 uses 0.3 times the horse’s nose-to-eye distance (Mathis et al., 2021); MPII’s PCKh, described in human pose estimation, uses the head. FreiHAND reports the area under the PCK curve over a range of 3D thresholds (Zimmermann et al., 2019), and face landmark benchmarks report the mean error divided by the outer-eye-corner distance, or by the distance between pupil centres, which gives different numbers for the same predictions (Wu et al., 2018). Names also drift: Keypoint-5 and CUB-200-2011 results are reported as “percentage of correct parts” (PCP), a keypoint counted as correct within 1.5 standard deviations of the annotators’ spread, which is not the limb-based PCP of human pose (Wu et al., 2016).

APK. For the detection setting, Yang and Ramanan’s APK treats each keypoint hypothesis with a score as a detection, counts it correct within $\alpha \max(h, w)$ of a ground-truth keypoint, and reports the area under the precision–recall curve (Tulsiani and Malik, 2015).

OKS and keypoint AP. COCO’s object keypoint similarity scores a whole object at once, playing the role of IoU in box detection:

$$\mathrm{OKS} = \frac{\sum_i \exp\!\left( -d_i^2 / 2 s^2 \kappa_i^2 \right) \delta(v_i > 0)}{\sum_i \delta(v_i > 0)}, \qquad \kappa_i = 2\sigma_i,$$

with $s^2$ the object’s segment area and $\sigma_i^2 = E[d_i^2 / s^2]$ the spread of human annotations of keypoint $i$, which COCO’s organizers measured on 5,000 doubly annotated images (COCO keypoint evaluation page). Keypoint AP then averages average precision over OKS thresholds from 0.50 to 0.95; the person-specific details, such as area ranges and ignored crowds, are in human pose estimation and COCO. AP-10K and APT-36K adopt this metric for animals (Yu et al., 2021; Yang et al., 2022).

Two quantities in OKS have no settled value for new categories. The first is $\sigma_i$. Neither the AP-10K nor the Animal-Pose paper reports annotator spreads: the official AP-10K configuration states that the paper gave too little information and takes COCO’s human values instead, assigning the eye and nose values to the animal’s eyes and nose, the ear value (0.035) to the neck and tail root, the wrist and ankle values to the front and back paws, and so on (AP-10K repository, ap10k.py); MMPose’s Animal-Pose configuration does the same with its own assignments, plus a value of 0.10 that is not among COCO’s, and its MacaquePose configuration uses COCO’s values in COCO’s order (MMPose configs). APT-36K states that it uses the constants defined by AP-10K. The second is $s^2$: AP-10K’s paper does not say what the area field holds, and MMPose, when an annotation has no area, falls back to 0.53 times the box area (base_coco_style_dataset.py). The example below shows how much these choices move the score.

3D metrics. For 3D hands, InterHand2.6M reports the mean per-joint position error (MPJPE), the mean Euclidean distance in millimetres between predicted and true joints after aligning each hand’s root joint, separately for single and interacting hands, and the mean relative-root position error (MRRPE), the error in the left hand’s root position relative to the right hand’s (Moon et al., 2020).

Example: PCK and OKS on synthetic quadrupeds

The example generates side-view quadrupeds with AP-10K’s 17 keypoints, in two body plans, “stocky” (legs 0.35 and neck 0.15 of a body unit $L$) and “long-legged” (0.80 and 0.55), with random limb and neck angles. Each animal is drawn as a union of capsules on a binary image, so it has a segment area (its pixel count) and a tight box. No network is trained: “predictions” are the true keypoints plus Gaussian noise of a chosen size, so that only the metric varies. Four simulated predictors are compared: a fixed error of 2 pixels per axis whatever the animal’s size (P_px); an error of $0.03 L$ for every keypoint (P_rel); and two predictors with the same mean squared error as P_rel but distributed differently, one precise on the eyes and nose ($0.01 L$) and less precise elsewhere (P_face), the other the reverse ($0.06 L$ on the head, P_body). PCK uses the longer side of the box. OKS follows cocoeval.py’s computeOks with AP-10K’s σ values. Since each animal has exactly one prediction and nothing is missed, the script reports, in place of COCO AP, the share of animals whose OKS reaches each threshold from 0.50 to 0.95, averaged over the ten thresholds (“OKS recall”); in our reasoning, this is about what interpolated AP would be if the predictor’s scores ranked the animals by OKS. Two choices the sources leave open are printed both ways: σ (AP-10K’s values or a uniform σ equal to their mean, 0.067) and the area (the silhouette’s pixel count, as COCO’s segment area, or MMPose’s 0.53 × box area). Each table cell uses 500 animals from seed 0.

import numpy as np

# AP-10K keypoints in annotation order, and the sigmas of its official config
# (COCO person sigmas reassigned: eyes, nose, ears -> neck and tail root, ...).
NAMES = ["l_eye", "r_eye", "nose", "neck", "tail_root", "l_shoulder", "l_elbow",
         "l_front_paw", "r_shoulder", "r_elbow", "r_front_paw", "l_hip", "l_knee",
         "l_back_paw", "r_hip", "r_knee", "r_back_paw"]
SIG_AP10K = np.array([.025, .025, .026, .035, .035, .079, .072, .062, .079, .072,
                      .062, .107, .087, .089, .107, .087, .089])
SIG_UNIFORM = np.full(17, SIG_AP10K.mean())
HEAD = [0, 1, 2]                                   # eyes and nose
SHAPES = {"stocky": (0.35, 0.15), "long-legged": (0.80, 0.55)}   # (leg, neck) in L


def quadruped(rng, L, shape):
    """Side-view quadruped: 17 keypoints (px, image y down) and silhouette capsules."""
    leg, neck = SHAPES[shape]
    j = lambda s=0.25: rng.normal(0, s)
    P = {}
    y0 = leg + 0.12                                   # height of the back
    P["tail_root"] = np.array([0.0, y0])
    for side, dx, dy in (("l", 0.0, 0.0), ("r", 0.03, 0.02)):   # far side offset
        P[side + "_hip"] = np.array([0.08 + dx, y0 - 0.05 + dy])
        P[side + "_shoulder"] = np.array([0.62 + dx, y0 - 0.05 + dy])
        for top, mid, end in (("hip", "knee", "back_paw"), ("shoulder", "elbow", "front_paw")):
            a, b = j(), j()
            P[side + "_" + mid] = P[side + "_" + top] + 0.5 * leg * np.array([np.sin(a), -np.cos(a)])
            P[side + "_" + end] = P[side + "_" + mid] + 0.5 * leg * np.array([np.sin(b), -np.cos(b)])
    base = np.array([0.70, y0 + 0.02])
    t = np.deg2rad(60) + j(0.15)
    P["neck"] = base + 0.3 * neck * np.array([np.cos(t), np.sin(t)])
    head = base + neck * np.array([np.cos(t), np.sin(t)]) + np.array([0.04, 0.0])
    P["nose"] = head + np.array([0.11, -0.03])
    P["l_eye"], P["r_eye"] = head + np.array([0.03, 0.03]), head + np.array([0.05, 0.035])
    kp = np.array([P[n] for n in NAMES]) * L
    kp[:, 1] = -kp[:, 1]                              # image y points down
    hips, shs = (P["l_hip"] + P["r_hip"]) / 2, (P["l_shoulder"] + P["r_shoulder"]) / 2
    caps = [(hips, shs, 0.13), (base, head, 0.05), (head, head, 0.07)]
    for s in "lr":
        for a, b in (("hip", "knee"), ("knee", "back_paw"), ("shoulder", "elbow"), ("elbow", "front_paw")):
            caps.append((P[s + "_" + a], P[s + "_" + b], 0.035))
    caps = [(a * L * [1, -1], b * L * [1, -1], r * L) for a, b, r in caps]
    return kp, caps


def silhouette(caps):
    """Rasterize capsules; return pixel area and tight box (x0, y0, x1, y1)."""
    pts = np.array([p for a, b, r in caps for p in (a - r, a + r, b - r, b + r)])
    lo = np.floor(pts.min(0)) - 1
    hi = np.ceil(pts.max(0)) + 1
    m = np.zeros((int(hi[1] - lo[1]), int(hi[0] - lo[0])), bool)
    for a, b, r in caps:                       # each capsule only inside its own box
        x0, y0 = (np.floor(np.minimum(a, b) - r) - lo).astype(int)
        x1, y1 = (np.ceil(np.maximum(a, b) + r) - lo).astype(int) + 1
        yy, xx = np.mgrid[y0:y1, x0:x1] + lo[::-1, None, None] + 0.5   # pixel centres
        X, ab = np.stack([xx, yy], -1), b - a
        t = np.clip(((X - a) @ ab) / max(ab @ ab, 1e-12), 0, 1)
        m[y0:y1, x0:x1] |= np.linalg.norm(X - (a + t[..., None] * ab), axis=-1) <= r
    ys, xs = np.nonzero(m)
    return m.sum(), (xs.min(), ys.min(), xs.max() + 1, ys.max() + 1)


def oks(pred, gt, area, sig):
    """cocoeval.computeOks for one animal, all 17 keypoints labeled; also per-keypoint ks."""
    e = ((pred - gt) ** 2).sum(1) / (2 * sig) ** 2 / (area + np.spacing(1)) / 2
    ks = np.exp(-e)
    return ks.mean(), ks


def pck(pred, gt, box, alpha):
    """PCK with the box's longer side as normalizer (Yang & Ramanan; MP-100)."""
    side = max(box[2] - box[0], box[3] - box[1])
    return (np.linalg.norm(pred - gt, axis=1) <= alpha * side).mean()


THR = np.linspace(0.5, 0.95, 10)


def evaluate(rng, L, shapes, err, n=500, sig=SIG_AP10K, area_mode="mask"):
    """err(L) -> per-keypoint std (17,) in px. Returns PCK@.05/.1/.2, mean OKS, OKS recall."""
    P, O, KS, ratio = [], [], [], []
    for i in range(n):
        kp, caps = quadruped(rng, L, shapes[i % len(shapes)])
        area, box = silhouette(caps)
        w, h = box[2] - box[0], box[3] - box[1]
        a = area if area_mode == "mask" else 0.53 * w * h      # MMPose fallback
        pred = kp + err(L)[:, None] * rng.standard_normal((17, 2))
        P.append([pck(pred, kp, box, al) for al in (0.05, 0.1, 0.2)])
        o, ks = oks(pred, kp, a, sig)
        O.append(o); KS.append(ks); ratio.append(np.sqrt(a) / max(w, h))
    O = np.array(O)
    rec = np.mean([(O >= t).mean() for t in THR])
    return 100 * np.mean(P, 0), O.mean(), 100 * rec, np.mean(KS, 0), np.mean(ratio)


c_face = np.sqrt((17 * 0.03**2 - 3 * 0.01**2) / 14)
c_body = np.sqrt((17 * 0.03**2 - 3 * 0.06**2) / 14)
PRED = {
    "P_px  (2 px)":   lambda L: np.full(17, 2.0),
    "P_rel (0.03 L)": lambda L: np.full(17, 0.03 * L),
    "P_face":         lambda L: np.where(np.isin(np.arange(17), HEAD), 0.01, c_face) * L,
    "P_body":         lambda L: np.where(np.isin(np.arange(17), HEAD), 0.06, c_body) * L,
}

print("1. Scale (stocky animals, sigma = AP-10K config, OKS area = silhouette)")
print("predictor        L    PCK@.05 PCK@.1 PCK@.2  meanOKS  OKS-recall")
for name in ("P_px  (2 px)", "P_rel (0.03 L)"):
    for L in (40, 80, 160, 320):
        p, o, r, _, _ = evaluate(np.random.default_rng(0), L, ["stocky"], PRED[name])
        print(f"{name:15s} {L:4d}   {p[0]:6.1f} {p[1]:6.1f} {p[2]:6.1f}   {o:6.3f}   {r:6.1f}")

print("\n2. Shape (P_rel, L = 160, sigma = AP-10K config)")
print("shape        sqrt(area)/side  PCK@.05 PCK@.1  OKS-recall(mask)  OKS-recall(0.53*box)")
for shape in SHAPES:
    p, o, r, _, q = evaluate(np.random.default_rng(0), 160, [shape], PRED["P_rel (0.03 L)"])
    _, _, rb, _, qb = evaluate(np.random.default_rng(0), 160, [shape], PRED["P_rel (0.03 L)"],
                               area_mode="box")
    print(f"{shape:12s}  {q:5.2f} / {qb:5.2f}       {p[0]:6.1f} {p[1]:6.1f}   {r:8.1f}          {rb:8.1f}")

print("\n3. Sigma (L = 160, both shapes, OKS area = silhouette)")
print("predictor  PCK@.1   OKS-recall(AP-10K sigma)  OKS-recall(uniform sigma)  ks eyes/hips (AP-10K)")
for name in ("P_rel (0.03 L)", "P_face", "P_body"):
    p, _, ra, ks, _ = evaluate(np.random.default_rng(0), 160, list(SHAPES), PRED[name])
    _, _, ru, _, _ = evaluate(np.random.default_rng(0), 160, list(SHAPES), PRED[name],
                              sig=SIG_UNIFORM)
    print(f"{name:14s} {p[1]:5.1f}   {ra:10.1f}               {ru:10.1f}              "
          f"{ks[[0, 1]].mean():.2f} / {ks[[11, 14]].mean():.2f}")

1. Scale (stocky animals, sigma = AP-10K config, OKS area = silhouette)
predictor        L    PCK@.05 PCK@.1 PCK@.2  meanOKS  OKS-recall
P_px  (2 px)      40     33.1   81.4   99.9    0.596     24.3
P_px  (2 px)      80     82.3   99.9  100.0    0.823     69.8
P_px  (2 px)     160     99.9  100.0  100.0    0.941     93.2
P_px  (2 px)     320    100.0  100.0  100.0    0.984    100.0
P_rel (0.03 L)    40     69.6   99.0  100.0    0.770     59.0
P_rel (0.03 L)    80     70.5   99.2  100.0    0.775     59.8
P_rel (0.03 L)   160     70.6   99.2  100.0    0.775     59.8
P_rel (0.03 L)   320     70.3   99.1  100.0    0.774     59.6

2. Shape (P_rel, L = 160, sigma = AP-10K config)
shape        sqrt(area)/side  PCK@.05 PCK@.1  OKS-recall(mask)  OKS-recall(0.53*box)
stocky         0.57 /  0.61         70.6   99.2       59.8              64.1
long-legged    0.46 /  0.65         94.3  100.0       71.9              85.5

3. Sigma (L = 160, both shapes, OKS area = silhouette)
predictor  PCK@.1   OKS-recall(AP-10K sigma)  OKS-recall(uniform sigma)  ks eyes/hips (AP-10K)
P_rel (0.03 L)  99.6         66.0                     79.5              0.50 / 0.95
P_face          99.3         76.5                     79.8              0.90 / 0.94
P_body          96.8         67.6                     84.0              0.20 / 0.98

The script runs in about 15 seconds with NumPy 2.3.5. Its first run took nearly two minutes; the only change since then is that silhouette() rasterizes each capsule inside its own bounding box instead of the whole animal’s, which draws the same pixels, and the printed output was checked to be identical before and after. The first block shows that both metrics are normalized by the object’s size, and so are blind to size only when the errors scale with it. The predictor whose errors grow with the animal scores the same at every size, while a fixed 2-pixel error, the kind of floor a coarse heatmap grid imposes, costs a 40-pixel animal two thirds of its PCK@0.05 and most of its OKS recall and costs a 320-pixel animal nothing. The loose thresholds hide this: PCK@0.2 is near 100 everywhere.

The second block went against the plan’s hypothesis, which was that PCK would be “nearly the same for both shapes, while OKS (normalized by sqrt(area)) is lower for the long-legged shape, whose area is small relative to its box”. Instead both rose for the long-legged animals, because the errors were tied to the body unit $L$ while the long-legged animal’s box and area are both larger relative to $L$; the comparison does not isolate shape. What the block does show is that the two normalizers move apart: the long-legged animal’s $\sqrt{\text{area}}$ is a smaller fraction of its box side (0.46 against 0.57), and PCK@0.05 gained about twice as much as OKS recall. The area convention alone, with the predictions unchanged, moved the long-legged animals’ OKS recall by 13.6 points and the stocky ones’ by 4.3, because a thin-limbed silhouette fills less of its box than 0.53.

The third block holds the mean squared error fixed and changes only where the error falls. With AP-10K’s σ, a relative error that is the same for every keypoint leaves the eyes at a mean similarity of 0.50 and the hips at 0.95, since COCO’s annotators placed eyes about four times more consistently than hips, and these constants now judge an animal’s eyes. As a result the predictor that is precise on the head wins (76.5 against 67.6), while under a uniform σ the order reverses (84.0 against 79.8). PCK at 0.1, which applies one threshold to every keypoint, is close to its ceiling for all three. Which predictor is “better” is therefore partly a property of σ, which for animals has not been measured.

The example is a simulation of metrics, not of a detector. It leaves out misses, false detections, left–right swaps and occluded keypoints, all of which matter for real models.

Applications

• Behaviour science and conservation. Animal keypoints measure movement without markers. Animal Kingdom’s authors cite conservation, wildlife management and neuroscience among the uses (Ng et al., 2022), and DeepLabCut’s README lists applications to mice, rats, flies, fish, cheetahs, race horses and robots (as of November 2026). Skeleton sequences from animals or people feed skeleton-based action recognition, and APT-36K combines keypoints with object tracking across video frames (Yang et al., 2022).
• Hands in interaction. Hand keypoints support gesture recognition, robotics and augmented reality (Zimmermann et al., 2019) and human–computer interaction (Moon et al., 2020); triangulated over many views they give markerless 3D hand motion capture (Simon et al., 2017).
• Vehicles. ApolloCar3D uses car keypoints to recover the 3D pose and shape of each car for autonomous-driving perception (Song et al., 2019), and CarFusion reconstructs moving vehicles at intersections for traffic analysis (Reddy, Vo and Narasimhan, 2018).
• Viewpoint and 3D structure. Keypoints on rigid objects constrain camera pose relative to the object and its 3D shape: Tulsiani and Malik (2015) combine them with viewpoint, and StarMap and 3D-INN lift detected 2D keypoints to 3D (Zhou et al., 2018; Wu et al., 2016).

Open Problems

• Generalization beyond the training categories. AP-10K’s inter-family results and MP-100’s cross-super-category results both show large drops on categories far from those seen in training, with vehicles the hardest super-category for every CAPE method tested and furniture next for all but ProtoNet (Yu et al., 2021; Xu et al., 2022; Shi et al., 2023).
• Metrics for new categories. OKS was calibrated on human annotators; for animals the constants are borrowed, and PCK’s threshold and normalizer change from benchmark to benchmark. In our reading of the example, rankings between models can depend on these conventions.
• Unified keypoint definitions. Each dataset defines its own points; MP-100 and UniKPT merge datasets but keep their definitions, and text-prompted models such as X-Pose are one attempt to describe keypoints in words instead of indices (Yang et al., 2024).
• Occlusion and annotation quality. Hidden hand joints and car keypoints remain hard to label by hand, which is why the largest hand and car datasets are labelled through multi-view geometry or CAD fitting; human annotators also disagree when keypoint definitions are long (Song et al., 2019).