Object Keypoint Similarity

Object keypoint similarity (OKS) scores how well one predicted set of keypoints matches one annotated object, such as a person’s 17 COCO body joints, with a single number between 0 and 1. It was introduced for the COCO keypoint challenge to play the role that intersection over union plays for boxes: thresholding OKS decides which detections count as true positives, and average precision is then computed as for boxes (COCO keypoint evaluation page). Each keypoint contributes an unnormalized Gaussian of its distance to the annotation, whose width grows with the object’s size and with how loosely human annotators place that kind of keypoint. The person-level protocol, with its area ranges, ignored crowds and the full port of the official evaluation, is described in human pose estimation; the borrowed constants and area conventions for animals and other categories are in keypoint detection. This article covers the similarity itself: what its constants mean, what it implies when errors are Gaussian, Ronchi and Perona’s (2017) taxonomy of the errors that lower it, and how evaluation code, toolkit forks and OKS-based non-maximum suppression compute it.

Formula

For a ground-truth object with keypoints $\mathbf{x}_i$, visibility flags $v_i$ and scale $s$, and a prediction $\hat{\mathbf{x}}_i$, the keypoint similarity of keypoint $i$ and the object keypoint similarity are

$$\mathrm{ks}_i = \exp\!\left( -\frac{d_i^2}{2 s^2 \kappa_i^2} \right), \qquad \mathrm{OKS} = \frac{\sum_i \mathrm{ks}_i \, \delta(v_i > 0)}{\sum_i \delta(v_i > 0)},$$

with $d_i = \lVert \hat{\mathbf{x}}_i - \mathbf{x}_i \rVert_2$ (COCO keypoint evaluation page). The page defines $s$ as the square root of the object’s segment area, so $s^2$ is that area, and sets $\kappa_i = 2\sigma_i$, where $\sigma_i^2 = E[d_i^2 / s^2]$ was measured from 5,000 redundantly annotated validation images. The visibility flag is $v = 0$ for unlabelled, 1 for labelled but not visible and 2 for labelled and visible; the average runs over the labelled keypoints of the ground truth, and the detector’s own flags are not used. For people, the page lists $\sigma$ for nine keypoint types, from .025 for the eyes to .107 for the hips; human pose estimation gives the full list, and the code below uses it.

OKS lies in $[0, 1]$: 1 when every labelled keypoint is exact, near 0 when every one is off by several $s\kappa_i$. The official code, computeOks in cocoeval.py, writes the exponent as d² / (2σ)² / (area + eps) / 2, which is the same expression with the ground truth’s segment area for $s^2$, and stores the constants multiplied by ten (.26, .25, … divided by 10.0). Ronchi and Perona state the same Eq. 1 but call $s$ “the area of the instance”; the COCO page and the code use the square root of the area for $s$ and the area for $s^2$.

A useful derived quantity is the distance at which a keypoint’s similarity falls to one half, $r_{50,i} = \kappa_i s \sqrt{2 \ln 2}$. Because $\kappa$ is 4.1 times larger for a hip than for the nose, a hip may be four times farther off for the same credit; human pose estimation tabulates these pixel tolerances for two person sizes.

Intuition

OKS is a soft count of correct keypoints. Instead of a hard radius, as in PCK, each keypoint earns partial credit that decays smoothly with distance, and the decay scale is set per keypoint type by human disagreement. The COCO organizers chose $\kappa_i = 2\sigma_i$ so that an error of one, two or three annotator standard deviations of $d_i/s$ gives a similarity of $e^{-1/8} = .88$, $e^{-4/8} = .61$ and $e^{-9/8} = .32$. The page then reads these through the 68–95–99.7 rule, expecting 68%, 95% and 99.7% of human keypoints to reach those similarities, and reports 75%, 95% and 98.7% in practice.

Because OKS is an average over the labelled keypoints, a person with two labelled keypoints and a person with seventeen count equally in AP (the page says each object has equal importance), but the first person’s OKS rests on two numbers and is far noisier. Unlabelled keypoints are simply ignored, so a detector is neither rewarded nor punished for keypoints the annotators skipped.

Expected similarity under Gaussian error

If a prediction’s error in each image axis is independent Gaussian with standard deviation $\tau$, then $d^2 / \tau^2$ follows a chi-squared distribution with two degrees of freedom, and half of that is exponential with rate 1. Writing $r = \tau^2 / (s\kappa)^2$, this gives (our derivation)

$$\mathrm{ks} = U^{\,r}, \quad U \sim \mathrm{Uniform}(0, 1), \qquad E[\mathrm{ks}] = \frac{1}{1 + r}, \qquad P(\mathrm{ks} \ge t) = 1 - t^{1/r}.$$

The similarity of a keypoint with Gaussian error is a power of a uniform variable, and its mean depends only on the ratio of the error spread to the keypoint’s tolerance. Two consequences follow. First, a target mean similarity fixes the error spread: $E[\mathrm{ks}] = 0.97$ needs $r = 0.031$, and the example below uses this to inject errors of a chosen quality. Second, the 68–95–99.7 reading on the COCO page applies to a one-dimensional error. If $E[d^2] = \sigma^2 s^2$ in two dimensions, the per-axis variance is $\sigma^2 s^2 / 2$, so $r = 1/8$ for every keypoint type, and the fractions of keypoints reaching .88, .61 and .32 are $1 - e^{-1} = 63.2\%$, $1 - e^{-4} = 98.2\%$ and $1 - e^{-9} = 99.99\%$ (our arithmetic). The page’s measured 75%, 95% and 98.7% are closer to the one-dimensional rule at the first two levels and below both models at the third; in our reading the measured spread is not an isotropic Gaussian, which fits the page’s own remark that the real data contain 1–2% outliers, mostly left–right confusions and annotations of the wrong person.

Interpretation

An OKS of 0.5, the loosest COCO threshold, is reached, for instance, when every labelled keypoint lies at its $r_{50}$, or when half are exact and the rest far off; an OKS of 0.95 requires near-annotator precision on every labelled keypoint. On the doubly annotated validation data, human performance at OKS 0.50 is nearly perfect (95%), the median human OKS is about .91, and human performance drops rapidly above .95 (COCO keypoint evaluation page). COCO’s primary metric averages AP over OKS thresholds 0.50:0.05:0.95, so, in our reading, the strictest thresholds sit at or above the level where human annotators themselves disagree. The rest of the protocol, including AP for medium and large people, 20 detections per image and the handling of people without labelled keypoints, is in human pose estimation.

An OKS value is comparable across people of different sizes and across keypoint types, which is its purpose, but not across datasets with different constants: keypoint detection shows how borrowed $\sigma$ values and area conventions move the score for animals.

Error diagnosis: Ronchi and Perona’s taxonomy

Ronchi and Perona (2017) split what lowers keypoint AP into localization errors of individual keypoints, scoring errors of whole detections, background false positives and false negatives, and measured each on the two leading methods of the 2016 COCO keypoint challenge, the bottom-up method of Cao et al. (OpenPose, “Cmu”) and the top-down G-RMI method (“Grmi”). For keypoint $i$ of a detection matched to person $p$, with keypoint similarities computed by the formula above:

• Good: $\mathrm{ks} \ge .85$ with the correct keypoint, within the error margin of human annotators.
• Jitter: $.5 \le \mathrm{ks} < .85$, a small error near the correct location. The paper notes that the limits can be chosen for the application: .5 is COCO’s loosest threshold, and above .85 annotators disagree substantially (around 30%).
• Inversion: $\mathrm{ks} < .5$ with the correct keypoint but $\ge .5$ with another keypoint of the same person; their study considers only left–right confusions.
• Swap: $\mathrm{ks} < .5$ with the correct keypoint but $\ge .5$ with a keypoint in the considered set $J$ of a different person.
• Miss: $\mathrm{ks} < .5$ with every keypoint in the set considered, of every person.

Their Fig. 5(a) finds 73.5% of Cmu’s keypoints good, 13.9% jitter, 7.1% miss, 3.7% inversion and 1.9% swap, and 73.3%, 14%, 7.8%, 3.9% and 1.1% for Grmi. The text rounds the good share to about 75% for both methods, calls their jitter and inversion rates approximately equal, and says Cmu has twice as many swaps and Grmi about 1% more misses; the figure’s values are 1.9% against 1.1% swaps and 7.1% against 7.8% misses. To measure each class’s cost, the paper repositions a keypoint at the distance giving ks = .85 for jitter and .5 for a miss, and, for inversions and swaps, at the same distance from the true location as the prediction had from the wrong part, then recomputes OKS and AP. The text ranks misses as the most costly errors in AP (about 15%), followed by inversions (about 4%), “relative to their low frequency”, at OKS .75; in absolute terms the bars of Fig. 5(d), read from the plot, show misses at about 14–16%, jitter at about 5–6%, inversions at about 4% and swaps at 1–3% (the authors focus on .75 because it correlates almost perfectly with the averaged AP). On the COCO page’s analysis plot for Cao et al. at OKS .9, AP rises from .327 to .415 after correcting misses, .448 after swaps, .545 after inversions and .859 after jitter, cumulatively, and .879 after rescoring.

Scoring. A scoring error occurs when, of two detections near the same person, the one with the higher confidence has the lower OKS. The paper argues that the best score a detection can have is its maximum OKS with any ground-truth instance, computed by an oracle at evaluation time, because it is both monotonic in OKS and calibrated; with such scores AP rose by about 5% on average over thresholds and by up to 10% at OKS .95 (Fig. 6a). Table 2 attributes this to more matches (64 more for Cmu, 156 for Grmi) and higher-OKS matches (590 and 430), and counts 407 scoring errors for Cmu and 82 for Grmi. Background errors weighed differently: the paper reports that false negatives affect AP far more than false positives, about 40% against 5% on average. The COCO page describes an analyze() function that produces the cumulative plots.

Implementations

• cocoapi cocoeval.py (commit 8c9bcc3) is the reference. Besides computeOks, it handles people with no labelled keypoints by measuring the distance from each predicted keypoint to a region around the ground-truth box, as human pose estimation describes, and it never uses predicted visibilities.
• xtcocotools (commit 17252e7), a fork of the COCO API, keeps the formula but adds options. With use_area=False, for datasets whose annotations have no segment area such as CrowdPose and “AIC” (the docstring’s examples), it replaces the area by 0.53 times the box area, both in OKS and in the area-range test. It accepts custom sigmas and adds iouType values for whole-body, foot, face and hand keypoints and for CrowdPose (keypoints_crowd, which decides whether a person has labelled keypoints from the annotation’s num_keypoints, a count that in CrowdPose covers only visible joints, as a code comment notes).
• OKS-based NMS in HRNet’s code (commit 6f69e46). Top-down pipelines produce one pose per person box and may produce near-duplicate poses for one person. HRNet’s coco.py first rescores each pose by the box score times the mean of its keypoint confidences above IN_VIS_THRE (0.2 in the W32 $256 \times 192$ COCO configuration), then suppresses duplicates with oks_nms (or soft_oks_nms, which decays the remaining scores by a Gaussian of their OKS, in the manner of Soft-NMS; see non-maximum suppression) at OKS_THRE 0.9. The similarity it uses, oks_iou, differs from the evaluation’s in two ways. It divides by the mean of the two poses’ areas, and the area passed in is the area of the person’s crop box (np.prod(s*200, 1) in function.py), that is, the detected box widened to the model’s aspect ratio and scaled by 1.25 per side (_xywh2cs in coco.py), not a segment area. Its optional visibility filter, list(vg > t) and list(vd > t), is Python’s and on two lists, which returns the second list, so only the second pose’s flags would be used; coco.py calls oks_nms without in_vis_thre, so that branch does not run in HRNet’s own COCO pipeline. These are our readings of the code; the example below checks the first two.

Example: expected similarity, error classes and rescoring

The script has four parts. Part A checks the closed form for expected keypoint similarity by Monte Carlo, at three error spreads for the wrist on a person of scale 100 pixels, and at COCO’s annotator spread, where it also prints the OKS of $k$ independent keypoints. Part B builds 40 synthetic scenes of three frontal stick figures (written for this example: 17 COCO keypoints, heights 140 to 300 pixels on a $512 \times 512$ canvas, each figure’s silhouette rendered with scikit-image so that its pixel count is the segment area). It places predicted keypoints at known positions of each class, displaced to ks between .9 and 1 (good) or between .55 and .8 (jitter), exactly on the person’s mirror keypoint (inversion) or on another person’s same keypoint (swap), or displaced to ks between .05 and .3 away from every candidate (miss), keeping a class only when the distances make it unambiguous, and checks that a classifier written from the paper’s definitions recovers every label.

The paper leaves three choices open, and the code fixes them as follows. The candidate set for keypoint $i$ is $J = \{i, \text{mirror}(i)\}$ for every class, and an inversion takes precedence over a swap; the code counts keypoints that meet both conditions. Similarities against another person’s keypoint can use that person’s scale (“target”) or the detection’s person’s scale (“own”), and every result is printed under both. In Part B a detection is analysed against the person it was generated from; in Part C against the person with which it has the highest OKS, if that OKS is at least 0.1.

Part C0 checks a compact AP function, written from the definition rather than ported (greedy matching in score order to the unmatched person with the highest OKS at or above the threshold, ranking pooled over images, precision envelope, 101 recall points, without COCO’s area ranges, crowd handling or detection limit), on a hand case with two people and three detections, and on perfect detections. Part C simulates a detector on 200 scenes of two to four people per seed, seeds 0 to 4. Every person gets one detection and, with probability 0.3, a duplicate; an image gets a background detection with probability 0.5. Each predicted keypoint is, with probabilities .02, .04, .07 and .14, placed near the nearest other person’s same keypoint (swap), near the person’s mirror keypoint (inversion), anywhere in the image (miss), or with Gaussian error of mean ks .70 (jitter), and otherwise with Gaussian error of mean ks .97, both from Part A’s closed form; the rates are loosely those of Ronchi and Perona’s Fig. 5(a), and the classes in the tables come from the classifier, not from the injection. Since no network is run, the “model score” is our analogue of a detector’s confidence: half the detection’s OKS with the person it was generated from plus half a uniform random number (half a random number for background detections), so the rescoring results test this analogue, not a real detector. The oracle score is the maximum OKS with any person in the image, as in the paper. The script then computes AP at OKS .50, .75 and .95 for the original detections, after correcting each class alone with the paper’s rule (moving jitter and misses along their own direction from the true keypoint, and keeping the offset from the wrong part for inversions and swaps, is our choice of direction), and cumulatively in the COCO page’s order (miss, swap, inversion, jitter, then oracle scores). Part D re-types HRNet’s oks_iou with its MIT licence and compares it with evaluation OKS. The script uses NumPy 2.3.5 and scikit-image 0.25.2 and runs in about 8 seconds.

import os
os.environ["OMP_NUM_THREADS"] = os.environ["OPENBLAS_NUM_THREADS"] = "1"
os.environ["MKL_NUM_THREADS"] = "1"
import numpy as np
from skimage.draw import polygon, disk

# COCO person keypoints and the sigmas of cocoeval.py (setKpParams); kappa = 2 sigma.
SIGMAS = np.array([.26, .25, .25, .35, .35, .79, .79, .72, .72, .62, .62,
                   1.07, 1.07, .87, .87, .89, .89]) / 10.0
KAPPA = 2 * SIGMAS
MIRROR = np.array([0, 2, 1, 4, 3, 6, 5, 8, 7, 10, 9, 12, 11, 14, 13, 16, 15])
W = 512                                      # square canvas, pixels
SEEDS = (0, 1, 2, 3, 4)                               # scenes for Part C


def figure(rng, H, cx, feet):
    """Frontal stick figure of height ~H: 17 COCO keypoints and its segment area."""
    lean = rng.normal(0, 0.1)
    up, side = np.array([np.sin(lean), -np.cos(lean)]), np.array([np.cos(lean), np.sin(lean)])
    pel = np.array([cx, feet - 0.47 * H]); neck = pel + 0.33 * H * up
    hc, hr = neck + 0.08 * H * up, 0.06 * H
    K = np.zeros((17, 2)); K[0] = hc - 0.015 * H * up
    caps = [(pel, neck, 0.07 * H)]
    for k, sg in ((0, 1), (1, -1)):          # k = 0 left (image +x), 1 right
        o = sg * side
        K[1 + k] = hc + 0.01 * H * up + 0.022 * H * o          # eye
        K[3 + k] = hc + 0.055 * H * o                          # ear
        K[5 + k] = neck - 0.02 * H * up + 0.11 * H * o         # shoulder
        K[11 + k] = pel + 0.075 * H * o                        # hip
        a = rng.uniform(-0.3, 2.6); b = a + rng.uniform(-1.4, 0.3)
        K[7 + k] = K[5 + k] + 0.16 * H * (-np.cos(a) * up + np.sin(a) * o)
        K[9 + k] = K[7 + k] + 0.14 * H * (-np.cos(b) * up + np.sin(b) * o)
        a = rng.uniform(-0.15, 0.5); b = a + rng.uniform(-0.4, 0.2)
        K[13 + k] = K[11 + k] + 0.23 * H * (-np.cos(a) * up + np.sin(a) * o)
        K[15 + k] = K[13 + k] + 0.23 * H * (-np.cos(b) * up + np.sin(b) * o)
        caps += [(K[5 + k], K[7 + k], .025 * H), (K[7 + k], K[9 + k], .025 * H),
                 (K[11 + k], K[13 + k], .03 * H), (K[13 + k], K[15 + k], .03 * H)]
    caps += [(K[5], K[6], .03 * H), (K[11], K[12], .035 * H)]
    sil = np.zeros((W, W), bool)              # the person's silhouette: capsules + head
    for a, b, r in caps:
        n = np.array([-(b - a)[1], (b - a)[0]]); n = r * n / (np.linalg.norm(n) + 1e-12)
        q = np.array([a + n, b + n, b - n, a - n])
        sil[polygon(q[:, 1], q[:, 0], (W, W))] = True
        for c in (a, b):
            sil[disk((c[1], c[0]), r, shape=(W, W))] = True
    sil[disk((hc[1], hc[0]), hr, shape=(W, W))] = True
    return K, float(sil.sum())


def scene(rng, n_people, vis_p=0.9):
    G = []
    for _ in range(n_people):
        H = rng.uniform(140, 300)
        K, area = figure(rng, H, rng.uniform(0.15 * W, 0.85 * W), rng.uniform(H + 10, W - 10))
        vis = np.where(rng.random(17) < vis_p, 2, 0); vis[0] = 2
        G.append(dict(kp=K, vis=vis, area=area))
    return G


def ks(p, g, area, kappa):
    """Keypoint similarity, as in cocoeval.computeOks: exp(-d^2 / (2 area kappa^2))."""
    d2 = ((np.asarray(p) - np.asarray(g)) ** 2).sum(-1)
    return np.exp(-d2 / (kappa ** 2) / (area + np.spacing(1)) / 2)


def oks(P, g):
    """Object keypoint similarity of predicted keypoints P (17 x 2) with person g."""
    m = g["vis"] > 0
    return ks(P, g["kp"], g["area"], KAPPA)[m].mean()


# ---------------------------------------------------------------- Part A
def part_a():
    print("A. Expected keypoint similarity under isotropic Gaussian error (per-axis sd tau)")
    print("   closed form E[ks] = 1 / (1 + tau^2 / (s kappa)^2); wrist, s = 100 px, n = 10^6")
    rng = np.random.default_rng(0)
    s, kap, n = 100.0, KAPPA[9], 1_000_000
    for tau in (3.0, 6.0, 12.0):
        r = tau ** 2 / (s * kap) ** 2
        k = ks(tau * rng.standard_normal((n, 2)), 0.0, s * s, kap)
        se = np.sqrt((1 / (1 + 2 * r) - 1 / (1 + r) ** 2) / n)
        cf = 1 / (1 + r)
        print(f"   tau {tau:4.1f}: closed {cf:.6f}  Monte Carlo {k.mean():.6f}"
              f"  |diff| {abs(k.mean() - cf):.2e}  4 SE {4 * se:.2e}  ok {abs(k.mean() - cf) <= 4 * se}")
    print("   error at COCO's annotator spread (E[d^2] = sigma^2 s^2): P(ks >= exp(-m^2/8))")
    k = ks(np.sqrt(0.5) * SIGMAS[9] * s * rng.standard_normal((n, 2)), 0.0, s * s, kap)
    for m in (1, 2, 3):
        t, p = np.exp(-m * m / 8), 1 - np.exp(-m * m)
        f = (k >= t).mean(); se = np.sqrt(p * (1 - p) / n)
        print(f"   m = {m}: ks >= {t:.3f}  closed {p:.6f}  Monte Carlo {f:.6f}"
              f"  |diff| {abs(f - p):.2e}  4 SE {4 * se:.2e}  ok {abs(f - p) <= 4 * se}")
    print("   OKS of k labelled keypoints, each at the annotator spread (independent errors)")
    for kk in (1, 4, 10, 17):
        u = rng.random((200_000, kk)) ** (1 / 8)          # ks = U^(1/8) in this model
        o = u.mean(1)
        print(f"   k = {kk:2d}: mean {o.mean():.4f}  sd {o.std():.4f}  median {np.median(o):.4f}"
              f"  P(OKS>=.50) {(o >= .5).mean():.4f}  P(OKS>=.95) {(o >= .95).mean():.4f}")


# ---------------------------------------------------------------- Part B
def classify(P, p, G, choice="target"):
    """Ronchi and Perona's localization classes for the keypoints of detection P,
    analysed against ground-truth person p; J = {i, mirror(i)}.
    choice: scale for ks against another person's keypoint ('target' or 'own')."""
    g = G[p]
    lab, wrong, amb = np.full(17, "", dtype=object), np.zeros((17, 2)), 0
    for i in range(17):
        if g["vis"][i] == 0:
            continue
        k = ks(P[i], g["kp"][i], g["area"], KAPPA[i])
        if k >= .85:
            lab[i] = "good"; continue
        if k >= .5:
            lab[i] = "jitter"; continue
        best_inv, best_swp = -1.0, -1.0
        j = MIRROR[i]
        if j != i and g["vis"][j] > 0:
            v = ks(P[i], g["kp"][j], g["area"], KAPPA[j])
            if v >= .5:
                best_inv, inv_pos = v, g["kp"][j]
        for q, h in enumerate(G):
            if q == p:
                continue
            for j in {i, MIRROR[i]}:
                if h["vis"][j] == 0:
                    continue
                a = h["area"] if choice == "target" else g["area"]
                v = ks(P[i], h["kp"][j], a, KAPPA[j])
                if v >= .5 and v > best_swp:
                    best_swp, swp_pos = v, h["kp"][j]
        amb += (best_inv >= .5) and (best_swp >= .5)
        if best_inv >= .5:
            lab[i], wrong[i] = "inversion", inv_pos
        elif best_swp >= .5:
            lab[i], wrong[i] = "swap", swp_pos
        else:
            lab[i] = "miss"
    return lab, wrong, amb


CLASSES = ("good", "jitter", "inversion", "swap", "miss")


def r50(area, kappa):
    """Distance at which ks falls to 0.5."""
    return kappa * np.sqrt(2 * np.log(2) * area)


def inject_b(rng, G):
    """Noise-free injections whose class follows from distances alone."""
    D, L = [], []
    for p, g in enumerate(G):
        P, lab = g["kp"].copy(), []
        for i in range(17):
            for _ in range(50):
                c = CLASSES[rng.integers(5)]
                ro = r50(g["area"], KAPPA[i])
                def far(x, people):           # x farther than r50 from every J target
                    for q in people:
                        h = G[q]
                        for j in {i, MIRROR[i]}:
                            for a in (h["area"], g["area"]):          # both scale choices
                                if np.linalg.norm(x - h["kp"][j]) <= r50(a, KAPPA[j]):
                                    return False
                    return True
                others = [q for q in range(len(G)) if q != p]
                if c in ("good", "jitter", "miss"):
                    t = rng.uniform(.9, 1.0) if c == "good" else (
                        rng.uniform(.55, .8) if c == "jitter" else rng.uniform(.05, .3))
                    th = rng.uniform(0, 2 * np.pi)
                    x = g["kp"][i] + ro * np.sqrt(np.log(t) / np.log(.5)) * np.array([np.cos(th), np.sin(th)])
                    if c == "miss":
                        j = MIRROR[i]
                        if (j != i and np.linalg.norm(x - g["kp"][j]) <= r50(g["area"], KAPPA[j])) \
                                or not far(x, others):
                            continue
                elif c == "inversion":
                    j = MIRROR[i]
                    x = g["kp"][j].copy()
                    if j == i or np.linalg.norm(x - g["kp"][i]) <= ro or not far(x, others):
                        continue
                else:                                                 # swap
                    if not others:
                        continue
                    q = others[rng.integers(len(others))]
                    x = G[q]["kp"][i].copy()
                    j = MIRROR[i]
                    if np.linalg.norm(x - g["kp"][i]) <= ro or (
                            j != i and np.linalg.norm(x - g["kp"][j]) <= r50(g["area"], KAPPA[j])):
                        continue
                break
            else:
                c, x = "good", g["kp"][i].copy()
            P[i] = x; lab.append(c)
        D.append(P); L.append(lab)
    return D, L


def part_b():
    print("\nB. Ronchi-Perona classes recovered from noise-free injections (40 scenes, 3 people)")
    rng = np.random.default_rng(100)
    scenes = [scene(rng, 3, vis_p=1.0) for _ in range(40)]
    for choice in ("target", "own"):
        rng = np.random.default_rng(101)
        conf = np.zeros((5, 5), int); amb = 0
        for G in scenes:
            D, L = inject_b(rng, G)
            for p, (P, lab) in enumerate(zip(D, L)):
                got, _, a = classify(P, p, G, choice); amb += a
                for c, h in zip(lab, got):
                    conf[CLASSES.index(c), CLASSES.index(h)] += 1
        print(f"   scale for other people's keypoints: {choice}")
        print("   injected \\ classified " + " ".join(f"{c:>9s}" for c in CLASSES))
        for c, row in zip(CLASSES, conf):
            print(f"   {c:>21s} " + " ".join(f"{v:9d}" for v in row))
        print(f"   agreement {np.trace(conf)}/{conf.sum()}  ambiguous (inversion and swap) {amb}")


# ---------------------------------------------------------------- Part C
def ap(images, thr, rec=np.linspace(0, 1, 101)):
    """COCO-style AP at one OKS threshold, written from the definition:
    greedy matching in score order, pooled ranking, 101-point interpolation.
    images: list of (O, scores) with O[d, g] the OKS of detection d with person g."""
    S, T, n_gt = [], [], 0
    for O, sc in images:
        n_gt += O.shape[1]
        used = np.zeros(O.shape[1], bool)
        for d in np.argsort(-sc, kind="mergesort"):
            o = np.where(used, -1.0, O[d])
            g = int(np.argmax(o)) if o.size else -1
            tp = g >= 0 and o[g] >= thr
            if tp:
                used[g] = True
            S.append(sc[d]); T.append(tp)
    order = np.argsort(-np.array(S), kind="mergesort")
    tp = np.cumsum(np.array(T)[order]); fp = np.cumsum(~np.array(T)[order])
    rc, pr = tp / n_gt, tp / (tp + fp)
    pr = np.maximum.accumulate(pr[::-1])[::-1]               # precision envelope
    idx = np.searchsorted(rc, rec, side="left")
    return np.where(idx < len(pr), pr[np.minimum(idx, len(pr) - 1)], 0.0).mean()


THRS = (.50, .75, .95)


def hand_case():
    from fractions import Fraction as F
    O = np.array([[.8, 0.], [.6, .3], [0., .9]]); sc = np.array([.9, .8, .7])
    want = {.50: F(253, 303), .75: F(253, 303), .85: F(17, 101), .95: F(0)}
    print("\nC0. AP on a hand case (2 people, 3 detections) and on perfect detections")
    for t, w in want.items():
        v = ap([(O, sc)], t)
        print(f"   OKS {t:.2f}: code {v:.12f}  hand {float(w):.12f}  ok {abs(v - float(w)) < 1e-12}")
    rng = np.random.default_rng(7)
    imgs = []
    for _ in range(20):
        G = scene(rng, 3)
        O = np.array([[oks(g["kp"], h) for h in G] for g in G])
        imgs.append((O, rng.random(3)))
    print("   perfect detections: " + "  ".join(f"AP@{t:.2f} {ap(imgs, t):.6f}" for t in THRS))


def gauss(rng, g, i, r):
    """Gaussian error of per-axis variance r (s kappa_i)^2, so E[ks] = 1/(1+r)."""
    return np.sqrt(r * g["area"]) * KAPPA[i] * rng.standard_normal(2)


R_GOOD, R_JIT = 1 / .97 - 1, 1 / .70 - 1                    # E[ks] = 0.97 and 0.70


def detect(rng, G, p):
    """Simulated detection of person p with an injected error mix (rates near
    Ronchi and Perona's Fig. 5a for their bottom-up method)."""
    g, P = G[p], G[p]["kp"].copy()
    others = [q for q in range(len(G)) if q != p]
    for i in range(17):
        u = rng.random()
        if u < .02 and others:                                    # swap: nearest other person
            q = min(others, key=lambda q: np.linalg.norm(G[q]["kp"][i] - g["kp"][i]))
            P[i] = G[q]["kp"][i] + gauss(rng, G[q], i, R_GOOD)
        elif u < .06 and MIRROR[i] != i:                          # inversion
            P[i] = g["kp"][MIRROR[i]] + gauss(rng, g, MIRROR[i], R_GOOD)
        elif u < .13:                                             # miss: anywhere in the image
            P[i] = rng.uniform(0, W, 2)
        elif u < .27:
            P[i] = g["kp"][i] + gauss(rng, g, i, R_JIT)
        else:
            P[i] = g["kp"][i] + gauss(rng, g, i, R_GOOD)
    return P


def make_c(seed, n_img=200):
    rng = np.random.default_rng(seed)
    data = []
    for _ in range(n_img):
        G = scene(rng, int(rng.integers(2, 5)))
        dets = []
        for p in range(len(G)):
            for _ in range(1 + (rng.random() < .3)):              # 30% duplicates
                P = detect(rng, G, p)
                dets.append((P, .5 * oks(P, G[p]) + .5 * rng.random()))
        for _ in range(int(rng.integers(0, 2))):                   # background detection
            H = rng.uniform(140, 300)
            P, _ = figure(rng, H, rng.uniform(0, W), rng.uniform(H, W))
            dets.append((P, .5 * rng.random()))
        data.append((G, dets))
    return data


def correct(P, lab, wrong, g, types):
    """Ronchi and Perona's corrections: jitter to ks .85, miss to ks .5 along the
    same direction; inversion and swap to the offset they had from the wrong part."""
    P = P.copy()
    for i in range(17):
        if lab[i] not in types:
            continue
        v = P[i] - g["kp"][i]; u = v / (np.linalg.norm(v) + 1e-12)
        if lab[i] in ("jitter", "miss"):
            t = .85 if lab[i] == "jitter" else .5
            P[i] = g["kp"][i] + u * KAPPA[i] * np.sqrt(-2 * np.log(t) * g["area"])
        else:
            P[i] = g["kp"][i] + (P[i] - wrong[i])
    return P


def evaluate(data, labels, types, oracle):
    imgs = []
    for (G, dets), labs in zip(data, labels):
        Ps = []
        for (P, s), (p, lab, wrong) in zip(dets, labs):
            Ps.append(correct(P, lab, wrong, G[p], types) if p >= 0 else P)
        O = np.array([[oks(P, g) for g in G] for P in Ps])
        sc = O.max(1) if oracle else np.array([s for _, s in dets])
        imgs.append((O, sc))
    return [ap(imgs, t) for t in THRS]


ROWS = [("original, model score", (), False), ("original, oracle score", (), True),
        ("alone: miss", ("miss",), False), ("alone: swap", ("swap",), False),
        ("alone: inversion", ("inversion",), False), ("alone: jitter", ("jitter",), False),
        ("cumul.: + miss", ("miss",), False), ("cumul.: + swap", ("miss", "swap"), False),
        ("cumul.: + inversion", ("miss", "swap", "inversion"), False),
        ("cumul.: + jitter", ("miss", "swap", "inversion", "jitter"), False),
        ("cumul.: + oracle score", ("miss", "swap", "inversion", "jitter"), True)]


def part_c():
    res = {}
    freq = {c: np.zeros((len(SEEDS), 5)) for c in ("target", "own")}
    diff = np.zeros(len(SEEDS), int)
    for si, seed in enumerate(SEEDS):
        data = make_c(seed)
        L = {}
        for choice in ("target", "own"):
            labels = []
            for G, dets in data:
                labs = []
                for P, _ in dets:
                    o = [oks(P, g) for g in G]
                    p = int(np.argmax(o)) if max(o) >= .1 else -1      # analysed person
                    if p < 0:
                        labs.append((-1, None, None)); continue
                    lab, wrong, _ = classify(P, p, G, choice)
                    labs.append((p, lab, wrong))
                labels.append(labs)
            L[choice] = labels
            allab = [c for labs in labels for p, lab, _ in labs if p >= 0 for c in lab if c]
            freq[choice][si] = [100 * np.mean([c == k for c in allab]) for k in CLASSES]
            for name, types, oracle in ROWS:
                res[choice, name, seed] = evaluate(data, labels, types, oracle)
        diff[si] = sum(int(a != b) for la, lb in zip(L["target"], L["own"])
                       for (p, x, _), (_, y, _) in zip(la, lb) if p >= 0 for a, b in zip(x, y))
    print(f"\nC. AP at OKS .50/.75/.95, 200 scenes of 2-4 people per seed, seeds {SEEDS}")
    print("   keypoint classes (% of labelled keypoints of analysed detections, mean over seeds):")
    for choice in ("target", "own"):
        print(f"   scale {choice:6s}: " + "  ".join(f"{c} {v:5.2f}" for c, v in zip(CLASSES, freq[choice].mean(0))))
    print(f"   keypoints labelled differently by the two scale choices, per seed: {[int(x) for x in diff]}")
    for choice in ("target", "own"):
        print(f"   scale for other people's keypoints: {choice}")
        print("   row                        AP@.50 mean (min-max)    AP@.75 mean (min-max)    AP@.95 mean (min-max)")
        for name, _, _ in ROWS:
            r = np.array([res[choice, name, s] for s in SEEDS])
            print(f"   {name:24s} " + "  ".join(
                f"{m:.4f} ({a:.4f}-{b:.4f})" for m, a, b in zip(r.mean(0), r.min(0), r.max(0))))
    print("   hypothesis checks per seed (unrounded comparisons; min gain over seeds and thresholds)")
    for choice in ("target", "own"):
        g = lambda n, s: np.array(res[choice, n, s]) - np.array(res[choice, ROWS[0][0], s])
        for t in ("miss", "swap", "inversion"):
            gains = np.array([g("alone: " + t, s) for s in SEEDS])
            print(f"   [{choice}] H2 alone {t:9s} raises AP at all 3 thresholds: "
                  f"{[bool((x > 0).all()) for x in gains]}  min gain {gains.min():.5f}")
        gj = np.array([g("alone: jitter", s) for s in SEEDS])
        print(f"   [{choice}] H3 jitter gain at .75 and .95 > gain at .50: "
              f"{[bool(x[1] > x[0] and x[2] > x[0]) for x in gj]}  gains (seed {SEEDS[0]}) "
              + " ".join(f"{v:.5f}" for v in gj[0]))
        o1 = np.array([np.array(res[choice, ROWS[1][0], s]) - np.array(res[choice, ROWS[0][0], s]) for s in SEEDS])
        o2 = np.array([np.array(res[choice, ROWS[10][0], s]) - np.array(res[choice, ROWS[9][0], s]) for s in SEEDS])
        print(f"   [{choice}] H4 oracle >= model score, original: {[bool((x >= 0).all()) for x in o1]}"
              f"  min {o1.min():.5f}; corrected: {[bool((x >= 0).all()) for x in o2]}  min {o2.min():.5f}")
        seq = [0] + list(range(6, 11))
        st = np.array([[np.array(res[choice, ROWS[a][0], s]) for a in seq] for s in SEEDS])
        step = np.diff(st, axis=1)
        print(f"   [{choice}] H5 cumulative steps non-decreasing: {[bool((x >= 0).all()) for x in step]}"
              f"  strictly increasing: {[bool((x > 0).all()) for x in step]}  min step {step.min():.5f}")


# ---------------------------------------------------------------- Part D
# oks_iou below is re-typed from HRNet's lib/nms/nms.py (commit 6f69e46), whose
# header reads "Copyright (c) Microsoft / Licensed under the MIT License."
# The repository's LICENSE file:
# MIT License
#
# Copyright (c) 2019 Leo Xiao
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.
def oks_iou(g, d, a_g, a_d, sigmas=None, in_vis_thre=None):
    if not isinstance(sigmas, np.ndarray):
        sigmas = np.array([.26, .25, .25, .35, .35, .79, .79, .72, .72, .62, .62, 1.07, 1.07, .87, .87, .89, .89]) / 10.0
    vars = (sigmas * 2) ** 2
    xg = g[0::3]
    yg = g[1::3]
    vg = g[2::3]
    ious = np.zeros((d.shape[0]))
    for n_d in range(0, d.shape[0]):
        xd = d[n_d, 0::3]
        yd = d[n_d, 1::3]
        vd = d[n_d, 2::3]
        dx = xd - xg
        dy = yd - yg
        e = (dx ** 2 + dy ** 2) / vars / ((a_g + a_d[n_d]) / 2 + np.spacing(1)) / 2
        if in_vis_thre is not None:
            ind = list(vg > in_vis_thre) and list(vd > in_vis_thre)
            e = e[ind]
        ious[n_d] = np.sum(np.exp(-e)) / e.shape[0] if e.shape[0] != 0 else 0.0
    return ious


def part_d():
    print("\nD. HRNet's OKS for NMS (oks_iou) against evaluation OKS")
    rng = np.random.default_rng(3)
    G = scene(rng, 1, vis_p=1.0); g = G[0]
    P = g["kp"] + np.array([gauss(rng, g, i, R_JIT) for i in range(17)])
    flat = lambda K, v: np.column_stack([K, v]).ravel()
    ev = oks(P, g)
    for f in (1.0, 2.0):
        v = oks_iou(flat(g["kp"], np.full(17, 2.0)), flat(P, np.full(17, 2.0))[None],
                    g["area"], np.array([f * g["area"]]))[0]
        print(f"   detection area = {f:.0f} x ground-truth area: oks_iou {v:.6f}  evaluation OKS {ev:.6f}"
              f"  |diff| {abs(v - ev):.2e}")
    vg = np.where(np.arange(17) % 2 == 0, 1.0, 0.0); vd = np.where(np.arange(17) < 9, 1.0, 0.0)
    ind = list(vg > .2) and list(vd > .2)
    print(f"   'list(vg > t) and list(vd > t)' equals the detection mask: {ind == list(vd > .2)};"
          f" equals the elementwise AND: {ind == list((vg > .2) & (vd > .2))}")


if __name__ == "__main__":
    part_a()
    part_b()
    hand_case()
    part_c()
    part_d()

A. Expected keypoint similarity under isotropic Gaussian error (per-axis sd tau)
   closed form E[ks] = 1 / (1 + tau^2 / (s kappa)^2); wrist, s = 100 px, n = 10^6
   tau  3.0: closed 0.944704  Monte Carlo 0.944725  |diff| 2.11e-05  4 SE 2.09e-04  ok True
   tau  6.0: closed 0.810287  Monte Carlo 0.810359  |diff| 7.26e-05  4 SE 6.26e-04  ok True
   tau 12.0: closed 0.516389  Monte Carlo 0.516500  |diff| 1.11e-04  4 SE 1.14e-03  ok True
   error at COCO's annotator spread (E[d^2] = sigma^2 s^2): P(ks >= exp(-m^2/8))
   m = 1: ks >= 0.882  closed 0.632121  Monte Carlo 0.633021  |diff| 9.00e-04  4 SE 1.93e-03  ok True
   m = 2: ks >= 0.607  closed 0.981684  Monte Carlo 0.981813  |diff| 1.29e-04  4 SE 5.36e-04  ok True
   m = 3: ks >= 0.325  closed 0.999877  Monte Carlo 0.999866  |diff| 1.06e-05  4 SE 4.44e-05  ok True
   OKS of k labelled keypoints, each at the annotator spread (independent errors)
   k =  1: mean 0.8890  sd 0.0991  median 0.9170  P(OKS>=.50) 0.9962  P(OKS>=.95) 0.3362
   k =  4: mean 0.8889  sd 0.0498  median 0.8952  P(OKS>=.50) 1.0000  P(OKS>=.95) 0.0887
   k = 10: mean 0.8888  sd 0.0315  median 0.8912  P(OKS>=.50) 1.0000  P(OKS>=.95) 0.0107
   k = 17: mean 0.8889  sd 0.0241  median 0.8903  P(OKS>=.50) 1.0000  P(OKS>=.95) 0.0011

B. Ronchi-Perona classes recovered from noise-free injections (40 scenes, 3 people)
   scale for other people's keypoints: target
   injected \ classified      good    jitter inversion      swap      miss
                    good       436         0         0         0         0
                  jitter         0       462         0         0         0
               inversion         0         0       377         0         0
                    swap         0         0         0       381         0
                    miss         0         0         0         0       384
   agreement 2040/2040  ambiguous (inversion and swap) 0
   scale for other people's keypoints: own
   injected \ classified      good    jitter inversion      swap      miss
                    good       436         0         0         0         0
                  jitter         0       462         0         0         0
               inversion         0         0       377         0         0
                    swap         0         0         0       381         0
                    miss         0         0         0         0       384
   agreement 2040/2040  ambiguous (inversion and swap) 0

C0. AP on a hand case (2 people, 3 detections) and on perfect detections
   OKS 0.50: code 0.834983498350  hand 0.834983498350  ok True
   OKS 0.75: code 0.834983498350  hand 0.834983498350  ok True
   OKS 0.85: code 0.168316831683  hand 0.168316831683  ok True
   OKS 0.95: code 0.000000000000  hand 0.000000000000  ok True
   perfect detections: AP@0.50 1.000000  AP@0.75 1.000000  AP@0.95 1.000000

C. AP at OKS .50/.75/.95, 200 scenes of 2-4 people per seed, seeds (0, 1, 2, 3, 4)
   keypoint classes (% of labelled keypoints of analysed detections, mean over seeds):
   scale target: good 76.64  jitter  7.08  inversion  3.47  swap  2.02  miss 10.79
   scale own   : good 76.64  jitter  7.08  inversion  3.47  swap  2.03  miss 10.78
   keypoints labelled differently by the two scale choices, per seed: [8, 8, 9, 13, 4]
   scale for other people's keypoints: target
   row                        AP@.50 mean (min-max)    AP@.75 mean (min-max)    AP@.95 mean (min-max)
   original, model score    0.8903 (0.8821-0.9043)  0.6527 (0.6165-0.6758)  0.0110 (0.0065-0.0130)
   original, oracle score   0.8896 (0.8728-0.9045)  0.7506 (0.7132-0.7778)  0.0475 (0.0396-0.0594)
   alone: miss              0.8968 (0.8903-0.9043)  0.8153 (0.7845-0.8362)  0.0120 (0.0065-0.0172)
   alone: swap              0.8903 (0.8821-0.9043)  0.7222 (0.6913-0.7418)  0.0153 (0.0128-0.0184)
   alone: inversion         0.8937 (0.8821-0.9043)  0.7556 (0.7150-0.7790)  0.0161 (0.0116-0.0189)
   alone: jitter            0.8919 (0.8821-0.9043)  0.6960 (0.6535-0.7211)  0.0167 (0.0089-0.0233)
   cumul.: + miss           0.8968 (0.8903-0.9043)  0.8153 (0.7845-0.8362)  0.0120 (0.0065-0.0172)
   cumul.: + swap           0.8968 (0.8903-0.9043)  0.8591 (0.8389-0.8727)  0.0159 (0.0128-0.0195)
   cumul.: + inversion      0.8968 (0.8903-0.9043)  0.8922 (0.8826-0.9043)  0.0273 (0.0235-0.0314)
   cumul.: + jitter         0.8968 (0.8903-0.9043)  0.8938 (0.8876-0.9043)  0.0491 (0.0415-0.0564)
   cumul.: + oracle score   0.8956 (0.8771-0.9085)  0.8925 (0.8771-0.9077)  0.1878 (0.1562-0.2139)
   scale for other people's keypoints: own
   row                        AP@.50 mean (min-max)    AP@.75 mean (min-max)    AP@.95 mean (min-max)
   original, model score    0.8903 (0.8821-0.9043)  0.6527 (0.6165-0.6758)  0.0110 (0.0065-0.0130)
   original, oracle score   0.8896 (0.8728-0.9045)  0.7506 (0.7132-0.7778)  0.0475 (0.0396-0.0594)
   alone: miss              0.8968 (0.8903-0.9043)  0.8156 (0.7857-0.8365)  0.0120 (0.0065-0.0172)
   alone: swap              0.8903 (0.8821-0.9043)  0.7238 (0.6913-0.7418)  0.0155 (0.0128-0.0184)
   alone: inversion         0.8937 (0.8821-0.9043)  0.7556 (0.7150-0.7790)  0.0161 (0.0116-0.0189)
   alone: jitter            0.8919 (0.8821-0.9043)  0.6960 (0.6535-0.7211)  0.0167 (0.0089-0.0233)
   cumul.: + miss           0.8968 (0.8903-0.9043)  0.8156 (0.7857-0.8365)  0.0120 (0.0065-0.0172)
   cumul.: + swap           0.8968 (0.8903-0.9043)  0.8593 (0.8389-0.8727)  0.0159 (0.0128-0.0195)
   cumul.: + inversion      0.8968 (0.8903-0.9043)  0.8922 (0.8826-0.9043)  0.0274 (0.0235-0.0314)
   cumul.: + jitter         0.8968 (0.8903-0.9043)  0.8938 (0.8876-0.9043)  0.0492 (0.0415-0.0564)
   cumul.: + oracle score   0.8954 (0.8767-0.9084)  0.8923 (0.8767-0.9078)  0.1878 (0.1562-0.2140)
   hypothesis checks per seed (unrounded comparisons; min gain over seeds and thresholds)
   [target] H2 alone miss      raises AP at all 3 thresholds: [True, False, True, True, False]  min gain 0.00000
   [target] H2 alone swap      raises AP at all 3 thresholds: [False, False, False, False, False]  min gain 0.00000
   [target] H2 alone inversion raises AP at all 3 thresholds: [False, False, False, True, True]  min gain 0.00000
   [target] H3 jitter gain at .75 and .95 > gain at .50: [True, True, True, False, True]  gains (seed 0) 0.00000 0.04180 0.01110
   [target] H4 oracle >= model score, original: [True, True, False, False, True]  min -0.01671; corrected: [True, True, False, False, True]  min -0.01320
   [target] H5 cumulative steps non-decreasing: [True, True, False, False, True]  strictly increasing: [False, False, False, False, False]  min step -0.01320
   [own] H2 alone miss      raises AP at all 3 thresholds: [True, False, True, True, False]  min gain 0.00000
   [own] H2 alone swap      raises AP at all 3 thresholds: [False, False, False, False, False]  min gain 0.00000
   [own] H2 alone inversion raises AP at all 3 thresholds: [False, False, False, True, True]  min gain 0.00000
   [own] H3 jitter gain at .75 and .95 > gain at .50: [True, True, True, False, True]  gains (seed 0) 0.00000 0.04180 0.01110
   [own] H4 oracle >= model score, original: [True, True, False, False, True]  min -0.01671; corrected: [True, True, False, False, True]  min -0.01363
   [own] H5 cumulative steps non-decreasing: [True, True, False, False, True]  strictly increasing: [False, False, False, False, False]  min step -0.01363

D. HRNet's OKS for NMS (oks_iou) against evaluation OKS
   detection area = 1 x ground-truth area: oks_iou 0.699588  evaluation OKS 0.699588  |diff| 0.00e+00
   detection area = 2 x ground-truth area: oks_iou 0.776714  evaluation OKS 0.699588  |diff| 7.71e-02
   'list(vg > t) and list(vd > t)' equals the detection mask: True; equals the elementwise AND: False

Part A. The closed form held: every Monte Carlo mean and fraction lies within four standard errors of it (the checks named C1 and C2 in our plan held at all six rows). Part A is an illustration of the derivation rather than a test of it: a planning script had already checked the mean by Monte Carlo at one spread before the example was written, and the closed-form values were printed before the criteria were fixed. The k-keypoint rows show what averaging does. The mean OKS is the same 0.889 for any number of keypoints, but with one labelled keypoint a person passes OKS .95 a third of the time and with seventeen almost never, because the average of many independent similarities concentrates near its mean; in this model the median for ten keypoints, 0.891, is close to the median human OKS of about .91 that the COCO page reports, although real keypoint errors are neither independent nor Gaussian.

Part B. The classifier recovered all 2,040 injected labels under both scale choices, with no keypoint meeting both the inversion and the swap condition (check C3 held). The scenes were built so that every label is unambiguous by distances alone, so this checks the implementation of the definitions, not how often real errors are ambiguous.

Part C0. The AP function matched the hand values to twelve decimals at all four thresholds and gave exactly 1 for perfect detections (checks C4 and C5 held). In the hand case, the second detection loses the first person to the higher-scored first detection and becomes a false positive between two true positives, which is why AP at .50 is 253/303 rather than 1.

Part C. The classified frequencies, about 77% good, 7% jitter and 11% miss, differ from the injected mix because Gaussian jitter of mean ks .70 falls outside the jitter band about half the time: by Part A’s closed form it lands above .85 with probability 0.32 and below .5 with probability 0.20 (our arithmetic), and the latter are counted as misses when no other keypoint is near. The two scale choices relabel between 4 and 13 keypoints per seed and change no mean AP by more than 0.0016. Correcting misses gives the largest gain at OKS .75 of the single-class corrections, as in the paper’s Fig. 5(d), and correcting jitter mostly matters at the strict thresholds. Our planned outcomes, stated per seed and identical under both scale choices:

• Correcting a class alone raises AP at all three thresholds (H2): held for misses on seeds 0, 2 and 3 and failed on seeds 1 and 4; failed for swaps on every seed, where AP at .50 is unchanged in mean, minimum and maximum; held for inversions on seeds 3 and 4 and failed on seeds 0, 1 and 2.
• Correcting jitter raises AP more at .75 and at .95 than at .50 (H3): held on seeds 0, 1, 2 and 4, failed on seed 3.
• Oracle scores give AP at least as high as model scores (H4): held on seeds 0, 1 and 4 and failed on seeds 2 and 3, both on the original and on the fully corrected detections. On the original detections the failures are at OKS .50: at .75 and .95 the lowest oracle AP over seeds is above the highest model AP, while at .50 the oracle mean (0.8896) is below the model mean (0.8903). On the corrected detections the oracle raises AP at .95 on every seed by the same reading of the ranges. Ronchi and Perona argue that maximum-OKS scores achieve the highest performance possible for the given detections; on our analogue that held at the strict thresholds but not at .50 on two seeds. In our reading, one way the construction allows this is through the duplicates: at a fixed threshold, a duplicate of an already matched person is a false positive however high its OKS with that person (unless it also reaches the threshold with another unmatched person), and the oracle gives it a high score. The paper also applies Soft-NMS alongside the oracle rescoring, which lowers the scores of overlapping duplicates; our analogue applies no suppression.
• The cumulative sequence never lowers AP (H5, non-decreasing): held on seeds 0, 1 and 4 and failed on seeds 2 and 3, where the most negative step (−0.01320 under the target choice) equals the oracle step’s minimum printed under H4. Each step raises AP strictly: failed on every seed; AP at .50, for instance, is the same after the swap, inversion and jitter steps as after the miss step to the printed precision. This disagrees with the COCO page, which describes each curve of its analysis as “guaranteed to be strictly higher” than the previous one: on our data, at fixed thresholds, some steps change nothing and the rescoring step lowers mean AP at .50 and .75. The page’s curves are precision–recall curves at OKS .9 on real detectors, and ours are AP values on a simulated detector, so the disagreement concerns the guarantee, not the page’s measured values.

Part D. With equal areas, HRNet’s oks_iou reproduces the evaluation OKS exactly (check C6 held); with a detection area twice the ground truth’s, its normalizer is 1.5 times larger and the same pose scores 0.777 instead of 0.700. The and expression returns the detection’s mask, not the elementwise AND of the two masks (check C7 held), which confirms the reading above.

When to Use It

OKS fits evaluation tasks where every object of a category has the same fixed set of named keypoints, annotations come with an object size, and detections must be matched to objects as in detection: multi-person pose in COCO style, and animal or object keypoints once constants are chosen. It is also the natural similarity for suppressing duplicate poses, with the caveat that NMS implementations normalize differently from evaluation. For single-object tasks, or when a per-keypoint accuracy is wanted, PCK-style thresholds on a normalized distance are simpler to interpret.

Failure Cases

• Few labelled keypoints. A person with one or two labelled keypoints has an OKS as noisy as a single similarity (the example’s $k = 1$ row), yet counts as much in AP as a fully labelled one.
• Constants from another population. The $\sigma_i$ were measured on COCO people; reused for animals or new categories without measurement, OKS no longer means “within annotator error” (keypoint detection).
• Area conventions. Segment area, 0.53 times box area (xtcocotools), or a person-box area (HRNet’s NMS) give different values for the same prediction; Part D shows a 0.08 change from doubling the detection’s area.
• Errors averaged away. A pose with a left–right inversion of two limbs can still pass OKS .5, so AP at loose thresholds hides such structural errors; Ronchi and Perona’s classes make them visible.
• Score dependence. As for any AP, the ranking of detections matters as much as their localization; in the example, replacing the simulated confidence with the oracle raised AP at OKS .95 about fourfold.

Variants

• Per-dataset constants and keypoint sets. COCO-WholeBody, CrowdPose and other datasets use the same form with their own keypoints and $\sigma$, through xtcocotools’ iouType and sigmas options.
• Box-area OKS. xtcocotools’ use_area=False (0.53 times the box area) for annotations without segment areas.
• OKS-NMS and soft OKS-NMS. HRNet’s oks_nms and soft_oks_nms use OKS with a mean-area normalizer as the overlap in greedy or Soft-NMS suppression.
• Error-class breakdowns. Ronchi and Perona’s localization, scoring and background analysis reports AP after each class is corrected, separately or cumulatively.

Related Metrics

• Intersection over union is the box and mask counterpart that OKS replaces in the AP machinery.
• Average precision turns OKS matches into the COCO keypoint metrics; human pose estimation ports the official keypoint evaluation.
• PCK and PCKh count a keypoint as correct within a normalized radius; they are described in Percentage of Correct Keypoints, keypoint detection, human pose estimation and MPII Human Pose.
• Mean per joint position error measures 3D joint error in millimetres, as 3D human pose estimation describes.