Skeleton-Based Action Recognition

Skeleton-based action recognition names the action a person performs from the motion of their body joints alone: a sequence of joint coordinates goes in, a label such as “hand waving” or “falling down” comes out. It is action recognition, a task of video understanding, with a pose signal, measured by a depth sensor or estimated from video, in place of pixels.

Problem Definition

A skeleton sequence is a tensor $\mathbf{X} \in \mathbb{R}^{T \times J \times C}$: $T$ frames, $J$ joints and $C$ channels per joint, $(x, y, z)$ from a depth sensor or $(x, y, c)$, image position and confidence, from a 2D pose estimator. Scenes with several people add a person axis, usually capped at two. The output is one of $K$ action classes for the trimmed sequence; how frames are sampled and combined over time follows temporal modeling.

Where skeletons come from. NTU RGB+D was recorded with Microsoft Kinect v2 sensors, whose body tracker gives the 3D positions of 25 joints per tracked person in camera coordinates, alongside RGB, depth and infrared frames (Shahroudy et al., 2016). For ordinary video, human pose estimation (not yet covered here) supplies them. ST-GCN’s authors ran the OpenPose toolbox (Cao et al., 2017), which groups detected body parts into people with part affinity fields, vector fields along the limbs, on Kinetics, keeping 18 joints per person as $(X, Y, C)$ (Yan, Xiong & Lin, 2018). PoseConv3D used a person detector and an HRNet pose estimator, judging its 2D poses better than Kinect’s 3D ones (Duan et al., 2022, Figure 1).

Why Skeletons, and What They Lose

Skeletons are unaffected by background and lighting changes (Duan et al., 2022), and 3D skeletons are view-invariant in essence, though prone to tracker errors (Shahroudy et al., 2016). They are small: one Kinect body is $25 \times 3 = 75$ numbers per frame, against about 6.2 million values for a $1920 \times 1080$ RGB frame (our arithmetic).

What they lose is objects and scenes. ST-GCN reached 30.7% top-1 on Kinetics from OpenPose skeletons, against 57.0% for an RGB model reported by the Kinetics authors; on 30 classes its authors chose as strongly related to body motion, its mean class accuracy was 72.4%, against 70.4% for an RGB CNN and 72.8% for a flow CNN (Yan, Xiong & Lin, 2018, Tables 2 and 4). Yet they complement pixels: adding ST-GCN to a TSN RGB model raised Kinetics accuracy from 70.3% to 71.2%, against 71.1% when adding optical flow (Table 5; see multimodal fusion).

Approaches

Recurrent Networks and Pseudo-Images

The NTU RGB+D paper proposed a part-aware LSTM (P-LSTM): the body is split into torso, two hands and two legs, each with its own memory cell and input, forget and modulation gates, and a shared output gate. Skeletons were first moved into a body coordinate system: origin at the middle of the spine, x from right to left shoulder, y from spine base to spine, scaled by that spine length. A two-layer P-LSTM reached 62.93% (cross-subject) and 70.27% (cross-view), against 60.69% and 67.29% for a two-layer LSTM and 60.23% and 65.22% for the best hand-crafted method (Shahroudy et al., 2016, Table 3).

Another family draws the sequence as images for a pretrained CNN. Ke et al. (2017) chained each frame’s joints, took their positions relative to the shoulders and hips in cylindrical coordinates and scaled each channel into $(J - 1) \times T$ gray images for VGG19 features, reaching 79.57% (cross-subject) and 84.83% (cross-view) on NTU RGB+D (Table 1). Neighbouring rows are neighbours in the chosen joint order, not in the body.

ST-GCN: Spatial-Temporal Graph Convolution

ST-GCN (Yan, Xiong & Lin, 2018) uses graph convolution, which updates each node of a graph from its neighbours’ features. The graph has one node $v_{ti}$ per joint and frame, spatial edges along the body’s natural connections and temporal edges linking each joint to itself in the next frame. Graph neighbours have no fixed order, so the neighbourhood $B(v_{ti})$ (joints at most one edge away) is split into $K$ subsets by a label map $l_{ti}$, each with its own weights, and each term is divided by its subset’s size $Z_{ti}$ (Eq. 5):

$$f_{out}(v_{ti}) = \sum_{v_{tj} \in B(v_{ti})} \frac{1}{Z_{ti}(v_{tj})} \, f_{in}(v_{tj}) \cdot \mathbf{w}\big(l_{ti}(v_{tj})\big).$$

Partitions. Uni-labeling ($K = 1$) amounts to weighting the average neighbour feature, which loses local differences. Distance partitioning ($K = 2$) separates the root joint from its neighbours, so two weight vectors can encode relative translation. Spatial configuration ($K = 3$) separates the root, the centripetal neighbours closer to the skeleton’s centre of gravity (the mean of the frame’s joints) and the centrifugal ones farther away, using $r_i$, joint $i$’s average distance to that centre over the training frames (Eq. 8):

$$l_{ti}(v_{tj}) = \begin{cases} 0 & \text{if } r_j = r_i \\ 1 & \text{if } r_j < r_i \\ 2 & \text{if } r_j > r_i . \end{cases}$$

Implementation. With one frame’s adjacency $\mathbf{A}$ and self-connections $\mathbf{I}$ split as $\mathbf{A} + \mathbf{I} = \sum_j \mathbf{A}_j$, the layer is printed as (Eq. 10)

$$\mathbf{f}_{out} = \sum_j \mathbf{\Lambda}_j^{-\frac{1}{2}} \mathbf{A}_j \mathbf{\Lambda}_j^{-\frac{1}{2}} \mathbf{f}_{in} \mathbf{W}_j, \qquad \Lambda_j^{ii} = \sum_k A_j^{ik} + \alpha, \quad \alpha = 0.001 .$$

Learnable edge importance replaces $\mathbf{A}_j$ by $\mathbf{A}_j \otimes \mathbf{M}$, an element-wise mask initialized to ones. The temporal convolution extends each neighbourhood over $\Gamma = 9$ frames. The official code (commit 221c0e1) departs from Eq. 10: it normalizes the adjacency by node degree and assigns centripetal and centrifugal neighbours by hop distance to a fixed centre joint.

The Kinetics ablation reached 19.3% top-1 with uni-labeling, 29.1% with distance partitioning, 29.9% with spatial configuration and 30.7% with edge importance, against 22.0% for the same sparse graph with unshared weights (Table 1). The temporal-convolution baseline on concatenated joints is printed at 25.1% in the AAAI version’s Table 1 but 20.3% in arXiv v2, which is also the Temporal Conv. entry of the AAAI Table 2. On NTU RGB+D, ST-GCN reached 81.5% cross-subject and 88.3% cross-view (Table 3).

Adaptive Graphs and Bones

2s-AGCN (Shi et al., 2019) lets the graph change with the layer and the sample:

$$\mathbf{f}_{out} = \sum_{k}^{K_v} \mathbf{W}_k \mathbf{f}_{in} (\mathbf{A}_k + \mathbf{B}_k + \mathbf{C}_k),$$

where $\mathbf{A}_k$ is the normalized body graph, $\mathbf{B}_k$ a freely learned matrix that, unlike a multiplicative mask, can add connections the body lacks, and $\mathbf{C}_k = \operatorname{softmax}(\mathbf{f}_{in}^T \mathbf{W}_{\theta k}^T \mathbf{W}_{\phi k} \mathbf{f}_{in})$ a per-sample graph from dot products of embedded joint features, as in self-attention. A second stream reads bones, vectors from each bone’s joint nearer the centre of gravity to the other, and the two streams’ softmax scores are added. On cross-view, after their training changes raised the ST-GCN baseline from 88.3% to 92.7%, the adaptive graph reached 93.7%, bones alone 93.2% and both streams 95.1% (Tables 1 and 2). CTR-GCN (Chen et al., 2021) refines a shared topology separately for each channel and sample, and fuses four streams: joint, bone, joint motion and bone motion.

Heatmap Volumes and 3D CNNs: PoseConv3D

PoseConv3D (Duan et al., 2022), often called PoseC3D, argues that graph networks on coordinates are sensitive to small coordinate perturbations, hard to fuse early with RGB, and costly per added person. It draws each frame’s 2D pose as $K$ joint heatmaps weighted by confidence (Eq. 1),

$$J_{kij} = \exp\left(-\frac{(i - x_k)^2 + (j - y_k)^2}{2\sigma^2}\right) c_k ,$$

or as limb heatmaps, accumulates all people into the same maps, and stacks them into a $K \times T \times H \times W$ volume, cropped around the people and sampled uniformly over time, which a 3D convolutional network (SlowOnly) classifies.

Datasets and Benchmarks

• NTU RGB+D (Shahroudy et al., 2016): 56,880 samples of 60 classes from 40 subjects, filmed by three Kinect v2 cameras at −45°, 0° and +45°, whose height and distance changed over 17 setups. Cross-subject: 20 named subjects for training (40,320 samples), 20 for testing (16,560). Cross-view: cameras 2 and 3 for training (37,920), camera 1 for testing (18,960). ST-GCN prints smaller splits (39,889/16,390 and 37,462/18,817).
• NTU RGB+D 120 (Liu et al., 2020, arXiv v2): 114,480 samples (113,945 in CTR-GCN’s description) of 120 classes from 106 subjects in 32 setups of varied location and background. Cross-subject splits the subjects 53/53; cross-setup trains on even-numbered setups and tests on odd ones.
• Kinetics-Skeleton: ST-GCN’s released OpenPose skeletons of Kinetics-400, with 240,000 training and 20,000 validation clips padded to $T = 300$ frames (Yan, Xiong & Lin, 2018), keeping two people per clip by average joint confidence (Shi et al., 2019).

Evaluation Metrics and Results

Results are top-1 accuracy on each test split. NTU defines no validation set, and some papers call its test split “validation”. The printed results below differ in input (Kinect 3D or estimated 2D poses), streams and test clips.

Method (year)	Input and streams	NTU-60 X-Sub	NTU-60 X-View	NTU-120 X-Sub	NTU-120 X-Set
Part-aware LSTM (2016; 2-layer in the NTU-60 columns)	Kinect 3D joints	62.93	70.27	25.5	26.3
Clips + CNN + MTLN (2017)	Kinect 3D joints, pseudo-images	79.57	84.83	–	–
ST-GCN (2018)	Kinect 3D joints	81.5	88.3	70.7*	73.2*
2s-AGCN (2019)	joint stream / bone stream / both	– / – / 88.5	93.7 / 93.2 / 95.1	–	–
CTR-GCN (2021)	bone / joint + bone / four streams	– / – / 92.4	– / – / 96.8	85.7 / 88.7 / 88.9	87.5 / 90.1 / 90.6
PoseConv3D (2022)	HRNet 2D joints / joints + limbs, 10 clips	93.7 / 94.1	96.6 / 97.1	86.0 / 86.9	89.6 / 90.3

Sources: Shahroudy et al. (2016, Table 3); Liu et al. (2020, Table 4); Ke et al. (2017, Table 1); Yan, Xiong & Lin (2018, Table 3); Shi et al. (2019, Tables 2 and 3); Chen et al. (2021, Tables 5 and 6); Duan et al. (2022, Table 8). *Listed by Duan et al. (2022), not by the ST-GCN paper. On Kinetics-Skeleton, top-1 rose from 30.7% (ST-GCN) to 36.1% (2s-AGCN, Table 4) and 47.7% (PoseConv3D with its own 2D poses, joints and limbs, Table 8).

Example: View Variation, a Graph Layer and an MLP

The example animates a 16-joint 3D stick figure in four actions (right-hand waving, clapping, raising both arms once, walking in place): 20-frame clips with random speed, phase and body size, seen from a random direction (any rotation about the vertical axis, up to 0.3 rad of tilt) 2.5–4 m away, with 5 cm of joint noise. The NumPy part builds the spatial-configuration partition, compares Eq. 10 as printed with degree normalization, runs one layer, and checks that NTU-style view normalization (here from the clip’s mean pose) removes the viewpoint.

import numpy as np

# A 16-joint stick figure. Body frame: x to the person's left, y up, z forward (metres).
NAMES = ["pelvis", "spine", "neck", "head", "l_shoulder", "l_elbow", "l_wrist",
         "r_shoulder", "r_elbow", "r_wrist", "l_hip", "l_knee", "l_ankle",
         "r_hip", "r_knee", "r_ankle"]
EDGES = [(0, 1), (1, 2), (2, 3), (2, 4), (4, 5), (5, 6), (2, 7), (7, 8), (8, 9),
         (0, 10), (10, 11), (11, 12), (0, 13), (13, 14), (14, 15)]
CLASSES = ["wave", "clap", "raise_arms", "walk_in_place"]
J, T, C = len(NAMES), 20, 3

def unit(*v):
    v = np.array(v, float)
    return v / np.linalg.norm(v)

def pose(cls, t, w, ph, t0):
    """Joint positions (J, 3) at time t in [0, 1) for one action."""
    s = np.sin(w * t + ph)
    arm = {"l": [unit(0.15, -1, 0)] * 2, "r": [unit(-0.15, -1, 0)] * 2}
    leg = {"l": [unit(0, -1, 0)] * 2, "r": [unit(0, -1, 0)] * 2}
    if cls == "wave":                       # right forearm swings sideways above the head
        a = 0.6 * s
        arm["r"] = [unit(-0.5, 0.85, 0), unit(np.sin(a), np.cos(a), 0)]
    elif cls == "clap":                     # forearms close in front of the chest and open again
        k = 0.2 + 0.7 * (0.5 + 0.5 * s)
        arm["l"] = [unit(0.25, -0.3, 0.9), unit(-k, 0.3, 0.8)]
        arm["r"] = [unit(-0.25, -0.3, 0.9), unit(k, 0.3, 0.8)]
    elif cls == "raise_arms":               # both arms rise sideways once, from down to overhead
        a = np.pi * np.clip((t - t0) / 0.4, 0, 1)
        arm["l"] = [unit(np.sin(a), -np.cos(a), 0)] * 2
        arm["r"] = [unit(-np.sin(a), -np.cos(a), 0)] * 2
    else:                                   # walk in place: knees lift in turn, arms swing
        for side, sign in (("l", 1), ("r", -1)):
            b = 0.9 * max(0.0, sign * s)
            leg[side] = [unit(0, -np.cos(b), np.sin(b)), unit(0, -1, -0.6 * np.sin(b))]
            g = -0.4 * sign * s
            arm[side] = [unit(0.1 * sign, -np.cos(g), np.sin(g))] * 2
    X = np.zeros((J, 3))
    X[0], X[1], X[2], X[3] = (0, 1.0, 0), (0, 1.25, 0), (0, 1.5, 0), (0, 1.68, 0)
    for side, sh, hip, sx in (("l", 4, 10, 1), ("r", 7, 13, -1)):
        X[sh] = (0.18 * sx, 1.45, 0)
        X[sh + 1] = X[sh] + 0.30 * arm[side][0]
        X[sh + 2] = X[sh + 1] + 0.27 * arm[side][1]
        X[hip] = (0.10 * sx, 0.95, 0)
        X[hip + 1] = X[hip] + 0.45 * leg[side][0]
        X[hip + 2] = X[hip + 1] + 0.45 * leg[side][1]
    return X

def rot_y(a):
    c, s = np.cos(a), np.sin(a)
    return np.array([[c, 0, s], [0, 1, 0], [-s, 0, c]])

def rot_x(a):
    c, s = np.cos(a), np.sin(a)
    return np.array([[1, 0, 0], [0, c, -s], [0, s, c]])

def make_dataset(n_per_class, rng, noise=0.05):
    """Sequences (N, T, J, 3) in camera coordinates and labels (N,)."""
    xs, ys = [], []
    for y, cls in enumerate(CLASSES):
        for _ in range(n_per_class):
            w = 2 * np.pi * rng.uniform(1.5, 3.0)            # 1.5 to 3 cycles per clip
            ph, t0 = rng.uniform(0, 2 * np.pi), rng.uniform(0.05, 0.45)
            seq = np.stack([pose(cls, t, w, ph, t0) for t in np.arange(T) / T])
            seq *= rng.uniform(0.85, 1.15)                    # body size
            R = rot_x(rng.uniform(-0.3, 0.3)) @ rot_y(rng.uniform(0, 2 * np.pi))  # random view
            seq = seq @ R.T + [rng.uniform(-0.5, 0.5), -1.0, rng.uniform(2.5, 4.0)]
            xs.append(seq + noise * rng.standard_normal(seq.shape))  # joint jitter
            ys.append(y)
    return np.array(xs), np.array(ys)

def view_normalize(seq):
    """NTU-style body frame, from the clip's mean pose (averaging suppresses jitter):
    origin at the spine joint, x along right->left shoulder, y along pelvis->spine,
    scaled by the pelvis-spine length."""
    f = seq.mean(0)
    x = f[4] - f[7]
    x /= np.linalg.norm(x)
    y = f[1] - f[0]
    y -= (y @ x) * x
    y /= np.linalg.norm(y)
    R = np.stack([x, y, np.cross(x, y)])      # rows: new axes
    return (seq - f[1]) @ R.T / np.linalg.norm(f[1] - f[0])

def spatial_configuration_adjacency(edges, r, alpha=1e-3):
    """ST-GCN partition into root / centripetal / centrifugal subsets A_k (A_0 + A_1 + A_2 = A + I).
    Returns A_k, Eq. 10 as printed (Lambda_k^-1/2 A_k Lambda_k^-1/2) and a degree-normalized
    version D^-1 A_k with D the degree of A + I, which is what we use."""
    A = np.zeros((3, J, J))
    nbrs = [(i, i) for i in range(J)] + [(i, j) for a, b in edges for i, j in ((a, b), (b, a))]
    for i, j in nbrs:                         # j is a neighbour of root i
        k = 0 if r[j] == r[i] else (1 if r[j] < r[i] else 2)
        A[k, i, j] = 1
    lam = A.sum(2) + alpha
    eq10 = A / np.sqrt(lam[:, :, None]) / np.sqrt(lam[:, None, :])
    return A, eq10, A / A.sum((0, 2))[None, :, None]

def st_gcn_layer(f, A_hat, W, w_t):
    """f: (T, J, C_in); W: (K, C_in, C_out); w_t: (G, C_out), a per-channel temporal kernel
    (ST-GCN mixes channels in its temporal convolution too)."""
    g = sum(A_hat[k] @ f @ W[k] for k in range(len(W)))    # sum_k A_k f W_k, as in Eq. 10
    G = len(w_t)
    gp = np.pad(g, ((G // 2, G // 2), (0, 0), (0, 0)))
    return np.stack([sum(gp[t + q] * w_t[q] for q in range(G)) for t in range(len(g))])

rng = np.random.default_rng(0)
X, y = make_dataset(10, rng)
print("dataset", X.shape, "labels", np.bincount(y))

# r_i: mean distance from the frame's centre of gravity to joint i, over the training frames
r = np.linalg.norm(X - X.mean(2, keepdims=True), axis=3).mean((0, 1))
A, eq10, A_hat = spatial_configuration_adjacency(EDGES, r)
I_plus_A = np.eye(J)
for a, b in EDGES:
    I_plus_A[a, b] = I_plus_A[b, a] = 1
print("A0 + A1 + A2 == A + I:", np.array_equal(A.sum(0), I_plus_A))
for i in (5, 2, 11):
    print(f"{NAMES[i]:>8}:", ", ".join(f"{name} {[NAMES[j] for j in np.flatnonzero(A[k, i])]}"
          for k, name in enumerate(["root", "centripetal", "centrifugal"])))
for name, M in (("Eq. 10 as printed", eq10), ("degree-normalized", A_hat)):
    print(f"l_elbow row, {name}:", {NAMES[j]: round(float(M[:, 5, j].sum()), 3)
                                    for j in np.flatnonzero(A.sum(0)[5])})

W = rng.standard_normal((3, C, 8)) * 0.3
out = st_gcn_layer(X[0], A_hat, W, rng.standard_normal((9, 8)) * 0.3)
print("one ST-GCN layer:", X[0].shape, "->", out.shape)

# The same clean sequence seen from two views, before and after view normalization
clean = np.stack([pose("wave", t, 2 * np.pi * 2, 0.3, 0.2) for t in np.arange(T) / T])
v1, v2 = clean @ rot_y(0.4).T, clean @ rot_y(2.5).T + [0.3, 0, 3.0]
print("max |view1 - view2| raw: %.2f m, after view normalization: %.2f body units"
      % (np.abs(v1 - v2).max(), np.abs(view_normalize(v1) - view_normalize(v2)).max()))

dataset (40, 20, 16, 3) labels [10 10 10 10]
A0 + A1 + A2 == A + I: True
 l_elbow: root ['l_elbow'], centripetal [], centrifugal ['l_shoulder', 'l_wrist']
    neck: root ['neck'], centripetal ['spine', 'l_shoulder', 'r_shoulder'], centrifugal ['head']
  l_knee: root ['l_knee'], centripetal ['l_hip'], centrifugal ['l_ankle']
l_elbow row, Eq. 10 as printed: {'l_shoulder': 0.707, 'l_elbow': 0.999, 'l_wrist': 22.355}
l_elbow row, degree-normalized: {'l_shoulder': 0.333, 'l_elbow': 0.333, 'l_wrist': 0.333}
one ST-GCN layer: (20, 16, 3) -> (20, 16, 8)
max |view1 - view2| raw: 3.10 m, after view normalization: 0.00 body units

Averaged over these clips, the hanging elbow lies closer to the centre of gravity than the shoulder, so the data-driven partition makes the shoulder centrifugal for the elbow. In our analysis, Eq. 10 as printed gives the elbow–wrist edge a weight of $1/\sqrt{(2 + \alpha)\,\alpha} \approx 22.4$: the elbow has two centrifugal neighbours, while the wrist has none, so its $\Lambda$ entry is only $\alpha$. We use degree normalization, as the official code does.

The PyTorch part (PyTorch 2.14.1, run after the NumPy part) trains an MLP on the 960 flattened coordinates and a two-unit ST-GCN (16 channels, spatial-configuration partition, edge importance, temporal kernel 9) on 10 clips per class, in camera and body-centred coordinates, and tests on 50 new clips per class, over seeds 0–4 (data from NumPy seeds 100–104).

import torch
import torch.nn as nn

torch.set_num_threads(4)
print("torch", torch.__version__)

class STGCNUnit(nn.Module):
    """Spatial graph convolution over K subsets with a learnable edge-importance mask M,
    then a temporal convolution of kernel size 9 over each joint; input (N, C, T, V)."""
    def __init__(self, c_in, c_out, A):
        super().__init__()
        self.register_buffer("A", torch.tensor(A, dtype=torch.float32))   # (K, V, V)
        self.M = nn.Parameter(torch.ones_like(self.A))                     # initialized to all ones
        self.gcn = nn.Conv2d(c_in, c_out * len(A), 1)                      # the K weight matrices W_k
        self.tcn = nn.Sequential(nn.BatchNorm2d(c_out), nn.ReLU(),
                                 nn.Conv2d(c_out, c_out, (9, 1), padding=(4, 0)),
                                 nn.BatchNorm2d(c_out))
        self.res = nn.Conv2d(c_in, c_out, 1)

    def forward(self, x):
        n, _, t, v = x.shape
        h = self.gcn(x).view(n, len(self.A), -1, t, v)
        h = torch.einsum("nkctj,kij->ncti", h, self.A * self.M)            # sum_k (A_k * M_k) f W_k
        return torch.relu(self.tcn(h) + self.res(x))

class TinySTGCN(nn.Module):
    def __init__(self, A, n_cls=4):
        super().__init__()
        self.bn = nn.BatchNorm2d(3)                        # shared by all joints
        self.units = nn.Sequential(STGCNUnit(3, 16, A), STGCNUnit(16, 16, A))
        self.fc = nn.Linear(16, n_cls)

    def forward(self, x):                                  # x: (N, T, V, C)
        return self.fc(self.units(self.bn(x.permute(0, 3, 1, 2))).mean((2, 3)))

def mlp():
    return nn.Sequential(nn.Flatten(), nn.BatchNorm1d(T * J * 3),
                         nn.Linear(T * J * 3, 128), nn.ReLU(), nn.Linear(128, 4))

def fit(model, x, y, steps=120):
    opt = torch.optim.Adam(model.parameters(), lr=1e-2, weight_decay=1e-4)
    for _ in range(steps):                                 # full-batch training
        opt.zero_grad()
        nn.functional.cross_entropy(model(x), y).backward()
        opt.step()
    return model.eval()

def accuracy(model, x, y):
    with torch.no_grad():
        return (model(x).argmax(1) == y).float().mean().item()

as_t = lambda a: torch.tensor(np.asarray(a), dtype=torch.float32)
results = {}
for seed in range(5):
    rng = np.random.default_rng(100 + seed)
    Xtr, ytr = make_dataset(10, rng)                       # 10 training clips per class
    Xte, yte = make_dataset(50, rng)                       # 50 test clips per class, new views
    r = np.linalg.norm(Xtr - Xtr.mean(2, keepdims=True), axis=3).mean((0, 1))
    A_seed = spatial_configuration_adjacency(EDGES, r)[2]
    for view in ("camera", "body-centred"):
        f = (lambda s: s) if view == "camera" else view_normalize
        xtr, xte = (as_t([f(s) for s in X]) for X in (Xtr, Xte))
        for name, make in (("MLP", mlp), ("ST-GCN", lambda: TinySTGCN(A_seed))):
            torch.manual_seed(seed)
            m = fit(make(), xtr, torch.tensor(ytr))
            results.setdefault((view, name), []).append(accuracy(m, xte, torch.tensor(yte)))

print("test accuracy (4 classes, chance 0.25), seeds 0-4")
for (view, name), accs in results.items():
    print(f"{view:>12} {name:>6}: " + " ".join(f"{a:.3f}" for a in accs)
          + f" | mean {np.mean(accs):.3f}")

torch 2.14.1
test accuracy (4 classes, chance 0.25), seeds 0-4
      camera    MLP: 0.940 0.960 0.865 0.885 0.905 | mean 0.911
      camera ST-GCN: 0.500 0.715 0.735 0.685 0.645 | mean 0.656
body-centred    MLP: 0.995 0.995 1.000 0.995 1.000 | mean 0.997
body-centred ST-GCN: 1.000 1.000 1.000 1.000 1.000 | mean 1.000

Two runs gave identical output, in about 19 seconds on our CPU. In camera coordinates the MLP reached 0.865–0.960 and the tiny ST-GCN only 0.500–0.735; after view normalization both reached 0.995–1.000, so on this toy the coordinate frame mattered more than the architecture. Our untested explanation: a two-unit graph network that averages over all joints and frames cannot easily undo a per-clip rotation and translation, while the MLP’s first layer can combine any joints. The toy does not test the papers’ deeper networks on large datasets.

Challenges

Pose errors. Kinect’s tracker sometimes took seats or tables for bodies; the NTU authors discarded skeletons whose horizontal spread exceeded 0.8 of their vertical spread (Shahroudy et al., 2016). On FineGYM, dropping one limb keypoint in every frame lowered the mean class accuracy of the graph network MS-G3D from 92.0% to 77.7% and of PoseConv3D from 92.4% to 91.5% (Duan et al., 2022, Table 4).

Occlusion. Joints hidden by the body, objects or other people are missing or guessed.

Several people. NTU’s authors normalized every body relative to the main actor, the one with the most 3D motion, and 2s-AGCN pads a missing second body with zeros. A graph network’s cost grows with the number of people; on Volleyball clips with 13 people, PoseConv3D drew everyone into one heatmap volume and reached 91.3% top-1, 2.1 points above a graph network (Duan et al., 2022).

View variation. 3D skeletons can be rotated into a body frame, as above; 2D skeletons cannot. On NTU RGB+D 120, RGB and depth models scored lower on cross-setup than on cross-subject, but a skeleton LSTM scored higher (57.9% against 55.7%), which the authors attribute to the view invariance of 3D skeletons (Liu et al., 2020, Table 5).

Object-dependent actions. The skeleton model’s most confused pairs on NTU RGB+D 120 include “put on jacket” → “put on bag/backpack”, “play magic cube” → “counting money” and “take a photo of other person” → “shoot at other person with a gun” (Liu et al., 2020, Table 6). Fusing pose with RGB helps: RGBPose-Conv3D reached 97.0% cross-subject and 99.6% cross-view on NTU RGB+D (Duan et al., 2022, Table 9).