Stacked Hourglass Network

The stacked hourglass network is a convolutional network for single-person human pose estimation, introduced by Alejandro Newell, Kaiyu Yang and Jia Deng in a 2016 paper at the European Conference on Computer Vision (ECCV). Its building block, the hourglass, is an encoder–decoder that pools a feature map down to $4 \times 4$ and brings it back up, adding at each resolution a branch that skipped the bottleneck. Several hourglasses are placed one after another; each ends in a set of joint heatmaps that is trained against the ground truth (intermediate supervision) and fed back into the features for the next hourglass. The network outputs one heatmap per joint, as described in heatmap regression. The model scored 90.9% PCKh@0.5 on the MPII Human Pose test set, and the design was later reused for bottom-up multi-person pose and as the backbone of the keypoint-based detectors CornerNet and CenterNet.

The ECCV paper could not be read for this article: its open copy is on arXiv and the publisher’s copy on SpringerLink, neither of which the site uses, and the authors’ pages link only to arXiv. The architecture and training procedure below are taken from the authors’ released training code, pose-hg-train (commit 2fef691, the repository’s last commit, of January 2017), and the results from the MPII results page and from later papers that tabulate them. No statement below is taken from the paper’s text: the paper is listed among the references only to identify the work, and each detail is attributed to the code or to the later source that states it.

Problem Addressed

Single-person pose estimation takes a crop centred on a person, at a roughly known scale, and locates each joint. By 2015 the leading methods regressed one heatmap per joint with a fully convolutional network (Tompson et al.; see heatmap regression). Two needs pull against each other. Locating a wrist precisely requires features at high resolution. Deciding which of several wrist-like blobs is the right wrist, or which is left and which is right, requires context from the whole body: the elbow, the shoulder, the orientation of the torso. A network of stride-1 convolutions sees only as far as its receptive field, and making that field cover the whole person with small kernels takes many layers.

Two answers appeared in 2016. Convolutional pose machines (Wei et al., CVPR 2016) chained stages whose receptive fields grew until they covered the body, and supervised each stage. The stacked hourglass network instead used pooling to reach a $4 \times 4$ map whose cells see the whole $256 \times 256$ input, then upsampled and merged features from every resolution, so that each output cell combines global context with local detail; it also supervised every stage. The associative embedding paper by Newell, Huang and Deng later summarized the model in these terms: it combines global and local features, capturing the structure of the whole body while keeping the fine detail needed for precise localization (NIPS 2017, §3.1).

Architecture

The released network takes a $256 \times 256$ RGB crop and returns, for each of the eight hourglasses, 16 heatmaps (MPII’s 16 joints) of $64 \times 64$ cells (opts.lua: inputRes 256, outputRes 64, nStack 8, nFeats 256). All hourglasses work at the same 256 channels; nothing is shared between them.

Stem

A $7 \times 7$ convolution with stride 2 and 64 channels, with batch normalization and ReLU, reduces the crop to $128 \times 128$; a residual module to 128 channels, a $2 \times 2$ max-pooling to $64 \times 64$, and two more residual modules, the second widening to 256 channels, complete the stem (hg.lua, createModel). All later processing in the full-resolution path happens at $64 \times 64$, a quarter of the input.

Residual Module

Every block in the network is the same residual module (Residual.lua): batch normalization, ReLU and a $1 \times 1$ convolution to half the output channels; batch normalization, ReLU and a $3 \times 3$ convolution at that width; batch normalization, ReLU and a $1 \times 1$ convolution back to the output width. A shortcut adds the input, through a $1 \times 1$ convolution only when the channel counts differ. In the code’s ordering, normalization and activation come before each convolution and nothing follows the addition, the pre-activation form of a residual connection (He et al., “Identity Mappings in Deep Residual Networks”, 2016). The $3 \times 3$ convolutions are the only spatial filters; the network has no kernel larger than $3 \times 3$ after the stem.

Hourglass Module

The hourglass is defined recursively in hg.lua. An hourglass of order $n$ applied to a feature map $x$ at some resolution computes

$$\begin{aligned} u &= R_{\text{up}}(x), \\ \ell &= R_{3}\big(H_{n-1}\big(R_{1}(\operatorname{maxpool}_{2 \times 2}(x))\big)\big), \\ H_n(x) &= u + \operatorname{upsample}_{\text{nearest}, 2\times}(\ell), \end{aligned}$$

where each $R$ is a residual module with its own weights, and $H_0$ is a single residual module. The upper branch $u$ keeps the input’s resolution; the lower branch halves it, recurses, and is upsampled by nearest-neighbour repetition and added back. The released network calls hourglass(4, ...) on the $64 \times 64$ map, so four poolings reach $4 \times 4$, and the result returns to $64 \times 64$ after four upsamplings. Each resolution thus has one residual module on the way down, one in the skip branch, and one on the way up (nModules 1 sets how many are used at each place).

Two properties follow from this definition. First, the module is symmetric: the same number of residual modules work at each resolution on the way down and on the way up, unlike encoder–decoders with a heavy encoder and a light decoder; later papers single out this symmetry when they classify pose networks, among them the HRNet paper (Sun et al., 2019, Fig. 2). Second, merging is by addition, not concatenation, so the channel count stays at 256 everywhere. The associative embedding paper describes the merge in the same way, as upsampling followed by elementwise addition (Fig. 1). SimpleBaseline’s description of the hourglass mentions “skip layer feature concatenation” (Xiao, Wu and Wei, 2018, §1), which does not match the released code. Compared with U-Net, which also has skip connections at every scale, the hourglass skips through a residual module rather than copying the feature map, adds rather than concatenates, and is repeated rather than used once.

A $4 \times 4$ cell at the bottleneck corresponds to a $64 \times 64$ block of the input, so a $3 \times 3$ convolution there already lets each cell draw on most of the crop and the bottleneck’s convolutions together cover all of it (our reading of the geometry); the cells at $64 \times 64$ add detail from their own neighbourhood through the skip branches. This is the mechanism the example under Training tests on a small scale.

Stacking and Intermediate Prediction

After each hourglass, one more residual module and a $1 \times 1$ convolution with batch normalization and ReLU produce the stack’s features $f_i$, and a $1 \times 1$ convolution maps them to the 16 heatmaps $\hat{\mathbf{y}}_i$. For every hourglass except the last, the input to the next one is

$$x_{i+1} = x_i + W_f f_i + W_y \hat{\mathbf{y}}_i,$$

where $W_f$ and $W_y$ are $1 \times 1$ convolutions back to 256 channels (hg.lua). The next hourglass therefore sees the previous hourglass’s input, its features, and its heatmaps re-embedded into feature space, and can correct the heatmaps with the whole body in view again. The associative embedding paper describes the stack as repeated bottom-up and top-down processing that produces a series of intermediate predictions, the last hourglass giving the final one (Fig. 1).

The released Torch code is written in Lua for Torch7. The PyTorch transcription below follows hg.lua and Residual.lua with their default options and counts parameters by component. It runs with PyTorch 2.14.1 in a second or two.

import torch
import torch.nn as nn
import torch.nn.functional as F

class Residual(nn.Module):                    # Residual.lua: pre-activation bottleneck
    def __init__(self, i, o):
        super().__init__()
        bn_relu = lambda c: [nn.BatchNorm2d(c), nn.ReLU()]
        self.conv = nn.Sequential(*bn_relu(i), nn.Conv2d(i, o // 2, 1),
                                  *bn_relu(o // 2), nn.Conv2d(o // 2, o // 2, 3, padding=1),
                                  *bn_relu(o // 2), nn.Conv2d(o // 2, o, 1))
        self.skip = nn.Identity() if i == o else nn.Conv2d(i, o, 1)
    def forward(self, x):
        return self.conv(x) + self.skip(x)

class Hourglass(nn.Module):                   # hg.lua: hourglass(n, f, inp), nModules = 1
    def __init__(self, n, f):
        super().__init__()
        self.up1, self.low1, self.low3 = Residual(f, f), Residual(f, f), Residual(f, f)
        self.low2 = Hourglass(n - 1, f) if n > 1 else Residual(f, f)
    def forward(self, x):
        low = self.low3(self.low2(self.low1(F.max_pool2d(x, 2))))
        return self.up1(x) + F.interpolate(low, scale_factor=2, mode="nearest")

class StackedHourglass(nn.Module):            # hg.lua: createModel()
    def __init__(self, n_stack=8, f=256, n_joints=16):
        super().__init__()
        self.stem = nn.Sequential(nn.Conv2d(3, 64, 7, 2, 3), nn.BatchNorm2d(64), nn.ReLU(),
                                  Residual(64, 128), nn.MaxPool2d(2),
                                  Residual(128, 128), Residual(128, f))       # 256 -> 64
        self.hgs = nn.ModuleList(Hourglass(4, f) for _ in range(n_stack))     # 64 -> 4 -> 64
        self.feat = nn.ModuleList(nn.Sequential(Residual(f, f), nn.Conv2d(f, f, 1),
                                                nn.BatchNorm2d(f), nn.ReLU()) for _ in range(n_stack))
        self.out = nn.ModuleList(nn.Conv2d(f, n_joints, 1) for _ in range(n_stack))
        self.back_feat = nn.ModuleList(nn.Conv2d(f, f, 1) for _ in range(n_stack - 1))
        self.back_pred = nn.ModuleList(nn.Conv2d(n_joints, f, 1) for _ in range(n_stack - 1))
    def forward(self, x):
        x, outs = self.stem(x), []
        for i in range(len(self.hgs)):
            y = self.feat[i](self.hgs[i](x))
            outs.append(self.out[i](y))                        # one heatmap set per stack
            if i < len(self.hgs) - 1:                          # add predictions back
                x = x + self.back_feat[i](y) + self.back_pred[i](outs[-1])
        return outs

count = lambda m: sum(p.numel() for p in m.parameters())
net = StackedHourglass()
outs = net.eval()(torch.zeros(1, 3, 256, 256))
print("outputs:", len(outs), "x", tuple(outs[0].shape))
print(f"stem {count(net.stem) / 1e6:.2f}M, one hourglass {count(net.hgs[0]) / 1e6:.2f}M, "
      f"per-stack head {(count(net.feat[0]) + count(net.out[0])) / 1e6:.2f}M")
print(f"total, 8 stacks: {count(net) / 1e6:.2f}M parameters")
for n in (1, 2, 4):
    print(f"total, {n} stack(s): {count(StackedHourglass(n_stack=n)) / 1e6:.2f}M")

outputs: 8 x (1, 16, 64, 64)
stem 0.35M, one hourglass 2.79M, per-stack head 0.28M
total, 8 stacks: 25.43M parameters
total, 1 stack(s): 3.43M
total, 2 stack(s): 6.57M
total, 4 stack(s): 12.86M

Almost all of the parameters sit in the hourglasses themselves; the stem and the heads are small, so the size grows nearly linearly with the number of stacks, about 3.1M per added hourglass. The count of the eight-stack network is about 1.3% above the 25.1M that the HRNet paper lists for Newell et al.’s network (Sun et al., 2019, Table 4; its Table 1 gives the same figure for an 8-stage hourglass); the table does not say how it counted, for example whether batch normalization parameters were included. The 13 residual modules of one order-4 hourglass (three per level for four levels, plus the one at the bottom) account for its 2.79M.

Training

Objective

Every stack’s heatmaps are compared with the same target heatmaps by mean squared error, and the losses of all stacks are summed with equal weight: the released code builds a ParallelCriterion of one MSE criterion per stack and repeats the ground-truth heatmaps once per stack (pose.lua). With $S$ stacks, $J$ joints and target maps $\mathbf{y}_j$,

$$\mathcal{L} = \sum_{i=1}^{S} \frac{1}{J H W} \sum_{j=1}^{J} \big\lVert \hat{\mathbf{y}}_{i,j} - \mathbf{y}_j \big\rVert_2^2 ,$$

where $H \times W = 64 \times 64$. Each target is a 2D Gaussian centred on the annotated joint, with standard deviation 1 in heatmap cells (opts.lua: hmGauss 1); joints without an annotation get an empty map. The choice of the spread, and the reading of coordinates off the maps, belong to heatmap regression. The associative embedding paper confirms the scheme for the original model: the same ground truth is enforced on every prediction (Fig. 1 caption).

Supervising intermediate predictions was not new in 2016; its rationale in pose estimation was argued most directly by the convolutional pose machines paper. Wei et al. tracked the distribution of gradient magnitudes at several depths of a three-stage network over the first training epochs and found that without intermediate losses the gradients near the input were tightly peaked around zero, while with them the distribution stayed broad throughout (Fig. 5). On LSP, a three-stage model trained jointly with intermediate supervision outperformed the same model trained without it, and stage-wise training, followed or not by joint fine-tuning (Fig. 6b). In the hourglass network the loss after each hourglass has a second role: because the predicted heatmaps are added back into the features, the intermediate target defines what the re-embedded heatmaps mean to the next hourglass.

Procedure

The defaults in opts.lua and the sampling code in pose.lua describe how the released code trains on MPII:

• Optimizer. RMSprop with learning rate $2.5 \times 10^{-4}$, smoothing constant 0.99 and $\epsilon = 10^{-8}$, no momentum and no weight decay; mini-batches of 6 crops, 8,000 iterations per epoch for 100 epochs. opts.lua also defines a threshold on validation accuracy growth for stopping early, but no other file at this commit reads it.
• Crops. Each sample is cropped around the person’s annotated centre at the annotated scale and resized to $256 \times 256$.
• Augmentation. The scale is multiplied by $2^{s}$ with $s$ drawn from a normal distribution with standard deviation 0.25 and clipped to $\pm 0.5$, a factor between 0.71 and 1.41; a rotation is drawn from a normal distribution with standard deviation 30° and clipped to $\pm 60°$, but set to zero 60% of the time; the crop is flipped horizontally half the time, with left and right joints swapped; and each colour channel is multiplied by a factor drawn uniformly from 0.6 to 1.4. Data augmentation of this kind is standard; the details matter mainly when results are reproduced.
• Validation. By default a fixed held-out set of 2,958 MPII training images, described in the code as the one used by Tompson et al. (2015).

The README notes that the training time reported in the paper was measured with an older cuDNN, that the switch to cuDNN 4 halved it, and that training from scratch then took under 3 days on MPII and about 1 day on FLIC on one NVIDIA Titan X. The official PyTorch reimplementation trained its released models with Adam at learning rate $10^{-3}$ and batch size 16 instead, because they did better on validation, for about 200,000 iterations (README of pytorch_stacked_hourglass, as of November 2026).

Example: Context from the Hourglass

The example asks whether the multi-scale path of one hourglass supplies context that a single-scale network of the same size cannot. Each synthetic $48 \times 48$ image shows a thin segment, 16 to 30 pixels long, with identical $3 \times 3$ bright squares at both ends, and a disc of radius 4 elsewhere. The two keypoints are the “head”, the end nearer the disc centre, and the “tail”. The disc lies at least 10 pixels beyond either end, outside the $13 \times 13$ window that a stack of six $3 \times 3$ convolutions sees, and the two distances differ by at least 6 pixels. Within that window the ends look alike, so a network that sees only locally can find the ends but not tell them apart. A label-free check before training confirmed this: a hand rule that finds the three bright blobs and calls the cap nearer the largest blob the head was right on all of 500 images, while assigning the two caps at random was right on 51%.

Two networks with about 27,000 parameters each are trained on 2,000 images to output two heatmaps at full resolution (Gaussians with $\sigma = 1.5$ pixels, mean squared error, as in the released code’s loss): a single-scale network of six $3 \times 3$ convolution layers with 24 channels, and one hourglass of order 3 with 12 channels ($48 \to 6 \to 48$), built from residual blocks of two $3 \times 3$ convolutions, followed by the $1 \times 1$ convolution, batch normalization and ReLU that hg.lua places before its output layer. Both use Adam at learning rate $10^{-3}$, batch size 32 and 400 steps; Adam in place of the code’s RMSprop was a choice made in the example’s plan before the first run, for speed at this scale. Keypoints are read off by argmax on 500 test images; a prediction is correct within 3 pixels. The script takes the seed as its argument; seeds 0, 1 and 2 were run, and the output below is seed 0. It ran with PyTorch 2.14.1 and NumPy 2.5.4 with four threads and took between about two and four minutes per seed on a laptop CPU, longer than most examples here, because both networks have to be trained from scratch.

import sys, time
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

SEED = int(sys.argv[1]) if len(sys.argv) > 1 else 0
torch.set_num_threads(4)
torch.manual_seed(SEED)
S = 48                                   # image and heatmap size (pixels)

def make_sample(rng):
    """A segment with identical end caps and a disc; the head is the end nearer the disc."""
    while True:
        a = rng.uniform(4, S - 5, 2)
        th = rng.uniform(0, 2 * np.pi)
        L = rng.uniform(16, 30)
        b = a + L * np.array([np.cos(th), np.sin(th)])
        if not np.all((b >= 4) & (b <= S - 5)):
            continue
        a, b = np.round(a), np.round(b)
        d = rng.uniform(6, S - 7, 2)
        da, db = np.linalg.norm(d - a), np.linalg.norm(d - b)
        t = np.clip(np.dot(d - a, b - a) / np.dot(b - a, b - a), 0, 1)
        dseg = np.linalg.norm(d - (a + t * (b - a)))
        if min(da, db) >= 14 and abs(da - db) >= 6 and dseg >= 7:
            break
    img = np.zeros((S, S), np.float32)
    for s in np.linspace(0, 1, int(4 * L) + 1):
        x, y = np.round(a + s * (b - a)).astype(int)
        img[y, x] = 0.6
    for p in (a, b):
        x, y = p.astype(int)
        img[y - 1:y + 2, x - 1:x + 2] = 1.0
    yy, xx = np.mgrid[0:S, 0:S]
    img[(xx - d[0]) ** 2 + (yy - d[1]) ** 2 <= 16] = 1.0
    img += rng.normal(0, 0.05, img.shape).astype(np.float32)
    head, tail = (a, b) if da < db else (b, a)
    return img, np.stack([head, tail])

def make_set(n, seed, sigma=1.5):
    rng = np.random.default_rng(seed)
    imgs, kps = zip(*[make_sample(rng) for _ in range(n)])
    kps = np.stack(kps)                                       # (n, 2, 2) as (x, y)
    yy, xx = np.mgrid[0:S, 0:S]
    hm = np.exp(-((xx - kps[..., 0, None, None]) ** 2 + (yy - kps[..., 1, None, None]) ** 2)
                / (2 * sigma ** 2))
    return (torch.tensor(np.stack(imgs))[:, None], torch.tensor(hm, dtype=torch.float32),
            torch.tensor(kps, dtype=torch.float32))

def cbr(i, o):
    return nn.Sequential(nn.Conv2d(i, o, 3, padding=1), nn.BatchNorm2d(o), nn.ReLU())

class Residual(nn.Module):
    def __init__(self, c):
        super().__init__()
        self.f = nn.Sequential(cbr(c, c), nn.Conv2d(c, c, 3, padding=1), nn.BatchNorm2d(c))
    def forward(self, x):
        return F.relu(x + self.f(x))

class Hourglass(nn.Module):
    """Recursive module as in hg.lua: upper branch at this resolution, lower branch pooled."""
    def __init__(self, n, c):
        super().__init__()
        self.up1, self.low1, self.low3 = Residual(c), Residual(c), Residual(c)
        self.low2 = Hourglass(n - 1, c) if n > 1 else Residual(c)
    def forward(self, x):
        low = self.low3(self.low2(self.low1(F.max_pool2d(x, 2))))
        return self.up1(x) + F.interpolate(low, scale_factor=2, mode="nearest")

models = {
    "single-scale CNN": nn.Sequential(cbr(1, 24), *[cbr(24, 24) for _ in range(5)],
                                      nn.Conv2d(24, 2, 1)),        # receptive field 13 x 13
    "hourglass": nn.Sequential(cbr(1, 12), Hourglass(3, 12),          # 48 -> 6 -> 48
                               nn.Conv2d(12, 12, 1), nn.BatchNorm2d(12), nn.ReLU(),  # hg.lua's lin()
                               nn.Conv2d(12, 2, 1)),
}

def macs(model):
    total = []
    hook = lambda m, i, o: total.append(o[0].numel() * m.in_channels * m.kernel_size[0] ** 2)
    hs = [m.register_forward_hook(hook) for m in model.modules() if isinstance(m, nn.Conv2d)]
    model.eval(); model(torch.zeros(1, 1, S, S)); [h.remove() for h in hs]
    return sum(total)

x_tr, y_tr, _ = make_set(2000, seed=100 + SEED)
x_te, _, k_te = make_set(500, seed=999)

def decode(hm):                                   # argmax per channel -> (x, y)
    idx = hm.flatten(2).argmax(-1)
    return torch.stack([idx % S, idx // S], -1).float()

t0 = time.time()
for name, model in models.items():
    opt = torch.optim.Adam(model.parameters(), lr=1e-3)
    g = torch.Generator().manual_seed(SEED)
    model.train()
    for step in range(400):
        idx = torch.randint(0, len(x_tr), (32,), generator=g)
        loss = F.mse_loss(model(x_tr[idx]), y_tr[idx])
        opt.zero_grad(); loss.backward(); opt.step()
    model.eval()
    with torch.no_grad():
        p = decode(model(x_te))                                    # (500, 2, 2)
    d = lambda i, j: (p[:, i] - k_te[:, j]).norm(dim=-1) <= 3      # pred i near truth j
    print(f"{name:17s} params {sum(q.numel() for q in model.parameters()):6d}  "
          f"MACs {macs(model) / 1e6:5.1f}M  final loss {loss.item():.5f}")
    print(f"   head correct {d(0, 0).float().mean():.3f}  tail correct {d(1, 1).float().mean():.3f}"
          f"  swapped {((d(0, 1).float() + d(1, 0).float()) / 2).mean():.3f}"
          f"  on either end {(((d(0, 0) | d(0, 1)).float() + (d(1, 1) | d(1, 0)).float()) / 2).mean():.3f}")
print(f"seed {SEED}, {time.time() - t0:.0f} s")

single-scale CNN  params  26618  MACs  60.3M  final loss 0.00158
   head correct 0.498  tail correct 0.472  swapped 0.515  on either end 1.000
hourglass         params  26990  MACs  12.5M  final loss 0.00190
   head correct 0.722  tail correct 0.696  swapped 0.291  on either end 1.000
seed 0, 232 s

The last column separates finding an end from naming it. Both networks put every prediction on one of the two ends, so both learned to localize the caps to within 3 pixels. The single-scale network then names them at chance: head and tail are each right about half the time and swapped the other half, as expected from a model that cannot see the disc. The hourglass, with the same number of parameters and a fifth of the multiply–accumulates, names the ends correctly about 70% of the time and swaps them in under 30%. Its advantage can only come from the pooled path, since that is the only part of the network that sees both the caps and the disc.

The other two seeds show how much the hourglass’s result depends on the run. With seed 1 it was right on 75.2% of heads and 74.0% of tails, with 25.4% swapped; with seed 2 on 97.2% and 98.0%, with 2.4% swapped, and a final training loss below the single-scale network’s. The single-scale network stayed at chance with every seed, between 46% and 51% head and tail accuracy and 50% to 52% swaps. Within 400 steps, then, the hourglass always learned some of the context rule and in one run of three learned nearly all of it; our reading, not a measured result, is that the slower runs were still learning it. The plan for this example expected the hourglass’s head and tail accuracy to be “well above the flat net (> 80%)”; two of the three seeds fell short of 80%. The example says nothing about intermediate supervision, which a single hourglass does not test.

Two changes were made after the first run. The first version of the hourglass, as the plan specified it (“then 1×1 conv to 2 heatmaps”), fed the sum of its branches straight into the output convolution, without the $1 \times 1$ convolution, batch normalization and ReLU that hg.lua applies first (its lin function); the single-scale network already ended in a normalized layer. A diagnostic run that printed the training loss every 25 steps showed that this hourglass started with outputs far larger than the targets and had not converged by step 400. The missing layer was added, as in hg.lua, which raised the hourglass’s parameter count from 26,810 to 26,990; the data, schedule, optimizer, metrics and seeds were left unchanged. The plan named seeds 0 and 1; seed 2 was run afterwards, with the final script, and every seed that was run is reported here.

Inference

At test time the network receives a crop around the person at the annotated centre and scale, as in training; on MPII these come with the single-person evaluation, so the model is not a person detector. The released code runs each crop and its horizontal mirror, flips the mirror’s heatmaps back and swaps left and right joints, and averages the two (train.lua). Only the last hourglass’s heatmaps are used (pose.lua). Each joint is placed at its heatmap maximum, moved a quarter of a cell toward the higher neighbour, and mapped back from the $64 \times 64$ grid to the image through the crop transform; the quarter shift and why it helps are explained in heatmap regression. The HRNet paper lists the eight-stack network at 19.1 GFLOPs for a $256 \times 256$ input (Sun et al., 2019, Table 4).

Performance

MPII single person. The MPII results page lists the stacked hourglass network (Newell et al., ECCV’16) at 90.9% PCKh@0.5 on the test set, with 98.2% for the head, 96.3% shoulders, 91.2% elbows, 87.1% wrists, 90.1% hips, 87.4% knees and 83.6% ankles; HRNet’s Table 3 gives the same values. On the same page, convolutional pose machines, trained with LSP and LSP Extended added, are listed at 88.5%, and Bulat and Tzimiropoulos (ECCV’16) at 89.7%. The MPII Human Pose article describes the benchmark, its PCKh metric and the history of the leaderboard.

Held-out validation. The README of the official PyTorch reimplementation states that the original paper reported a validation accuracy of 0.881 on the code’s held-out split, that the reimplementation’s eight-hourglass model reaches 0.902 and its two-hourglass model 0.883, and that both were trained longer than the original (as of November 2026). These are validation numbers on the authors’ own split and are not comparable with test-set results.

FLIC. The released code also trains on FLIC (README), but the paper’s FLIC results could not be checked against a readable source for this article.

COCO keypoints. The stacked hourglass network was designed for single-person crops; on COCO it was evaluated by others as a top-down model on detected people. SimpleBaseline (Table 3) and HRNet (Table 1) list an eight-stage hourglass at 66.9 AP on COCO val2017 with $256 \times 192$ inputs, and SimpleBaseline also at 67.1 with $256 \times 256$, results that SimpleBaseline states it cites from the Cascaded Pyramid Network paper (Chen et al., 2018), which was not read for this article; HRNet’s table does not name a source. SimpleBaseline gives the person detection AP behind these numbers as 55.3. HRNet lists this model at 25.1M parameters and 14.3 GFLOPs at $256 \times 192$, against 73.4 AP for HRNet-W32 at a similar parameter count, both listed without ImageNet pretraining (Table 1). The SimpleBaseline authors note that these hourglass results were not produced by them, so the gap may partly reflect implementation.

Strengths

• Global and local evidence in one map. Every output cell combines features that have seen the whole crop with features from its own neighbourhood, so a joint whose identity depends on the rest of the body, such as the left or right of a symmetric pair, can be told apart; the example above shows the effect on a toy task.
• Iterative refinement. Each hourglass can revise the previous heatmaps with the full context, and intermediate supervision gives every stage a direct training signal.
• Simple, uniform design. One residual module and one recursive function make up the whole network; depth is changed by the number of stacks, and the parameter count grows almost linearly with it.
• A general dense predictor. The same network outputs any set of per-pixel maps, which made it easy to reuse for tags, corners and centres.

Limitations

• Single-person crops. The model needs a person’s centre and scale. Multi-person use requires a detector in front (top-down) or a grouping mechanism such as associative embedding.
• Quarter-resolution output. Heatmaps at $64 \times 64$ quantize positions to 4 input pixels per cell, which decoding tricks only partly recover (heatmap regression). In the associative embedding work, refining the multi-person predictions with a single-person hourglass helped most at a single test scale, and the authors’ text says most on small figures, which they read as output resolution limiting performance; their COCO table, which has no small category, shows the larger gain for medium-sized people (AP 0.498 to 0.575) than for large ones (0.670 to 0.706) (NIPS 2017, Table 5).
• Cost. Repeating full-resolution hourglasses is expensive: 19.1 GFLOPs at $256 \times 256$, about twice HRNet-W32’s 9.5 GFLOPs for similar parameters and higher accuracy on MPII (Sun et al., 2019, Table 4). As a detection backbone, Hourglass-104 needed 5 days of training in CenterNet, even when fine-tuned from ExtremeNet, against 2.5 days on 8 TITAN V GPUs for DLA-34 and ResNet-101 (Zhou et al., 2019, §5).
• Resolution is recovered, not kept. HRNet’s authors grouped the hourglass with designs that encode to low resolution and decode back, and argued for keeping a high-resolution branch throughout instead (Sun et al., 2019, §1–2).

Historical Importance

The stacked hourglass network and convolutional pose machines both appeared in 2016, and both trained a multi-stage heatmap network with a loss after every stage. They differ in how they reach context. A convolutional pose machine re-reads image features at every stage, shares the image-feature weights across stages after the first, and grows its receptive field with stacks of large kernels on a stride-8 map, $368 \times 368$ inputs and a background belief map (Wei et al., 2016, §3–4). The hourglass network re-processes its own features, shares nothing between stacks, reaches context by pooling to $4 \times 4$, and works on a stride-4 map with residual modules. Wei et al. themselves noted that residual units were complementary to their design and could replace each stage (§1, footnote 2).

In 2018 the SimpleBaseline authors described it as the dominant approach on MPII and the basis for all of its leading methods (Xiao, Wu and Wei, 2018, §1); HRNet’s authors describe the methods of Sun et al.’s global and local normalization (ICCV 2017), Chu et al. (CVPR 2017), Yang et al. (ICCV 2017), Ke et al. (2018) and Tang, Yu and Wu (ECCV 2018) as extensions of the stacked hourglass, and these make up five of the sixteen rows of their MPII comparison table (Sun et al., 2019, §4 and Table 3). Its second life was in object detection, where it became the backbone of the keypoint-based detectors described in anchor-free object detection.

Variants

• Multi-context attention (Chu et al., CVPR 2017). An eight-stack hourglass baseline with attention maps generated at several resolutions inside each hourglass, conditional random fields over the attention, and “hourglass residual units” that add a small hourglass branch inside each residual module, giving a nested hourglass. It scored 91.5% PCKh@0.5 on the MPII test set (MPII results page; Sun et al., 2019, Table 3).
• Associative embedding (Newell, Huang and Deng, NIPS 2017). A bottom-up multi-person model that outputs, besides each joint’s detection heatmap, a tag heatmap per joint; joints whose tags agree are grouped into one person. Its grouping loss and decoding are described in human pose estimation. The network was a four-hourglass model with $512 \times 512$ inputs and $128 \times 128$ outputs, with the features widened at each drop in resolution (256, 384, 512, 640, 768) and $3 \times 3$ convolutions in place of residual modules (§3.1, §4). It reached 77.5 AP on MPII Multi-Person (Table 1) and 65.5 AP on COCO test-dev with multi-scale testing and refinement by a single-person hourglass (Tables 3 and 5).
• Hourglass-104 in CornerNet (Law and Deng, ECCV 2018). Two hourglasses with strided convolutions instead of max-pooling, five reductions of resolution with 256, 384, 384, 384 and 512 channels, two residual modules before each nearest-neighbour upsampling and in each skip connection, and four 512-channel residual modules at the bottom, after a stem that reduces the image four times; the network is 104 layers deep (§3.5). Intermediate supervision is kept, but the intermediate predictions are not added back, because the authors found that this hurt; instead the first hourglass’s input and output, each through a $3 \times 3$ convolution with batch normalization, are added and passed on. With $511 \times 511$ inputs and $128 \times 128$ outputs, CornerNet reached 40.5 AP on COCO test-dev single-scale and 42.1 multi-scale (Table 4).
• Hourglass-104 in CenterNet (Zhou, Wang and Krähenbühl, 2019). The “Objects as Points” detector, one of two 2019 detectors named CenterNet, used CornerNet’s Hourglass-104 unchanged as its most accurate backbone and fine-tuned it from an ExtremeNet model for detection. It reached 42.2 AP on COCO validation with flip testing at 7.8 FPS and 45.1 with multi-scale testing (Table 1); see CenterNet for the detector.

Implementations

The official training code, princeton-vl/pose-hg-train, is written in Lua for Torch7 with cuDNN and trains on MPII and FLIC; its last commit is from January 2017, and its licence is a BSD-style licence held by the University of Michigan, with parts built on Soumith Chintala’s ImageNet training code (LICENSE, README; as of November 2026). The README points to a pretrained model on the project’s University of Michigan page. The official PyTorch reimplementation, princeton-vl/pytorch_stacked_hourglass, by Chris Rockwell and adapted from the associative embedding code, trains and evaluates single-person models on MPII, provides pretrained two- and eight-hourglass models, and is released under the BSD 3-Clause licence (as of November 2026). CornerNet’s and CenterNet’s official repositories contain the Hourglass-104 variant.