CenterNet

CenterNet, in the form introduced by Xingyi Zhou, Dequan Wang and Philipp Krähenbühl in the 2019 report “Objects as Points”, is a one-stage, anchor-free object detector that represents each object by a single point, the centre of its bounding box. A fully convolutional encoder–decoder network predicts, at a quarter of the input resolution, one heatmap of box centres per class, together with the box width and height and a sub-cell offset; at test time the local maxima of the heatmaps, found with a $3 \times 3$ max-pooling, become the detections, and no box-based non-maximum suppression (NMS) is applied. The same network regresses other properties at the centre point: depth, 3D dimensions and orientation for 3D boxes, and joint offsets for multi-person human pose. The paper exists as an arXiv technical report (v2 of 25 April 2019 is the version read here), and its official code is released under the MIT license.

Another detector of the same year is also called CenterNet: “CenterNet: Keypoint Triplets for Object Detection” by Duan, Bai, Xie, Qi, Huang and Tian (ICCV 2019). It extends CornerNet’s corner pairs with a third, central keypoint and is a different method; it is described under Variants. In this article, CenterNet without qualification means “Objects as Points”.

Problem Addressed

Zhou et al. described the detectors of 2019 as enumerating a nearly exhaustive list of candidate boxes, as anchors or as proposals, classifying each, and then removing duplicates with IoU-based NMS, a post-processing step they called hard to differentiate and train. CornerNet (Law and Deng, 2018) and ExtremeNet had already turned detection into keypoint estimation, with corners or extreme points found as peaks in heatmaps, but both had to group several keypoints into each object, a combinatorial step that, the authors wrote, slows them down considerably. CenterNet keeps the keypoint machinery and drops the grouping: an object is one keypoint, and everything else about it is read off at that point.

The authors related the design to anchor-based one-stage detectors: a centre point can be seen as a single anchor with no shape, assigned by location alone rather than by box overlap, with no foreground and background IoU thresholds and exactly one positive per object. Because each object has one positive, duplicates are rare enough that peak extraction replaces NMS, and a fine output stride of 4 (against 16 in the detectors they compare with) is meant to make several anchors per location unnecessary. The anchor-free object detection article places this rule beside those of FCOS and FoveaBox, and label assignment treats one-positive rules in general.

Architecture

One backbone produces a single feature map at output stride $R = 4$ , and small heads predict $C + 4$ channels at every location: $C$ class heatmaps $\hat Y \in [0, 1]^{\frac{W}{R} \times \frac{H}{R} \times C}$ (80 for COCO), a two-channel size map $\hat S$ and a two-channel offset map $\hat O$ . Each head is a $3 \times 3$ convolution, a ReLU and a $1 \times 1$ convolution on the shared features; the paper uses 256 channels in the $3 \times 3$ layer, and the released code sets this width to 256 for DLA and 64 for the other backbones by default (opts.py). There is no feature pyramid: every object, large or small, is detected on the one stride-4 map.

Backbones

The paper tests four encoder–decoder networks, all producing a $128 \times 128$ output for a $512 \times 512$ input:

Hourglass-104, the network that CornerNet adapted from the stacked hourglass design for pose estimation: a $4\times$ downsampling stem followed by two hourglass modules, each a symmetric five-level down- and up-sampling network with skip connections. It is used unchanged from CornerNet and, for detection, fine-tuned from an ExtremeNet model to save computation.
ResNet-18 and ResNet-101 with three up-convolution stages, following the SimpleBaseline pose network of Xiao et al. The authors reduced the up-sampling channels to 256, 128 and 64 and added a $3 \times 3$ deformable convolution before each transposed convolution, whose kernels are initialized as bilinear interpolation.
DLA-34, Deep Layer Aggregation in its up-sampling form for dense prediction, with extra skip connections from the lower layers and every convolution in the up-sampling stages replaced by a $3 \times 3$ deformable convolution.

The ResNet and DLA encoders start from ImageNet weights; their up-sampling layers start from random weights. The released code takes the deformable convolutions from the DCNv2 project and the hourglass network from CornerNet’s code (README, License section).

Detection Heads

For an object $k$ of class $c_k$ with box $(x_1^{(k)}, y_1^{(k)}, x_2^{(k)}, y_2^{(k)})$ , the centre is $p_k = \left(\frac{x_1^{(k)} + x_2^{(k)}}{2}, \frac{y_1^{(k)} + y_2^{(k)}}{2}\right)$ and the size $s_k = (x_2^{(k)} - x_1^{(k)}, y_2^{(k)} - y_1^{(k)})$ , in input pixels. The heatmap channel $c_k$ should peak at the low-resolution centre $\tilde p_k = \lfloor p_k / R \rfloor$ , the size map should hold $s_k$ there, and the offset map should hold the discretization error $p_k / R - \tilde p_k$ , which the stride-4 grid would otherwise lose. The size and offset heads are shared by all classes (for the size head, “to limit the computational burden”); the code also offers a class-specific size head (--cat_spec_wh), which the paper does not evaluate. Size is regressed without normalization. The paper defines $s_k$ from the box in image coordinates and speaks of raw pixel coordinates, but its discussion of the size weight gives the target’s range as 0 to $w/R$ or $h/R$ , and the released code computes the size after mapping the box onto the stride-4 grid (ctdet.py); in that discussion and in the code, the units are pixels of the output map, so a 100-pixel-wide object has a width target of 25.

Heads for 3D Detection

For monocular 3D detection the paper adds three heads at the centre point. Depth $d$ is predicted through the output transform $d = 1/\sigma(\hat d) - 1$ , with $\sigma$ the sigmoid, a transform the authors take from Eigen et al.’s depth estimation; the 3D dimensions (height, width, length in metres) are regressed directly; and the orientation is encoded with eight scalars, following Mousavian et al.’s Deep3DBox: two overlapping angular bins, $[-\tfrac{7\pi}{6}, \tfrac{\pi}{6}]$ and $[-\tfrac{\pi}{6}, \tfrac{7\pi}{6}]$ , each with two scores for a softmax classification and the sine and cosine of the in-bin offset from the bin centre (supplementary Appendix B). The orientation is decoded as $\hat\theta = \operatorname{arctan2}(\hat a_{j1}, \hat a_{j2}) + m_j$ in the bin $j$ with the higher score. The 2D box heads stay as they are.

Heads for Human Pose

For multi-person pose, each person is detected as a centre point (one class, $C = 1$ ), and the $k$ joints (17 on COCO) become a $2k$ -dimensional property of that point: a head $\hat J$ regresses each joint’s offset from the centre in pixels. This alone is, in the authors’ words, a regression-based one-stage pose estimator, a counterpart of the two-stage regression methods of Toshev et al. (DeepPose) and Sun et al. To refine it, a second set of $k$ joint heatmaps $\hat\Phi$ with its own offset head is trained as in bottom-up human pose estimation, and at test time each regressed joint $l_j = (\hat x, \hat y) + \hat J_{\hat x \hat y j}$ snaps to the nearest heatmap detection of the same joint type with confidence above 0.1, considering only detections inside the person’s box. The centre offset therefore serves as the grouping cue that bottom-up methods otherwise learn separately.

Training

Centre Heatmaps and the Penalty-Reduced Focal Loss

The heatmap target follows CornerNet. Each ground-truth centre is splatted onto its class channel as an unnormalized Gaussian,

Y_{xyc} = \exp\left( -\frac{(x - \tilde p_x)^2 + (y - \tilde p_y)^2}{2\sigma_p^2} \right),

with a standard deviation $\sigma_p$ that adapts to object size, and where two Gaussians of the same class overlap the element-wise maximum is kept. The paper names $\sigma_p$ only as size-adaptive and cites CornerNet; in the released code, the radius $r$ comes from CornerNet’s routine for the radius within which a displaced box keeps IoU of at least 0.7 with the true one, and the Gaussian, drawn only within that radius, has $\sigma_p = (2r + 1)/6$ , about a third of the radius as in CornerNet (gaussian_radius and draw_umich_gaussian in utils/image.py; ctdet.py). Gaussian targets, their resolution and their decoding in general are the subject of heatmap regression.

Only one cell per object has the value 1, and the training objective treats every other cell as a negative whose penalty shrinks near a centre. It is a pixel-wise logistic regression with a variant of the focal loss of Lin et al. (2017), as in CornerNet:

L_k = -\frac{1}{N} \sum_{xyc} \begin{cases} (1 - \hat Y_{xyc})^{\alpha} \log \hat Y_{xyc} & \text{if } Y_{xyc} = 1, \\ (1 - Y_{xyc})^{\beta} \, \hat Y_{xyc}^{\alpha} \log (1 - \hat Y_{xyc}) & \text{otherwise,} \end{cases}

with $N$ the number of keypoints in the image, so that the positive terms are normalized to one per object, and $\alpha = 2$ , $\beta = 4$ in all experiments. The factor $\hat Y^{\alpha}$ is the focal term: confident negatives contribute little, which counters the imbalance between the $128 \times 128 = 16{,}384$ cells of each class map and the handful of centres in an image. The factor $(1 - Y)^{\beta}$ is the penalty reduction: a cell next to a centre, where the target Gaussian is close to 1, costs almost nothing when the network fires there, while a cell far from every object costs the full focal amount. The released loss is the one from CornerNet’s code (_neg_loss in losses.py). In CornerNet’s ablation on its validation split, removing the penalty reduction lowered AP from 38.5 to 32.9 (Law and Deng, 2018); the anchor-free object detection article gives this ablation together with CornerNet’s radius rule.

Size and Offset Losses

The size and offset heads are trained with L1 losses at the ground-truth centres only; every other location is ignored (the paper writes the index of $\hat S$ in Eq. 3 as $p_k$ ):

L_{\text{size}} = \frac{1}{N} \sum_{k=1}^{N} \left| \hat S_{\tilde p_k} - s_k \right|, \qquad L_{\text{off}} = \frac{1}{N} \sum_{p} \left| \hat O_{\tilde p} - \left( \frac{p}{R} - \tilde p \right) \right|.

The total objective is

L_{\text{det}} = L_k + \lambda_{\text{size}} L_{\text{size}} + \lambda_{\text{off}} L_{\text{off}}, \qquad \lambda_{\text{size}} = 0.1,\ \lambda_{\text{off}} = 1 .

The small size weight compensates for the unnormalized targets, which, as the authors put it, range up to the output width or height rather than up to 1. In the paper’s ablations (COCO validation, DLA-34, 140 epochs; Table 3), $\lambda_{\text{size}} = 0.2$ gave 33.5 AP, 0.1 gave 36.3 and 0.02 gave 35.4, and a smooth L1 size loss gave 33.9 against 36.3 for L1, which the authors attribute to L1’s better accuracy at fine scales. The 3D heads and the pose joint offsets add L1 losses of the same form (softmax plus L1 for orientation), the joint heatmaps a focal loss with its own offsets, all with weights of 1. How these losses compare with IoU-based box losses is discussed in bounding box regression.

Procedure

COCO models train on $512 \times 512$ crops with random flipping, random scaling between 0.6 and 1.3, cropping and colour jittering, using Adam. The ResNets and DLA-34 train with batch size 128 on 8 GPUs at learning rate $5 \times 10^{-4}$ for 140 epochs, with the rate divided by 10 at epochs 90 and 120; Hourglass-104 follows ExtremeNet, with batch size 29 on 5 GPUs at $2.5 \times 10^{-4}$ for 50 epochs and a drop at epoch 40. The authors report 2.5 days on 8 TITAN V GPUs for ResNet-101 and DLA-34 and 5 days for Hourglass-104. A longer schedule of 230 epochs raised DLA-34 from 36.3 to 37.4 AP on COCO validation (Table 3d); the authors used 140 epochs for ablations and 230 epochs for DLA-34 in their comparisons with other detectors. The model zoo calls the two schedules 1x and 2x, with the 2x drops at epochs 180 and 210, and notes up to 0.4 AP of variation from training randomness (MODEL_ZOO.md). The 3D branch is trained without augmentation, since cropping or scaling changes the 3D measurements.

Example: The Cost of a Near Miss

The first part of the example computes $L_k$ for one object, a $96 \times 64$ -pixel box at stride 4, when the predicted peak lands on the true centre, one cell off, or four cells off, with the paper’s $\beta = 4$ and with $\beta = 0$ , which removes the penalty reduction and leaves a focal loss against a one-hot target. The target is drawn with the radius rule and $\sigma_p$ of the released code; the prediction is a blob of the same shape with peak 0.9 on a background of 0.01. It runs with NumPy 2.3.5 and SciPy 1.16.3; the second part, below, continues the same script.

import numpy as np
from scipy.ndimage import maximum_filter

R = 4                                   # output stride

def gaussian_radius(h, w, min_overlap=0.7):
    """CornerNet's radius rule as in the official CenterNet code (utils/image.py)."""
    b1, c1 = h + w, w * h * (1 - min_overlap) / (1 + min_overlap)
    r1 = (b1 + np.sqrt(b1 ** 2 - 4 * c1)) / 2
    b2, c2 = 2 * (h + w), (1 - min_overlap) * w * h
    r2 = (b2 + np.sqrt(b2 ** 2 - 16 * c2)) / 2
    a3, b3, c3 = 4 * min_overlap, -2 * min_overlap * (h + w), (min_overlap - 1) * w * h
    r3 = (b3 + np.sqrt(b3 ** 2 - 4 * a3 * c3)) / 2
    return min(r1, r2, r3)

def gaussian_at(shape, cx, cy, h, w):
    """Unnormalized Gaussian at integer cell (cx, cy); sigma = (2r + 1) / 6 as in the code."""
    r = max(0, int(gaussian_radius(np.ceil(h), np.ceil(w))))
    sigma = (2 * r + 1) / 6
    yy, xx = np.mgrid[0:shape[0], 0:shape[1]]
    g = np.exp(-((xx - cx) ** 2 + (yy - cy) ** 2) / (2 * sigma ** 2))
    g[(np.abs(xx - cx) > r) | (np.abs(yy - cy) > r)] = 0      # drawn only inside the radius
    return g

def focal_loss(pred, target, alpha=2, beta=4):
    """Penalty-reduced pixel-wise focal loss, Eq. (1) of Objects as Points."""
    pos = target == 1
    pos_loss = ((1 - pred) ** alpha * np.log(pred))[pos].sum()
    neg_loss = ((1 - target) ** beta * pred ** alpha * np.log(1 - pred))[~pos].sum()
    return -(pos_loss + neg_loss) / pos.sum()

# --- Example 1: the cost of a near miss --------------------------------------
H = W = 64
box_h, box_w = 64 / R, 96 / R                    # a 96 x 64 px box, in output cells
target = gaussian_at((H, W), 32, 32, box_h, box_w)
r = int(gaussian_radius(np.ceil(box_h), np.ceil(box_w)))
print(f"radius {r} cells, sigma {(2 * r + 1) / 6:.2f} cells")
for shift in (0, 1, 4):
    blob = gaussian_at((H, W), 32 + shift, 32, box_h, box_w)
    pred = np.clip(0.01 + 0.89 * blob, 1e-4, 1 - 1e-4)        # peak 0.9, background 0.01
    print(f"peak {shift} cell(s) off:  beta=4 loss {focal_loss(pred, target):6.3f}   "
          f"beta=0 loss {focal_loss(pred, target, beta=0):6.3f}")

radius 5 cells, sigma 1.83 cells
peak 0 cell(s) off:  beta=4 loss  0.203   beta=0 loss  7.467
peak 1 cell(s) off:  beta=4 loss  0.462   beta=0 loss  8.438
peak 4 cell(s) off:  beta=4 loss  8.099   beta=0 loss 11.293

With the penalty reduction, a peak one cell from the centre a little more than doubles the loss of a perfect one, while a peak four cells away, near the edge of the 5-cell radius, multiplies it by about forty. Without it, even the perfect prediction costs about 7.5, because its own blob spills onto the cells around the centre and every one of them counts as a full negative; the one-cell miss then adds only about 13% more. The penalty reduction is what lets the loss tell a near miss from a far one, and, in our reading, what keeps a network from being pushed toward a single, isolated spike.

Inference

Peak Extraction Instead of NMS

At test time the heatmaps are searched for peaks in each class independently: a response is kept if it is greater than or equal to its eight neighbours, which a $3 \times 3$ max-pooling with stride 1 finds in one operation, and the top 100 are kept. Each peak at integer location $(\hat x_i, \hat y_i)$ with score $\hat Y_{\hat x_i \hat y_i c}$ gives the box

\left( \hat x_i + \delta\hat x_i - \frac{\hat w_i}{2},\; \hat y_i + \delta\hat y_i - \frac{\hat h_i}{2},\; \hat x_i + \delta\hat x_i + \frac{\hat w_i}{2},\; \hat y_i + \delta\hat y_i + \frac{\hat h_i}{2} \right),

with $(\delta\hat x_i, \delta\hat y_i) = \hat O_{\hat x_i, \hat y_i}$ and $(\hat w_i, \hat h_i) = \hat S_{\hat x_i, \hat y_i}$ , read at the peak. In the released code the peak test is hmax == heat after max_pool2d with kernel 3 and padding 1, so ties are all kept; the top 100 are taken per class and then the top 100 over all classes; and boxes are computed on the output grid (_nms, _topk and ctdet_decode in decode.py) and mapped back to the image afterwards (post_process in detectors/ctdet.py). The authors call the peak extraction a sufficient alternative to NMS that can run efficiently on devices.

The claim was tested directly: running IoU-based NMS on top of the peaks moved DLA-34 with flip testing from 39.2 to 39.7 AP on COCO validation and left Hourglass-104 at 42.2, and the authors left NMS out. The reason, in their account, is that each object has one positive cell in training, so a trained network rarely produces two separated peaks for one object. Peak extraction is not suppression by overlap: two different objects whose centres lie at least two cells apart keep their peaks however much their boxes overlap, as long as each centre stays a local maximum. The non-maximum suppression article notes that this is close to the older, pixel-wise sense of the term.

Test-Time Augmentation and Speed

The paper reports three test modes: none, flip (the network outputs for the image and its mirror are averaged before decoding; the released code averages the heatmaps and sizes and takes the offsets from the unflipped image), and flip plus five scales (0.5, 0.75, 1, 1.25 and 1.5), whose detections are merged with NMS. In the released code that merge is Gaussian Soft-NMS (soft_nms with method 2 in detectors/ctdet.py, external/nms.pyx), which also runs when the --nms flag is set. Images are tested at their original resolution, zero-padded to a multiple of the network’s maximum stride (32 pixels for ResNet and DLA, 128 for the hourglass). In Table 3a, testing at the original resolution gave 36.3 AP against 36.2 for a fixed $512 \times 512$ , and training and testing at $384 \times 384$ ran 1.7 times faster but lost 3 AP. Speeds in the paper were measured on one machine with a Titan Xp GPU, PyTorch 0.4.1 and CUDA 9.0, and include decoding.

Example: Decoding Overlapping Objects

The second part of the script decodes 50 synthetic $512 \times 512$ scenes with two classes. Each scene holds three pairs of same-class boxes whose IoU lies between 0.5 and 0.8, the situation of people in a crowd, and four boxes placed at random. The heatmaps are CenterNet’s targets on the $128 \times 128$ output grid. In place of a trained network’s size and offset heads, every cell reports the sub-cell offset of the object whose Gaussian is highest there, and a size for that object. Four decoders are compared: $3 \times 3$ peaks, peaks followed by IoU-NMS at 0.5 per class, every cell above threshold followed by NMS (the decoding of a dense detector), and every cell above threshold with no suppression. All use a score threshold of 0.3, the default of the released code for displaying detections (--vis_thresh). Detections are matched to objects of the same class at IoU 0.5 in score order; a detection that overlaps an already matched object is counted as a duplicate.

What a trained network’s heatmap looks like is a modelling choice the sources leave open, so the script prints two: the target itself, as in the paper’s error analysis that replaces the predicted heatmap with the ground truth, and a degraded version in which each object’s Gaussian is scaled to a random peak between 0.5 and 0.95 and independent Gaussian noise of standard deviation 0.05 is added. What the size head reports away from the centres is also left open, since CenterNet supervises size only at centres (the released code also offers --dense_wh, a weighted size regression near the centre), so the script prints two stand-ins for it as well: the object’s exact size at every cell, and a size that is exact only at the object’s centre cell and elsewhere has its width and height each multiplied by an independent log-normal factor $e^{\epsilon}$ , $\epsilon \sim \mathcal N(0, 0.25^2)$ . The plan written before the first run expected exact recovery from peaks on the target, duplicates from noise maxima on the degraded maps, and losses in overlapping pairs whenever NMS is applied. That run had only the exact size stand-in; in fact-checking, the second stand-in was added, with its spread fixed before it was run, because the dense-decoding results depend on it, and the first run’s numbers are unchanged in the output. The script takes about 2 seconds.

# --- Example 2: decoding without NMS ------------------------------------------
def iou(a, b):
    ix = max(0, min(a[2], b[2]) - max(a[0], b[0]))
    iy = max(0, min(a[3], b[3]) - max(a[1], b[1]))
    inter = ix * iy
    return inter / ((a[2] - a[0]) * (a[3] - a[1]) + (b[2] - b[0]) * (b[3] - b[1]) - inter)

def make_scene(rng, n_pairs=3, n_random=4, size=512):
    boxes, cls, pair = [], [], []
    for p in range(n_pairs):                      # same-class pairs with IoU 0.5-0.8
        c = int(rng.integers(2))
        while True:
            w, h = rng.uniform(40, 160, 2)
            x0, y0 = rng.uniform(0, size - w - 60), rng.uniform(0, size - h - 60)
            a = (x0, y0, x0 + w, y0 + h)
            dx, dy = rng.uniform(-0.3, 0.3, 2) * (w, h)
            s = rng.uniform(0.85, 1.15)
            bx0, by0 = np.clip((x0 + dx, y0 + dy), 0, None)
            b = (bx0, by0, min(size, bx0 + w * s), min(size, by0 + h * s))
            if 0.5 <= iou(a, b) <= 0.8:
                break
        boxes += [a, b]; cls += [c, c]; pair += [p, p]
    for _ in range(n_random):
        w, h = rng.uniform(30, 200, 2)
        x0, y0 = rng.uniform(0, size - w), rng.uniform(0, size - h)
        boxes.append((x0, y0, x0 + w, y0 + h)); cls.append(int(rng.integers(2))); pair.append(-1)
    return np.array(boxes), np.array(cls), np.array(pair)

def render(boxes, cls, shape, gains=None):
    """Heatmap (element-wise max per class); size and offset maps of the dominant object."""
    hm = np.zeros((2,) + shape)
    best = np.zeros((2,) + shape); owner = -np.ones((2,) + shape, int)
    for k, (b, c) in enumerate(zip(boxes, cls)):
        ct = np.array([(b[0] + b[2]) / 2, (b[1] + b[3]) / 2]) / R
        ci = ct.astype(int)
        g = gaussian_at(shape, ci[0], ci[1], (b[3] - b[1]) / R, (b[2] - b[0]) / R)
        take = g > best[c]
        best[c][take] = g[take]; owner[c][take] = k
        hm[c] = np.maximum(hm[c], g * (1 if gains is None else gains[k]))
    return hm, owner

def boxes_from_cells(cells, boxes, owner, log_noise=None):
    """Decode (c, y, x, score) cells with the owning object's size and sub-cell offset.
    With log_noise, the size is exact only at the object's centre cell."""
    out = []
    for c, y, x, s in cells:
        k = owner[c, y, x]
        if k < 0:
            continue
        b = boxes[k]; ct = np.array([(b[0] + b[2]) / 2, (b[1] + b[3]) / 2]) / R
        off = ct - ct.astype(int)
        cx, cy = (x + off[0]) * R, (y + off[1]) * R
        w, h = b[2] - b[0], b[3] - b[1]
        if log_noise is not None and (x, y) != tuple(ct.astype(int)):
            w, h = w * np.exp(log_noise[c, 0, y, x]), h * np.exp(log_noise[c, 1, y, x])
        out.append((c, s, (cx - w / 2, cy - h / 2, cx + w / 2, cy + h / 2)))
    return out

def nms(dets, thr=0.5):
    keep = []
    for d in sorted(dets, key=lambda d: -d[1]):
        if all(k[0] != d[0] or iou(k[2], d[2]) <= thr for k in keep):
            keep.append(d)
    return keep

def evaluate(dets, boxes, cls, pair):
    matched = np.zeros(len(boxes), bool); tp = dup = fp = 0
    for c, s, b in sorted(dets, key=lambda d: -d[1]):
        ious = [iou(b, g) if cls[k] == c else 0 for k, g in enumerate(boxes)]
        free = [i if not matched[k] else -1 for k, i in enumerate(ious)]
        k = int(np.argmax(free))
        if free[k] >= 0.5:
            matched[k] = True; tp += 1
        elif max(ious) >= 0.5:
            dup += 1
        else:
            fp += 1
    return np.array([tp, dup, fp, (~matched).sum(), (~matched & (pair >= 0)).sum()])

rng = np.random.default_rng(0)
rng_size = np.random.default_rng(1)               # size errors of the second stand-in
shape, thr = (128, 128), 0.3
names = ["peaks (3x3 max-pool)", "peaks + NMS 0.5", "all cells + NMS 0.5", "all cells, no suppression"]
cases = [(m, z) for m in ("A", "B") for z in ("exact", "noisy")]
totals = {mz: {n: np.zeros(5, int) for n in names} for mz in cases}
n_obj = collisions = 0
for scene in range(50):
    boxes, cls, pair = make_scene(rng)
    n_obj += len(boxes)
    cts = [(c, int((b[0] + b[2]) / 2 / R), int((b[1] + b[3]) / 2 / R)) for b, c in zip(boxes, cls)]
    collisions += len(cts) - len(set(cts))
    gains = rng.uniform(0.5, 0.95, len(boxes))
    noise = rng.normal(0, 0.05, (2,) + shape)
    log_noise = rng_size.normal(0, 0.25, (2, 2) + shape)
    for m in ("A", "B"):
        if m == "A":
            hm, owner = render(boxes, cls, shape)
        else:
            hm, owner = render(boxes, cls, shape, gains)
            hm = np.clip(hm + noise, 0, 1)
        peak = (maximum_filter(hm, size=(1, 3, 3), mode="constant") == hm) & (hm >= thr)
        dense = hm >= thr
        cells_p = [(c, y, x, hm[c, y, x]) for c, y, x in zip(*np.nonzero(peak))]
        cells_d = [(c, y, x, hm[c, y, x]) for c, y, x in zip(*np.nonzero(dense))]
        for z, ln in (("exact", None), ("noisy", log_noise)):
            dp, dd = boxes_from_cells(cells_p, boxes, owner, ln), boxes_from_cells(cells_d, boxes, owner, ln)
            for n, dets in zip(names, (dp, nms(dp), nms(dd), dd)):
                totals[(m, z)][n] += evaluate(dets, boxes, cls, pair)

print(f"\n50 scenes, {n_obj} objects ({50 * 6} in overlapping pairs), "
      f"same-class centre collisions: {collisions}")
hm_label = {"A": "heatmap = target", "B": "heatmap = scaled target + noise"}
sz_label = {"exact": "size exact at every cell", "noisy": "size exact only at centre cells"}
for m, z in cases:
    print(f"{hm_label[m]}; {sz_label[z]}:")
    print(f"  {'decoder':27s} {'TP':>5s} {'dup':>5s} {'FP':>5s} {'missed':>6s} {'(in pairs)':>10s}")
    for n in names:
        tp, dup, fp, miss, missp = totals[(m, z)][n]
        print(f"  {n:27s} {tp:5d} {dup:5d} {fp:5d} {miss:6d} {missp:10d}")

50 scenes, 500 objects (300 in overlapping pairs), same-class centre collisions: 0
heatmap = target; size exact at every cell:
  decoder                        TP   dup    FP missed (in pairs)
  peaks (3x3 max-pool)          500     0     0      0          0
  peaks + NMS 0.5               349     0     0    151        150
  all cells + NMS 0.5           470     0     0     30         30
  all cells, no suppression     500 20416     0      0          0
heatmap = target; size exact only at centre cells:
  decoder                        TP   dup    FP missed (in pairs)
  peaks (3x3 max-pool)          500     0     0      0          0
  peaks + NMS 0.5               349     0     0    151        150
  all cells + NMS 0.5           488   130   890     12         12
  all cells, no suppression     500 18769  1647      0          0
heatmap = scaled target + noise; size exact at every cell:
  decoder                        TP   dup    FP missed (in pairs)
  peaks (3x3 max-pool)          484   173     0     16         16
  peaks + NMS 0.5               361     0     0    139        139
  all cells + NMS 0.5           444     0     0     56         56
  all cells, no suppression     500 14994     0      0          0
heatmap = scaled target + noise; size exact only at centre cells:
  decoder                        TP   dup    FP missed (in pairs)
  peaks (3x3 max-pool)          477   155    25     23         21
  peaks + NMS 0.5               377    17    23    123        120
  all cells + NMS 0.5           484   237   553     16         16
  all cells, no suppression     500 13852  1142      0          0

On the target heatmaps every peak is a centre cell, so the two size stand-ins give the same peak results. Peak extraction recovers every object once, with no duplicates; no two objects of one class shared a centre cell in these scenes, so the collision case that the paper counts on COCO does not arise. Adding NMS at 0.5 then removes one box of every overlapping pair, as it must, since each pair overlaps by at least the threshold, and one other box elsewhere: the suppression that box-based detectors need for duplicates is exactly what deletes the second person in a crowd. Decoding every cell without suppression shows why a dense detector needs NMS at all: every cell inside each Gaussian above 0.3 becomes a box, about forty duplicates per object.

Dense decoding followed by NMS lost far fewer pair members than peaks followed by NMS under both size stand-ins, against the plan’s expectation, but what else it kept depends on the stand-in. With the exact size at every cell its output is clean. A likely cause, in our reading, is that a cell on the far flank of the second object’s Gaussian decodes that object’s exact size at a centre shifted by whole cells away from the first object, and such a box can stay below the NMS threshold with the first object’s box while still matching the second. With sizes exact only at centres, dense decoding with NMS leaves hundreds of duplicates and false positives, about two spurious boxes per object on the target heatmaps; in our reading, boxes with wrong sizes overlap one another too little to be suppressed. Dense decoding plus NMS is therefore only as good as the boxes predicted away from the centres, which a CenterNet head, supervised only at centres, is not trained to provide.

The degraded heatmaps show the other side. Noise and unequal peak heights create local maxima on the flanks of strong peaks, and peak extraction, which suppresses only immediate neighbours, turns them into duplicates, or into false positives when the size off the centre is wrong. The misses fall almost all in overlapping pairs; a likely reason is that a weaker centre lying on the flank of a stronger neighbour of the same class stops being a local maximum. NMS removes the duplicates (all of them with exact sizes, most of them otherwise) but again costs one member of most pairs. Which effect dominates on real images depends on how clean the network’s peaks are; in our reading, the paper’s half-point gain from NMS for DLA-34 and zero gain for Hourglass-104 suggest that trained networks sit close to the clean case, with the stronger backbone the cleaner of the two.

Performance

All results below are the authors’ own, from the arXiv v2 report unless stated otherwise.

2D Detection on COCO

On COCO val2017 (Table 1; DLA-34 with the 230-epoch schedule), the authors reported:

Backbone	AP, no aug.	AP, flip	AP, flip + multi-scale	FPS (no aug. / flip / multi-scale)
Hourglass-104	40.3	42.2	45.1	14 / 7.8 / 1.4
DLA-34	37.4	39.2	41.7	52 / 28 / 4
ResNet-101	34.6	36.2	39.3	45 / 25 / 4
ResNet-18	28.1	30.0	33.2	142 / 71 / 12

The text compares the ResNet-101 model, at 34.8 AP and 45 FPS, with RetinaNet’s 34.4 AP at 18 FPS on the same backbone; Table 1 gives 34.6 for that model, and the text’s 34.8 is not in the table. On the same backbone and the same machine, the authors reported that CenterNet-Hourglass-104 with flip testing outperformed CornerNet (40.6 AP at 4.1 FPS) and ExtremeNet (40.3 AP at 3.1 FPS) in both speed and accuracy on the validation set, and attributed the speed to fewer output heads and simpler decoding.

On COCO test-dev (Table 2), CenterNet-DLA-34 reached 39.2 AP single-scale and 41.6 multi-scale at 28 FPS, and CenterNet-Hourglass-104 reached 42.1 and 45.1 at 7.8 FPS, against 40.5 and 42.1 for CornerNet and 40.2 and 43.7 for ExtremeNet, both with Hourglass-104. Judging from the FPS values, which match the flip-testing column of Table 1, the single-scale entries include flip testing. The authors described 45.1 AP as the best of the one-stage detectors they compared with; two-stage detectors in the same table, such as TridentNet at 48.4, were more accurate and slower.

The paper’s error analysis (supplementary Table 7, COCO validation; the baseline’s 36.3 AP matches the 140-epoch DLA-34 of the ablations, though the appendix does not name the backbone) replaces each head’s output with the ground truth: the true size map raised AP to 41.9, the true heatmap to 54.2, true heatmap and size to 83.1, and all three heads to 99.5. The authors concluded that the centre heatmap, not the size regression, limits accuracy. On centre collisions, they counted 614 pairs of same-class objects that share a centre cell at stride 4 among the 860,001 objects of COCO train2017, under 0.1%; the anchor-free object detection article compares this with anchor collisions and measures it on synthetic scenes. The text compares this with “20.0% for Faster-RCNN with 15 anchors” that miss objects at an IoU threshold of 0.5, while the appendix computes that figure for RetinaNet’s anchors, as 170,220 forced assignments, which is 19.8% of the objects. The text also mentions a multiple-anchor version of CenterNet that regresses several sizes per location, which did not work, and refers to the supplement for it; the supplement of v2 contains no such section.

Pascal VOC

On Pascal VOC 2007 test, trained on VOC 2007 and 2012 trainval, with flip testing and mAP at IoU 0.5 (supplementary Table 6), CenterNet-DLA reached 79.3 at $384 \times 384$ (50 FPS) and 80.7 at $512 \times 512$ (33 FPS), and ResNet-18 72.6 at $384 \times 384$ (142 FPS). Hourglass-104 was not trained on VOC because, the authors wrote, it did not converge within two days when trained from scratch. The model zoo lists 77.1 mAP for ResNet-101 at $384 \times 384$ , where the paper’s table gives 77.6.

3D Detection on KITTI

The 3D model uses DLA-34 at $1280 \times 384$ and trains for 70 epochs. Its evaluation on KITTI is the average precision for cars at 11 recall points and IoU 0.5, on the 2D box (AP), on orientation (AOS) and on the bird’s-eye-view box (BEV AP). Because the validation AP fluctuated by up to 10 points, the authors trained five models and report mean and standard deviation, each comparison on the competing method’s own validation split (Table 4):

Split	Method	AP (easy / moderate / hard)	BEV AP (easy / moderate / hard)
Deep3DBox’s	Deep3DBox	98.8 / 97.2 / 81.2	30.0 / 23.7 / 18.8
Deep3DBox’s	CenterNet	90.2 / 80.4 / 71.1	31.4 / 26.5 / 23.8
Mono3D’s	Mono3D	95.8 / 90.0 / 80.6	30.5 / 22.4 / 19.1
Mono3D’s	CenterNet	97.1 / 87.9 / 79.3	31.5 / 29.7 / 28.1

The authors summarized the results as on par in AP and AOS and slightly better in BEV AP, at two orders of magnitude higher speed; on Deep3DBox’s split the table shows a clearly lower 2D AP, by 8.6 to 16.8 points, and a lower AOS, by 13.3 to 21.7 points (85.3 / 75.0 / 66.2 against 98.6 / 96.7 / 80.5).

Human Pose on COCO

On COCO test-dev keypoint AP, which uses object keypoint similarity in place of box IoU (Table 5), direct regression from the centre reached 51.7 with DLA-34 and 55.0 with Hourglass-104, and snapping to the joint heatmaps raised these to 57.9 and 63.0, against 61.8 for CMU-Pose, 62.8 for Pose-AE, 63.1 for Mask R-CNN and 66.5 for PersonLab. The authors noted that direct regression struggles mostly at high similarity thresholds. The pose models were fine-tuned from the detection models, for 320 epochs (DLA-34) and 150 epochs (Hourglass-104). The README lists 64.0 (Hourglass-104) and 58.9 (DLA-34) keypoint AP on COCO validation, a different split.

Strengths

Simple decoding. A detection is a heatmap peak plus two lookups; there is no anchor design, no IoU-based assignment and, at single scale, no NMS.
Speed range. One design runs from 142 FPS at 28.1 AP (ResNet-18) to 45.1 AP with multi-scale testing (Hourglass-104), as reported on one machine.
Extensible output. Any per-object quantity can be regressed at the centre point, which the paper used for 3D boxes and pose and later work used for tracking.
Few duplicates by construction. One positive per object keeps duplicate peaks rare, and peak extraction keeps heavily overlapping objects that box NMS would merge, as long as their centres fall on different cells.

Limitations

Centre collisions. Two objects of the same class whose centres fall on one stride-4 cell yield one detection; the authors measured this as under 0.1% of COCO training objects. Because the size and offset heads are shared across classes, objects of different classes at one cell also share one size prediction.
One output scale. All objects are predicted on a single stride-4 map with no pyramid; the per-object Gaussian radius and the raw-pixel size targets, weighted by $\lambda_{\text{size}} = 0.1$ , carry the burden of scale. The paper’s ablations show AP sensitive to that weight and to the size loss.
Heatmap quality limits accuracy. In the error analysis the heatmap, not the size head, accounts for most of the gap to perfect detection.
Augmented testing reintroduces NMS. Multi-scale results are merged with NMS (Soft-NMS in the released code), so the NMS-free property holds for single-scale and flip testing only.
Custom operators and heavy backbones. The ResNet and DLA variants depend on deformable convolutions, built from the DCNv2 code; the Hourglass-104 model takes 5 days to train on COCO and was not trained on VOC because it did not converge from scratch in two days.
Regression-only pose is weak. Direct joint regression trailed heatmap-based methods by several points of keypoint AP and needed heatmap snapping to compete.

Historical Importance

CenterNet is the third step of a short lineage in which object detection was recast as keypoint estimation with the tools of human pose estimation. CornerNet (Law and Deng, 2018) detected an object as two corners found in per-class heatmaps at stride 4 with an Hourglass-104 backbone, grouped them with associative embeddings borrowed from multi-person pose, trained them with the penalty-reduced focal loss, and added corner pooling because corners usually lie outside the object. ExtremeNet, which Zhou et al. cite, detects the top-, left-, bottom- and right-most points and the centre of each object, and like CornerNet needs a grouping stage to assemble them. CenterNet kept the backbone, the heatmap targets and the loss of both, and removed the grouping step entirely by detecting the centre alone; Zhou et al. read their accuracy gain over CornerNet and ExtremeNet as evidence that centres are easier to detect than corners or extreme points, the opposite of CornerNet’s argument that corners are easier to localize because a corner depends on two sides of the box and a centre on all four.

With FCOS, of the same year, CenterNet made the anchor-free design competitive on COCO in 2019, and it is one of the detectors that later analyses of anchor-free detection, such as Zhang et al.’s study of training-sample selection, group as keypoint-based. Its centre-point formulation was reused by the same group for tracking (CenterTrack, in the repository’s update of April 2020) and for LiDAR 3D detection (CenterPoint), and the README lists third-party systems built on it, such as FairMOT for multi-object tracking.

Variants

CenterNet: Keypoint Triplets (Duan et al., 2019). A different detector with the same name, built on CornerNet rather than on “Objects as Points”. Duan et al. measured that many of CornerNet’s boxes were wrong, with a false discovery rate of 32.7% at IoU 0.05 for CornerNet511-104 on COCO validation, and attributed this to CornerNet’s inability to look inside a box. Their network adds a centre heatmap and its offsets to CornerNet’s two corner heatmaps, embeddings and offsets. A box proposed by a pair of corners is kept only if a centre keypoint of the same class falls inside its central region, which is a scaled copy of the box shrunk by a factor $n$ ( $n = 3$ for boxes whose scale is below 150 and 5 for larger ones), and its score becomes the mean of the three keypoints’ scores. Center pooling adds the maximum responses along the horizontal and vertical lines through a location, and cascade corner pooling extends CornerNet’s corner pooling by looking first along the box boundary and then into the object. Inference keeps CornerNet’s flip testing and uses Soft-NMS. On COCO test-dev, with $511 \times 511$ training inputs, the authors reported 44.9 AP single-scale and 47.0 multi-scale with Hourglass-104 (CornerNet: 40.5 and 42.1), 41.6 and 43.5 with Hourglass-52, and 44.0 and 46.3 with HRNet-W64 (Table 2); the text gives 45.6 for the HRNet-W64 multi-scale result, which differs from the table. Inference took 340 ms per image with Hourglass-104 on a Tesla P100, against 300 ms for CornerNet. The two CenterNets thus use a centre point for opposite purposes: Zhou et al. detect an object as its centre alone, Duan et al. use a centre to verify a pair of corners.

CenterTrack and CenterPoint. The official repository’s README announces CenterTrack (April 2020), which extends the centre-point detector to tracking, described in multi-object tracking, and CenterPoint (June 2020). CenterPoint (Yin, Zhou and Krähenbühl, 2021) detects 3D objects in LiDAR point clouds by finding their centres with a keypoint detector in a bird’s-eye-view map, regresses 3D size, orientation and velocity there, refines them in a second stage with point features on the object, and reduces tracking to greedy closest-point matching.

CenterNet2. Zhou, Koltun and Krähenbühl’s “Probabilistic two-stage detection” (2021, an arXiv report not read here) turns CenterNet into the first stage of a two-stage detector. According to the official repository, the first stage estimates object probabilities with a class-agnostic one-stage detector and the second stage classifies conditionally on them, and the best model reached 56.4 mAP on COCO test-dev (README, as of November 2026). The same repository contains a Detectron2-based reimplementation of CenterNet that it reports at 42.5 mAP on COCO val at 71 FPS on a Titan RTX (70 FPS in the README’s summary), and an FPN version of CenterNet.

Implementations

The official PyTorch code, xingyizhou/CenterNet, is released under the MIT license and borrows parts from CornerNet (hourglass network, loss functions), DLA, DCNv2, human-pose-estimation.pytorch and others under their own licenses (README). It provides training scripts for every experiment in the paper and pretrained models for detection on COCO and VOC, 3D detection on KITTI, and pose on COCO, hosted on Google Drive; its experiments used PyTorch 0.4.1, CUDA 9.0 and cuDNN 7.1 (MODEL_ZOO.md). The task names in the code are ctdet for 2D detection, ddd for 3D detection and multi_pose for pose. As of November 2026 the most recent commit is from November 2022. The README also lists third-party reimplementations in Detectron2, Keras, MXNet and TensorRT.