PointPillars

PointPillars is a single-stage 3D object detector for LiDAR point clouds. It divides the ground plane into a regular grid and treats each cell as a pillar, a voxel of unlimited height. A simplified PointNet encodes the points of each non-empty pillar into one feature vector, the vectors are scattered back to their grid positions to form a bird’s-eye-view pseudo-image, and from there the detector is an ordinary 2D convolutional neural network with an SSD-style head that regresses oriented 3D boxes. Alex H. Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang and Oscar Beijbom of nuTonomy presented it at CVPR 2019, after an arXiv report of December 2018. It descends from VoxelNet (Zhou and Tuzel, 2018), which first learned the encoding of LiDAR voxels end to end, and SECOND (Yan, Mao and Li, 2018), which sped VoxelNet up with sparse 3D convolutions; PointPillars removed the 3D convolutions altogether. The authors reported state-of-the-art bird’s-eye-view and 3D results on the KITTI benchmark at 62 Hz, and the network became the LiDAR baseline of the nuScenes dataset and of the Waymo Open Dataset paper.

Problem Addressed

A LiDAR sweep is sparse and three-dimensional, so an image detector cannot be applied to it directly. Before learned encoders, the usual remedy was to project the points onto the ground plane, divide it into a grid of about 10 cm, and compute hand-crafted features in each cell, as MV3D, AVOD, PIXOR and Complex-YOLO did; the result is a pseudo-image that a standard detection network can process. The bird’s-eye view keeps object scales and keeps convolutions from blurring depth (Lang et al., 2019, §1). PointPillars’ authors argue that such fixed encodings may not generalize to new configurations without significant engineering.

VoxelNet replaced the fixed encoding with a learned one. It divides space into 3D voxels (0.2 × 0.2 × 0.4 m for cars), randomly samples at most 35 points in each, and encodes them with stacked voxel feature encoding (VFE) layers: a shared fully connected layer per point, a max over the voxel’s points, and concatenation of the pooled feature back onto each point, so that two layers, VFE-1 (7, 32) and VFE-2 (32, 128), let points exchange information within a voxel (Zhou & Tuzel, 2018, §2.1 and §3.1). Three 3D convolutional middle layers then collapse the vertical axis to a 128 × 400 × 352 feature map for a region proposal network. The two papers disagree on its speed. VoxelNet’s authors report 33 ms per scan on a Titan X GPU (§4.2), while the PointPillars paper gives 225 ms (4.4 Hz), 190 ms of it in the encoder, which is also the speed listed for VoxelNet in its KITTI tables (Lang et al., 2019, §1.1, §6 and Table 1). SECOND, known here only through the PointPillars paper and its code, replaced the dense 3D convolutions with sparse ones and reached 20 Hz, but as PointPillars describes it the 3D convolutions remained the bottleneck (§1); the sparse convolutional networks article describes SECOND and the 48 ms its open-source encoder took.

PointPillars keeps the learned encoder but makes every voxel a pillar. With no vertical binning there is no vertical resolution to tune, the encoder needs no 3D convolution, and everything after it is dense 2D convolution, which GPUs run efficiently (§1).

Architecture

The network has three stages (the paper’s Fig. 2): a pillar feature network that turns the point cloud into a sparse pseudo-image, a 2D convolutional backbone, and a detection head. Points are in the LiDAR’s coordinate frame, with $z$ vertical. The paper’s range for cars is $x \in (0, 70.4)$, $y \in (-40, 40)$ and $z \in (-3, 1)$ m (§4.2), the same as VoxelNet’s; the official configuration uses a slightly smaller $[0, 69.12) \times [-39.68, 39.68) \times [-3, 1)$ m.

Pillars and point decorations

The $x$–$y$ plane is divided into a grid of cells, 0.16 m on a side by default; the paper writes the cell area as $0.16^2$ m². Each point $l = (x, y, z, r)$, with $r$ the reflectance, is assigned to the pillar of its cell and decorated with five more values: its offsets $(x_c, y_c, z_c)$ from the arithmetic mean of the points in its pillar, and its offsets $(x_p, y_p)$ from the pillar’s centre in $x$ and $y$. Each point thus becomes a vector $\hat{l} \in \mathbb{R}^D$ with $D = 9$ (§2.1). The cluster offsets come from VoxelNet; the pillar offsets are PointPillars’ addition, and the authors argue that they standardize each point’s local context without depending on the other points, whereas the cluster centre changes under augmentation and subsampling (§7.3). The $z$ offset from the pillar centre is left out because it is the same for every point. Other decorations could serve other point clouds, such as radar or RGB-D, and the nuScenes baseline later added each point’s time offset (see Performance).

Most pillars are empty. At $0.16^2$ m² bins, the paper reports 6,000–9,000 non-empty pillars in a KITTI HDL-64E scan over the usual range, a sparsity of about 97%, and few points in each non-empty one. PointPillars exploits this by fixing two caps: at most $P$ non-empty pillars per sample and at most $N$ points per pillar, so that each sample becomes a dense tensor of shape $(D, P, N)$. Pillars or samples with too much data are subsampled, and those with too little are padded with zeros (§2.1). The defaults are $P = 12{,}000$ and $N = 100$ (§4.2).

Pillar feature network

Each decorated point goes through the same linear layer, batch normalization and a ReLU, and a max over the $N$ points of the pillar gives its feature:

$$\mathbf{f}_j = \max_{i = 1, \dots, N} \operatorname{ReLU}\!\big(\operatorname{BN}(W \hat{l}_{j,i})\big) \in \mathbb{R}^C, \qquad C = 64,$$

for pillar $j$, turning the $(D, P, N)$ tensor into a $(C, P)$ one (§2.1, §3.1). This is PointNet’s shared per-point network and symmetric max pooling, applied to each pillar with a single layer; the linear layer is a $1 \times 1$ convolution over the tensor. VoxelNet used two VFE layers in sequence; using one saved 2.5 ms in the authors’ PyTorch implementation without affecting detection, by their account (§6). In the official code the class is PFNLayer, and its comment notes that the paper’s results used a single layer.

Pseudo-image

The $P$ feature vectors are scattered back to the grid cells their pillars came from, giving a pseudo-image of size $(C, H, W)$ in which empty cells are zero (§2.1). For the car configuration, $69.12 / 0.16 = 432$ cells along $x$ and $79.36 / 0.16 = 496$ along $y$. The scatter is a single indexed assignment (PointPillarsScatter in the code), and this step is what replaces VoxelNet’s convolutional middle layers.

Backbone

The backbone follows VoxelNet’s region proposal network (§2.2). A top-down path of three blocks produces features at decreasing resolution: $\text{Block}(S, L, F)$ has $L$ $3 \times 3$ convolutions with $F$ output channels, each followed by BatchNorm and ReLU, the first with the stride needed to bring the block to stride $S$ relative to the pseudo-image. The outputs of the three blocks are each upsampled by a transposed convolution, $\text{Up}(S_{in}, S_{out}, F)$, followed by BatchNorm and ReLU, and concatenated. The configuration in §3.1 is

$$\text{Block}_1(S, 4, C),\; \text{Block}_2(2S, 6, 2C),\; \text{Block}_3(4S, 6, 4C);\quad \text{Up}_1(S, S, 2C),\; \text{Up}_2(2S, S, 2C),\; \text{Up}_3(4S, S, 2C),$$

giving $6C = 384$ channels at stride $S$, with $S = 2$ for the car network and $S = 1$ for the pedestrian and cyclist network, whose objects are smaller. Two of the speed-ups the authors list are here: lowering the first block’s width to 64 to match the encoder saved 4.5 ms, and halving the upsampled outputs to 128 channels saved 3.9 ms (§6). The official configuration expresses the blocks as layer_nums: [3, 5, 5], the convolutions after each block’s strided first convolution, which gives the paper’s 4, 6 and 6.

Detection head

The head follows SSD’s setup (§2.3): convolutions on the concatenated features, $1 \times 1$ in the official code, predict for every cell and every anchor the class scores, seven box residuals and a direction class. Each class has one anchor size, applied at two orientations, 0° and 90°: $(w, l, h) = (1.6, 3.9, 1.5)$ m with centre height $-1$ m for cars, $(0.6, 0.8, 1.73)$ m for pedestrians and $(0.6, 1.76, 1.73)$ m for cyclists, both at $-0.6$ m (§4.2). The anchor boxes are matched to ground truth by 2D intersection over union in the bird’s-eye view, as in SSD; height and elevation play no part in matching and become extra regression targets. An anchor is positive if it has the highest IoU with a ground-truth box or exceeds the positive threshold (0.6 for cars, 0.5 for pedestrians and cyclists), negative below the negative threshold (0.45 and 0.35), and ignored otherwise. The authors note that swapping the head is all another task, such as segmentation, would need.

Training

PointPillars uses the losses introduced by SECOND (§3.2). Boxes and anchors are $(x, y, z, w, l, h, \theta)$, and the regression targets are

$$\Delta x = \frac{x^{gt} - x^a}{d^a},\quad \Delta y = \frac{y^{gt} - y^a}{d^a},\quad \Delta z = \frac{z^{gt} - z^a}{h^a},\quad \Delta w = \log \frac{w^{gt}}{w^a},\quad \Delta l = \log \frac{l^{gt}}{l^a},\quad \Delta h = \log \frac{h^{gt}}{h^a},$$ $$\Delta \theta = \sin\!\big(\theta^{gt} - \theta^a\big), \qquad d^a = \sqrt{(w^a)^2 + (l^a)^2},$$

with the localization loss $\mathcal{L}_{loc} = \sum_{b \in (x, y, z, w, l, h, \theta)} \text{SmoothL1}(\Delta b)$. The sine makes the loss blind to a 180° flip of the box, so the heading is learned separately as a softmax classification over discretized directions, $\mathcal{L}_{dir}$. The classification loss is the focal loss of RetinaNet, $\mathcal{L}_{cls} = -\alpha_a (1 - p^a)^\gamma \log p^a$ with $\alpha = 0.25$ and $\gamma = 2$, where $p^a$ is the anchor’s class probability. The total loss is

$$\mathcal{L} = \frac{1}{N_{pos}} \big( \beta_{loc} \mathcal{L}_{loc} + \beta_{cls} \mathcal{L}_{cls} + \beta_{dir} \mathcal{L}_{dir} \big), \qquad \beta_{loc} = 2,\ \beta_{cls} = 1,\ \beta_{dir} = 0.2,$$

with $N_{pos}$ the number of positive anchors. In the official code the sine is applied inside the loss (the angle residual itself is stored as a plain difference and both sides are encoded as $\sin a \cos b$ and $\cos a \sin b$), and the direction target has two bins, whether the ground-truth yaw is positive.

Weights are initialized randomly from a uniform distribution, without pre-training. Adam starts at a learning rate of $2 \times 10^{-4}$, decayed by 0.8 every 15 epochs, for 160 epochs with batch size 2 on the validation split and 320 epochs with batch size 4 for the test submission (§3). KITTI’s 7,481 training samples were split into 3,712 for training and 3,769 for validation; for the test submission a mini-val set of 784 samples was held out and the network trained on the remaining 6,733, as the paper gives the figures (7,481 − 784 would be 6,697). One network is trained for cars and one for pedestrians and cyclists, and only the points that project into the camera image are used, because KITTI annotates only objects visible in the image (§4.1).

Data augmentation follows SECOND’s ground-truth sampling: a database of all ground-truth boxes with the points inside them is built, and 15, 0 and 8 boxes of cars, pedestrians and cyclists are pasted into each training sample. Each box is then rotated by an angle drawn uniformly from $[-\pi/20, \pi/20]$ and translated by $\mathcal{N}(0, 0.25)$ in $x$, $y$ and $z$; the whole scene is mirrored along the $x$ axis, rotated and scaled, and translated by $\mathcal{N}(0, 0.2)$ to simulate localization noise (§4.3). The paper gives no values for the global rotation and scaling; the official configuration uses $[-\pi/4, \pi/4]$ and $[0.95, 1.05]$. Its per-box standard deviation of 0.25 m, and the 0.2 m default of the global translation in the preprocessing code, match the paper’s $\mathcal{N}(\cdot)$ arguments read as standard deviations. Contrary to VoxelNet’s and SECOND’s recommendations, the authors found that minimal per-box augmentation worked better, and that pedestrian detection degraded significantly with more of it; they suggest ground-truth sampling makes heavy per-box augmentation unnecessary (§7.2). The README of the code puts training of a 0.16 m (xyres_16) configuration at about 20 hours for 160 epochs on one GTX 1080 Ti.

Inference

At inference the scan is cropped to the range and to the points visible in the image, organized into pillars and decorated, encoded, scattered, and passed through the backbone and head; non-maximum suppression then removes overlapping boxes. PointPillars uses axis-aligned non-maximum suppression with an IoU threshold of 0.5, which the authors found about as accurate as rotated NMS and much faster (§4.2). The official configuration keeps the 1,000 highest-scoring boxes before suppression and 300 after, with a score threshold of 0.05.

The authors timed the car network on a desktop with an Intel i7 CPU and a GTX 1080 Ti GPU (§6): 1.4 ms to load and filter the points, 2.7 ms to form and decorate the pillars, 2.9 ms to upload the tensor to the GPU, 1.3 ms to encode it, 0.1 ms to scatter, 7.7 ms for the backbone and heads and 0.1 ms for NMS on the CPU, 16.2 ms in all. The encoder’s 1.3 ms is two orders of magnitude below VoxelNet’s 190 ms. The final kernels were built with NVIDIA TensorRT, a 45.5% speed-up over the PyTorch pipeline’s 42.4 Hz. The authors add two caveats to their own figures: on KITTI only about 10% of the LiDAR points, those projecting into the front camera, are processed, and timings in the literature are measured on high-power desktop GPUs rather than the embedded hardware a vehicle may carry.

Example: Point Caps and Pillar Caps by Range

The script below (NumPy 2.5.4) builds synthetic LiDAR scans with known boxes and asks how many points the point and pillar caps discard, and where. It runs in about 10 seconds on a laptop CPU, and two runs printed identical output.

The scene generator places, in front of a 64-beam spinning sensor 1.73 m above flat ground, 6–12 cars (a 3.9 × 1.6 m body 0.7 m tall on a 0.3 m clearance, with a 2.0 × 1.5 m cabin on top, 1.5 m high in all), 2–4 vans (one box 1.6–2.0 m tall), 2–4 low barriers, 3–6 poles and two building fronts, at random positions and headings. Every object gets a random reflectivity. The beams are spaced evenly from −24.8° to +2.0°, the 26.8° vertical field the LiDAR article lists for the HDL-64E S3, with 0.09° azimuth steps over the front half; even spacing and the rest of the sensor are our simplifications, not a model of that product. The ray caster intersects each ray with every box in the box’s own frame and keeps the nearest hit within 100 m, with 2 cm range noise. Coordinates are in a ground frame, $x$ forward, $y$ left and $z$ up from the ground, so the official $z$ range of $[-3, 1)$ m in the sensor frame becomes $[-1.27, 2.73)$ m.

The script applies the caps to 20 scans over the official car range, at the five operating points of the paper’s Fig. 5 (grids of 0.12 to 0.28 m with 16,000, 12,000, 12,000, 8,000 and 8,000 pillars) and $N = 100$. The caps follow the official points_to_voxel kernel rather than the paper’s description: points are visited in order, a pillar keeps its first 100 points, and as soon as a point would open pillar $P + 1$ the loop stops, so that point and every later one, in any pillar, are dropped. A function that visits points one by one in that way gave the same kept points as the vectorized version below on three scans, two of them over the pillar cap. Which points come first is therefore a modelling choice. The paper says the data are randomly sampled, and the official configuration shuffles points during training (shuffle_points: true) but not during evaluation (false), where the stored order applies; the sources do not describe that order. The script prints three orders: shuffled, beam-major (every azimuth of the lowest beam, then the next beam) and azimuth-major (all beams of one firing, firing by firing). The last column is the mean height of the 100 kept points minus the mean height of all points, over the pillars that hit the point cap.

import numpy as np

# ---- a synthetic scan and the point and pillar caps ----
# A 64-beam spinning sensor 1.73 m above flat ground (z = 0): beams evenly spaced
# from -24.8 to +2.0 degrees, 0.09 degree azimuth steps over the front half.
H = 1.73
ELEV = np.radians(np.linspace(-24.8, 2.0, 64))
AZ = np.radians(np.arange(-90, 90, 0.09))

def rot(yaw):
    c, s = np.cos(yaw), np.sin(yaw)
    return np.array([[c, -s, 0], [s, c, 0], [0, 0, 1.0]])

def make_scene(rng):
    """Oriented boxes (cx, cy, z0, l, w, h, yaw, reflectivity) and the car footprints."""
    boxes, cars, used = [], [], []
    def place(n, rad):
        out = []
        for _ in range(n):
            for _ in range(50):
                x, y = rng.uniform(2, 45), rng.uniform(-22, 22)
                if np.hypot(x, y) > 4 and all(np.hypot(x - a, y - b) > rad + c for a, b, c in used):
                    used.append((x, y, rad)); out.append((x, y)); break
        return out
    for x, y in place(rng.integers(6, 13), 2.2):      # cars: body plus cabin
        yaw, r = rng.uniform(-np.pi, np.pi), rng.uniform(0.1, 0.9)
        boxes.append((x, y, 0.3, 3.9, 1.6, 0.7, yaw, r))
        dx, dy, _ = rot(yaw) @ [-0.3, 0, 0]
        boxes.append((x + dx, y + dy, 1.0, 2.0, 1.5, 0.5, yaw, r))
        cars.append((x, y, 3.9, 1.6, yaw))
    for x, y in place(rng.integers(2, 5), 2.6):       # vans: one taller box
        boxes.append((x, y, 0.3, rng.uniform(4.2, 5.0), 1.9, rng.uniform(1.6, 2.0),
                      rng.uniform(-np.pi, np.pi), rng.uniform(0.1, 0.9)))
    for x, y in place(rng.integers(2, 5), 2.0):       # barriers
        boxes.append((x, y, 0.0, rng.uniform(2, 4), 0.4, rng.uniform(0.8, 1.0),
                      rng.uniform(-np.pi, np.pi), rng.uniform(0.1, 0.9)))
    for x, y in place(rng.integers(3, 7), 0.5):       # poles
        boxes.append((x, y, 0.0, 0.3, 0.3, 3.0, 0.0, rng.uniform(0.1, 0.9)))
    for side in (-1, 1):                              # building fronts
        boxes.append((35, side * rng.uniform(24, 30), 0.0, 70, 1.0, rng.uniform(3, 6),
                      0.0, rng.uniform(0.1, 0.9)))
    return boxes, cars

def scan(boxes, rng):
    """Nearest hit of every ray; points (x, y, z, intensity) in beam-major firing order."""
    E, A = np.meshgrid(ELEV, AZ, indexing="ij")
    d = np.stack([np.cos(E) * np.cos(A), np.cos(E) * np.sin(A), np.sin(E)], -1).reshape(-1, 3)
    o = np.array([0, 0, H])
    t = np.full(len(d), np.inf); refl = np.zeros(len(d))
    with np.errstate(divide="ignore"):
        tg = -H / d[:, 2]
    g = d[:, 2] < 0
    t[g], refl[g] = tg[g], 0.2                       # ground
    for cx, cy, z0, l, w, h, yaw, r in boxes:        # slab test in each box's frame
        R = rot(yaw)
        ol, dl = R.T @ (o - [cx, cy, z0 + h / 2]), d @ R
        half = np.array([l, w, h]) / 2
        with np.errstate(divide="ignore", invalid="ignore"):
            t1, t2 = (-half - ol) / dl, (half - ol) / dl
        tn, tf = np.nanmax(np.minimum(t1, t2), 1), np.nanmin(np.maximum(t1, t2), 1)
        hit = (tn <= tf) & (tn > 0) & (tn < t)
        t[hit], refl[hit] = tn[hit], r
    ray = np.flatnonzero(np.isfinite(t) & (t < 100))
    pts = o + d[ray] * (t[ray] + rng.normal(0, 0.02, len(ray)))[:, None]
    inten = np.clip(refl[ray] + rng.normal(0, 0.05, len(ray)), 0, 1)
    return np.column_stack([pts, inten]).astype(np.float32), ray

def apply_caps(P, size, n_cap, p_cap, box):
    """The official code's rule: points are visited in order; a pillar is opened at its
    first point; a pillar keeps its first n_cap points; when a point would open pillar
    number p_cap + 1, that point and every later point are dropped."""
    x0, x1, y0, y1 = box
    ny = round((y1 - y0) / size)
    key = np.floor((P[:, 0] - x0) / size).astype(np.int64) * ny + \
          np.floor((P[:, 1] - y0) / size).astype(np.int64)
    uniq, first, inv = np.unique(key, return_index=True, return_inverse=True)
    stop = np.sort(first)[p_cap] if len(uniq) > p_cap else len(P)
    srt = np.lexsort((np.arange(len(P)), key))           # by pillar, then by order
    slot = np.empty(len(P), np.int64)
    slot[srt] = np.arange(len(P)) - np.searchsorted(key[srt], key[srt], side="left")
    before = np.arange(len(P)) < stop
    return before & (slot < n_cap), before, len(uniq), inv

BOX = (0.0, 69.12, -39.68, 39.68)                 # the official car configuration's x-y range
ZR = (H - 3.0, H + 1.0)                           # its z range [-3, 1] m in the sensor frame
GRIDS = [(0.12, 16000), (0.16, 12000), (0.20, 12000), (0.24, 8000), (0.28, 8000)]
BINS = [0, 10, 20, 40, 80]
orders = {"shuffled": None, "beam-major": None, "azimuth-major": None}
stats = {(o, s): [] for o in orders for s, _ in GRIDS}
n_points = []
for k in range(20):
    rng = np.random.default_rng(100 + k)
    P, ray = scan(make_scene(rng)[0], rng)
    m = (P[:, 0] >= BOX[0]) & (P[:, 0] < BOX[1]) & (P[:, 1] >= BOX[2]) & (P[:, 1] < BOX[3]) \
        & (P[:, 2] >= ZR[0]) & (P[:, 2] < ZR[1])
    P, ray = P[m], ray[m]
    n_points.append(len(P))
    beam, col = ray // len(AZ), ray % len(AZ)
    orders["shuffled"] = rng.permutation(len(P))
    orders["beam-major"] = np.lexsort((col, beam))
    orders["azimuth-major"] = np.lexsort((beam, col))
    for oname, idx in orders.items():
        Q = P[idx]
        rbin = np.digitize(np.hypot(Q[:, 0], Q[:, 1]), BINS) - 1
        for size, p_cap in GRIDS:
            kept, before, n_pil, inv = apply_caps(Q, size, 100, p_cap, BOX)
            lostN = [np.mean(before[rbin == b] & ~kept[rbin == b]) for b in range(4)]
            lostP = [np.mean(~before[rbin == b]) for b in range(4)]
            cnt = np.bincount(inv[before], minlength=n_pil)
            full = np.flatnonzero(cnt > 100)                  # pillars that hit the point cap
            sel = np.isin(inv, full) & before
            zall = np.bincount(inv[sel], Q[sel, 2], n_pil)[full] / cnt[full]
            zkept = np.bincount(inv[sel & kept], Q[sel & kept, 2], n_pil)[full] / 100
            stats[(oname, size)].append([n_pil, n_pil > p_cap, *lostN, *lostP,
                                         np.mean(zkept - zall) * 100 if len(full) else 0])

print("20 scans, mean points in range: %.0f" % np.mean(n_points))
print("order          grid  pillars  over P   lost to point cap (%) by range      lost to pillar cap (%)"
      "              z bias")
print("                                         0-10  10-20  20-40  40-70 m     0-10  10-20  20-40  40-70 m   (cm)")
for oname in orders:
    for size, p_cap in GRIDS:
        a = np.array(stats[(oname, size)])
        print(f"{oname:13s} {size:5.2f} {a[:, 0].mean():8.0f} {int(a[:, 1].sum()):3d}/20  "
              + " ".join(f"{100 * v:6.2f}" for v in a[:, 2:6].mean(0)) + "   "
              + " ".join(f"{100 * v:6.2f}" for v in a[:, 6:10].mean(0)) + f" {a[:, 10].mean():7.1f}")

Output:

20 scans, mean points in range: 123608
order          grid  pillars  over P   lost to point cap (%) by range      lost to pillar cap (%)              z bias
                                         0-10  10-20  20-40  40-70 m     0-10  10-20  20-40  40-70 m   (cm)
shuffled       0.12    16310  13/20    4.06   0.26   0.00   0.00    18.02  18.06  18.07  17.81    -0.0
shuffled       0.16    11416   6/20    5.77   1.34   0.00   0.00     9.30   9.26   9.31   9.01     0.0
shuffled       0.20     8585   0/20    7.05   2.49   0.00   0.00     0.00   0.00   0.00   0.00    -0.1
shuffled       0.24     6836   1/20    9.32   3.39   0.05   0.00     1.40   1.39   1.40   1.43    -0.1
shuffled       0.28     5654   0/20   14.92   4.58   0.31   0.00     0.00   0.00   0.00   0.00     0.0
beam-major     0.12    16310  13/20    4.16   0.21   0.00   0.00     0.51   3.04  27.54  46.09   -17.0
beam-major     0.16    11416   6/20    5.82   1.33   0.00   0.00     0.20   0.73  11.37  19.27   -24.1
beam-major     0.20     8585   0/20    7.05   2.50   0.00   0.00     0.00   0.00   0.00   0.00   -31.0
beam-major     0.24     6836   1/20    9.40   3.40   0.05   0.00     0.00   0.08   1.82   2.99   -10.9
beam-major     0.28     5654   0/20   14.92   4.59   0.31   0.00     0.00   0.00   0.00   0.00    -7.7
azimuth-major  0.12    16310  13/20    4.30   0.42   0.00   0.00     5.71   5.47   6.39   0.00    -1.9
azimuth-major  0.16    11416   6/20    5.82   1.45   0.00   0.00     2.63   2.11   2.71   0.00    -2.2
azimuth-major  0.20     8585   0/20    7.05   2.49   0.00   0.00     0.00   0.00   0.00   0.00    -2.1
azimuth-major  0.24     6836   1/20    9.39   3.44   0.05   0.00     0.22   0.20   0.33   0.00    -0.9
azimuth-major  0.28     5654   0/20   14.92   4.57   0.31   0.00     0.00   0.00   0.00   0.00    -0.6

The point cap falls on the nearest pillars. Within 10 m it removes 4% of the points at the finest grid and 15% at the coarsest, beyond 20 m almost nothing, because a pillar near the sensor collects hundreds of returns (from the ground and from vertical surfaces hit by many beams) and a distant one only a few. The share lost depends on the grid size and hardly at all on the point order. The pillar cap behaves differently. These scans have more non-empty pillars than the 6,000–9,000 the paper reports for KITTI, in our reading because they cover the whole front half rather than the camera’s view, and the cap was exceeded in 13 of 20 scans at 0.12 m, 6 at 0.16 m and 1 at 0.24 m. When it is exceeded, the order decides what is lost. With shuffled points the kernel stops at a random point in the list, so every later point is gone, whatever its pillar: the scan is thinned evenly, by the same share at every range (18% of all points on average at 0.12 m), including pillars that were already open. With beam-major order the upper beams come last, and they are the ones that reach far, so the losses fall on distant points: 46% of the points beyond 40 m at 0.12 m and 19% at 0.16 m, where the paper’s default grid runs. With azimuth-major order the last firings, one side of the sector, are lost at every range up to the building fronts; nothing beyond 40 m is lost there because, in these scenes, the last firings point sideways at buildings 24–30 m away (our reading of the scene’s construction). Order also changes which points a full pillar keeps. Shuffled, the kept points have the same mean height as all of them; beam-major, a full pillar keeps its lowest beams’ points and its mean height is 8–31 cm too low. The cluster offsets $(x_c, y_c, z_c)$ are computed from the kept points, so they inherit that bias.

Performance

KITTI

PointPillars was evaluated on the KITTI object benchmark with its official metrics, average precision in the bird’s-eye view and in 3D, 2D detection and average orientation similarity (AOS), with methods ranked at moderate difficulty; the KITTI article gives the matching rules and difficulty levels. On the test set (the paper’s Tables 1 and 2):

Method	Input	Speed (Hz)	BEV mAP, moderate	BEV car, moderate	3D mAP, moderate	3D car, moderate
AVOD-FPN	LiDAR and image	10	64.11	83.79	55.62	71.88
F-PointNet	LiDAR and image	5.9	65.39	84.00	57.35	70.39
VoxelNet	LiDAR	4.4	58.25	79.26	49.05	65.11
SECOND	LiDAR	20	60.56	79.37	56.69	73.66
PointPillars	LiDAR	62	66.19	86.10	59.20	74.99

The authors write that PointPillars beats the other LiDAR-only methods in every class and difficulty except easy cars, and also beats the fusion methods on cars and cyclists. The tables support the first claim. For the second, they support it for cyclists at every difficulty but for cars only at moderate and hard difficulty: several fusion methods score higher on easy cars (in 3D, for example, 81.94 for AVOD-FPN and 81.20 for F-PointNet against 79.05). Fig. 1 (bird’s-eye view) adds that the fusion methods remained ahead on pedestrians. On AOS (Table 3) PointPillars reached 68.86 mAOS at moderate difficulty, against 54.53 for SECOND and 63.19 for AVOD-FPN, the only other 3D detectors in the table that predict orientation, and its 68.16 on moderate cyclists exceeded the best image-only method. On the validation split, the moderate results were 87.7, 67.9 and 66.8 BEV AP and 77.4, 61.8 and 64.9 3D AP for cars, pedestrians and cyclists (footnote 2).

The ablations, all on the KITTI validation set by BEV mAP, separate the encoder from the rest of the system:

• Spatial resolution. Larger pillars are faster, with fewer non-empty pillars and a smaller pseudo-image. Over grids of $0.12^2$ to $0.28^2$ m² (Fig. 5) the authors report 105 Hz at $0.28^2$ m² with performance similar to previous methods, and that the loss in accuracy came mainly from pedestrians and cyclists while cars were stable (§7.1).
• Decorations. With $x$, $y$, $z$ only, BEV mAP was 66.6, and adding reflectance gave 70.5. On top of these, the cluster offsets alone gave 70.4, the pillar offsets alone 71.4, and both together 72.6 (Table 4).
• Encoders. Several encoders were implemented in SECOND’s code base with everything else fixed (Table 5). At $0.16^2$ m², the fixed encoders of MV3D, Complex-YOLO and PIXOR scored 72.8, 72.0 and 72.9 and the learned VoxelNet and PointPillars encoders 74.4 and 73.7; at $0.28^2$ m² the fixed encoders fell to 65.6–67.6 and the learned ones to 71.9 and 72.0. The authors conclude that learned encoders are better at every resolution and that the gap widens with bin size. They also write that VoxelNet’s encoder is stronger than theirs; in the table it is ahead at the two finer resolutions, tied at $0.24^2$ m² and behind by 0.1 at $0.28^2$ m². Their argument for pillars is the operating point at a similar inference time, not accuracy at equal resolution. They also note that their re-implementations beat the published results of the original encoders by 1–10 mAP, and attribute about 3 mAP of that, for the fixed encoders, to ground-truth sampling.

nuScenes

The PointPillars paper reports no nuScenes results. PointPillars became the LiDAR baseline in the nuScenes paper, whose authors include four of PointPillars’ six (Caesar et al., 2020). nuScenes detection matches predictions by the 2D centre distance on the ground plane, averages AP over thresholds of 0.5, 1, 2 and 4 m and ten classes, and combines mAP with five true-positive errors into the nuScenes detection score (NDS); the bird’s-eye-view perception article gives the definitions. The baseline is a single network for all classes, modified to regress velocity and assigning each class its most common attribute (§4.1). Because the protocol allows the past 0.5 s of data, it accumulates up to ten earlier sweeps, moved into the keyframe’s frame, and adds each point’s time offset as an extra decoration. On the validation set, one sweep gave 31.8 NDS and 21.9 mAP and ten gave 44.8 and 28.8, with the velocity error falling from 1.21 to 0.30 m/s; with ten sweeps, pre-training on KITTI, on ImageNet or not at all gave 44.8, 44.9 and 44.2 NDS (Table 3). On the test set the baseline scored 45.3 NDS and 30.5 mAP, against 38.4 and 30.4 for the monocular MonoDIS (Table 4). PointPillars was stronger on cars (68.4 against 47.8 AP) and pedestrians (59.7 against 37.0), MonoDIS on bicycles (24.5 against 1.1) and traffic cones (48.7 against 30.8), which the authors attribute to bicycles giving few LiDAR returns, cones being easy to see in images, and MonoDIS’s sampling of rare classes. The paper also reports that PointPillars and SSD+3D performed similarly at KITTI-sized amounts of training data and that PointPillars pulled ahead with more (Fig. 6). Its text says MonoDIS had the larger scale errors, but both the mean IoUs it quotes (74% against 71%) and Table 4 (mASE 0.26 against 0.29) give MonoDIS the smaller error.

PointPainting (Vora et al., 2020), from the same group, built an improved baseline, PointPillars+, with pillars of 0.2 m instead of the original 0.25 m, more layers early in the network, attributes estimated from the predicted velocities and heights, sample weighting by annotation count, and global yaw augmentation reduced from $\pi$ to $\pi/6$ (§3.3). On the nuScenes test set it raised mAP from 30.5 to 40.1 and NDS from 45.3 to 55.0, and painting the points with image segmentation scores raised it further to 46.4 mAP and 58.1 NDS (Tables 3 and 4).

Waymo Open Dataset

The Waymo Open Dataset paper also used a re-implemented PointPillars as its 3D detection baseline, trained on single frames from all the vehicle’s LiDARs with 0.33 m pillars over ±85 m, a 512 × 512 pseudo-image, and stride 1 in the first block for vehicles as well as pedestrians (Sun et al., 2020, §5.1). Its results by distance on the test set show how accuracy falls with range: vehicle 3D APH at LEVEL 1 was 81.9 within 30 m, 58.5 at 30–50 m and 34.9 beyond 50 m (Table 6).

Strengths

• Speed from 2D convolutions. All the work after the per-pillar PointNet is dense 2D convolution, and the authors’ timing puts the encoder at 1.3 ms of a 16.2 ms pipeline.
• No vertical binning. A pillar has no height resolution to tune, and the learned encoder needs no hand-designed features, which the authors argue makes it adaptable to other point cloud configurations, such as several LiDAR scans or radar. The nuScenes baseline’s time-offset decoration is an instance; on radar, the nuScenes authors report that a preliminary study with PointPillars did not achieve promising results (Caesar et al., 2020, §4.2).
• A reusable encoder. The pseudo-image can feed any 2D detection architecture or head (§2.3), which is how later work reused it: the nuScenes and Waymo baselines, PointPainting’s painted input and CenterPoint’s center-based head all keep the PointPillars encoder.
• Speed–accuracy trade-off through one parameter. The grid size trades accuracy for speed along the curve of Fig. 5.

Limitations

• Vertical structure is pooled away. One max-pooled vector per pillar must summarize everything above a cell. The encoder ablation shows the learned pillar encoder slightly behind VoxelNet’s voxel encoder at fine resolutions, and later work identified the lack of a sparse encoder as a cause of the remaining gap to detectors built on sparse 3D convolutions (Shi, Li & Ma, 2022, §3.1).
• Caps. Points above $N$ per pillar and pillars above $P$ per sample are discarded; the example shows that the point cap falls on the nearest, densest pillars and that what the pillar cap removes depends on the order of the points.
• Small and thin objects. The failure cases the authors show include false negatives on occluded or distant objects, false positives on similar classes such as vans and trams, pedestrians and cyclists confused with each other, and pedestrians confused with poles and tree trunks (§5, Fig. 4). On nuScenes the baseline reached 1.1 AP on bicycles.
• Range. The Waymo baseline’s vehicle 3D APH fell from 81.9 within 30 m to 34.9 beyond 50 m (Sun et al., 2020, Table 6).
• Anchors and per-class networks. On KITTI one network is trained for cars and another for pedestrians and cyclists, each with hand-set anchors and matching thresholds. Wang et al. (2020) argue that anchors bring hyperparameters and class imbalance, and Yin et al. (2021) that axis-aligned anchors fit rotating objects poorly.
• Evaluation scope. The reported speed processes only the camera’s field of view, and the paper’s timings are on a desktop GPU, as its authors note (§6).

Historical Importance

PointPillars made learned LiDAR encoders fast. VoxelNet had shown in 2018 that an end-to-end learned encoder beats hand-crafted bird’s-eye-view features; PointPillars showed in 2019 that the 3D convolutions in between were not needed for strong results, and that a pillar encoder followed by a 2D detection network could lead the KITTI tables at tens of hertz. Its authors framed the contribution as an encoder, and its title calls it one. The nuScenes paper, PointPainting and the Waymo Open Dataset paper adopted it as their LiDAR baseline, and later pillar detectors such as CenterPoint-Pillar, Pillar-OD and PillarNet are measured against it. The pseudo-image also connected LiDAR detection to the bird’s-eye-view perception of camera and fusion methods, in which a grid on the ground plane is the common frame.

Variants

• PointPillars+ (Vora et al., 2020): the improved nuScenes baseline described above, with 0.2 m pillars and the other changes listed in PointPainting’s §3.3. Painted PointPillars appends image segmentation scores to each point as extra decorations, and on the nuScenes test set its largest per-class gain was 16.8 AP on traffic cones (Table 3). The sensor fusion article describes painting as point-level fusion.
• CenterPoint-Pillar (Yin et al., 2021): keeps the PointPillars encoder (0.32 m pillars on Waymo, 0.2 m on nuScenes) and replaces the anchor head with a CenterNet-style heatmap of object centres. With the PointPillars encoder, the change raised LEVEL 2 mAPH on the Waymo validation set from 57.5 to 62.0 (Table 5) and nuScenes validation mAP from 46.2 to 50.3 and NDS from 59.1 to 60.2 (Table 6).
• Pillar-OD (Wang et al., 2020): predicts box parameters per pillar instead of per anchor, adds a cylindrical view to the bird’s-eye view in multi-view feature learning, as MVF did with a spherical view, and projects pillar features back to points by bilinear rather than nearest-neighbour interpolation. The authors present it as an extension of PointPillars and MVF.
• PillarNet (Shi, Li & Ma, 2022): attributes the gap between pillar and voxel detectors mainly to PointPillars’ lack of a sparse encoder and of a neck for spatial-semantic fusion, notes that tying the output resolution to the pillar size makes fine pillars costly, and adds, after the pillar features, a hierarchical encoder of sparse 2D convolutions and a neck that fuses its outputs, with a center-based head. On the nuScenes validation set, the new encoder raised mAP by 7.61 and NDS by 5.94 over CenterPoint-PointPillars (50.26 mAP and 60.22 NDS, as reported by the code base the authors used; Table 4).

Implementations

The paper’s code is the GitHub repository nutonomy/second.pytorch, a fork of the SECOND code (traveller59/second.pytorch) with the changes for PointPillars; its README says it is not an official nuTonomy codebase but can reproduce the published results, and that it reproduces the first arXiv version of the paper. As of November 2026 the repository is archived, its last commit dates from 6 May 2019, it is under the MIT licence, and its README warns that it is not maintained and recommends the original SECOND repository, which it says can also reproduce PointPillars on nuScenes. The configuration files for the four grid sizes of each network are in second/configs/pointpillars/.

Reading the code at that commit next to the paper shows where they differ:

• Ranges and anchors. The ranges are slightly smaller than the paper’s (above), and the car anchor is 1.56 m tall rather than 1.5. The configuration stores anchor heights of −1.78 m for cars and −1.465 m for pedestrians and cyclists; these equal the paper’s centres minus half the anchor heights, which in our reading means the code stores box bottoms, as its box encoder adds half the height before computing residuals.
• Sampling. The paper says that excess data are randomly sampled. The kernel keeps the first $N$ points of each pillar in the order they arrive and stops at the first point that would open pillar $P + 1$; a docstring tells users to shuffle the points first, and the configurations shuffle during training only.
• Padding. The pillar feature network zeroes the decorations of padded slots, but those zeros still pass through the linear layer, BatchNorm and ReLU, and enter the max as a constant.
• Settings the paper omits. Weight decay of $10^{-4}$, the global rotation and scaling ranges, and the NMS limits are in the configuration only.