Point Cloud Segmentation

Point cloud segmentation assigns a label to every point of a point cloud. In semantic segmentation the label is a class such as floor, chair, road or car; in instance segmentation points are also grouped into individual objects; in part segmentation the points of a single object are labelled with its parts, such as the wings, tail and engines of an aeroplane. It is the 3D counterpart of semantic segmentation of images, sometimes called point-wise classification or 3D semantic parsing, and it is used to interpret indoor scans for mapping and robotics and LiDAR sweeps for driving. Because a point cloud has no grid and no order, the image methods do not carry over directly. Networks either consume the points as a set, as PointNet and its successors do, quantize them into sparse voxels for sparse convolutional networks, project them into a 2D range image, or relate them by attention as point transformers do. Methods are compared on a few benchmarks with their own protocols: S3DIS and ScanNet indoors, SemanticKITTI outdoors, and the ShapeNet part annotations for objects.

Problem Definition

The input is a set of $N$ points $\mathbf{x}_i \in \mathbb{R}^3$, each with optional features $\mathbf{f}_i$ such as colour or LiDAR remission. A semantic segmentation method outputs a class $y_i \in \{1, \dots, C\}$ for every point. An instance segmentation method outputs a set of masks, each a subset of the points with one class and a confidence, usually only for countable “thing” classes; walls and floors are left to the semantic labels. Part segmentation labels the points of one object, whose category is usually given, with one of that category’s parts. Panoptic segmentation, as SemanticKITTI defines it, asks for a semantic label for every point and an instance identity for the points of thing classes (panoptic segmentation in images defines the same split).

Three assumptions shape the problem. The labels must not depend on the order in which points are stored, since a cloud of $N$ points can be stored in $N!$ orders (the PointNet article sets out this requirement). The density of points varies: a spinning LiDAR samples nearby surfaces densely and distant ones sparsely, and SemanticKITTI’s authors found that every baseline they tested got worse with distance from the sensor (their Figure 4). And the ground truth is defined on a particular set of points, which is not always the set a method predicts on: the ScanNet benchmark scores labels on the vertices of each scan’s mesh, and voxel and range-image methods predict on their own grids and must carry their labels back to the points.

Input and Output

Setting	Input points	Labels	Example benchmark
Indoor scans	XYZ and RGB from RGB-D or laser scans of rooms, often millions of points per room	semantic, instance	S3DIS, ScanNet
Outdoor LiDAR	XYZ and remission of one sensor revolution, in the order of $10^5$ points	semantic, panoptic, moving or static	SemanticKITTI
Objects	points sampled from a CAD model, with its category	parts	ShapeNet part

The output is a label per input point, or a set of point masks. Most networks also output per-point class scores, which are needed for instance confidences and for averaging predictions over overlapping crops of large scenes.

Variants

• Semantic segmentation of scenes, the most common benchmark task.
• Instance segmentation, scored by average precision over masks, as for images (instance segmentation).
• Part segmentation of single objects, with the object category known.
• Panoptic and 4D panoptic segmentation. SemanticKITTI’s 4D task asks for instance identities that stay the same over a whole test sequence, so it combines segmentation with tracking (tasks page).
• Multiple-scan segmentation with moving and non-moving classes. Behley et al. (2019) split six movable classes into moving and non-moving versions; the single-scan task merges each pair, because one scan of a LiDAR that cannot measure velocity does not show motion, while the multi-scan task scores all 25 classes.
• Semantic scene completion, which also labels unobserved space, is a different output, a voxel grid; occupancy prediction covers it.

Challenges

• No grid, no order. Convolutions need neighbours at fixed offsets. Points have neighbours at arbitrary offsets, and a network must find them, by k nearest neighbours, a radius search or a voxel hash, before it can compute anything local.
• Density. Sampling density varies with range, incidence angle and sensor. The PointNet++ authors found that features learned on dense regions may not transfer to sparse ones, and designed multi-scale grouping for it; KPConv’s authors, citing earlier work on Monte Carlo point convolutions, hold that k-nearest-neighbour neighbourhoods are not robust to non-uniform sampling, and use radius neighbourhoods with grid subsampling instead (Thomas et al., 2019, §1, §3.1).
• Scale. A rotating 64-beam scanner returns on the order of 130,000 points per scan (Milioto et al., 2019, §III-B), and S3DIS rooms can hold millions of points (Jiang et al., 2020, §4.1). Behley et al. could run PointNet and PointNet++ on SemanticKITTI only after downsampling each scan to 50,000 points because of memory limits (§4.1).
• Class imbalance. Road, sidewalk, building, vegetation and terrain dominate SemanticKITTI’s labels, while the rarest class, motorcyclist, still has more than 100,000 labelled points (Behley et al., 2019, §3.2). Classes with few examples, such as motorcyclists and trucks, seemed more difficult for all baselines, and classes with few points per scan, such as bicycles and poles, were also hard (§4.1).
• Thin and small objects. Voxel networks merge the points of a voxel, and repeated downsampling coarsens the deepest layers; the sparse convolutional networks article describes how SPVNAS’s authors argue this hurts pedestrians and cyclists.
• Labels from reconstructions. Indoor ground truth is painted on reconstructed meshes or fused scans, so it inherits the reconstruction’s errors and its unlabelled regions, which the benchmarks then ignore (ScanNet describes the ignore rules).

Approaches

Before deep learning

Behley et al. (2019, §4.1) summarize the earlier work: point-wise classification with a feature extractor, such as spin images, and a conventional classifier, such as a support vector machine, with conditional random fields added in many methods to make the labels of neighbouring points consistent. Armeni et al. (2016), who introduced the data that became S3DIS, parsed whole buildings top-down: they first split a building into disjoint spaces such as rooms and corridors, detecting walls by a “peak-gap-peak” signature in point density histograms, and then found structural elements and furniture in each space with a 3D sliding-window detector on hand-designed global and local features. They argued that for their applications detecting each element as a whole, despite occlusion and clutter, suits better than labelling every point (§4). With deep learning the feature extractor and classifier were replaced by networks trained end to end, first on voxel grids with 3D convolutions and then on representations that avoid the memory of dense voxels (Behley et al., 2019, §4.1).

Point-based networks

PointNet (Qi et al., 2017) applies one shared multilayer perceptron to every point, max-pools the results into a global feature, and for segmentation concatenates that global feature to each point’s own feature before a per-point classifier. Each point therefore sees its own coordinates and a summary of the whole cloud, but nothing of its neighbourhood, a limitation its authors addressed in PointNet++ (Qi, Yi, Su and Guibas, 2017). PointNet++ groups points around sampled centroids by a ball query, runs a small PointNet on the coordinates of each group relative to its centroid, repeats this at coarser levels, and propagates features back to every point by inverse-distance interpolation; the PointNet article gives the design and its published results on S3DIS, ScanNet and ShapeNet part.

Other point networks build the neighbourhood differently. Graph networks connect each point to its k nearest neighbours and pass messages along the edges. DGCNN (Wang et al., 2019), whose journal paper could not be read for this article, is described by its official repository as proposing a module called EdgeConv for classification and segmentation, and by the Point Transformer authors as performing graph convolutions on kNN graphs. KPConv’s authors draw the distinction that graph convolutions learn filters on edge relationships, whereas a point convolution weights neighbours by their 3D offsets (Thomas et al., 2019, §2).

Point convolutions define a kernel directly in continuous space. Kernel Point Convolution (KPConv; Thomas et al., 2019) places $K$ kernel points $\tilde{\mathbf{x}}_k$ inside a ball of radius $r$, each carrying a weight matrix $W_k$. For a point $\mathbf{x}$ with neighbours $\mathbf{x}_i$ within the radius and offsets $\mathbf{y}_i = \mathbf{x}_i - \mathbf{x}$, the convolution is (Eqs. 1–3)

$$(\mathcal{F} * g)(\mathbf{x}) = \sum_{\lVert \mathbf{x}_i - \mathbf{x} \rVert \le r} \; \sum_{k < K} \max\Big(0,\; 1 - \frac{\lVert \mathbf{y}_i - \tilde{\mathbf{x}}_k \rVert}{\sigma}\Big) \, W_k \, \mathbf{f}_i ,$$

so each neighbour’s feature is multiplied by every kernel weight in proportion to its closeness to that kernel point, with influence distance $\sigma$. The authors chose this linear correlation over a Gaussian to ease gradient propagation when the kernel points move. In the deformable version a rigid KPConv predicts a shift of every kernel point at each location; because shifted kernel points could drift away from the data and stop receiving gradients, a regularizer pulls each kernel point towards its nearest input point and pushes overlapping kernel points apart (§3.2). Density is controlled by grid subsampling: each layer’s support points are the barycentres of the points in the non-empty cells of a grid whose cell size doubles at each pooling layer, and the radius and $\sigma$ scale with the cell size (§3.3). With $K = 15$ kernel points, the segmentation network KP-FCNN is a ResNet-like encoder with a nearest-upsampling decoder and skip links. Scenes too large to process whole are segmented in spheres, picked at random during training and regularly at test time so that every point is predicted several times and its probabilities averaged (§4.2). Rigid kernels scored higher on ModelNet40 classification and on the ScanNet and Semantic3D test sets, deformable kernels on part segmentation, S3DIS and Paris-Lille-3D (Tables 1 and 2). The authors conclude that deformable kernels pay off on large and diverse datasets, explain the Semantic3D result by that dataset’s lack of diversity, and report that on ScanNet their validation experiments favoured deformable kernels although the test score did not (§4.2–4.3).

Voxel and sparse-convolution networks

Voxel networks quantize the cloud into a grid and convolve it. Dense 3D grids grow with the cube of the resolution, so the networks used for segmentation store only occupied voxels and convolve them through precomputed neighbour lists: submanifold sparse convolutions (Graham, Engelcke and van der Maaten, 2018) and generalized sparse convolutions in MinkowskiNet (Choy, Gwak and Savarese, 2019). The sparse convolutional networks article covers the operators, the libraries and the published results; in outline, these networks take whole rooms at once (Choy et al., 2019, §7.3). Their labels are per voxel: voxels whose points carry different labels are ignored in training, and predictions are transferred back to the original points, or to the benchmark’s mesh vertices, after inference. Point–voxel hybrids such as SPVConv (Tang et al., 2020) add a per-point branch at full resolution to recover small objects that coarse voxels merge.

Projection-based networks

A spinning LiDAR fires its beams in a regular pattern, so one revolution maps naturally onto a 2D image with one row per beam and one column per firing angle (LiDAR describes the range image). Projection-based methods label this image with a 2D segmentation network. SqueezeSeg, as Behley et al. and Milioto et al. describe it, projected the scan spherically, applied a lightweight fully convolutional network and smoothed the result with a conditional random field. Behley et al. replaced its backbone with Darknet backbones of 21 and 53 layers and 25 and 50 million parameters, removed its vertical downsampling, and reached 47.4% and 49.9% mIoU against 29.5% for SqueezeSeg, which they took as evidence that the earlier models were limited by their capacity (§4.1).

RangeNet++ (Milioto, Vizzo, Behley and Stachniss, 2019) is the published form of this approach, and its 53-layer backbone, RangeNet53, has the same per-class test scores as DarkNet53Seg in the SemanticKITTI paper (our comparison of the two tables). Each point $(x, y, z)$ with range $r = \lVert \mathbf{p} \rVert$ is mapped to image coordinates by (Eq. 1)

$$u = \tfrac{1}{2}\big[1 - \arctan(y, x)\,\pi^{-1}\big]\, w, \qquad v = \big[1 - (\arcsin(z\, r^{-1}) + f_{\text{up}})\, f^{-1}\big]\, h ,$$

where $f = f_{\text{up}} + f_{\text{down}}$ is the sensor’s vertical field of view and $(h, w)$ the image size. Each pixel stores range, $x$, $y$, $z$ and remission, a $5 \times h \times w$ tensor; points are written in order of decreasing range, so a pixel keeps the closest of the points that fall into it (§III-A, §III-C). The encoder downsamples only horizontally, keeping all 64 rows. Training uses a cross-entropy weighted per class by $1/\log(f_c + \epsilon)$ for class frequency $f_c$ (Eq. 2).

The projection loses points: a scan of 130,000 points rendered at $64 \times 512$ keeps 32,768 of them, and the hour-glass network blurs labels across depth discontinuities, so background points that share a pixel with a foreground object receive its label as a “shadow”. RangeNet++ therefore labels every original point through its saved pixel coordinates, and then replaces each label by a vote among its $k$ nearest neighbours, found among the $S \times S$ pixels around it, ranked by range difference weighted by an inverse Gaussian kernel and cut off beyond a maximum distance (§III-C, §III-D and Algorithm 1). On the SemanticKITTI test set this post-processing raised RangeNet53 from 49.9% to 52.2% mIoU at $64 \times 2048$, and from 39.3% to 41.9% at $64 \times 512$ (Table I). The authors also propose a border-IoU, computed only on points within a given distance of a label change in the range image; the post-processing raised it most for points closest to such changes (Fig. 6).

Transformers

Self-attention is itself a set operator: it does not depend on the order or the number of its inputs, which Zhao et al. (2021) give as the reason why transformers suit point clouds. The Point Transformer applies vector self-attention within the $k$ nearest neighbours $\mathcal{X}(i)$ of each point (Eqs. 3–4):

$$\mathbf{y}_i = \sum_{\mathbf{x}_j \in \mathcal{X}(i)} \rho\big(\gamma(\varphi(\mathbf{x}_i) - \psi(\mathbf{x}_j) + \boldsymbol{\delta})\big) \odot \big(\alpha(\mathbf{x}_j) + \boldsymbol{\delta}\big), \qquad \boldsymbol{\delta} = \theta(\mathbf{p}_i - \mathbf{p}_j),$$

where $\varphi$, $\psi$ and $\alpha$ are linear maps, $\gamma$ and the position encoding $\theta$ are small MLPs, $\rho$ is a softmax, and the attention weights are vectors that scale each feature channel separately, unlike the scalar weights of the original transformer. The network is a U-Net with five stages, farthest point sampling by a factor of 4 between stages and $k = 16$ throughout (§3.5). Its ablations on S3DIS Area 5 (Tables 5–7) show what the design rests on: mIoU fell from 70.4% to 64.6% with scalar instead of vector attention and to 64.6% without position encoding, and $k = 16$ was better than both smaller and larger neighbourhoods (59.6% at $k = 4$, 67.7% at $k = 64$).

Point Transformer V3 (PTv3; Wu et al., 2024) gives up exact neighbourhoods for speed. It sorts the points along space-filling curves, Z-order and Hilbert curves and their variants with permuted axes, by encoding each point’s grid cell as an integer, groups consecutive points of the sorted sequence into patches, and applies ordinary dot-product attention within each patch, changing the curve from one block to the next so that patches cover different neighbourhoods (§4.1–4.2). Positions enter through a sparse convolution with a skip connection before each attention layer instead of relative position encodings. In the authors’ profile of the previous version, PTv2, the kNN search took 28% and the relative position encoding 26% of the forward time (§3, Fig. 2). The authors write that this let them enlarge the attention’s receptive field from 16 to 1,024 points, and they report 3.3 times lower inference latency and 10.2 times lower memory than PTv2 in their Figure 1 benchmark. In their ablation on the ScanNet validation set, mIoU rose with patch size from 75.0% at 16 points to 77.3% at 1,024 (Table 4). Results marked as trained with multi-dataset joint training (Point Prompt Training) are reported beside those trained from scratch.

Instance segmentation

Instance methods group points into objects. SGPN (Wang, Yu, Huang and Neumann, 2018) learns a feature per point with a network inspired by PointNet and PointNet++ (PointNet itself in its S3DIS experiments) and forms an $N_p \times N_p$ similarity matrix of feature distances between all pairs of points; each row is a proposed group, scored by a predicted confidence and given a class by a semantic branch (§3). Following PointNet’s S3DIS set-up, it worked on 1 m × 1 m blocks, with 4,096 points per block in training and all points of the block at test time (§4.1). PointGroup (Jiang et al., 2020) works bottom-up on a sparse-convolution U-Net. A semantic branch labels each point, and an offset branch predicts a vector $\mathbf{o}_i$ that moves each point towards the centroid of its instance, trained with an L1 loss and a loss on the direction (Eqs. 1–3). Points of the same predicted class are then grouped by breadth-first search within a radius $r$, once on the original coordinates and once on the shifted ones $\mathbf{x}_i + \mathbf{o}_i$, since clustering on the original coordinates may merge nearby objects of the same class, while the shifted coordinates suffer from inaccurate offsets, especially at the boundaries of large objects (§3.1, §4.2.2, Algorithm 1). A small network, ScoreNet, scores the candidate clusters, and non-maximum suppression removes duplicates. With 2 cm voxels, $r = 3$ cm and a minimum of 50 points per cluster, PointGroup reported 63.6% AP at IoU 0.5 on the ScanNet v2 test set, against 54.9% for the previous best entry (Table 1, benchmark of 15 November 2019), and 64.0% on S3DIS with six-fold cross-validation (Table 5). PTv3’s instance results use the PointGroup framework with PTv3 as the backbone (Wu et al., 2024, Table 8).

Example: Global Feature Against Local Neighbourhoods

The script below (PyTorch 2.14.1, NumPy 2.5.4, SciPy 1.18.1) trains three small segmentation networks on synthetic room scans and scores them by mIoU on test rooms at three point densities. Each room is axis-aligned, between 4 and 6 m by 3 and 5 m, with 2.4 m walls and no ceiling, in metres with the floor at $z = 0$, $z$ up and the room centred at the origin in $x$ and $y$. It holds one or two columns, one to three cabinets against the walls, and one or two tables with two to four chairs each; the generator tries 50 positions for each object and leaves it out if none fits. Six classes are labelled: floor, wall, column, cabinet, table and chair. Each scan is 4,096 points sampled uniformly by area on the floor, the walls and the faces of the objects, jittered by 1 cm; the floor is left out inside the footprints of columns and cabinets, but the wall is sampled behind a cabinet, whose back face lies in the wall plane, so points there carry both labels. 200 rooms train the networks and 50 test them.

• PointNet (global only) follows PointNet’s segmentation design in its vanilla form: a per-point MLP on the coordinates, a max-pooled global feature, and a per-point classifier on the point feature concatenated with the global feature.
• Local grouping adds one PointNet++-style grouping step at every point: the coordinates of its 16 neighbours relative to the point pass through a small MLP and are max-pooled into a local feature, which joins the point feature before the global pooling.

The sources build neighbourhoods in two ways: PointNet++ by a ball query with a fixed radius, Point Transformer (and DGCNN, as Point Transformer’s authors describe it) by $k$ nearest neighbours. Both are run. kNN takes the 16 nearest points. Ball takes the same 16 but replaces those farther than $r = 0.29$ m by the point itself; $r$ was fixed before the first run from a separate check that measured the median distance to the 16th neighbour in the first 50 training rooms as 0.292 m, so at training density about half of the points keep all 16 neighbours inside the ball (our reasoning from the median). The networks are trained once at full density and tested on the 50 rooms at 100%, 50% and 25% of their points, with neighbourhoods recomputed on each thinned cloud. IoU is computed from one confusion matrix over all test points, as ScanNet’s evaluation script accumulates over its test scans. The script takes the network seed as its argument; seeds 0, 1 and 2 were run and the output of seed 0 is shown. One seed took between about 8 and 40 minutes on a laptop CPU with four threads, depending on the load from other processes.

"""Per-point labels on synthetic rooms: PointNet's global feature against local neighbourhood grouping."""
import sys
import numpy as np
import torch
from torch import nn
from scipy.spatial import cKDTree

CLASSES = ["floor", "wall", "column", "cabinet", "table", "chair"]
H = 2.4  # wall height (m)


def box_faces(x0, x1, y0, y1, z0, z1, label, top=True, bottom=True, sides=True):
    """Faces of an axis-aligned box as (origin, u, v, label)."""
    f = []
    o = np.array
    if top:
        f.append((o([x0, y0, z1]), o([x1 - x0, 0, 0]), o([0, y1 - y0, 0]), label))
    if bottom:
        f.append((o([x0, y0, z0]), o([x1 - x0, 0, 0]), o([0, y1 - y0, 0]), label))
    if sides:
        for x in (x0, x1):
            f.append((o([x, y0, z0]), o([0, y1 - y0, 0]), o([0, 0, z1 - z0]), label))
        for y in (y0, y1):
            f.append((o([x0, y, z0]), o([x1 - x0, 0, 0]), o([0, 0, z1 - z0]), label))
    return f


def make_room(rng, n=4096):
    Lx, Ly = rng.uniform(4, 6), rng.uniform(3, 5)
    hx, hy = Lx / 2, Ly / 2
    faces, cyls, foot, solid = [], [], [], []   # foot: occupied rectangles; solid: no floor inside

    def free(r, margin=0.1):
        x0, x1, y0, y1 = r
        if x0 < -hx or x1 > hx or y0 < -hy or y1 > hy:
            return False
        return all(x1 + margin <= a0 or a1 + margin <= x0 or y1 + margin <= b0 or b1 + margin <= y0
                   for a0, a1, b0, b1 in foot)

    # walls (inner faces)
    o = np.array
    faces += [(o([-hx, -hy, 0]), o([0, Ly, 0]), o([0, 0, H]), 1), (o([hx, -hy, 0]), o([0, Ly, 0]), o([0, 0, H]), 1),
              (o([-hx, -hy, 0]), o([Lx, 0, 0]), o([0, 0, H]), 1), (o([-hx, hy, 0]), o([Lx, 0, 0]), o([0, 0, H]), 1)]
    # cabinets against a wall
    for _ in range(rng.integers(1, 4)):
        for _try in range(50):
            w, d, h = rng.uniform(0.5, 1.0), rng.uniform(0.4, 0.6), rng.uniform(0.8, 1.8)
            side = rng.integers(4)
            if side < 2:   # against x = -hx or x = +hx: depth along x
                y = rng.uniform(-hy, hy - w)
                r = (-hx, -hx + d, y, y + w) if side == 0 else (hx - d, hx, y, y + w)
            else:
                x = rng.uniform(-hx, hx - w)
                r = (x, x + w, -hy, -hy + d) if side == 2 else (x, x + w, hy - d, hy)
            if free(r, 0.05):
                foot.append(r); solid.append(r)
                faces += box_faces(*r, 0, h, 3, bottom=False)
                break
    # columns
    for _ in range(rng.integers(1, 3)):
        for _try in range(50):
            rad = rng.uniform(0.15, 0.25)
            cx, cy = rng.uniform(-hx + 0.4, hx - 0.4), rng.uniform(-hy + 0.4, hy - 0.4)
            r = (cx - rad, cx + rad, cy - rad, cy + rad)
            if free(r, 0.2):
                foot.append(r); solid.append(r)
                cyls.append((cx, cy, rad, 0.0, H, 2))
                break
    # tables with chairs
    for _ in range(rng.integers(1, 3)):
        for _try in range(50):
            w, d, zt = rng.uniform(0.9, 1.4), rng.uniform(0.6, 0.9), rng.uniform(0.70, 0.78)
            cx, cy = rng.uniform(-hx + 1, hx - 1), rng.uniform(-hy + 1, hy - 1)
            r = (cx - w / 2, cx + w / 2, cy - d / 2, cy + d / 2)
            big = (r[0] - 0.6, r[1] + 0.6, r[2] - 0.6, r[3] + 0.6)
            if not free(big, 0.1):
                continue
            foot.append(big)
            faces += box_faces(r[0], r[1], r[2], r[3], zt - 0.05, zt, 4)
            for lx in (r[0] + 0.03, r[1] - 0.08):
                for ly in (r[2] + 0.03, r[3] - 0.08):
                    faces += box_faces(lx, lx + 0.05, ly, ly + 0.05, 0, zt - 0.05, 4, top=False, bottom=False)
            # chairs: one per chosen side, facing the table, back away from it
            sides = rng.permutation(4)[:rng.integers(2, 5)]
            for s in sides:
                zs = rng.uniform(0.42, 0.48)
                if s < 2:      # along x sides (in front of y0 or y1 edge)
                    sx = rng.uniform(r[0], r[1] - 0.45)
                    sy = r[2] - 0.5 if s == 0 else r[3] + 0.05
                    back = (sx, sx + 0.45, sy, sy + 0.05) if s == 0 else (sx, sx + 0.45, sy + 0.40, sy + 0.45)
                else:
                    sy = rng.uniform(r[2], r[3] - 0.45)
                    sx = r[0] - 0.5 if s == 2 else r[1] + 0.05
                    back = (sx, sx + 0.05, sy, sy + 0.45) if s == 2 else (sx + 0.40, sx + 0.45, sy, sy + 0.45)
                seat = (sx, sx + 0.45, sy, sy + 0.45)
                faces += box_faces(*seat, zs - 0.05, zs, 5)
                faces += box_faces(*back, zs, zs + 0.45, 5)
                for lx in (seat[0] + 0.02, seat[1] - 0.06):
                    for ly in (seat[2] + 0.02, seat[3] - 0.06):
                        faces += box_faces(lx, lx + 0.04, ly, ly + 0.04, 0, zs - 0.05, 5, top=False, bottom=False)
            break

    # areas
    floor_area = Lx * Ly - sum((a1 - a0) * (b1 - b0) for a0, a1, b0, b1 in solid)
    areas = [floor_area] + [np.linalg.norm(np.cross(u, v)) for _, u, v, _ in faces] + \
            [2 * np.pi * c[2] * (c[4] - c[3]) for c in cyls]
    areas = np.array(areas)
    counts = rng.multinomial(n, areas / areas.sum())
    pts, lab = [], []
    # floor with rejection inside solid footprints
    k = counts[0]
    fl = np.empty((0, 2))
    while len(fl) < k:
        c = np.c_[rng.uniform(-hx, hx, 2 * k + 10), rng.uniform(-hy, hy, 2 * k + 10)]
        inside = np.zeros(len(c), bool)
        for a0, a1, b0, b1 in solid:
            inside |= (c[:, 0] > a0) & (c[:, 0] < a1) & (c[:, 1] > b0) & (c[:, 1] < b1)
        fl = np.vstack([fl, c[~inside]])
    pts.append(np.c_[fl[:k], np.zeros(k)]); lab.append(np.zeros(k, int))
    for (orig, u, v, l), m in zip(faces, counts[1:1 + len(faces)]):
        st = rng.uniform(0, 1, (m, 2))
        pts.append(orig + st[:, :1] * u + st[:, 1:] * v); lab.append(np.full(m, l))
    for (cx, cy, rad, z0, z1, l), m in zip(cyls, counts[1 + len(faces):]):
        t = rng.uniform(0, 2 * np.pi, m)
        pts.append(np.c_[cx + rad * np.cos(t), cy + rad * np.sin(t), rng.uniform(z0, z1, m)]); lab.append(np.full(m, l))
    P = np.vstack(pts) + rng.normal(scale=0.01, size=(n, 3))
    Y = np.concatenate(lab)
    perm = rng.permutation(n)
    return P[perm].astype(np.float32), Y[perm]


def make_rooms(count, seed, n=4096):
    rng = np.random.default_rng(seed)
    rooms = [make_room(rng, n) for _ in range(count)]
    return np.stack([r[0] for r in rooms]), np.stack([r[1] for r in rooms])


torch.set_num_threads(4)
C, K, R = len(CLASSES), 16, 0.29
seed = int(sys.argv[1]) if len(sys.argv) > 1 else 0     # network seed: 0, 1 and 2 were run
epochs = 40


def neighbours(X, mode):
    """(B, n, K) indices of the K nearest points; for 'ball', those beyond R become the point itself."""
    out = []
    for x in X:
        d, idx = cKDTree(x).query(x, K)
        if mode == "ball":
            idx = np.where(d <= R, idx, idx[:, :1])
        out.append(idx)
    return torch.tensor(np.stack(out), dtype=torch.long)


def mlp(*s):
    layers = []
    for a, b in zip(s[:-1], s[1:]):
        layers += [nn.Linear(a, b), nn.ReLU()]
    return nn.Sequential(*layers)


class Seg(nn.Module):
    def __init__(self, local):
        super().__init__()
        self.local = local
        self.point = mlp(3, 64, 64)
        d = 64
        if local:
            self.group = mlp(3, 64, 64)
            d = 128
        self.lift = mlp(d, 128, 256)
        self.head = nn.Sequential(mlp(d + 256, 256, 128), nn.Linear(128, C))

    def forward(self, X, idx=None):                      # X (B, n, 3), idx (B, n, K)
        f = self.point(X)
        if self.local:
            nb = torch.gather(X[:, None].expand(-1, X.shape[1], -1, -1), 2,
                              idx[..., None].expand(-1, -1, -1, 3))          # (B, n, K, 3)
            f = torch.cat([f, self.group(nb - X[:, :, None]).max(2).values], -1)
        g = self.lift(f).max(1, keepdim=True).values     # global feature (B, 1, 256)
        return self.head(torch.cat([f, g.expand(-1, X.shape[1], -1)], -1))


def miou(y, p):
    cm = np.bincount(C * y.ravel() + p.ravel(), minlength=C * C).reshape(C, C)
    iou = np.diag(cm) / (cm.sum(0) + cm.sum(1) - np.diag(cm))
    return iou.mean(), iou


Xtr, Ytr = make_rooms(200, seed=100)                   # 200 training rooms
Xte, Yte = make_rooms(50, seed=200)                    # 50 test rooms
methods = {"PointNet (global only)": None, "local, kNN": "knn", "local, ball r=0.29": "ball"}
idx_tr = {m: neighbours(Xtr, m) for m in ("knn", "ball")}

# test sets at three densities (same subsets for every method)
rng = np.random.default_rng(1)
tests = {}
for frac in (1.0, 0.5, 0.25):
    n = int(4096 * frac)
    sub = np.stack([rng.permutation(4096)[:n] for _ in range(len(Xte))])
    X = np.take_along_axis(Xte, sub[..., None], 1); Y = np.take_along_axis(Yte, sub, 1)
    tests[frac] = (torch.tensor(X), Y, {m: neighbours(X, m) for m in ("knn", "ball")})

Xtr_t, Ytr_t = torch.tensor(Xtr), torch.tensor(Ytr)


def predict(net, X, idx):
    with torch.no_grad():
        return torch.cat([net(X[i:i + 10], None if idx is None else idx[i:i + 10]).argmax(-1)
                          for i in range(0, len(X), 10)]).numpy()


for name, mode in methods.items():
    torch.manual_seed(seed); g = torch.Generator().manual_seed(seed)
    net = Seg(local=mode is not None)
    opt = torch.optim.Adam(net.parameters(), lr=1e-3)
    for ep in range(epochs):
        perm = torch.randperm(len(Xtr), generator=g)
        for i in range(0, len(Xtr), 8):
            b = perm[i:i + 8]
            opt.zero_grad()
            out = net(Xtr_t[b], None if mode is None else idx_tr[mode][b])
            nn.functional.cross_entropy(out.reshape(-1, C), Ytr_t[b].reshape(-1)).backward()
            opt.step()
    tr = miou(Ytr, predict(net, Xtr_t, None if mode is None else idx_tr[mode]))[0]
    row = []
    for frac, (X, Y, idx) in tests.items():
        row.append(miou(Y, predict(net, X, None if mode is None else idx[mode])))
    npar = sum(p.numel() for p in net.parameters())
    print(f"seed {seed} {name:24s} {npar:7d} params, train mIoU {tr:.3f} | test mIoU at 100/50/25%: "
          + " ".join(f"{r[0]:.3f}" for r in row), flush=True)
    for frac, r in zip(tests, row):
        print(f"    {frac:>4.0%} per class " + " ".join(f"{c} {v:.2f}" for c, v in zip(CLASSES, r[1])))

Output (seed 0):

seed 0 PointNet (global only)    161606 params, train mIoU 0.571 | test mIoU at 100/50/25%: 0.565 0.562 0.562
    100% per class floor 0.96 wall 0.86 column 0.50 cabinet 0.14 table 0.52 chair 0.42
     50% per class floor 0.96 wall 0.86 column 0.49 cabinet 0.13 table 0.51 chair 0.42
     25% per class floor 0.96 wall 0.85 column 0.48 cabinet 0.13 table 0.53 chair 0.42
seed 0 local, kNN                190598 params, train mIoU 0.693 | test mIoU at 100/50/25%: 0.683 0.653 0.575
    100% per class floor 0.96 wall 0.88 column 0.61 cabinet 0.32 table 0.74 chair 0.57
     50% per class floor 0.96 wall 0.88 column 0.59 cabinet 0.28 table 0.74 chair 0.47
     25% per class floor 0.94 wall 0.88 column 0.49 cabinet 0.21 table 0.74 chair 0.20
seed 0 local, ball r=0.29        190598 params, train mIoU 0.677 | test mIoU at 100/50/25%: 0.668 0.637 0.605
    100% per class floor 0.96 wall 0.88 column 0.59 cabinet 0.33 table 0.72 chair 0.52
     50% per class floor 0.95 wall 0.86 column 0.54 cabinet 0.29 table 0.71 chair 0.47
     25% per class floor 0.94 wall 0.82 column 0.51 cabinet 0.24 table 0.66 chair 0.46

Across seeds 0, 1 and 2, test mIoU at full density was 0.565–0.584 for PointNet, 0.670–0.700 for kNN grouping and 0.668–0.672 for ball grouping. At 25% density it was 0.562–0.579, 0.559–0.575 and 0.587–0.605.

What local structure adds. Floor and wall are easy for every network, since the floor is at $z = 0$ and the walls lie at the room’s extent, which in our reading the global maximum can encode. The four object classes are where local grouping helps: at full density it raises the IoU of column, cabinet, table and chair by 0.09 to 0.22 each for seed 0. A PointNet point sees only its own coordinates and a summary of the whole room. A point 0.3 m above the floor may lie on a chair leg, a table leg, a column or a cabinet side, and in our reading the global feature, a single 256-vector per room, cannot record where each of several randomly placed objects stands. The grouped offsets show the shape around the point: a flat patch, a curved one or a thin vertical one. The gain, about 0.1 mIoU, is smaller than the hypothesis in our plan expected (“by a wide margin”). At full density cabinets stay the hardest class for all three networks. Part of every cabinet, its back face, shares the wall plane with wall points by construction; in our reading their sides also look locally like walls and their tops like table tops.

Density. PointNet’s scores hardly move when three quarters of the points are removed: its per-point features do not depend on other points, and only the global maximum can change. The local networks lose accuracy, the kNN network most, from 0.683 to 0.575 for seed 0, with chairs falling from 0.57 to 0.20. On a surface, the radius that holds 16 points grows with the inverse square root of the density, so at 25% the 16 nearest neighbours span about twice the radius they had in training (our reasoning). The network then sees offsets of a scale it never learned, and in our reading a neighbourhood around a chair point now reaches into the table and the floor. The ball neighbourhood keeps its 0.29 m scale and instead contains fewer real points, padded with the centre; it lost less, and stayed above PointNet at 25% for all three seeds, while the kNN network fell to PointNet’s level or below it (above it by 0.013 for seed 0, below it for seeds 1 and 2). The result is in line with the PointNet++ and KPConv authors’ concern about non-uniform density and with KPConv’s preference for radius neighbourhoods; the example trains at one density only and does not test their remedies, multi-scale grouping with random input dropout, or grid subsampling.

Limits of the example. The rooms are clean synthetic surfaces without colour, the networks are small and trained for 40 epochs from a single data set, and the neighbourhood is a single scale. The ranking of kNN and ball grouping at full density differs between seeds, so the example does not separate them there.

Datasets and Benchmarks

S3DIS. The Stanford Large-Scale 3D Indoor Spaces dataset grew out of Armeni et al. (2016), whose CVPR paper describes five areas of three buildings, about 6,020 m² in all, scanned with a Matterport scanner and annotated with 12 semantic elements: ceiling, floor, wall, beam, column, window, door, table, chair, sofa, bookcase and board. The released version used by later work has six areas; Point Transformer’s authors describe it as 271 rooms from three buildings with every point labelled with one of 13 categories, the 12 elements plus clutter (Zhao et al., 2021, §4.1), and KPConv’s as 273 million points (Thomas et al., 2019, §4.2). (The ScanNet article, following Table 1 of the ScanNet paper, gives 265 rooms.) Two protocols are used:

• Six-fold cross-validation: one model per area, trained on the other five and tested on it, with the results combined over the six test areas. The PointNet repository’s README describes this protocol and notes that the 3-fold split of the S3DIS paper was not public at the time.
• Area 5: train on areas 1–4 and 6, test on area 5. KPConv’s authors, following SegCloud, recommend it to better measure generalization (§4.2), and MinkowskiNet’s call it Fold #1 (Choy et al., 2019, §7.3 and Table 4).

The metrics are mean IoU over the 13 classes, mean class accuracy and overall point accuracy. Results from the two protocols are not comparable. Rooms are large and dense, so networks train on crops, such as PointNet’s 1 m × 1 m blocks of 4,096 sampled points, KPConv’s spheres, or PointGroup’s crops of at most 250,000 points after subsampling S3DIS scenes to about a quarter of their points (§4.1).

Reported Area 5 results, from the tables of Zhao et al. (2021, Table 1) and Wu et al. (2024, Table 5), all as mIoU over 13 classes:

Method	Family	Area 5 mIoU (%)
PointNet	point MLP	41.1
SPGraph	superpoint graph	58.0
MinkowskiNet	sparse voxels	65.4
KPConv (deformable)	point convolution	67.1
Point Transformer	local vector attention	70.4
PTv3, trained from scratch	serialized attention	73.4
PTv3, multi-dataset training	serialized attention	74.7

The PointNet row is the Area 5 figure in Zhao et al.’s table; the PointNet paper itself reports the six-fold protocol, which its article gives.

ScanNet. ScanNet (Dai et al., 2017) has 1,513 RGB-D scans of indoor rooms reconstructed into meshes; version 2 splits them 1,201 for training and 312 for validation, with 100 test scans whose labels are hidden on the benchmark server. The 3D semantic label task scores 20 classes on the vertices of each scan’s mesh, from one confusion matrix over all test scans; the instance task scores 18 classes, without wall and floor, by AP at IoU 0.25 and 0.5 and averaged over thresholds. That article sets out the ignore rules and the details of the official scripts, which affect what a reported number means. Papers report either the hidden test set or the validation set: PTv3 reports 77.5% validation and 77.9% test mIoU trained from scratch (Table 5), and KPConv 68.6% test with rigid kernels (Table 2).

SemanticKITTI. Behley et al. (2019) annotated every scan of the 22 sequences of the KITTI odometry benchmark, recorded with a Velodyne HDL-64E around Karlsruhe, with dense per-point labels over the full 360° field of view: 23,201 scans for training (sequences 00–10) and 20,351 for testing (11–21), whose labels are withheld for server-side evaluation. To label consistently, the annotators worked on scans superimposed using poses from a laser-based SLAM system, in tiles of 100 m × 100 m, for over 1,700 hours in all including verification (§3.1). Of 28 annotated classes, the benchmark drops outlier, other-structure and other-object, leaving 25; the single-scan task merges moving and non-moving classes into 19, and the multi-scan task keeps all 25. Inputs are coordinates and remission, and a label is required for every point of each scan. The metric is mIoU over classes, $\frac{1}{C}\sum_c \mathrm{TP}_c / (\mathrm{TP}_c + \mathrm{FP}_c + \mathrm{FN}_c)$. In the paper’s baselines sequence 08 served as validation set (Table 2), and the official site adds that test submissions go through CodaBench competitions and that the leaderboard lists the numbers from each method’s paper (tasks page, November 2026). Test-set single-scan mIoU, as published:

Method	Representation	mIoU (%)	Source
PointNet	points (downsampled to 50,000)	14.6	Behley et al., Table 2
PointNet++	points (downsampled to 50,000)	20.1	Behley et al., Table 2
SqueezeSeg	range image	29.5	Behley et al., Table 2
TangentConv	tangent planes	40.9	Behley et al., Table 2
DarkNet53Seg / RangeNet53	range image $64 \times 2048$	49.9	both papers
RangeNet53++	range image + kNN	52.2	Milioto et al., Table I
SPVNAS	sparse point–voxel	66.4	Wu et al., Table 7
PTv3, from scratch	serialized attention	74.2	Wu et al., Table 7

The two 2019 papers print different test scores for three of the same baselines, for example 17.4% (Behley et al.) and 20.0% (Milioto et al.) for SPGraph and 40.9% and 35.9% for TangentConv; neither explains the difference. The sparse convolutional networks article gives the MinkowskiNet and SPVNAS results from the SPVNAS paper.

ShapeNet part. The part annotations of Yi et al. (2016) on 16 ShapeNetCore categories, with 50 parts in total and two to six per category, are the standard part segmentation benchmark; the category is given to the network. The ShapeNet article describes the data, the splits and the two copies used by the PointNet and PointNet++ code. Papers count the shapes differently: 16,881 in PointNet, 16,880 with 14,006 for training and 2,874 for testing in Point Transformer (§4.3), and 16,681 in KPConv (§4.1). Reported instance mIoU ranges from 83.7% for PointNet through 85.1% for PointNet++ and 86.4% for deformable KPConv to 86.6% for Point Transformer (Zhao et al., 2021, Table 4), differences of a few tenths between the stronger methods.

Others. KPConv’s authors also evaluate on Semantic3D, fixed terrestrial laser scans with more than 4 billion points in 8 classes, and Paris-Lille-3D, mobile scans of more than 2 km of streets in 10 classes (§4.2); PTv3’s tables add ScanNet200, nuScenes and the Waymo Open Dataset.

Evaluation Metrics

• Mean IoU. The standard measure for semantic segmentation is the intersection over union of each class, $\mathrm{TP}_c / (\mathrm{TP}_c + \mathrm{FP}_c + \mathrm{FN}_c)$, averaged over classes. ScanNet’s script accumulates one confusion matrix over all test scans, so large scans count more than small ones; ignored labels and ignored classes differ between benchmarks.
• Accuracy. S3DIS papers also report overall point accuracy (OA) and mean class accuracy (mAcc). Overall accuracy weights classes by their number of points, so in our reading it is dominated by walls, floors and ceilings indoors, while mIoU weights every class equally, which is why papers lead with it.
• Part IoU. ShapeNet part scores each shape by the mean IoU over its category’s parts, counting a part absent from both prediction and ground truth as 1, then averages over shapes (instance mIoU) or over categories (class mIoU); KPConv’s Table 1 prints both, and the ShapeNet article shows how they differ.
• Instance AP. Mask AP at point or vertex IoU thresholds of 0.25 and 0.5, and averaged over higher thresholds, per class and then averaged (average precision); S3DIS papers often add mean precision and recall at IoU 0.5 (Jiang et al., 2020, §4.1).
• Panoptic quality. SemanticKITTI’s panoptic task uses PQ, with a match counted as true positive above IoU 0.5, and its 4D task the LiDAR Segmentation and Tracking Quality, LSTQ (tasks page).
• Diagnostics. Behley et al. plot mIoU against distance from the sensor (Figure 4), and Milioto et al. score only points near label changes (border-IoU), both to expose errors that the overall mean hides.

Reported numbers are comparable only within one protocol: S3DIS Area 5 against six-fold, ScanNet validation against test, a method trained from scratch against one trained on several datasets, single-scan against multi-scan. The comparison tables of later papers do not always agree with the papers they cite. PTv3’s Table 5 lists Point Transformer’s six-fold S3DIS mIoU as 65.4%, against 73.5% in Point Transformer’s own Table 2, and KPConv’s and Point Transformer’s tables give different ShapeNet part scores for DGCNN (84.7% and 85.1% instance mIoU).

Applications

• Driving and mobile robots. Per-point labels of LiDAR scans separate drivable ground, sidewalks, vehicles, people and vegetation; SemanticKITTI was built for this setting, and Milioto et al. designed RangeNet++ to run at the sensor’s frame rate. Autonomous driving perception and sensor fusion describe how such outputs are used together with cameras.
• Indoor mapping and buildings. Labelling scans of rooms and buildings into structural elements and furniture, the setting of Armeni et al. (2016), for building analysis and indoor maps.
• Robotics. Segmented points of objects and support surfaces feed grasping and placement in robotic manipulation.
• Detection and completion. Segmentation backbones are also used for 3D detection: PTv3 reports Waymo detection results with the same backbone (Wu et al., 2024, Table 10). Occupancy prediction labels whole voxel grids, including unobserved space.

Open Problems

• Sparse and distant points. All SemanticKITTI baselines lost accuracy with range, and rare or thin classes such as motorcyclists, trucks, bicycles and poles stayed hard (Behley et al., 2019, §4.1).
• Motion. The multi-scan baselines in Behley et al. (Table 4) struggled to separate moving from non-moving objects, and never assigned the non-moving class correctly to bicyclists and motorcyclists.
• Efficiency at scale. PTv3’s design trades exact neighbourhoods for speed and memory so that larger models fit, and its authors argue that model performance depends more on scale than on intricate design (abstract).
• Data. Labelled 3D data are scarce and domain-specific: indoor RGB-D and outdoor LiDAR differ in density and accuracy (Behley et al., 2019, §2), and PTv3’s best results use joint training on several datasets. How far one model transfers across sensors and scene types is open.
• Comparable evaluation. Two protocols on S3DIS, hidden and public test splits, and differing numbers for the same baseline in different papers make cross-paper comparison harder than the leaderboards suggest.