PointNet

PointNet is a deep network that takes a point cloud as it comes, an unordered set of 3D points, and outputs either one class label for the whole cloud or one label per point. Charles R. Qi, Hao Su, Kaichun Mo and Leonidas J. Guibas of Stanford introduced it at CVPR 2017, after an arXiv report in 2016. Its core is simple: the same multilayer perceptron is applied to every point independently, and the per-point features are combined by max pooling, which gives the same result for every ordering of the points. The same paper uses one architecture for shape classification, object part segmentation and semantic segmentation of indoor scenes. PointNet++ (Qi, Yi, Su and Guibas, NIPS 2017) applies PointNet recursively to nested local neighbourhoods, so that features capture local structure at increasing scales, as the layers of a convolutional neural network do on images. Both are fully connected networks on sets rather than convolutional networks on grids.

Problem Addressed

Convolutional networks need regular inputs, such as pixel grids or voxel grids, to share weights across positions. A point cloud has neither a grid nor an order: its points can be stored in any of $N!$ orders, and the spacing between them varies across the scene. Before PointNet, as its authors summarize the field, deep networks on 3D data therefore converted the input first. Volumetric CNNs such as 3D ShapeNets (Wu et al., 2015) and VoxNet ran 3D convolutions on a voxel grid, and multi-view CNNs such as MVCNN rendered the shape into 2D images and applied image networks. The PointNet authors argue that the conversion makes the data “unnecessarily voluminous” and introduces quantization artifacts, that volumetric networks are limited in resolution by sparsity and the cost of 3D convolution, and that multi-view methods are hard to extend to scene understanding or to per-point tasks such as point classification. The 3D representations article compares these representations in general.

The PointNet paper (§4.1) names three properties of a point set that a network must respect:

• Unordered. The output must be invariant to the $N!$ permutations of the input points.
• Interaction among points. Points lie in a metric space, so neighbouring points form meaningful subsets, and the model should capture local structure.
• Invariance under transformations. Rotating and translating the whole cloud should not change its class or the labels of its points.

In our reading, PointNet meets the first property exactly, the third approximately with a learned alignment, and the second only through its global feature; PointNet++ was designed for the second.

Architecture

The classification network (the paper’s Figure 2) takes $n$ points as an $n \times 3$ array of coordinates. An input transform aligns the points; a shared MLP with layer sizes (64, 64) lifts each point to 64 features; a feature transform aligns these features; a second shared MLP (64, 128, 1024) lifts each point to 1,024 features; max pooling over the points gives a 1,024-dimensional global feature; and an MLP (512, 256, $k$) outputs scores for $k$ classes. Every layer has batch normalization and a ReLU, and the last MLP uses dropout. “Shared” means that the same weights are applied to each point separately, which the official TensorFlow code implements as 2D convolutions whose kernels cover one point at a time.

Shared per-point MLP

The per-point MLP $h$ maps each point $\mathbf{x}_i \in \mathbb{R}^3$ to a feature vector $h(\mathbf{x}_i) \in \mathbb{R}^K$, with $K = 1{,}024$ in the paper. In the basic setting a point is represented by its three coordinates only; other channels such as colour or normals can be appended. Because $h$ sees one point at a time, nothing up to the pooling layer depends on the order of the points, and the cost is linear in $n$. Each channel $h_j$ is a function over space, which the paper calls a point function; the authors describe the network as learning criteria that select the informative points of the cloud and encode why they were selected (§1).

Max pooling as a symmetric function

A function of $n$ vectors is symmetric if its value does not depend on their order; addition and the element-wise maximum are examples. PointNet approximates a general set function $f$ by applying a symmetric function to transformed elements (the paper’s Eq. 1),

$$f(\{\mathbf{x}_1, \dots, \mathbf{x}_n\}) \approx \gamma\Big(\max_{i = 1, \dots, n} h(\mathbf{x}_i)\Big),$$

where the maximum is taken separately in each of the $K$ channels and $\gamma$ is the classification MLP. The authors considered two other routes to order invariance (§4.2). Sorting the points into a canonical order cannot work in general, they argue, because an ordering of a high-dimensional space that is stable under small perturbations would amount to a map onto the real line that preserves proximity, which does not exist. Training a recurrent network on randomly permuted sequences makes it only approximately invariant and is hard to scale to the thousands of elements in a point cloud.

Their control experiment on ModelNet40 (Figure 5, overall accuracy) compares these options:

Aggregation	Accuracy (%)
MLP on unsorted points	24.2
MLP on sorted points	45.0
LSTM	78.5
Attention-weighted sum	83.0
Average pooling	83.8
Max pooling	87.1

The per-point MLP in this comparison has five hidden layers (64, 64, 64, 128, 1024) and the final MLP (512, 256); the figure shows no transforms, and its max-pooling accuracy equals the no-transform row of Table 5.

The theory comes with a universal-approximation result (Theorem 1): any function on sets of $n$ points in $[0, 1]^m$ that is continuous with respect to the Hausdorff distance between point sets can be approximated arbitrarily well by $\gamma \circ \max \circ h$, given a large enough $K$. The proof is in the supplementary material; the main text gives its key idea, that in the worst case the network can learn to convert the cloud into a volumetric representation by partitioning space into equal voxels, and adds that in practice the network learns a “much smarter strategy”. Zaheer et al. (2017), in Deep Sets, published the same year, proved that every permutation-invariant function of a set drawn from a countable universe has the form $\rho\big(\sum_{x} \phi(x)\big)$, a sum instead of a maximum; for uncountable universes such as $\mathbb{R}$ they could prove it only for sets of fixed size. They reported 90 ± 0.3% on ModelNet40 with 5,000 points per cloud.

Input and feature transforms

The class of a shape should not depend on its pose. PointNet adds a joint alignment network, in the spirit of spatial transformer networks for images, that predicts a transformation from the cloud itself and applies it to the coordinates. The mini-network, called T-net, resembles the main network and is built from the same modules: a shared MLP, max pooling and fully connected layers. In the official code (transform_nets.py), it applies a shared MLP (64, 128, 1024), max pooling and fully connected layers (512, 256), and its last layer, which outputs the 9 entries of a $3 \times 3$ matrix, has weights and biases initialized to zero with the identity added as a constant, so that training begins from the identity transform. The paper calls the predicted matrix an affine transformation; it is applied by matrix multiplication of the point coordinates.

A second T-net predicts a $64 \times 64$ matrix $A$ that is applied to the 64-dimensional point features after the first shared MLP. Because this matrix has many more parameters, the authors add a regularization term to the softmax classification loss that pushes it towards an orthogonal matrix, which loses no information (Eq. 2):

$$L_{\text{reg}} = \lVert I - A A^\top \rVert_F^2 .$$

The ablation (Table 5, ModelNet40 overall accuracy) shows that the transforms help together but not separately: 87.1% with no transform, 87.9% with the input transform, 86.9% with the feature transform alone, 87.4% with the feature transform and the regularizer, and 89.2% with both transforms and the regularizer. The transforms make the network neither rotation-invariant nor guaranteed to align shapes; they are learned from the training data and its augmentations.

Segmentation network

Segmentation needs per-point outputs that depend both on the point and on the shape it belongs to. The segmentation network concatenates the 1,024-dimensional global feature to each point’s 64-dimensional feature from after the feature transform, giving $n \times 1{,}088$, and applies shared MLPs (512, 256, 128) and (128, $m$) to output $m$ scores per point. The authors report that this network can also predict per-point normals, which they take as evidence that it summarizes each point’s neighbourhood (the figure is in the supplementary). The part segmentation version is “a modified version” of this network, detailed in the supplementary; the ShapeNet article describes how the official code feeds it the object category as a one-hot vector.

Critical point set

Max pooling has a consequence that the authors analyse in Theorem 2. Write $\mathbf{u}(S) = \max_{\mathbf{x} \in S} h(\mathbf{x})$ for the global feature of a set $S$ and $f = \gamma \circ \mathbf{u}$. Then for every $S$ there are two sets $\mathcal{C}_S \subseteq S \subseteq \mathcal{N}_S$ such that

$$f(T) = f(S) \quad \text{whenever } \mathcal{C}_S \subseteq T \subseteq \mathcal{N}_S, \qquad |\mathcal{C}_S| \le K .$$

The critical point set $\mathcal{C}_S$ holds the points that attain the maximum in at least one channel; each of the $K$ channels has one winner, so there are at most $K$ of them. It follows from the definition that deleting a non-critical point leaves every channel’s maximum in place, and that adding a point changes nothing unless it raises some channel above the current maximum. The upper-bound shape $\mathcal{N}_S$ is the set of all points whose channel values $h_1(\mathbf{p}), \dots, h_K(\mathbf{p})$ are no larger than the global feature; the authors construct it by passing every point of a cube of edge length 2 through the network. They call $K$ the bottleneck dimension, and take the theorem, with the continuity of $h$, as the explanation of PointNet’s robustness to perturbation, corruption and extra noise points. Their Figure 7 shows critical point sets that, in their words, summarize the skeleton of the shape.

Training

For ModelNet40 classification, the authors sample 1,024 points uniformly on the mesh faces according to face area and normalize them into a unit sphere. During training each cloud is rotated by a random angle about the up axis and each point jittered by Gaussian noise with zero mean and standard deviation 0.02, a form of data augmentation. The loss is the softmax cross-entropy plus the regularization term on the feature transform.

The defaults of the official train.py and pointnet_cls.py (commit 2618f72) fill in what the paper leaves out: Adam with learning rate 0.001, batch size 32, 250 epochs, the learning rate multiplied by 0.7 every 200,000 training samples with a floor of 0.00001, dropout with keep probability 0.7 after the first two fully connected layers, and a regularization weight of 0.001. The code computes the regularizer with TensorFlow’s l2_loss, which is half the sum of squares, over the whole batch, so the term added to the mean cross-entropy is $0.0005 \sum_b \lVert A_b A_b^\top - I \rVert_F^2$ (our reading of the code).

For semantic segmentation on S3DIS, the rooms are split into blocks of 1 m by 1 m, and each point is a 9-dimensional vector of XYZ, RGB and its location normalized to the room (from 0 to 1). During training 4,096 points are sampled per block on the fly; at test time every point is labelled.

Inference

The network accepts any number of points, because the shared MLP is applied per point and max pooling reduces any number of rows to one. The paper’s Table 6 counts parameters and floating-point operations per sample for classification:

Model	Parameters	FLOPs per sample
PointNet (vanilla, no transforms)	0.8M	148M
PointNet	3.5M	440M
Subvolume (3D CNN)	16.6M	3,633M
MVCNN	60.0M	62,057M

PointNet’s time and space grow linearly with the number of points. The authors report that on a GPU (their “1080X”) in TensorFlow it processed more than a million points per second, about 1,000 objects per second for classification or about two rooms per second for semantic segmentation. The table’s note adds that Subvolume and MVCNN pooled over multiple rotations or views, without which their accuracy was much lower.

Example: Order, Dropout, Outliers and Critical Points

The script below (PyTorch 2.14.1, NumPy 2.5.4) trains a small PointNet and two MLPs that read the concatenated coordinates of all points, on synthetic point clouds of four shapes with known labels: sphere, cube, cylinder with caps and cone with base. Each shape is sampled with 256 points on its surface in proportion to area, stretched by a random factor between 0.6 and 1.4 along each axis, turned by a random angle about the up axis $z$, jittered, centred and scaled into the unit ball; 400 shapes per class train the networks and 100 per class test them. The PointNet is the vanilla design without transforms: a shared MLP (3, 64, 128, 256), max pooling, and an MLP (256, 128, 4), without batch normalization or dropout. The MLP baseline flattens the $256 \times 3$ array into a 768-vector, with layers (768, 256, 256, 4).

An MLP on coordinates must be given the points in some order, which the sources leave open, so the script trains it both ways: once with the points of every shape sorted by height $z$, which turns about $z$ leave unchanged, and once in a fresh random order every epoch. Every network is then tested on the clean test set, after random point dropout that keeps 50% or 25% of each shape’s points, with 10% or 25% of the points replaced by uniform outliers in the unit ball, and with each test shape’s points in a random order. PointNet can take the remaining points directly; the MLP needs exactly 256 points, so the dropped slots are filled either by repeating the kept points or with zeros, another open choice that is printed both ways. The last part extracts the critical point sets of the PointNet and an upper-bound set on a grid. A second PointNet, trained with outliers, is described after the output. All four networks are trained once, with seed 0, and the script runs in about 20 seconds on a laptop CPU with four threads.

import numpy as np
import torch
from torch import nn

torch.set_num_threads(4)
N = 256                                   # points per shape
NAMES = ["sphere", "cube", "cylinder", "cone"]

def sample_shape(c, rng, n=N):
    """n points on the surface of one shape, area-weighted, then scaled, turned and jittered."""
    if c == 0:    # sphere
        p = rng.normal(size=(n, 3)); p /= np.linalg.norm(p, axis=1, keepdims=True)
    elif c == 1:  # cube surface
        face = rng.integers(0, 6, n); uv = rng.uniform(-1, 1, (n, 2))
        p = np.empty((n, 3)); ax = face // 2; sgn = np.where(face % 2 == 0, -1.0, 1.0)
        for i in range(n):
            others = [a for a in range(3) if a != ax[i]]
            p[i, ax[i]] = sgn[i]; p[i, others] = uv[i]
    elif c == 2:  # cylinder with caps
        side = rng.uniform(0, 1, n) < 4 / 6; th = rng.uniform(0, 2 * np.pi, n)
        r = np.where(side, 1.0, np.sqrt(rng.uniform(0, 1, n)))
        z = np.where(side, rng.uniform(-1, 1, n), np.where(rng.uniform(0, 1, n) < 0.5, -1.0, 1.0))
        p = np.stack([r * np.cos(th), r * np.sin(th), z], 1)
    else:         # cone with base
        lat = rng.uniform(0, 1, n) < np.sqrt(5) / (np.sqrt(5) + 1); th = rng.uniform(0, 2 * np.pi, n)
        t = np.sqrt(rng.uniform(0, 1, n)); r = t; z = np.where(lat, 1 - 2 * t, -1.0)
        p = np.stack([r * np.cos(th), r * np.sin(th), z], 1)
    p = p * rng.uniform(0.6, 1.4, 3)                         # random scale per axis
    a = rng.uniform(0, 2 * np.pi)                             # random turn about the up axis z
    R = np.array([[np.cos(a), -np.sin(a), 0], [np.sin(a), np.cos(a), 0], [0, 0, 1]])
    p = p @ R.T + rng.normal(scale=0.02, size=p.shape)
    p -= p.mean(0); p /= np.linalg.norm(p, axis=1).max()     # centre, scale into the unit ball
    return p

def make_set(n_per_class, seed):
    rng = np.random.default_rng(seed)
    X = [sample_shape(c, rng) for c in range(4) for _ in range(n_per_class)]
    y = [c for c in range(4) for _ in range(n_per_class)]
    return torch.tensor(np.array(X), dtype=torch.float32), torch.tensor(y)

def reorder(X, idx):
    return torch.gather(X, 1, idx[:, :, None].expand(-1, -1, 3))

def sort_z(X):                       # a canonical order: by height (unchanged by turns about z)
    return reorder(X, X[:, :, 2].argsort(dim=1))

def shuffle(X, g):                   # an independent random order for every shape
    return reorder(X, torch.stack([torch.randperm(X.shape[1], generator=g) for _ in X]))

class PointNet(nn.Module):
    """Shared per-point MLP h (3-64-128-256), max pooling, MLP gamma (256-128-4)."""
    def __init__(self):
        super().__init__()
        self.h = nn.Sequential(nn.Linear(3, 64), nn.ReLU(), nn.Linear(64, 128), nn.ReLU(),
                               nn.Linear(128, 256), nn.ReLU())
        self.gamma = nn.Sequential(nn.Linear(256, 128), nn.ReLU(), nn.Linear(128, 4))
    def feature(self, X):            # (B, n, 3) -> (B, 256) global feature, any n
        return self.h(X).max(dim=1).values
    def forward(self, X):
        return self.gamma(self.feature(X))

def MLP():                           # one weight per coordinate slot: input is the 768-vector
    return nn.Sequential(nn.Flatten(), nn.Linear(3 * N, 256), nn.ReLU(),
                         nn.Linear(256, 256), nn.ReLU(), nn.Linear(256, 4))

def train_outliers(X, g):             # training-time outliers: 0-25% of each shape's points
    X = X.clone()
    for i in range(len(X)):
        k = int(torch.rand(1, generator=g).item() * 0.25 * N)
        u = torch.randn(k, 3, generator=g); u /= u.norm(dim=1, keepdim=True)
        u *= torch.rand(k, 1, generator=g) ** (1 / 3)
        X[i, torch.randperm(N, generator=g)[:k]] = u
    return X

def train(net, X, y, order, seed=0, epochs=20, with_outliers=False):
    torch.manual_seed(seed); g = torch.Generator().manual_seed(seed)
    opt = torch.optim.Adam(net.parameters(), lr=1e-3)
    for _ in range(epochs):
        Xe = sort_z(X) if order == "sorted" else shuffle(X, g)
        if with_outliers: Xe = train_outliers(Xe, g)
        perm = torch.randperm(len(X), generator=g)
        for i in range(0, len(X), 32):
            b = perm[i:i + 32]
            opt.zero_grad()
            nn.functional.cross_entropy(net(Xe[b]), y[b]).backward()
            opt.step()
    return net

Xtr, ytr = make_set(400, seed=0)
Xte, yte = make_set(100, seed=1)
torch.manual_seed(0); pointnet = train(PointNet(), Xtr, ytr, "random")
torch.manual_seed(0); mlp_sorted = train(MLP(), Xtr, ytr, "sorted")
torch.manual_seed(0); mlp_random = train(MLP(), Xtr, ytr, "random")
torch.manual_seed(0); pointnet_o = train(PointNet(), Xtr, ytr, "random", with_outliers=True)

# Corrupted test sets, the same for every model.
g = torch.Generator().manual_seed(123)
def keep(X, frac):                   # random point dropout: a random subset of each shape
    k = int(frac * N)
    return reorder(X, torch.stack([torch.randperm(N, generator=g)[:k] for _ in X]))
def outliers(X, frac):               # replace a fraction of points by uniform points in the unit ball
    X = X.clone(); k = int(frac * N)
    for i in range(len(X)):
        u = torch.randn(k, 3, generator=g); u /= u.norm(dim=1, keepdim=True)
        u *= torch.rand(k, 1, generator=g) ** (1 / 3)
        X[i, torch.randperm(N, generator=g)[:k]] = u
    return X
tests = {"clean": Xte, "keep 50%": keep(Xte, 0.5), "keep 25%": keep(Xte, 0.25),
         "outl 10%": outliers(Xte, 0.10), "outl 25%": outliers(Xte, 0.25)}

def fill(X, how):                    # the MLP needs exactly N points: fill dropped slots
    if X.shape[1] == N: return X
    if how == "repeat":
        return X.repeat(1, N // X.shape[1], 1)
    return torch.cat([X, torch.zeros(len(X), N - X.shape[1], 3)], 1)

def acc(net, X):
    with torch.no_grad(): return (net(X).argmax(1) == yte).float().mean().item()

print(f"{'model (input order)':32s}" + "".join(f"{k:>10s}" for k in tests) + f"{'shuffled':>10s}")
rows = [("PointNet", pointnet, None, None),
        ("MLP, z-sorted, repeat fill", mlp_sorted, "sorted", "repeat"),
        ("MLP, z-sorted, zero fill", mlp_sorted, "sorted", "zero"),
        ("MLP, random order, repeat fill", mlp_random, "random", "repeat"),
        ("MLP, random order, zero fill", mlp_random, "random", "zero"),
        ("PointNet, outliers in training", pointnet_o, None, None)]
for name, net, order, how in rows:
    accs = []
    for X in tests.values():
        if order == "sorted":        # sort the points that are present; zeros go at the end
            X = fill(sort_z(X), how) if how == "zero" else sort_z(fill(X, how))
        elif order == "random":
            X = shuffle(fill(X, how), g)
        accs.append(acc(net, X))
    accs.append(acc(net, shuffle(Xte, g)))
    print(f"{name:32s}" + "".join(f"{a:10.3f}" for a in accs))

with torch.no_grad():
    d = (pointnet(Xte) - pointnet(shuffle(Xte, g))).abs().max().item()
print(f"PointNet, max |logit change| when every test shape is reshuffled: {d:.1e}")

# Critical point sets: the points that win at least one of the 256 max-pooling channels.
with torch.no_grad():
    H = pointnet.h(Xte)                              # (400, 256 points, 256 channels)
    F, arg = H.max(dim=1)                            # global features and winning point per channel
    sizes, same_feat, same_pred, changed = [], 0.0, 0, 0
    for i in range(len(Xte)):
        crit = arg[i].unique()
        sizes.append(len(crit))
        only = Xte[i, crit][None]
        same_feat = max(same_feat, (pointnet.feature(only) - F[i]).abs().max().item())
        same_pred += int(pointnet(only).argmax() == pointnet(Xte[i:i + 1]).argmax())
        rest = Xte[i, [j for j in range(N) if j not in set(crit.tolist())]][None]
        changed += int(pointnet(rest).argmax() != pointnet(Xte[i:i + 1]).argmax())
    sizes = torch.tensor(sizes, dtype=torch.float32)
    print(f"critical set size over {len(Xte)} test shapes: median {sizes.median():.0f}, "
          f"range {sizes.min():.0f}-{sizes.max():.0f} of {N} points (at most 256 = channels)")
    print(f"critical points only: max |feature change| {same_feat:.1e}, "
          f"same class for {same_pred}/{len(Xte)} shapes")
    print(f"critical points removed: class changes for {changed}/{len(Xte)} shapes")
    for name, c in zip(NAMES, range(4)):
        s = sizes[yte == c]
        print(f"  {name:9s} median critical set {s.median():.0f}")

    # Upper-bound set N_S for one shape: grid points whose every h-channel stays below F.
    t = torch.linspace(-1, 1, 41)
    grid = torch.stack(torch.meshgrid(t, t, t, indexing="ij"), -1).reshape(-1, 3)
    i = 0                                            # first test shape (a sphere)
    inside = (pointnet.h(grid) <= F[i]).all(dim=1)
    extra = torch.cat([Xte[i], grid[inside]])[None]
    print(f"upper-bound set of test shape 0: {inside.sum().item()} of {len(grid)} grid points; "
          f"adding them changes the feature by {(pointnet.feature(extra) - F[i]).abs().max().item():.1e}")

Output:

model (input order)                  clean  keep 50%  keep 25%  outl 10%  outl 25%  shuffled
PointNet                             0.923     0.920     0.908     0.382     0.260     0.923
MLP, z-sorted, repeat fill           0.725     0.675     0.640     0.707     0.613     0.265
MLP, z-sorted, zero fill             0.725     0.250     0.250     0.707     0.613     0.262
MLP, random order, repeat fill       0.472     0.452     0.498     0.405     0.428     0.452
MLP, random order, zero fill         0.465     0.252     0.250     0.440     0.420     0.445
PointNet, outliers in training       0.815     0.798     0.793     0.795     0.663     0.815
PointNet, max |logit change| when every test shape is reshuffled: 0.0e+00
critical set size over 400 test shapes: median 43, range 23-61 of 256 points (at most 256 = channels)
critical points only: max |feature change| 0.0e+00, same class for 400/400 shapes
critical points removed: class changes for 22/400 shapes
  sphere    median critical set 49
  cube      median critical set 38
  cylinder  median critical set 48
  cone      median critical set 34
upper-bound set of test shape 0: 21986 of 68921 grid points; adding them changes the feature by 0.0e+00

Order. Reshuffling every test shape leaves PointNet’s logits exactly unchanged, not merely within rounding: each point passes through the same MLP whatever its position in the array, and a maximum does not depend on the order in which its arguments are compared. The MLP trained on height-sorted points is reasonable as long as the points come in that order and falls to chance when they do not. As in the paper’s Figure 5, the MLP on sorted points beats the MLP on unsorted points, and both stay well below the max-pooling network. The MLP trained on random orders never learns a usable rule: it stays below 50% under every condition, and its clean accuracy differs between its two rows only because each row draws new random orders, so its answer depends on the order. The first layer of an MLP gives each coordinate slot its own weights, so it must learn the same shape once for every arrangement of the points, the problem that the multilayer perceptron article shows for pixels.

Dropout. Keeping a quarter of the points costs PointNet about 1.5 points of accuracy. Each channel’s maximum over a subset is at most its maximum over the whole set, and on a densely sampled surface a neighbour of the lost winner usually comes close (our reading). The MLP needs every slot filled: repeating the kept points costs the sorted MLP 8.5 points at 25%, and filling with zeros drops both MLPs to chance. In our reading, the zeros put many points at the centre of the ball, where no training surface had points, into slots whose weights expect surface coordinates.

Outliers. Uniform outliers that the network never saw in training are a different matter: 10% of them cut PointNet’s accuracy from 0.923 to 0.382, much more than they hurt the sorted MLP. The upper-bound set suggests why. For the first test shape, only 21,986 of the 68,921 grid points of the cube $[-1, 1]^3$, about a third, lie in $\mathcal{N}_S$; any point outside it raises at least one channel above the shape’s maximum and so changes the global feature. The outliers fill the unit ball rather than the cube, but with 25 of them in a shape it is likely that several fall outside (our reasoning), and the channels they take over then describe the outliers rather than the shape. Theorem 2 promises invariance only to points inside $\mathcal{N}_S$, and the network decides how large $\mathcal{N}_S$ is through training.

The last row was added after the first run. That run’s PointNet, trained on clean shapes only, lost most of its accuracy to outliers, while the paper states its outlier robustness for a network that has seen outliers during training (§5.2). We therefore trained the same network again with 0–25% of the points of every training shape, drawn anew each epoch, replaced by uniform points in the unit ball, checked in a separate trial run with differently drawn test outliers that it learned the clean task, and added it to the script; the other lines of the output are unchanged from the first run. Trained this way, the network keeps 0.795 at 10% outliers and 0.663 at 25%, but its clean accuracy is 0.815 rather than 0.923. With one training seed per network, the size of that trade-off is not established by this example.

Critical points. The critical sets hold a median of 43 of the 256 points (23 to 61 over the 400 test shapes), well under the bound of $K = 256$, so in this network many channels share their winners; cubes and cones need fewer critical points than spheres and cylinders. Feeding the network only the critical points reproduces every global feature exactly and every predicted class, as Theorem 2 states. Removing the critical points instead changes the class of 22 of the 400 shapes: the theorem makes no claim about sets that lack some critical points, and in our reading the remaining neighbours usually take over the maxima with similar values. Adding all 21,986 grid points of the upper-bound set to the first shape leaves its feature unchanged, also as the theorem states.

Performance

ModelNet40 classification

Wu et al. (2015) built ModelNet, 151,128 CAD models in 660 categories, for 3D ShapeNets; their own experiments used 40 categories with 100 models each, rotated into 12 poses. The ModelNet40 benchmark used by PointNet, PointNet++ and Deep Sets has 12,311 models from 40 categories, split into 9,843 for training and 2,468 for testing. The PointNet paper’s Table 1 reports accuracy both averaged over classes and overall:

Method	Input	Views	Average class (%)	Overall (%)
3DShapeNets	volume	1	77.3	84.7
VoxNet	volume	12	83.0	85.9
Subvolume	volume	20	86.0	89.2
MVCNN	images	80	90.1	–
Baseline (MLP on hand-crafted features)	point	–	72.6	77.4
PointNet	point	1	86.2	89.2

The authors describe the result as the best among methods on 3D input, with a small gap to the multi-view MVCNN that they attribute to fine geometric detail captured by rendered images. In the robustness tests (Figure 6, the max-pooling network of Figure 5 on points normalized into a unit sphere), deleting 50% of the points lowered accuracy by 2.4 points when the original 1,024 points had been chosen by furthest point sampling and by 3.8 points when they had been chosen at random. With outliers scattered uniformly in the unit sphere, the authors evaluate two networks that had seen outliers in training, one on coordinates and one with point density added, and report more than 80% accuracy at an outlier ratio of 20%.

ShapeNet part segmentation

The part segmentation data comes from Yi et al. (2016): 16,881 shapes from 16 ShapeNet categories annotated with 50 parts, most categories with two to five parts, with labels on sampled points. The metric is a part-averaged intersection over union (IoU) per shape, counting a part absent from both ground truth and prediction as 1. PointNet’s Table 2 reports a mean of 83.7%, against 81.4% for Yi et al. and 79.4% for the authors’ 3D CNN baseline; the ShapeNet article shows that this mean is the average over shapes rather than over categories, and why the two differ. The network was best in most categories but not all, for example 74.9% against Yi et al.’s 75.7% on cars. On incomplete scans from six random viewpoints per model, generated with the Blensor Kinect simulator, the authors trained the same network on the complete shapes and partial scans and report that it lost only 5.3% mean IoU.

S3DIS semantic segmentation

The Stanford 3D semantic parsing data of Armeni et al. (2016), now called S3DIS, consists of Matterport scans of indoor areas. The PointNet paper describes 6 areas with 271 rooms, every point labelled with one of 13 categories (12 classes such as chair, table, floor and wall, plus clutter); the repository’s README says that version 1.2 of the dataset was used. The CVPR paper of Armeni et al. describes an earlier state: five areas from three buildings covering about 6,020 square metres, 12 semantic elements, and a k-fold evaluation with each building as a fold. The PointNet paper says it follows the same k-fold protocol as Armeni et al., but the README gives the split actually used: six-fold, each model trained on five areas and tested on the sixth, and notes that the S3DIS paper’s own three-fold split was not public at the time.

Over the six test areas, PointNet reached a mean IoU of 47.71% over the 13 classes and 78.62% point accuracy (Table 3), against 20.12% and 53.19% for an MLP on hand-crafted point features (the same 9 inputs plus local density, curvature and normal). A detection system built on its segmentation, with connected components as object proposals, reached a mean average precision of 24.24 at 3D IoU 0.5 over table, chair, sofa and board, against 18.22 for the method of Armeni et al. (Table 4), though lower on sofas (4.76 against 6.78). The semantic segmentation article covers the task in general.

Strengths

• Exact order invariance by construction, with no augmentation over permutations needed.
• Any number of points, at a cost linear in the number of points, and few parameters: 3.5M for classification against 60.0M for MVCNN (Table 6).
• One design for several tasks: classification from the global feature, segmentation from per-point features joined with it.
• Robust to missing points, through the critical point set: any subset that keeps the critical points gives the same feature.
• A reusable module. A shared MLP followed by max pooling turns any set of points, or of feature vectors, into a fixed-size vector, so it serves as an encoder inside larger systems (see Historical Importance).

Limitations

• No local structure. Each point is embedded alone, and the only interaction between points is the final maximum. The PointNet++ authors give this as PointNet’s main limitation: it does not capture the local structures induced by the metric, which limits its ability to recognize fine-grained patterns and to generalize to complex scenes. They add that set networks that ignore the metric are sensitive to the translation and normalization of the whole set.
• Fragile maxima under unseen corruption. A single point that raises a channel above its maximum changes the global feature. The paper’s outlier robustness holds for networks trained with outliers; the example above shows how far accuracy can fall without them.
• Pose. The T-nets are learned from the training data and guarantee no invariance; the ModelNet40 training rotates shapes only about the up axis.
• Scenes in blocks. For S3DIS the paper labels 1 m by 1 m blocks, with each point’s position in the room as extra input; on ScanNet, PointNet++’s Figure 6 shows PointNet capturing the room layout but failing to find the furniture.

Historical Importance

PointNet appeared at CVPR 2017. Its authors describe it as the first network to work directly on raw point clouds for these benchmarks, where earlier deep methods had converted the data to voxels, rendered views or hand-crafted feature vectors. Its theory, the universal approximation result and the critical point set, explained why such a simple network should work, and the shared-MLP-plus-max-pooling encoder became a standard component. The PointNet README lists VoxelNet, which learns a feature for each voxel of a LiDAR scan with a PointNet before 3D convolutions, and Frustum PointNets, which run point networks inside the 3D frustum of each 2D detection (see sensor fusion for autonomous driving). Other published articles on this site describe further uses: FlowNet3D’s set convolution layers for scene flow, PointNetLK for point cloud registration, the PointNet-style point embedding of DenseFusion in robotic manipulation, and VectorNet’s polyline encoder in motion forecasting, whose authors call it a generalization of PointNet. Deep Sets (Zaheer et al., 2017) reached the sum form of permutation-invariant functions in the same year from the side of general machine learning on sets.

Variants

PointNet (vanilla)

The paper’s ablations and Table 6 use a “vanilla” PointNet without the input and feature transforms: 0.8M parameters and 87.1% overall accuracy on ModelNet40 (Table 5; PointNet++’s Table 2 gives 87.2% for the same model). The semantic segmentation network in the official repository is also, by its README, a simple model based on vanilla PointNet.

PointNet++

PointNet++ (Qi, Yi, Su and Guibas, NIPS 2017) applies PointNet recursively on a nested partitioning of the input, so that, as in a CNN, early layers see small neighbourhoods and later layers larger ones. Its README calls it version 2.0 of the PointNet architecture.

Set abstraction. A set abstraction level takes $N$ points with $d$-dimensional coordinates and $C$-dimensional features, an $N \times (d + C)$ matrix, and outputs $N' \times (d + C')$ for $N' < N$ subsampled points. It has three layers:

• Sampling. Iterative farthest point sampling picks $N'$ centroids, each the point farthest from those already chosen, which covers the set better than random sampling for the same number of centroids.
• Grouping. A ball query collects the points within radius $r$ of each centroid, up to a limit of $K$ points. The authors prefer it to $k$ nearest neighbours because a fixed radius gives every local region the same scale, which they consider better for local pattern recognition such as semantic labelling.
• PointNet layer. Each group’s coordinates are translated into the centroid’s local frame, $\mathbf{x}_i - \hat{\mathbf{x}}$, and a small PointNet maps the group, with its point features, to one feature vector for the centroid.

The official classification model with single-scale grouping (SSG; pointnet2_cls_ssg.py) has two set abstraction levels, 512 centroids with $r = 0.2$ and up to 32 points, then 128 centroids with $r = 0.4$ and up to 64 points, followed by one PointNet over all remaining points and three fully connected layers.

Non-uniform density. Scans are dense near the sensor and sparse far away. Features learned on dense regions may not generalize to sparse ones, while small neighbourhoods in sparse regions contain too few points; the authors report that small neighbourhoods, which help CNNs, did not help on point sets. They propose two density-adaptive layers:

• Multi-scale grouping (MSG) groups each centroid’s neighbourhood at several radii, runs a PointNet at each scale and concatenates the features. In the official MSG classifier, the first level uses radii 0.1, 0.2 and 0.4 with up to 16, 32 and 128 points around 512 centroids, and the second radii 0.2, 0.4 and 0.8 around 128 centroids.
• Multi-resolution grouping (MRG) concatenates two vectors per region: the summary of its subregions from the level below, and a PointNet over all raw points in the region. When the region is sparse, the second vector should be weighted more. MRG avoids running PointNets on large neighbourhoods at the lowest level, which makes MSG expensive.

To learn how to combine the scales, the network is trained with random input dropout: for each training shape a dropout ratio $\theta$ is drawn uniformly from $[0, p]$ with $p = 0.95$, and each point is dropped with probability $\theta$. At test time all points are kept.

Feature propagation. For segmentation, features are propagated from the $N_l$ points of level $l$ back to the $N_{l-1}$ points of the level before by inverse-distance-weighted interpolation over the $k$ nearest neighbours (the paper’s Eq. 2, with $p = 2$ and $k = 3$ by default),

$$f^{(j)}(\mathbf{x}) = \frac{\sum_{i=1}^{k} w_i(\mathbf{x}) f_i^{(j)}}{\sum_{i=1}^{k} w_i(\mathbf{x})}, \qquad w_i(\mathbf{x}) = \frac{1}{d(\mathbf{x}, \mathbf{x}_i)^p},$$

concatenated with skip-linked features from the set abstraction level and passed through a “unit pointnet”, a shared MLP comparable to a $1 \times 1$ convolution, until every original point has a feature.

Results. On ModelNet40 with 1,024 points and coordinates only, PointNet++ reached 90.7% accuracy, against 89.2% for PointNet, and 91.9% with face normals as extra features and 5,000 points (Table 2). On MNIST digits converted to 2D point sets of 512 pixel locations, its error was 0.51%, against 0.78% for PointNet and 1.30% for vanilla PointNet (Table 1). When test points were dropped at random, MSG with random input dropout lost less than 1% from 1,024 to 256 points, while single-scale grouping trained without dropout failed on sparse inputs (Figure 4).

On ScanNet, with the 1,201 training and 312 test scenes of the ScanNet paper, RGB removed and predictions converted to per-voxel labels following that paper, the authors report per-voxel accuracy (their Figure 5):

Method	ScanNet	Non-uniform virtual scans
3DCNN (ScanNet paper)	0.730	–
PointNet	0.739	0.680
PointNet++ SSG	0.833	0.727
PointNet++ MSG + DP	0.845	0.804
PointNet++ MRG + DP	0.834	0.762

The second column tests models trained on uniformly sampled points on synthesized virtual scans with non-uniform density; how the scans were generated is in the supplementary material. Figure 5 labels the PointNet bar with reference [19], which in its reference list is a different paper; the text cites PointNet as [20].

PointNet++ also works in non-Euclidean metric spaces. On SHREC15 non-rigid shapes (1,200 shapes in 50 categories, five-fold cross-validation), grouping by geodesic distance with intrinsic point features reached 96.09%, against 94.49% with Euclidean neighbourhoods and the same features, 60.18% with Euclidean neighbourhoods and coordinates, and 93.03% for the previous method, DeepGM (Table 3). The authors name the inference speed of the MSG and MRG layers as a problem for future work.

Implementations

As of November 2026, the official repositories are charlesq34/pointnet, last changed in September 2022 (commit 2618f72), and charlesq34/pointnet2, last changed in November 2018 (commit 4292663), both under the MIT license and written for TensorFlow 1. The PointNet README says the code was tested with Python 2.7 and TensorFlow 1.0.1, and points to a third-party PyTorch implementation; the training script downloads ModelNet40 as HDF5 files of 2,048 points per shape (416 MB), and a separate script downloads the ShapeNet part data. In train.py, the help text of the learning-rate decay rate says 0.8 while the default value is 0.7. PointNet++ needs custom TensorFlow operators for sampling, grouping and interpolation, compiled from the repository, and was tested with TensorFlow 1.2. Its models folder holds single- and multi-scale classification, part segmentation and semantic segmentation networks but no MRG model, and in its train.py the call that applies random point dropout to each training batch is commented out, so the default training script does not use the random input dropout described in the paper.