Deep Learning on 3D Data

Deep learning on 3D data, sometimes called 3D deep learning or geometric deep learning on shapes and scenes, is the family of neural networks that take 3D geometry as input or produce it as output: a point cloud from a LiDAR or a depth camera, a voxel grid, a polygon mesh, a set of rendered images, or a function such as a signed distance function. Images come as one regular pixel grid, so image networks could take the convolutional neural network as their common design. 3D data has no single native form, and each network design starts from a choice of representation. That choice decides which operations are cheap, which symmetries the network has by construction, and how cost grows with resolution. This article surveys the representations that 3D networks consume, the main designs for each, the tasks and benchmarks on which they are compared, and their trade-offs; the 3D representations article compares the representations as data structures, and the linked articles carry the details of each design.

Definition

A 3D network is a function $f_\theta$ whose input or output is a 3D shape or scene in one of five forms:

• Voxel grids, dense or sparse: values on a regular 3D lattice, processed by 3D convolutions.
• Multi-view images: renderings or projections of the 3D data into 2D, such as depth images, range images or a bird’s-eye view, processed by 2D convolutions.
• Point sets: unordered sets of 3D coordinates with optional attributes, processed by networks that do not depend on the order of the points.
• Meshes and graphs: vertices, edges and faces, or a graph of neighbouring points, processed by operators defined on the connectivity.
• Implicit functions: a network that maps a 3D coordinate to an occupancy or a signed distance, usually as the output of a reconstruction or generation system.

Most systems mix forms: a point network may encode the points inside each voxel, a voxel network may predict an implicit function, and a detector may project voxel features into a 2D bird’s-eye view.

Intuition

Three properties of 3D data shape every design.

• No canonical grid. A point cloud has no order and uneven spacing, and a mesh has a different connectivity for every shape. Convolution, the operation that made image networks work, needs a grid on which the same weights can be shifted; either the data is moved onto a grid (voxels, rendered views) or a new operator is defined on the data as it is (point, graph and mesh networks).
• Most of space is empty. A surface is two-dimensional, so a dense grid of linear resolution $n$ spends $n^3$ cells on a shape that occupies about $n^2$ of them. Dense 3D networks are therefore limited to coarse grids, and much of the field’s history is the search for representations whose cost grows with the surface rather than the volume.
• No canonical orientation. An image has an up direction; a 3D object can be presented in any rotation. Benchmarks usually supply aligned shapes (ModelNet10, for example, comes in a canonical pose), and the networks learn whatever orientation conventions the training data have.

Formal Definition

The designs differ in the symmetries they build in. Write $X = \{\mathbf{x}_1, \dots, \mathbf{x}_n\}$ for a point set and $\pi$ for a permutation of its elements. A network on point sets must be permutation-invariant for a label of the whole set, $f(\pi X) = f(X)$, and permutation-equivariant for a label per point, $f(\pi X) = \pi f(X)$. PointNet meets the first by applying one function $h$ to every point and pooling with a symmetric function (Qi et al., 2017, Eq. 1):

$$f(X) \approx \gamma\Big(\max_{i = 1, \dots, n} h(\mathbf{x}_i)\Big).$$

A convolution on a grid is translation-equivariant: shifting the input shifts the output. On a voxel grid this holds for translations by whole voxels, and a global pooling layer at the end makes the classification invariant to them (exactly so, in our reasoning, only away from the grid’s borders and, after strided pooling, for shifts by multiples of the total stride); nothing in a standard 3D CNN makes it invariant to rotations.

A multi-view network applies one image network $g$ to each of $V$ views $I_1, \dots, I_V$ and pools across them element-wise; Su et al. (2015) use the maximum,

$$f(I_1, \dots, I_V) = \gamma\Big(\max_{v = 1, \dots, V} g(I_v)\Big),$$

so its output does not depend on the order of the views. When the cameras sit at equal steps of $360^\circ / V$ around the up axis, turning the object by one step permutes the views, and the output is unchanged (reasoning from the formula; the example below checks it).

An implicit-function network instead maps a query point $\mathbf{p} \in \mathbb{R}^3$, and a code $\mathbf{z}$ describing the shape, to a value: an occupancy probability $f_\theta(\mathbf{p}, \mathbf{z}) \in [0, 1]$ in occupancy networks (Mescheder et al., 2019) or a signed distance in DeepSDF (Park et al., 2019). The surface is a level set of $f_\theta$, which can be extracted at any resolution.

Representations and Designs

Voxel grids

The first deep networks on 3D shapes were volumetric. 3D ShapeNets (Wu et al., 2015) represented each shape as a binary voxel grid, 1 inside the mesh and 0 outside, of $30 \times 30 \times 30$ cells (a $24^3$ shape with three cells of padding on each side), and learned a convolutional deep belief network on it, without pooling in the hidden layers, for recognition, shape completion from a single depth map and next-best-view planning. To train it, the authors built ModelNet, 151,128 CAD models in 660 categories. VoxNet (Maturana and Scherer, 2015) was a supervised 3D CNN for range data from LiDAR, RGB-D and CAD models: a $32^3$ occupancy grid, two convolutional layers, one pooling layer and two fully connected layers, 921,736 parameters in all. It compared three ways of filling the grid from range measurements (binary occupancy, density and hit grids), and it handled orientation about the gravity axis by training on copies rotated in equal steps and pooling the outputs over the same rotations at test time.

These networks were compact but coarse, and their accuracy trailed that of networks on rendered views. Qi, Su, Nießner, Dai, Yan and Guibas (2016) analysed the gap on ModelNet40 and found two causes. Part of it was resolution: a multi-view network sees $227 \times 227$ images, while a volumetric network of similar cost sees $30^3$ voxels. But when they rendered the $30^3$ grids themselves as images (each occupied voxel drawn as a sphere) and trained a multi-view network on them, it was still 4.8 points better than the volumetric network, against 7.3 points on standard renderings, so the volumetric architecture was also weaker. Their two volumetric designs, one trained with auxiliary classification of sub-volumes and one that projects the volume to 2D with elongated “anisotropic probing” kernels, matched their multi-view network at 3D resolution 30 when combined with pooling over 20 orientations. In their controlled comparison on one split with azimuth and elevation augmentation (Table 2, ModelNet40 accuracy), VoxNet reached 83.8% on single orientations and 85.9% with orientation pooling, and the anisotropic probing network 84.4% and 89.9%. They concluded that 3D resolution was likely the bottleneck of volumetric CNNs.

Resolution is what later voxel designs bought back by skipping empty space. OctNet (Riegler, Ulusoy and Geiger, 2017) stores features in a hybrid of a coarse grid and shallow octrees, so that large empty or uniform regions hold a single feature vector; its authors report that, with batches of 32 on a 12 GB GPU, a dense network fits in memory only for inputs up to $64^3$, while OctNet fits at $256^3$. On ModelNet10 they found diminishing returns in classification accuracy beyond $32^3$ (some classes, such as dresser and night stand, stayed confused at every resolution), while orientation estimation and point labelling gained from higher resolution. O-CNN (Wang, Liu, Guo, Sun and Tong, 2017) is an octree-based network of the same year. Sparse convolutional networks compute only at the occupied (“active”) voxels. Graham, Engelcke and van der Maaten (2018) point out that an ordinary convolution on sparse data dilates it, so that the number of active sites grows rapidly from layer to layer; their submanifold sparse convolution fixes the active sites, so the sparsity stays unchanged through many layers and deep networks for point-wise segmentation become practical. The sparse convolutional networks article covers the operators, the later generalized form, the libraries and their use in LiDAR detection and indoor segmentation.

Multi-view images

The multi-view CNN (MVCNN; Su, Maji, Kalogerakis and Learned-Miller, 2015) renders a mesh into shaded images and classifies them with an image network pretrained on ImageNet. For upright shapes it places 12 cameras every 30° around the vertical axis, elevated 30°; without that assumption it uses 80 views, four rotations of each of 20 cameras at the vertices of an icosahedron. All views pass through the same convolutional layers, a view-pooling layer takes the element-wise maximum across views, and further layers classify the pooled feature. On ModelNet40 the authors report 89.9% accuracy with 12 views and 90.1% with 80 (Table 1, with fine-tuning on ModelNet40), against 77.3% for 3D ShapeNets on the same split; with a single view the fine-tuned image network already reached 85.1%. Qi et al. (2016), training on the full ModelNet40 training split, later reached 92.0% instance accuracy with their own 20-view MVCNN and 93.8% with a multi-resolution version that adds sphere renderings of voxel grids at 3D resolutions 30 and 60 (Table 3). Rendered views inherit everything image networks have, pretraining included, but they see only the visible surface, and the 12-view set-up assumes a known up direction.

Projections into 2D remain common for scans. A spinning LiDAR’s sweep maps onto a range image with one row per beam, and projection-based segmentation networks label that image with a 2D network (point cloud segmentation describes RangeNet++ and its post-processing). Driving detectors flatten 3D features into a bird’s-eye-view image: PointPillars (Lang et al., 2019) encodes the points of each vertical column (“pillar”) with PointNets into a pseudo-image that a 2D convolutional detector processes, and BEV perception covers the camera-based versions.

Point sets

PointNet (Qi, Su, Mo and Guibas, 2017) consumes the raw points: a shared multilayer perceptron embeds every point, max pooling gives a global feature, and the same backbone serves classification and segmentation. Its authors argue that converting points to voxels or images makes the data unnecessarily voluminous and introduces quantization artifacts. With 3.5 million parameters (0.8 million without its transform networks), against 60.0 million for MVCNN (Table 6), it reached 89.2% overall accuracy on ModelNet40 from 1,024 points. Its weakness is that points interact only through the final maximum, so it captures no local structure; PointNet++ (Qi, Yi, Su and Guibas, 2017) applies PointNet to nested neighbourhoods found by farthest point sampling and ball queries, and reported 90.7% ModelNet40 accuracy against PointNet’s 89.2% with the same 1,024 points and coordinates only (Table 2). The PointNet article covers both models, their theory and their published results.

Later point networks define neighbourhood operators directly on the points, among them point convolutions such as KPConv and self-attention within local neighbourhoods such as the Point Transformer. Point cloud segmentation describes these designs with their ablations and benchmark results, and transformers explains the attention mechanism they adapt.

Graphs and meshes

A point cloud becomes a graph by connecting each point to its nearest neighbours. DGCNN (Wang et al., 2019) builds such a graph anew at every layer, in the space of the current features rather than of the coordinates. In the authors’ PyTorch implementation, each EdgeConv layer finds the $k$ nearest neighbours of each point (20 by default), forms for every edge the feature pair $(\mathbf{x}_j - \mathbf{x}_i, \mathbf{x}_i)$, applies a shared layer to each edge and takes the maximum over the neighbours; its README describes EdgeConv as a module for classification and segmentation that can be plugged into existing architectures. (The journal paper could not be read for this article.)

A mesh carries its own graph. MeshCNN (Hanocka et al., 2019) treats the edges of a triangle mesh as the analogue of pixels. As its project page describes it, each edge starts with five geometric features (the dihedral angle, two inner angles and two edge-length ratios), a convolution combines an edge with the four other edges of its two triangles, in an order made symmetric so that the features are invariant to rotation, translation and uniform scale, and pooling is a learned edge collapse that simplifies the mesh while keeping its topology; unpooling restores the resolution for segmentation. When the output must be a mesh, many systems instead deform a template of fixed connectivity, which, as Mescheder et al. (2019) note, does not allow arbitrary topologies; parametric human body models are the best-known case, and differentiable rendering lets such meshes be fitted to images by gradient descent.

Implicit functions

For reconstruction and generation the output representation matters as much as the input. Occupancy networks (Mescheder et al., 2019) represent a surface as the decision boundary of a network that classifies points as inside or outside, conditioned on an encoding of the input (an image, a point cloud or a coarse voxel grid), and extract meshes at any resolution with a hierarchical, octree-like procedure (Multiresolution IsoSurface Extraction); they evaluate reconstructions by volumetric IoU, Chamfer-$L_1$ distance and normal consistency. DeepSDF (Park et al., 2019) regresses a signed distance instead, from a latent code per shape learned without an encoder (“auto-decoder”). Both avoid the cubic memory of a voxel output and the fixed topology of a template mesh, at the price of many network evaluations per shape. Neural implicit representations covers this family, surface reconstruction places it among the methods that turn points into surfaces, and neural radiance fields and 3D Gaussian splatting are the corresponding representations of appearance for novel view synthesis, usually optimized per scene rather than predicted by a trained network.

Networks that regress geometry directly from images form a further branch. DUSt3R predicts a 3D point for every pixel of a pair of uncalibrated images, and 3D foundation models groups it with later feed-forward models of the same kind.

Tasks and Benchmarks

• Shape classification. ModelNet40, the 40-category subset of ModelNet with 12,311 models split into 9,843 for training and 2,468 for testing, is the usual benchmark, scored by overall (instance) accuracy and by accuracy averaged over classes. Early papers did not all use this split: Qi et al. (2016) note that 3D ShapeNets and MVCNN trained and tested on the first 80 and 20 models of each category, while VoxNet used the split on the ModelNet website, so their numbers are not strictly comparable. ModelNet10 is a 10-category subset; OctNet’s authors describe it as 3,991 training and 908 test shapes in a canonical pose.
• Part segmentation. ShapeNet part annotations, scored by part-averaged IoU per shape; the ShapeNet article explains the protocol and its averaging.
• Semantic and instance segmentation of scenes. S3DIS and ScanNet indoors, SemanticKITTI outdoors; point cloud segmentation gives their protocols and results by representation.
• 3D object detection. Oriented 3D boxes from LiDAR, cameras or both, on KITTI and larger driving benchmarks; 3D object detection covers the task and its matching rules.
• Reconstruction and generation. Shapes from single images, sparse points or coarse voxels, scored for example by volumetric IoU, Chamfer distance and normal consistency, as in the occupancy networks paper, often on ShapeNet categories.
• Registration and scene flow. Learned features and correspondences for point cloud registration and scene flow.

Properties

The representations trade the same few quantities against each other: memory and computation as resolution grows, what is built in and what must be learned, and which tasks are natural. The table summarizes the designs above; the cost column is reasoning from the representations’ sizes (3D representations).

Representation	Typical design	Cost vs linear resolution $n$	Built in	Natural tasks	Main limits
Dense voxels	3D CNN	$O(n^3)$	translation equivariance	classification, completion	coarse grids ($30^3$–$64^3$ in early work)
Octree / sparse voxels	OctNet, sparse convolution	near $O(n^2)$	translation equivariance	scene segmentation, LiDAR detection	irregular memory access, quantization
Multi-view / projections	MVCNN, range-image and BEV networks	$O(V n^2)$	view-order invariance, 2D pretraining	classification, retrieval, driving scans	visible surface only, chosen viewpoints
Point sets	PointNet, PointNet++, point convolution, point transformers	$O(N)$ to $O(Nk)$	permutation invariance	classification, segmentation	neighbourhood search, density changes
Graphs / meshes	DGCNN, MeshCNN	$O(Nk)$ or number of edges	connectivity, intrinsic features	segmentation, shape analysis	graph building, varying connectivity
Implicit functions	occupancy networks, DeepSDF	network size, many queries	continuous output	reconstruction, generation	slow extraction, per-query cost

None of the standard designs is invariant to arbitrary rotations: they learn the orientations their training data contain, which is why benchmarks provide upright shapes, why VoxNet assumes that the grid’s $z$ axis is aligned with gravity, and why MVCNN’s 12-view set-up assumes upright shapes. VoxNet’s own ablation (Table I) shows what happens without augmentation: trained on shapes in a canonical pose and tested on randomly oriented shapes, it reached 0.61 average class accuracy on ModelNet40, against 0.82 with rotation augmentation and 0.83 with augmentation and voting over 12 rotations.

Examples

The script below (PyTorch 2.14.1, NumPy 2.5.4) trains three networks of about 60,000 parameters each on the same synthetic shape classification task, each on a different representation of the same shapes, and tests them under rotations and changes of resolution. The five classes are an ellipsoid, a box, a cylinder, a cone and a torus, each with random dimensions and defined by its signed distance function (for the ellipsoid, an approximation with the exact zero set), so that every representation can be computed exactly from one shape:

• points: 512 points on the surface, drawn from a thin band around the zero level set and projected onto it by Newton steps;
• voxels: a $16^3$ grid over the cube $[-1, 1]^3$, either solid (a voxel is 1 if its centre is inside the shape, as in 3D ShapeNets) or a shell (1 if its centre lies within half a voxel diagonal of the surface, closer to OctNet’s voxels that intersect the mesh). The sources use both, so the script trains both;
• multi-view images: 12 orthographic depth images of $32 \times 32$ pixels from cameras every 30° around the up axis $z$, elevated 30°, the camera layout of MVCNN’s first set-up, rendered by sphere tracing; a pixel holds $2.5 - t$ for a ray that hits the surface at distance $t$ from the image plane (nearer is brighter) and 0 otherwise.

All three use one fixed frame, the cube $[-1, 1]^3$ with $z$ up, and no per-shape normalization, so they see the same sizes. The point network is a vanilla PointNet (shared MLP 3–64–128–256, max pooling, MLP 256–64–5). The voxel network has three 3D convolutions with two max-pooling layers and a global maximum over the grid. The multi-view network applies one 2D CNN of the same shape to every view, takes the global spatial maximum of each view, and pools the views element-wise by maximum before the classifier, as MVCNN’s view pooling does (MVCNN places it near its last convolutional layer). Every network is trained on 200 shapes per class, each turned by a random angle about $z$, for 20 epochs with Adam, and tested on 100 new shapes per class:

• base: the test shapes, turned about $z$ as in training, at the training resolution;
• any rot: the same shapes in uniformly random 3D rotations;
• half and double: the same shapes at half and double resolution (128 and 2,048 points, $8^3$ and $32^3$ voxels, $16^2$ and $64^2$ pixels), fed to the networks as they are, which global pooling allows;
• half->base and double->base: the grids at half and double resolution resampled by linear interpolation to the training resolution, the other way of feeding them, printed because neither is the obvious choice.

The last lines check two symmetries: the multi-view network’s logits after turning every test shape by 30° about $z$, which permutes the 12 cameras exactly, and by 15°, which does not; and the voxel networks’ logits after rotating each grid by 90° about $z$, an exact permutation of its cells. The shapes are fixed; the command-line argument sets the training seed (initialization and batch order). One run takes about three minutes on a laptop CPU with four threads.

import math
import sys
import numpy as np
import torch
from torch import nn
import torch.nn.functional as F

torch.set_num_threads(4)
SEED = int(sys.argv[1]) if len(sys.argv) > 1 else 0   # training seed (initialization and batch order)
NAMES = ["ellipsoid", "box", "cylinder", "cone", "torus"]
RANGES = [[(0.35, 0.85)] * 3,                 # ellipsoid semi-axes
          [(0.25, 0.55)] * 3,                 # box half-extents
          [(0.30, 0.65)] * 2,                 # cylinder radius, half-height (axis z)
          [(0.35, 0.65)] * 2,                 # cone base radius, half-height (apex up)
          [(0.40, 0.60), (0.12, 0.25)]]       # torus radii (in the xy plane)

def sdf_obj(c, q, P):
    """Signed distance in the object frame for B shapes of class c: q (B, ..., 3), P (B, k)."""
    a = lambda i: P[:, i].view(-1, *[1] * (q.dim() - 2))
    x, y, z = q[..., 0], q[..., 1], q[..., 2]
    if c == 0:                                # ellipsoid (an approximation; exact zero set)
        r = torch.stack([a(0), a(1), a(2)], -1)
        k0, k1 = (q / r).norm(dim=-1), (q / r**2).norm(dim=-1)
        return k0 * (k0 - 1) / k1.clamp_min(1e-9)
    if c == 1:                                # box
        d = q.abs() - torch.stack([a(0), a(1), a(2)], -1)
        return d.clamp_min(0).norm(dim=-1) + d.amax(-1).clamp_max(0)
    if c == 2:                                # capped cylinder
        d = torch.stack([torch.hypot(x, y) - a(0), z.abs() - a(1)], -1)
        return d.clamp_min(0).norm(dim=-1) + d.amax(-1).clamp_max(0)
    if c == 3:                                # cone: base radius r at z = -h, apex at z = +h
        r, h, qx = a(0), a(1), torch.hypot(x, y)
        cax, cay = qx - torch.minimum(qx, torch.where(z < 0, r, 0.0)), z.abs() - h
        t = ((-qx * -r + (h - z) * 2 * h) / (r**2 + 4 * h**2)).clamp(0, 1)
        cbx, cby = qx - r * t, z - h + 2 * h * t
        s = torch.where((cbx < 0) & (cay < 0), -1.0, 1.0)
        return s * torch.sqrt(torch.minimum(cax**2 + cay**2, cbx**2 + cby**2))
    return torch.hypot(torch.hypot(x, y) - a(0), z) - a(1)      # torus

def sdf(g, p):                                # g = (class, params, rotations); p world points (B, ..., 3)
    c, P, R = g
    q = (p.reshape(len(P), -1, 3) @ R).reshape(p.shape)          # world -> object frame: R^T p
    return sdf_obj(c, q, P)

def rot_z(a):
    a = torch.as_tensor(a, dtype=torch.float64)
    c, s, o, l = torch.cos(a), torch.sin(a), torch.zeros_like(a), torch.ones_like(a)
    return torch.stack([c, -s, o, s, c, o, o, o, l], -1).reshape(-1, 3, 3)

def rot_any(rng, B):                          # uniform random rotations from unit quaternions
    q = rng.normal(size=(B, 4)); q /= np.linalg.norm(q, axis=1, keepdims=True); w, x, y, z = q.T
    M = np.stack([1-2*(y*y+z*z), 2*(x*y-w*z), 2*(x*z+w*y), 2*(x*y+w*z), 1-2*(x*x+z*z),
                  2*(y*z-w*x), 2*(x*z-w*y), 2*(y*z+w*x), 1-2*(x*x+y*y)], 1)
    return torch.tensor(M).reshape(B, 3, 3)

def make_shapes(n, seed):                     # n shapes per class, each turned about the up axis z
    rng = np.random.default_rng(seed)
    return [(c, torch.tensor(np.stack([rng.uniform(lo, hi, n) for lo, hi in RANGES[c]], 1)),
             rot_z(rng.uniform(0, 2 * math.pi, n))) for c in range(5)]

def points(g, n, gen):
    """n points per shape on the surface: rejection in a band |sdf| < 0.05, then Newton steps."""
    B = len(g[1]); got = [[] for _ in range(B)]
    while min(sum(len(t) for t in o) for o in got) < n:
        p = torch.rand(B, 20 * n, 3, generator=gen, dtype=torch.float64) * 2 - 1
        keep = sdf(g, p).abs() < 0.05
        for b in range(B):
            got[b].append(p[b][keep[b]])
    p = torch.stack([torch.cat(o)[:n] for o in got])
    for _ in range(3):
        e = torch.eye(3, dtype=torch.float64) * 1e-5
        grad = torch.stack([(sdf(g, p + e[k]) - sdf(g, p - e[k])) / 2e-5 for k in range(3)], -1)
        p = p - (sdf(g, p) / (grad**2).sum(-1).clamp_min(1e-12))[..., None] * grad
    return p.float()

def voxels(g, R, mode):
    """R^3 grid over [-1, 1]^3: 'solid' = centre inside, 'shell' = centre within half a voxel diagonal."""
    c = -1 + (torch.arange(R, dtype=torch.float64) + 0.5) * 2 / R
    grid = torch.stack(torch.meshgrid(c, c, c, indexing="ij"), -1).expand(len(g[1]), R, R, R, 3)
    d = sdf(g, grid)
    occ = d <= 0 if mode == "solid" else d.abs() <= math.sqrt(3) / R
    return occ.float()[:, None]

CAMS = []                                     # 12 cameras every 30 degrees about z, elevated 30 degrees
for k in range(12):
    az, el = math.radians(30 * k), math.radians(30)
    fwd = -torch.tensor([math.cos(el) * math.cos(az), math.cos(el) * math.sin(az), math.sin(el)], dtype=torch.float64)
    right = torch.linalg.cross(fwd, torch.tensor([0.0, 0, 1], dtype=torch.float64)); right /= right.norm()
    CAMS.append((fwd, right, torch.linalg.cross(right, fwd)))

def render(g, S, steps=80):
    """Orthographic depth images by sphere tracing; pixel = 2.5 - t on a hit (nearer is brighter), 0 if none."""
    u = -1 + (torch.arange(S, dtype=torch.float64) + 0.5) * 2 / S
    U, V = torch.meshgrid(u, -u, indexing="xy")
    views = []
    for fwd, right, up in CAMS:
        o = (-1.5 * fwd + U[..., None] * right + V[..., None] * up).expand(len(g[1]), S, S, 3)
        t = torch.zeros(o.shape[:-1], dtype=torch.float64)
        for _ in range(steps):
            t = (t + 0.9 * sdf(g, o + t[..., None] * fwd).clamp_min(0)).clamp_max(3.0)
        hit = sdf(g, o + t[..., None] * fwd).abs() < 2e-3
        views.append(torch.where(hit, 2.5 - t, 0.0))
    return torch.stack(views, 1).float()     # (B, 12, S, S)

def build(groups, n_pts, R_vox, S_img, gen):
    cat = lambda f: torch.cat([f(g) for g in groups])
    return {"point": cat(lambda g: points(g, n_pts, gen)),
            "voxel solid": cat(lambda g: voxels(g, R_vox, "solid")),
            "voxel shell": cat(lambda g: voxels(g, R_vox, "shell")),
            "multi-view": cat(lambda g: render(g, S_img))}

class PointNet(nn.Module):                    # shared MLP 3-64-128-256, max over points
    def __init__(s):
        super().__init__()
        s.f = nn.Sequential(nn.Linear(3, 64), nn.ReLU(), nn.Linear(64, 128), nn.ReLU(), nn.Linear(128, 256), nn.ReLU())
        s.g = nn.Sequential(nn.Linear(256, 64), nn.ReLU(), nn.Linear(64, 5))
    def forward(s, x):
        return s.g(s.f(x).amax(1))

class VoxelCNN(nn.Module):                    # 3D convolutions, global max over the grid
    def __init__(s):
        super().__init__()
        s.f = nn.Sequential(nn.Conv3d(1, 16, 3, padding=1), nn.ReLU(), nn.MaxPool3d(2),
                            nn.Conv3d(16, 32, 3, padding=1), nn.ReLU(), nn.MaxPool3d(2),
                            nn.Conv3d(32, 48, 3, padding=1), nn.ReLU())
        s.g = nn.Sequential(nn.Linear(48, 64), nn.ReLU(), nn.Linear(64, 5))
    def forward(s, x):
        return s.g(s.f(x).amax((2, 3, 4)))

class MultiViewCNN(nn.Module):                # one 2D CNN for all views, element-wise max over views
    def __init__(s):
        super().__init__()
        s.f = nn.Sequential(nn.Conv2d(1, 32, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
                            nn.Conv2d(32, 64, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
                            nn.Conv2d(64, 64, 3, padding=1), nn.ReLU())
        s.g = nn.Sequential(nn.Linear(64, 64), nn.ReLU(), nn.Linear(64, 5))
    def forward(s, x):
        B, V, S, _ = x.shape
        f = s.f(x.reshape(B * V, 1, S, S)).amax((2, 3)).reshape(B, V, -1)
        return s.g(f.amax(1))

def train(model, X, y, epochs=20, seed=SEED):
    torch.manual_seed(seed); g = torch.Generator().manual_seed(seed)
    net = model()                             # initialized after seeding
    opt = torch.optim.Adam(net.parameters(), lr=1e-3)
    for _ in range(epochs):
        perm = torch.randperm(len(X), generator=g)
        for i in range(0, len(X), 32):
            b = perm[i:i + 32]
            opt.zero_grad(); F.cross_entropy(net(X[b]), y[b]).backward(); opt.step()
    return net

def logits(net, X):
    with torch.no_grad():
        return net(X)

gen = torch.Generator().manual_seed(0)
train_g, test_g = make_shapes(200, seed=0), make_shapes(100, seed=1)
any_g = [(c, P, rot_any(np.random.default_rng(2 + c), len(P))) for c, P, _ in test_g]   # same shapes, any rotation
ytr, yte = torch.arange(5).repeat_interleave(200), torch.arange(5).repeat_interleave(100)
Xtr = build(train_g, 512, 16, 32, gen)
tests = {"base": build(test_g, 512, 16, 32, gen), "any rot": build(any_g, 512, 16, 32, gen),
         "half": build(test_g, 128, 8, 16, gen), "double": build(test_g, 2048, 32, 64, gen)}
models = {"point": PointNet, "voxel solid": VoxelCNN, "voxel shell": VoxelCNN, "multi-view": MultiViewCNN}
nets = {k: train(m, Xtr[k], ytr) for k, m in models.items()}

def acc(k, X):
    return (logits(nets[k], X).argmax(1) == yte).float().mean().item()
def resample(X, size):                        # linear interpolation back to the training resolution
    mode = "trilinear" if X.dim() == 5 else "bilinear"
    return F.interpolate(X, size=(size,) * (X.dim() - 2), mode=mode, align_corners=False)

print("parameters:", {k: sum(p.numel() for p in nets[k].parameters()) for k in nets})
print(f"{'network':13s}{'base':>7s}{'any rot':>9s}{'half':>7s}{'double':>8s}{'half->base':>12s}{'double->base':>14s}")
for k in nets:
    row = [acc(k, tests[c][k]) for c in ("base", "any rot", "half", "double")]
    if k != "point":
        size = 32 if k == "multi-view" else 16
        row += [acc(k, resample(tests["half"][k], size)), acc(k, resample(tests["double"][k], size))]
    print(f"{k:13s}" + "".join(f"{v:{w}.3f}" for v, w in zip(row, [7, 9, 7, 8, 12, 14])))
for mode in ("solid", "shell"):
    empty = (tests["half"]["voxel " + mode].sum((1, 2, 3, 4)) == 0).sum().item()
    print(f"8^3 {mode} grids with no occupied voxel: {empty} of 500")

# Turns about z: 30 degrees permutes the 12 cameras; a 90-degree turn of the voxel grid permutes its cells.
base_mv = logits(nets["multi-view"], tests["base"]["multi-view"])
for deg in (30, 15):
    turned = [(c, P, rot_z(math.radians(deg)).expand(len(P), 3, 3) @ R) for c, P, R in test_g]
    L = logits(nets["multi-view"], torch.cat([render(g, 32) for g in turned]))
    print(f"multi-view, {deg:2d}-degree turn: max |logit change| {(L - base_mv).abs().max().item():.1e}, "
          f"class changed for {(L.argmax(1) != base_mv.argmax(1)).sum().item()} of 500")
for k in ("voxel solid", "voxel shell"):
    X = tests["base"][k]
    L0, L = logits(nets[k], X), logits(nets[k], torch.rot90(X, 1, (2, 3)))
    print(f"{k}, 90-degree grid turn: max |logit change| {(L - L0).abs().max().item():.1e}, "
          f"class changed for {(L.argmax(1) != L0.argmax(1)).sum().item()} of 500")

Output (python script.py 0, training seed 0):

parameters: {'point': 58373, 'voxel solid': 59285, 'voxel shell': 59285, 'multi-view': 60229}
network         base  any rot   half  double  half->base  double->base
point          0.946    0.288  0.876   0.972
voxel solid    0.964    0.386  0.418   0.374       0.810         0.892
voxel shell    0.996    0.336  0.370   0.572       0.614         0.908
multi-view     1.000    0.294  0.558   0.728       0.940         1.000
8^3 solid grids with no occupied voxel: 15 of 500
8^3 shell grids with no occupied voxel: 0 of 500
multi-view, 30-degree turn: max |logit change| 0.0e+00, class changed for 0 of 500
multi-view, 15-degree turn: max |logit change| 5.8e+00, class changed for 0 of 500
voxel solid, 90-degree grid turn: max |logit change| 3.3e+00, class changed for 5 of 500
voxel shell, 90-degree grid turn: max |logit change| 4.7e+00, class changed for 0 of 500

Base task. All three networks learn the task from their own representation: on shapes posed as in training, seed 0 gives 0.946 for the point network, 0.964 and 0.996 for the two voxel grids and 1.000 for the multi-view network. The script was also run with training seeds 1 and 2. Over the three seeds the base accuracies were 0.946–1.000 (point), 0.918–0.964 (solid voxels), 0.896–0.996 (shell voxels) and 1.000 (multi-view each time). With one data set and three seeds per network, differences of a few points in the base column do not rank the representations; the larger effects below hold for every seed.

Rotations. Turned arbitrarily in 3D, every network falls to 0.28–0.39 over the three seeds, much nearer the 0.2 of guessing than its base accuracy. None of them is invariant to rotation, and each learned the up axis of its training data; in our reading, a cylinder lying on its side is a new object for all three. This is the effect behind VoxNet’s canonical-pose ablation and behind the fixed up axis of MVCNN’s 12-view set-up, here for rotations out of the vertical rather than about it, the axis of VoxNet’s augmentation and voting. The structural lines show where invariance does exist. Turning a shape by 30° about $z$ moves every camera onto its neighbour’s position, the 12 depth images are permuted, and the element-wise maximum makes the logits identical, exactly, for every seed. A 15° turn produces new images and logits that change by up to 5.8 for seed 0, although no predicted class changed for any seed: that robustness was learned from the turned training shapes, not built in. The voxel networks have the same relationship to the grid: a 90° turn of the grid only permutes its cells, yet the logits change and, for seed 0, 5 of 500 solid-grid predictions change class, because 3D convolutions are equivariant to translations of the grid, not to its rotations.

Resolution. The point network barely notices the number of points: with 128 or 2,048 points it stays at 0.876–0.996 and 0.956–1.000 over the three seeds. The maximum over a shared per-point function depends on where the surface is, and a denser or sparser sample of the same surface gives a similar maximum (our reading; the PointNet example tests point dropout and its critical points in detail). The grid networks fed off-resolution grids directly collapse: at half resolution the solid grid gives 0.064–0.418 and the shell 0.146–0.370 over the seeds, and at double resolution 0.344–0.484 and 0.438–0.572. A grid at another resolution shows the same shape at another scale in cells, so every kernel covers a different metric size than in training and the features the classifier learned no longer appear; the multi-view network, also a grid network, drops in the same way, to 0.270–0.596 and 0.728–0.882. Resampling the grids to the training resolution restores most of the accuracy: at double resolution 0.882–0.948 (solid), 0.824–0.922 (shell) and 1.000 (multi-view), and at half resolution 0.930–0.978 for the multi-view network. What resampling cannot restore is detail that a coarse grid never held. At $8^3$, 15 of the 500 test shapes (the same count for every seed, since the data are fixed) have no voxel centre inside them and become empty solid grids, and the solid and shell networks reach only 0.810–0.840 and 0.614–0.748 on resampled half-resolution grids. In this example, then, the resolution of the input matters for grids in two ways: as a change of scale, which resampling removes, and as a loss of detail, which it does not. Qi et al. (2016) and OctNet’s authors measured the effect of resolution on ModelNet’s CAD shapes.

This example was changed once after its first run. In that run the point network was built before the random seed was set, so two identical runs gave different point-network results (base accuracy 0.940 and 1.000); the other networks were unaffected. We moved the construction of every network after the seeding, added the training seed as an argument, and, because the first run had shown how much the point network’s result depends on its initialization, ran seeds 1 and 2 as well and report the ranges above. The data, networks, epochs and conditions are those of the first run.

Common Misconceptions

• “Voxels lose to points and views because 3D convolution is weak.” Qi et al. (2016) found both an architecture gap and a resolution gap, and closed the first at equal 3D resolution. Sparse voxel networks later removed much of the resolution limit and process whole rooms and LiDAR sweeps (sparse convolutional networks).
• “Point networks are rotation-invariant.” PointNet is invariant to the order of the points, not to their orientation; its transform networks are learned from the training data and guarantee nothing.
• “A representation that keeps more information gives a better network.” A $30^3$ grid holds about as many values as a $227^2$ image (Qi et al., 2016), yet the earlier volumetric networks that Qi et al. compared scored below multi-view networks on such images; what a network can exploit depends on the operators available for the representation, its pretraining and its training data.

Where It Is Used

• Recognition and segmentation of scans: indoor mapping and robotics (point cloud segmentation, robotic manipulation), and LiDAR perception for driving (3D object detection, PointPillars, occupancy prediction, sensor fusion for autonomous driving).
• Reconstruction: learned implicit surfaces in surface reconstruction and 3D reconstruction, and feed-forward geometry from images (DUSt3R, 3D foundation models).
• Human and object modelling: fitting parametric human body models and estimating 6-DoF object pose from images and depth.
• Registration and motion: learned features for point cloud registration and scene flow.