3D Occupancy Prediction

3D occupancy prediction labels every small cube of space around a vehicle or robot as free or occupied and, if occupied, says what occupies it: road, car, pedestrian, vegetation, building, or an object of no listed class. The output is a voxel grid, often called a semantic occupancy grid, and the networks that produce it from cameras are often called occupancy networks. Where 3D object detection describes each object by a box and bird’s-eye-view perception flattens the scene onto a ground-plane grid, an occupancy grid keeps the height axis and the irregular shapes of objects: the arm of a crane, an overhanging branch, a pile of debris that matches no category. The task grew out of two older lines of work, the occupancy grids of mobile robotics and semantic scene completion from depth, and became a driving benchmark in 2023 with labels built from accumulated LiDAR scans. Much of what is distinctive about it lies in evaluation: which voxels can be scored at all, given that the sensors never saw many of them.

Problem Definition

Tian et al. (2023), who built the Occ3D benchmark, state the task for cameras as follows. The input is a sequence of $T$ past frames, each with $N$ surround-view images $I_{i,t} \in \mathbb{R}^{H_i \times W_i \times 3}$ and known intrinsic matrices $K_i$ and extrinsics $[R_i \mid t_i]$ (camera parameters). The output assigns each voxel of a fixed grid around the ego vehicle an occupancy state and a semantic label. In the ground truth the state is “occupied”, “free” or “unobserved”: a voxel on a vehicle is (“occupied”, “vehicle”), a voxel of empty air is (“free”, none), and occupied voxels of objects outside the label set are “general objects”. In the CVPR 2023 challenge built on this format, models predict only free or an occupied class and not the masks; the unobserved state is a property of the labels, used to decide which voxels are scored.

Written compactly, a model is a function

$$f : \{I_{i,t}, K_i, R_i, t_i\} \;\longmapsto\; \hat{Y} \in \{0, 1, \dots, C\}^{X \times Y \times Z},$$

with 0 for free and $1, \dots, C$ for the semantic classes. The same formulation holds with LiDAR, radar or depth as input; only the encoder changes.

Relation to occupancy grids

Dividing space into cells that each hold a belief about occupancy is a classical technique of mobile robotics. Tian et al. (2023) cite Moravec and Elfes’s “High Resolution Maps from Wide Angle Sonar” (1985) for occupancy grid mapping, together with later work; Elfes’s “Using Occupancy Grids for Mobile Robot Perception and Navigation” (1989) is another early paper on the grids. Neither was read for this article, and nothing here is attributed to them beyond their titles. Tian et al. summarize occupancy grid mapping as estimating probabilistic maps from a sequence of noisy range measurements, solvable in a Bayesian framework, and they distinguish it from occupancy prediction on two counts: grid mapping needs range sensors and assumes the scene is static, whereas occupancy prediction can work from images alone and in dynamic scenes.

The Bayesian update behind such grids follows from Bayes’ rule if cells are treated as independent and the world as static (this derivation is ours, not taken from the unread papers). With $l_t = \log \frac{p(o \mid z_{1:t})}{1 - p(o \mid z_{1:t})}$ the log-odds that a cell is occupied after measurements $z_{1:t}$, and $l_0$ the prior log-odds,

$$l_t = l_{t-1} + \log \frac{p(o \mid z_t)}{1 - p(o \mid z_t)} - l_0 ,$$

so each range reading adds evidence for occupancy at its endpoint and for free space along the beam. The bird’s-eye-view perception article describes how Pyramid Occupancy Networks used such an update to fuse cameras and time steps on a ground-plane grid. Learned occupancy prediction replaces the hand-made sensor model $p(o \mid z_t)$ by a network that reasons about appearance and context, and adds semantics.

Relation to semantic scene completion

Song et al. (2017) named semantic scene completion (SSC): producing a complete voxel grid of occupancy and semantic labels for a scene from a single depth map, including voxels hidden behind the visible surface. Their motivating example is inferring the seat and legs of a chair from its visible top behind a table. Occupancy prediction inherits SSC’s output format and its two scores, but the boundary between the tasks is drawn differently by different authors. Tian et al. (2023) write that SSC focuses on inferring occluded regions from visible parts, while occupancy prediction “does not intend to estimate the invisible regions”, and that SSC usually deals with static scenes. Other occupancy benchmarks score hidden voxels too (see Evaluation Metrics below), and Liu et al. (2024) argue that scoring only visible voxels reduces occupancy to depth estimation with categories. In practice the two names now cover one family of methods, separated mainly by which voxels are scored.

A naming clash is worth knowing: Occupancy Networks (Mescheder et al., 2019), described in the 3D representations article, represent object shape as a continuous function learned by a network. Liu et al. (2024) note that the name was first used for that work, whereas the driving “occupancy networks” mostly draw on bird’s-eye-view methods and predict semantics on a discrete grid.

Relation to BEV perception and 3D detection

Occ3D’s authors motivate the task by two limits of box-based 3D detection: a box erases geometric detail, such as a mechanical arm protruding from a construction vehicle, and categories in the open world cannot all be listed, so objects such as trash cans on the street are often not annotated. Huang et al. (2023) make the parallel argument against bird’s-eye-view features: a single ground-plane feature per cell summarizes a whole vertical pillar, which, they argue, suffices for coarse boxes of common objects but not for fine 3D structure. The methods themselves stay close to BEV perception: Occ3D’s baselines replace the detection heads of BEVDet and BEVFormer with an occupancy decoder while keeping their BEV encoders, and SurroundOcc and CTF-Occ lift image features with BEVFormer’s deformable spatial cross-attention, applied to 3D voxel queries instead of ground-plane queries.

Input and Output

The inputs are those of the autonomous-driving perception stack: one to six cameras with calibration, often a short history of frames with ego poses, and in LiDAR and fusion variants one or more LiDAR sweeps. The output is a dense label grid. The grid’s extent, voxel size and coordinate frame differ between benchmarks:

Benchmark	Region (m)	Voxel (m)	Grid	Classes
NYUv2 / SUNCG (Song et al., 2017)	4.8 × 4.8 × 2.88, camera frustum	0.02	240 × 144 × 240	11 + free
SemanticKITTI SSC	51.2 ahead, ±25.6 sideways, 6.4 high	0.2	256 × 256 × 32	19 + free
Occ3D-nuScenes	x, y ∈ [−40, 40], z ∈ [−1, 5.4]	0.4	200 × 200 × 16	16 + general object + free
SurroundOcc (nuScenes)	x, y ∈ [−50, 50], z ∈ [−5, 3]	0.5	200 × 200 × 16	16 + free
OpenOccupancy (nuScenes-Occupancy)	x, y ∈ [−51.2, 51.2], z ∈ [−5, 3]	0.2	512 × 512 × 40	16 + noise (ignored) + free

The Occ3D-Waymo grid is described inconsistently: the paper’s Table 1 and §3.2 give 0.05 m voxels over ±80 m (3200 × 3200 × 128), the README at commit 6e8563a lists a 0.1 × 0.1 × 0.2 m version over ±80 m with z from −5 to 7.8 m (1600 × 1600 × 64) and a 0.4 m version over ±40 m with z from −1 to 5.4 m (200 × 200 × 16), and the paper’s experiments use 0.4 m voxels within ±40 m and z from −5 to 7.8 m. Occ3D-Waymo has 14 classes plus general object and free.

The output is usually large and mostly empty. The 200 × 200 × 16 nuScenes grids hold 640,000 voxels; Liu et al. (2024) report that more than 90% of them are free, and Song et al. (2017) report a ratio of empty to occupied voxels of about 9:1 in their indoor scenes.

Variants

• Monocular, indoor and outdoor. MonoScene (Cao and de Charette, 2022) predicts SSC from a single RGB image, on NYUv2 indoors and on SemanticKITTI from one colour camera (cam2).
• Surround-view cameras. TPVFormer, SurroundOcc, CTF-Occ and SparseOcc use all six nuScenes cameras (five on Waymo, which has no rear camera) and predict the full surrounding grid.
• LiDAR and fusion. SemanticKITTI’s SSC benchmark takes a single voxelized LiDAR scan as input. OpenOccupancy reports camera, LiDAR and fused baselines; Occ3D adds a LiDAR-only and a camera-LiDAR fusion baseline on Waymo (methods in sensor fusion for autonomous driving).
• Sparse supervision. TPVFormer trains from sparse LiDAR point labels and predicts all voxels at test time; with no benchmark for that setting, its authors evaluated it only qualitatively and on proxy tasks.
• Panoptic occupancy. SparseOcc replaces semantic queries by instance queries to separate individual objects in the grid, the 3D counterpart of panoptic segmentation.
• Occupancy flow. Occ3D notes that its framework supports extra per-voxel attributes such as instance IDs and motion vectors and leaves them to future work; the SparseOcc repository lists an entry in the CVPR 2024 “Occupancy and Flow” challenge.

Challenges

• Cubic cost. Voxel features cost $O(XYZ)$ in memory and compute, against $O(XY)$ for a BEV map (Huang et al., 2023). SparseOcc’s authors give 2 to 3 frames per second for existing dense methods on an A100 GPU, the motivation for its sparse design.
• Ground truth. Wang et al. (2023) call direct human annotation of large-scale occupancy labels almost impractical, and Tian et al. (2023) note that images alone lack the depth to annotate from. Labels are therefore built from accumulated LiDAR, which, as Occ3D’s authors put it, raises three problems: LiDAR is sparse, accumulated points include surfaces that are occluded in the current view, and 3D voxels misalign with 2D images through calibration and pose errors.
• Moving objects. Accumulating scans over time smears moving objects unless their points are aggregated in their own box frames; objects without boxes (an animal) or that deform (a pedestrian swinging their arms) remain blurred (Tian et al., 2023, Limitations).
• Unobserved space. Even accumulated LiDAR leaves voxels inside objects and behind walls unseen. Whether such voxels are labelled free, ignored or scored decides what the benchmark measures.
• Class imbalance. Free space dominates all voxels, and a few classes dominate the occupied ones: the SemanticKITTI class frequencies printed in MonoScene’s and TPVFormer’s tables are 39.3% for vegetation, 15.3% for road and 14.1% for building, against 0.03% for bicycle and for motorcycle; Song et al. (2017) balanced empty and occupied samples during training, and SparseOcc weights each class by the inverse of its voxel share.
• Depth from images. A camera does not measure depth, so a camera-based model must place surfaces along each ray from appearance and context, as in monocular depth estimation.

Approaches

From depth: semantic scene completion

SSCNet (Song et al., 2017) is a 3D convolutional network that maps a single depth image, encoded as a volume, to one of $N+1$ labels for every voxel in the camera frustum. The encoding is a truncated signed distance function (the representation fused in TSDF fusion) with two changes: distances are computed to the nearest point anywhere on the observed surface rather than along the line of sight, to remove view dependence, and the value is flipped, $d_\text{flipped} = \operatorname{sign}(d)(d_{\max} - d)$, so the strongest gradients sit on the surface rather than in empty space along occlusion boundaries. A dilation-based context module enlarges the receptive field, which the authors found crucial: a chair seen in isolation is hard to complete, but with the table and floor around it the shape is constrained. Training data came from SUNCG, a dataset of more than 45,000 synthetic indoor scenes built from labelled object meshes, voxelized at 2 cm. On the NYU test set with Kinect depth, training on SUNCG and fine-tuning on NYU gave a scene-completion IoU of 56.6 and an average SSC IoU of 30.5 over eleven classes (Table 1).

Behley et al. (2019) carried SSC outdoors with SemanticKITTI. Because the recording car drives past objects, superimposing future LiDAR scans records their backs and sides, which a single initial scan misses through self-occlusion. The input is one voxelized scan; the target is the accumulated, labelled scene. Their baselines were SSCNet and the two-stream TS3D, which adds an image segmentation; replacing TS3D’s backbone by SATNet, which handles the full output resolution, gave much better scores than the SSCNet-based variants (Table 5).

From images: lifting to voxels

MonoScene (Cao and de Charette, 2022), which its authors present as the first SSC method to work from a single RGB image both indoors and outdoors, bridges a 2D and a 3D network. A 2D U-Net extracts multi-scale image features; its “features line of sight projection” (FLoSP) projects every voxel centroid into the image and samples features at scales 1, 2, 4 and 8, so each voxel receives the features of every pixel on its line of sight and the 3D U-Net decides which depth they belong to. A 3D context relation prior layer predicts relations between supervoxels and voxels, four kinds formed by whether at least one of the pair is free or both are occupied and whether their classes are similar or different, to give the network a global receptive field. Two losses act on groups of voxels: a scene-class affinity loss that maximizes differentiable per-class precision, recall and specificity, and a frustum proportion loss that aligns the class distribution inside image-space frustums, giving supervision beyond occlusions. On the SemanticKITTI hidden test set it reached an IoU of 34.16 and an mIoU of 11.08 from RGB alone, against 34.00 and 8.97 for the best RGB-inferred baseline (JS3C-Net run on a point cloud inferred from the image); on the NYUv2 test set, 42.51 and 26.94 (Table 1).

TPVFormer (Huang et al., 2023) keeps three orthogonal feature planes instead of a full volume: a top view $T^{HW} \in \mathbb{R}^{H \times W \times C}$, a side view $T^{DH}$ and a front view $T^{WD}$. A 3D point’s feature is the sum of the features bilinearly sampled at its projections onto the three planes, so the representation can be queried at any resolution, and its cost grows with the plane areas rather than the volume. A transformer encoder fills the planes with image features through cross-attention from plane queries to the images and then lets the planes attend to each other. Trained only with sparse LiDAR point labels, it predicts every voxel at test time. On nuScenes LiDAR segmentation (test set), where the points are used only as query locations, the camera-only TPVFormer-Base reached 69.4 mIoU, comparable to some LiDAR methods of the time (PolarNet 69.4) but below the best (LidarMultiNet 81.4) (Table 1). On SemanticKITTI SSC (test set) it scored 34.25 IoU and 11.26 mIoU, slightly above MonoScene, with 6.0 million parameters against MonoScene’s 15.7 million (Table 2).

SurroundOcc (Wei et al., 2023) lifts multi-scale features from all six cameras into 3D voxel queries with spatial 2D-3D attention, upsamples the volume with 3D convolutions and supervises every level. Its second contribution is dense labels (see Datasets and Benchmarks). Trained and tested on those labels on the nuScenes validation set, it reported 31.49 IoU and 20.30 mIoU, against 30.86 and 17.10 for TPVFormer retrained on the same dense labels and 30.50 and 16.75 for BEVFormer (Table 1).

CTF-Occ (Tian et al., 2023) also uses voxel queries and cross-attention, but coarse to fine: at each level a binary classifier scores voxels as empty or not, only the top-$K$ most uncertain or non-empty tokens (a ratio of 0.2) attend to image features, 3D convolutions mix the volume, and the result is upsampled. An implicit decoder, an MLP taking the voxel feature and a 3D coordinate inside the voxel, can output labels at arbitrary resolution.

FB-OCC is described by its official repository as an occupancy method based on forward-backward view transformation; its technical report is on arXiv and was not read, so its architecture is not described further here. The repository states that it won the Outstanding Champion and Innovation Awards of the CVPR 2023 Autonomous Driving Challenge, and Liu et al. (2024) describe it as the winner of the CVPR 2023 occupancy challenge, with several elaborate components: forward-backward view transformation, a depth network and joint depth and semantic pre-training.

SparseOcc (Liu et al., 2024) drops dense volumes altogether. A sparse voxel decoder starts from coarse voxel queries, upsamples each kept voxel into eight children per layer, scores their occupancy and keeps only the top $k$ (32,000 voxels at full resolution, 5% of 640,000, was best in their ablation), sampling image features at 3D points around each voxel as SparseBEV does for detection. A mask transformer in the style of Mask2Former then assigns class masks to the kept voxels, sampling image features only at points inside each predicted mask. The authors report 17.3 frames per second with eight frames of input on an A100 GPU (Table 1); in a single-frame ablation the sparse decoder ran at 24.0 frames per second against 6.3 for a dense coarse-to-fine decoder with the same RayIoU (Table 2). The trade-off they list: voxels pruned early cannot be recovered later.

Industry descriptions

Liu et al. (2024) cite a 2021 Tesla AI Day presentation for the vision-centric occupancy task, which they cite as a video. That presentation was not watched for this article, and no written official description by Tesla was found, so its design is not described here.

Why the field moved this way

Read as a whole (our summary), the sequence above follows the sources of ground truth and the cost of 3D. SSC began indoors, where synthetic scenes give complete labels and the volume is small (Song et al., 2017). SemanticKITTI made real outdoor labels possible by accumulating LiDAR along a drive, but only for a volume ahead of the car. The bird’s-eye-view methods of 2020–2022 showed that surround-camera features could be lifted into a common metric frame; occupancy prediction reused that machinery and kept the height axis once dense surround labels existed (Occ3D, SurroundOcc, OpenOccupancy, all 2023). The cubic cost of the volume then pushed methods towards factorized (TPVFormer), coarse-to-fine (CTF-Occ, SurroundOcc) and sparse (SparseOcc) designs, and the weaknesses of voxel scores under visibility masks pushed evaluation towards ray-based scores.

Datasets and Benchmarks

• NYUv2 and SUNCG. Song et al. (2017) voxelized mesh annotations of the 1,449 NYU depth maps, with 33 object meshes in 7 categories and other categories approximated by boxes or planes; because the meshes do not always align with the depth, they tested on both the Kinect depth and depth rendered from the annotation. SUNCG provided the synthetic training scenes. The NYU Depth V2 article describes the underlying dataset.
• SemanticKITTI SSC. Built from the KITTI odometry sequences: 19,130 input-target pairs for training, 815 for validation and 3,992 for testing, with test targets withheld. Each target voxel takes the majority label of its points; voxels without points are empty. Voxels never visible from any pose of the car, such as those inside objects or behind walls, are found by ray tracing and ignored in training and evaluation (Behley et al., 2019).
• Occ3D-nuScenes and Occ3D-Waymo. Tian et al. (2023) build labels in three stages. Voxel densification: static background points are accumulated across frames in the world frame, points of moving objects in their box frames, unlabelled sweeps get labels from their $K$ nearest labelled points, and holes are filled by meshing (VDBFusion, a TSDF method, for non-ground classes; local surface fits for the ground). Occlusion reasoning: rays are cast from the LiDAR origins to every point; a voxel is occupied if it holds points, free if a ray passes through it, and unobserved otherwise (the LiDAR mask). Rays are then cast from each camera (to every occupied voxel centre in §3.3.2, through every pixel in Appendix D’s Algorithm 3); along each ray, voxels up to and including the first occupied one are observed by the camera and the rest are not (the camera mask). Image-guided refinement: along camera rays, voxels in front of the first voxel whose label matches the pixel’s segmentation label are set free, which trims objects that LiDAR noise and pose drift made too large. Each nuScenes sample carries six images, the voxel labels and both masks. The split counts disagree across the paper: §3.2 gives 600 training, 150 validation and 150 test scenes; §6.1 gives 700 training and 150 validation scenes; Appendix A gives 700, 150 and 150; the datasheet gives 600, 150 and 250 (and, in another answer, 200 test scenes); the README at commit 6e8563a lists 600, 150 and 250. Liu et al. (2024) describe Occ3D-nuScenes with the 700/150/150 split of nuScenes. The CVPR 2023 3D Occupancy Prediction Challenge used nuScenes-based labels in the same format, with the same masks; its README asks users to cite OccNet (“Scene as Occupancy”) and Occ3D.
• SurroundOcc labels. Wei et al. (2023) stitch all LiDAR frames of a 20-second nuScenes sequence at 20 Hz (about 400 frames and 13 million points), separately for static scenes and for movable objects cut out by their boxes, reconstruct a mesh by Poisson surface reconstruction, voxelize it, and assign each dense voxel the label of its nearest labelled voxel. They note that the nearest-neighbour step is sensitive to noise in the original LiDAR labels. Because nuScenes test labels are not public, they train on the training set and evaluate on validation.
• OpenOccupancy (nuScenes-Occupancy). Wang et al. (2023) start from accumulated LiDAR labels, add pseudo-labels from a trained model only in voxels the LiDAR left empty, and have annotators purify the result in 3D and camera views, about 4,000 human hours in total. The result has about 400,000 occupied voxels per frame, roughly twice the initial annotation, with 28,130 training and 6,019 validation frames. Their baselines found LiDAR better on large structured regions (drivable surface, sidewalk, vegetation) and cameras better on small objects (bicycle, pedestrian, motorcycle, traffic cone), with fusion best in mIoU: the fused M-CONet reached 26.5 IoU and 20.5 mIoU on the validation set, against 21.6 and 13.6 for the camera-only C-CONet and 30.1 and 15.9 for the LiDAR-only L-CONet (Table 2).

The three nuScenes label sets differ in region, voxel size, densification and visibility handling, so scores on one cannot be compared with scores on another.

Evaluation Metrics

Voxel IoU and mIoU

SemanticKITTI adopted the two voxel scores of Song et al. (2017), and the later benchmarks report the same pair. Scene completion IoU treats all occupied classes as one and scores the binary occupied/free prediction; semantic scene completion mIoU averages the per-class intersection over union over the occupied classes, without the free class:

$$\mathrm{IoU} = \frac{TP_o}{TP_o + FP_o + FN_o}, \qquad \mathrm{mIoU} = \frac{1}{C} \sum_{c=1}^{C} \frac{TP_c}{TP_c + FP_c + FN_c}.$$

Cao and de Charette (2022) point out that the two interact: invalidating semantic labels can raise IoU at the cost of mIoU. In the Occ3D challenge code the counts are accumulated over the whole evaluation set, in one confusion matrix, before dividing.

Which voxels are scored

The voxel set is where benchmarks differ most:

• SSCNet scores semantic IoU on observed and occluded voxels, scores scene completion on occluded voxels only, and leaves out voxels outside the view or outside the room (following Firman et al.).
• SemanticKITTI ignores voxels never visible from any pose of the recording drive.
• MonoScene notes that indoor practice scores only observed surfaces and occluded voxels while outdoor practice scores all voxels, and evaluates on all voxels in both settings, calling it the harder choice.
• Occ3D scores only voxels observed in the current camera view: its README states that only the camera mask is used for evaluation and that unobserved voxels are excluded from mIoU (the paper says voxels observed in both the LiDAR and the camera views). In the challenge code, mIoU averages 17 classes (16 classes and “others”), excluding free.
• SurroundOcc’s official evaluation scores every voxel of its dense grid not marked with the ignore label 255, so hidden parts of the dense labels count.
• OpenOccupancy’s protocol states only that the noise class is ignored.

These choices change what a score rewards. Without a mask, a camera model is penalized for voxels it had no means of seeing; with a camera mask, the voxels behind each visible surface drop out of the score.

RayIoU

Liu et al. (2024) argue that the camera mask makes voxel mIoU exploitable. A model trained with the mask receives no supervision behind surfaces, fills that space with duplicated predictions, and so predicts a thick surface. Because the mask removes most voxels behind the true surface from the score, a thick prediction that starts one voxel too close is penalized only for the one free voxel in front, while a prediction one voxel too far scores nothing, however thick (their Fig. 5 gives IoU 0.5 and 0 for a thick wall predicted one voxel closer or farther). They report that training BEVFormer with the visible mask inflates its mIoU by 5 to 15 points while its predictions become visibly thicker and noisier (Fig. 4). They also object that a thin prediction off by one voxel scores zero.

Their RayIoU scores query rays instead of voxels. Rays emulate a LiDAR: they are cast into the predicted grid and into the ground-truth grid, and each yields the distance to the first occupied voxel and its class; rays that hit nothing in the ground truth are dropped. A ray is a true positive for class $c$ when both grids give class $c$ and the L1 distance error is below a threshold, and

$$\mathrm{RayIoU} = \frac{1}{C} \sum_{c=1}^{C} \frac{TP_c}{TP_c + FP_c + FN_c},$$

averaged over thresholds of 1, 2 and 4 m. Rays are resampled so that near and far ranges carry similar weight, and they originate from 8 LiDAR positions along the ego path (past, current and future), which lets the score reach surfaces hidden from the current position. In the official code (loaders/ray_metrics.py), the first ten ray elevations point downward at $-(\pi/2 - \arctan(k+1))$ for $k = 0, \dots, 9$, so that their ground intersections are spaced evenly, further elevations continue at the last spacing up to about 0.21 rad, azimuths are spaced 1° apart, and the free class is excluded from the mean.

Their Table 1 (Occ3D-nuScenes validation) shows how far the two scores can disagree. FB-Occ with 16 frames (ResNet-50, 90 epochs) scored 33.5 RayIoU and 39.1 mIoU; SparseOcc with 8 frames (ResNet-50, 24 epochs) scored 34.0 RayIoU but 30.1 mIoU; BEVFormer with 4 frames (ResNet-101, 24 epochs) scored 32.4 RayIoU and 39.2 mIoU. The repository’s README states that SparseOcc, being fully sparse, always predicts a thin surface and so cannot exploit the old metric. The repository’s README at commit af4d9df reports newer v1.1 results (36.8 RayIoU after 24 epochs) and says the implementation differs slightly from the paper.

Reported results

On Occ3D-nuScenes (Table 3 of Tian et al., observed voxels only, the table does not name the split; ResNet-101 for the BEV baselines), the authors’ reproductions gave mIoU 6.06 for MonoScene, 27.83 for TPVFormer, 26.88 for BEVFormer with an occupancy head and 28.53 for CTF-Occ. On Occ3D-Waymo (Table 4), CTF-Occ reached 18.73 mIoU, a single-frame LiDAR-only baseline 29.74 and BEVFormer with LiDAR fusion 39.05.

Example

The script below builds a small street scene with known ground truth, renders a depth image and a label image from a camera, turns them into occupancy predictions by ray casting, and scores them on different voxel sets and by a ray-based score. The scene is a 25.6 × 25.6 × 3.2 m grid of 0.2 m voxels (262,144 voxels) holding a road slab, two cars (the second partly hidden behind the first), a pedestrian, a pole and a building wall, all filled solid, so every voxel has a label, including those no camera ray reaches. The camera sits 1.6 m above the road with a 90° horizontal field of view and a 192 × 96 image.

Ray casting follows the logic of Occ3D’s visibility step, simplified: voxels sampled along a ray before its measured depth are free, the voxel at the hit takes the pixel’s label, and voxels no ray reaches are unknown; where rays disagree, occupied wins over free. Running this on the noise-free image gives the camera visibility mask. Two predictions are built from the image: thin, exactly the ray-cast surface, and thick, which extends every hit 0.6 m (three voxels) behind the surface with the same label, imitating the thickened surfaces Liu et al. (2024) describe. Unknown voxels are predicted free. Each is built from the exact depth and from depth with Gaussian noise of 0.15 m. The voxel scores are IoU and mIoU (free excluded, classes with an empty union skipped, as with np.nanmean in the official codes) on visible voxels, on hidden voxels (occluded or outside the field of view) and on all voxels. The ray score follows SparseOcc’s definition on a smaller ray set: 3,060 LiDAR-like rays (2° azimuth steps over 360°, elevations from −30° to +10°) from the camera position and from a later ego position 6 m ahead, averaged over thresholds of 1, 2 and 4 m. Traversal samples each ray every quarter voxel rather than walking voxels exactly. It runs in about 5 seconds with NumPy 2.3.5.

import numpy as np

VOX, LO, SHAPE = 0.2, np.array([0.0, -12.8, -0.2]), (128, 128, 16)   # 25.6 x 25.6 x 3.2 m
CLASSES = ["road", "car", "pedestrian", "pole", "building"]           # labels 1..5, 0 = free
BOXES = [  # (label, min corner, max corner) in metres; x forward, y left, z up
    (1, (0.0, -12.8, -0.2), (25.6, 12.8, 0.0)),    # road slab
    (2, (8.0, -2.6, 0.0), (12.4, -0.8, 1.6)),       # car A
    (2, (14.0, -3.4, 0.0), (18.4, -1.6, 1.6)),      # car B, partly behind car A
    (3, (15.0, 1.0, 0.0), (15.6, 1.6, 1.8)),        # pedestrian
    (4, (11.0, 3.0, 0.0), (11.4, 3.4, 3.0)),        # pole
    (5, (22.0, -12.8, 0.0), (24.0, 12.8, 3.0)),     # building wall
]
STEP, TMAX = VOX / 4, 40.0
TS = np.arange(STEP / 2, TMAX, STEP)                # sample distances along each ray

def ground_truth():
    """Complete semantic grid: every voxel whose centre lies in a box gets its label."""
    c = LO + (np.indices(SHAPE).reshape(3, -1).T + 0.5) * VOX
    gt = np.zeros(int(np.prod(SHAPE)), np.int64)
    for lab, lo, hi in BOXES:
        gt[np.all((c >= lo) & (c <= hi), axis=1)] = lab
    return gt.reshape(SHAPE)

def camera_rays(w=192, h=96, hfov=90.0):
    f = (w / 2) / np.tan(np.radians(hfov / 2))
    uu, vv = np.meshgrid(np.arange(w) + 0.5 - w / 2, np.arange(h) + 0.5 - h / 2)
    d = np.stack([np.full_like(uu, f), -uu, -vv], -1).reshape(-1, 3)
    return d / np.linalg.norm(d, axis=1, keepdims=True)

def render(origin, dirs):
    """Depth and label image: first ray-box intersection (slab method); inf = sky."""
    depth, label = np.full(len(dirs), np.inf), np.zeros(len(dirs), np.int64)
    with np.errstate(divide="ignore", invalid="ignore"):
        for lab, lo, hi in BOXES:
            t1, t2 = (np.array(lo) - origin) / dirs, (np.array(hi) - origin) / dirs
            tmin = np.nanmax(np.minimum(t1, t2), axis=1)
            tmax = np.nanmin(np.maximum(t1, t2), axis=1)
            hit = (tmax >= tmin) & (tmin > 0) & (tmin < depth)
            depth[hit], label[hit] = tmin[hit], lab
    return depth, label

def lookup(p):
    ijk = np.floor((p - LO) / VOX).astype(np.int64)
    return ijk, np.all((ijk >= 0) & (ijk < SHAPE), axis=-1)

def cast(origin, dirs, depth, label, thick=0.0):
    """Ray-cast a depth image into a grid: -1 unknown, 0 free (before the hit), the
    pixel's label at the hit and for `thick` metres behind it. Occupied wins over free."""
    free, occ = np.zeros(SHAPE, bool), np.zeros(SHAPE, np.int64)
    for k in range(0, len(dirs), 2048):
        d, dep, lab = dirs[k:k + 2048], depth[k:k + 2048], label[k:k + 2048]
        ijk, ok = lookup(origin + d[:, None, :] * TS[None, :, None])
        m = ok & (TS[None, :] < np.minimum(dep, TMAX)[:, None])
        free[tuple(ijk[m].T)] = True
        m = ok & (TS[None, :] >= dep[:, None]) & (TS[None, :] < dep[:, None] + thick)
        occ[tuple(ijk[m].T)] = np.broadcast_to(lab[:, None], m.shape)[m]
        hit = np.isfinite(dep)
        ijk, ok = lookup(origin + d[hit] * (dep[hit] + 1e-4)[:, None])
        occ[tuple(ijk[ok].T)] = lab[hit][ok]
    grid = np.where(free, 0, -1)
    return np.where(occ > 0, occ, grid)

def voxel_scores(pred, gt, mask):
    p, g = pred[mask], gt[mask]
    iou = np.sum((p > 0) & (g > 0)) / np.sum((p > 0) | (g > 0))
    per_class = [np.sum((p == c) & (g == c)) / np.sum((p == c) | (g == c))
                 for c in range(1, 6) if np.any((p == c) | (g == c))]   # skip 0/0, as nanmean
    return iou, np.mean(per_class)

def first_hit(grid, origin, dirs):
    ijk, ok = lookup(origin + dirs[:, None, :] * TS[None, :, None])
    lab = np.where(ok, grid[tuple(np.where(ok[..., None], ijk, 0).transpose(2, 0, 1))], 0)
    hit = lab > 0
    first = np.argmax(hit, axis=1)
    found = hit.any(axis=1)
    return np.where(found, lab[np.arange(len(dirs)), first], 0), np.where(found, TS[first], TMAX)

def lidar_rays():   # 360 deg in 2 deg azimuth steps, elevation -30..+10 deg in 2.5 deg steps
    az, el = np.meshgrid(np.radians(np.arange(0, 360, 2.0)), np.radians(np.arange(-30, 10.1, 2.5)))
    return np.stack([np.cos(el) * np.cos(az), np.cos(el) * np.sin(az), np.sin(el)], -1).reshape(-1, 3)

def ray_iou(pred, gt, origins, dirs, thresholds=(1.0, 2.0, 4.0)):
    """First-hit class and distance per query ray; rays whose ground truth hits nothing
    are dropped; TP = same class and |depth error| < threshold; mean over classes and thresholds."""
    tp, n_gt, n_pred = np.zeros((len(thresholds), 6)), np.zeros(6), np.zeros(6)
    for o in origins:
        lg, dg = first_hit(gt, o, dirs)
        lp, dp = first_hit(pred, o, dirs)
        keep = lg > 0
        lg, dg, lp, dp = lg[keep], dg[keep], lp[keep], dp[keep]
        n_gt += np.bincount(lg, minlength=6)
        n_pred += np.bincount(lp, minlength=6)
        for j, thr in enumerate(thresholds):
            tp[j] += np.bincount(lg[(lg == lp) & (np.abs(dp - dg) < thr)], minlength=6)
    union = n_gt + n_pred - tp
    present = union[0, 1:] > 0                          # skip 0/0, as nanmean
    return np.mean((tp / np.maximum(union, 1))[:, 1:][:, present])

gt = ground_truth()
cam, later = np.array([0.0, 0.0, 1.6]), np.array([6.0, 0.0, 1.6])
dirs = camera_rays()
depth, label = render(cam, dirs)
visible = cast(cam, dirs, depth, label) >= 0          # camera visibility mask from noise-free rays
print(f"voxels: {gt.size}, occupied {np.mean(gt > 0):.1%}, visible {np.mean(visible):.1%}, "
      f"occupied and visible {np.sum(visible & (gt > 0))} of {np.sum(gt > 0)}")
qrays = lidar_rays()
rng = np.random.default_rng(0)
print("noise   surface  | IoU  mIoU visible | IoU  mIoU hidden | IoU  mIoU all | RayIoU cam  cam+later")
for sigma in [0.0, 0.15]:
    noisy = depth + sigma * rng.standard_normal(depth.shape)
    for name, thick in [("thin", 0.0), ("thick", 0.6)]:
        pred = cast(cam, dirs, noisy, label, thick)
        pred[pred < 0] = 0                              # unknown voxels predicted free
        row = [voxel_scores(pred, gt, m) for m in (visible, ~visible, np.ones(SHAPE, bool))]
        r1, r2 = ray_iou(pred, gt, [cam], qrays), ray_iou(pred, gt, [cam, later], qrays)
        print(f"{sigma:.2f} m  {name:6s}  | " + " | ".join(f"{a:.3f} {b:.3f}" for a, b in row)
              + f"  |   {r1:.3f}      {r2:.3f}")

voxels: 262144, occupied 14.8%, visible 52.4%, occupied and visible 3552 of 38893
noise   surface  | IoU  mIoU visible | IoU  mIoU hidden | IoU  mIoU all | RayIoU cam  cam+later
0.00 m  thin    | 1.000 1.000 | 0.000 0.000 | 0.091 0.262  |   0.821      0.747
0.00 m  thick   | 0.978 0.851 | 0.175 0.335 | 0.249 0.396  |   0.743      0.789
0.15 m  thin    | 0.296 0.312 | 0.023 0.047 | 0.063 0.115  |   0.630      0.688
0.15 m  thick   | 0.468 0.448 | 0.186 0.288 | 0.232 0.309  |   0.668      0.766

The first line shows the scale of the problem: the camera sees about half of the grid, but only 3,552 of the 38,893 occupied voxels, because each solid object shows only its front face. The exact ray-cast surface is therefore perfect on visible voxels and worthless on hidden ones, and the score on all voxels (0.091 IoU) is dominated by what the camera never saw. A benchmark that scores all voxels asks for completion of hidden geometry; one that scores only visible voxels asks for something close to depth estimation with labels, which is Liu et al.’s (2024) objection to the camera mask.

Depth noise of 0.15 m, less than one voxel, cuts the thin surface’s visible IoU from 1.000 to 0.296: a surface displaced by one voxel along its ray produces both a false positive and a false negative. The thick surface loses less (0.468), because the mask hides most voxels behind the true surface, where thickening adds false positives, while the extension still covers the true surface voxel when the measured depth falls short. This is the effect SparseOcc describes: under a camera mask, thickness is rewarded. Without noise, however, the thick surface is not free of cost on visible voxels (0.978 IoU and 0.851 mIoU). Our reading, from the code’s construction: the thin surface matches the ground truth on visible voxels, and the thick one differs from it only where an extension wrote a label, so the losses are visible voxels that extensions filled. That can happen at silhouette edges, where the space behind an object is crossed by neighbouring rays and cast lets occupied win over free; behind the pole (0.4 m deep) and the pedestrian (0.6 m) the extension also leaves the object. That mIoU falls much more than IoU suggests that the losses sit in classes with few voxels, which mIoU weights like the road. On hidden voxels the thick surface gains (0.175 IoU against 0), since the cars and building really are solid behind their faces.

The ray score moves much less with noise (0.821 to 0.630 for the thin surface), because a first hit within 1 m of the true distance counts as correct. Before the run we expected the thin and thick surfaces to score the same from the camera position, since their first hits along camera rays coincide; they did not (0.821 against 0.743). The output does not show why; what follows is our reading of the code’s construction. The query rays cover 360° and reach down to −30°, the camera ±45° horizontally and about ±27° vertically, so many query rays meet voxels that both predictions leave free; this keeps both scores below 1 but affects them equally. Inside the field of view a query ray runs between camera rays, so it can enter a voxel filled by a neighbouring ray’s extension before it reaches the true surface, which hurts the thick surface. At the building, 22 m away, the image’s pixel spacing at the centre (22/96 ≈ 0.23 m) exceeds the 0.2 m voxel, so the thin wall can have unhit voxels that a query ray passes through and the thick surface may fill. Such holes cannot lower the voxel scores on visible voxels, because the visibility mask is computed from the same camera rays. Adding the later origin, whose rays start inside the scene, reverses the order: the thick surface scores 0.789 against 0.747. Our reading is that those rays reach surfaces hidden from the camera, some of which the thickened prediction happens to fill. The example is a toy (one frame, box-shaped objects, predictions built by hand rather than learned), but it shows that the same prediction can rank first or last depending on the voxel set and the score.

Applications

The main application is camera-based and fused perception for automated driving, where an occupancy grid can describe drivable space, the shape of irregular obstacles and objects of no listed class. Occ3D’s authors argue that general objects, though rare, matter for safety because detectors with fixed categories usually miss them. SparseOcc’s authors suggest that the road surface could be supplied by a high-definition map or online mapping, leaving the occupancy network the sparser remaining structure. In robotics, Song et al. (2017) motivated scene completion by grasping and obstacle avoidance, which need complete 3D shape from partial views, and by object retrieval, which needs semantics; robot perception and visual navigation describe those settings.

Open Problems

• Label quality. Labels are reconstructions, not measurements. Occ3D’s authors list calibration errors between LiDAR and cameras, motion blur on dynamic objects without boxes and on deforming objects, and coarse general-object labels as limits; SurroundOcc’s authors note label noise propagated by nearest-neighbour assignment.
• What to score. Each voxel set has the drawbacks described above, and RayIoU depends on the choice of ray origins, ray density and distance thresholds. In our reading no score is yet agreed, and published methods report different ones.
• Hidden geometry. Predicting the backs of objects and space behind occluders, the original aim of scene completion, is what masked benchmarks leave out, and the ground truth for it is incomplete (the interior of objects is never scanned, as the SparseOcc README notes).
• Resolution and cost. The common 0.4 to 0.5 m voxels are coarse next to a pole, a curb or a child; finer grids grow cubically, and sparse methods trade completeness for speed, since voxels pruned early cannot return.
• Beyond the training classes and cities. General-object classes absorb everything unlisted, and the main benchmarks come from two datasets recorded in a few cities; how occupancy networks behave on unseen object types, sensors and places is largely unmeasured in the sources read here (our observation).