ScanNet

ScanNet is a dataset of RGB-D video of indoor spaces, reconstructed into textured surface meshes and labelled object by object, introduced at CVPR 2017 by Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser and Matthias Nießner of Stanford University, Princeton University and the Technical University of Munich. The first release (February 2017) has 1,513 scans of 707 distinct spaces, about 2.5 million RGB-D frames, camera poses, meshes and instance-level semantic labels on the meshes. Version 2 (June 2018) re-annotated those scans, added 100 test scans and opened an online benchmark whose 3D semantic and instance segmentation tasks became the standard comparison for semantic segmentation and instance segmentation of indoor scenes. ScanNet200 (2022) evaluates the same scans with 200 categories, and ScanNet++ (2023) is a separate, laser-scanned successor.

Purpose

Dai et al. (2017) argued that RGB-D datasets of the time were orders of magnitude smaller than 2D image datasets and were annotated with 2D polygons or bounding boxes on single frames, or labelled in 3D by the authors themselves, which kept them small. The paper’s two questions were how to let many untrained people capture and annotate scans, and whether the resulting data would train better 3D models for scene understanding. The design answer was a pipeline of a tablet capture app, automatic reconstruction on a server and crowdsourced labelling in a web browser. The paper also reconstructed and annotated the sequences of NYU Depth v2 with the same pipeline, all but nine for which too few frames got valid poses (arXiv appendix A.4).

Dataset Size

Capture. Scans were recorded with a Structure sensor, a commodity depth camera similar in design to the Kinect v1, attached to an iPad Air 2. Depth (640 × 480) and colour (1296 × 968) are synchronised in hardware at 30 Hz, and the app also stores IMU data and timestamps. Depth is stored as 16-bit values with zlib compression and colour as H.264 video at 15 Mbps. Each device was calibrated on the server from sequences its user recorded of a printed checkerboard on a flat surface. While scanning, the app shows a “featurefulness” bar, a log-scale measure of detected colour features, as a rough guide to tracking robustness in the region being scanned (Section 3.1).

Reconstruction. On the server, BundleFusion (Dai et al., 2017, ACM Transactions on Graphics) estimates the camera poses at 1 cm voxel resolution. The frames are then fused into a TSDF with VoxelHashing and meshed with marching cubes at 4 mm voxels. Cleaning merges close vertices and removes small or duplicate components, and decimation produces lower-resolution meshes, each halving the face count. Each mesh is rotated so that $z$ points up, using the floor plane found with an IMU gravity prior, with a PCA of the vertices fixing the rotation about $z$. Short scans, scans with high residual error or few aligned frames, and visibly misaligned reconstructions were discarded (Section 3.2). The arXiv version’s appendix gives a median processing time of 11.3 min per scan (mean 14.9 min).

Counts. Table 2 of the paper reports, for the initial snapshot:

Statistic	ScanNet (2017)
Scans	1,513
RGB-D frames	2,492,518
Floor area (average / total)	22.6 / 34,453 m²
Surface area (average / total)	51.6 / 78,595 m²
Labelled objects (average / total)	24.1 / 36,213

The scans come from 707 distinct spaces, about one month of capture by 20 users at locations in several countries. The abstract speaks of “1513 scenes” where the body says scans; in the release a scan is named scene<space>_<scan>, so scene0000_00 and scene0000_01 are two scans of the same space. The v2 split lists add 100 test scans, scene0707_00 to scene0806_00, one per new space, which makes 1,613 scans of 807 spaces (our count from the lists in the repository). The project page and README still describe the dataset as “more than 1500 scans” and 2.5 million views (checked October 2026).

Each scan provides the raw sensor stream (.sens, with colour, depth, poses and intrinsics), a high-resolution mesh (_vh_clean.ply), the cleaned and decimated mesh used for annotation and evaluation (_vh_clean_2.ply), its over-segmentation, the aggregated annotation, and 2D label and instance images rendered from the 3D labels for every frame (labels only outside the test set). Version 2 added a scene type and a 4 × 4 axis-alignment matrix per scan (changelog, 11 June 2018).

Classes

Crowd workers typed free-form labels. The labels were cleaned with a curated list of canonical names, synonyms and misspellings, labels used fewer than five times were dropped, and the remaining ones were mapped to WordNet synsets and to the category sets of NYU v2, ModelNet and ShapeNet (Section 4; appendix C.5 of the arXiv version). The release distributes the mapping as scannetv2-labels.combined.tsv.

The benchmark uses the NYU40 label set of NYU Depth v2 and evaluates a 20-class subset: wall, floor, cabinet, bed, chair, sofa, table, door, window, bookshelf, picture, counter, desk, curtain, refrigerator, shower curtain, toilet, sink, bathtub and other furniture (NYU40 ids 1–12, 14, 16, 24, 28, 33, 34, 36, 39). Instance tasks drop wall and floor, leaving 18 classes. Every other NYU40 class, such as blinds, shelves, lamps or “other prop”, is ignored in evaluation.

ScanNet200 (Rozenberszki, Litany & Dai, 2022) goes back to the raw labels: 607 raw categories, 550 after merging near-duplicates, of which the 200 with the most instances form the benchmark vocabulary, so that every category has at least 10 samples. The categories are split by the number of labelled surface points in the training set into head (66), common (68) and tail (66) sets. Scene types have 20 labels, of which the scene classification task evaluates 13, from apartment and bathroom to office and “misc.” (benchmark documentation; evaluate_scene_type.py).

Annotation Types

Instance labels on the mesh. The labelling interface loads the low-resolution mesh and a conservative over-segmentation computed with a normal-based graph cut. Workers paint segments with a label by clicking and dragging, much as in a 2D paint program, so the unit of annotation is a segment, not a vertex or a polygon (Section 3.3). A connectivity check forbids giving one label to disconnected surfaces; the paper reports that without it some workers painted many surfaces with a few labels or merged several objects. A turntable animation at the start was needed because many workers otherwise never moved the camera. Autocomplete offers labels that other workers have used more than five times, while new text is always allowed. Each worker had to cover at least 25% of the surface and was encouraged to cover 50%; workers were paid $0.50 per task (appendix C.4).

The paper reports 3,391 annotation tasks for the 1,513 scans, by more than 500 Mechanical Turk workers, with 2.3 workers per scan on average. In the arXiv appendix, annotation took a median of 16.8 min per scan summed over workers (mean 22.3 min). Overlapping annotations are merged into a consensus: a majority vote per segment gives the class, a greedy union of annotations that share at least 50% of their segments gives the instances, and the final labelling intersects the two (appendix C.5). Labels are then propagated to the high-resolution mesh by nearest neighbours and to the RGB-D frames by rendering the labelled mesh from each camera pose. The projected 2D labels covered 76% of pixels on average (appendix A.3).

Version 2 annotations. The v2 release replaced the annotations of all 1,513 scans, raising surface coverage to about 90% from 63%, and removed from the meshes the segments that annotators marked “remove” (changelog).

CAD model alignment. A second task asked workers to place ShapeNet CAD models on an already labelled scan: clicking an object retrieved ShapeNetCore models of the same category and placed one over the object’s oriented bounding box, which the worker then adjusted. The paper reports 107 such scene interpretations on a subset of 52 scans by 106 workers, with 681 placed models (296 distinct). Finding a close match in ShapeNet was the main difficulty. The later Scan2CAD dataset (Avetisyan et al., 2019) annotated alignments separately: 14,225 ShapeNet models matched by keypoints to objects in 1,506 ScanNet scans (abstract), with its own benchmark.

Splits

The repository’s Tasks/Benchmark directory lists the scans of each split (commit 3830fce):

Version	Train	Validation	Test
v1	1,045	156	312
v2	1,201	312	100

The v2 training list is exactly the union of the v1 training and validation lists, and the v2 validation list is exactly the v1 test list (our check). The paper’s Table 3 counts 1,201 training and 312 test scans for semantic voxel labelling, which matches v1 with validation merged into training, but 1,205 training scans for object classification. In both versions all scans of a space fall in the same split: no space id appears in two lists (our check), so a model is never tested on another scan of a room it has seen.

The 100 v2 test scans have hidden labels. For the 2D tasks the release offers scannet_frames_25k, about 25,000 frames sampled roughly every 100th frame, and the benchmark evaluates the frames listed in scannet_frames_test. The benchmark’s policy allows parameter tuning on the 1,513 released scans only, asks that the test server be used once for a final system, and blocks updating a method’s test results for two weeks after a submission. ScanNet200 keeps the same scene splits.

Tasks

The 2017 paper used ScanNet for three tasks:

• 3D object classification of partially scanned objects cut out by their bounding boxes, with 17 categories. Table 3 gives 9,305 training and 2,606 test instances; the text gives 9,677 training instances.
• Semantic voxel labelling: a label for every visible surface voxel, over 20 classes plus free space. Scenes are voxelized at 2 cm, and the network labels one column at a time from a 1.5 m × 1.5 m × 3 m volume of about 4.8 cm voxels around it.
• 3D model retrieval of ShapeNetCore models for scanned objects.

The benchmark opened with v2 in June 2018 with five tasks on the 100 test scans: 3D semantic label prediction per mesh vertex, 3D semantic instance prediction (a binary vertex mask, a class and a confidence per instance), the corresponding 2D semantic label and 2D semantic instance tasks on test frames, all four on the 20-class label set, and scene type classification. ScanNet200 adds 3D semantic label and 3D semantic instance tasks with the 200 categories. CAD alignment is evaluated by the separate Scan2CAD benchmark. 3D object detection with axis-aligned boxes is a common use of ScanNet without an official benchmark (see Common Uses).

Evaluation

The benchmark scores every 3D prediction on the vertices of each test scan’s _vh_clean_2.ply mesh. Methods that work on points, voxels or images must transfer their labels to those vertices. The protocols below follow the official scripts in BenchmarkScripts at commit 3830fce; the two 3D scripts were last changed in July and November 2018. They are written for Python 2.7 and adapted from the Cityscapes evaluation. We ran them on the example below after a mechanical port to Python 3 (print statements turned into calls, except X, e into except X as e, the removed NumPy aliases np.float and np.bool replaced by float and bool, and a stub for the plyfile import, which evaluation does not use).

3D semantic label (evaluate_semantic_label.py). For each vertex:

• A vertex whose ground-truth label is not one of the 20 classes is skipped. This covers unannotated vertices (label 0) and all other NYU40 classes, so a prediction there costs nothing.
• A prediction outside the 20 classes is counted as an “unknown” column: it is a false negative for the true class but a false positive for none.
• A single confusion matrix is accumulated over all test scans, and the per-class IoU, $\mathrm{TP}/(\mathrm{TP}+\mathrm{FP}+\mathrm{FN})$, is computed from it. Large scans therefore weigh more than small ones.

The script prints the 20 per-class IoUs and no mean. The leaderboard’s “avg iou” is the unweighted mean of the 20 class columns, as the displayed rows confirm to rounding (our arithmetic on three rows, October 2026). A class with no vertices in either ground truth or prediction gets a bare NaN, and the script then fails while printing. That does not happen on the full test set, where every class has ground truth (each has non-zero IoUs on the leaderboard), but it can on a small validation subset.

3D semantic instance (evaluate_semantic_instance.py, with util_3d.py). Ground truth stores one integer per vertex, $1000 \times \text{class} + \text{instance}$. Predictions are listed per scan as a mask file, a class id and a confidence. For each of the 18 classes and each IoU threshold $t$:

• Minimum size. Predicted masks with fewer than 100 vertices are discarded. Ground-truth instances with fewer than 100 vertices are left out of the ground truth.
• Matching. Overlap is the vertex IoU between a mask and a ground-truth instance of the same class, and a match needs IoU strictly greater than $t$. Each ground-truth instance, in id order, takes the first not-yet-used prediction above $t$, in the order of the submission file. A second prediction above $t$ on the same instance is a false positive carrying the lower of the two scores; the true positive keeps the higher. Under the Python 2.7 the scripts were written for, predictions are held in a dictionary whose order is arbitrary; in our reading the order matters only when one mask overlaps two ground-truth instances above $t$, which is possible only for $t < 0.5$.
• Unmatched predictions are false positives unless enough of them lies on ignored vertices. The script adds the vertices that fall on void (wall, floor, unannotated and non-benchmark classes) to those on too-small instances of the same class. If that count is more than $t$ of the mask, the prediction is dropped. A large, confident “table” painted on the floor costs nothing.
• AP. Per class, the precision–recall curve has one point per distinct confidence, plus a point at recall 0 and precision 1; unmatched ground-truth instances stay in the recall denominator. AP is the trapezoidal area under this uninterpolated curve, with no precision envelope; the average precision article explains the usual interpolated variants. A class with ground truth but no prediction scores 0, and a class without ground truth is NaN and is left out of the means.
• Averages. The script reports AP at $t = 0.5$ (AP 50%), at $t = 0.25$ (AP 25%), and “AP” averaged over thresholds and classes.

The thresholds for “AP” are np.arange(0.5, 0.95, 0.05), and NumPy’s arange excludes its end point. The code therefore averages nine thresholds, 0.50 to 0.90, although the benchmark page describes the range as [0.5:0.95:0.05]. The 2D instance script in the same repository uses np.arange(0.5, 1., 0.05), the ten thresholds 0.50 to 0.95 of Cityscapes, and reports no AP 25%. The 3D leaderboard shows AP 50% by default. One more edge case from our runs: if a class has both ground truth and predictions, but every prediction is ignored and every ground-truth instance is missed, the script stops with an IndexError instead of scoring 0.

The 2D tasks use Cityscapes-style scripts on the same label sets, with a 100-pixel minimum for 2D instances. Scene type classification reports per-class IoU and recall over the 13 evaluated types.

Example: a synthetic room scored the official way

The code builds a 5 m × 4 m room as a point cloud on a 5 cm grid, standing in for mesh vertices: two walls, a floor, a cabinet, a bed, two chairs, a table, a toy chair too small to count and a lamp labelled “other prop”. A patch of wall is left unannotated, like a window nobody painted. It then scores a simulated semantic prediction and a list of simulated instance predictions with the rules above.

import numpy as np

# ScanNet benchmark classes (NYU40 ids). Semantic task: 20 classes; instance task: 18 (no wall, floor).
SEM_IDS = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 16, 24, 28, 33, 34, 36, 39]
INST_IDS = SEM_IDS[2:]
NAMES = {1: "wall", 2: "floor", 3: "cabinet", 4: "bed", 5: "chair", 6: "sofa", 7: "table"}

def grid(a, b, step=0.05):                     # cell centres on [a, b]
    return np.arange(a + step / 2, b, step)

def box(x0, x1, y0, y1, z0, z1):               # top and four sides of a box, 5 cm grid
    P = [np.array([(x, y, z1) for x in grid(x0, x1) for y in grid(y0, y1)])]
    for y in (y0, y1):
        P.append(np.array([(x, y, z) for x in grid(x0, x1) for z in grid(z0, z1)]))
    for x in (x0, x1):
        P.append(np.array([(x, y, z) for y in grid(y0, y1) for z in grid(z0, z1)]))
    return np.vstack(P)

# Synthetic room: (nyu40 label, box) per object; label 40 = otherprop (not evaluated)
objects = [(3, (0.0, 0.5, 2.5, 4.0, 0, 1.0)),   # cabinet
           (4, (3.0, 5.0, 0.0, 1.6, 0, 0.5)),   # bed
           (5, (1.5, 2.0, 1.0, 1.5, 0, 0.9)),   # chair A
           (5, (2.2, 2.7, 1.0, 1.5, 0, 0.9)),   # chair B
           (7, (1.5, 2.7, 1.6, 2.6, 0, 0.75)),  # table
           (5, (4.5, 4.7, 3.5, 3.7, 0, 0.2)),   # toy chair, under 100 points
           (40, (4.0, 4.3, 3.0, 3.3, 0, 1.2))]  # lamp (otherprop)
floor = np.array([(x, y, 0.0) for x in grid(0, 5) for y in grid(0, 4)])
under = np.zeros(len(floor), bool)
for _, (x0, x1, y0, y1, _, _) in objects:      # no floor points under objects
    under |= (floor[:, 0] > x0) & (floor[:, 0] < x1) & (floor[:, 1] > y0) & (floor[:, 1] < y1)
floor = floor[~under]
wall_x = np.array([(0.0, y, z) for y in grid(0, 4) for z in grid(0, 2.5)])
wall_y = np.array([(x, 4.0, z) for x in grid(0, 5) for z in grid(0, 2.5)])
parts = [(1, wall_x), (1, wall_y), (2, floor)] + [(l, box(*b)) for l, b in objects]
xyz = np.vstack([p for _, p in parts])
sem = np.concatenate([np.full(len(p), l) for l, p in parts])
inst = np.concatenate([np.full(len(p), l * 1000 + k + 1) for k, (l, p) in enumerate(parts)])
x, y, z = xyz.T
unlabelled = (sem == 1) & (y == 4.0) & (x > 2) & (x < 3) & (z > 0.8) & (z < 2.0)  # a window left unlabelled
sem[unlabelled], inst[unlabelled] = 0, 0
ids = {k: l * 1000 + k + 1 for k, (l, _) in enumerate(parts)}   # label * 1000 + object index
BED, CHAIR_A, CHAIR_B, TABLE, TOY = (ids[k] for k in (4, 5, 6, 7, 8))
print("points", len(xyz), {NAMES.get(l, int(l)): int((sem == l).sum()) for l in np.unique(sem)})

# ---- 3D semantic label task: confusion over evaluated GT points, IoU per class ----
pred = sem.copy()
chairB = inst == CHAIR_B
pred[chairB & (x < 2.35)] = 6                     # part of chair B called sofa
pred[(sem == 1) & (z > 2.2)] = 40                 # top of the walls called otherprop (not evaluated)
pred[sem == 0] = 7                                # unlabelled GT points: anything goes
pred[sem == 40] = 3                               # otherprop GT points: anything goes
pred[(sem == 2) & (x > 2.8) & (y < 1.8)] = 4      # floor next to the bed called bed

def semantic_iou(gt, pr):
    keep = np.isin(gt, SEM_IDS)                   # GT outside the 20 classes is ignored
    g, p = gt[keep], pr[keep]
    p = np.where(np.isin(p, SEM_IDS), p, 41)      # predictions outside them: miss, never FP
    iou = {}
    for c in SEM_IDS:
        tp = np.sum((g == c) & (p == c))
        fn = np.sum(g == c) - tp
        fp = np.sum((p == c) & (g != c))
        iou[c] = tp / (tp + fp + fn) if tp + fp + fn else np.nan
    return iou

iou = semantic_iou(sem, pred)
print("IoU", {NAMES[c]: round(float(v), 3) for c, v in iou.items() if not np.isnan(v)})
print("mean over classes present: %.3f; over all 20 with absent = 0: %.3f"
      % (np.nanmean(list(iou.values())), np.nansum(list(iou.values())) / 20))

# ---- 3D semantic instance task: AP per class, greedy matching as in the official script ----
def m(cond): return np.flatnonzero(cond)
preds = [(4, 0.90, m((inst == BED) & (x > 3.05))),                        # bed, most of it
         (5, 0.80, m(inst == CHAIR_A)),                                   # chair A
         (5, 0.60, m((inst == CHAIR_A) & (z < 0.6))),                     # duplicate of A
         (5, 0.70, m((chairB & (x < 2.35)) | ((sem == 2) & (abs(x - 2.45) < 0.3) & (y < 1.0)))),
         (7, 0.50, m((inst == TABLE) & (x < 2.3))),                       # table, partial
         (7, 0.95, m((sem == 2) & (x > 3.5) & (y > 2.0))),                # a "table" on the floor
         (5, 0.85, m((inst == TOY) | ((sem == 1) & (x > 4.4) & (z < 0.15)))),  # toy chair + wall
         (6, 0.40, m(chairB & (x < 2.35)))]                               # sofa: no sofa in GT
MIN_SIZE = 100                                     # vertices, for GT and predictions
void = ~np.isin(inst // 1000, INST_IDS)            # wall, floor, unlabelled, otherprop
GT = {g: set(m(inst == g)) for g in np.unique(inst) if g // 1000 in INST_IDS}

def iou_sets(a, b): return len(a & b) / len(a | b)

for l, s, idx in preds:
    best = max([iou_sets(set(idx), q) for g, q in GT.items() if g // 1000 == l] or [0])
    print("pred %-6s score %.2f  points %4d  best IoU %.2f  on void %.2f"
          % (NAMES[l], s, len(idx), best, void[idx].mean()))

def ap_class(c, t):
    gts = [q for g, q in GT.items() if g // 1000 == c and len(q) >= MIN_SIZE]
    small = set().union(*[q for g, q in GT.items() if g // 1000 == c and len(q) < MIN_SIZE])
    P = [(s, set(idx)) for l, s, idx in preds if l == c and len(idx) >= MIN_SIZE]
    if not gts:
        return np.nan                              # class absent: left out of the mean
    if not P:
        return 0.0
    used, true, score = set(), [], []
    for q in gts:                                  # each GT takes the first unused prediction above t
        best = None
        for i, (s, p) in enumerate(P):
            if i in used or iou_sets(p, q) <= t:
                continue
            if best is None:
                best = s; used.add(i)
            else:                                  # a second one above t: the lower score is a FP
                true.append(0); score.append(min(best, s)); best = max(best, s)
        if best is not None:
            true.append(1); score.append(best)
    for s, p in P:                                 # unmatched predictions: FP unless mostly ignored
        if any(iou_sets(p, q) > t for q in gts):
            continue
        ignored = np.sum(void[list(p)]) + len(p & small)
        if ignored / len(p) <= t:
            true.append(0); score.append(s)
    order = np.argsort(score)[::-1]
    tp, s_sorted = np.cumsum(np.array(true)[order]), np.array(score)[order]
    last = np.r_[s_sorted[1:] != s_sorted[:-1], True]   # one point per distinct score
    prec = np.r_[1.0, tp[last] / (np.flatnonzero(last) + 1)]
    rec = np.r_[0.0, tp[last] / len(gts)]               # missed GTs stay in the denominator
    return np.sum((rec[1:] - rec[:-1]) * (prec[1:] + prec[:-1]) / 2)  # trapezoids, no envelope

ths = np.append(np.arange(0.5, 0.95, 0.05), 0.25)  # as in the official script
print("thresholds averaged:", np.round(ths[:-1], 2))
A = np.array([[ap_class(c, t) for t in ths] for c in INST_IDS])
for c, row in zip(INST_IDS, A):
    if not np.all(np.isnan(row)):
        print("%-8s AP %.3f  AP50 %.3f  AP25 %.3f" % (NAMES[c], row[:-1].mean(), row[0], row[-1]))
print("mean     AP %.3f  AP50 %.3f  AP25 %.3f" % (np.nanmean(A[:, :-1]), np.nanmean(A[:, 0]), np.nanmean(A[:, -1])))
A95 = np.array([[ap_class(c, 0.95)] for c in INST_IDS])     # the threshold the arange leaves out
print("AP over ten thresholds 0.50:0.95 would be %.3f" % np.nanmean(np.hstack([A[:, :-1], A95])))

points 23440 {0: 480, 'wall': 8520, 'floor': 5688, 'cabinet': 1900, 'bed': 2720, 'chair': 1720, 'table': 1800, 40: 612}
IoU {'wall': 0.873, 'floor': 0.947, 'cabinet': 1.0, 'bed': 0.899, 'chair': 0.815, 'sofa': 0.0, 'table': 1.0}
mean over classes present: 0.791; over all 20 with absent = 0: 0.277
pred bed    score 0.90  points 2348  best IoU 0.86  on void 0.00
pred chair  score 0.80  points  820  best IoU 1.00  on void 0.00
pred chair  score 0.60  points  480  best IoU 0.59  on void 0.00
pred chair  score 0.70  points  558  best IoU 0.30  on void 0.43
pred table  score 0.50  points 1100  best IoU 0.61  on void 0.00
pred table  score 0.95  points 1148  best IoU 0.00  on void 1.00
pred chair  score 0.85  points  116  best IoU 0.69  on void 0.31
pred sofa   score 0.40  points  318  best IoU 0.00  on void 0.00
thresholds averaged: [0.5  0.55 0.6  0.65 0.7  0.75 0.8  0.85 0.9 ]
cabinet  AP 0.000  AP50 0.000  AP25 0.000
bed      AP 0.889  AP50 1.000  AP25 1.000
chair    AP 0.500  AP50 0.500  AP25 1.000
table    AP 0.333  AP50 1.000  AP25 1.000
mean     AP 0.431  AP50 0.625  AP25 0.750
AP over ten thresholds 0.50:0.95 would be 0.400

The semantic scores show what the ignore rules forgive. The table keeps IoU 1.0 although every unannotated wall point was predicted as table, and the cabinet keeps 1.0 although the lamp was called cabinet: ground truth outside the 20 classes is never scored. The tops of the walls, called “other prop”, lower the wall IoU but add no false positive to any class. The floor called bed and the part of chair B called sofa do reach other classes, as false positives of bed and of sofa, which has no ground truth and scores 0. The two means differ more than any single error: the seven classes that occur in ground truth or prediction average 0.791, but a benchmark mean over all 20 classes would count the 13 absent ones. The official script returns NaN for them and stops, so a validation subset needs its own convention, stated alongside the result.

The instance scores show the matching rules. The most confident prediction, a “table” lying entirely on the floor, is ignored at every threshold because all of its vertices are void. The toy-chair prediction is ignored too, because it covers a ground-truth chair below 100 vertices and some wall. The sofa prediction has no sofa to match, and since the class has no ground truth it drops out of the mean instead of counting as an error. Chair B’s prediction (IoU 0.30) is a true positive only at $t = 0.25$, which is why chair reaches AP 25% of 1 but AP 50% of 0.5. The duplicate on chair A is a false positive at every threshold, but it scores below the true positives and so costs nothing under this uninterpolated area. The bed (IoU 0.86) passes eight of the nine averaged thresholds, giving 0.889. Had 0.95 been included, as the documentation states, the overall AP would be 0.400 instead of 0.431. The missed cabinet costs a full class.

The official 3D scripts, ported as described, gave the same per-class IoUs (difference 0.0) and the same AP, AP 50% and AP 25% for every class. The semantic script printed the seven IoUs and then stopped at door, the first class with a bare NaN, so we compared all values through its get_iou function. In our own check outside the code shown, on further random sets of instance predictions for the same room, our function matched the official evaluate_matches exactly whenever the official code ran; on the other sets it raised the IndexError described above.

License

ScanNet has no open data license. The data are released under the ScanNet Terms of Use, a one-page agreement with Princeton University and Stanford University (checked October 2026; the server dates the PDF to February 2017). To obtain the data, a researcher fills in the agreement with the requester’s and the principal investigator’s names, emails and affiliation, signs it, and sends it from an institutional address to the project’s mailing list (README). The terms:

• allow use only for non-commercial research and educational purposes;
• give no warranty;
• make the researcher responsible for their use, including for copies of copyrighted 3D models made from the database;
• allow sharing with research colleagues only if they also accept the terms;
• let the universities end access at any time;
• bind the researcher’s employer when it is a commercial company;
• are governed by the law of New Jersey.

The terms cover the database as a whole and do not license the scans and the annotations separately. All images were captured by the project, not collected from the web. The code in the GitHub repository is under the MIT License. ScanNet++ is distributed separately through its own website, which requires registration.

Limitations and Bias

• The reference geometry is a reconstruction. Meshes and poses come from BundleFusion on the same commodity RGB-D data that methods are evaluated on, not from an independent measurement. ScanNet therefore suits scene understanding better than accuracy benchmarks for 3D reconstruction or SLAM (our reasoning). Laser-scanned benchmarks such as Tanks and Temples and ETH3D were built for that, and ScanNet++ adds laser-scanned geometry to indoor scenes.
• Labels are crowdsourced and incomplete. v1 meshes were about 63% covered and v2 about 90% (changelog). Annotation follows a pre-computed over-segmentation, so boundaries can be no finer than the segments. The vote-based consensus can still contain errors.
• Free evaluation regions. The other 20 NYU40 classes and the unannotated surface are never scored. In the instance task wall and floor are void as well, so predictions on them are free (Evaluation).
• Who and where. The scans were captured over about a month by 20 users in several countries (Section 4), in spaces such as offices, apartments, bathrooms, classrooms and libraries. The paper does not say which countries or institutions were scanned; it gives the distribution of scene types only as a chart (Figure 5).
• People and privacy. The paper, project page, benchmark documentation and terms of use (checked October 2026) do not describe consent from the occupants of the scanned spaces or any filtering of people or personal items. ScanNet200’s category list includes “person” among its common classes. The ScanNet++ changelog, by contrast, mentions anonymization of its DSLR images and iPhone video.
• Benchmark mechanics. One test set of 100 scans, hidden labels and a two-week resubmission block limit overfitting to the test set. Validation-set numbers, such as VoteNet’s below, use the v2 validation set (312 scans in the official list) and are not comparable with test-set results.

Common Uses

• 3D semantic and instance segmentation. Training and evaluating point-based, sparse-voxel and transformer networks on the 20-class and ScanNet200 tasks, as the leaderboard entries show. Papers report either the hidden test set or the 312-scan validation set.
• 3D object detection. VoteNet (Qi et al., 2019) derived axis-aligned boxes from the instance labels for 18 classes, since ScanNet v2 has no oriented boxes, and reported mean AP at box IoU 0.25 and 0.5 on the validation set. The axis-aligned convention follows 3D-SIS by Hou, Dai and Nießner, as cited by Qi et al.
• RGB-D sequences. The raw frames with depth and BundleFusion poses are also used outside the benchmark, for example to train and evaluate indoor feature matching. Such uses rely on the released poses as reference, without an official protocol, and inherit the reconstruction’s errors (Limitations).
• Shape and scene retrieval. The paper’s own object classification and CAD retrieval tasks, and later CAD alignment through Scan2CAD.

Historical Milestones

• 2017. The paper’s geometry-only network for semantic voxel labelling reached 73.0% voxel accuracy over the paper’s 312 test scenes (Table 7). For object classification, training on ScanNet gave 74.9% on the ScanNet test set, against 39.5% when training on ShapeNet alone (Table 4): real scans were needed to classify real scans.
• 2018. v2 and the online benchmark were released on 11 June 2018.
• 2019. MinkowskiNet (Choy, Gwak & Savarese, 2019), a deep sparse 3D convolutional network that processes whole rooms, reported 67.9% mIoU on the 3D semantic label test set at 5 cm voxels. The paper reports a gain of about 19 points over the best work published by the CVPR deadline; the highest-scoring cited entry from before the deadline in Table 1 is 3DMV at 48.4%, which also uses images. A 2 cm version reached 72.1% after the deadline (CVPR version, text and Table 1; the arXiv version’s Table 1 lists 73.4% while its text keeps 72.1%). In the same year VoteNet reported 58.6 mAP@0.25 and 33.5 mAP@0.5 on the v2 validation set from geometry only, against 40.2 and 22.5 for 3D-SIS with five RGB views (Qi et al., 2019, Table 2).
• 2022. ScanNet200 added a 200-category setting to the same scans.
• 2023. ScanNet++ (Yeshwanth et al., 2023) introduced laser scans, DSLR images and iPhone RGB-D for 460 scenes in the paper; its website lists v1 (November 2023) with 380 released scenes and v2 (December 2024) with 1,006 (changelog, checked October 2026).

The current 3D semantic and instance leaderboards are on the benchmark website.

Related Datasets

• NYU Depth v2: the earlier Kinect dataset whose 40-class label set ScanNet evaluates and whose sequences ScanNet reconstructed and re-annotated in 3D.
• SUN RGB-D, SceneNN and PiGraphs: the comparison points in the paper’s Table 1. SUN RGB-D has 10,335 single frames with 2D polygons and 3D boxes. SceneNN (100 scans in Table 1, 94 in the arXiv appendix) and PiGraphs (26) were reconstructed and labelled in 3D by their authors.
• Stanford Large-Scale 3D Indoor Spaces (S3DIS; Armeni et al.; BuildingParser in the paper’s Table 1): 265 rooms captured with a Matterport camera and labelled by hand, released as fused point clouds without the raw RGB-D data.
• ScanNet++: higher-fidelity successor from the same group, with laser-scanned geometry and benchmarks for novel view synthesis and 3D semantics.
• ShapeNet: source of the CAD models for ScanNet’s alignment and retrieval tasks.
• Tanks and Temples and ETH3D: reconstruction benchmarks with laser-scanned ground truth, for evaluating geometry rather than semantics.

Resources

• ScanNet project page and changelog.
• ScanNet GitHub repository: capture app, reconstruction and segmentation tools, split lists, label map instructions and BenchmarkScripts. Data access requires the signed Terms of Use.
• ScanNet Benchmark: documentation, submission formats and current leaderboards for the ScanNet and ScanNet200 tasks.
• Scan2CAD benchmark for CAD model alignment.
• ScanNet++: dataset, documentation and benchmarks.