nuScenes

nuScenes (“nuTonomy scenes”) is a driving dataset recorded with the full sensor set of an autonomous vehicle: six cameras, five radars and one LiDAR, all covering 360°, together with GPS and an IMU. It was built by nuTonomy, now Motional. A teaser set came out in September 2018 and the full dataset of 1,000 scenes in March 2019 (Caesar et al., 2020, footnote 2; devkit changelog). The CVPR 2020 paper by Holger Caesar, Oscar Beijbom and colleagues describes the data and defines the benchmark’s detection and tracking tasks. The authors call it the first dataset to carry the full sensor suite of an autonomous vehicle. Each scene lasts 20 seconds and has 3D boxes for 23 object classes, annotated at 2 Hz.

The benchmark brought in two metrics that later papers report alongside or instead of plain average precision. The nuScenes detection score (NDS) combines mean AP with five errors of the matched boxes. AMOTA averages a recall-normalized tracking accuracy over recall levels. This article covers the dataset, its versions and terms, and both metrics as the official devkit computes them. 3D object detection explains the centre-distance matching and the AP rule, and this article links there rather than repeating them.

Purpose

When nuScenes appeared, image datasets for driving were plentiful, but few multimodal datasets carried the range sensors that vehicles use. KITTI, the earlier multimodal benchmark, labels objects only in the front camera’s view and has 22 scenes (Caesar et al., 2020, §1.2 and Table 1). The nuScenes authors set out to give detection and tracking methods a dataset with all three sensor types, annotated all around the vehicle. They also chose scenes for difficulty and diversity: dense traffic, rare classes, dangerous situations, night and rain (§2).

Dataset Size

The paper reports 84 logs with 15 hours of driving, 242 km in all, from Boston (Seaport and South Boston) and Singapore (One North, Holland Village and Queenstown). From these, the authors manually selected 1,000 scenes of 20 s each (§2). Two Renault Zoe cars with identical sensor layouts did the recording (§2, Table 2):

• six cameras at 12 Hz, 1600 × 900 pixels;
• a 32-beam spinning LiDAR at 20 Hz, with a range of up to 70 m;
• five 77 GHz radars at 13 Hz, with a range of up to 250 m.

Each camera exposure is triggered as the LiDAR sweeps across the centre of the camera’s field of view. Localization against a prebuilt LiDAR map reaches errors of 10 cm or less.

The counts on the dataset page and in the paper differ slightly:

Source	Camera images	LiDAR sweeps	Radar sweeps	Boxes	Keyframes
Dataset page (November 2026)	about 1.4M	390k	1.4M	1.4M	40k
Paper, Table 1	1.4M	400k	1.3M	1.4M	40k

Table 1 also gives the size as 5.5 hours, which is the duration of the 1,000 selected scenes, not the 15 hours of driving. Separately, the page’s annotation-statistics table counts 1,166,187 cuboids over the 23 categories. It does not say which splits that count covers, and it does not match the 1.4M figure above it.

The paper reports, per keyframe, 7 pedestrians and 20 vehicles on average. Its keyframes come 55% from Boston and 45% from the three Singapore areas; 19.4% are in rain and 11.6% at night (§2).

Classes

The annotation taxonomy has 23 categories, from adult pedestrians and cars to strollers, ambulances and debris. The dataset page lists their cuboid counts: 493,322 cars, 208,240 adult pedestrians and 152,087 barriers, against 49 ambulances and 787 animals. The paper reports a ratio of 1:10,000 between the rarest and the commonest class, against 1:36 in KITTI (§2).

For the detection challenge, the authors merged similar classes and removed classes with fewer than 10,000 annotations, which leaves 10 detection classes (supplement, Class mapping and Table 5). The devkit’s category_to_detection_name maps the categories to these classes:

• car;
• truck;
• bus (bendy and rigid);
• trailer;
• construction vehicle;
• pedestrian (adult, child, construction worker and police officer);
• motorcycle;
• bicycle;
• traffic cone;
• barrier.

Personal mobility devices, strollers, wheelchairs, animals, debris, pushable objects, bicycle racks and emergency vehicles are not evaluated.

The tracking task drops barrier, traffic cone and construction vehicle as mostly static, which leaves seven classes (tracking README; tracking_nips_2019.json).

There are eight attributes, and each detection class may carry only some of them (detection_name_to_rel_attributes):

• vehicles: moving, parked or stopped;
• pedestrians: moving, standing, or sitting or lying down;
• cycles: with or without a rider;
• cones and barriers: none.

The page counts 420,226 parked-vehicle attributes, 149,203 moving and 65,975 stopped.

Annotation Types

Annotators labelled every keyframe at 2 Hz with each object’s category, attributes (visibility, activity and pose) and a cuboid given as $x, y, z$, width, length, height and yaw. An object is annotated throughout a scene if at least one LiDAR or radar point falls on it (§2). The acknowledgements name Scale.ai as the annotation provider. The intermediate sensor frames between keyframes are released without labels.

Each box also carries a velocity, which the evaluation compares with the predicted one. The devkit stores sizes as (width, length, height) and orientations as quaternions (detection README, results format).

Later releases added further annotation layers to the same scenes:

• Maps. Human-annotated semantic maps came first; the map expansion of July 2019 added 11 vector layers such as lanes, crossings and stop lines.
• CAN bus. The expansion of February 2020 adds vehicle bus data.
• nuScenes-lidarseg. One of 32 semantic labels for every LiDAR point of a keyframe.
• Panoptic nuScenes (July 2021). Gives each of those points an instance and a track label as well.

Two passages of the dataset page differ on lidarseg. One says it was released in July 2020 with 1.4 billion points in 40,000 point clouds and 1,000 scenes. The download section says August 2020, with 1.1 billion points in 34,000 point clouds from the 850 annotated train and val scenes. A third passage, on Panoptic nuScenes, says the semantic labels were first released as nuScenes-lidarseg in October 2019. The devkit changelog dates the lidarseg code to 31 August 2020.

Privacy: according to the dataset page, faces and licence plates are detected with object detectors run for high recall, and the detections are then blurred in the camera images.

Splits

The devkit’s splits.py hard-codes the scene lists. Counted at commit b40adc4, they hold:

• train: 700 scenes, made of two disjoint halves of 350 (train_detect and train_track), so that a detector and a tracker can be trained on separate data;
• val: 150 scenes;
• test: 150 scenes;
• mini: 10 scenes, 8 for training and 2 for validation, for exploring the data.

Together, train, val and test hold 1,000 distinct scene names. The dataset page gives the same 700/150/150 split and describes mini as a subset of trainval. In the hard-coded lists, two of the eight mini_train scenes belong to val.

Annotations for the test scenes are not released. The supplement states that train, val and test follow the same distribution of country, rain and night (supplement, §C).

Tasks

• 3D detection: boxes, velocities and attributes for the 10 classes. A method may use only about 0.5 s of past sensor data: at most 6 past camera images, 6 past radar sweeps and 10 past LiDAR sweeps (detection README). Results are split into LiDAR, vision and open tracks by input modality.
• Tracking: online 3D multi-object tracking of the seven tracking classes, using past and current data from the scene only (tracking README).
• Prediction: future trajectories of agents. Motion forecasting describes this set-up.
• LiDAR semantic and panoptic segmentation: on lidarseg and Panoptic nuScenes.

Derived benchmarks built on the scenes, such as the occupancy grids in occupancy prediction, define their own labels and metrics.

Evaluation

The official evaluation is the devkit’s Python code. Test-set results come only from the evaluation server, which runs on Hugging Face as of November 2026. Each user or team may hold at most one account and submit at most three results per year (detection README).

Detection: preprocessing, matching and AP

Before scoring, the devkit removes:

• every ground-truth and predicted box beyond its class range: 50 m for the vehicle classes, 40 m for pedestrians, motorcycles and bicycles, and 30 m for cones and barriers (detection_cvpr_2019.json);
• bicycles and motorcycles inside bicycle racks;
• ground-truth boxes that contain no LiDAR or radar point.

A results file with more than 500 predicted boxes in any keyframe is rejected (load_prediction).

Predictions are matched to ground truth by 2D centre distance on the ground plane, greedily in order of score. AP is computed at thresholds of 0.5, 1, 2 and 4 m and averaged over thresholds and classes. 3D object detection describes the matching, the authors’ reasons for it and calc_ap.

One point bears on the comparison of code and paper. The paper calls AP the “normalized area” for recall and precision above 10% and says that operating points below 10% are removed (§3.1). The code does not remove points with precision below 0.1. It subtracts 0.1 from every precision value, clips at zero and divides by 0.9, so a precision of 0.5 counts as about 0.44.

True-positive errors

For every class, five errors are measured on the matches at a 2 m threshold (accumulate, calc_tp, evaluate.py):

• ATE: centre distance on the ground plane, in metres;
• ASE: one minus the IoU of the two boxes after aligning centre and orientation. scale_iou computes it from the sizes alone, as the product of the per-axis minima over the union of the volumes;
• AOE: the smallest yaw difference in radians, on a period of $2\pi$, except barriers on a period of $\pi$;
• AVE: the L2 norm of the 2D velocity difference, in m/s;
• AAE: one minus attribute accuracy.

The class exemptions are written into evaluate.py. Traffic cones get no AOE, AVE or AAE, and barriers no AVE or AAE; both are set to NaN. A ground-truth box with no attribute also gives NaN for AAE, which affects about 0.4% of annotations (detection README).

The errors are not plain averages over the matches. accumulate walks the predictions in score order, records the error of each match, and takes the cumulative mean, so the value at match $k$ is the mean error of the $k$ most confident matches. It then interpolates these cumulative means onto 101 recall points through the confidence at each recall point. calc_tp averages them from recall index 11 (recall 0.11) up to the last index whose interpolated confidence is non-zero, which is the highest recall reached. If that index is below 11, so that 10% recall is not exceeded, all five errors of the class are set to 1. The paper describes the same rule as the average of the cumulative mean at each achieved recall level above 10% (§3.1).

The code comments describe that last index two ways. calc_tp calls it the “First instance of confidence = 0”, while the max_recall_ind property it calls computes the last non-zero confidence. The behaviour follows the property.

The nuScenes detection score

DetectionMetrics first averages each error over the classes with nanmean, so exempt classes drop out of the denominator: mATE and mASE average 10 classes, mAOE 9, and mAVE and mAAE 8. It then turns each mean error into a score $\max(0, 1 - \text{mTP})$. The clip is applied to the class mean, not to each class. The score is

$$\text{NDS} = \frac{1}{10}\Big[5\,\text{mAP} + \sum_{\text{mTP} \in \mathbb{TP}} \big(1 - \min(1, \text{mTP})\big)\Big],$$

which the code computes as $(5\,\text{mAP} + \sum \text{scores}) / (5 + 5)$ with mean_ap_weight = 5. Half of NDS is detection and half is the quality of the matched boxes (§3.1, Eq. 3). Because mATE, mAOE and mAVE can exceed 1 (§3.1), a mean error of 1 or more scores 0. A large error and an enormous one therefore cost the same.

Tracking: AMOTA and AMOTP

The tracking evaluation matches by centre distance below 2 m and computes the CLEAR MOT counts with the motmetrics library. As the nuScenes paper reports, the averaged metrics come from Weng and Kitani’s 3D tracking benchmark on KITTI. The devkit computes them as follows (algo.py, metrics.py and evaluate.py of the tracking code):

1. Thresholds. For each class it finds 40 score thresholds that would give recalls of 0.1, 0.123, …, 1, interpolating the scores of the matches (compute_thresholds, num_thresholds = 40, min_recall = 0.1).
2. MOTAR. At each threshold it computes the recall-normalized accuracy $$\text{MOTAR} = \max\Big(0,\ 1 - \frac{\text{IDS} + \text{FP} + \text{FN} - (1 - r)P}{rP}\Big),$$ where $P$ is the number of ground-truth boxes of the class and $r$ is the measured recall TP/$P$. That measured recall differs from the target recall of the threshold, as the motar docstring notes. At 10% recall, plain MOTA counts at least 90% of the boxes as misses; the $(1 - r)P$ term removes those misses (tracking README).
3. Averaging. AMOTA is the mean of MOTAR over the 40 thresholds, and AMOTP the mean of MOTP, the average distance of the matches.
4. Unreached recall. A recall level the tracker never reaches counts as the worst value: 0 for AMOTA and 2 m for AMOTP (metric_worst).
5. Classes. Both are averaged over the seven classes.

The secondary metrics (MOTA, MOTP, IDS, fragmentations and others) are taken at the single threshold with the highest MOTA. Two duration measures were added: the time until a track is first detected (TID) and the longest gap (LGD).

The definitions in the paper, the README and the code differ in places:

• Names. The paper names the main metric sAMOTA and its per-threshold term sMOTA$_r$ (§3.2). The README and code use AMOTA and MOTAR.
• Recall points. The README’s equation sums over $r \in \{\frac{1}{n-1}, \dots, 1\}$ and says in words that recall below 0.1 is left out. The code spaces its 40 points evenly from 0.1 to 1.
• Threshold for the secondary metrics. The paper picks the threshold with the highest sMOTA$_r$ (§3.2). The README and evaluate.py pick the one with the highest MOTA.
• Untracked objects in TID and LGD. The paper and the README assign the whole track duration to an object that is never tracked. At b40adc4, track_initialization_duration and longest_gap_duration leave such objects out of both averages.
• Weight factor. The motar function has a weight alpha, which its docstring says was 0.2 before. The changelog dates the removal of the weighting to 11 December 2019 (devkit v1.0.5), and the tracking README to 10 December 2019. Tracking numbers computed before and after that change are not comparable.

Example: what NDS rewards

The script below scores five synthetic detectors with the detection score, to show which defects NDS punishes and by how much. The scoring code is a line-by-line port of the devkit at commit b40adc4: accumulate, calc_ap, calc_tp, cummean, angle_diff, scale_iou, attr_acc, velocity_l2, the class exemptions of evaluate.py and the class means and NDS of DetectionMetrics. The devkit itself was not run, because it needs pyquaternion, which is not installed here. The port therefore takes yaw directly instead of through quaternions. A short helper converts quaternions to yaw only where the devkit’s own tests pass quaternions. The range filter is not ported, because every box is built inside its class range, and the script prints that check.

Before anything else, the script checks the port against the devkit’s own unit tests in test_utils.py and test_algo.py, each within that test’s tolerance. It ports the tests that check computed values; test_attr_acc, test_calc_ap, test_calc_tp and the cases that expect an assertion error are not ported. That includes test_nd_score, which runs the whole pipeline on random boxes from fixed seeds and expects an NDS of 0.08606662159639042.

Synthetic scenes. The scenes are synthetic, not nuScenes data:

• 40 keyframes, each with two boxes of every detection class, so 80 per class;
• the two boxes of a class lie on opposite sides of the vehicle, at least 10 m apart, so a 2 m match cannot take the wrong one;
• coordinates are metres in a vehicle frame, the yaw is measured about the vertical axis from the $x$ axis, sizes are (width, length, height), and the distance from the origin stands in for the devkit’s distance to the ego vehicle;
• attributes are drawn from each class’s allowed set; moving objects get a speed along their heading, and the others get zero velocity;
• every object has one score, drawn uniformly from 0.05 to 1 and shared by all detectors.

Detectors. Each is an error model built from the ground truth, with one prediction per object:

• (A) copies the boxes exactly;
• (B) sets every velocity to zero;
• (C) turns every heading by 180°;
• (D) shifts every centre by Gaussian noise whose standard deviation falls with the score, from 1 m at the lowest score to 0.05 m at score 1;
• (E) detects only 4 of the 80 bicycles.

Predictions fixed before the run. Each was derived from the code. They were written into the plan after a planning script had computed values for them from the construction and the formulas, without the ported pipeline, so the example illustrates them rather than testing them blind:

• A scores NDS 1 up to rounding;
• B and C each lose exactly 0.1, since the velocity and orientation mean errors exceed 1 and clip to a score of 0;
• C gives barriers an AOE of 0 (period $\pi$) and mAOE $= 8\pi/9$ over the nine classes that count;
• under D, the devkit’s ATE is below the plain mean of the matched errors in every class, and the other four errors stay 0;
• under E, the bicycle class gets AP 0 and all five errors set to 1, and the other nine classes are unchanged.

The plain mean for D. That prediction was derived from the construction. In planning it was checked with each sampled error replaced by its expected size, every object matched, no 2 m cut-off, and the paper’s average of cumulative means without the devkit’s interpolation through confidence. That is a different set-up from the example’s sampled noise, which is what the run tests.

Changes before the first run. A pilot run of the same script preceded the first run. Two print-only changes were made after it with the editor’s approval, and neither decides any check: a NumPy warning about an empty mean is silenced in the unit-test block, and the bicycle AP under E is printed as a plain number. After the first run, the licence notice in the header comment was extended with the licence file at the pinned commit; no code changed.

Running it. The script uses NumPy 2.3.5 and runs in under a second. Two runs printed identical output.

"""What the nuScenes detection score rewards, on synthetic boxes.

The scoring functions below are a line-by-line port of the nuScenes devkit
(nutonomy/nuscenes-devkit, commit b40adc4, python-sdk/nuscenes/eval/detection/
algo.py, data_classes.py, evaluate.py and eval/common/utils.py), with yaw taken
directly instead of through pyquaternion. Original code Copyright 2021 Motional,
licensed under the Apache License, Version 2.0
(https://github.com/nutonomy/nuscenes-devkit/blob/b40adc467b919192899405d9b77871afee8efa07/LICENSE.txt,
http://www.apache.org/licenses/LICENSE-2.0); modified (ported to plain dicts).
NumPy 2.3.5, one BLAS thread. No dataset file is used.
"""
import os
os.environ["OMP_NUM_THREADS"] = "1"; os.environ["OPENBLAS_NUM_THREADS"] = "1"
import random
import warnings
import numpy as np

# ---------------- devkit port (b40adc4) ----------------
CFG = dict(dist_ths=[0.5, 1.0, 2.0, 4.0], dist_th_tp=2.0, min_recall=0.1, min_precision=0.1,
           mean_ap_weight=5)
TP_METRICS = ["trans_err", "scale_err", "orient_err", "vel_err", "attr_err"]
NELEM = 101

def center_distance(g, p):
    return np.linalg.norm(np.array(p["xy"][:2]) - np.array(g["xy"][:2]))

def velocity_l2(g, p):
    return np.linalg.norm(np.array(p["vel"]) - np.array(g["vel"]))

def angle_diff(x, y, period):
    diff = (x - y + period / 2) % period - period / 2
    if diff > np.pi:
        diff = diff - (2 * np.pi)
    return diff

def yaw_diff(g, p, period=2 * np.pi):
    return abs(angle_diff(g["yaw"], p["yaw"], period))

def attr_acc(g, p):
    return np.nan if g["attr"] == "" else float(g["attr"] == p["attr"])

def scale_iou(g, p):
    sa, sr = np.array(g["size"]), np.array(p["size"])
    assert all(sa > 0) and all(sr > 0)
    inter = np.prod(np.minimum(sa, sr))
    return inter / (np.prod(sa) + np.prod(sr) - inter)

def cummean(x):
    if sum(np.isnan(x)) == len(x):
        return np.ones(len(x))
    sum_vals = np.nancumsum(x.astype(float))
    count_vals = np.cumsum(~np.isnan(x))
    return np.divide(sum_vals, count_vals, out=np.zeros_like(sum_vals), where=count_vals != 0)

def no_predictions():
    md = dict(recall=np.linspace(0, 1, NELEM), precision=np.zeros(NELEM), confidence=np.zeros(NELEM))
    md.update({k: np.ones(NELEM) for k in TP_METRICS})
    return md

def accumulate(gt, pred, cls, dist_th, plain=None):
    """gt, pred: dict sample -> list of boxes (insertion order = devkit order).
    If `plain` is a dict, the raw match errors are stored in it (not a devkit output)."""
    npos = len([1 for s in gt for b in gt[s] if b["cls"] == cls])
    if npos == 0:
        return no_predictions()
    pred_list = [b for s in pred for b in pred[s] if b["cls"] == cls]
    confs = [b["score"] for b in pred_list]
    sortind = [i for (v, i) in sorted((v, i) for (i, v) in enumerate(confs))][::-1]
    tp, fp, conf = [], [], []
    md = {k: [] for k in TP_METRICS + ["conf"]}
    taken = set()
    for ind in sortind:
        p = pred_list[ind]
        min_dist, match = np.inf, None
        for gi, g in enumerate(gt.get(p["sample"], [])):
            if g["cls"] == cls and (p["sample"], gi) not in taken:
                d = center_distance(g, p)
                if d < min_dist:
                    min_dist, match = d, gi
        if min_dist < dist_th:
            taken.add((p["sample"], match))
            tp.append(1); fp.append(0); conf.append(p["score"])
            g = gt[p["sample"]][match]
            md["trans_err"].append(center_distance(g, p))
            md["vel_err"].append(velocity_l2(g, p))
            md["scale_err"].append(1 - scale_iou(g, p))
            period = np.pi if cls == "barrier" else 2 * np.pi
            md["orient_err"].append(yaw_diff(g, p, period=period))
            md["attr_err"].append(1 - attr_acc(g, p))
            md["conf"].append(p["score"])
        else:
            tp.append(0); fp.append(1); conf.append(p["score"])
    if plain is not None:
        plain.update({k: np.array(md[k], dtype=float) for k in TP_METRICS})
    if len(md["trans_err"]) == 0:
        return no_predictions()
    tp = np.cumsum(tp).astype(float); fp = np.cumsum(fp).astype(float); conf = np.array(conf)
    prec = tp / (fp + tp); rec = tp / float(npos)
    rec_interp = np.linspace(0, 1, NELEM)
    prec = np.interp(rec_interp, rec, prec, right=0)
    conf = np.interp(rec_interp, rec, conf, right=0)
    out = dict(recall=rec_interp, precision=prec, confidence=conf)
    for k in TP_METRICS:
        tmp = cummean(np.array(md[k]))
        out[k] = np.interp(conf[::-1], md["conf"][::-1], tmp[::-1])[::-1]
    return out

def max_recall_ind(md):
    nz = np.nonzero(md["confidence"])[0]
    return 0 if len(nz) == 0 else nz[-1]

def calc_ap(md, min_recall, min_precision):
    prec = np.copy(md["precision"])[round(100 * min_recall) + 1:]
    prec -= min_precision
    prec[prec < 0] = 0
    return float(np.mean(prec)) / (1.0 - min_precision)

def calc_tp(md, min_recall, metric):
    first, last = round(100 * min_recall) + 1, max_recall_ind(md)
    if last < first:
        return 1.0
    return float(np.mean(md[metric][first:last + 1]))

def evaluate(gt, pred, classes, dist_for_all=None):
    """DetectionEval.evaluate + DetectionMetrics. Returns label APs, TP errors,
    mAP, mTP, TP scores, NDS and the raw 2 m match errors (for plain means)."""
    aps, tps, raw = {}, {}, {}
    for c in classes:
        mds = {}
        for d in CFG["dist_ths"]:
            r = {} if d == CFG["dist_th_tp"] else None
            mds[d] = accumulate(gt, pred, c, d if dist_for_all is None else dist_for_all, plain=r)
            if r is not None:
                raw[c] = r
        aps[c] = {d: calc_ap(mds[d], CFG["min_recall"], CFG["min_precision"]) for d in CFG["dist_ths"]}
        tps[c] = {}
        for m in TP_METRICS:
            if c == "traffic_cone" and m in ["attr_err", "vel_err", "orient_err"]:
                tps[c][m] = np.nan
            elif c == "barrier" and m in ["attr_err", "vel_err"]:
                tps[c][m] = np.nan
            else:
                tps[c][m] = calc_tp(mds[CFG["dist_th_tp"]], CFG["min_recall"], m)
    mean_dist_aps = {c: np.mean(list(aps[c].values())) for c in classes}
    mean_ap = float(np.mean(list(mean_dist_aps.values())))
    mtp = {m: float(np.nanmean([tps[c][m] for c in classes])) for m in TP_METRICS}
    scores = {m: max(0.0, 1.0 - mtp[m]) for m in TP_METRICS}
    nds = float(CFG["mean_ap_weight"] * mean_ap + np.sum(list(scores.values())))
    nds = nds / float(CFG["mean_ap_weight"] + len(scores))
    return dict(aps=aps, mean_dist_aps=mean_dist_aps, mAP=mean_ap, tps=tps, mTP=mtp,
                scores=scores, NDS=nds, raw=raw)
# ---------------- end of port ----------------

# Quaternion helpers, used only to run the devkit's own tests that pass quaternions.
def quat_axis_angle(axis, angle):
    axis = np.array(axis, float) / np.linalg.norm(axis)
    return (np.cos(angle / 2), *(np.sin(angle / 2) * axis))

def quaternion_yaw(q):
    """Yaw of the rotated x axis, from the unit-normalized quaternion (w, x, y, z)."""
    q = np.array(q, float); n = np.linalg.norm(q)
    w, x, y, z = q / n if n > 0 else q
    r00, r10 = 1 - 2 * (y * y + z * z), 2 * (x * y + w * z)
    return np.arctan2(r10, r00)

# ---------------- C1, C2: the devkit's own unit tests (b40adc4) ----------------
def box(**kw):
    b = dict(sample="", cls="car", xy=(0, 0, 0), size=(1, 1, 1), yaw=0.0, vel=(0, 0),
             attr="vehicle.parked", score=-1.0)
    b.update(kw); return b

def check_group(name, devs, tol):
    devs = np.array(devs, float)
    print(f"  {name:<34s} {int(np.sum(devs < tol)):3d}/{len(devs):<3d} passed, max deviation {devs.max():.1e}"
          f" (tolerance {tol:g})")
    return bool(np.all(devs < tol))

print("C1: devkit unit tests reproduced by the port")
ok = []
ok.append(check_group("scale_iou (exact cases)", [abs(scale_iou(box(size=a), box(size=b)) - t) for a, b, t in
          [((4, 4, 4), (4, 4, 4), 1), ((2, 2, 2), (1, 1, 1), 1 / 8), ((1, 1, 1), (2, 2, 2), 1 / 8)]], 1e-15))
ok.append(check_group("scale_iou (arbitrary, 7 places)",
          [abs(scale_iou(box(size=(0.96, 0.37, 0.69)), box(size=(0.32, 0.01, 0.39))) - 0.00509204)], 5e-8))
rad = lambda x: x / 180 * np.pi
ok.append(check_group("angle_diff", [abs(abs(angle_diff(rad(a), rad(b), rad(p))) - rad(t)) for a, b, p, t in
          [(90, 0, 360, 90), (90, 0, 180, 90), (90, 0, 90, 0), (0, 90, 90, 0), (0, 180, 180, 0),
           (0, 180, 360, 180), (0, 180 + 360 * 200, 360, 180)]], 5e-8))
ok.append(check_group("center_distance", [abs(center_distance(box(xy=a), box(xy=b)) - t) for a, b, t in
          [((4, 4, 5), (4, 4, 5), 0), ((0, 0, 0), (0, 0, 0), 0), ((4, 4, 4), (3, 3, 3), np.sqrt(2)),
           ((-1, -1, -1), (1, 1, 1), np.sqrt(8)), ((4.2, 2.8, 4.2), (-1.45, 3.5, 3.9),
                                                   np.sqrt((-1.45 - 4.2) ** 2 + (3.5 - 2.8) ** 2))]], 5e-8))
ok.append(check_group("velocity_l2", [abs(velocity_l2(box(vel=a), box(vel=b)) - t) for a, b, t in
          [((4, 4), (4, 4), 0), ((-1, -1), (1, 1), np.sqrt(8)),
           ((8.2, 1.4), (6.4, -9.4), np.sqrt((6.4 - 8.2) ** 2 + (-9.4 - 1.4) ** 2))]], 5e-8))
nan = np.nan
cm_cases = [((nan, 5), (0, 5)), ((5, 2, nan), (5, 3.5, 3.5)), ((nan, 4.5, nan), (0, 4.5, 4.5)),
            ((nan,) * 4, (1, 1, 1, 1)), ((nan,), (1,)), ((4,), (4.0,)),
            ((nan, 3.58, 2.14, nan, 9, 1.48, nan), (0, 3.58, 2.86, 2.86, 4.906666, 4.05, 4.05))]
ok.append(check_group("cummean (decimal 6)", [np.max(np.abs(cummean(np.array(x, float)) - np.array(t)))
                                               for x, t in cm_cases], 1.5e-6))
yd = []
qa = quat_axis_angle((0, 0, 1), np.pi / 8)
yd.append(abs(abs(angle_diff(quaternion_yaw(qa), quaternion_yaw(qa), 2 * np.pi)) - 0))
yd.append(abs(abs(angle_diff(quaternion_yaw(qa), quaternion_yaw(quat_axis_angle((0, 1, 0), np.pi / 8)),
                             2 * np.pi)) - np.pi / 8))
for yaw_in in np.linspace(-10, 10, 100):
    yt = yaw_in % (2 * np.pi)
    yt = 2 * np.pi - yt if yt > np.pi else yt
    yd.append(abs(abs(angle_diff(quaternion_yaw(quat_axis_angle((0, 0, 1), 0)),
                                 quaternion_yaw(quat_axis_angle((0, 0, 1), yaw_in)), 2 * np.pi)) - yt))
yd.append(abs(abs(angle_diff(quaternion_yaw(quat_axis_angle((0, 0, 1), 1.1 * np.pi)),
                             quaternion_yaw(quat_axis_angle((0, 0, 1), 0.9 * np.pi)), 2 * np.pi)) - 0.2 * np.pi))
ok.append(check_group("yaw_diff (via quaternion helper)", yd, 5e-8))

def md_for(gts, preds, cls, dist_th):
    G = {s: [box(sample=s, **b) for b in L] for s, L in gts.items()}
    P = {s: [box(sample=s, **b) for b in L] for s, L in preds.items()}
    return accumulate(G, P, cls, dist_th)

car1 = dict(xy=(1, 1, 1), cls="car", score=1.0); car2 = dict(xy=(3, 3, 1), cls="car", score=0.7)
bic1 = dict(xy=(5, 5, 1), cls="bicycle", score=1.0)
ap = lambda g, p: calc_ap(md_for(g, p, "car", 2.0), 0.1, 0.1)
E = {"sample1": []}
ap_cases = [(E, {"sample1": [car1]}, 0.0), ({"sample1": [car1]}, E, 0.0), (E, E, 0.0),
            ({"sample1": [car1]}, {"sample1": [car1]}, 1.0),
            ({"sample1": [car1, car2]}, {"sample1": [car1]}, 0.4 / 0.9),
            ({"sample1": [car1]}, {"sample1": [car1, car2]}, 1.0),
            ({"sample1": [car2]}, {"sample1": [car1, car2]}, ((0.8 * 0.4) / 2) / (0.9 * 0.9)),
            ({"sample1": [car1]}, {"sample1": [car1, bic1]}, 1.0),
            ({"sample1": [car1], "sample2": [car2]}, {"sample1": [car1], "sample2": [car2]}, 1.0),
            ({"sample1": [car1], "sample2": []}, {"sample1": [car1], "sample2": []}, 1.0),
            ({"sample1": [car1], "sample2": [car2]}, {"sample1": [car1], "sample2": []}, 0.4 / 0.9)]
ok.append(check_group("TestAPSimple (11 cases, 0.01)", [abs(ap(g, p) - t) for g, p, t in ap_cases], 0.01 + 1e-12))

tp = lambda g, p, m: calc_tp(md_for(g, p, "car", 2.0), 0.1, m)
tpd = []
c1 = dict(xy=(1, 1, 1), score=1.0); c2 = dict(xy=(3, 3, 1), score=1.0)
b1 = dict(xy=(1, 1, 1), score=1.0, cls="bicycle", attr="cycle.with_rider")
for m in TP_METRICS:
    tpd.append(abs(tp({"sample1": [c1]}, {"sample1": [c2]}, m) - 1.0))
for m in TP_METRICS:
    tpd.append(abs(tp({"sample1": [c1]}, {"sample1": [b1]}, m) - 1.0))
c1b = dict(xy=(1, 1, 1), score=0.3)
for m in TP_METRICS:
    tpd.append(abs(tp({"sample1": [c1]}, {"sample1": [c1]}, m) - 0.0))
    tpd.append(abs(tp({"sample1": [c1]}, {"sample1": [c1b]}, m) - 0.0))
g1 = dict(xy=(1, 1, 1)); g2 = dict(xy=(10, 10, 1), size=(2, 2, 2)); g3 = dict(xy=(20, 20, 1), size=(2, 4, 2))
p1 = dict(xy=(1, 1, 1), score=1.0); p2 = dict(xy=(11, 10, 1), size=(2, 2, 2), score=0.9)
p3 = dict(xy=(100, 10, 1), size=(2, 2, 2), score=0.8); p4 = dict(xy=(20, 20, 1), size=(2, 4, 2), score=0.7)
p5 = dict(xy=(21, 20, 1), size=(2, 4, 2), score=0.7); p4b = dict(xy=(21, 20, 1), size=(2, 4, 2), score=0.7)
t2 = ((0 + 0) / 2 + (0 + 0.5) / 2) / (2 * 0.9)
t3 = ((0 + 0) / 2 + (0 + 0.5) / 2 + (0.5 + 0.33) / 2) / (3 * 0.9)
for g, p, t in [({"sample1": [g2]}, {"sample1": [p2]}, 1.0),
                ({"sample1": [g1, g2]}, {"sample1": [p1, p2]}, t2),
                ({"sample1": [g1, g2]}, {"sample1": [p1, p2, p3]}, t2),
                ({"sample1": [g1, g2, g3]}, {"sample1": [p1, p2, p4]}, t3),
                ({"sample1": [g2, g3]}, {"sample1": [p2, p5]}, 1.0),
                ({"sample1": [g2]}, {"sample1": [p2]}, 1.0),
                ({"sample1": [g1], "sample2": [g2]}, {"sample1": [p1], "sample2": [p2]}, t2),
                ({"sample1": [g1], "sample2": [g2], "sample3": []},
                 {"sample1": [p1], "sample2": [p2, p3], "sample3": []}, t2),
                ({"sample1": [g2, g3], "sample2": [g3]}, {"sample1": [p2], "sample2": [p4b]}, 1.0)]:
    tpd.append(abs(tp(g, p, "trans_err") - t))
ok.append(check_group("TestTPSimple (29 cases, 0.01)", tpd, 0.01 + 1e-12))

# C2: test_nd_score, with the devkit test's seeds and its _mock_results
REL = {"pedestrian": ["pedestrian.moving", "pedestrian.sitting_lying_down", "pedestrian.standing"],
       "bicycle": ["cycle.with_rider", "cycle.without_rider"], "motorcycle": ["cycle.with_rider", "cycle.without_rider"],
       "barrier": [], "traffic_cone": []}
for c in ["car", "bus", "construction_vehicle", "trailer", "truck"]:
    REL[c] = ["vehicle.moving", "vehicle.parked", "vehicle.stopped"]
CFG_CLASSES = ["car", "truck", "bus", "trailer", "construction_vehicle", "pedestrian", "motorcycle",
               "bicycle", "traffic_cone", "barrier"]

def mock_results(nsamples, ngt, npred, name):
    def random_attr():
        rel = REL[name]
        return "" if len(rel) == 0 else rel[np.random.randint(0, len(rel))]
    def one(s, scale):
        xy = tuple(np.random.rand(2) * scale)
        size = tuple(np.random.rand(3) * 4); rot = tuple(np.random.rand(4)); vel = tuple(np.random.rand(3)[:2] * 4)
        score = random.random(); attr = random_attr(); _ego = random.random()
        return dict(sample=str(s), cls=name, xy=(xy[0], xy[1], 0.0), size=size, yaw=quaternion_yaw(rot),
                    vel=vel, attr=attr, score=score)
    gt = {str(s): [one(s, 15) for _ in range(ngt)] for s in range(nsamples)}
    pred = {str(s): [one(s, 10) for _ in range(npred)] for s in range(nsamples)}
    return gt, pred

random.seed(42); np.random.seed(42)
aps2, tps2 = {}, {}
with warnings.catch_warnings():          # nanmean over an all-exempt single class warns
    warnings.simplefilter("ignore", RuntimeWarning)
    for c in CFG_CLASSES:
        g, p = mock_results(30, 3, 25, c)
        r = evaluate(g, p, [c], dist_for_all=2)
        aps2[c], tps2[c] = r["aps"][c], r["tps"][c]
mAP2 = float(np.mean([np.mean(list(aps2[c].values())) for c in CFG_CLASSES]))
mtp2 = {m: float(np.nanmean([tps2[c][m] for c in CFG_CLASSES])) for m in TP_METRICS}
nds2 = (5 * mAP2 + sum(max(0.0, 1 - v) for v in mtp2.values())) / 10
print(f"C2: test_nd_score: port {nds2!r}, devkit target 0.08606662159639042, "
      f"|difference| {abs(nds2 - 0.08606662159639042):.1e} (tolerance 1e-12)")
print("C1 all groups passed:", all(ok), "| C2 passed:", abs(nds2 - 0.08606662159639042) < 1e-12)

# ---------------- data generator (verbatim from reach.py) ----------------
CLASS_RANGE = {"car": 50, "truck": 50, "bus": 50, "trailer": 50, "construction_vehicle": 50,
               "pedestrian": 40, "motorcycle": 40, "bicycle": 40, "traffic_cone": 30, "barrier": 30}
CLASSES = list(CLASS_RANGE)
SIZE_WLH = {"car": (1.9, 4.6, 1.7), "truck": (2.5, 7.0, 3.0), "bus": (2.9, 11.0, 3.4),
            "trailer": (2.9, 12.0, 3.9), "construction_vehicle": (2.8, 6.5, 3.2),
            "pedestrian": (0.7, 0.7, 1.8), "motorcycle": (0.8, 2.1, 1.5), "bicycle": (0.6, 1.7, 1.3),
            "traffic_cone": (0.4, 0.4, 1.0), "barrier": (2.5, 0.5, 1.0)}
ATTRS = {"pedestrian": ["pedestrian.moving", "pedestrian.sitting_lying_down", "pedestrian.standing"],
         "motorcycle": ["cycle.with_rider", "cycle.without_rider"],
         "bicycle": ["cycle.with_rider", "cycle.without_rider"],
         "traffic_cone": [], "barrier": []}
for c in ["car", "truck", "bus", "trailer", "construction_vehicle"]:
    ATTRS[c] = ["vehicle.moving", "vehicle.parked", "vehicle.stopped"]
MOVING = {"vehicle.moving": (3.0, 12.0), "pedestrian.moving": (0.8, 1.8), "cycle.with_rider": (2.0, 6.0)}
N_SAMPLES, PER_CLASS = 40, 2          # 40 keyframes, 2 boxes of every class in each

def make_gt(rng):
    """Ground-truth boxes: dicts with sample, class, xy (ego frame, m), size (w,l,h),
    yaw (rad), velocity (vx,vy m/s), attribute ('' for cones and barriers), score
    (the score every derived detector gives this object, U(0.05, 1))."""
    gt = []
    for s in range(N_SAMPLES):
        for c in CLASSES:
            theta = rng.uniform(-np.pi, np.pi)
            for k in range(PER_CLASS):                  # the two boxes of a class face each other
                r = rng.uniform(5.0, CLASS_RANGE[c] - 5.0)
                ang = theta + k * np.pi
                yaw = rng.uniform(-np.pi, np.pi)
                size = tuple(float(v) for v in np.array(SIZE_WLH[c]) * rng.uniform(0.9, 1.1, 3))
                attr = ATTRS[c][rng.integers(len(ATTRS[c]))] if ATTRS[c] else ""
                speed = rng.uniform(*MOVING[attr]) if attr in MOVING else 0.0
                gt.append(dict(sample=f"s{s:02d}", cls=c, xy=np.array([r * np.cos(ang), r * np.sin(ang)]),
                               size=size, yaw=float(yaw),
                               vel=np.array([speed * np.cos(yaw), speed * np.sin(yaw)]),
                               attr=attr, score=float(rng.uniform(0.05, 1.0))))
    return gt
# ---------------------------------------------------------------------------------

gt_list = make_gt(np.random.default_rng(0))
by_sample = lambda boxes: {s: [b for b in boxes if b["sample"] == s] for s in dict.fromkeys(b["sample"] for b in gt_list)}
GT = by_sample(gt_list)
wrap = lambda a: (a + np.pi) % (2 * np.pi) - np.pi
noise_rng = np.random.default_rng(1)
D_noise = [noise_rng.normal(0.0, 0.05 + (1.0 - g["score"]), 2) for g in gt_list]
bic_keep = [i for i, g in enumerate(gt_list) if g["cls"] == "bicycle"][:4]
DET = {
    "A exact": [dict(g) for g in gt_list],
    "B zero velocity": [dict(g, vel=np.zeros(2)) for g in gt_list],
    "C heading + 180 deg": [dict(g, yaw=wrap(g["yaw"] + np.pi)) for g in gt_list],
    "D noise falls with score": [dict(g, xy=g["xy"] + e) for g, e in zip(gt_list, D_noise)],
    "E bicycle recall 0.05": [dict(g) for i, g in enumerate(gt_list) if g["cls"] != "bicycle" or i in bic_keep],
}

# C3: properties of the generated data (label-free)
print("\nC3: data checks")
cnt = {c: sum(g["cls"] == c for g in gt_list) for c in CLASSES}
print("  GT boxes per class:", sorted(set(cnt.values())), "| total", len(gt_list))
print("  max ego distance / class range: GT %.4f, D predictions %.4f" % (
    max(np.hypot(*g["xy"]) / CLASS_RANGE[g["cls"]] for g in gt_list),
    max(np.hypot(*p["xy"]) / CLASS_RANGE[p["cls"]] for p in DET["D noise falls with score"])))
sep = min(np.hypot(*(a["xy"] - b["xy"])) for s in GT for i, a in enumerate(GT[s]) for b in GT[s][i + 1:]
          if a["cls"] == b["cls"])
print("  min same-class separation within a keyframe: %.4f m" % sep)
print("  every attribute valid for its class:", all((g["attr"] in REL[g["cls"]]) or (g["attr"] == "" and not REL[g["cls"]])
                                                   for g in gt_list))
print("  E keeps %d bicycle predictions" % sum(p["cls"] == "bicycle" for p in DET["E bicycle recall 0.05"]))
print("  plain mean GT speed (m/s):", " ".join(f"{c[:5]} {np.mean([np.hypot(*g['vel']) for g in gt_list if g['cls'] == c]):.4f}"
                                                for c in CLASSES[:8]))

# ---------------- score every detector ----------------
SHORT = ["ATE", "ASE", "AOE", "AVE", "AAE"]
R = {}
for name, preds in DET.items():
    r = R[name] = evaluate(GT, by_sample(preds), CLASSES)
    print(f"\n{name}")
    print(f"  {'class':<21s}{'AP':>7s}" + "".join(f"{h:>8s}" for h in SHORT) + f"{'plainATE':>9s}")
    for c in CLASSES:
        pl = r["raw"][c]["trans_err"].mean() if len(r["raw"][c]["trans_err"]) else np.nan
        print(f"  {c:<21s}{r['mean_dist_aps'][c]:7.4f}" + "".join(f"{r['tps'][c][m]:8.4f}" for m in TP_METRICS)
              + f"{pl:9.4f}")
    print("  mAP by threshold:", " ".join(f"{d} m {np.mean([r['aps'][c][d] for c in CLASSES]):.4f}" for d in CFG["dist_ths"]),
          f"| mAP {r['mAP']:.4f}")
    plain = {}
    for m in TP_METRICS:
        vals = []
        for c in CLASSES:
            if np.isnan(r["tps"][c][m]):
                continue
            x = r["raw"][c][m]
            vals.append(np.nanmean(x) if len(x) and not np.all(np.isnan(x)) else np.nan)
        plain[m] = np.nanmean(vals)
    print("  mTP (devkit):      " + " ".join(f"m{h} {r['mTP'][m]:.4f}" for h, m in zip(SHORT, TP_METRICS)))
    print("  plain mean errors: " + " ".join(f"m{h} {plain[m]:.4f}" for h, m in zip(SHORT, TP_METRICS)))
    print("  TP scores:         " + " ".join(f"{h} {r['scores'][m]:.4f}" for h, m in zip(SHORT, TP_METRICS)))
    print(f"  NDS {r['NDS']:.6f}")

# ---------------- hypothesis checks ----------------
A, B, Cc, Dd, Ee = (R[k] for k in DET)
print("\nChecks (absolute deviations)")
nonex = lambda r: [r["tps"][c][m] for c in CLASSES for m in TP_METRICS if not np.isnan(r["tps"][c][m])]
print(f"H1 A: max|AP-1| {max(abs(A['mean_dist_aps'][c] - 1) for c in CLASSES):.1e}, "
      f"max per-threshold |AP-1| {max(abs(A['aps'][c][d] - 1) for c in CLASSES for d in CFG['dist_ths']):.1e}, "
      f"max TP error {max(nonex(A)):.1e}, |NDS-1| {abs(A['NDS'] - 1):.1e}")
dap = lambda r: max(abs(r["aps"][c][d] - A["aps"][c][d]) for c in CLASSES for d in CFG["dist_ths"])
print(f"H2 B: max|dAP| {dap(B):.1e}, max|dmTP| other four "
      f"{max(abs(B['mTP'][m] - A['mTP'][m]) for m in TP_METRICS if m != 'vel_err'):.1e}, "
      f"mAVE {B['mTP']['vel_err']:.4f}, AVE score {B['scores']['vel_err']:.4f}, "
      f"|NDS_A-NDS_B-0.1| {abs(A['NDS'] - B['NDS'] - 0.1):.1e}, cone/barrier AVE "
      f"{B['tps']['traffic_cone']['vel_err']}/{B['tps']['barrier']['vel_err']}")
print(f"H3 C: max|dAP| {dap(Cc):.1e}, barrier AOE {Cc['tps']['barrier']['orient_err']:.1e}, "
      f"max|AOE-pi| other 8 {max(abs(Cc['tps'][c]['orient_err'] - np.pi) for c in CLASSES if c not in ('barrier', 'traffic_cone')):.1e}, "
      f"cone AOE {Cc['tps']['traffic_cone']['orient_err']}, mAOE {Cc['mTP']['orient_err']:.6f}, "
      f"|mAOE-8pi/9| {abs(Cc['mTP']['orient_err'] - 8 * np.pi / 9):.1e}, AOE score {Cc['scores']['orient_err']:.4f}, "
      f"|NDS_A-NDS_C-0.1| {abs(A['NDS'] - Cc['NDS'] - 0.1):.1e}")
gaps = {c: Dd["raw"][c]["trans_err"].mean() - Dd["tps"][c]["trans_err"] for c in CLASSES}
pm = np.mean([Dd["raw"][c]["trans_err"].mean() for c in CLASSES])
print("H4 D: plain ATE - devkit ATE per class: " + " ".join(f"{c[:5]} {gaps[c]:.4f}" for c in CLASSES))
print(f"H4 D: min gap {min(gaps.values()):.4f} m; plain mATE {pm:.4f} vs devkit mATE {Dd['mTP']['trans_err']:.4f} "
      f"(gap {pm - Dd['mTP']['trans_err']:.4f}); max other TP error {max(Dd['tps'][c][m] for c in CLASSES for m in TP_METRICS[1:] if not np.isnan(Dd['tps'][c][m])):.1e}")
nds_e = (5 * 0.9 + (0.9 + 0.9 + 8 / 9 + 7 / 8 + 7 / 8)) / 10
print(f"H5 E: bicycle AP {float(Ee['mean_dist_aps']['bicycle'])!r}, bicycle TP errors "
      f"{[Ee['tps']['bicycle'][m] for m in TP_METRICS]}, max|dAP| other classes "
      f"{max(abs(Ee['aps'][c][d] - A['aps'][c][d]) for c in CLASSES if c != 'bicycle' for d in CFG['dist_ths']):.1e}, "
      f"max|dTP| other classes {max(abs(Ee['tps'][c][m] - A['tps'][c][m]) for c in CLASSES if c != 'bicycle' for m in TP_METRICS if not np.isnan(A['tps'][c][m])):.1e}, "
      f"|mAP_E-0.9 mAP_A| {abs(Ee['mAP'] - 0.9 * A['mAP']):.1e}, NDS_E {Ee['NDS']:.9f}, |NDS_E-0.893888889| {abs(Ee['NDS'] - nds_e):.1e}")

C1: devkit unit tests reproduced by the port
  scale_iou (exact cases)              3/3   passed, max deviation 0.0e+00 (tolerance 1e-15)
  scale_iou (arbitrary, 7 places)      1/1   passed, max deviation 8.6e-09 (tolerance 5e-08)
  angle_diff                           7/7   passed, max deviation 1.7e-13 (tolerance 5e-08)
  center_distance                      5/5   passed, max deviation 0.0e+00 (tolerance 5e-08)
  velocity_l2                          3/3   passed, max deviation 0.0e+00 (tolerance 5e-08)
  cummean (decimal 6)                  7/7   passed, max deviation 6.7e-07 (tolerance 1.5e-06)
  yaw_diff (via quaternion helper)   103/103 passed, max deviation 4.4e-16 (tolerance 5e-08)
  TestAPSimple (11 cases, 0.01)       11/11  passed, max deviation 6.2e-03 (tolerance 0.01)
  TestTPSimple (29 cases, 0.01)       29/29  passed, max deviation 2.8e-03 (tolerance 0.01)
C2: test_nd_score: port 0.08606662159639042, devkit target 0.08606662159639042, |difference| 0.0e+00 (tolerance 1e-12)
C1 all groups passed: True | C2 passed: True

C3: data checks
  GT boxes per class: [80] | total 800
  max ego distance / class range: GT 0.8991, D predictions 0.9178
  min same-class separation within a keyframe: 10.6596 m
  every attribute valid for its class: True
  E keeps 4 bicycle predictions
  plain mean GT speed (m/s): car 1.5468 truck 1.8310 bus 2.9361 trail 2.0133 const 2.4377 pedes 0.4770 motor 2.2859 bicyc 1.4154

A exact
  class                     AP     ATE     ASE     AOE     AVE     AAE plainATE
  car                   1.0000  0.0000  0.0000  0.0000  0.0000  0.0000   0.0000
  truck                 1.0000  0.0000  0.0000  0.0000  0.0000  0.0000   0.0000
  bus                   1.0000  0.0000  0.0000  0.0000  0.0000  0.0000   0.0000
  trailer               1.0000  0.0000  0.0000  0.0000  0.0000  0.0000   0.0000
  construction_vehicle  1.0000  0.0000  0.0000  0.0000  0.0000  0.0000   0.0000
  pedestrian            1.0000  0.0000  0.0000  0.0000  0.0000  0.0000   0.0000
  motorcycle            1.0000  0.0000  0.0000  0.0000  0.0000  0.0000   0.0000
  bicycle               1.0000  0.0000  0.0000  0.0000  0.0000  0.0000   0.0000
  traffic_cone          1.0000  0.0000  0.0000     nan     nan     nan   0.0000
  barrier               1.0000  0.0000  0.0000  0.0000     nan     nan   0.0000
  mAP by threshold: 0.5 m 1.0000 1.0 m 1.0000 2.0 m 1.0000 4.0 m 1.0000 | mAP 1.0000
  mTP (devkit):      mATE 0.0000 mASE 0.0000 mAOE 0.0000 mAVE 0.0000 mAAE 0.0000
  plain mean errors: mATE 0.0000 mASE 0.0000 mAOE 0.0000 mAVE 0.0000 mAAE 0.0000
  TP scores:         ATE 1.0000 ASE 1.0000 AOE 1.0000 AVE 1.0000 AAE 1.0000
  NDS 1.000000

B zero velocity
  class                     AP     ATE     ASE     AOE     AVE     AAE plainATE
  car                   1.0000  0.0000  0.0000  0.0000  1.3539  0.0000   0.0000
  truck                 1.0000  0.0000  0.0000  0.0000  1.4378  0.0000   0.0000
  bus                   1.0000  0.0000  0.0000  0.0000  2.8786  0.0000   0.0000
  trailer               1.0000  0.0000  0.0000  0.0000  1.8717  0.0000   0.0000
  construction_vehicle  1.0000  0.0000  0.0000  0.0000  2.0633  0.0000   0.0000
  pedestrian            1.0000  0.0000  0.0000  0.0000  0.3815  0.0000   0.0000
  motorcycle            1.0000  0.0000  0.0000  0.0000  2.3194  0.0000   0.0000
  bicycle               1.0000  0.0000  0.0000  0.0000  1.2271  0.0000   0.0000
  traffic_cone          1.0000  0.0000  0.0000     nan     nan     nan   0.0000
  barrier               1.0000  0.0000  0.0000  0.0000     nan     nan   0.0000
  mAP by threshold: 0.5 m 1.0000 1.0 m 1.0000 2.0 m 1.0000 4.0 m 1.0000 | mAP 1.0000
  mTP (devkit):      mATE 0.0000 mASE 0.0000 mAOE 0.0000 mAVE 1.6917 mAAE 0.0000
  plain mean errors: mATE 0.0000 mASE 0.0000 mAOE 0.0000 mAVE 1.8679 mAAE 0.0000
  TP scores:         ATE 1.0000 ASE 1.0000 AOE 1.0000 AVE 0.0000 AAE 1.0000
  NDS 0.900000

C heading + 180 deg
  class                     AP     ATE     ASE     AOE     AVE     AAE plainATE
  car                   1.0000  0.0000  0.0000  3.1416  0.0000  0.0000   0.0000
  truck                 1.0000  0.0000  0.0000  3.1416  0.0000  0.0000   0.0000
  bus                   1.0000  0.0000  0.0000  3.1416  0.0000  0.0000   0.0000
  trailer               1.0000  0.0000  0.0000  3.1416  0.0000  0.0000   0.0000
  construction_vehicle  1.0000  0.0000  0.0000  3.1416  0.0000  0.0000   0.0000
  pedestrian            1.0000  0.0000  0.0000  3.1416  0.0000  0.0000   0.0000
  motorcycle            1.0000  0.0000  0.0000  3.1416  0.0000  0.0000   0.0000
  bicycle               1.0000  0.0000  0.0000  3.1416  0.0000  0.0000   0.0000
  traffic_cone          1.0000  0.0000  0.0000     nan     nan     nan   0.0000
  barrier               1.0000  0.0000  0.0000  0.0000     nan     nan   0.0000
  mAP by threshold: 0.5 m 1.0000 1.0 m 1.0000 2.0 m 1.0000 4.0 m 1.0000 | mAP 1.0000
  mTP (devkit):      mATE 0.0000 mASE 0.0000 mAOE 2.7925 mAVE 0.0000 mAAE 0.0000
  plain mean errors: mATE 0.0000 mASE 0.0000 mAOE 2.7925 mAVE 0.0000 mAAE 0.0000
  TP scores:         ATE 1.0000 ASE 1.0000 AOE 0.0000 AVE 1.0000 AAE 1.0000
  NDS 0.900000

D noise falls with score
  class                     AP     ATE     ASE     AOE     AVE     AAE plainATE
  car                   0.7518  0.4028  0.0000  0.0000  0.0000  0.0000   0.6016
  truck                 0.7790  0.3400  0.0000  0.0000  0.0000  0.0000   0.5825
  bus                   0.7244  0.4047  0.0000  0.0000  0.0000  0.0000   0.6640
  trailer               0.7723  0.3700  0.0000  0.0000  0.0000  0.0000   0.5675
  construction_vehicle  0.7744  0.3696  0.0000  0.0000  0.0000  0.0000   0.6237
  pedestrian            0.7171  0.4092  0.0000  0.0000  0.0000  0.0000   0.6101
  motorcycle            0.7909  0.3149  0.0000  0.0000  0.0000  0.0000   0.5352
  bicycle               0.7414  0.3879  0.0000  0.0000  0.0000  0.0000   0.5951
  traffic_cone          0.7523  0.3741  0.0000     nan     nan     nan   0.6657
  barrier               0.7999  0.3266  0.0000  0.0000     nan     nan   0.5687
  mAP by threshold: 0.5 m 0.3472 1.0 m 0.7370 2.0 m 0.9572 4.0 m 1.0000 | mAP 0.7603
  mTP (devkit):      mATE 0.3700 mASE 0.0000 mAOE 0.0000 mAVE 0.0000 mAAE 0.0000
  plain mean errors: mATE 0.6014 mASE 0.0000 mAOE 0.0000 mAVE 0.0000 mAAE 0.0000
  TP scores:         ATE 0.6300 ASE 1.0000 AOE 1.0000 AVE 1.0000 AAE 1.0000
  NDS 0.843177

E bicycle recall 0.05
  class                     AP     ATE     ASE     AOE     AVE     AAE plainATE
  car                   1.0000  0.0000  0.0000  0.0000  0.0000  0.0000   0.0000
  truck                 1.0000  0.0000  0.0000  0.0000  0.0000  0.0000   0.0000
  bus                   1.0000  0.0000  0.0000  0.0000  0.0000  0.0000   0.0000
  trailer               1.0000  0.0000  0.0000  0.0000  0.0000  0.0000   0.0000
  construction_vehicle  1.0000  0.0000  0.0000  0.0000  0.0000  0.0000   0.0000
  pedestrian            1.0000  0.0000  0.0000  0.0000  0.0000  0.0000   0.0000
  motorcycle            1.0000  0.0000  0.0000  0.0000  0.0000  0.0000   0.0000
  bicycle               0.0000  1.0000  1.0000  1.0000  1.0000  1.0000   0.0000
  traffic_cone          1.0000  0.0000  0.0000     nan     nan     nan   0.0000
  barrier               1.0000  0.0000  0.0000  0.0000     nan     nan   0.0000
  mAP by threshold: 0.5 m 0.9000 1.0 m 0.9000 2.0 m 0.9000 4.0 m 0.9000 | mAP 0.9000
  mTP (devkit):      mATE 0.1000 mASE 0.1000 mAOE 0.1111 mAVE 0.1250 mAAE 0.1250
  plain mean errors: mATE 0.0000 mASE 0.0000 mAOE 0.0000 mAVE 0.0000 mAAE 0.0000
  TP scores:         ATE 0.9000 ASE 0.9000 AOE 0.8889 AVE 0.8750 AAE 0.8750
  NDS 0.893889

Checks (absolute deviations)
H1 A: max|AP-1| 4.4e-16, max per-threshold |AP-1| 4.4e-16, max TP error 0.0e+00, |NDS-1| 2.2e-16
H2 B: max|dAP| 0.0e+00, max|dmTP| other four 0.0e+00, mAVE 1.6917, AVE score 0.0000, |NDS_A-NDS_B-0.1| 8.3e-17, cone/barrier AVE nan/nan
H3 C: max|dAP| 0.0e+00, barrier AOE 1.0e-16, max|AOE-pi| other 8 4.4e-16, cone AOE nan, mAOE 2.792527, |mAOE-8pi/9| 4.4e-16, AOE score 0.0000, |NDS_A-NDS_C-0.1| 8.3e-17
H4 D: plain ATE - devkit ATE per class: car 0.1988 truck 0.2426 bus 0.2593 trail 0.1975 const 0.2541 pedes 0.2009 motor 0.2203 bicyc 0.2072 traff 0.2917 barri 0.2422
H4 D: min gap 0.1975 m; plain mATE 0.6014 vs devkit mATE 0.3700 (gap 0.2315); max other TP error 0.0e+00
H5 E: bicycle AP 0.0, bicycle TP errors [1.0, 1.0, 1.0, 1.0, 1.0], max|dAP| other classes 0.0e+00, max|dTP| other classes 0.0e+00, |mAP_E-0.9 mAP_A| 1.1e-16, NDS_E 0.893888889, |NDS_E-0.893888889| 0.0e+00

Port checks. The port reproduces every unit test it was checked against, and test_nd_score to the last digit. Within the tests’ tolerances, the largest deviations are in the AP and TP cases, at most 0.0062. Those tests compare with values worked out by hand and allow 0.01, because the 101-point recall grid quantizes the curves. The data checks confirm the construction: 80 boxes per class, every box and every noisy prediction of D inside its class range, and same-class boxes at least 10.66 m apart.

What each defect costs. All five predictions held.

• A, exact boxes, held. Every class has AP 1 and every error 0, and NDS prints as 1.000000. The deviation from 1 is $2.2 \times 10^{-16}$, inherited from the floating-point rounding of AP, whose largest deviation from 1 is $4.4 \times 10^{-16}$.
• B, zero velocity, held. The devkit’s mAVE is 1.6917 m/s, so the velocity score clips to 0 and NDS falls by exactly 0.1 to 0.900000. AP and the other four errors do not change, and cones and barriers report no AVE. Any mAVE above 1 m/s would have cost the same. The nuScenes image baseline OFT set velocities to zero (§4.1) and has an mAVE of 1.73 m/s in Table 4, so on this term it scores no better than any other velocity error above 1 m/s. Even though the scores are drawn independently of speed, the devkit’s mAVE is below the plain mean of the same errors (1.8679 m/s).
• C, headings turned by 180°, held. Eight classes have an AOE of π. The barrier has 0, because its orientation is measured on a period of π, and the cone has none. mAOE is 2.792527, which is $8\pi/9$ to within $4.4 \times 10^{-16}$. The orientation score is 0, and NDS again falls by exactly 0.1. Driving in the wrong direction and ignoring motion cost the same here, and neither touches mAP.
• D, noise that falls with score, held. In every class, the devkit’s ATE is below the plain mean of the matched translation errors, by 0.1975 m (trailer) to 0.2917 m (traffic cone). The class means are 0.3700 m against 0.6014 m, and the other four errors stay 0. In our reading, this follows from the cumulative mean: the $k$-th most confident match enters every cumulative mean from $k$ onwards, so the confident matches, which this detector makes accurate, carry more weight than the plain mean gives them. The same noise moves AP mostly at the tight threshold, with mAP 0.3472 at 0.5 m against 1.0000 at 4 m. NDS is 0.843177, of which the AP loss costs $5 \times (1 - 0.7603)/10 \approx 0.12$ and the translation term $0.37/10$.
• E, one class below 10% recall, held. The bicycle class gets AP 0 and all five errors set to 1, although its four matches are exact, as the plain-error column shows. The other nine classes are unchanged, and mAP drops to 0.9. Because the exemptions change the denominators, the one class weighs $1/10$ in mATE and mASE, $1/9$ in mAOE and $1/8$ in mAVE and mAAE, which is why the five TP scores differ. NDS is 0.893889, the value derived before the run.

The example tests our ported functions on synthetic boxes with simple error models. It does not show how real detectors trade these terms against each other.

License

As of November 2026, the nuScenes Terms of Use, last updated on 16 November 2021, state that “Unless specifically labeled otherwise”, the datasets are provided under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Public License (CC BY-NC-SA 4.0), with additional terms. Where the two conflict, the Dataset Terms prevail. Downloading or using the data counts as agreeing to both.

The same terms cover nuImages, nuPlan, nuReasoning and nuReality. Third-party data within the datasets may carry other conditions. The dataset page describes the data as “strictly for non-commercial purposes” and invites commercial users to contact Motional for a commercial licence. Downloading requires an account and acceptance of the terms (devkit README).

The devkit, including the evaluation code, is released separately under the Apache License 2.0.

Limitations and Bias

• Two cities. All data come from parts of Boston and Singapore. The supplement reports relative mAP drops of 1–8% for three baselines on the Singapore part of the val set, which the authors think is likely due to different object appearance and label distributions.
• Night and rain. Night scenes have few objects, and the authors note that objects with bad visibility are harder to annotate. The baselines’ mAP fell by 36% (LiDAR) to 55–58% (cameras) relative to the full val set at night. A scene counts as rainy if raindrops are on the windshield, whether or not it is raining (supplement, Table 6 and §C).
• Sensor limits. The 32-beam LiDAR reaches about 70 m. Evaluation stops at 30–50 m depending on the class, so distant objects are not scored.
• A long tail. Classes with fewer than 10,000 annotations are removed from the benchmark, so rare road users such as wheelchairs, strollers and ambulances are not evaluated at all.
• A forgiving matching rule. Centre-distance matching at up to 4 m accepts boxes that 3D IoU would reject. Size and orientation enter only through ASE and AOE on matched boxes, at a combined weight of 0.2 of NDS.
• Clipped errors. As the example shows, NDS cannot tell a mean velocity error of 1 m/s from one of 10 m/s, or a mean heading error of 1 rad from one of π. A detector that sets every velocity to zero loses no more than one with large velocity errors.

Common Uses

• Training and evaluating LiDAR, camera-only and fusion 3D detectors. PointPillars served as the LiDAR baseline in the nuScenes paper, and camera methods in bird’s-eye-view perception and sensor fusion report their main results in NDS on nuScenes.
• Evaluating 3D multi-object tracking by detection, ranked by AMOTA.
• Trajectory prediction, LiDAR segmentation and derived occupancy benchmarks built on the same scenes.

Historical Milestones

• 2018. The devkit for the teaser data was released in September, and the first evaluation code in December (changelog).

• March 2019. The full dataset, the paper and devkit v1.0.0 were released.

• CVPR 2019. The first detection challenge was won by MEGVII with 63.3% NDS (detection README). On the test set in the paper’s Table 4:

  Method	Input	NDS	mAP
  Megvii (challenge entry)	LiDAR	63.3%	52.8%
  PointPillars (baseline)	LiDAR	45.3%	30.5%
  MonoDIS (challenge entry)	camera	38.4%	30.4%
  OFT (baseline)	camera	21.2%	12.6%

  PointPillars and MonoDIS reached almost the same mAP but different NDS, mainly through translation errors (0.52 against 0.74 m) and velocity errors (0.32 against 1.55 m/s). The text also says that MonoDIS “had larger scale errors”, while Table 4 lists a lower mASE for MonoDIS (0.26) than for PointPillars (0.29); the table is reported here.

• NIPS 2019. The first tracking challenge was won by StanfordIPRL-TRI with 55.0% AMOTA (sAMOTA in the paper) on the test set, against 15.1% for the AB3DMOT baseline on Megvii’s detections (supplement, Table 8; tracking README).

• 2020. The CAN bus expansion, the prediction challenge, nuImages and nuScenes-lidarseg were released. Detection challenges were won with 69.0% NDS (ICRA 2020, Noah CV Lab fusion) and 71.4% (NeurIPS 2020, CenterPoint).

• 2021. The ICRA 2021 challenge was won with 74.9% NDS (CenterPoint-Fusion and FusionPainting). Panoptic nuScenes v1.0 followed in July, and the Terms of Use were updated in November.

Related Datasets

• KITTI. The earlier multimodal driving benchmark, with front-camera labels and IoU matching.
• Waymo Open Dataset. Appeared after the first nuScenes release, with a size comparable to nuScenes but five times its annotation frequency (Table 1).
• Lyft Level 5. Released in the nuScenes database schema, so the nuScenes devkit can read it (§1.2).
• Sibling datasets under the same terms. nuImages (93,000 annotated camera images) and nuPlan (over 15,000 driving logs from four cities, by the dataset page) share the nuScenes Terms of Use.
• Other benchmarks. For pedestrian tracking in crowds, MOTChallenge is the 2D counterpart.

Resources

• nuScenes: the official site, with the data description, downloads (account required) and the detection and tracking leaderboards for current results.
• nuScenes devkit: the official code, with the detection and tracking evaluation, configuration files and challenge rules.
• Terms of Use: the licence terms quoted above.