Waymo Open Dataset

The Waymo Open Dataset is a set of driving datasets released by Waymo LLC and recorded by its vehicles. The first part, now called the Perception dataset, has 1,150 scenes of 20 seconds from San Francisco, Mountain View and Phoenix, with LiDAR and camera data and hand-labelled 3D and 2D boxes. Pei Sun, Henrik Kretzschmar, Dragomir Anguelov and colleagues at Waymo and Google described it in a CVPR 2020 paper. The citation on the dataset’s terms page gives the year 2019, and the repository README names a licence version of August 2019. As of November 2026 the family has three members (README): the Perception dataset, the Motion dataset of object trajectories with 3D maps, and the End-to-End Driving dataset of camera data and high-level commands.

The detection benchmark introduced APH, an average precision in which every true positive counts only as much as its heading is right. Results are reported at two difficulty levels, LEVEL_1 and LEVEL_2. This article covers the dataset family, its versions and licence, and AP and APH as the official C++ metrics code computes them. 3D object detection introduces the IoU matching and the paper’s formulas, and this article goes into the code.

Purpose

The authors wanted a dataset large and varied enough to study how 3D perception scales and how it generalizes across places. They describe the geographic coverage as the area of the union of all ego poses dilated by 150 m (§3.5; Table 1’s caption describes a 75 m radius): 40 km² in Phoenix and 36 km² in San Francisco and Mountain View together, 76 km² in all, which they report as 15.2 times the effective coverage of the largest earlier camera and LiDAR dataset (abstract, Table 1). The test scenes come from a geographic holdout area, so that models are evaluated on places they were not trained on (§1). The paper also delivers the LiDAR as range images with per-pixel vehicle poses, to make research on input representations other than point sets easier (§1, §3.4).

Dataset Size

The paper’s Table 1 compares the dataset with KITTI, nuScenes and Argoverse. For the Waymo Open Dataset it lists:

Quantity	Value
Scenes	1,150
Annotated LiDAR frames	230K
Hours	6.4
3D boxes	12M
2D boxes	9.9M
LiDARs / cameras	5 / 5
Average points per frame	177K
Visited area	76 km²

The scenes are sampled at 10 Hz (§4.2), and 1,150 scenes of 20 s make the 6.4 hours. Five LiDARs and five cameras were mounted on the vehicles (§3.1). The top LiDAR’s range is restricted to 75 m in the released data and that of the four others to 20 m, and the first two returns of each pulse are provided (Table 2), together with intensity and elongation per return (§3.4); the LiDAR article explains returns and range images. The front and two front-side cameras give 1920 × 1280 images and the two side cameras 1920 × 1040, cropped and downsampled from the raw sensor images (Table 3). Camera and LiDAR are synchronized to within −6 to 7 ms with 99.7% confidence (§3.4).

The paper’s counts of 2D labels disagree among themselves. The text reports “around 12M labeled 2D image objects” and around 254k image track IDs (§3.5), the introduction around 250k camera tracks, Table 1 9.9M 2D boxes, and Table 5’s per-class 2D counts (9.0M, 2.7M and 81k) sum to about 11.8M. Table 1’s caption notes that 3D labels projected to 2D are not counted in its 2D boxes. The 3D figures agree: around 12M LiDAR boxes and around 113k LiDAR track IDs.

The dataset grew after the paper (release history, LOG.md). Later perception versions mainly added labels and formats; the one addition of scenes the release notes name is the 80 camera-only test segments of v1.3.1, and the readable pages give no updated totals; the data pages behind the sign-in were not read for this article.

Classes

Four classes are labelled in 3D: vehicles, pedestrians, cyclists and signs (§3.3). Table 5 gives their counts:

	Vehicle	Pedestrian	Cyclist	Sign
3D objects	6.1M	2.8M	67k	3.2M
3D track IDs	60k	23k	620	23k
2D objects	9.0M	2.7M	81k	–
2D track IDs	194k	58k	1.7k	–

Cyclists are rare against the other classes, and signs have no 2D labels. The labelling specifications in the repository fix the boundaries between classes: motorcycles and motorcyclists are labelled as vehicles, trains and trams are not labelled, people riding kick scooters, segways or skateboards are pedestrians, and a person riding a bicycle is a cyclist. Vehicles with trailers get two boxes. No Label Zones are created for areas of little immediate relevance to driving, such as the opposite side of a highway or a roadside parking lot with many vehicles.

Annotation Types

• 3D boxes. Every LiDAR label is a 7-degree-of-freedom upright box $(c_x, c_y, c_z, l, w, h, \theta)$ in the vehicle frame, with $l$, $w$, $h$ along the box’s $x$, $y$, $z$ axes and a tracking ID that persists through the scene (§3.3). The vehicle frame has $x$ forward, $y$ left and $z$ up (§3.2). label.proto defines the heading as the angle that rotates $+x$ to the normal of the box’s front face, normalized to $[-\pi, \pi)$.
• 2D boxes. Camera labels are tight axis-aligned boxes with their own tracking IDs, annotated separately from the LiDAR labels (§3.3).
• Difficulty. Each 3D label can carry a detection and a tracking difficulty level (label.proto).
• Later additions (LOG.md): 3D semantic segmentation and 2D and 3D keypoints in v1.3.0 (March 2022); 80 camera-only test segments for a camera 3D detection challenge in v1.3.1 (April 2022); 2D video panoptic segmentation in v1.4.0 (June 2022); 3D maps in v1.4.2 (March 2023); and an object-centric asset set with over 1.2 million images and LiDAR observations of vehicles and pedestrians from v2.0.0 (August 2023).

From v2.0.0 (March 2023) the Perception dataset is also released in a modular format, so users can download only the components they need; v2.0.0 includes all features of v1.4.2 except maps. Versions v1.4.3 and v2.0.1 (March 2024) improved the 3D semantic segmentation labels.

Splits

The paper fixes 798 training, 202 validation and 150 test scenes (§4). The test labels are hidden, and the paper evaluates its baselines on “the 150-scene hidden test set” (§5.1). Table 4 breaks down the training and validation scenes:

	Phoenix	Mountain View	San Francisco	Day	Night	Dawn
Training	286	103	409	646	79	73
Validation	93	21	88	160	23	19

Both rows sum to their split sizes by city and by time of day.

Tasks

The paper defines 3D and 2D object detection and tracking (§4). The 3D detection task asks for upright boxes of the four classes from any of the sensors, optionally using earlier frames (§4.1.1); 2D detection uses camera images only (§4.1.2); tracking is scored with MOTA and MOTP (§4.2, see MOTA).

Later releases added further tasks with their own metrics and challenges: motion prediction, occupancy and flow prediction, simulated agents, pose estimation, 3D semantic segmentation, 2D video panoptic segmentation and camera-only 3D detection (LOG.md). The home page lists the 2025 challenges as interaction prediction, sim agents, scenario generation and vision-based end-to-end driving; they closed on 22 May 2025, and the leaderboards remain open for benchmarking (as of November 2026).

• Motion dataset. Added in March 2021 with trajectories and 3D maps for over 100,000 segments (LOG.md), and described by Ettinger et al. (2021). The README gives 103,354 scenes. Motion forecasting describes its windows, mining and metrics. Versions run to v1.3.0 (October 2024), with LiDAR for the first second of each training window from v1.2.0 and camera tokens and embeddings from a pre-trained VQ-GAN, not raw images, from v1.2.1.
• End-to-End Driving dataset. Released in March 2025 (v1.0.0). By April 2025 it had 2,037 training runs, 479 validation runs with rater feedback labels and scenario tags, and 1,505 test runs (LOG.md).

Evaluation

The official detection metrics are C++ code in the repository, which also provides TensorFlow helpers; test-set results go through the challenge leaderboards on the official site. The description below follows the code at commit 99a4cb3, chiefly the command-line tool compute_detection_metrics_main.cc, whose comment says that it assumes 3D upright boxes by default. The code needs TensorFlow and Bazel, which are not installed here, so it was read line by line and its scoring functions were ported to Python for the example below.

Matching

• Score cutoffs. The tool evaluates 101 fixed cutoffs, $0, 0.01, \dots, 0.99, 1$ (GetConfig). At each, it keeps the predictions with score $\ge$ the cutoff (BuildSubsets).
• Overlap. For 3D boxes the IoU is the overlap area of the two rotated footprints times the overlap of their vertical extents, divided by the union of the volumes (ComputeIoU3d). Boxes with a side of 0.01 m or less, or of $10^6$ m or more, get IoU 0. Predictions and ground truth of different types get IoU 0.
• Thresholds. 0.7 for vehicles and 0.5 for pedestrians, signs and cyclists (GetConfig). A pair can match if its IoU is at least the threshold (CanMatch).
• Assignment. In each frame, at each cutoff and for each class, HungarianMatcher finds the assignment of predictions to ground truth that maximizes the sum of matching weights. A weight is the IoU quantized to an integer, std::round(weight * kMaxIoU) with kMaxIoU $= 10^6$ (matcher.h, iou.cc), because the Hungarian solver needs exact integer arithmetic (hungarian.h). Pairs that cannot match get weight 0, and an assigned pair that cannot match is dropped afterwards. The Hungarian algorithm explains the assignment.

LEVEL_1 and LEVEL_2

The tool first drops every ground-truth box with no LiDAR point. A box that carries no human-annotated difficulty is then set to LEVEL_2 if it has 5 or fewer LiDAR points and to LEVEL_1 otherwise (compute_detection_metrics_main.cc). The paper adds that LEVEL_2 also takes boxes “the labeler annotates as hard” (§5.1).

One matching serves both levels. ComputeDetectionMeasurementFromMatchingResult then counts, at each level (detection_metrics.cc, metrics_utils.cc):

• a true positive for every matched prediction, whatever the level of its ground truth;
• a false positive for every unmatched prediction, except predictions flagged as overlapping a No Label Zone;
• a false negative for every unmatched ground-truth box whose level is at most the evaluated level.

LEVEL_2 is cumulative and counts every box, as the paper says (§3.3). LEVEL_1 is not a score restricted to LEVEL_1 boxes: unmatched LEVEL_2 boxes are not misses there, but predictions matched to LEVEL_2 boxes still count as hits. The example below shows the consequence.

From counts to AP and APH

At each cutoff, ToDetectionMetrics turns the merged counts into

$$p = \frac{TP}{TP + FP}, \qquad h = \frac{\sum_{\text{TP}} a}{TP + FP}, \qquad r = \frac{TP}{TP + FN},$$

where $a$ is the heading accuracy of a true positive. The heading-weighted curve uses the same recall as the plain one; a code comment explains that no matcher takes heading into account. Where recall is 0, precision is set to 1. ComputeHeadingAccuracy normalizes both headings to $[-\pi, \pi)$, takes the absolute difference $\Delta$, replaces it by $2\pi - \Delta$ when it exceeds $\pi$, and returns

$$a = \min\big(1, \max(0,\ 1 - \Delta/\pi)\big).$$

A correct heading scores 1, one off by 90° scores 0.5 and one pointing backwards scores 0.

ComputeMeanAveragePrecision then turns each curve into a number:

1. It maps each recall to the largest precision seen at that recall, and adds the point (recall 0, precision 1).
2. It walks from high to low recall, keeping the running maximum of precision. Where two consecutive recalls are more than $\delta + 10^{-6}$ apart, it inserts points every $\delta$ that carry the running maximum from the higher-recall side, which is the conservative choice. The configuration’s desired_recall_delta defaults to $\delta = 0.05$ (metrics.proto).
3. It replaces the precision of the recall-0 point by that of the next point, because the added (0, 1) point is fake.
4. It integrates by the trapezoid rule.

The same function gives AP from $p$ and APH from $h$. Its values are held in single precision (float).

Code and paper

The paper’s §4.1.1 and the code differ in several places. None of these is called a bug here; they are differences of form.

• Heading weight. The paper writes the weight as $\min(|\tilde\theta - \theta|, 2\pi - |\tilde\theta - \theta|)/\pi$, which is the normalized heading error and is 0 for a perfect heading. The code returns one minus it. 3D object detection and sensor fusion for autonomous driving also read the paper’s expression as the normalized error and give the one-minus form as their own reading; ComputeHeadingAccuracy settles it.
• Integral. The paper’s Eq. 5 and 6 integrate the interpolated precision $\max_{r' \ge r} p(r')$ over recall and multiply by 100. The code integrates the conservatively filled curve with the trapezoid rule and reports a fraction.
• Worked example. The paper illustrates gap filling with $p(0) = 1$, $p(1) = 0$, $\delta = 0.05$ and says that AP is 0.05 after the augmentation. Walking the code gives 0, because the inserted points carry precision 0 and the recall-0 point is overridden by the next point’s precision. The example below prints this.
• Cutoffs. The paper keeps predictions with scores “above” each threshold; the code keeps scores at or above it.
• Signs. The paper decides sign true positives by box-centre distance. The tool’s configuration matches signs by IoU at 0.5.
• Difficulty. The paper’s LEVEL_2 rule includes the labeller’s “hard” flag. The command-line tool sets levels from point counts only for boxes without an annotated level.

The paper also reports AP and APH for bird’s-eye-view boxes, with 5 degrees of freedom (§5.1). The tool’s other box types compute IoU on the rotated footprints (TYPE_2D) or on axis-aligned boxes (TYPE_AA_2D).

nuScenes, by contrast, matches by centre distance and scores heading separately as an orientation error of the matched boxes within its detection score, NDS; the nuScenes article defines it. There, as that article’s example shows, a detector that turns every heading by 180° loses exactly 0.1 of NDS, the same as any mean orientation error of 1 radian or more. APH instead discounts each true positive’s contribution to precision.

Example: AP against APH as the code computes them

The script below scores five synthetic detectors with ported Waymo metrics, to show what APH and the two levels reward. The scoring code is ported line by line from the repository at commit 99a4cb3: ComputeMeanAveragePrecision, ComputeHeadingAccuracy and NormalizeAngle, IsTP, IsFP and IsDetectionFN, ComputeDetectionMeasurementFromMatchingResult, ToDetectionMetrics, the matcher’s CanMatch, MatchingWeight and QuantizedWeight, and ComputeIoU3d with kMaxIoU, using single precision where the C++ uses float. Cutoffs, thresholds and the LEVEL rule are those of the command-line tool. The official code was not run, because it needs TensorFlow and Bazel. Three parts are ours: the polygon clipping (Sutherland–Hodgman), the synthetic data, and SciPy’s linear_sum_assignment, which stands in for the Hungarian solver in hungarian.cc, not read for this article. In this layout every prediction overlaps at most one ground-truth box, as the printed data checks show, so the maximum-weight assignment is unique and any correct solver gives the same matches.

Before anything else, the script checks the port against the repository’s unit tests: the five ComputeMeanAveragePrecision cases and three ComputeHeadingAccuracy cases of metrics_utils_test.cc, the IsTP, IsFP and IsFN cases of the same file, and the MatcherIoU and HungarianMatch cases of matcher_test.cc. It also prints the code’s value for the paper’s worked example.

Synthetic scenes. The scenes are synthetic, not Waymo data:

• 200 frames; in each, a 4 × 3 grid of slots 12 m apart, with ±1 m jitter, holds 8 vehicles and 4 pedestrians, so 1,600 vehicles and 800 pedestrians;
• vehicles are 4.2–5.2 m long, 1.9–2.2 m wide and 1.5–1.9 m tall, and pedestrians 0.7–1.0 × 0.6–0.9 × 1.6–1.9 m; every box stands on flat ground ($c_z = h/2$) with a uniform random heading;
• coordinates follow label.proto: metres, length along the heading, heading in radians about $z$ from the $x$ axis;
• within each class, every fourth box gets 1–5 LiDAR points and the others 6–399, so a quarter of the boxes are LEVEL_2 by the tool’s rule; no box has 0 points;
• every object has one score, drawn uniformly from 0.05 to 0.999 and shared by all detectors.

The code computes 3D IoU, as the tool does. Because every prediction keeps the height and ground position of its object, 3D IoU equals bird’s-eye-view IoU for every pair here.

Detectors. Each is an error model built from the ground truth, with one prediction per selected object:

• (A) copies every box exactly;
• (B) is A with the heading of every second object in score order, the highest first, turned by π;
• (C) is A with Gaussian heading noise of standard deviation 10°;
• (D) detects only the LEVEL_1 objects;
• (E) detects only the LEVEL_2 objects.

None adds false positives of its own, so B to E each differ from A in one defect. The script prints AP and APH, APH with the paper’s printed weight taken literally (the weight $\Delta/\pi$ in place of $1 - \Delta/\pi$), AP with $\delta = 1$ (no gap filling), and the largest recall step between consecutive cutoffs.

Checks.

• C1: the five AP unit tests, within the tests’ tolerance of $10^{-6}$;
• C2: the three heading-accuracy tests, within $10^{-6}$;
• C3: the twelve IsTP, IsFP and IsFN expectations;
• C4: the matcher tests, exactly;
• C5: AP = APH = 1 for A, within $10^{-6}$, for each class and level;
• C6: the data checks, including that no prediction overlaps more than one ground-truth box and that B’s flipped boxes keep the full matching weight $10^6$;
• C7: two runs print identical output.

Predictions fixed before the run. Each was derived from the code and the construction:

• H1. B’s AP equals A’s, and B’s APH is 0.5, both within $10^{-6}$. A box turned by π has the same footprint, so the matches do not change. With the top-ranked object flipped, the heading-weighted precision at $T$ true positives is $\lfloor T/2 \rfloor / T$, exactly 0.5 for even $T$, and $T$ is even (1,600 or 800) at the highest recall, so the running maximum is 0.5 everywhere.
• H2. With the literal weight, A’s APH is at most $10^{-6}$, and B’s lies between $0.5 - 10^{-6}$ and 0.505. The literal reading therefore ranks B above A.
• H3. Under C: (a) vehicle AP is below 0.999 at both levels; (b) pedestrian AP is 1 within $10^{-6}$; (c) for every class and level, APH < AP and APH/AP $\ge 1 - m/\pi$, where $m$ is the largest heading error among the true positives. The bound holds pointwise for the two curves, and the running maximum and the trapezoid rule preserve it.
• H4. Every recall step is below 0.05, so gap filling never fires and AP with $\delta = 1$ equals AP with $\delta = 0.05$ exactly.
• H5. D scores AP = APH = 1 at LEVEL_1 and 0.75 at LEVEL_2; E scores 0.25 at both levels, all within $10^{-6}$. Precision is 1 at every cutoff, since no detector here adds false positives. D’s LEVEL_2 recall is 1,200/1,600 and 600/800. At LEVEL_1, E’s 400 (or 200) hits count, while the 1,200 (or 600) LEVEL_1 boxes it misses are false negatives.
• H6. The code returns 0 for the paper’s worked example, at $\delta = 0.05$ and at $\delta = 1$, against the paper’s 0.05.

A planning script computed the reachable values for the checks before the plan was written: the unit tests, and detector A on the same data. It also printed the IoU of nominal vehicle and pedestrian boxes against pure rotations, and the code’s value for the paper’s example. H3 (a) and (b) were written after it printed that rotation table, H6 after it printed the example’s value, and H4 after it printed detector A’s largest recall steps (A’s rows only), so those are illustrations rather than blind tests. Aerial Object Detection’s second example shows how angle errors cost IoU for boxes of different aspect ratios (aerial object detection); the planning table is not reproduced here. The planning script was also timed once before the plan was written, to estimate the runtime.

Changes before the first run. The first pilot run stopped with a division by zero in our polygon clipper when it reached detector B. The edges of a box and of its copy turned by π lie on the same four lines. With an exact inside test, rounding could put an endpoint of a collinear edge just outside the other box, and the clipper then tried to intersect two collinear lines. With the editor’s approval, the inside test was given a tolerance of $10^{-9}$, the intersection routine now returns the edge’s endpoint when the two lines are parallel, and a printed check was added: the smallest IoU of any ground-truth box against its copy turned by π, which must be at least $1 - 10^{-9}$. Nothing else changed. The lines printed before the crash were the same in the repeated pilot, and the repeated pilot’s script is the script below, run unchanged.

Running it. The script uses NumPy 2.3.5 and SciPy 1.16.3 under Python 3.13 and runs in about 3.5 seconds on the drafting machine. Two runs printed identical output.

"""Waymo Open Dataset 3D detection AP and APH, as the official metrics code
computes them, on synthetic boxes with known ground truth.

The scoring functions are ported (re-typed in Python, modified) from the
Waymo Open Dataset repository at commit 99a4cb3: ComputeMeanAveragePrecision,
ComputeHeadingAccuracy, NormalizeAngle, IsTP, IsFP and IsDetectionFN
(metrics/metrics_utils.cc); ComputeDetectionMeasurementFromMatchingResult and
ToDetectionMetrics (metrics/detection_metrics.cc); HungarianMatcher::Match,
CanMatch, MatchingWeight and QuantizedWeight (metrics/matcher.cc, matcher.h);
ComputeIoU3d and kMaxIoU (metrics/iou.cc); the score cutoffs, IoU thresholds
and LEVEL rule of tools/compute_detection_metrics_main.cc; and the expected
values of metrics_utils_test.cc and matcher_test.cc.
  Copyright 2019 The Waymo Open Dataset Authors.
  Copyright (c) 2023 Waymo LLC.
  Licensed under the Apache License, Version 2.0; modified (ported to Python).
  https://github.com/waymo-research/waymo-open-dataset/blob/99a4cb3ff07e2fe06c2ce73da001f850f628e45a/LICENSE
The polygon clipping, the Hungarian solver (SciPy) and the synthetic data are ours.
"""
import os
os.environ["OMP_NUM_THREADS"] = "1"; os.environ["OPENBLAS_NUM_THREADS"] = "1"
import math
import numpy as np
from scipy.optimize import linear_sum_assignment

f32 = np.float32
K_MAX_IOU = 1000 * 1000                              # iou.cc
TYPES = {"VEHICLE": 1, "PEDESTRIAN": 2}              # label.proto
IOU_THR = {1: f32(0.7), 2: f32(0.5)}                 # GetConfig()
CUTOFFS = [f32(i * 0.01) for i in range(100)] + [f32(1.0)]

# ---------- rotated upright boxes (clipping is ours; IoU follows ComputeIoU3d) ----------
def corners(cx, cy, l, w, h):
    c, s = math.cos(h), math.sin(h)
    return [(cx + c*dx - s*dy, cy + s*dx + c*dy)
            for dx, dy in ((l/2, w/2), (-l/2, w/2), (-l/2, -w/2), (l/2, -w/2))]

def clip(subject, clipper):                           # Sutherland-Hodgman, convex
    def inside(p, a, b):
        return (b[0]-a[0])*(p[1]-a[1]) - (b[1]-a[1])*(p[0]-a[0]) >= -1e-9
    def inter(p, q, a, b):
        x1, y1 = p; x2, y2 = q; x3, y3 = a; x4, y4 = b
        den = (x1-x2)*(y3-y4) - (y1-y2)*(x3-x4)
        if den == 0.0:                                # collinear edges
            return q
        t = ((x1-x3)*(y3-y4) - (y1-y3)*(x3-x4)) / den
        return (x1 + t*(x2-x1), y1 + t*(y2-y1))
    out = subject
    for i in range(len(clipper)):
        a, b = clipper[i], clipper[(i+1) % len(clipper)]
        inp, out = out, []
        if not inp:
            break
        for j in range(len(inp)):
            p, q = inp[j], inp[(j+1) % len(inp)]
            if inside(q, a, b):
                if not inside(p, a, b):
                    out.append(inter(p, q, a, b))
                out.append(q)
            elif inside(p, a, b):
                out.append(inter(p, q, a, b))
    return out

def area(poly):
    if len(poly) < 3:
        return 0.0
    return abs(sum(poly[i][0]*poly[(i+1) % len(poly)][1] - poly[(i+1) % len(poly)][0]*poly[i][1]
                   for i in range(len(poly)))) / 2

def iou3d(b1, b2):
    for b in (b1, b2):
        if b["l"] <= 1e-2 or b["w"] <= 1e-2 or b["hgt"] <= 1e-2:
            return 0.0
    zmin = max(b1["cz"]-b1["hgt"]/2, b2["cz"]-b2["hgt"]/2)
    zmax = min(b1["cz"]+b1["hgt"]/2, b2["cz"]+b2["hgt"]/2)
    if zmin > zmax:
        return 0.0
    inter = area(clip(corners(b1["cx"], b1["cy"], b1["l"], b1["w"], b1["h"]),
                      corners(b2["cx"], b2["cy"], b2["l"], b2["w"], b2["h"]))) * (zmax - zmin)
    uni = b1["l"]*b1["w"]*b1["hgt"] + b2["l"]*b2["w"]*b2["hgt"] - inter
    if uni <= 1e-6:
        return 0.0
    return max(min(inter/uni, 1.0), 0.0)

# ---------- ports ----------
def normalize_angle(rad):                             # WrapToRange(-pi, pi, rad)
    mn, mx = -math.pi, math.pi
    x, y = rad - mn, mx - mn
    q = x * (1 / y); t = math.trunc(q); mod = x - t*y; sh = y + mod
    return (mod if mod >= 0 else (0.0 if sh >= y else sh)) + mn

def heading_accuracy(pd_h, gt_h, literal=False):      # ComputeHeadingAccuracy
    p = f32(normalize_angle(pd_h)); g = f32(normalize_angle(gt_h))
    d = f32(abs(p - g))
    if float(d) > math.pi:
        d = f32(2.0*math.pi - float(d))
    if literal:                                       # the paper's printed weight
        return f32(min(1.0, max(0.0, float(d)/math.pi)))
    return f32(min(1.0, max(0.0, 1.0 - float(d)/math.pi)))

def quantized_weight(iou):                            # Matcher::QuantizedWeight
    return int(round(float(f32(iou) * f32(K_MAX_IOU))))

def hungarian_match(ious, thr):                       # HungarianMatcher::Match
    npd, ngt = ious.shape
    pdm, gtm = [-1]*npd, [-1]*ngt
    if npd == 0 or ngt == 0:
        return pdm, gtm
    n = max(npd, ngt)
    W = np.zeros((n, n), dtype=np.int64)
    for i in range(npd):
        for j in range(ngt):
            W[i, j] = 0 if ious[i, j] < thr else quantized_weight(ious[i, j])
    rows, cols = linear_sum_assignment(W, maximize=True)
    perm = dict(zip(rows.tolist(), cols.tolist()))
    for i in range(npd):
        j = perm[i]; can = False
        if j >= ngt:
            j = -1
        else:
            can = not (ious[i, j] < thr)
        pdm[i] = j if can else -1
        if j >= 0:
            gtm[j] = i if can else -1
    return pdm, gtm

def is_tp(pd_matches, i):
    return pd_matches[i] >= 0

def is_fp(pd_matches, i, overlap_with_nlz):
    return pd_matches[i] < 0 and not overlap_with_nlz[i]

def is_detection_fn(gt_matches, i, gt_levels, level):
    if gt_matches[i] >= 0:
        return False
    return gt_levels[i] <= level

def compute_map(precisions, recalls, delta):          # ComputeMeanAveragePrecision
    if len(precisions) == 0:
        return f32(0.0)
    rp = {f32(0.0): f32(1.0)}
    for p, r in zip(precisions, recalls):
        r = f32(r); rp[r] = max(rp.get(r, f32(0.0)), f32(p))
    keys = sorted(rp)
    d, err = f32(delta), f32(1e-6)
    pr, last, maxp = [], keys[0], f32(0.0)
    for r in reversed(keys):
        while last - r > d + err:
            last = f32(last - d); pr.append((maxp, last))
        maxp = max(rp[r], maxp); pr.append((maxp, r)); last = r
    if len(pr) >= 2:
        pr[-1] = (pr[-2][0], pr[-1][1])
    m = f32(0.0)
    for i in range(1, len(pr)):
        m = f32(float(m) + 0.5*float(f32(pr[i-1][1] - pr[i][1]))*float(f32(pr[i-1][0] + pr[i][0])))
    return m

def curves(meas, k_ha):                               # ToDetectionMetrics
    P, R, PH = [], [], []
    for m in meas:
        tp, fp, fn, sha = m[0], m[1], m[2], m[k_ha]
        if tp + fp <= 0:
            p = ph = f32(0.0)
        else:
            p = f32(tp) / f32(tp + fp); ph = f32(sha) / f32(tp + fp)
        r = f32(0.0) if tp + fn <= 0 else f32(tp) / f32(tp + fn)
        if r == 0.0:
            p = ph = f32(1.0)
        P.append(p); R.append(r); PH.append(ph)
    return P, R, PH

def evaluate(gts, preds, typ):
    """Per frame and cutoff: match, then ComputeDetectionMeasurementFromMatchingResult
    for LEVEL_1 and LEVEL_2; measurements merged over frames."""
    meas = {lev: [[0, 0, 0, f32(0), f32(0)] for _ in CUTOFFS] for lev in (1, 2)}
    info = dict(max_overlap=0, min_tp_weight=K_MAX_IOU, max_tp_err_deg=0.0)
    for g_all, p_all in zip(gts, preds):
        g = [o for o in g_all if o["type"] == typ]
        p = [o for o in p_all if o["type"] == typ]
        I = np.zeros((len(p), len(g)), dtype=np.float32)
        for i, a in enumerate(p):
            for j, b in enumerate(g):
                I[i, j] = f32(iou3d(a, b))
        if len(p) and len(g):
            info["max_overlap"] = max(info["max_overlap"], int((I > 0).sum(axis=1).max()))
        nlz = [False] * len(p)
        levels = [o["level"] for o in g]
        for ci, c in enumerate(CUTOFFS):
            sub = [i for i in range(len(p)) if p[i]["score"] >= c]
            pdm, gtm = hungarian_match(I[sub, :], IOU_THR[typ])
            sub_nlz = [nlz[i] for i in sub]
            for lev in (1, 2):
                tp = fp = fn = 0; sha = f32(0.0); shl = f32(0.0)
                for k, i in enumerate(sub):
                    if is_tp(pdm, k):
                        ha = heading_accuracy(p[i]["h"], g[pdm[k]]["h"])
                        sha = f32(sha + ha)
                        shl = f32(shl + heading_accuracy(p[i]["h"], g[pdm[k]]["h"], literal=True))
                        tp += 1
                        if ci == 0 and lev == 1:
                            info["min_tp_weight"] = min(info["min_tp_weight"], quantized_weight(I[i, pdm[k]]))
                            info["max_tp_err_deg"] = max(info["max_tp_err_deg"], 180.0*(1.0 - float(ha)))
                    if is_fp(pdm, k, sub_nlz):
                        fp += 1
                for j in range(len(g)):
                    if is_detection_fn(gtm, j, levels, lev):
                        fn += 1
                m = meas[lev][ci]
                m[0] += tp; m[1] += fp; m[2] += fn; m[3] = f32(m[3] + sha); m[4] = f32(m[4] + shl)
    return meas, info

# ---------- synthetic data ----------
def make_data(seed=40, frames=200, nveh=8):
    rng = np.random.default_rng(seed)
    gts, counter = [], {1: 0, 2: 0}
    for fr in range(frames):
        objs = []
        slots = [(gx*12.0, gy*12.0) for gx in range(4) for gy in range(3)]
        for k, (sx, sy) in enumerate(slots):
            t = 1 if k < nveh else 2
            if t == 1:
                l, w, hg = rng.uniform(4.2, 5.2), rng.uniform(1.9, 2.2), rng.uniform(1.5, 1.9)
            else:
                l, w, hg = rng.uniform(0.7, 1.0), rng.uniform(0.6, 0.9), rng.uniform(1.6, 1.9)
            idx = counter[t]; counter[t] += 1
            pts = int(rng.integers(1, 6)) if idx % 4 == 3 else int(rng.integers(6, 400))
            objs.append(dict(type=t, cx=sx + rng.uniform(-1, 1), cy=sy + rng.uniform(-1, 1),
                             l=l, w=w, hgt=hg, cz=hg/2, h=rng.uniform(-math.pi, math.pi),
                             points=pts, level=2 if pts <= 5 else 1,   # tool's LEVEL rule
                             score=f32(rng.uniform(0.05, 0.999))))
        gts.append(objs)
    return gts

def detectors(gts):
    A = [[dict(o) for o in fr] for fr in gts]
    B = [[dict(o) for o in fr] for fr in gts]
    for t in TYPES.values():
        objs = [o for fr in B for o in fr if o["type"] == t]
        for rank, o in enumerate(sorted(objs, key=lambda o: -float(o["score"]))):
            if rank % 2 == 0:
                o["h"] = normalize_angle(o["h"] + math.pi)
    rng = np.random.default_rng(41)
    C = [[dict(o, h=normalize_angle(o["h"] + rng.normal(0.0, math.radians(10.0)))) for o in fr]
         for fr in gts]
    D = [[dict(o) for o in fr if o["level"] == 1] for fr in gts]
    E = [[dict(o) for o in fr if o["level"] == 2] for fr in gts]
    return {"A exact": A, "B half flipped": B, "C noise 10 deg": C,
            "D misses LEVEL_2": D, "E only LEVEL_2": E}

# ---------- run ----------
print("C1 ComputeMeanAveragePrecision tests (metrics_utils_test.cc, tolerance 1e-6)")
ap_cases = [(([], []), 0.05, 0.0), (([0.1], [0.2]), 0.05, 0.2*0.1),
            (([0.1, 0.05], [0.2, 1.0]), 0.05, 0.05*0.8 + 0.1*0.2 + (0.1-0.05)*0.5*0.05),
            (([0.1, 0.05], [0.2, 1.0]), 1.0, 0.2*0.1 + 0.5*(0.1+0.05)*0.8),
            (([0.1, 0.0, 0.05], [0.2, 0.4, 1.0]), 0.05, 0.05*0.8 + 0.1*0.2 + (0.1-0.05)*0.5*0.05)]
c1 = []
for (p, r), d, want in ap_cases:
    got = float(compute_map(p, r, d)); c1.append(abs(got - want))
    print(f"  p={p} r={r} delta={d}: got {got:.9f} want {want:.9f} |diff| {abs(got-want):.1e}")
print("C2 ComputeHeadingAccuracy tests (tolerance 1e-6)")
c2 = []
for pd, gt, want in ((-0.1, 0.1, 1-0.2/math.pi), (-math.pi+0.1, math.pi-0.1, 1-0.2/math.pi), (0.0, math.pi, 0.0)):
    got = float(heading_accuracy(pd, gt)); c2.append(abs(got - want))
    print(f"  pd {pd:+.6f} gt {gt:+.6f}: got {got:.9f} want {want:.9f} |diff| {abs(got-want):.1e}")
print("C3 IsTP / IsFP / IsFN tests")
res = [is_tp([1, 2, -1], 0), is_tp([1, 2, -1], 1), not is_tp([1, 2, -1], 2)]
res += [not is_fp([-1, 0, -1], 0, [True, False, False]), not is_fp([-1, 0, -1], 1, [True, False, False]),
        is_fp([-1, 0, -1], 2, [True, False, False])]
gm, lv = [-1, 0, -1], [2, 1, 1]
res += [is_detection_fn(gm, 0, lv, 2), not is_detection_fn(gm, 0, lv, 1), not is_detection_fn(gm, 1, lv, 2),
        not is_detection_fn(gm, 1, lv, 1), is_detection_fn(gm, 2, lv, 2), is_detection_fn(gm, 2, lv, 1)]
print(f"  {sum(res)}/{len(res)} expectations met")
print("C4 matcher tests (matcher_test.cc)")
box = lambda L: dict(cx=0.0, cy=0.0, cz=0.0, l=L*100, w=1.0, hgt=1.0, h=0.0)
ws = [quantized_weight(iou3d(box(L), box(1.0))) for L in (0.0, 0.2, 0.6, 1.0)]
print(f"  MatcherIoU weights {ws} (want [0, 200000, 600000, 1000000])")
I = np.array([[0.60, 0], [1.0, 0.70], [0.65, 0.71], [0, 0]], dtype=np.float32)
hm = hungarian_match(I, f32(0.5))
e1 = hungarian_match(np.zeros((0, 2), dtype=np.float32), f32(0.5))
e2 = hungarian_match(np.zeros((0, 0), dtype=np.float32), f32(0.5))
print(f"  HungarianMatch {hm} (want ([-1, 0, 1, -1], [1, 2])); empty pd {e1}; both empty {e2}")
c4 = (ws == [0, 200000, 600000, 1000000] and hm == ([-1, 0, 1, -1], [1, 2])
      and e1 == ([], [-1, -1]) and e2 == ([], []))

print("Paper's worked example p(0)=1, p(1)=0 (paper: AP = 0.05)")
pe = [float(compute_map([1.0, 0.0], [0.0, 1.0], d)) for d in (0.05, 1.0)]
print(f"  code delta=0.05: {pe[0]:.7f}  code delta=1: {pe[1]:.7f}")

gts = make_data()
print("C6 data")
for name, t in TYPES.items():
    objs = [o for fr in gts for o in fr if o["type"] == t]
    print(f"  {name}: {len(objs)} boxes, {sum(o['level'] == 2 for o in objs)} LEVEL_2, "
          f"min points {min(o['points'] for o in objs)}")
sc = np.array([float(o["score"]) for fr in gts for o in fr], dtype=np.float32)
gap_sc = float(np.min(np.abs(sc[:, None] - np.array(CUTOFFS, dtype=np.float32)[None, :])))
print(f"  min |score - cutoff| {gap_sc:.1e}")
flip_iou = min(iou3d(o, dict(o, h=normalize_angle(o["h"] + math.pi))) for fr in gts for o in fr)
print(f"  clipper check: min IoU(box, heading+pi copy) {flip_iou:.9f} pass {flip_iou >= 1 - 1e-9}")

R = {}
hdr = f"  {'class':<10} {'level':<7} {'TP':>5} {'FP':>4} {'FN':>5} {'AP':>10} {'APH':>10} {'APH_lit':>10} {'AP_d=1':>10} {'gap':>7}"
for det, preds in detectors(gts).items():
    print(f"\n{det}\n{hdr}")
    for name, t in TYPES.items():
        meas, info = evaluate(gts, preds, t)
        for lev in (1, 2):
            P, Rc, PH = curves(meas[lev], 3)
            _, _, PL = curves(meas[lev], 4)
            ap, aph = compute_map(P, Rc, 0.05), compute_map(PH, Rc, 0.05)
            apl, ap1 = compute_map(PL, Rc, 0.05), compute_map(P, Rc, 1.0)
            gap = max(abs(float(Rc[i]) - float(Rc[i+1])) for i in range(len(Rc) - 1))
            tp, fp, fn = meas[lev][0][:3]
            R[(det[0], t, lev)] = dict(ap=float(ap), aph=float(aph), apl=float(apl), ap1=float(ap1),
                                       gap=gap, **info)
            print(f"  {name:<10} LEVEL_{lev} {tp:>5} {fp:>4} {fn:>5} {float(ap):>10.7f} {float(aph):>10.7f} "
                  f"{float(apl):>10.7f} {float(ap1):>10.7f} {gap:>7.4f}")
    print(f"  max GT boxes overlapped by one prediction {max(R[(det[0], t, 1)]['max_overlap'] for t in TYPES.values())}"
          + (f"; min quantized weight of TPs {min(R[('B', t, 1)]['min_tp_weight'] for t in TYPES.values())}"
             if det[0] == "B" else "")
          + ("; largest heading error of TPs (deg): " + ", ".join(
              f"{n} {R[('C', t, 1)]['max_tp_err_deg']:.2f}" for n, t in TYPES.items()) if det[0] == "C" else ""))

print("\nChecks and hypotheses")
keys = [(t, lev) for t in TYPES.values() for lev in (1, 2)]
print(f"C1 max |diff| {max(c1):.1e} pass {max(c1) <= 1e-6} | C2 max |diff| {max(c2):.1e} pass {max(c2) <= 1e-6}"
      f" | C3 pass {all(res)} | C4 pass {c4}")
c5 = max(max(abs(1 - R[('A',) + k]['ap']), abs(1 - R[('A',) + k]['aph'])) for k in keys)
print(f"C5 A: max |1-AP|, |1-APH| {c5:.1e} pass {c5 <= 1e-6}")
h1a = max(abs(R[('B',) + k]['ap'] - R[('A',) + k]['ap']) for k in keys)
h1b = max(abs(R[('B',) + k]['aph'] - 0.5) for k in keys)
print(f"H1 B: max |AP_B-AP_A| {h1a:.1e}, max |APH_B-0.5| {h1b:.1e} held {h1a <= 1e-6 and h1b <= 1e-6}")
h2a = max(R[('A',) + k]['apl'] for k in keys)
h2b = [R[('B',) + k]['apl'] for k in keys]
print(f"H2 literal: max APH_lit(A) {h2a:.1e}; APH_lit(B) range {min(h2b):.7f}-{max(h2b):.7f} "
      f"held {h2a <= 1e-6 and min(h2b) >= 0.5 - 1e-6 and max(h2b) <= 0.505}")
for t, name in ((1, "vehicle"), (2, "pedestrian")):
    for lev in (1, 2):
        r = R[('C', t, lev)]; ratio = r['aph'] / r['ap']; bound = 1 - r['max_tp_err_deg'] / 180.0
        print(f"H3 C {name} LEVEL_{lev}: AP {r['ap']:.7f} APH {r['aph']:.7f} APH/AP {ratio:.6f} "
              f"bound {bound:.6f} | (c) held {r['aph'] < r['ap'] and ratio >= bound - 1e-6}")
h3a = max(R[('C', 1, lev)]['ap'] for lev in (1, 2))
h3b = max(abs(1 - R[('C', 2, lev)]['ap']) for lev in (1, 2))
print(f"H3 (a) vehicle max AP_C {h3a:.7f} held {h3a < 1 - 1e-3} | (b) pedestrian max |1-AP_C| {h3b:.1e} held {h3b <= 1e-6}")
allk = [(d,) + k for d in "ABCDE" for k in keys]
h4g = max(R[k]['gap'] for k in allk); h4d = max(abs(R[k]['ap1'] - R[k]['ap']) for k in allk)
print(f"H4 delta=1: max recall gap {h4g:.4f}, max |AP(d=1)-AP(d=0.05)| {h4d:.1e} held {h4g < 0.05 and h4d == 0.0}")
dev = [abs(R[('D', t, 1)][m] - 1) for t in TYPES.values() for m in ('ap', 'aph')]
dev += [abs(R[('D', t, 2)][m] - 0.75) for t in TYPES.values() for m in ('ap', 'aph')]
dev += [abs(R[('E', t, lev)][m] - 0.25) for t in TYPES.values() for lev in (1, 2) for m in ('ap', 'aph')]
print(f"H5 D/E: max deviation from 1, 0.75, 0.25 {max(dev):.1e} held {max(dev) <= 1e-6}")
print(f"H6 paper example: code {pe[0]:.7f} (delta 0.05), {pe[1]:.7f} (delta 1), paper 0.05; "
      f"held {pe[0] == 0.0 and pe[1] == 0.0}")

C1 ComputeMeanAveragePrecision tests (metrics_utils_test.cc, tolerance 1e-6)
  p=[] r=[] delta=0.05: got 0.000000000 want 0.000000000 |diff| 0.0e+00
  p=[0.1] r=[0.2] delta=0.05: got 0.020000001 want 0.020000000 |diff| 1.4e-09
  p=[0.1, 0.05] r=[0.2, 1.0] delta=0.05: got 0.061250001 want 0.061250000 |diff| 1.2e-09
  p=[0.1, 0.05] r=[0.2, 1.0] delta=1.0: got 0.080000006 want 0.080000000 |diff| 5.7e-09
  p=[0.1, 0.0, 0.05] r=[0.2, 0.4, 1.0] delta=0.05: got 0.061250005 want 0.061250000 |diff| 4.9e-09
C2 ComputeHeadingAccuracy tests (tolerance 1e-6)
  pd -0.100000 gt +0.100000: got 0.936338007 want 0.936338023 |diff| 1.5e-08
  pd -3.041593 gt +3.041593: got 0.936338007 want 0.936338023 |diff| 1.5e-08
  pd +0.000000 gt +3.141593: got 0.000000048 want 0.000000000 |diff| 4.8e-08
C3 IsTP / IsFP / IsFN tests
  12/12 expectations met
C4 matcher tests (matcher_test.cc)
  MatcherIoU weights [0, 200000, 600000, 1000000] (want [0, 200000, 600000, 1000000])
  HungarianMatch ([-1, 0, 1, -1], [1, 2]) (want ([-1, 0, 1, -1], [1, 2])); empty pd ([], [-1, -1]); both empty ([], [])
Paper's worked example p(0)=1, p(1)=0 (paper: AP = 0.05)
  code delta=0.05: 0.0000000  code delta=1: 0.0000000
C6 data
  VEHICLE: 1600 boxes, 400 LEVEL_2, min points 1
  PEDESTRIAN: 800 boxes, 200 LEVEL_2, min points 1
  min |score - cutoff| 3.8e-06
  clipper check: min IoU(box, heading+pi copy) 1.000000000 pass True

A exact
  class      level      TP   FP    FN         AP        APH    APH_lit     AP_d=1     gap
  VEHICLE    LEVEL_1  1600    0     0  1.0000000  1.0000000  0.0000000  1.0000000  0.0179
  VEHICLE    LEVEL_2  1600    0     0  0.9999998  0.9999998  0.0000000  0.9999998  0.0175
  PEDESTRIAN LEVEL_1   800    0     0  1.0000000  1.0000000  0.0000000  1.0000000  0.0227
  PEDESTRIAN LEVEL_2   800    0     0  0.9999998  0.9999998  0.0000000  0.9999998  0.0200
  max GT boxes overlapped by one prediction 1

B half flipped
  class      level      TP   FP    FN         AP        APH    APH_lit     AP_d=1     gap
  VEHICLE    LEVEL_1  1600    0     0  1.0000000  0.5000000  0.5013221  1.0000000  0.0179
  VEHICLE    LEVEL_2  1600    0     0  0.9999998  0.4999999  0.5011449  0.9999998  0.0175
  PEDESTRIAN LEVEL_1   800    0     0  1.0000000  0.5000000  0.5030870  1.0000000  0.0227
  PEDESTRIAN LEVEL_2   800    0     0  0.9999998  0.4999999  0.5026221  0.9999998  0.0200
  max GT boxes overlapped by one prediction 1; min quantized weight of TPs 1000000

C noise 10 deg
  class      level      TP   FP    FN         AP        APH    APH_lit     AP_d=1     gap
  VEHICLE    LEVEL_1  1490  110    87  0.8845657  0.8497984  0.0349986  0.8845657  0.0181
  VEHICLE    LEVEL_2  1490  110   110  0.8712189  0.8370197  0.0343889  0.8712189  0.0156
  PEDESTRIAN LEVEL_1   800    0     0  1.0000000  0.9566659  0.0475757  1.0000000  0.0227
  PEDESTRIAN LEVEL_2   800    0     0  0.9999998  0.9565599  0.0470420  0.9999998  0.0200
  max GT boxes overlapped by one prediction 1; largest heading error of TPs (deg): VEHICLE 20.09, PEDESTRIAN 36.79

D misses LEVEL_2
  class      level      TP   FP    FN         AP        APH    APH_lit     AP_d=1     gap
  VEHICLE    LEVEL_1  1200    0     0  0.9999999  0.9999999  0.0000000  0.9999999  0.0175
  VEHICLE    LEVEL_2  1200    0   400  0.7500000  0.7500000  0.0000000  0.7500000  0.0131
  PEDESTRIAN LEVEL_1   600    0     0  0.9999995  0.9999995  0.0000000  0.9999995  0.0233
  PEDESTRIAN LEVEL_2   600    0   200  0.7500002  0.7500002  0.0000000  0.7500002  0.0175
  max GT boxes overlapped by one prediction 1

E only LEVEL_2
  class      level      TP   FP    FN         AP        APH    APH_lit     AP_d=1     gap
  VEHICLE    LEVEL_1   400    0  1200  0.2500000  0.2500000  0.0000000  0.2500000  0.0075
  VEHICLE    LEVEL_2   400    0  1200  0.2500000  0.2500000  0.0000000  0.2500000  0.0063
  PEDESTRIAN LEVEL_1   200    0   600  0.2500000  0.2500000  0.0000000  0.2500000  0.0125
  PEDESTRIAN LEVEL_2   200    0   600  0.2499999  0.2499999  0.0000000  0.2499999  0.0113
  max GT boxes overlapped by one prediction 1

Checks and hypotheses
C1 max |diff| 5.7e-09 pass True | C2 max |diff| 4.8e-08 pass True | C3 pass True | C4 pass True
C5 A: max |1-AP|, |1-APH| 1.8e-07 pass True
H1 B: max |AP_B-AP_A| 0.0e+00, max |APH_B-0.5| 8.9e-08 held True
H2 literal: max APH_lit(A) 0.0e+00; APH_lit(B) range 0.5011449-0.5030870 held True
H3 C vehicle LEVEL_1: AP 0.8845657 APH 0.8497984 APH/AP 0.960696 bound 0.888373 | (c) held True
H3 C vehicle LEVEL_2: AP 0.8712189 APH 0.8370197 APH/AP 0.960746 bound 0.888373 | (c) held True
H3 C pedestrian LEVEL_1: AP 1.0000000 APH 0.9566659 APH/AP 0.956666 bound 0.795632 | (c) held True
H3 C pedestrian LEVEL_2: AP 0.9999998 APH 0.9565599 APH/AP 0.956560 bound 0.795632 | (c) held True
H3 (a) vehicle max AP_C 0.8845657 held True | (b) pedestrian max |1-AP_C| 1.8e-07 held True
H4 delta=1: max recall gap 0.0233, max |AP(d=1)-AP(d=0.05)| 0.0e+00 held True
H5 D/E: max deviation from 1, 0.75, 0.25 4.8e-07 held True
H6 paper example: code 0.0000000 (delta 0.05), 0.0000000 (delta 1), paper 0.05; held True

Port checks. C1 to C4 held. The AP tests agree with the expected values to within $5.7 \times 10^{-9}$ and the heading tests to within $4.8 \times 10^{-8}$, both far inside the tests’ $10^{-6}$; the heading accuracy of a backward heading prints as $4.8 \times 10^{-8}$ rather than 0, as single precision allows. All twelve IsTP, IsFP and IsFN expectations are met, including the prediction in a No Label Zone that is not a false positive, and the matcher reproduces the quantized weights and the Hungarian assignment of the official tests. C6 held: 1,600 and 800 boxes with 400 and 200 at LEVEL_2, no box without points, no score on a cutoff, no prediction overlapping more than one box, B’s true positives at full weight $10^6$, and the repaired clipper gives IoU 1.000000000 for every box against its turned copy. C7 held.

What the levels and the heading weight do.

• A, exact boxes: C5 held. AP and APH are 1.0000000 at LEVEL_1 and 0.9999998 at LEVEL_2 for both classes, a largest deviation of $1.8 \times 10^{-7}$. By construction precision is 1 at every cutoff and recall runs from 0 to 1, so in exact arithmetic the trapezoid sum is exactly 1; the deviation is rounding in the single-precision trapezoid sum.

• B, half the headings reversed: H1 held. AP is the same as A’s to the last digit (difference 0), and APH is 0.5000000 at LEVEL_1 and 0.4999999 at LEVEL_2, at most $8.9 \times 10^{-8}$ from 0.5. AP cannot see a reversed heading, while APH halves the score. The footprint is identical, so the matching weight stays at $10^6$.

• Literal weight: H2 held. Taken literally, the paper’s weight gives the exact detector A an APH of 0.0000000 and B one of 0.5011449 to 0.5030870, so the reading ranks the detector with reversed headings far above the perfect one. The code’s weight ranks them the other way, 1 against 0.5. The literal column is also near 0 for C (0.034 to 0.048) and exactly 0 for D and E, whose headings are exact.

• C, heading noise of 10°: H3 held.

  • (a) Vehicle AP falls to 0.8845657 at LEVEL_1 and 0.8712189 at LEVEL_2: 110 of the 1,600 rotated vehicle boxes fall below IoU 0.7 and each becomes both a false positive and a missed object. Of those 110 missed boxes, 87 are LEVEL_1, which is why LEVEL_1 counts 87 false negatives and LEVEL_2 counts 110.
  • (b) Every pedestrian still matches (AP 1.0000000 and 0.9999998), although the largest pedestrian heading error is 36.79° against 20.09° for vehicles. The construction differs between the classes in two ways that can account for this, as our reading: pedestrians need IoU 0.5 rather than 0.7, and their footprints (0.7–1.0 by 0.6–0.9 m) are far less elongated than the vehicles’ (4.2–5.2 by 1.9–2.2 m), and aerial object detection’s second example shows elongated boxes losing IoU faster under an angle error. The output does not separate the two.
  • (c) APH is below AP everywhere. APH/AP is 0.960696 and 0.960746 for vehicles against a bound of 0.888373, and 0.956666 and 0.956560 for pedestrians against 0.795632.

  So heading noise costs vehicles twice, once through IoU in AP and again through the weight in APH, and costs pedestrians only through the weight.

• Recall steps: H4 held. The largest step between consecutive cutoffs is 0.0233, below $\delta = 0.05$, and AP with $\delta = 1$ equals AP with gap filling exactly in every row. With 101 cutoffs and continuous scores the filling has nothing to fill. It matters for sparse curves such as the unit tests’ third and fourth cases, where the same two points give 0.06125 with filling and 0.08 without.

• D and E: H5 held. The largest deviation from the predicted 1, 0.75 and 0.25 is $4.8 \times 10^{-7}$. D loses nothing at LEVEL_1, because LEVEL_2 boxes it misses are not false negatives there. E finds no LEVEL_1 object at all and still scores 0.2500000 at LEVEL_1, because its 400 vehicle and 200 pedestrian hits on LEVEL_2 boxes count as true positives at LEVEL_1 while the boxes it misses there are the 1,200 and 600 LEVEL_1 boxes. A LEVEL_1 number is therefore a score over all predictions in which only the misses are restricted to LEVEL_1 boxes, and it can rise with hits on hard boxes.

• The paper’s example: H6 held. The code gives 0.0000000 with and without gap filling, where the paper states 0.05.

The example tests our port on synthetic boxes with simple error models. It shows what the code computes, not how real detectors trade localization against heading.

License

As of November 2026 the terms page publishes the “Waymo Dataset License Agreement for Non-Commercial Use (March 2025)”. Its conditions include:

• Acceptance. It applies “By downloading or using the Dataset”. Access goes through a Google sign-in on the dataset site.
• Grant. A non-exclusive, royalty-free, personal licence to use, reproduce and modify the dataset “for Non-commercial Purposes only”, defined as research, teaching, scientific publication and personal experimentation, including benchmarking for publication.
• Derivative IP. The agreement defines derivative IP to include models, weights and algorithms trained or tuned with the dataset, and requires a set attribution in any derivative IP.
• Sharing. Small extracts may illustrate research publications; the dataset itself may be passed on only to those who have registered at waymo.com/open and agreed to the terms; trained models may be published under the agreement’s non-commercial conditions, with a notice and a copy of the agreement.
• Restrictions. The dataset and models trained on it may not be used in the operation of a vehicle, in production systems or for other primarily commercial purposes. A limited non-assert clause covers claims against Waymo over the user’s derivative IP.
• Commercial use requires contacting Waymo.

The repository README’s metadata still names the “Waymo Dataset License Agreement for Non-Commercial Use (August 2019)” and links the same terms page. The two dates are reported here as printed.

The code repository is licensed separately. Its LICENSE file applies the Apache License 2.0 to everything except the src/waymo_open_dataset/wdl_limited folder, which carries a BSD 3-clause licence with a limited patent licence tied to use with the dataset (README); that folder was not opened for this article. The terms page states that Waymo takes reasonable care to remove or hide personally identifiable information, including faces and licence plates, and offers a form for removal requests.

Limitations and Bias

• Three American regions. All scenes come from San Francisco, Mountain View and Phoenix. The paper’s domain experiment, on LEVEL_2 3D APH on the validation set, shows the gap between them: a vehicle detector trained on San Francisco scored 36.7 on the suburban domain against 44.0 for one trained there, and a pedestrian detector trained on the suburbs scored 29.7 in San Francisco against 47.6 (Table 9). Autonomous driving perception notes that the paper’s text gives larger losses than these table differences.
• Restricted range. The released LiDAR data stop at 75 m for the top sensor and 20 m for the others (Table 2), and LiDAR annotations cover objects within 75 m (§4).
• What LEVEL_1 means. As the example shows, LEVEL_1 counts hits on LEVEL_2 boxes, so it is not a score on easy objects alone.
• A linear heading weight. APH treats a 90° error as half a hit. A reversed heading scores 0, but AP does not change, so a detector that reverses headings can still rank high on AP.
• Licence limits. The non-commercial terms extend to models trained on the data, which limits their use in products.
• Hidden numbers. Totals for later versions are on pages behind sign-in, and the paper’s own 2D counts disagree, as noted above.

Common Uses

• 3D detection from LiDAR. The paper’s baseline was a re-implemented PointPillars, scored by AP and APH at both levels and by range.
• Domain adaptation and data scaling. The city split supports experiments on transfer between regions (§5.3), and the paper studies how the PointPillars baseline improves with more training data (§5.4).
• Camera, fusion and tracking. Camera-only 3D detection has its own challenge and metrics, and sensor fusion work compares methods on the dataset. 3D and 2D tracking are scored with MOTA (§4.2).
• Motion forecasting and simulation. The Motion dataset serves trajectory prediction and the sim-agents and scenario-generation challenges (motion forecasting).

Historical Milestones

• 2019. The citation on the terms page gives the year 2019 and the README names the licence version of August 2019; the release history (LOG.md) starts only in March 2021.

• CVPR 2020. Sun et al. published the paper with baselines on the hidden test set (Table 6). The 3D results for the PointPillars baseline, LEVEL_1/LEVEL_2:

  Class	3D AP	3D APH	BEV AP	BEV APH
  Vehicle	63.3/55.6	62.8/55.1	80.1/71.9	79.1/71.0
  Pedestrian	62.1/55.9	50.2/45.1	70.0/63.8	56.1/51.1

  At LEVEL_1 the gap between 3D AP and APH is 0.5 for vehicles and 11.9 for pedestrians. By range, vehicle 3D APH at LEVEL_1 fell from 81.9 within 30 m to 34.9 beyond 50 m. The 2D baseline, a Faster R-CNN with a ResNet-101, reached 63.7/53.3 AP on vehicles and 55.8/52.7 on pedestrians (§5.1).

• Data size. Section 5.4 reports LEVEL_2 AP/APH for the baseline trained on 10% to 100% of the training scenes, rising for vehicles from 29.7/28.9 to 49.8/49.4 (Table 10). The text says these models were evaluated on the test set, while Table 10’s caption says the validation set; the caption is reported here.

• March 2021. The Motion dataset was added (LOG.md).

• 2022. Perception v1.3.0 to v1.4.1 added segmentation, keypoints, camera-only test segments and video panoptic labels.

• March 2023. Perception v2.0.0 came out in the modular format.

• March–April 2025. The End-to-End Driving dataset was released, and the licence agreement dated March 2025 is the one on the terms page as of November 2026.

Related Datasets

• KITTI. The earlier multimodal benchmark, with 22 scenes and one LiDAR in the paper’s Table 1, and three difficulty levels on front-camera labels (KITTI).
• nuScenes. About the same number of scenes (1,000) with radar as well, annotated at 2 Hz; detection is scored by NDS with centre-distance matching rather than APH (nuScenes).
• Argoverse. Its tracking set in Table 1 has 113 scenes with two LiDARs and maps.
• Waymo Open Motion Dataset. The family’s motion member, described by Ettinger et al. (2021) and covered in motion forecasting. The repository README cites that paper with pages 9710–9719, while the published motion forecasting article gives 9690–9699; the difference is not resolved here.

Resources

• Waymo Open Dataset: the official site, with data access (Google sign-in), challenges and leaderboards for current results.
• waymo-open-dataset repository: data formats, tutorials and the official metrics code.
• Terms: the licence agreement, challenge rules, citation and privacy information quoted above.