Autonomous Driving Perception

Autonomous driving perception turns what a vehicle’s sensors measure into a description of the scene that a planner can act on: where the road and its lanes are, where the vehicle itself is on a map, which vehicles, cyclists and pedestrians are around it, where they are in 3D, how fast they move, and where they are likely to go next. Nearly every major task of computer vision appears in it, including object detection, multi-object tracking, semantic segmentation, depth estimation and SLAM, but under conditions few other applications combine: the output must be metric and three-dimensional, cover the whole surroundings, reach far enough to stop at speed, arrive within a fraction of a second, and hold up at night, in rain and for objects the training data rarely shows.

This article is the overview of that stack. It describes the problem and its history, from Ernst Dickmanns’s vision-guided van in the 1980s through the DARPA challenges to learned 3D detectors, and then each component in a paragraph or two with links: sensor fusion, bird’s-eye-view perception, lane detection, mapping and localization, and prediction. A worked example turns stopping distances into the range and resolution a sensor must deliver.

Problem

Driving systems have mostly been built as a chain of modules. Stanley, the vehicle that won the 2005 DARPA Grand Challenge, had a perception layer that mapped sensor data into internal models (the vehicle’s state and maps of the terrain from lasers, camera and radar), followed by a control layer whose path planner set the trajectory from those models (Thrun et al., 2006). Hu et al. (2023) still describe modern systems as modular tasks in sequence: perception, prediction and planning. Perception’s part of the chain produces several outputs at once:

• Objects. Every relevant road user as a 3D box with a class, position, size and heading, and, on nuScenes, a velocity and attributes such as whether a pedestrian is moving (Caesar et al., 2020).
• Tracks. The same objects linked over time, so that each has an identity and a motion history.
• The static scene. Drivable area, lane boundaries and centrelines, crossings and traffic controls, either detected online or read from a stored map.
• The ego pose. The vehicle’s own position and orientation, in a map or in a world frame, precise enough to place it within its lane.
• Predictions. The likely future paths of the other road users, which the planner needs as much as their present positions.

Three requirements separate this from perception for a photo or a video. The first is range at speed. A vehicle needs to see far enough ahead to stop or evade, and that distance grows with the square of speed. Stanley’s laser scanners detected obstacles to about 22 m, which its team found sufficient for speeds up to 25 mph (40 km/h); since they estimated the course required 35 mph, they added a camera-based terrain classifier that reached beyond 70 m (Thrun et al., 2006). The example below works through this arithmetic. The second is latency: every 100 ms between a measurement and the action taken on it is 1.4 m of travel at 50 km/h. The third is coverage, in space and in conditions. nuScenes was designed around a full sensor suite with a 360° field of view (Caesar et al., 2020), and its scenes were chosen to include night, rain and dense urban traffic.

What a system must perceive also depends on the task it is designed for. Janai et al. (2020) describe the classification of driving automation that the Society of Automotive Engineers (SAE) first released in 2014: six levels, from 0 (no autonomy) to 5 (full autonomy).

History

Dickmanns and VaMoRs. Janai et al. (2020) date the first self-driving prototypes that did not rely on infrastructure built into the road to 1986, with Ernst Dickmanns’s team at the Bundeswehr University Munich and Carnegie Mellon University’s Navlab team as the pioneers. Dickmanns’s own records describe the test vehicle VaMoRs, a 5-tonne Daimler-Benz van equipped from 1985 with a steering motor, braking actuation and a camera platform on the dashboard that could turn on two axes. In 1986 it made its first fully autonomous runs, with both steering and speed guided by vision, on a skidpan at Daimler-Benz at up to 36 km/h; in 1987 it drove on a free stretch of Autobahn at up to 96 km/h, the van’s top speed. The image processing ran on Intel 80x86 processors at 5 MHz (Dickmanns, dyna-vision). The approach was modular and model-based: as Janai et al. summarize it, a model of the vehicle and the road was used to estimate their state continuously and to control the vehicle.

The work continued within the European PROMETHEUS project. In November 1995 the team’s Mercedes sedan VaMP drove from Munich to Odense in Denmark; Dickmanns’s account gives more than 1,600 km of autonomous lateral and longitudinal guidance from black-and-white images, at speeds of 60 to 180 km/h, with a longest uninterrupted stretch of about 160 km and more than 400 lane changes performed autonomously after the safety driver triggered them (Dickmanns, dyna-vision). Janai et al. (2020) give the top speed as 175 km/h; both sources put the autonomous share of the distance at about 95%.

Navlab and ALVINN. At Carnegie Mellon, ALVINN (Pomerleau, 1988) took the opposite route: a three-layer neural network trained by backpropagation to map a 30 × 32 video image and an 8 × 32 laser range image directly to a steering direction. Trained on 1,200 simulated road images, it drove the Navlab, a modified Chevrolet van carrying three Sun computers, at 0.5 m/s along a 400 m wooded path, comparable, the author reported, to the best conventional vision-based system at CMU under the conditions tested. Janai et al. (2020) describe it as an early demonstration of imitation learning optimized end to end, and record that in 1995 the Navlab team drove from Washington, D.C., to San Diego 98% autonomously, with speed controlled by the driver.

The DARPA challenges. DARPA launched the Grand Challenge in 2003 (Thrun et al., 2006). On 13 March 2004, 15 vehicles started a 142-mile course through the Mojave Desert; none finished, and the best covered 7.5 miles (DARPA, 2014; Thrun et al., 2006). Janai et al. (2020) give the route as 240 km, longer than the 142 miles (about 229 km) that DARPA and Thrun et al. state. On 8 October 2005, of 195 registered teams, 23 raced and 5 finished a course DARPA gives as 132 miles; Stanford’s Stanley won in 6 hours 53 minutes (Thrun et al., 2006; DARPA, 2014).

Stanley shows what perception meant then. Five laser range finders on the roof scanned the terrain ahead to 25 m, a colour camera looked further, and two 24 GHz radars covered the front to 200 m (Thrun et al., 2006). Its camera had no hand-labelled training data. Instead, the system projected a patch of ground that the lasers had just classified as drivable into the image, used those pixels to update a mixture of Gaussians modelling the colour of drivable surface, and classified the rest of the image with it, adapting as the terrain changed. Because colour changes do not always mean obstacles, the vision map was used only to choose the speed: when it found no drivable corridor within 40 m, Stanley slowed to the speed at which the lasers sufficed. It is a form of self-supervision: one sensor labels training data for another.

The 2007 Urban Challenge moved the problem into a staged city at Victorville, California, where vehicles had to obey traffic rules among other moving vehicles; six of 11 teams completed the course, and Carnegie Mellon’s Tartan Racing team won (DARPA, 2014). Janai et al. (2020) note that most successful teams relied on the new spinning multi-beam LiDAR, which gave precise depth all around the vehicle, and Chang et al. (2019) point out that the three winning entries relied on a map that DARPA supplied. Both, dense 3D sensing and prior maps, became standard parts of the stack.

Benchmarks and deep learning. In 2012, KITTI put a sensor-equipped car’s data into public benchmarks for stereo, optical flow, visual odometry and object detection with orientation estimation, built on 3D object annotations; its authors found that methods ranking high on laboratory datasets such as Middlebury performed below average on their street scenes (Geiger et al., 2012). Learned 3D detectors on LiDAR point clouds followed, such as VoxelNet (Zhou & Tuzel, 2018), together with a second generation of datasets, released or first described in 2019, that recorded the full surroundings of the vehicle with several sensor types: nuScenes, the Waymo Open Dataset and Argoverse. Since about 2020, camera-only methods that build a bird’s-eye-view representation, online construction of map elements, and networks that train perception, prediction and planning together have extended the stack, each described below.

Visual Data

Driving vehicles carry three kinds of exteroceptive sensor, complementary in what they measure, and the LiDAR article compares them in detail.

• Cameras. Digital cameras give colour and texture, which identify traffic lights, signs, lane paint and object classes, at a resolution that keeps distant objects several pixels tall; depth they give only through stereo or inference. Surround rigs use several cameras: nuScenes has six at 1600 × 900 pixels and 12 Hz, the front and side cameras with a 70° field of view and the rear camera with 110° (Caesar et al., 2020); Argoverse has seven ring cameras at 1920 × 1200 and 30 Hz plus a forward stereo pair (Chang et al., 2019). Waymo’s cameras use rolling shutters, so each image row is exposed at a different time (Sun et al., 2020).
• LiDAR. Metric range along each laser beam, at night as by day, but sparse and with little appearance. nuScenes’s single 32-beam sensor spins at 20 Hz with a range of up to 70 m; Argoverse uses two 32-beam sensors with a range of 200 m; the Waymo Open Dataset combines five LiDARs, one on the roof, whose data are released up to 75 m, and four around the vehicle, released up to 20 m (Caesar et al., 2020; Chang et al., 2019; Sun et al., 2020).
• Radar. Range and radial velocity from radio echoes, with long reach and robustness to weather but coarse angular resolution. nuScenes has five 77 GHz radars with up to 250 m range (Caesar et al., 2020). The sensor fusion article describes how radar data are processed and combined with images.
• Position and motion sensors. GNSS receivers, inertial measurement units and wheel odometry give the ego motion; nuScenes lists RTK positioning to 20 mm in its sensor table, but its authors localize against a LiDAR map instead, because GPS-based localization is vulnerable to outages, more so in dense cities (Caesar et al., 2020).
• Maps. Stored maps act as a further input: lane geometry and connectivity, drivable area and ground height (Chang et al., 2019).

Vision Tasks and Methods

3D object detection

Detection in driving means 3D boxes in the vehicle’s frame, usually seven numbers (centre, length, width, height and heading) plus a class and, increasingly, a velocity. Before learned detectors, as Lang et al. (2019) summarize it, LiDAR pipelines removed the background, clustered the remaining points in space and time, and classified the clusters. VoxelNet (Zhou & Tuzel, 2018) replaced the hand-made encodings of the point cloud, such as bird’s-eye-view projections, with a learned one: it divides the cloud into voxels, encodes the points of each voxel with a small network, and passes the volume to a region proposal network, trained end to end. PointPillars (Lang et al., 2019) organizes the points into vertical columns instead, so that the encoded grid is a 2D pseudo-image that ordinary 2D convolutions can process; on KITTI the authors reported 62 Hz, two to four times faster than previous methods. CenterPoint (Yin et al., 2021) argued that boxes are an awkward target for objects that can face any direction, and detects objects as centre points on a bird’s-eye-view heatmap, regressing size, heading and velocity at each peak and refining them in a second stage. With a single model it reported 65.5 NDS for detection and 63.8 AMOTA for tracking on the nuScenes test set.

Camera-only 3D detection must recover depth, which a single image does not contain. The current approaches lift image features into a bird’s-eye-view grid or let 3D queries gather features from the images; the bird’s-eye-view perception article describes them and their results. Combining LiDAR, camera and radar detections, at the level of points, features, a shared grid or finished boxes, is the subject of sensor fusion for autonomous driving. The general methods of 2D detection, from anchors to transformers, belong to the object detection article.

Tracking

Tracking turns per-frame detections into objects with identities and velocities, which prediction and planning need. Most driving trackers follow tracking by detection: a motion model such as a Kalman filter predicts each track, new detections are associated with the predictions, and unmatched tracks are kept for a few frames before being deleted, as in SORT. Working in 3D and in the vehicle’s frame plausibly simplifies association compared with image-space tracking, since objects rarely overlap when seen from above (our reasoning). CenterPoint shows how far this can be simplified when the detector predicts velocity: each detected centre is moved back by its velocity to the previous frame and matched greedily to the nearest track, and unmatched tracks survive for three frames (Yin et al., 2021). Maps help too: Argoverse’s baseline tracker used lane direction, drivable area and ground height to improve 3D tracking (Chang et al., 2019). The general methods, from data association to track management, are in multi-object tracking.

Fusion and the bird’s-eye view

Two components hold the stack together, and each has its own article. Sensor fusion handles the geometry and timing that make measurements comparable (calibration between sensors, synchronization, compensation for the vehicle’s own motion) and chooses where the streams meet; its article describes point-level, feature-level, bird’s-eye-view and late fusion, and radar–camera fusion. Bird’s-eye-view perception turns camera images into a metric top-down grid around the vehicle, by inverse perspective mapping, depth-based lifting or attention; that grid has become the common frame in which detection, map segmentation, tracking and planning work and in which other sensors are fused.

Lanes and the road

The vehicle must know where it may drive. Lane detection finds lane markings and boundaries in camera images and returns them as curves; its article follows the field from edge detection, Hough voting and curve fitting to segmentation networks and lane-level detectors, and describes the benchmarks on which they are compared. Drivable area and other road surfaces can be segmented in the image or directly in the bird’s-eye view, and map elements such as lane dividers and crossings can be detected online as vectors (Li et al., 2022), described under mapping below. Traffic lights and signs are recognized from camera images, since only cameras see colour.

Mapping and localization

Many research systems have used a prior map and localized the vehicle in it. Satellite positioning alone is not enough: Levinson et al. (2007) found that centimetre accuracy could not be achieved reliably in cities with GPS-based inertial systems. Their method built, offline, a map of the ground’s infrared reflectivity from LiDAR scans at 5 cm resolution, aligned with SLAM-like relaxation and with other vehicles removed by keeping only the flat road surface, and localized the moving vehicle by correlating new LiDAR scans with that map in a particle filter, with accuracy around 10 cm. nuScenes localized its data the same way in principle, against a LiDAR map built offline with Monte Carlo localization, to within 10 cm (Caesar et al., 2020).

High-definition (HD) maps add semantics to geometry. Argoverse released three: a vector map of lane centrelines as a graph of lane segments with their predecessors, successors, turn directions and traffic-control flags; a 1 m raster of drivable area; and a 1 m raster of ground height, which helps remove ground points from LiDAR data (Chang et al., 2019). Such maps are expensive to keep current. Li et al. (2022) describe the conventional pipeline as capturing point clouds, building a globally consistent map with SLAM and annotating it by hand, which takes much human effort; their HDMapNet instead predicts vectorized map elements around the vehicle in the bird’s-eye view from its cameras, its LiDAR or both at run time. Where no map is available, visual odometry and SLAM estimate the vehicle’s motion and build a map as it drives.

Prediction

A planner must know where other road users will be, not only where they are. Argoverse defines motion forecasting as observing 2 s of an object’s bird’s-eye-view track at 10 Hz and predicting the next 1 to 3 s (Chang et al., 2019). Because the future is uncertain (a car at an intersection may turn either way), methods output several trajectories, and Argoverse scores the best of the top $K$ by average displacement error (ADE) and final displacement error (FDE). In its baselines over a 3 s horizon, a constant-velocity model had an ADE of 3.55 m and an FDE of 7.89 m, and an LSTM encoder–decoder using the lane graph had 2.25 m and 4.67 m; the map-based baselines generated on average 5.9 hypotheses from the lane graph, while the others made a single prediction (Chang et al., 2019, Table 3). The authors conclude that the vector map matters most for the longer horizon, while adding the surrounding vehicles’ motion did not help their LSTM.

Joint and end-to-end models

A chain of modules accumulates errors, and each module is trained for its own metric rather than for driving. Hu et al. (2023) argue for designing the whole chain around planning. Their UniAD encodes multi-camera images into a bird’s-eye-view feature map with BEVFormer’s encoder, then runs transformer modules for tracking, online mapping, motion forecasting and future occupancy, connected by queries, and ends with a planner that predicts the vehicle’s waypoints while avoiding the predicted occupied regions; it was trained and evaluated on nuScenes. ALVINN’s direct mapping from pixels to steering was the earliest form of this idea, and end-to-end driving, in which a single network maps sensor data to driving decisions, is now a research area of its own.

Example

The script below sizes the perception range a vehicle needs. A simple stopping model assumes that the vehicle keeps its speed $v$ for a reaction time $T$ (perception, decision and actuation together) and then brakes at a constant deceleration $a$, so an obstacle must be detected at a distance of at least

$$d = vT + \frac{v^2}{2a}.$$

The deceleration of 5.5 m/s² is VaMoRs’s listed maximum (Dickmanns, dyna-vision); 3.0 m/s², a gentler stop such as a wet road might allow, and the reaction times of 0.2 to 1.0 s are assumptions chosen to span a range, not measured values. For each speed the script then asks what a nuScenes-like sensor set would see of a pedestrian of an assumed 1.75 m height at the required distance: its height in pixels in the nuScenes front camera (1600 pixels across 70°) and in the Waymo front camera (1920 pixels across ±25.2°), both treated as ideal pinhole cameras, and the number of nuScenes LiDAR beams that would cross it, assuming its 32 beams are spread evenly over the 40° vertical field (the paper gives the field, not the spacing). Speeds of 25 and 35 mph are included because Stanley’s team planned with them. It runs with NumPy 2.3.5.

import numpy as np

# Sensor geometry from the dataset papers (Caesar et al., 2020; Sun et al., 2020).
f_nus = 800 / np.tan(np.radians(35.0))     # nuScenes front camera: 1600 px wide, 70 deg HFOV
f_way = 960 / np.tan(np.radians(25.2))     # Waymo front camera: 1920 px wide, +-25.2 deg HFOV
beam = np.radians(40.0 / 31)               # nuScenes LiDAR: 32 beams over -30..+10 deg,
                                           # assumed evenly spaced
LIDAR_MAX = 70.0                           # nuScenes LiDAR range (m)
PED_H = 1.75                               # pedestrian height (m), assumed

# Stopping model: the vehicle keeps its speed for the reaction time T (perception,
# decision and actuation), then brakes at a constant deceleration a.
def required_range(v, T, a):
    return v * T + v**2 / (2 * a)

speeds_kmh = sorted([30, 50, 80, 100, 130, 25 * 1.609344, 35 * 1.609344])  # incl. 25, 35 mph
# (T in s, a in m/s^2): 5.5 m/s^2 is VaMoRs's listed maximum deceleration
# (Dickmanns, dyna-vision); the reaction times and 3.0 m/s^2 are assumed.
cases = [(0.2, 5.5), (0.5, 5.5), (0.5, 3.0), (1.0, 3.0)]

print(f"focal lengths: nuScenes {f_nus:.0f} px, Waymo {f_way:.0f} px")
print("speed   required range (m) for T [s] / a [m/s^2]    pedestrian at the 0.5/3.0 range")
print("km/h  " + "".join(f"  {T:.1f}/{a:.1f}" for T, a in cases)
      + "    nuS px  Waymo px  LiDAR rows")
for kmh in speeds_kmh:
    v = kmh / 3.6
    d = [required_range(v, T, a) for T, a in cases]
    r = d[2]
    rows = (PED_H / r) / beam
    lidar = f"{rows:5.1f}" if r <= LIDAR_MAX else "  out of range"
    print(f"{kmh:5.1f} " + "".join(f"{x:9.1f}" for x in d)
          + f"   {f_nus * PED_H / r:6.1f}   {f_way * PED_H / r:6.1f}    {lidar}")

# Distance travelled during each 0.1 s of latency, and per sensor frame.
for kmh in (50, 130):
    v = kmh / 3.6
    print(f"{kmh} km/h: {v * 0.1:.2f} m per 100 ms; {v / 20:.2f} m per 20 Hz LiDAR sweep,"
          f" {v / 12:.2f} m per 12 Hz camera frame")

# Range at which a pedestrian is 25 px tall (the minimum box height of KITTI's
# moderate level), and at which it spans two LiDAR beams.
print(f"pedestrian 25 px tall at: nuScenes {f_nus * PED_H / 25:.0f} m, "
      f"Waymo {f_way * PED_H / 25:.0f} m; spans 2 LiDAR beams at {PED_H / (2 * beam):.0f} m")

focal lengths: nuScenes 1143 px, Waymo 2040 px
speed   required range (m) for T [s] / a [m/s^2]    pedestrian at the 0.5/3.0 range
km/h    0.2/5.5  0.5/5.5  0.5/3.0  1.0/3.0    nuS px  Waymo px  LiDAR rows
 30.0       8.0     10.5     15.7     19.9    127.0    226.8      4.9
 40.2      13.6     16.9     26.4     32.0     75.7    135.2      2.9
 50.0      20.3     24.5     39.1     46.0     51.1     91.3      2.0
 56.3      25.4     30.1     48.6     56.4     41.1     73.4      1.6
 80.0      49.3     56.0     93.4    104.5     21.4     38.2      out of range
100.0      75.7     84.0    142.5    156.4     14.0     25.1      out of range
130.0     125.8    136.6    235.4    253.4      8.5     15.2      out of range
50 km/h: 1.39 m per 100 ms; 0.69 m per 20 Hz LiDAR sweep, 1.16 m per 12 Hz camera frame
130 km/h: 3.61 m per 100 ms; 1.81 m per 20 Hz LiDAR sweep, 3.01 m per 12 Hz camera frame
pedestrian 25 px tall at: nuScenes 80 m, Waymo 143 m; spans 2 LiDAR beams at 39 m

The quadratic term dominates. At urban speeds every assumption gives a required range of a few tens of metres, well inside the LiDAR’s 70 m, but at 130 km/h the braking distance alone is over 100 m even with VaMoRs’s deceleration, and with the gentler assumptions the vehicle must recognize the pedestrian at 235 to 253 m. The reaction time matters less, though not negligibly: each extra 100 ms costs 3.6 m at 130 km/h, a little more than the 3.0 m the vehicle covers between two camera frames at 12 Hz.

What the sensors deliver shrinks as the required range grows. Under the 0.5 s, 3.0 m/s² assumption, the pedestrian at the distance needed for 50 km/h is about 51 pixels tall in the nuScenes front camera and is crossed by two LiDAR beams; at 80 km/h it is beyond the LiDAR’s stated range and 21 pixels tall, below the 25-pixel minimum height of KITTI’s moderate difficulty level (applied here only as a reference point); at 130 km/h it is 8.5 pixels. The narrower Waymo front camera keeps it 1.8 times taller; in our reading, this is a reason to give long-range perception narrow, high-resolution cameras, at the cost of more cameras for the same coverage. The same arithmetic, in our reading, explains Stanley’s design: the required ranges for 25 mph (40.2 km/h), 14 to 32 m, straddle its 22 m laser reach, while those for 35 mph, 25 to 56 m, needed the camera’s longer reach. The example ignores everything that makes real detection harder at range, such as occlusion, lower contrast and fewer LiDAR returns from dark surfaces, so its numbers are optimistic bounds, not sensor requirements.

Datasets

Public datasets defined much of the field’s progress, and each reflects its sensors and cities. KITTI has its own article; the other three are summarized here from their papers.

Dataset	Year	Where	Sensors	Size and labels
KITTI	2012	Karlsruhe	2 grayscale and 2 colour cameras (front), 1 Velodyne 64-beam LiDAR, GPS/IMU	Separate benchmarks; more than 200k 3D object annotations
nuScenes	2019	Boston, Singapore	6 cameras, 1 LiDAR (32 beams), 5 radars, GPS/IMU	1,000 scenes of 20 s; 3D boxes for 23 classes at 2 Hz; map layers
Waymo Open Dataset	2019	San Francisco, Phoenix, Mountain View	5 cameras (front and sides), 5 LiDARs	1,150 scenes of 20 s; about 12M 3D LiDAR boxes
Argoverse	2019	Pittsburgh, Miami	7 ring cameras, 2 stereo cameras, 2 LiDARs (32 beams)	100 tracking segments with 10,572 tracked objects; 333,441 forecasting sequences; vector maps

• KITTI (Geiger et al., 2012) recorded around Karlsruhe with front-facing cameras, a spinning LiDAR and a GPS/IMU system; its article describes the benchmarks, metrics and their limits.
• nuScenes (Caesar et al., 2020) was, in its authors’ words, the first dataset to carry the full autonomous-vehicle sensor suite, with 360° coverage by cameras, LiDAR and radar. Its 1,000 scenes were selected from 15 hours of driving, with rain in 19.4% and night in 11.6% of the annotated keyframes; its fine-grained classes are severely imbalanced, at about 1:10,000 between the rarest and most common. Faces and licence plates are blurred, found with detectors set for very high recall (supplementary material of the arXiv version).
• The Waymo Open Dataset (Sun et al., 2020) has 798 training, 202 validation and 150 test scenes, the test scenes taken from a geographically separate area so that generalization to unseen places can be measured. Its LiDAR data come as range images, with the first two returns of each pulse.
• Argoverse (Chang et al., 2019) was built around maps: about 290 km of mapped lanes with the vector, drivable-area and ground-height maps described above, a 3D tracking set annotated in 17 categories, and a forecasting benchmark mined from 1,006 hours of fleet driving for vehicles at intersections, turning, changing lanes or in dense traffic.

These datasets are also where camera-only, LiDAR and fused methods are compared; the sensor fusion and bird’s-eye-view perception articles list the results reported on them.

Challenges

• Range and small objects. Distant objects are a few pixels and a few LiDAR points, as the example shows. The Waymo baseline, a re-implemented PointPillars, illustrates the effect: its 3D vehicle APH on the test set fell from 81.9 within 30 m to 34.9 beyond 50 m (LEVEL 1; Sun et al., 2020, Table 6).
• Weather, light and sensor limits. Cameras lose contrast at night and in glare, LiDAR returns are scattered by fog and rain, as the LiDAR article describes, and radar is coarse; no single sensor covers every condition, which is the case for fusion.
• Domain shift. A model trained in one city transfers imperfectly to another. In the Waymo baselines (LEVEL 2 3D APH on the validation set), a vehicle detector trained on San Francisco scored 36.7 on the suburban domain against 44.0 for one trained there, and a pedestrian detector trained only on the suburbs scored 29.7 in San Francisco against 47.6 for one trained there (Sun et al., 2020, Table 9). The text gives the two losses as 8.0 and 19.8 APH, larger than the table’s differences (7.3 and 17.9), and suggests that the pedestrian gap may come from the few pedestrians in the suburban data.
• The long tail. Rare classes and situations (unusual vehicles, animals, debris, construction zones) are poorly represented in any dataset; nuScenes’s class imbalance is one measure of it.
• Occlusion and prediction. Objects hidden behind others must be inferred, and other road users’ intentions are uncertain, which is why forecasting is evaluated with several hypotheses.
• Latency and compute. Every model runs on the vehicle within a fixed budget, from sensor capture to output; real-time vision pipelines and edge inference cover the engineering.
• Maps that change. A prior map is only as good as its last update, and building one takes human effort (Li et al., 2022).

Evaluation

Perception components are evaluated on recorded data against human or sensor-derived ground truth, each with its own metric:

• 3D detection. KITTI scores average precision with 3D or bird’s-eye-view IoU thresholds, in three difficulty levels. nuScenes matches by centre distance on the ground and combines mAP with errors of position, size, heading, velocity and attributes in the nuScenes detection score (NDS), defined in the bird’s-eye-view perception article. Waymo reports AP and APH, which weights true positives by heading accuracy, at two difficulty levels, LEVEL 2 adding objects that labellers marked as hard or that have at most five LiDAR points, and broken down by range (Sun et al., 2020).
• Tracking. KITTI ranks trackers by HOTA since 2021; nuScenes uses AMOTA, which averages a recall-adjusted MOTA over recall thresholds, because plain MOTA was often zero on its harder data (Caesar et al., 2020).
• Forecasting. ADE and FDE of the best of the top $K$ predictions (Chang et al., 2019).
• Maps. HDMapNet introduced semantic-level and instance-level metrics for predicted map elements (Li et al., 2022).

These metrics treat all errors alike, wherever they occur. Philion et al. (2020) argued that a detector that misses a parked car far away should not be penalized as much as one that misses a car just in front, and proposed a planner-centric metric, PKL: a learned planner predicts a distribution over the ego vehicle’s future positions once from the ground-truth objects and once from a detector’s output, and the metric is the KL divergence between the two, zero for a perfect detector. In scenes where this metric and standard metrics disagreed, human judges sided with the planner-centric metric 79% of the time. Joint models such as UniAD are evaluated on every task they perform, and on planning by the L2 error of the planned trajectory and a collision rate (Hu et al., 2023).

All of these are measurements on recorded benchmark data. They are not measurements of on-road safety, and the papers cited here make no safety claims from them.

Failure Modes and Risks

• Missed and late detections. An object detected too late is, for the planner, the same as one not detected; the example shows how quickly the margin shrinks with speed.
• Phantom obstacles. False detections cause unnecessary braking or swerving. Stanley’s team showed that pitching of the vehicle combined with small pose errors made a laser mapper without probabilistic treatment of the measurements see an obstacle on an empty road, large enough to force the vehicle off the road; their probabilistic analysis removed it (Thrun et al., 2006).
• Appearance that misleads. Stanley’s vision module labelled grass as non-drivable until the vehicle drove onto it, which is why its output controlled only speed, not steering (Thrun et al., 2006).
• Localization failure. GPS outages, as KITTI’s data show, and a map that no longer matches the road can both place the vehicle wrongly (Caesar et al., 2020; Li et al., 2022).
• Calibration and timing errors between sensors, which misplace fused measurements; the sensor fusion article measures their effect.
• Error propagation. In a modular chain, a missed detection becomes a missing track and a missing prediction, a concern that motivates joint models (Hu et al., 2023).
• Privacy. Driving data record bystanders’ faces and licence plates; both nuScenes and Argoverse blur them before release (Caesar et al., 2020, arXiv supplementary material; Chang et al., 2019).

Deployment Considerations

• Redundancy across sensors. Since each sensor fails differently, systems combine cameras, LiDAR and radar, and fused designs that keep working when one sensor fails are a research topic of their own, discussed in the sensor fusion article.
• Self-supervision from the vehicle’s own sensors. Stanley trained its camera classifier online from laser labels (Thrun et al., 2006), and camera networks that are trained with LiDAR depth as supervision, such as BEVDepth in the bird’s-eye-view perception article, follow the same idea.
• Maps as a dependency. HD maps give precise lanes and localization but must be built and maintained; online map construction reduces that dependency (Li et al., 2022).
• Calibration and time synchronization, checked in service, not only at the factory; see camera calibration and the sensor fusion article.
• Data from many places and conditions. The Waymo domain-gap results (Sun et al., 2020) suggest, in our reading, training and test data from every operating area, and dataset bias applies to driving data as to any other.
• Evaluation tied to the task. Benchmark scores rank methods, but what matters is the effect on driving decisions, as planner-centric metrics try to measure (Philion et al., 2020).