End-to-End Autonomous Driving

End-to-end autonomous driving trains one model to map a vehicle’s sensor data to what it should do next, either the low-level controls (steering, throttle and brake) or a short planned trajectory that a simple controller then follows. The alternative it is defined against is the modular stack described in autonomous driving perception: separately built and separately evaluated modules for perception, motion forecasting and planning, connected by hand-designed interfaces such as object lists and maps. CARLA’s authors call the problem sensorimotor control (Dosovitskiy et al., 2017); the most common training method, behaviour cloning, is supervised learning on recorded pairs of observations and the expert driver’s actions.

The idea is old. ALVINN steered a van along a path in 1988 with a three-layer network (Pomerleau, 1988), and the published perception article tells that history in a paragraph. This article covers what has been learned since about the concept itself: why imitation errors compound once the learned policy drives, how recovery data, high-level commands and privileged teachers address that, how recent models keep intermediate tasks such as detection and forecasting inside a network trained for planning, and why the field’s evaluation moved from comparing predictions with logged driving (open loop) to letting the model drive in simulation (closed loop), after several studies found that the two disagree. A worked example trains an image-to-steering policy on rendered roads and compares it with a hand-built detect-then-control pipeline in both settings.

Definition

A driving policy $\pi$ maps an observation $o_t$ (camera images, LiDAR, the vehicle’s speed and other measurements) and, usually, a navigation input $c_t$ (a route command or goal point) to an action $a_t$:

$$a_t = \pi_\theta(o_t, c_t).$$

The policy is end-to-end when its parameters $\theta$ are trained jointly, through one objective tied to driving, from the sensor input to the action. Two meanings are in use, and papers do not always say which they intend.

• Direct sensor-to-action learning. No intermediate output is defined or supervised. ALVINN, the conditional imitation models of Codevilla et al. (2018; the ICRA paper, told apart below from the ECCV 2018 study by Codevilla, López, Koltun and Dosovitskiy) and many CARLA agents are of this kind; Hu et al. (2023) call this the “tabula-rasa” design and argue that it is inadequate in safety guarantees and interpretability, especially in highly dynamic urban scenarios.
• Planning-oriented, jointly trained stacks. The network still contains detection, tracking, mapping and forecasting modules with their own losses, but they pass learned features and queries to each other and to a planning head, and the whole is trained together with the planner’s loss. UniAD (Hu et al., 2023) and VAD (Jiang et al., 2023) are examples. Li et al. (2024) describe end-to-end driving in this broader sense, as learning from raw sensor data to planning trajectories or driving commands.

What the term excludes is a stack whose modules are trained separately for their own metrics and joined only at inference. Multi-task networks that share a backbone between several heads sit between the two; Hu et al. (2023) list TransFuser among these multi-task designs.

Intuition

A modular stack decides in advance what the planner needs to know: a list of 3D boxes, tracks, lanes, a map. Each module is optimized for its own benchmark, and errors pass down the chain without the downstream module being able to ask for something better; Hu et al. (2023) list information loss, error accumulation and misaligned features as the risks. An end-to-end model lets the training signal of the driving task decide what to extract from the pixels. Pomerleau (1988) already put the argument this way: the data rather than the programmer determine which image features matter, and in his case the network developed road filters on roads of fixed width and edge detectors on roads of varying width. He also reported that it took half an hour of training to reach a level of road following that had taken months of manual algorithm development.

The price is paid in three places, and they recur through this article.

1. The training data come from someone else’s driving. A cloned policy is trained on states the expert visited. Once it drives, its own small errors take it to states the expert never visited, where its predictions are unconstrained, and the errors compound. Pomerleau (1988) noted that a network trained on a human’s driving must also be shown how to recover once a mistake has been made.
2. The input does not always determine the action. At a fork or an intersection, the camera image does not say whether to turn left or right. ALVINN’s outputs oscillated between the two directions (Pomerleau, 1988), and Codevilla et al. (2018) point out that the mapping from image to control is then no longer a function.
3. Learning finds shortcuts. If a simple input such as the current speed predicts the expert’s next action well, a network will use it, whether or not it is looking at the road. This makes some evaluations misleading, as the evaluation section shows.

Formal Definition

Behaviour cloning and its error bound

Given demonstrations $\mathcal{D} = \{(o_i, c_i, a_i)\}$ recorded while an expert $\pi^*$ drives, behaviour cloning minimizes a supervised loss over the expert’s distribution of states $d_{\pi^*}$:

$$\hat\pi = \arg\min_{\pi} \; \mathbb{E}_{s \sim d_{\pi^*}} \big[\ell(s, \pi)\big].$$

Ross, Gordon and Bagnell (2011) formalize why this is not enough: over a horizon of $T$ steps, a policy that errs with probability $\epsilon$ on the expert’s states can incur an extra cost of up to $T^2 \epsilon$, a bound they note is tight. The reason is that after one mistake the learner may see observations unlike any in the demonstrations; training on the states the learner itself reaches can bring the growth down to linear in $T$. Their DAgger algorithm does this by iteration: run the current policy, have the expert label the states it reaches, add them to the dataset and retrain. Visuomotor policy learning states the bounds with their conditions and treats DAgger and its experiments in depth; embodied AI summarizes it in a paragraph. For driving, the difficulty, in our reading, is that DAgger needs an expert who can be queried in any state the learner reaches, which a human safety driver in a real car cannot easily provide; the simulator experts and privileged teachers below fill that role.

What the policy outputs

Three output representations appear in the literature:

• Controls. Steering, throttle and brake directly, as in ALVINN (a steering curvature) and in conditional imitation learning (Codevilla et al., 2018).
• Waypoints. A sequence of future positions $\{w_t = (x_t, y_t)\}_{t=1}^{T}$ in the vehicle’s bird’s-eye-view frame, converted to controls by a separate controller. TransFuser predicts $T = 4$ waypoints with a recurrent decoder, trains them with an L1 loss against the expert’s waypoints, and drives them with two PID controllers, one lateral and one longitudinal (Prakash et al., 2021). Its authors report that an image-only version predicting waypoints performed nearly seven times better than CILRS, which predicts controls, and attribute this likely to the PID controller, to goal locations given in the same bird’s-eye-view frame as the waypoints, and to the recurrent decoder.
• Trajectories scored or refined against predicted scene structure. UniAD decodes waypoints from a planning query and then, at inference only, optimizes them with Newton’s method so that they stay close to the network’s prediction while moving away from cells that its occupancy module predicts as occupied (Hu et al., 2023). VAD instead adds three losses during training that penalize planned trajectories passing close to predicted agents, crossing predicted road boundaries or pointing against the nearest lane’s direction (Jiang et al., 2023).

Evaluation, formally

Open-loop evaluation feeds the model logged sensor data and compares its output with what the human or expert did next. On nuScenes, the common measures are the L2 distance between planned and logged ego positions at 1, 2 and 3 s and the rate at which the planned trajectory overlaps other objects’ boxes (Jiang et al., 2023; Li et al., 2024). The model’s output never changes the inputs it sees next.

Closed-loop evaluation lets the model drive: its action changes the vehicle’s state and therefore the next observation. The CARLA Leaderboard’s main metric for route $i$ is the driving score

$$\mathrm{DS}_i = R_i \, P_i,$$

where $R_i$ is the percentage of the route completed and $P_i \in [0, 1]$ is an infraction penalty; the global scores are arithmetic means over routes, so the global driving score is not the product of the global route completion and the global penalty (CARLA Leaderboard, evaluation pages). How $P_i$ is computed changed between versions, and the change is itself a lesson in evaluation; it is described below. nuPlan and NAVSIM combine multiplicative penalties with a weighted average of other subscores in the same spirit:

$$\mathrm{PDMS} = \Big(\prod_{m \in \{\mathrm{NC}, \mathrm{DAC}\}} \mathrm{score}_m\Big) \times \frac{\sum_{w \in \{\mathrm{EP}, \mathrm{TTC}, \mathrm{C}\}} \mathrm{weight}_w \, \mathrm{score}_w}{\sum_{w} \mathrm{weight}_w},$$

with no at-fault collision (NC) and drivable-area compliance (DAC) as penalties, and ego progress (EP), time to collision (TTC) and comfort (C) weighted 5, 5 and 2 (Dauner et al., 2024).

History

ALVINN (1988). The autonomous driving perception article gives ALVINN’s inputs, its simulated training set and its drive on the Navlab. Details that matter here: the network had 29 hidden units and 45 outputs for steering curvature, trained towards the curvature that would bring the vehicle to the road centre 7 m ahead; after 40 epochs it chose a curvature within two units of the correct one about 90% of the time on new simulated images; and when its laser range input was replaced by a constant obstacle-free image, its author saw no noticeable degradation in driving and concluded that the range finder’s contributions were not necessary for reasonable performance (Pomerleau, 1988).

Deep networks on real cars (2016). Bojarski et al. (2016, posted on arXiv and not read here) trained a convolutional network to steer from camera images; as Codevilla et al. (2018) describe the system, it recorded from a forward camera and two cameras shifted to the left and right, added those views and synthetically reprojected ones with adjusted steering labels to simulate recovery from drift, and handled lane and road following only, with the human driver taking over for lane changes and turns.

Simulators, commands and recovery data (2017–2018). CARLA gave the field a shared urban simulator with two towns, Town 1 for training and Town 2 for testing, and sensors ranging from RGB cameras to ground-truth depth and segmentation pseudo-sensors (Dosovitskiy et al., 2017). Its authors compared a modular pipeline (semantic segmentation, a rule-based state machine and PID control), an imitation-learning agent and a reinforcement-learning agent on goal-directed navigation of increasing difficulty. The modular pipeline and the imitation agent performed similarly in most conditions, their success rates typically differing by less than 10%; the reinforcement-learning agent, trained on 10 million simulation steps against the imitation agent’s 14 hours of driving, did much worse. In the test town with dynamic obstacles, the modular pipeline completed 24% of episodes and the imitation agent 38%; the authors trace the pipeline’s failures in the new town to perception failing systematically under complex weather, after which the rule-based planner could not recover, and call the pipeline more fragile than the end-to-end method in this sense (Dosovitskiy et al., 2017, Table 1). Their infraction table shows the other side: the modular pipeline generally went furthest between collisions with cars and static objects, and the authors take the end-to-end agents’ results as evidence of susceptibility to rare events such as braking for a pedestrian.

Conditional imitation learning answered the intersection problem by giving the network the expert’s intention as an input: one of four commands, follow the lane, go straight, turn left or turn right at the next intersection, which at test time come from a route planner. In the branched variant, the command selects one of several output heads on a shared perception network. With 2 hours of human driving in Town 1, of which about 12 minutes had temporally correlated noise injected into the steering so that the human’s corrections were recorded, the branched model completed 88% of episodes in Town 1 and 64% in Town 2, against 20% and 26% for a model without commands (the text gives 24% for Town 2 where the table has 26%); trained without the noise-injected data, it completed 56% and 22% (Codevilla et al., 2018, Table 1). The labels for the noisy segments are the driver’s corrective steering, not the injected noise, so that the data show recoveries without showing the car veering away.

Larger cloning studies and privileged teachers (2019). Codevilla et al. (2019) scaled behaviour cloning to 100 hours of simulated driving (CARLA100) with a ResNet-34 model, CILRS, that also predicts the vehicle’s speed from the image, and proposed the NoCrash benchmark in which collisions end an episode. They report three limitations. Dataset bias: most driving is a few simple behaviours, so performance can degrade as more data are collected if diversity does not grow with it. Causal confusion, in the form they call the inertia problem: when the car is stopped, for example at a red light, the training data overwhelmingly show it staying stopped, so the policy learns a spurious link between low speed and no acceleration and has trouble starting again; the speed-prediction head reduced this but did not remove it. High variance: twelve training runs differing only in random seed gave success rates whose variance reached 42% on the dense-traffic task (Codevilla et al., 2019, Table 3). Learning by Cheating (Chen et al., 2019) split the problem in two: first train a privileged agent that sees the ground-truth bird’s-eye-view layout and other road users and imitates the expert, then train the camera-only agent to imitate the privileged one. Because the teacher can be queried in any state, including those the student reaches, the authors describe the second stage as automatic DAgger-like training, and because the teacher was trained with commands, it can supervise all of the student’s command branches at once. The resulting agent reached 100% success on all tasks of the original CARLA benchmark.

Sensor fusion and waypoints (2021). As CARLA added adversarial scenarios such as vehicles running red lights and pedestrians emerging from occlusion, image-only agents struggled. TransFuser fuses a front camera image and a LiDAR bird’s-eye-view histogram with transformers at four resolutions, so that each modality’s features can attend to the other’s over the whole scene, and predicts waypoints (Prakash et al., 2021). The sensor fusion article covers fusion methods in general. On ten long routes of 1–2 km in the held-out Town05, TransFuser reached a driving score of 33.15 against 38.60 for the privileged expert that generated its training data, 25.30 for a geometry-based fusion baseline and 7.05 for an updated version of Learning by Cheating that the authors retrained on their data (Prakash et al., 2021, Table 1a, mean over three training seeds each evaluated three times). On the short routes, geometric fusion and TransFuser had similar driving scores (54.32 and 54.52) with standard deviations above 4.

Planning-oriented models on nuScenes (2023). UniAD runs tracking, online mapping, motion forecasting and occupancy prediction as transformer modules on bird’s-eye-view features from BEVFormer’s encoder, connects them through queries, and ends with the planner described above; an ego-vehicle query passes through the tracking and forecasting modules so that the planner starts from features that already describe the vehicle’s own likely motion (Hu et al., 2023). It was trained in two stages, perception first for 6 epochs and then everything for 20. On nuScenes it reported an average L2 error of 1.03 m and an average collision rate of 0.31% over the 3 s planning horizon, about half of ST-P3’s (Hu et al., 2023, Table 7). VAD replaces UniAD’s dense rasters (the occupancy and semantic map) with vectors: map elements as polylines and agents’ forecasts as trajectories, which its planning query attends to and its constraint losses use. Its base model reported an average L2 of 0.72 m and a collision rate of 0.22% on nuScenes, at 4.5 frames per second on one RTX 3090 against UniAD’s 1.8 on an A100 (Jiang et al., 2023, Table 1); in CARLA’s Town05 it reached a driving score of 30.31 on the long routes, below TransFuser’s 33.15, and 64.29 on the short routes (Jiang et al., 2023, Table 4). 3D occupancy prediction and motion forecasting describe the modules these models contain.

The evaluation reckoning (2023–2024). The nuScenes numbers of this generation of models turned out to be hard to interpret, for reasons the next sections give, and benchmarks built for planning on real data (nuPlan’s closed-loop simulation, then NAVSIM) took over much of the comparison.

Evaluation

Closed loop in simulation: CARLA

The original CARLA benchmark scored goal-directed navigation by the percentage of successful episodes, an episode succeeding if the agent reached the goal within a time budget, and separately by the average distance driven between infractions; infractions did not end an episode (Dosovitskiy et al., 2017). Codevilla et al. (2019) objected that continuing after an infraction makes counting inaccurate (is a crash after leaving the road one infraction or two?), and their NoCrash benchmark ends an episode as a failure at any collision above a fixed magnitude, with three traffic densities, four training and two test weathers, and a training and a test town.

The CARLA Leaderboard evaluates agents on predefined routes with scripted scenarios based on the NHTSA pre-crash typology, given only sparse route points with high-level commands; using the simulator’s privileged information, such as ground truth or its planners, is forbidden. Its SENSORS track allows cameras, LiDAR, radar, GNSS, an IMU and a speedometer; the MAP track adds the HD map as an OpenDRIVE file. The infraction penalty has changed with the versions (CARLA Leaderboard, evaluation pages; leaderboard code):

Version	Penalty $P_i$	Coefficients
1.0	$\prod_j p_j^{n_j}$	pedestrian collision 0.50, vehicle 0.60, static 0.65, red light 0.70, stop sign 0.80
2.0	$\prod_j p_j^{n_j}$	as 1.0, plus failing to yield to an emergency vehicle 0.70, scenario timeout 0.70, minimum-speed infraction up to 0.70
2.1	$1 / (1 + \sum_j c_j n_j)$	pedestrian 1.0, vehicle 0.70, static 0.60, red light 0.40, emergency vehicle 0.40, scenario timeout 0.40, stop sign 0.25, minimum speed up to 0.40

Here $n_j$ counts infractions of type $j$. In every version, driving off-road reduces the penalty by the off-road percentage of the route, cancelling the route completion gained there, and a route ends early on events such as deviating more than 30 m from the route or not moving for 180 simulated seconds. The leaderboard’s home page explains the change made in March 2025: participants had found that the best strategy was not to complete the route but to end the simulation after some fraction of it, because under what the page calls the exponential nature of the penalty, continuing to drive risked more than standing still, so the organizers changed it to a linear one (in the denominator, as the table shows) and advised against comparing 2.1 scores with 2.0 scores. The leaderboard code at the three branches implements these formulas. As of November 2026 the page also records that the CARLA Autonomous Driving Challenge did not run in 2025.

Two properties of simulated closed-loop results matter when reading them. They are noisy: TransFuser’s driving scores are means over three training seeds each evaluated three times with standard deviations of about 4 points (Prakash et al., 2021), and NAVSIM’s authors contrast the variance among training seeds in CARLA with their own deterministic evaluation (Dauner et al., 2024). And simulated sensors differ from real ones: Li et al. (2024) cite this gap between simulator and real world as the background to the recent attention paid to open-loop evaluation on real logs.

Open loop on real logs, and why it misleads

Offline error does not predict driving. Codevilla et al. (ECCV 2018) trained 45 models in CARLA, varying the amount and kind of training data, the network depth and the training procedure, and correlated their offline errors on held-out driving with their online success rate in the test town. For validation data from the central camera on the expert’s own trajectories, the setup most papers used, the correlation between steering mean squared error and success rate had an absolute value of only 0.39. Validation data with injected action noise raised it to 0.54, and data from three cameras, including two looking 30° to the sides, to 0.77; the authors conclude that a good policy must predict the expert’s action away from the expert’s trajectories, and that validation data should include recoveries from perturbations. Changing the metric helped less: absolute error (0.61), quantized classification error (0.65) and a thresholded relative error (0.64) correlated better than squared error, while weighting by speed or accumulating errors over time did not improve on absolute error (their Figures 2 and 3, computed on the better-performing half of the models).

Ego status is a shortcut on nuScenes. By 2023, planning-oriented models were compared on nuScenes by L2 error and collision rate against the logged trajectory. Li et al. (2024) found that 73.9% of nuScenes data are straight driving, where keeping the current speed and heading or turning rate is usually enough, so a model that knows the ego status (velocity, acceleration, yaw angle) can score well without looking at the road. With a single metric implementation applied to the official checkpoints, a planner that simply continues straight at the current velocity had an average L2 of 0.83 m, and an MLP given only the ego status and the driving command (Ego-MLP) had 0.35 m, against 0.66 m for the released UniAD checkpoint and 0.37 m for VAD-Base with ego status in its planner (Li et al., 2024, Table 1). The released UniAD uses ego status in its BEV module, and Li et al. note that it outperforms the results in the UniAD paper (1.03 m there, quoted above). When they blanked every camera image given to that VAD model, its detection and mapping scores fell to zero while its average L2 changed from 0.37 to 0.46 m; when they multiplied the input speed by 1.5, its L2 rose to 3.20 m, and a speed of 100 m/s produced wildly impractical trajectories that may leave the local bird’s-eye-view area, so that the collision rate is no longer precise (Table 2). They also proposed a curb collision rate, the rate at which planned trajectories cross road boundaries, on which Ego-MLP was worse than UniAD, and they concluded that methods using ego status cannot be compared directly on these metrics. Their analysis builds on AD-MLP by Zhai et al. (2023, posted on arXiv and not read here), which Li et al. describe as reaching state-of-the-art planning results from ego status and the past trajectory alone.

Two further sources of disagreement show up in the same papers. The numbers depend on the implementation: VAD-Base without ego status reported a collision rate of 0.22% in its own paper (Jiang et al., 2023, Table 1), while Li et al.’s common implementation, which among other things uses a 0.1 m collision grid instead of 0.5 m, gives 0.54% for the official checkpoint, with the same L2 of 0.72 m (Li et al., 2024, Table 1). Li et al. list that checkpoint as using ego status in its BEV module, while VAD’s authors write that they deactivated ego status for open-loop evaluation. And the command can leak the answer: NAVSIM’s authors note that most nuScenes planners receive the endpoint of the human trajectory as a discrete direction command (Dauner et al., 2024).

Ego-forecasting and planning are misaligned on nuPlan too. nuPlan’s planning challenge, built on over 1,300 hours of real driving in Las Vegas, Pittsburgh, Boston and Singapore, scores planners in three modes (nuPlan devkit documentation). In open loop, the planned trajectory is compared with the logged one by displacement and heading errors at 3, 5 and 8 s, and a miss-rate check sets the score to zero. In closed loop with non-reactive agents, the planner’s trajectory is tracked by an LQR controller while the other agents replay the log; with reactive agents, vehicles follow the Intelligent Driver Model (IDM). The closed-loop score multiplies hard penalties (at-fault collisions with vehicles or vulnerable road users, drivable-area violations, driving more than 6 m into oncoming traffic, insufficient progress) with a weighted average of time to collision, progress along the expert’s route, speed-limit compliance, driving direction and comfort. Dauner et al. (2023) found a negative correlation between the two schemes: learned planners trained to forecast the ego vehicle’s motion did best in open loop and worse in closed loop, and rule-based planners did the opposite. An IDM-based planner from over twenty years earlier outperformed the learned ones in closed loop, and an MLP that saw only the ego state and the route’s centreline, ignoring other agents, set the best open-loop score. A preliminary version of their hybrid of the two won the 2023 nuPlan challenge. These planners take perception outputs as input rather than sensors, but the lesson about imitation metrics carries over.

NAVSIM: non-reactive simulation on real sensor data

NAVSIM (Dauner et al., 2024) evaluates sensor-based policies on real data without rendering new sensor views. The agent sees the current and up to three past frames of eight cameras and the LiDAR, plus the ego status, and plans a 4 s trajectory once; the trajectory is then executed by an LQR controller and a kinematic bicycle model at 10 Hz in a bird’s-eye-view abstraction of the logged scene, while the other agents follow their logged motion. Nothing reacts to anything, but the outcome can be scored with the closed-loop-style subscores of the PDM Score defined above. The authors filtered the OpenScene redistribution of nuPlan to remove scenes in which a constant-velocity agent already scores above 0.8 and scenes in which the human trajectory scores below 0.8 (to discard annotation errors), giving the navtrain and navtest splits of 103k and 12k samples. On a set of 396 filtered scenarios (navmini), they ran 37 rule-based and 114 learned planners in nuPlan’s closed-loop simulator and found that the PDM Score correlated better with the closed-loop score than nuPlan’s open-loop score did, both overall and within every planner type (their Figures 3 and 4).

On navtest, a constant-velocity agent scored 20.6 and an ego-status MLP 65.6, while TransFuser and PARA-Drive scored 84.0, the camera-only Latent TransFuser 83.8, UniAD 83.4 and the human driver 94.8 (Dauner et al., 2024, Table 1). The ego-status shortcut that dominated nuScenes does not suffice here; and UniAD and PARA-Drive, whose training the authors call more demanding (80 GPUs for 3 days for PARA-Drive against 1 GPU for 1 day for TransFuser), did not surpass TransFuser. Three TransFuser training seeds gave PDM Scores of 83.3, 84.0 and 84.4 (Table 2). As of November 2026, the project’s repository hosts NAVSIM v2, which adds a two-stage “pseudo-simulation” with synthetic observations near the planned trajectory and an extended score (EPDMS), and keeps v1 on a separate branch (NAVSIM README; the v2 paper by Cao et al., 2025, was not read).

Properties

• Distribution shift is built in. Every remedy in the history above changes the training distribution rather than the model: shifted cameras, noise injection with expert corrections, DAgger with a queryable expert, and privileged teachers queried on the student’s own states (Bojarski et al., as described by Codevilla et al., 2018; Codevilla et al., 2018; Ross et al., 2011; Chen et al., 2019).
• The navigation input cuts both ways. A route command makes the mapping at intersections a function of the input (Codevilla et al., 2018); the same input becomes a leak when it encodes the logged future, as on nuScenes (Dauner et al., 2024).
• Single runs are weak evidence. Closed-loop success varies with the training seed (Codevilla et al., 2019; Prakash et al., 2021), and open-loop scores vary with the metric implementation (Li et al., 2024).

Examples

Example: behaviour cloning against a detect-then-control pipeline

The script below builds a miniature of the comparison CARLA’s authors made and of the evaluation question Codevilla et al. (ECCV 2018) asked. Roads are generated from a curvature profile that is the sum of two sinusoids (radius of curvature down to about 25 m), 4 m wide with white edge lines. A kinematic bicycle (wheelbase 2.7 m) drives at a constant 10 m/s with 0.1 s steps; it has left the road when its reference point is more than 2 m from the centreline. Its only sensor is a forward camera 1.5 m above the ground: a noisy 48 × 24 image of the ground below the horizon (focal length 24 px), rendered from the true road geometry, with grass, road and lines at fixed grey levels.

• Expert: pure pursuit on the true centreline, aiming at the point 6 m ahead. It is privileged, like the CARLA experts of Codevilla et al. (ECCV 2018) and Prakash et al. (2021), and serves as both demonstrator and labeller.
• Modular pipeline: a threshold finds road pixels, each image row between 3 and 20 m gives the midpoint of the road run nearest the image centre, a quadratic fitted to those points on the ground estimates the centreline, and the same pure-pursuit law steers towards its point 6 m ahead. Nothing is learned.
• Behaviour cloning: a two-layer convolutional network maps the image to steering, trained with mean squared error on 16,000 frames from 40 roads (400 m each). Three training sets differ only in how many of the 40 demonstration drives had temporally correlated noise added to the executed steering, with the expert’s corrective steering as the label, as in conditional imitation learning: none (BC-clean), 4 (BC-10%, the share Codevilla et al., 2018, used) and all 40 (BC-100%). Each is trained with seeds 0, 1 and 2.

The open-loop test computes the mean absolute steering error against the expert on 10 held-out roads, once along the expert’s clean drive and once along a noise-perturbed drive (the analogue of Codevilla et al.’s validation data with action noise). The closed-loop test lets each policy drive 600 m on 20 further roads, with and without small random actuation noise (0.01 rad per step; the sources leave this choice open, so both are printed), and reports the fraction of roads completed, the mean distance driven before leaving the road, the mean absolute offset while on it, and the fraction still on the road after 100, 200, 400 and 600 m. Before the first run, a check confirmed that the expert stays on all test roads, that zero steering leaves the road after 28 m on average, that the clean demonstrations visit offsets below 0.08 m in 95% of frames against 0.34 m for the noisy ones, and that the pipeline’s estimate of the lateral offset has a mean absolute error of 0.12 m on clean and 0.15 m on perturbed frames. It runs with PyTorch 2.14.1, NumPy 2.5.4 and SciPy 1.18.1 with four PyTorch threads, in about 5.5 minutes on our machine; two runs gave identical output.

import numpy as np
import torch
from scipy.spatial import cKDTree

torch.set_num_threads(4)
L, V, DT, DMAX, HALF_W = 2.7, 10.0, 0.1, 0.5, 2.0   # wheelbase, speed, step, steer limit, road half-width

# --- Roads: centreline from a sum of two sinusoidal curvature profiles -----------------
def make_track(seed, length=700.0, ds=0.25):
    r = np.random.default_rng(seed)
    a, P, ph = r.uniform(0, 0.02, 2), [r.uniform(150, 300), r.uniform(60, 120)], r.uniform(0, 2*np.pi, 2)
    s = np.arange(0, length, ds)
    kappa = a[0]*np.sin(2*np.pi*s/P[0] + ph[0]) + a[1]*np.sin(2*np.pi*s/P[1] + ph[1])
    th = np.concatenate([[0.0], np.cumsum(kappa[:-1]*ds)])
    xy = np.stack([np.cumsum(np.cos(th))*ds, np.cumsum(np.sin(th))*ds], 1) - [ds, 0]
    return dict(xy=xy, t=np.stack([np.cos(th), np.sin(th)], 1), tree=cKDTree(xy))

def lateral(tr, q):                       # signed offset (left positive) and nearest index
    _, i = tr["tree"].query(q)
    t, r = tr["t"][i], q - tr["xy"][i]
    return t[..., 0]*r[..., 1] - t[..., 1]*r[..., 0], i

# --- Camera: 48 x 24 ground pixels below the horizon, f = 24 px, height 1.5 m -----------
F, H_CAM, W, H = 24.0, 1.5, 48, 24
rows, cols = np.meshgrid(np.arange(H) + 0.5, np.arange(W) + 0.5, indexing="ij")
Z = F*H_CAM/rows                           # forward distance of each pixel's ground point
X = (cols - W/2)*Z/F                       # rightward offset
FAR = Z > 40.0

def render(tr, state, rng):                # state rows: x, y, heading
    x, y, th = state[:, :1, None], state[:, 1:2, None], state[:, 2:3, None]
    gx = x + Z*np.cos(th) + X*np.sin(th)
    gy = y + Z*np.sin(th) - X*np.cos(th)
    d, _ = lateral(tr, np.stack([gx, gy], -1).reshape(-1, 2))
    d = np.abs(d).reshape(gx.shape)
    img = np.where(d < HALF_W, np.where(d > HALF_W - 0.15, 1.0, 0.6), 0.25)
    img[:, FAR] = 0.4
    return img + rng.normal(0, 0.05, img.shape)

# --- Expert: pure pursuit on the true centreline, 6 m ahead ----------------------------
def expert(tr, st):
    _, i = lateral(tr, st[:, :2])
    tgt = tr["xy"][np.minimum(i + 24, len(tr["xy"]) - 1)] - st[:, :2]
    fz = tgt[:, 0]*np.cos(st[:, 2]) + tgt[:, 1]*np.sin(st[:, 2])
    fy = -tgt[:, 0]*np.sin(st[:, 2]) + tgt[:, 1]*np.cos(st[:, 2])
    a = np.arctan2(fy, fz)
    return np.clip(np.arctan(2*L*np.sin(a)/np.hypot(fz, fy)), -DMAX, DMAX)

def step(st, delta):
    x, y, th = st.T
    return np.stack([x + V*np.cos(th)*DT, y + V*np.sin(th)*DT, th + V/L*np.tan(delta)*DT], 1)

# --- Modular pipeline: threshold, lane centre per row, quadratic fit, pure pursuit -----
ROWS = np.where((Z[:, 0] >= 3) & (Z[:, 0] <= 20))[0]
def modular(imgs, prev):
    out = prev.copy()
    for k, im in enumerate(imgs):
        zs, xs = [], []
        for r in ROWS:
            road = im[r] > 0.45
            if not road.any():
                continue
            c = int(np.argmin(np.where(road, np.abs(np.arange(W) - W/2 + 0.5), 1e9)))
            lo, hi = c, c
            while lo > 0 and road[lo - 1]: lo -= 1
            while hi < W - 1 and road[hi + 1]: hi += 1
            if lo == 0 or hi == W - 1:
                continue                   # run touches the border: centre unknown
            zs.append(Z[r, 0]); xs.append(((lo + hi + 1)/2 - W/2)*Z[r, 0]/F)
        if len(zs) >= 4:
            b = np.polyfit(zs, xs, 2)
            xt = np.polyval(b, 6.0)
            out[k] = np.clip(np.arctan(2*L*np.sin(np.arctan2(-xt, 6.0))/np.hypot(6.0, xt)), -DMAX, DMAX)
    return out

# --- Data collection: clean or noise-injected expert driving --------------------------
def drive_expert(seed, steps, noisy, rng):
    tr, st, n = make_track(seed), np.zeros((1, 3)), 0.0
    S, A = [], []
    for _ in range(steps):
        a = expert(tr, st)
        S.append(st[0]); A.append(a[0])
        if noisy:
            n = 0.8*n + rng.normal(0, 0.02)
        st = step(st, np.clip(a + (n if noisy else 0.0), -DMAX, DMAX))
    S = np.array(S)
    return render(tr, S, rng), np.array(A), np.abs(lateral(tr, S[:, :2])[0])

def dataset(seeds, noisy_seeds, steps, rng):
    out = [drive_expert(s, steps, s in noisy_seeds, rng) for s in seeds]
    return [np.concatenate(z) for z in zip(*out)]

# --- Behaviour cloning: a small CNN from the image to steering ------------------------
def net():
    return torch.nn.Sequential(torch.nn.Conv2d(1, 8, 5, 2), torch.nn.ReLU(),
                               torch.nn.Conv2d(8, 16, 3, 2), torch.nn.ReLU(), torch.nn.Flatten(),
                               torch.nn.Linear(16*4*10, 64), torch.nn.ReLU(), torch.nn.Linear(64, 1))

def train(imgs, acts, seed, epochs=6):
    torch.manual_seed(seed)
    m, g = net(), torch.Generator().manual_seed(seed)
    opt = torch.optim.Adam(m.parameters(), 1e-3)
    Xt = torch.tensor(imgs[:, None] - 0.4, dtype=torch.float32)
    Yt = torch.tensor(acts[:, None]*10, dtype=torch.float32)
    for _ in range(epochs):
        for idx in torch.randperm(len(Xt), generator=g).split(128):
            opt.zero_grad()
            torch.nn.functional.mse_loss(m(Xt[idx]), Yt[idx]).backward()
            opt.step()
    return lambda im, prev: np.clip(m(torch.tensor(im[:, None] - 0.4, dtype=torch.float32))
                                    .detach().numpy()[:, 0]/10, -DMAX, DMAX)

# --- Closed loop: 20 test roads in parallel, 600 m each -------------------------------
TEST = [make_track(s) for s in range(2000, 2020)]
def closed_loop(policy, act_noise, steps=600):
    rng, nrng = np.random.default_rng(7), np.random.default_rng(8)
    st, alive = np.zeros((len(TEST), 3)), np.ones(len(TEST), bool)
    dist, absd, prev = np.zeros(len(TEST)), [[] for _ in TEST], np.zeros(len(TEST))
    for t in range(steps):
        noise = nrng.normal(0, act_noise, len(TEST))
        if policy == "expert":
            a = np.array([expert(tr, st[k:k+1])[0] for k, tr in enumerate(TEST)])
        else:
            imgs = np.concatenate([render(tr, st[k:k+1], rng) for k, tr in enumerate(TEST)])
            a = policy(imgs, prev)
        prev = a
        st = step(st, np.clip(a + noise, -DMAX, DMAX))
        for k, tr in enumerate(TEST):
            if alive[k]:
                d = abs(lateral(tr, st[k:k+1, :2])[0][0])
                if d > HALF_W:
                    alive[k] = False
                else:
                    dist[k] += V*DT; absd[k].append(d)
    return dist, np.mean(np.concatenate([np.array(a) for a in absd]))

if __name__ == "__main__":
    rng = np.random.default_rng(0)
    clean_val = dataset(range(1000, 1010), set(), 400, rng)
    noisy_val = dataset(range(1000, 1010), set(range(1000, 1010)), 400, rng)
    data = {name: dataset(range(40), set(noisy), 400, rng)
            for name, noisy in [("BC-clean", []), ("BC-10%", range(4)), ("BC-100%", range(40))]}
    print("visited |d| (m), 95th percentile: clean %.2f, noisy %.2f" %
          (np.percentile(data["BC-clean"][2], 95), np.percentile(data["BC-100%"][2], 95)))

    mae = lambda pol, ds: np.mean(np.abs(pol(ds[0], np.zeros(len(ds[0]))) - ds[1]))
    print("\nopen loop: steering MAE (rad) on held-out expert states")
    print("%-10s %8s %10s" % ("policy", "clean", "perturbed"))
    print("%-10s %8.4f %10.4f" % ("modular", mae(modular, clean_val), mae(modular, noisy_val)))
    policies = {"expert": ["expert"], "modular": [modular]}
    for name, (imgs, acts, _) in data.items():
        policies[name] = [train(imgs, acts, seed) for seed in (0, 1, 2)]
        m = np.array([[mae(p, clean_val), mae(p, noisy_val)] for p in policies[name]])
        print("%-10s %8.4f %10.4f   seeds 0-2, clean [%.4f, %.4f]" %
              (name, *m.mean(0), m[:, 0].min(), m[:, 0].max()))

    for act_noise in (0.0, 0.01):
        print("\nclosed loop, 20 roads x 600 m, actuation noise %.2f rad" % act_noise)
        print("%-10s %10s %9s %9s   on road at 100/200/400/600 m" % ("policy", "completed", "dist (m)", "mean |d|"))
        for name, pols in policies.items():
            runs = [closed_loop(p, act_noise) for p in pols]
            dists = np.array([r[0] for r in runs])           # seeds x roads
            comp = (dists >= 599.9).mean(1)
            alive = [(dists >= m - 0.1).mean() for m in (100, 200, 400, 600)]
            seeds = "  seeds [%.2f, %.2f]" % (comp.min(), comp.max()) if len(pols) > 1 else ""
            print("%-10s %10.2f %9.1f %9.3f   %s%s" % (name, comp.mean(), dists.mean(),
                  np.mean([r[1] for r in runs]), " ".join("%.2f" % a for a in alive), seeds))

visited |d| (m), 95th percentile: clean 0.08, noisy 0.34

open loop: steering MAE (rad) on held-out expert states
policy        clean  perturbed
modular      0.0042     0.0045
BC-clean     0.0018     0.0085   seeds 0-2, clean [0.0017, 0.0018]
BC-10%       0.0021     0.0039   seeds 0-2, clean [0.0019, 0.0024]
BC-100%      0.0024     0.0030   seeds 0-2, clean [0.0024, 0.0025]

closed loop, 20 roads x 600 m, actuation noise 0.00 rad
policy      completed  dist (m)  mean |d|   on road at 100/200/400/600 m
expert           1.00     600.0     0.030   1.00 1.00 1.00 1.00
modular          1.00     600.0     0.034   1.00 1.00 1.00 1.00
BC-clean         0.92     567.3     0.034   0.95 0.95 0.95 0.92  seeds [0.90, 0.95]
BC-10%           1.00     600.0     0.031   1.00 1.00 1.00 1.00  seeds [1.00, 1.00]
BC-100%          1.00     600.0     0.031   1.00 1.00 1.00 1.00  seeds [1.00, 1.00]

closed loop, 20 roads x 600 m, actuation noise 0.01 rad
policy      completed  dist (m)  mean |d|   on road at 100/200/400/600 m
expert           1.00     600.0     0.035   1.00 1.00 1.00 1.00
modular          1.00     600.0     0.038   1.00 1.00 1.00 1.00
BC-clean         0.87     544.0     0.044   0.95 0.90 0.88 0.87  seeds [0.85, 0.90]
BC-10%           1.00     600.0     0.036   1.00 1.00 1.00 1.00  seeds [1.00, 1.00]
BC-100%          1.00     600.0     0.036   1.00 1.00 1.00 1.00  seeds [1.00, 1.00]

The open-loop table ranks the policies in the wrong order for driving. The clean-trained network imitates the expert most closely on the expert’s own states, with less than half the modular pipeline’s error, yet it is the only policy that left the road: on one or two of the 20 roads per seed without actuation noise and on two or three with it. Its error on perturbed states is about five times its error on clean ones, because its training data contain almost no off-centre images, so it has never been shown what to do there; once small errors push it sideways, nothing pulls it back. The visuomotor policy learning example measures the same shift on a reaching task, on the states the policy visits itself. Here, error on perturbed states ranks BC-clean worst, as the closed loop does (BC-10% and BC-100% tie there), while error on clean states ranks it best, which is Codevilla et al.’s finding about validation data in miniature. Four noisy drives out of forty were enough for every seed to complete every road, consistent with their observation that a small amount of noise-injected data stabilized the policy.

The modular pipeline has the largest error on clean states, since its steering inherits the noise of a lane estimate made from 10 image rows, and it still completes every road: its control law is defined for any offset, and its perception does not depend on which states it was shown. The survival columns show the compounding only weakly: 5% of BC-clean’s runs have left the road by 100 m in both settings, and by 600 m 8% without actuation noise (all of the further departures after 400 m) and 13% with it, where departures accumulate at every checkpoint. The mean offset over the steps it does drive is close to the others’ without actuation noise and somewhat larger with it, which in our reading means its failures are mostly sudden losses of a few roads rather than a general loss of precision. The scale limits what this shows: one lane, no other traffic, a perfect simulator expert and 20 test roads, so the closed-loop differences rest on a handful of departures per seed. It illustrates the mechanisms the sources describe; it says nothing about which design drives better on real roads.

Common Misconceptions

• “End-to-end means no intermediate representations.” Prominent recent models such as UniAD and VAD keep detection, tracking, mapping and forecasting (and, in UniAD, occupancy) as supervised modules; what makes them end-to-end is that the modules pass learned features to a planner and are trained together for it (Hu et al., 2023; Jiang et al., 2023). Learning by Cheating even uses a hand-designed bird’s-eye-view map, but only for the teacher (Chen et al., 2019).
• “Low prediction error on driving logs means good driving.” Offline steering error on the expert’s trajectories correlated weakly with closed-loop success in CARLA (Codevilla et al., ECCV 2018), open-loop and closed-loop scores were negatively correlated across planners on nuPlan (Dauner et al., 2023), and on nuScenes a model given only the ego status matched the best perception-based planners in L2 error (Li et al., 2024). The example above shows the same effect at toy scale.
• “More demonstrations always help.” If the extra data are more of the dominant behaviour, they can make a cloned policy worse (Codevilla et al., 2019); what helped in the cited studies was data of the right kind, from perturbed or learner-induced states.
• “A modular stack is safe and an end-to-end one is not.” CARLA’s authors found their modular baseline’s performance bimodal, failing completely when its perception failed in the new town, and called it more fragile than the imitation agent in that sense; the modular baseline did better on collisions (Dosovitskiy et al., 2017). Which design wins depends on the benchmark, the data and how failures are counted, and the cited studies compare specific implementations, not the paradigms as such.
• “Closed-loop simulation settles the question.” Simulated sensors and simulated traffic differ from real ones (Li et al., 2024; Dauner et al., 2024), scores vary with training seeds (Codevilla et al., 2019; Prakash et al., 2021), and a metric’s design can reward the wrong behaviour, as the CARLA Leaderboard’s early-stopping strategy did under its multiplicative penalty.

Where It Is Used

End-to-end policies are a research subject first: CARLA and its leaderboard, nuPlan and NAVSIM are where most of the methods above were developed and compared. They depend on bird’s-eye-view perception for their scene encoders, on sensor fusion for combining cameras and LiDAR, and on motion forecasting and 3D occupancy prediction as modules or auxiliary tasks; transformers and cross-attention connect those modules in UniAD, VAD and TransFuser. The same imitation-learning questions arise for robots that act from pixels, which embodied AI and visuomotor policy learning treat, and vision-language-action models extend the idea with language conditioning. Industrial use is described in the cited papers only in passing; Hu et al. (2023) note that most industrial solutions deploy separate models per task. This article states no view on the safety or readiness of any deployed system; its results are those the sources report under their benchmarks’ conditions.