Motion Forecasting for Autonomous Driving

Motion forecasting predicts where the other road users around a self-driving vehicle will be over the next few seconds: given the tracked past of each vehicle, pedestrian and cyclist and a map of the road, it outputs future trajectories, usually several per agent, each with a probability. It is also called trajectory prediction, and Waymo’s papers call it behavior prediction (Chai et al., 2019; Gao et al., 2020). A planner needs it because whether to yield, merge or keep going depends on what the others will do, not only on where they are; the Argoverse 2 authors describe forecasts as the primary input to motion planning (Wilson et al., 2021).

The task sits between perception and planning. Its inputs are the 3D tracks produced by multi-object tracking and the high-definition map described in autonomous driving perception, which also gives Argoverse’s definition of the task in a paragraph. This article goes deeper: the problem and its multimodal outputs, why regressing a single trajectory averages the possible futures, the physics baselines, the change from rasterized to vectorized scene encoders, anchor-, goal- and transformer-based predictors, joint prediction of interacting agents, and the benchmarks with the exact definitions their evaluation code uses. A worked example trains a small predictor on synthetic intersection tracks and scores it the way Argoverse does. Motion forecasting is studied with logged data; nothing here describes how a vehicle should be operated.

Problem Definition

For one agent, let $\mathbf{s}_{-T_{\text{obs}}+1:0}$ be its observed states (2D positions, often with heading and velocity) and let $\mathbf{c}$ be the scene context: the other agents’ histories, the lane graph, crosswalks, traffic-light states. The task is to predict the future positions $\mathbf{s}_{1:T}$ up to a fixed horizon $T$. TNT’s authors write the target as the conditional distribution $p(\mathbf{s}_{1:T} \mid \mathbf{x})$ with $\mathbf{x} = (\mathbf{s}_{\text{past}}, \mathbf{c})$ (Zhao et al., 2020), and every benchmark below asks for a finite summary of it: $K$ trajectories $\{\mathbf{s}^k_{1:T}\}_{k=1}^{K}$ with confidences $c_k$.

The defining difficulty is that $p(\mathbf{s}_{1:T} \mid \mathbf{x})$ is multimodal. A car approaching an intersection may turn left, go straight or turn right, and nothing in its past need reveal which. Argoverse’s authors made this the centre of their evaluation, arguing that a vehicle must anticipate many plausible outcomes rather than only the most likely one, and following DESIRE and Social GAN in scoring the best of several predictions (Chang et al., 2019). The ground truth is a single logged future, so a benchmark sees one sample of the distribution per scenario.

Input and Output

Benchmarks fix the windows. Argoverse 1 observes 20 frames (2 s) at 10 Hz and predicts 10 to 30 frames (1 to 3 s) (Chang et al., 2019). Argoverse 2 scenarios are 11 s long at 10 Hz: 5 s observed, 6 s forecast (Wilson et al., 2021). The Waymo Open Motion Dataset (WOMD) scenes are 9.1 s, 91 steps at 10 Hz, with an 8 s horizon (Ettinger et al., 2021); its official metrics configuration uses 10 history samples plus the current one, expects predictions at 2 Hz, and measures at 3, 5 and 8 s (Waymo, motion tutorial). nuScenes prediction allows up to 2 s of history and asks for 6 s trajectories sampled at 2 Hz (nuScenes devkit README).

Inputs are the outputs of a perception stack, not raw sensor data. WOMD provides tracks from an offboard 3D detection and tracking pipeline, with the box, heading and velocity of each object and a validity flag per step, and a map of polylines and polygons sampled at 0.5 m: lane centres, lane boundaries, road edges, stop signs, crosswalks, speed bumps and traffic-signal states (Ettinger et al., 2021). Argoverse 1 gives the 2D bird’s-eye-view centroid of each tracked object and a city-scale vector map (Chang et al., 2019); Argoverse 2 gives position, velocity and orientation and a local map per scenario, with 3D lane boundaries and the drivable area as polygons (Wilson et al., 2021).

Outputs are $K$ trajectories per agent ($K = 6$ on Argoverse and WOMD; up to 25 on nuScenes) with confidences. WOMD’s submission format says the confidences should not be normalized across the set (Waymo, motion_metrics.proto), while Argoverse’s evaluation code normalizes the probabilities of the top $K$ before using them (argoverse-api).

Variants

• Marginal and joint prediction. Marginal prediction gives each agent its own set of futures, scored agent by agent. Joint prediction gives $K$ futures for a group of agents together, each a consistent scene; WOMD’s interactive split asks for the joint futures of two interacting agents and scores the pair together (Ettinger et al., 2021).
• Single-agent and multi-agent. Argoverse 1 scores one “interesting” agent per sequence (Chang et al., 2019); Argoverse 2 adds a set of scored actors to the focal agent (Wilson et al., 2021); WOMD’s standard split asks for up to 8 agents per scene (Ettinger et al., 2021).
• Agent classes. Argoverse 1 and nuScenes evaluate vehicles; WOMD evaluates vehicles, pedestrians and cyclists, and Argoverse 2 five classes of dynamic objects (Ettinger et al., 2021; Wilson et al., 2021).
• Occupancy instead of trajectories. The Argoverse 2 authors define the task as predicting future states or occupancy maps (Wilson et al., 2021), and MultiPath’s Gaussian mixtures can be evaluated on a grid to give an occupancy map (Chai et al., 2019). Occupancy forecasting is related to 3D occupancy prediction, which estimates the present occupancy of space.
• Forecasting inside an end-to-end model. Joint models such as UniAD run forecasting as one module among tracking, mapping and planning, as the autonomous driving perception article describes; end-to-end autonomous driving treats the whole chain together.

Challenges

• Multimodality. The futures form discrete choices (turn or not) with continuous variation inside each (speed, timing, curvature). A single prediction cannot represent both, and a metric that rewards a single prediction rewards the wrong thing (see below).
• Dependence on the map. Which turns exist, where lanes go and when signals change are given by the map, not the track. Argoverse’s baselines found that the vector map mattered most at the 3 s horizon, while adding social features did not help their LSTM (Chang et al., 2019).
• Interaction. Agents react to each other: a car yields to a pedestrian, a merging car waits for a gap. WOMD measured interactivity by how much a conditional predictor’s output for one agent changed when told another agent’s future: 73% of agent pairs in its interactive validation set had a KL divergence above 10, against 48% in the standard set (Ettinger et al., 2021).
• A long tail of easy cases. Most logged vehicles are parked or moving at nearly constant velocity, which Argoverse’s authors called “hardly a representation” of real forecasting problems; every benchmark therefore mines interesting cases (Chang et al., 2019). MultiPath’s authors capped each bin of a grid over curvature and distance at 5% of their dataset for the same reason (Chai et al., 2019).
• Perception noise. Histories come from automatic trackers. VectorNet’s authors note that Argoverse histories and futures are both machine-generated and noisy (Gao et al., 2020); WOMD invested in an offboard pipeline to reduce this noise (Ettinger et al., 2021).

Approaches

Physics and map heuristics

The oldest predictors extrapolate a kinematic model: constant velocity, constant acceleration, or constant turn rate and speed. These are the motion models a tracker’s Kalman filter already runs, used open-loop over several seconds instead of one frame. Their parameters matter: Argoverse’s constant-velocity baseline uses the mean velocity over the whole 2 s history (Chang et al., 2019), WOMD’s uses the velocity at the current step (Ettinger et al., 2021), and MultiPath’s “Linear” baseline fits each coordinate as a linear function of time, after higher-order polynomials did worse (Chai et al., 2019). They are useful as floors and as checks on a benchmark: on WOMD’s standard validation set the constant-velocity model had a vehicle minADE of 11.0 m at 8 s against 1.34 m for the best LSTM, which the authors take as evidence that the dataset contains nontrivial trajectories (Ettinger et al., 2021, Table 2).

Map heuristics came next. Argoverse’s map-based baselines localize the agent on the lane graph and generate hypotheses by breadth-first search along the branching lanes, on average 5.9 per scenario; LaneGCN’s authors describe this kind of method as constraining predictions to the map, which cannot capture rare or non-compliant behaviour (Chang et al., 2019; Liang et al., 2020).

Why single-trajectory regression averages modes

A network trained to output one trajectory with a squared-error loss converges, given enough data and capacity, toward the conditional mean, because the minimizer of $\mathbb{E}\,\|\mathbf{Y} - f(\mathbf{x})\|^2$ over $f$ is $f^*(\mathbf{x}) = \mathbb{E}[\mathbf{Y} \mid \mathbf{x}]$. If the history allows branches $b$ with probabilities $\pi_b$ and futures $\mathbf{y}_b$, then

$$f^*(\mathbf{x}) = \sum_b \pi_b \, \mathbf{y}_b ,$$

a trajectory between the branches that no agent follows: at a T-junction it cuts through the corner. MultiPath’s toy experiment shows the effect on synthetic three-way intersections: the single-trajectory model predicted a mean over all paths, with an uncertainty that grew as the paths diverged (Chai et al., 2019). TNT’s authors name mode averaging as the main obstacle to multimodal prediction by regression (Zhao et al., 2020).

Three families of remedies followed:

• Several outputs with winner-takes-all training. The network outputs $K$ trajectories and only the one closest to the ground truth is trained on each example, so the outputs can specialize. MultiPath calls this baseline Min-of-K and notes that its implicit, evolving assignment can suffer from mode collapse; on their toy intersection it collapsed to one or two active modes in five trials at four learning rates (Chai et al., 2019). LaneGCN uses this scheme with the winner chosen by final displacement, a smooth-$\ell_1$ loss on the winner and a max-margin loss that ranks its confidence above the others (Liang et al., 2020).
• Mixture distributions over fixed anchors. MultiPath predicts a Gaussian mixture at each future step: a softmax over $K$ anchor trajectories (intent uncertainty) and, for each anchor, Gaussian offsets per waypoint (control uncertainty). The anchors are fixed before training, obtained by k-means clustering of training trajectories (or, where k-means gave redundant clusters, by sampling trajectory space uniformly), and each example trains only the anchor nearest its ground truth (Chai et al., 2019).
• Latent-variable samplers. Conditional variational autoencoders (DESIRE) and GANs (Social GAN) draw futures by sampling. TNT’s authors argue that their latent intents cannot be interpreted and that probabilistic queries need test-time sampling (Zhao et al., 2020); MultiPath’s authors add non-determinism in a safety-critical system and the lack of a handle on approximation error (Chai et al., 2019).

On its in-house driving data (6 s horizon at 5 Hz), MultiPath reported minADE$_5$ of 0.58 m for the anchor model with offsets only, against 0.87 m for Min-of-K and 1.17 m for single-trajectory regression, averaged over five runs (Chai et al., 2019, Table 1).

Example: modes at a synthetic intersection

The example builds synthetic bird’s-eye-view tracks at 10 Hz with Argoverse 1’s windows: 2 s observed and 3 s predicted. Every agent drives at a constant speed between 8 and 12 m/s. In 60% of tracks, “approach” agents drive straight towards an intersection and then go straight (probability 0.4), turn left or turn right (0.3 each) along a 90° arc of radius 8 to 14 m, starting 0.3 to 1.2 s into the future; nothing in the history depends on the choice. The other 40% are already turning when observation ends. Each track gets a random pose, and the history gets 0.1 m of Gaussian tracking noise per axis.

Five predictors are compared: constant velocity with the velocity averaged over the whole history (Argoverse’s baseline) or over the last 0.5 s (closer to WOMD’s current-velocity baseline), both printed because the sources differ; constant turn rate and speed fitted to the last second; and a small multilayer perceptron trained twice on the same history in an agent-centred frame (last point at the origin, heading over the last 0.5 s along $+x$; Argoverse’s map-free baselines instead put the first observed point at the origin and the last on the $+x$ axis, Chang et al., 2019), once with one output trajectory and squared error, and once with six outputs and the same squared error applied to the winning output only, plus a cross-entropy loss that trains a confidence for each output to predict the winner. Scoring follows Argoverse 1’s evaluation code: the mode with the smallest final displacement is selected, minADE is that mode’s average displacement, and a miss is a minFDE above 2 m. The script ran in about 20 seconds on a CPU with one thread (PyTorch 2.14.1, NumPy 2.5.4).

import numpy as np
import torch

DT, T_OBS, T_FUT, MISS = 0.1, 20, 30, 2.0          # 10 Hz, 2 s history, 3 s future, Argoverse miss threshold
t_all = DT * (np.arange(T_OBS + T_FUT) - (T_OBS - 1))  # -1.9 ... 3.0 s

def rollout(v, omega, t_on, t_off, sub=10):
    """Constant speed v; turn rate omega during [t_on, t_off), else straight. Start at the origin heading +x."""
    pos, th, out, h = np.zeros((len(v), 2)), np.zeros(len(v)), [np.zeros((len(v), 2))], DT / sub
    for k in range(1, len(t_all)):
        for s in range(sub):
            t = t_all[k - 1] + (s + 0.5) * h
            w = np.where((t >= t_on) & (t < t_off), omega, 0.0)
            pos = pos + h * v[:, None] * np.stack([np.cos(th + 0.5 * w * h), np.sin(th + 0.5 * w * h)], 1)
            th = th + w * h
        out.append(pos.copy())
    return np.stack(out, 1)

def make_agents(n, rng):
    v, R = rng.uniform(8, 12, n), rng.uniform(8, 14, n)
    D = 0.5 * np.pi * R / v                                   # duration of a 90-degree turn
    approach = rng.random(n) < 0.6                            # turn (if any) starts in the future
    branch = rng.choice(3, n, p=[0.4, 0.3, 0.3])              # straight, left, right: not visible in the past
    side = np.where(rng.random(n) < 0.5, 1.0, -1.0)           # in-turn agents: already turning
    t0, ts = rng.uniform(0.3, 1.2, n), -rng.uniform(0.4, np.minimum(1.2, D - 0.3))
    sign = np.where(approach, np.select([branch == 1, branch == 2], [1.0, -1.0], 0.0), side)
    a = dict(v=v, R=R, approach=approach, branch=branch, t_on=np.where(approach, t0, ts))
    a["t_off"] = a["t_on"] + D
    tr = rollout(v, sign * v / R, a["t_on"], a["t_off"])
    ang, off = rng.uniform(-np.pi, np.pi, n), rng.uniform(-50, 50, (n, 2))   # random pose in the world
    rot = np.stack([np.cos(ang), -np.sin(ang), np.sin(ang), np.cos(ang)], 1).reshape(n, 2, 2)
    tr = np.einsum("nij,ntj->nti", rot, tr) + off[:, None]
    obs = tr[:, :T_OBS] + rng.normal(0, 0.1, (n, T_OBS, 2))  # tracker noise on the history only
    a["rot"], a["off"] = rot, off
    return obs, tr[:, T_OBS:], a

def normalize(obs):
    """Agent-centred frame: last observed point at the origin, recent heading along +x."""
    d = obs[:, -1] - obs[:, -6]
    c, s = d[:, 0] / np.linalg.norm(d, axis=1), d[:, 1] / np.linalg.norm(d, axis=1)
    Rn = np.stack([c, s, -s, c], 1).reshape(-1, 2, 2)
    return Rn, obs[:, -1]

def to_frame(x, Rn, o): return np.einsum("nij,n...j->n...i", Rn, x - o[:, None] if x.ndim == 3 else x - o[:, None, None])

# ---------- physics baselines (global frame) ----------
def constant_velocity(obs, span):
    vel = (obs[:, -1] - obs[:, -1 - span]) / (span * DT)
    return obs[:, -1, None] + vel[:, None] * DT * np.arange(1, T_FUT + 1)[None, :, None]

def constant_turn(obs):
    ch = obs[:, 10:20] - obs[:, 5:15]                        # 0.5 s chords, midpoints in the last second
    hd = np.unwrap(np.arctan2(ch[..., 1], ch[..., 0]), axis=1)
    tm = DT * (np.arange(5, 15) + 2.5) - 1.9
    om, b = np.linalg.lstsq(np.stack([tm, np.ones_like(tm)], 1), hd.T, rcond=None)[0]
    v = np.linalg.norm(obs[:, -1] - obs[:, -6], axis=1) / 0.5
    tf = DT * np.arange(1, T_FUT + 1)
    th0 = b[:, None]
    small = np.abs(om) < 1e-6
    oms = np.where(small, 1.0, om)[:, None]
    dx = np.where(small[:, None], v[:, None] * tf * np.cos(th0), v[:, None] / oms * (np.sin(th0 + oms * tf) - np.sin(th0)))
    dy = np.where(small[:, None], v[:, None] * tf * np.sin(th0), -v[:, None] / oms * (np.cos(th0 + oms * tf) - np.cos(th0)))
    return obs[:, -1, None] + np.stack([dx, dy], -1)

# ---------- metrics, as in argoverse-api eval_forecasting.py ----------
def score(pred, gt, prob=None):
    """pred (n, K, T, 2). The mode with min FDE is selected; minADE is that mode's ADE; miss if minFDE > 2 m."""
    fde = np.linalg.norm(pred[:, :, -1] - gt[:, None, -1], axis=-1)
    j = fde.argmin(1); n = np.arange(len(gt))
    ade = np.linalg.norm(pred[n, j] - gt, axis=-1).mean(1)
    out = dict(ADE=ade, FDE=fde[n, j], MR=(fde[n, j] > MISS).astype(float))
    if prob is not None:
        out["brierFDE"] = fde[n, j] + (1 - prob[n, j]) ** 2
    return out

# ---------- learned predictor ----------
class MLP(torch.nn.Module):
    def __init__(self, K):
        super().__init__(); self.K = K
        self.net = torch.nn.Sequential(torch.nn.Linear(2 * T_OBS, 128), torch.nn.ReLU(),
                                       torch.nn.Linear(128, 128), torch.nn.ReLU(),
                                       torch.nn.Linear(128, K * T_FUT * 2 + K))
    def forward(self, x):
        y = self.net(x.flatten(1))
        return y[:, :-self.K].view(-1, self.K, T_FUT, 2), y[:, -self.K:]

def train(K, x, y, epochs=60, seed=0):
    torch.manual_seed(seed); model = MLP(K); opt = torch.optim.Adam(model.parameters(), 1e-3)
    g = torch.Generator().manual_seed(seed)
    for ep in range(epochs):
        for idx in torch.randperm(len(x), generator=g).split(256):
            traj, logit = model(x[idx])
            err = ((traj - y[idx, None]) ** 2).sum(-1).mean(-1)          # (batch, K) squared error per mode
            win = err.argmin(1)                                            # winner takes all
            loss = err.gather(1, win[:, None]).mean()
            if K > 1:
                loss = loss + torch.nn.functional.cross_entropy(logit, win)
            opt.zero_grad(); loss.backward(); opt.step()
    return model

torch.set_num_threads(1)
rng = np.random.default_rng(0)
obs_tr, fut_tr, _ = make_agents(20000, rng)
obs_te, fut_te, info = make_agents(4000, rng)

def net_io(obs, fut):
    Rn, o = normalize(obs)
    return (torch.tensor(to_frame(obs, Rn, o) / 10, dtype=torch.float32),
            torch.tensor(to_frame(fut, Rn, o) / 10, dtype=torch.float32), Rn, o)
x_tr, y_tr, _, _ = net_io(obs_tr, fut_tr)
x_te, y_te, Rn_te, o_te = net_io(obs_te, fut_te)

def predict(model):
    with torch.no_grad():
        traj, logit = model(x_te)
    traj = 10 * traj.numpy(); prob = torch.softmax(logit, 1).numpy()
    traj = np.einsum("nji,nktj->nkti", Rn_te, traj) + o_te[:, None, None]   # back to the world frame
    return traj, prob

m1, m6 = train(1, x_tr, y_tr), train(6, x_tr, y_tr)
p1, _ = predict(m1); p6, pr6 = predict(m6)
top = pr6.argmax(1)
rows = {"CV (mean over 2 s)": constant_velocity(obs_te, 19)[:, None],
        "CV (last 0.5 s)": constant_velocity(obs_te, 5)[:, None],
        "constant turn": constant_turn(obs_te)[:, None],
        "MLP K=1": p1,
        "MLP K=6, top mode": p6[np.arange(len(p6)), top][:, None],
        "MLP K=6, all six": p6}
subsets = {"approach, straight": info["approach"] & (info["branch"] == 0),
           "approach, turning": info["approach"] & (info["branch"] > 0),
           "already turning": ~info["approach"], "all": np.ones(len(obs_te), bool)}

print("%-20s" % "minADE / minFDE / MR" + "".join("%26s" % k for k in subsets))
for name, pred in rows.items():
    s = score(pred, fut_te)
    print("%-20s" % name + "".join("%10.2f %6.2f %6.2f" % tuple(s[m][sub].mean() for m in ("ADE", "FDE", "MR"))
                                   for sub in subsets.values()))
print("brier-minFDE_6 (all): %.2f" % score(p6, fut_te, pr6)["brierFDE"].mean())

# Where does the single-mode endpoint land? Compare with this agent's three possible futures.
app = info["approach"]
ends = []
for sgn in (0.0, 1.0, -1.0):
    v, R = info["v"][app], info["R"][app]
    e = rollout(v, sgn * v / R, info["t_on"][app], info["t_off"][app])[:, -1]
    ends.append(np.einsum("nij,nj->ni", info["rot"][app], e) + info["off"][app])
ends = np.stack(ends, 1)                                            # (n, 3, 2) straight, left, right
mean_end = (np.array([0.4, 0.3, 0.3])[None, :, None] * ends).sum(1)
e1 = p1[app, 0, -1]
print("approach agents, MLP K=1 endpoint: %.2f m from the probability-weighted mean of the three branch endpoints,"
      " %.2f m from the nearest branch endpoint" % (np.linalg.norm(e1 - mean_end, axis=1).mean(),
                                                    np.linalg.norm(e1[:, None] - ends, axis=-1).min(1).mean()))
print("MLP K=6 modes whose endpoint is within 2 m of a branch endpoint, per approach agent: mean %.2f of 6;"
      " branches covered: %.2f of 3" % tuple(
          (lambda d: ((d.min(2) < MISS).sum(1).mean(), (d.min(1) < MISS).sum(1).mean()))(
              np.linalg.norm(p6[app, :, -1][:, :, None] - ends[:, None], axis=-1))))

# How much does the history leave open within one branch? Redraw the hidden turn start and radius.
turn = info["approach"] & (info["branch"] > 0)
r2 = np.random.default_rng(1)
v, sgn = info["v"][turn], np.where(info["branch"][turn] == 1, 1.0, -1.0)
R2, t2 = r2.uniform(8, 14, turn.sum()), r2.uniform(0.3, 1.2, turn.sum())
e_alt = rollout(v, sgn * v / R2, t2, t2 + 0.5 * np.pi * R2 / v)[:, -1]
e_alt = np.einsum("nij,nj->ni", info["rot"][turn], e_alt) + info["off"][turn]
print("approach agents that turn: same branch and speed, redrawn turn start and radius ->"
      " endpoint within 2 m of the true one for %.2f of agents" % (np.linalg.norm(e_alt - fut_te[turn, -1], axis=1) < MISS).mean())

# Spread over network initializations for the K=6 model.
for seed in range(5):
    p, pr = (p6, pr6) if seed == 0 else predict(train(6, x_tr, y_tr, seed=seed))
    s = score(p, fut_te)
    d = np.linalg.norm(p[app, :, -1][:, :, None] - ends[:, None], axis=-1)
    print("K=6 seed %d: MR_6 all %.2f, approach turning %.2f; branches covered %.2f of 3" % (
        seed, s["MR"].mean(), s["MR"][turn].mean(), (d.min(1) < MISS).sum(1).mean()))

minADE / minFDE / MR        approach, straight         approach, turning           already turning                       all
CV (mean over 2 s)        0.25   0.38   0.00      6.00  20.03   1.00     16.39  34.41   1.00      8.84  21.18   0.76
CV (last 0.5 s)           0.65   1.15   0.10      6.06  20.04   1.00     12.20  26.48   1.00      7.26  18.15   0.78
constant turn             0.97   2.15   0.48      6.07  20.01   1.00      6.46  14.94   0.98      5.01  13.71   0.87
MLP K=1                   2.02   7.19   1.00      5.45  17.09   1.00      0.85   1.21   0.12      2.77   8.30   0.64
MLP K=6, top mode         0.23   0.34   0.01      5.99  20.03   1.00      0.84   1.14   0.10      2.53   7.69   0.40
MLP K=6, all six          0.19   0.26   0.00      1.86   3.73   0.71      0.83   1.12   0.10      1.04   1.84   0.29
brier-minFDE_6 (all): 2.11
approach agents, MLP K=1 endpoint: 2.07 m from the probability-weighted mean of the three branch endpoints, 7.19 m from the nearest branch endpoint
MLP K=6 modes whose endpoint is within 2 m of a branch endpoint, per approach agent: mean 1.59 of 6; branches covered: 1.59 of 3
approach agents that turn: same branch and speed, redrawn turn start and radius -> endpoint within 2 m of the true one for 0.23 of agents
K=6 seed 0: MR_6 all 0.29, approach turning 0.71; branches covered 1.59 of 3
K=6 seed 1: MR_6 all 0.29, approach turning 0.70; branches covered 1.61 of 3
K=6 seed 2: MR_6 all 0.29, approach turning 0.70; branches covered 1.61 of 3
K=6 seed 3: MR_6 all 0.31, approach turning 0.71; branches covered 1.59 of 3
K=6 seed 4: MR_6 all 0.33, approach turning 0.73; branches covered 1.54 of 3

The physics models split the scenarios cleanly. On straight approaches, constant velocity averaged over the whole history is the best predictor in the table, because its model is exactly right and averaging over 2 s suppresses the tracking noise; the 0.5 s version is noisier for the same reason. Every physics model misses every approach agent that turns, since the turn has not begun. On agents already turning, the constant-turn model roughly halves the error of the better constant-velocity model, yet still misses almost all of them; by our reasoning, it underestimates the rate when the turn began recently, because the fitted chords still contain straight motion, and it keeps turning after the real 90° arc ends.

The single-output network learns what the history does determine: on agents already turning its average displacement is about an eighth of the constant-turn model’s. On approach agents it shows mode averaging. Its endpoint lies about 2 m from the probability-weighted mean of the agent’s three possible endpoints and about 7 m from the nearest of them, so it misses every approach agent, including the 40% who simply drive straight and whom constant velocity predicts almost perfectly. Over all agents its average displacement is lower than that of the three physics models, yet its miss rate is still 0.64: squared error rewards being near every branch at once, which is not what a miss rate measures.

With six outputs the network separates the branches. Its highest-confidence output is nearly identical to constant velocity on straight approaches, the most probable branch, which is why that output alone has a lower ADE and miss rate over all agents than the single-output network; the K=1 numbers of a multimodal model and of a regressor measure different things. All six outputs together cut minFDE on turning approaches from about 17–20 m to 3.7 m, but the miss rate there stays at 0.71, against the expectation written in the plan before the first run (“minFDE_6 well under 2 m and a low MR_6 on all subsets”). The redrawn-parameter line points to a cause: two turns of the same branch and speed, with turn start and radius drawn independently, end within 2 m of each other for only 23% of agents, so, by our reasoning, one output per turning branch, wherever it is placed, misses most turning agents, and covering that spread would take several outputs per branch. In a real benchmark the map would tell where the intersection is and what radius its lanes have, which is the information Argoverse found mattered most at longer horizons (Chang et al., 2019). Each approach agent has about 1.6 outputs near one of its three branch endpoints, one per branch reached; over five network initializations the overall miss rate ranged from 0.29 to 0.33 and the turning miss rate from 0.70 to 0.73, so the result does not depend on a lucky initialization. The brier-minFDE is 0.27 m above the minFDE, the average of $(1-p)^2$ for the probability $p$ the model gave its best output.

One change was made after the first run, prompted by its output. The first version printed the table, the brier-minFDE and the two lines on where endpoints land; it showed a high miss rate for turning agents even with six outputs. Two parts were then added: the line that redraws the hidden turn start and radius, to measure how much the history leaves undetermined within a branch, and the loop that retrains the six-output network with four more initializations and prints its miss rates. Both additions were first run as separate diagnostic scripts and then added to the example. The data, models, losses, training settings and the first version’s printed lines are unchanged. The check made before the first run, that the labels are multimodal and learnable, measured how far apart the branches end but not how much the endpoints spread within one branch; that spread is what the added line measures.

From rasters to vectors

The first learned scene encoders drew the scene as an image and used a convolutional network. MultiPath renders a top-down 400 × 400 pixel tensor covering 80 m × 80 m with 15 channels: road semantics, distance to the road edge, speed limit, five frames of traffic-light states and five frames of vehicle boxes; a ResNet-50 with a quarter of the usual depth channels extracts scene features, and an 11 × 11 patch around each agent, rotated into the agent’s frame, feeds the per-agent head (Chai et al., 2019). The appeal is that one image represents spatial relations and road semantics simply; the bird’s-eye-view rasters are the same kind of grid that BEV perception builds from cameras.

VectorNet replaced the image with the map’s own structure (Gao et al., 2020). Lanes, crosswalks and agent histories become polylines, each a sequence of vectors whose node features are the start and end points, attributes (type, timestamp, speed limit) and a polyline identifier. A subgraph network encodes each polyline with shared MLPs and max pooling (a generalization of PointNet, in the authors’ words), and a global graph with self-attention over all polylines models their interactions; an auxiliary task masks out polyline features and reconstructs them, which the authors relate to BERT. Its authors argue that rasterization is lossy and that convolutions have limited receptive fields. In the paper’s comparison on the Argoverse validation set with a single predicted trajectory (Table 4; its test-set results are in a separate table), VectorNet with the auxiliary task reached a displacement error at 3 s of 3.67 m against 4.49 m for their best rendered ResNet-18 baseline, with 72K parameters against 246K and 0.041 GFLOPs per agent against 10.56 GFLOPs for the backbone (Gao et al., 2020).

LaneGCN builds a graph of lane segments and convolves along it (Liang et al., 2020). Its LaneConv operator has a separate weight matrix for each connection type (predecessor, successor, left neighbour, right neighbour) and dilated versions that use powers of the predecessor and successor adjacency matrices to reach far along a lane in one step. A fusion network then passes information actor to lane, lane to lane, lane to actor and actor to actor. On the Argoverse test set at $K = 6$ it reported minADE 0.87 m, minFDE 1.36 m and a miss rate of 0.16, against 0.93 m, 1.49 m and 0.19 for the leaderboard’s first entry at submission time (Liang et al., 2020, Table 1).

Goals and anchors

Anchor- and goal-based methods make the discrete part of the future explicit. MultiPath’s anchors are whole trajectories fixed in advance. TNT predicts targets instead: points the agent may reach at the horizon (Zhao et al., 2020). It samples about $N = 1000$ candidate targets along lane centrelines (at least one per metre) or on a grid for pedestrians, scores them with a classifier and regresses an offset for each, keeps the top $M = 50$, generates one trajectory per target with a unimodal regressor, and finally scores the trajectories and selects $K = 6$ while suppressing near-duplicates. Its authors argue that targets are lower-dimensional than anchor trajectories and therefore easier to estimate, and that given a target the remaining motion is close to unimodal; replacing the regressor with a CVAE that sampled ten trajectories per target improved minADE$_6$ only from 0.73 to 0.71 m. On Argoverse validation, TNT reported minFDE$_6$ 1.29 m, minADE$_6$ 0.73 m and miss rate 0.09, against 1.68, 0.80 and 0.14 for MultiPath reimplemented with a VectorNet encoder (Zhao et al., 2020, Table 6). On the test set it reported 1.54 m, 0.94 m and 0.13 against 1.42 m, 0.97 m and 0.13 for the 2020 challenge winner; the text calls TNT “on par or better”, while the table shows the winner with the lower minFDE.

Transformers

The Argoverse 2 authors report that graph-based attention methods dominated their three Argoverse forecasting competitions (Wilson et al., 2021), and transformers carried attention through the whole model. Wayformer (Nayakanti et al., 2023) is a family of attention-based architectures for motion forecasting from Waymo; only its abstract was read for this article. The Motion Transformer, MTR (Shi et al., 2022), encodes agent and map polylines, each first summarized PointNet-style as in VectorNet, with a transformer encoder whose self-attention is local (each polyline attends to its nearest neighbours), and decodes with pairs of queries. Each static intention query is the learnable embedding of an intention point, one of 64 found by k-means on the endpoints of training trajectories, so each query takes charge of one region of possible destinations, much as an anchor does; a dynamic searching query refines the trajectory by gathering map features around its current prediction, layer by layer. Training uses hard assignment: the query whose intention point is nearest the ground-truth endpoint receives a Gaussian negative log-likelihood loss. Six outputs are chosen from the 64 by non-maximum suppression on endpoint distance with a 2.5 m threshold. An auxiliary head also regresses the future trajectory of every agent in the scene, and these forecasts are encoded as extra context, so the decoder sees where neighbours are going as well as where they have been.

On the WOMD test set (marginal), MTR reported minADE 0.6050 m, minFDE 1.2207 m, miss rate 0.1351 and mAP 0.4129 (Shi et al., 2022, Table 1). Its variant MTR-e2e, with only six queries and the winner chosen by predicted trajectory rather than by intention point, had better minADE and minFDE on validation (0.5160 and 1.0404 against 0.6046 and 1.2251) but lower mAP (0.3245 against 0.4164), which the authors attribute to the explicit meaning each intention query learns. The two metrics disagree about which model is better, which is the distinction WOMD’s mAP was designed to draw (see below).

Joint prediction

Marginal predictors can be combined into joint ones. For WOMD’s interactive split, the dataset authors took their marginal model’s $K$ futures for each of the two agents and kept the top $K$ of the $K^2$ pairs by the product of predicted probabilities; MTR did the same, choosing six of 36 pairs (Ettinger et al., 2021; Shi et al., 2022). By our reasoning, the product treats the two agents as independent, which is exactly what an interaction violates. In WOMD’s direct comparison, an LSTM trained to predict the pair jointly (features of the two agents concatenated, pairs matched jointly in the loss, one confidence per pair) had a vehicle minADE at 8 s of 3.81 m against 4.16 m for the same model trained marginally, and mAP 0.03 against 0.01, on the interactive validation set (Ettinger et al., 2021, Table 4). The Argoverse 2 evaluation code includes functions that score $K$ predicted “worlds”, each a joint future of all scored actors, by average and final displacement, misses and collisions between predicted actors (av2-api, metrics.py).

Datasets and Benchmarks

Benchmark	History / horizon	Rate	Size	Predicted agents
Argoverse 1 (2019)	2 s / 3 s	10 Hz	333,441 sequences of 5 s, Miami and Pittsburgh	one vehicle per sequence
Argoverse 2 (2021)	5 s / 6 s	10 Hz	250,000 scenarios of 11 s, six cities	focal agent and scored actors, five dynamic classes
Waymo Open Motion (2021)	1 s / 8 s	10 Hz (predictions at 2 Hz)	104k segments of 20 s cut into 9.1 s scenes, six cities	up to 8 per scene (standard); 2 interacting (interactive)
nuScenes prediction	up to 2 s / 6 s	2 Hz	1000 scenes of 20 s, Boston and Singapore	vehicles

Argoverse 1. The forecasting set was mined from 1006 driving hours for vehicles at intersections, turning, changing lanes or in dense traffic; each sequence has one challenging trajectory to predict, with the other tracks kept as context, and the train, validation and test splits come from disjoint parts of the two cities (Chang et al., 2019). The paper gives 333,441 sequences, split 211,691 / 41,146 / 80,604. The official repository’s README gives 327,790 (208,272 / 40,127 / 79,391), LaneGCN’s paper 205,942 / 39,472 / 78,143, and the Argoverse 2 and WOMD comparison tables list 324k; the sources do not explain the differences. The benchmark allows up to six predictions per agent, and its server ranked entries by minFDE at $K = 6$ when LaneGCN was submitted in 2020 (Liang et al., 2020).

Argoverse 2. The motion forecasting dataset has 250,000 non-overlapping scenarios, split 80/10/10 at random, from Austin, Detroit, Miami, Palo Alto, Pittsburgh and Washington, D.C.; each scenario has a local vector map, a focal agent fully observed for all 11 s, and scored actors, the other fully observed tracks within 30 m of the ego vehicle (Wilson et al., 2021). The focal agents were chosen by an “interestingness” score over object category, kinematics, map complexity, social context and relation to the ego vehicle, and the authors show that higher scores go with higher miss rates for a learned model. They designed the dataset to be harder after performance on Argoverse 1 began to plateau: longer horizon, more object classes, more tail events. Their baselines show the effect: constant velocity had minFDE 17.44 m at $K = 1$ on vehicle-like objects, and the graph-attention model WIMP still had a miss rate of 0.42 at $K = 6$ (Wilson et al., 2021, Table 4).

Waymo Open Motion Dataset. Interactions were mined from a larger corpus by composing hand-written predicates (for example, two agents crossing paths within a time gap) in a relational database; the mined pairs include merges, lane changes, unprotected turns, intersection turns, pedestrian–vehicle and cyclist–vehicle interactions, close-proximity interactions and high accelerations (Ettinger et al., 2021). The 104k 20-second segments (574 hours, 1750 km of unique roadway) are cut into 9.1 s scenes and split 70/15/15. The standard validation and test sets ask for up to 8 agents per scene, chosen with a bias against constant-velocity and straight paths; the interactive sets ask for the two mined interacting agents. MTR’s paper counts 487k training scenes and about 44k each for validation and test (Shi et al., 2022).

nuScenes prediction. The nuScenes dataset has 1000 scenes of 20 s with 3D boxes annotated at 2 Hz (Caesar et al., 2020). The prediction task, defined in the development kit rather than in the CVPR paper, asks for up to 25 modes per agent with probabilities, 6 s long at 2 Hz, from at most 2 s of history; because test annotations are not released, the leaderboard is ranked on the validation set and the top five entries’ code is run on the test set to pick winners (nuScenes devkit README).

Evaluation Metrics

All distances are Euclidean in metres in the bird’s-eye-view plane. For one agent with ground truth $\hat{\mathbf{s}}_{1:T}$ and predictions $\mathbf{s}^k_{1:T}$:

$$\mathrm{ADE}_k = \frac{1}{T} \sum_{t=1}^{T} \lVert \hat{\mathbf{s}}_t - \mathbf{s}^k_t \rVert_2 , \qquad \mathrm{FDE}_k = \lVert \hat{\mathbf{s}}_T - \mathbf{s}^k_T \rVert_2 .$$

The benchmarks differ in which mode they take the minimum over, and in what counts as a miss.

minADE$_K$ and minFDE$_K$. Argoverse 1’s evaluation code keeps the $K$ most probable predictions (the first $K$ if no probabilities are given), finds the mode $k^*$ with the smallest FDE, and reports $\mathrm{minFDE}_K = \mathrm{FDE}_{k^*}$ and $\mathrm{minADE}_K = \mathrm{ADE}_{k^*}$: minADE is the ADE of the best-endpoint mode, not the smallest ADE (argoverse-api, eval_forecasting.py). WOMD’s paper and its metric definitions take the minimum of each separately, $\min_k \mathrm{ADE}_k$ and $\min_k \mathrm{FDE}_k$, computed up to each measurement step (Ettinger et al., 2021; Waymo, motion_metrics.proto); so does nuScenes over the $k$ most probable modes, reported for $k = 1, 5, 10$ in the ICRA 2020 configuration (nuScenes devkit). The Argoverse 2 module computes ADE and FDE per predicted trajectory; how its server selects the mode is not in that file.

Miss rate. The fraction of agents for which no prediction counts as a match:

• Argoverse 1 and 2: a miss when $\mathrm{minFDE}_K > 2$ m (argoverse-api; av2-api default miss_threshold_m = 2.0; Wilson et al., 2021). The 2019 paper defines no miss rate; LaneGCN and TNT state the 2 m definition for the leaderboard (Liang et al., 2020; Zhao et al., 2020).
• nuScenes: a prediction is a miss when its largest pointwise distance from the ground truth over the whole horizon reaches 2 m; an agent counts as missed when all of its $k$ most probable predictions miss. The README says “greater than 2 meters”, while the code tests >= tolerance (nuScenes devkit).
• WOMD: the error at the measurement step is rotated into the ground-truth agent’s heading at that step and split into longitudinal and lateral parts, each compared with its own threshold: 2 and 1 m at 3 s, 3.6 and 1.8 m at 5 s, 6 and 3 m at 8 s (Ettinger et al., 2021; Waymo, motion tutorial). Both thresholds are scaled by the agent’s speed at the current time: by 0.5 below 1.4 m/s, by 1 above 11 m/s, and linearly in between. The paper writes separate scale factors $\gamma(v_x)$ and $\gamma(v_y)$; the official code uses one factor from the speed $\sqrt{v_x^2 + v_y^2}$ and divides the displacements by it (Waymo, motion_metrics.cc). For joint predictions a configuration is a match only if every agent in it matches.

Probability-weighted metrics. Argoverse 1’s code also reports brier-minFDE: with the top-$K$ probabilities renormalized to sum to one and $p_{k^*}$ the probability of the best-endpoint mode,

$$\text{brier-minFDE}_K = \mathrm{FDE}_{k^*} + (1 - p_{k^*})^2 ,$$

averaged over agents, and likewise brier-minADE; its p-minFDE adds $\min(-\log p_{k^*}, -\log 0.05)$ instead, and p-MR counts $1 - p_{k^*}$ for an agent that is not missed and 1 for one that is (argoverse-api, eval_forecasting.py). The Argoverse 2 functions compute $(1-p)^2$ per trajectory, renormalizing only if asked (av2-api, metrics.py). These terms penalize a forecast whose best mode was given little probability, which minADE and minFDE ignore.

mAP and soft mAP (WOMD). WOMD’s authors observed that minADE improves steadily as more predictions are allowed, whatever their confidences, while mAP peaked, at 3 predictions for 3 s and 5 s and at 6 for 8 s, in their best LSTM baseline (Ettinger et al., 2021, Figure 6). The mAP treats each predicted trajectory as a detection: across the dataset, predictions are sorted by confidence, a prediction matching the ground truth by the miss-rate definition is a true positive, only the highest-confidence matching prediction of each object counts, and further matches are false positives, as in average precision for object detection. Precision–recall curves are computed separately for buckets of ground-truth trajectory shape and the mean of their APs is reported; the paper lists eight shapes (straight, straight-left, straight-right, left, right, left u-turn, right u-turn, stationary), and the official code bins right u-turns with right turns (Ettinger et al., 2021; Waymo, motion_metrics.cc). Soft mAP is the same except that further matches of an already matched object are ignored rather than counted as false positives (Waymo, motion_metrics.proto). The proto’s comment describes the mAP as thresholding the FDE, whereas the code it documents uses the longitudinal and lateral match above.

Map and collision metrics. Argoverse 1’s code reports drivable area compliance, the fraction of each agent’s predicted trajectories that stay entirely within the drivable area, averaged over agents (argoverse-api), and nuScenes an off-road rate (nuScenes devkit). WOMD’s overlap rate counts an agent’s highest-confidence prediction as an overlap if its box intersects any other object valid at the current time, or any of the jointly predicted trajectories, at any step up to the measurement step (Ettinger et al., 2021).

Applications

Forecasts feed the motion planner of a modular driving stack, the chain described in autonomous driving perception. MultiPath’s authors argue that a planner needs both a small weighted set of discrete futures, to react to hypotheses such as yielding or following, and a closed-form likelihood, for probabilistic queries such as the expected risk of collision in a region of space and time (Chai et al., 2019); TNT’s authors note that its three stages mirror the steps of traditional robotics forecasting and planning systems (Zhao et al., 2020). Planning-oriented models that train perception, prediction and planning together use forecasting as an internal module, and their evaluation is part of end-to-end autonomous driving. The same methods are applied to pedestrians: TNT reports results on the Stanford Drone Dataset and an in-house pedestrians-at-intersections dataset, and MultiPath on Stanford Drone and CARLA (Zhao et al., 2020; Chai et al., 2019).

Open Problems

• Metrics that reward coverage over calibration. minADE and minFDE improve with any additional prediction, so a method can gain by spreading modes; WOMD’s mAP and Argoverse’s Brier variants add confidence, and the MTR results show the rankings can then disagree (Ettinger et al., 2021; Shi et al., 2022).
• Joint, interactive futures. Products of marginals assume independence; WOMD’s joint baseline improved on its marginal counterpart, and its authors present joint training as an early result to build on (Ettinger et al., 2021).
• The long tail. Benchmarks mine hard cases because most driving is easy, and the Argoverse 2 authors designed their dataset to reward methods that ensure safety in tail events rather than average performance on “easy miles” (Wilson et al., 2021).
• Dependence on perception and maps. Benchmarks give tracks and maps from offline or offboard pipelines; a deployed predictor sees online tracking errors and maps that may be out of date, and the Argoverse 2 authors list imperfect tracking among their dataset’s limitations (Wilson et al., 2021).
• Open-loop evaluation. Forecasts are scored against logged futures, which did not react to the predicting vehicle; in our reading, whether a better forecast leads to better driving can be measured only when forecasting is evaluated inside a planner, the subject of closed-loop evaluation in end-to-end driving.