Bird's-Eye-View Perception

Bird’s-eye-view (BEV) perception turns what a vehicle’s cameras see into a map of the ground around it, seen from above. The output is a grid in metres centred on the vehicle, in which every cell holds a learned feature vector or a label such as “drivable”, “lane boundary” or “car”. Object detection, map segmentation, tracking and motion planning can then all work in that one frame. The hard part is the change of viewpoint: a camera records a perspective projection, so going from pixels to ground positions means recovering the depth that the projection discarded. The methods differ in how they do it: by assuming the world is flat (inverse perspective mapping), by predicting a distribution over depth for every pixel and moving features along the camera ray (depth-based lifting), or by letting grid cells or object queries look up the image features that project onto them (attention-based view transformation).

Problem Definition

The input is a set of images from $n$ cameras, each with known intrinsic and extrinsic camera parameters. The output is a raster on a horizontal plane in the ego vehicle’s frame. Philion and Fidler (2020) state the problem in this form: given images $X_k \in \mathbb{R}^{3 \times H \times W}$ with extrinsics $E_k$ and intrinsics $I_k$, find a representation $y \in \mathbb{R}^{C \times X \times Y}$ in the BEV frame. The grid’s extent and cell size are design choices: Lift-Splat-Shoot uses 200 × 200 cells of 0.5 m covering ±50 m, and BEVFormer on nuScenes uses 200 × 200 cells of 0.512 m covering ±51.2 m (Li et al., 2022).

Several properties make the top-down frame attractive:

• It is metric. A cell is the same size near and far, so an object’s footprint does not shrink with distance, and distances between objects can be read off directly. Roddick, Kendall and Cipolla (2018) argued that in the perspective image the appearance and scale of objects vary drastically with depth, and that a domain with consistent scale makes 3D detection easier.
• It separates what the image superimposes. Cars queued in one lane overlap in a forward-facing image but occupy different cells from above. On a road, objects rarely stand on top of each other, so little is lost by collapsing height (our reasoning; the open problems below return to what is lost).
• It is a common frame for fusion. Every camera, every past frame (after compensating ego-motion), and every other sensor can be written into the same grid. Roddick and Cipolla (2020) used a Bayesian occupancy grid for this reason: evidence from several cameras and time steps accumulates by simple updates.
• It is the planner’s frame. Philion and Fidler (2020) motivated their model by the mismatch between image-frame predictions and the ego frame in which planning works, and trained a planner directly on the BEV output.

Input and Output

The usual input is a surround camera rig with its calibration. nuScenes provides six cameras covering 360 degrees, with five radars and one LiDAR on the same vehicle (Caesar et al., 2020); the Waymo Open Dataset has five cameras, which Li et al. note in the arXiv version of the BEVFormer paper cover only about 252° horizontally. Temporal methods also take the ego pose of past frames. Some methods use LiDAR during training only, as a source of depth labels, and remain camera-only at test time (Li et al., 2023).

The view transformer’s output is a BEV feature map, for instance $B_t \in \mathbb{R}^{H \times W \times C}$ in BEVFormer, which task heads decode into:

• 3D object detection: boxes with centre, size, yaw, velocity and attributes, as nuScenes asks for.
• BEV semantic segmentation: static map layers such as drivable area and lane boundaries, rasterized from an HD map, and object layers such as “vehicle”, obtained by projecting annotated 3D boxes onto the ground (Philion & Fidler, 2020).
• Planning inputs: a cost map over the grid, scored against candidate trajectories.

Variants

• Monocular or surround view. Early learned methods such as the Orthographic Feature Transform (OFT) and Pyramid Occupancy Networks transform one image at a time (Pyramid Occupancy Networks then accumulate several cameras in an occupancy grid); Lift-Splat-Shoot and later methods consume the whole rig at once.
• Dense grid or sparse queries. Most methods build a dense BEV feature map. DETR3D and PETR skip it and detect objects with a small set of queries in 3D space.
• Single frame or temporal. Temporal methods carry BEV features from earlier frames, aligned by ego-motion, which helps with velocity and occlusion.
• Camera-only or fused. When LiDAR or radar features are rasterized into the same grid, the BEV map becomes the meeting point of multimodal fusion, the subject of sensor fusion for autonomous driving. This article concerns the camera branch.

Challenges

• Depth is missing. Each pixel defines a ray, and the image does not say where along the ray the surface lies. Monocular depth estimation can supply an estimate, but its errors grow with range and propagate into the BEV map.
• Resolution falls with distance. A pixel covers more ground the farther away it looks, so the far part of the grid is filled from few pixels. The first example below measures this.
• Occlusion. The space behind a vehicle is not observed, but map and box labels cover it, so a network trained on them learns to fill it in. Philion and Fidler (2020) show the network extrapolating lanes and the body of a partly visible car when a camera is removed.
• Calibration. View transformations depend on the extrinsics, which shift during driving and between datasets. Philion and Fidler (2020) trained with noisy extrinsics and with randomly dropped cameras to make the model robust to both.
• Cost. A dense grid of 40,000 cells and a lifted point cloud of depth bins times pixels times cameras are expensive to build at every frame, which has driven much engineering in the field.

Approaches

Inverse perspective mapping

If every pixel shows a point on a flat ground plane, the change of view is exact and needs no learning. With the world frame chosen so that the ground is $Z = 0$ and the camera is $P = K[R \mid \mathbf{t}]$, a ground point projects as

$$\lambda \begin{pmatrix} u \\ v \\ 1 \end{pmatrix} = K \begin{bmatrix} \mathbf{r}_1 & \mathbf{r}_2 & \mathbf{r}_3 & \mathbf{t} \end{bmatrix} \begin{pmatrix} X \\ Y \\ 0 \\ 1 \end{pmatrix} = \underbrace{K \begin{bmatrix} \mathbf{r}_1 & \mathbf{r}_2 & \mathbf{t} \end{bmatrix}}_{H_g} \begin{pmatrix} X \\ Y \\ 1 \end{pmatrix},$$

so the ground and the image are related by the homography $H_g$, the camera matrix with its third column removed. Inverse perspective mapping (IPM) resamples the image onto a ground grid: for each cell centre $(X, Y)$ it computes the pixel $H_g(X, Y, 1)^\top$ and reads the image there, which is backward image warping. Mallot et al. (1991) used the remapping for obstacle detection from optical flow: as their abstract describes it, disturbances of the expected motion field are detected by a mechanism based on inverse perspective mapping.

The flat-world assumption is also IPM’s failure mode. Take a camera at height $h$ above the point on the ground beneath it, and a scene point at horizontal offset $(X, Y)$ from that point and height $Z$. The ray through it meets the ground at

$$(X', Y') = \frac{h}{h - Z}\,(X, Y), \qquad Z < h,$$

so every point above the ground is drawn farther away than it is, by a factor that grows without bound as $Z$ approaches the camera height. A vertical surface becomes a long streak pointing away from the camera, and points at or above the camera height never reach the ground at all. Roddick and Cipolla (2020) report this in their IPM baseline, which maps a semantic segmentation of the image onto ground planes fitted to LiDAR points: on the nuScenes split they used, it reached 40.1 IoU for drivable area but 4.9 for cars and 0.6 for pedestrians, because predictions are elongated along camera rays. IPM remains useful for things that really lie on the ground, such as road surface and the lane markings that lane detection looks for, when the ground is close to planar and the camera’s pitch and height are known.

Example: inverse perspective mapping of a rendered road

The code below renders a road scene by casting a ray through every pixel of a 320 × 240 camera, 1.5 m above the ground and pitched 8° down: three lane lines 3.5 m apart and a car-sized box, 1.2 m tall, in the right lane from 14 m to 18.5 m ahead. It builds $H_g$ from the known intrinsics and pose and resamples the label image onto a 5 cm ground grid. It then checks lane widths in the image and in the grid, compares IPM with the true ground labels where the ground is visible, and measures how much ground the box covers after IPM. It runs with NumPy 2.3.5 and SciPy 1.16.3.

import numpy as np
from scipy.ndimage import map_coordinates

# Camera: 1.5 m above flat ground, pitched 8 degrees down, 320x240 pixels.
# World: X right, Y forward, Z up. Camera: x right, y down, z forward.
W, H, f, h, pitch = 320, 240, 300.0, 1.5, np.deg2rad(8.0)
K = np.array([[f, 0, W / 2], [0, f, H / 2], [0, 0, 1]])
s, c = np.sin(pitch), np.cos(pitch)
R = np.array([[1, 0, 0], [0, -s, -c], [0, c, -s]])   # rows: right, down, forward
C = np.array([0.0, 0.0, h])                           # camera centre
t = -R @ C

# Scene: three lane lines (X = -3.5, 0, 3.5 m), road |X| < 5.5 m, grass outside,
# and a car-sized box in the right lane, 1.2 m tall.
LINES = np.array([-3.5, 0.0, 3.5])
BOX = np.array([[0.85, 2.65], [14.0, 18.5], [0.0, 1.2]])   # X, Y, Z extents (m)

def ground_label(X, Y):
    lab = np.where(np.abs(X) < 5.5, 2, 1)                    # 1 grass, 2 road
    on_line = np.min(np.abs(X[..., None] - LINES), axis=-1) < 0.075
    return np.where(on_line, 3, lab)                         # 3 lane marking

# Ray-cast every pixel: nearest hit of the box (slab test) or the ground plane.
u, v = np.meshgrid(np.arange(W) + 0.5, np.arange(H) + 0.5)
rays = np.stack([u, v, np.ones_like(u)], -1) @ np.linalg.inv(K).T @ R  # world dirs
with np.errstate(divide="ignore", invalid="ignore"):
    lam_g = np.where(rays[..., 2] < 0, -h / rays[..., 2], np.inf)
    t0 = (BOX[:, 0] - C) / rays
    t1 = (BOX[:, 1] - C) / rays
    t_near = np.max(np.minimum(t0, t1), -1)
    t_far = np.min(np.maximum(t0, t1), -1)
lam_b = np.where((t_near <= t_far) & (t_near > 0), t_near, np.inf)
lam = np.minimum(lam_g, lam_b)
P = C + lam[..., None] * rays                              # 3D hit points
label = np.where(np.isinf(lam), 0, ground_label(P[..., 0], P[..., 1]))
label = np.where(lam_b < lam_g, 4, label)                  # 0 sky, 4 car
depth = np.where(np.isinf(lam), np.inf, lam * (rays @ R.T)[..., 2])  # z-depth (m)

# Inverse perspective mapping: ground point (X, Y, 0) -> pixel by H = K [r1 r2 t].
Hg = K @ np.column_stack([R[:, 0], R[:, 1], t])
np.set_printoptions(suppress=True, precision=1)
print("ground-plane homography, scaled so H[2,2] = 1:\n", Hg / Hg[2, 2])
cell = 0.05
Xs = np.arange(-6, 6, cell) + cell / 2
Ys = np.arange(4, 40, cell) + cell / 2
GX, GY = np.meshgrid(Xs, Ys)                               # BEV grid, rows = Y
q = np.stack([GX, GY, np.ones_like(GX)], -1) @ Hg.T
px, py = q[..., 0] / q[..., 2], q[..., 1] / q[..., 2]
bev = map_coordinates(label, [py - 0.5, px - 0.5], order=0, mode="constant", cval=0)

def runs(row, value, coords):
    """Centres of consecutive runs of `value` along a 1D label row."""
    idx = np.flatnonzero(row == value)
    groups = np.split(idx, np.flatnonzero(np.diff(idx) > 1) + 1)
    return np.array([coords[g].mean() for g in groups if len(g)])

for Yd in (8.0, 25.0):
    row_img = int((Hg @ [0, Yd, 1])[1] / (Hg @ [0, Yd, 1])[2])   # image row of Y = Yd
    xi = runs(label[row_img], 3, u[0])[:2]                # lines X = -3.5 and 0
    xb = runs(bev[np.argmin(np.abs(Ys - Yd))], 3, Xs)[:2]
    print(f"Y = {Yd:4.1f} m: left lane is {xi[1] - xi[0]:5.1f} px wide in image row"
          f" {row_img}, {xb[1] - xb[0]:.2f} m wide in the BEV grid")

# Ground the camera sees directly: IPM is exact up to the image resolution.
visible = np.isfinite(lam_g) & (lam_g <= lam_b)
seen = map_coordinates(visible.astype(int), [py - 0.5, px - 0.5], order=0) == 1
truth = ground_label(GX, GY)
for lo, hi in ((4, 15), (15, 40)):
    m = seen & (GY >= lo) & (GY < hi)
    print(f"visible ground, Y = {lo}..{hi} m: IPM label = true label in "
          f"{np.mean(bev[m] == truth[m]):.1%} of cells")

car = bev == 4
print(f"car in IPM: {car.sum() * cell**2:.1f} m^2 over Y = {GY[car].min():.2f}"
      f"..{GY[car].max():.2f} m; true footprint {1.8 * 4.5:.1f} m^2 over Y = 14..18.5 m")
print(f"top edge (Z = 1.2 m) lands on the ground at Y = {14.0 * h / (h - 1.2):.0f} m")

Its output is:

ground-plane homography, scaled so H[2,2] = 1:
 [[1437.1  759.   160. ]
 [   0.   369.2 2254.6]
 [   0.     4.7    1. ]]
Y =  8.0 m: left lane is 128.5 px wide in image row 133, 3.50 m wide in the BEV grid
Y = 25.0 m: left lane is  43.0 px wide in image row 96, 3.57 m wide in the BEV grid
visible ground, Y = 4..15 m: IPM label = true label in 98.2% of cells
visible ground, Y = 15..40 m: IPM label = true label in 96.3% of cells
car in IPM: 93.4 m^2 over Y = 14.07..39.97 m; true footprint 8.1 m^2 over Y = 14..18.5 m
top edge (Z = 1.2 m) lands on the ground at Y = 70 m

The left lane is 3.5 m wide everywhere, yet in the image it spans about three times as many pixels at 8 m as at 25 m, which is the convergence of parallel lines towards the vanishing point. After IPM the lane lines are parallel again: the width is 3.50 m at 8 m and 3.57 m at 25 m, where one pixel already covers about 8 cm across, so the measurement is limited by the source image rather than by the mapping. The same limit explains the agreement with the true ground labels, which is lower beyond 15 m: the mapping is exact, but the 15 cm lane lines are only one or two pixels wide at that range, and nearest-neighbour sampling of a coarse image cannot place their edges to 5 cm.

The box shows the failure. Its true footprint is 8.1 m², but IPM paints “car” over more than ten times that area, from its front face at 14 m to the edge of the grid at 40 m. The streak is the formula above at work: the box’s top edge, 1.2 m up, is drawn where its ray meets the ground, at 70 m. The ground behind the box, which the camera never sees, is covered by the streak.

Depth-based lifting

The alternative is to estimate depth and place each pixel’s features at the right distance. Pseudo-LiDAR approaches, as Philion and Fidler (2020) summarize them, train one network for monocular depth and a separate one for detection in the bird’s-eye view, typically on a point cloud built from the predicted depth. OFT went the other way: it projects a fixed volume of voxels into the image and gives each voxel the image features it lands on, so a pixel contributes the same feature to every voxel along its ray, whatever the depth of the surface at that pixel (Roddick, Kendall & Cipolla, 2018; Philion & Fidler, 2020).

Lift-Splat-Shoot (Philion & Fidler, 2020) learns a depth distribution instead. For each pixel $p$ the image network predicts a context vector $\mathbf{c} \in \mathbb{R}^C$ and a categorical distribution $\boldsymbol{\alpha}$ over a fixed set of depths $D$ (4 to 45 m in steps of 1 m in their experiments). The lift places one point per depth bin along the pixel’s ray, with feature

$$\mathbf{c}_d = \alpha_d\, \mathbf{c}, \qquad d \in D,$$

the outer product of the distribution and the context. A one-hot $\boldsymbol{\alpha}$ puts the whole feature at a single depth, as pseudo-LiDAR does; a uniform $\boldsymbol{\alpha}$ spreads it equally along the ray, as OFT does; anything in between lets the network hedge when depth is ambiguous. The splat uses the camera parameters to move all points from all cameras into the ego frame and sum-pools them into “pillars”, vertical columns of unbounded height, one per BEV cell, giving a $C \times X \times Y$ tensor that a BEV network processes. To pool efficiently they sort points by pillar, take a cumulative sum and subtract it at pillar boundaries (the “cumsum trick”), with an analytic gradient. The shoot step scores 1,000 template trajectories, clustered from expert driving, against a learned cost map. Depth is never supervised directly: it is learned only through the BEV losses.

On the nuScenes validation set, Lift-Splat reached an IoU of 32.06 for car and 32.07 for vehicle segmentation, against 29.72 and 30.05 for their OFT re-implementation (Philion & Fidler, 2020, Table 1), and 72.94 for drivable area and 19.96 for lane boundaries (Table 2). The paper’s comparison with a LiDAR-based PointPillars model (Table 5) lists Lift-Splat’s map scores as 70.81 and 19.58 instead; the text does not explain the difference, and the ECCV version and arXiv v1 agree on both tables.

BEVDet (Huang et al., 2021) packaged the idea as a detector with four modules (image encoder, view transformer, BEV encoder and detection head) and added data augmentation in the BEV space, since augmentations of the image no longer regularize the BEV encoder once the view transformer decouples the two spaces. BEVDepth (Li et al., 2023) asked how good the learned depth was. Replacing it with depth from projected LiDAR points raised a Lift-Splat-style detector from 0.327 to 0.515 NDS on the nuScenes validation set (their Table 1), so they supervised the depth distribution with LiDAR depth during training and made the depth network aware of the camera parameters. With these changes, multi-frame fusion and a ConvNeXt backbone trained on the train and val sets, and with test-time augmentation, they reported 0.609 NDS on the nuScenes test set (Table 8).

Example: a toy lift and splat

The second part continues the script on the same rendered image, at every second pixel. Each pixel gets a one-hot context vector (ground, lane marking or car) and a depth distribution over Lift-Splat-Shoot’s bins from 4 to 45 m. Instead of being learned, the distributions are built from the true depth: a sharp Gaussian, a blurred one, and a uniform distribution that ignores depth, the OFT-like extreme. The features are lifted along each ray, splatted into 0.5 m pillars with the cumulative-sum trick (checked against a direct scatter-add), and scored by how much of the car’s feature mass lands within 0.5 m of its true footprint and how much of the lane mass lands within 0.5 m of a lane line.

# ---- a toy lift-splat on the same image (continues the script above) ----
# Context: one-hot [ground, lane, car] per pixel, at every 2nd pixel; sky has none.
lab2, dep2 = label[::2, ::2], depth[::2, ::2]
uu, vv = u[::2, ::2], v[::2, ::2]
ctx = np.stack([np.isin(lab2, (1, 2)), lab2 == 3, lab2 == 4], -1).astype(float)
ctx[dep2 > 45] = 0                                   # beyond the depth bins
bins = np.arange(4.0, 46.0, 1.0)                     # D: 4..45 m in 1 m steps
pts = (np.stack([uu, vv, np.ones_like(uu)], -1) @ np.linalg.inv(K).T)[..., None, :] \
      * bins[:, None]                                # camera coords, (h, w, D, 3)
Pw = (pts - t) @ R                                   # world coords: R^T (x - t)
cell2 = 0.5                                          # BEV: X -10..10, Y 0..40 m
ix = np.floor((Pw[..., 0] + 10) / cell2).astype(int)
iy = np.floor(Pw[..., 1] / cell2).astype(int)
inside = (ix >= 0) & (ix < 40) & (iy >= 0) & (iy < 80)
bin_id = (iy * 40 + ix)[inside]

def splat(feat):
    """Sum-pool lifted features into BEV pillars with the cumulative-sum trick."""
    order = np.argsort(bin_id, kind="stable")
    ids, f_sorted = bin_id[order], feat[order]
    csum = np.cumsum(f_sorted, axis=0)
    last = np.r_[ids[1:] != ids[:-1], True]          # last point of each pillar
    sums = np.diff(np.vstack([np.zeros((1, feat.shape[1])), csum[last]]), axis=0)
    out = np.zeros((80 * 40, feat.shape[1]))
    out[ids[last]] = sums
    return out.reshape(80, 40, -1)

GX2, GY2 = np.meshgrid(np.arange(-10, 10, cell2) + 0.25, np.arange(0, 40, cell2) + 0.25)
foot = (GX2 > 0.35) & (GX2 < 3.15) & (GY2 > 13.5) & (GY2 < 19.0)   # car +- 0.5 m
near_line = np.min(np.abs(GX2[..., None] - LINES), -1) < 0.5

for name, sigma in (("sharp, sigma 0.5 m", 0.5), ("blurred, sigma 4 m", 4.0),
                    ("uniform (no depth)", None)):
    if sigma is None:
        alpha = np.full(dep2.shape + bins.shape, 1 / len(bins))
    else:
        alpha = np.exp(-0.5 * ((bins - np.minimum(dep2, 45)[..., None]) / sigma) ** 2)
        alpha /= alpha.sum(-1, keepdims=True)        # a distribution over D per pixel
    feat = (alpha[..., None] * ctx[:, :, None, :])[inside]   # outer product c_d = alpha_d c
    bev2 = splat(feat)
    if sigma == 0.5:                                 # check the trick once
        ref = np.zeros((80 * 40, 3)); np.add.at(ref, bin_id, feat)
        assert np.allclose(ref.reshape(80, 40, 3), bev2)
    carm, lanem = bev2[..., 2], bev2[..., 1]
    ys = GY2[carm > 0]
    print(f"{name:20s} car mass in footprint {carm[foot].sum() / carm.sum():6.1%},"
          f" car spread along Y {np.sqrt(np.cov(ys, aweights=carm[carm > 0])):4.1f} m,"
          f" lane mass within 0.5 m of a line {lanem[near_line].sum() / lanem.sum():6.1%}")
print(f"lifted points: {feat.shape[0]:,} from {lab2.size:,} pixels x {len(bins)} depths")

Its output is:

sharp, sigma 0.5 m   car mass in footprint  89.3%, car spread along Y  0.9 m, lane mass within 0.5 m of a line  99.5%
blurred, sigma 4 m   car mass in footprint  41.6%, car spread along Y  4.0 m, lane mass within 0.5 m of a line  82.8%
uniform (no depth)   car mass in footprint  13.6%, car spread along Y 10.5 m, lane mass within 0.5 m of a line  77.5%
lifted points: 568,997 from 19,200 pixels x 42 depths

With nearly exact depth, nine tenths of the car’s features land on its footprint, and the rest spills into neighbouring cells because the bins are a metre apart. With a 4 m blur, less than half lands on the footprint and the car is smeared along the viewing direction with a spread of 4 m, the same as the blur, since depth error moves features along the ray and nowhere else. With no depth information, the car is spread over the whole ray, much as IPM spread it. The lane markings suffer less: they lie on the ground and run roughly along the camera’s viewing direction, so a depth error slides their features along the line rather than off it. In this toy the geometry is exact, so the depth distribution alone decides the quality of the lifted map; BEVDepth’s experiment above points the same way for a learned model.

Attention-based view transformation

Attention, the core operation of the transformer, reverses the direction of the mapping. Instead of pushing image features out into 3D, a query that lives in 3D or BEV space pulls in the image features that correspond to it, using the camera parameters to know where to look. Learned depth is not needed, because the query already has a 3D position.

DETR3D (Wang et al., 2021) extends DETR to multi-camera 3D detection without a dense BEV map. Each of a set of learned object queries decodes a 3D reference point, which is projected into every camera with the known projection matrices. Features are sampled there by bilinear interpolation from each level of a feature pyramid, averaged over the pyramid levels and the cameras in which the point is visible, and added to the query; the queries interact by self-attention, and training uses DETR’s set-to-set loss, so no non-maximum suppression is needed. The authors argue that this top-down design avoids the compounding errors of a separate depth predictor. PETR (Liu et al., 2022) avoids the projection and sampling step: it discretizes each camera’s frustum into points at several depths, transforms them to 3D coordinates, encodes those with a small network into a 3D position embedding, and adds it to the image features, so that object queries can attend to all image tokens with ordinary attention.

Cross-view transformers (Zhou & Krähenbühl, 2022) apply the idea to BEV segmentation: a learned map-view positional embedding cross-attends to image features carrying camera-aware positional embeddings built from each camera’s intrinsics and extrinsics, so the network learns the mapping between views without modelling it geometrically. The authors report 35 frames per second on a single RTX 2080 Ti GPU.

BEVFormer (Li et al., 2022) builds a dense BEV map with attention. Its queries are a learnable grid $Q \in \mathbb{R}^{H \times W \times C}$, one per BEV cell. In spatial cross-attention, the query $Q_p$ at cell $p$ with ground position $(x', y')$ is lifted to a pillar of $N_{\text{ref}}$ 3D reference points $(x', y', z'_j)$ at predefined heights, which are projected into the cameras by their $3 \times 4$ projection matrices $T_i$:

$$z_{ij} \begin{pmatrix} x_{ij} \\ y_{ij} \\ 1 \end{pmatrix} = T_i \begin{pmatrix} x' \\ y' \\ z'_j \\ 1 \end{pmatrix}, \qquad \operatorname{SCA}(Q_p, F_t) = \frac{1}{|\mathcal{V}_{\text{hit}}|} \sum_{i \in \mathcal{V}_{\text{hit}}} \sum_{j=1}^{N_{\text{ref}}} \operatorname{DeformAttn}\big(Q_p, (x_{ij}, y_{ij}), F_t^i\big),$$

where $\mathcal{V}_{\text{hit}}$ is the set of cameras into which the points project and $F_t^i$ are camera $i$’s features. DeformAttn is the deformable attention of Deformable DETR: around each reference point it samples a few locations at learned offsets and combines them with learned weights, so a query attends to a handful of image positions instead of all of them, a sparse form of cross-attention. On nuScenes they use $N_{\text{ref}} = 4$ heights spread uniformly from −5 m to 3 m, and six encoder layers. Li et al. introduce these anchor heights to capture cues that appear at different heights above a cell, so the cell’s content is not assumed to lie on the ground, which is the assumption that breaks IPM.

Temporal BEV

A single frame cannot reveal velocity and sees occluded objects poorly. BEVFormer’s temporal self-attention keeps the previous BEV map $B_{t-1}$, aligns it to the current frame by ego-motion so that each cell refers to the same place on the ground, and lets each query attend, again with deformable attention, both to the current queries and to the aligned history. The history enters only through the previous BEV map rather than a stack of earlier maps, which the authors argue costs less and carries less disturbing information. BEVDet4D (Huang & Huang, 2022) concatenates the aligned BEV features of the previous frame with the current ones and predicts velocity as the positional offset between the two; on the nuScenes validation set it reduced the velocity error mAVE of its small configuration from 0.909 to 0.337 m/s.

The effect on detection metrics is large. In BEVFormer’s nuScenes test results, the temporal model reached 0.569 NDS and an mAVE of 0.378 m/s, against 0.495 NDS and 0.842 m/s for BEVFormer-S, the same model without temporal self-attention (Li et al., 2022, Table 1, VoVNet-99 backbone).

The families compare as follows:

Family	Example	Where image features go	Needs depth	Main weakness
Inverse perspective mapping	IPM baseline (Roddick & Cipolla, 2020)	Onto the ground plane by a homography	No (assumes flat ground)	Everything above the ground smears along rays
Uniform lifting	OFT	Every voxel along the ray	No	Features spread over the whole ray
Depth-distribution lifting	Lift-Splat-Shoot, BEVDet, BEVDepth	Along the ray, weighted by predicted depth	Predicted, optionally LiDAR-supervised	BEV quality bounded by depth quality
Sparse 3D queries	DETR3D, PETR	Pulled into object queries	No	No dense BEV map for segmentation
Dense BEV queries	BEVFormer, cross-view transformers	Pulled into BEV grid queries	No (reference heights instead)	Cost of attention for every cell

Datasets and Benchmarks

• nuScenes (Caesar et al., 2020) is the benchmark on which the methods above report their main results: 1,000 scenes of 20 s each, recorded in Boston and Singapore with six cameras, five radars and one LiDAR, with 3D boxes for 23 classes on keyframes at 2 Hz and human-annotated semantic maps. Detection uses 10 of the 23 classes. Test-set results are compared on the benchmark’s leaderboard.
• The Waymo Open Dataset (Sun et al., 2020) has 1,150 scenes of 20 s with five LiDARs and five cameras. Because its cameras do not cover the full circle, BEV methods adapt the grid: the arXiv version of the BEVFormer paper uses a 300 × 220 grid on Waymo, covering −35 to 75 m along one axis and ±75 m along the other.
• Lyft Level 5 and Argoverse. Lift-Splat-Shoot also evaluated on Lyft Level 5 and showed transfer from nuScenes to Lyft’s different camera rig; Pyramid Occupancy Networks evaluated BEV segmentation of map and object classes on Argoverse and nuScenes.
• KITTI provides driving data from cameras and LiDAR with a 3D object detection benchmark, on which monocular BEV methods such as OFT were evaluated.

Evaluation Metrics

nuScenes detection (Caesar et al., 2020) matches predictions to ground truth by the 2D centre distance on the ground plane rather than by 3D IoU, because small objects such as pedestrians score zero IoU after a small translation error, which would make vision-only methods hard to compare. Average precision is computed from the precision–recall curve with operating points below 10% recall or precision removed, and averaged over the classes $\mathbb{C}$ and four distance thresholds $\mathbb{D} = \{0.5, 1, 2, 4\}$ m:

$$\text{mAP} = \frac{1}{|\mathbb{C}|\,|\mathbb{D}|} \sum_{c \in \mathbb{C}} \sum_{d \in \mathbb{D}} \text{AP}_{c,d}.$$

Five true-positive error metrics, computed for matches at 2 m and averaged over classes, measure the quality of matched boxes: translation error (mATE, metres), scale error (mASE, $1 - \text{IoU}$ after aligning position and orientation), orientation error (mAOE, radians), velocity error (mAVE, m/s) and attribute error (mAAE, $1 - \text{accuracy}$). The nuScenes detection score combines them:

$$\text{NDS} = \frac{1}{10}\Big[5\,\text{mAP} + \sum_{\text{mTP} \in \mathbb{TP}} \big(1 - \min(1, \text{mTP})\big)\Big],$$

so half of NDS is detection and half is box quality. Because velocity enters NDS, temporal methods gain on it beyond their gain in mAP: of the 0.074 NDS between BEVFormer and BEVFormer-S above, the lower velocity error accounts for about 0.046 and the higher mAP for about 0.023 (our arithmetic from Li et al., 2022, Table 1).

BEV segmentation is scored by intersection over union per class on the grid. The numbers depend on the grid’s extent and resolution, on how box labels are rasterized and on the split; Philion and Fidler (2020) note that the concurrent works they list used different grid definitions, so a true comparison was not possible. Waymo detection uses IoU-matched AP and APH, which weights each true positive by its heading accuracy (Sun et al., 2020).

Reported camera-only results on the nuScenes test set, each from the paper’s own table:

Method	Backbone and training data	NDS	mAP	Source
DETR3D	DD3D’s backbone, initialized from a DD3D checkpoint	0.479	0.412	Wang et al. (2021), Table 2
BEVDet	BEVDet-Base, train + val, test-time augmentation	0.482	0.422	Huang et al. (2021), Table 3
PETR	VoVNet-99, external data	0.504	0.441	Liu et al. (2022), Table 2
BEVFormer	VoVNet-99, depth-pretrained	0.569	0.481	Li et al. (2022), Table 1
BEVDet4D	BEVDet4D-Base, train + val, test-time augmentation	0.569	0.451	Huang & Huang (2022), Table 2
BEVDepth	ConvNeXt, train + val, multi-frame, test-time augmentation	0.609	0.520	Li et al. (2023), Table 8

The rows differ in backbone, image resolution, pretraining, training data and test-time augmentation, so they document the progress of the period rather than isolate the effect of the view transformation. Comparison tables in these papers also copy each other’s numbers, not always correctly: the DETR3D rows in the BEVDet and BEVDet4D tables give PGD’s mAP, mATE and mASE (0.386, 0.626, 0.245), not the 0.412, 0.641 and 0.255 that DETR3D reports.

Applications

The main application is camera-based perception for automated driving: 3D detection and tracking of road users, online construction of the local map (drivable area, lanes, crossings), and planning. BEV output feeds object tracking by detection; boxes in a metric frame, with predicted velocities, are plausibly easier to associate across frames than boxes in the image (our reasoning). Lift-Splat-Shoot used its BEV cost map for planning, framed as choosing among 1,000 template trajectories; its top-5 accuracy on nuScenes was 15.52%, against 19.27% for a planner on a single LiDAR scan (Philion & Fidler, 2020, Table 6). Mobile robots that move on a floor face the same problem of turning camera views into a ground map, and the flat-floor assumption of IPM often holds better indoors than on roads (our reasoning).

Open Problems

• Depth. Camera BEV methods still trail LiDAR where depth matters. Lift-Splat-Shoot’s car segmentation IoU was 32.06 against 40.26 for a model given a single LiDAR scan, and its performance at night fell well below the LiDAR model’s (Philion & Fidler, 2020). BEVDepth’s experiment with ground-truth depth shows how much remains to be gained from better depth.
• Generalization across rigs. A model trained on nuScenes and tested on Lyft’s cameras reached 21.35 IoU for cars, against 43.09 when trained on Lyft (Philion & Fidler, 2020, Tables 1 and 4). Calibration conditions the network, but distribution shift in cameras, mounting and environment remains.
• Fair comparison. Results depend strongly on the training setup. Used in place of BEVFormer’s encoder, with BEVFormer’s task heads, Lift-Splat reached 42.1 car IoU and 77.7 drivable-area IoU on the nuScenes validation set (Li et al., 2022, Table 3 of the ECCV version), well above the 32.06 and 72.94 of the original paper.
• Height. A BEV grid collapses the vertical axis: a BEV label cannot say whether a cell holds an overhanging barrier at windscreen height or an obstacle on the road, and it represents irregular objects only by their footprint. Three-dimensional occupancy grids keep the height axis at a higher cost (our reasoning).
• Range and resolution. Cells far from the vehicle are filled from few pixels, as the first example showed, while a uniform grid spends the same memory on every cell. Long-range perception with a fixed-resolution grid remains a trade-off between extent, cell size and compute (our reasoning).