Sensor Fusion for Autonomous Driving

Sensor fusion for autonomous driving combines the measurements of a vehicle’s cameras, LiDARs and radars into one estimate of the surrounding scene, most often a set of 3D boxes with classes, headings and velocities. The sensors fail in different ways, which is why vehicles carry all three, but their data do not line up by themselves. Each sensor sees the scene from its own position, at its own rate and in its own geometry: a perspective image, a sparse point cloud, a handful of radar returns with radial velocities. Fusion therefore needs three things: geometry (calibration and projection between sensor frames), timing (synchronization and compensation for the vehicle’s own motion), and a decision about where in the processing the streams meet.

The general idea of combining modalities, and the vocabulary of early, intermediate and late fusion, belong to multimodal fusion. The LiDAR article covers the sensor, its calibration against cameras and the correction of its own motion distortion. This article covers what is specific to driving: the frame and time bookkeeping, the families of camera–LiDAR and radar–camera methods, fusion in the bird’s-eye view, fusion over time through tracking, and the benchmarks that measure it.

Purpose

Each sensor answers a different question well, as the LiDAR article’s comparison of sensors sets out: LiDAR localizes surfaces in 3D but is sparse and carries little appearance, and cameras carry appearance but recover depth only by inference. Radar measures range and radial velocity directly and keeps working in rain, smoke and dust, but with coarse angular resolution and very sparse returns (Yao et al., 2024).

That complementarity did not automatically turn into better detectors. Vora et al. (2020) opened the PointPainting paper by noting that LiDAR-only methods outperformed fusion methods on the main benchmark datasets, with LiDAR-only methods at the top of the KITTI leaderboard, and illustrated the missed opportunity with a pedestrian and a pole 25 m away that look nearly identical in the point cloud although both are clearly visible in the image. Pang et al. (2020) made a similar observation about the KITTI 3D leaderboard, where LiDAR-only methods outperformed most fusion methods. By 2022, Bai et al. reported the opposite on the larger nuScenes and Waymo datasets, whose point clouds are sparser: LiDAR-only methods were generally surpassed by methods that use both sensors. Much of the work described below aimed at that gap.

Fusion also serves redundancy. Liang et al. (2022) pointed out that many fusion networks of the time looked up image features from LiDAR points or LiDAR proposals, so a LiDAR failure left them unable to predict anything at all, and designed their method so that each sensor’s branch works without the other.

When to Use It

The reported gains are largest where one sensor is weak:

• Small and distant objects. On the nuScenes test set, PointPainting’s largest per-class gain was on traffic cones (+16.8 AP), which the authors attribute to cones having very few LiDAR points (Vora et al., 2020, Table 3). CLOCs improved most at 40–50 m on the KITTI validation set (Pang et al., 2020, Fig. 4).
• Classes that look alike in 3D. Pedestrians against poles, a common LiDAR confusion that painting resolved in the authors’ qualitative examples (Vora et al., 2020).
• Poor light and bad weather. TransFusion’s gain over its LiDAR-only version on the nuScenes validation set was 6.0 mAP at night against 5.4 by day (Bai et al., 2022, Table 4), and BEVFusion reported its largest improvements over single-sensor models in rain (+10.7 mAP over a LiDAR-only CenterPoint) and at night (+12.8 mIoU in map segmentation over its camera-only version) (Liu et al., 2023, Table IV).
• Velocity. Radar measures radial velocity in a single frame; on the nuScenes validation set, with the same backbone, adding it to a monocular detector cut the mean velocity error from 1.426 to 0.540 m/s (Nabati & Qi, 2021, Table 1).

Fusion is not free, and it does not always help. Feng et al. (2021), reviewing deep multimodal detection and segmentation for driving, found no conclusive evidence that one fusion scheme is better than the others: performance depended on the sensors, the data and the architecture. On the Waymo Open Dataset, whose LiDAR is denser than nuScenes’, TransFusion’s image branch added little, which Bai et al. (2022) attribute partly to the LiDAR-only boxes already being well localized. Every fused system also inherits a dependence on calibration and synchronization that a single-sensor system avoids, a point Pang et al. (2020) make explicitly. The multimodal fusion article discusses how adding a modality can even hurt a network.

How It Works

Frames and projection

Every sensor has its own coordinate frame, and the vehicle has an ego frame fixed to its body. Calibration gives each sensor’s rigid pose in the ego frame; localization gives the ego pose in the world at each instant. A LiDAR point $\mathbf{p}$ reaches a camera pixel through the extrinsics $(R, \mathbf{t})$ and the intrinsic matrix $K$ of the pinhole camera model:

$$\mathbf{p}_c = R\,\mathbf{p} + \mathbf{t}, \qquad \begin{bmatrix} u \\ v \\ 1 \end{bmatrix} = \frac{1}{z_c}\, K\, \mathbf{p}_c ,$$

where $z_c$ is the point’s depth along the optical axis. The LiDAR article derives this chain and the camera parameters article explains $K$, $R$ and $\mathbf{t}$. Two details matter in practice. Points with $z_c \le 0$ lie behind the camera and must be discarded before the division, or they reappear mirrored in the image. And projection is many-to-one: a ray from the camera can pass through several LiDAR points, and because the two sensors sit at different positions, a point the LiDAR sees can be hidden from the camera by a nearer surface. Painting such a point with the colour or class of its pixel transfers the occluder’s label to it; the example below counts how often this happens.

When the sensors capture at different instants, the transform also has to cross time. PointPainting writes the nuScenes transform as a product of three rigid transforms:

$$T = T_{\text{camera} \leftarrow \text{ego}}\; T_{\text{ego}(t_c) \leftarrow \text{ego}(t_l)}\; T_{\text{ego} \leftarrow \text{lidar}},$$

from the LiDAR to the ego frame, from the ego pose at the LiDAR time $t_l$ to the ego pose at the image time $t_c$, and from the ego frame to the camera (Vora et al., 2020). The middle factor is ego-motion compensation. Rolling-shutter cameras add a further complication, since every image row has its own capture time: the Waymo Open Dataset ships LiDAR-to-image projections that solve for each point’s capture time under a constant-velocity model of the vehicle, an optimization the authors report converges in two or three iterations (Sun et al., 2020).

The two sensors also differ in density. Liu et al. (2023) estimate that with a typical 32-beam LiDAR only about 5% of camera features are matched to a LiDAR point, so moving camera features onto LiDAR points discards most of the image, while moving LiDAR points into the image distorts geometry, since points far apart in 3D become neighbours in 2D. Several of the methods below can be read as responses to this asymmetry.

Calibration

Camera intrinsics come from camera calibration, and the LiDAR article describes target-based and targetless LiDAR–camera extrinsic calibration and KITTI’s procedure. Two further problems are specific to vehicles in service.

The first is that calibration drifts. Levinson and Thrun (2013) built an online method that needs no target: depth discontinuities in the LiDAR scan should project onto image edges, so they score a calibration by how well projected discontinuities fall on an edge image smoothed by an inverse distance transform, accumulated over frames. In their experiments on real scenes, it detected miscalibrations within one second once the error exceeded 0.25° or 10 cm, and tracked gradual rotational drift with a mean error of 0.10°. Feng et al. (2021) also survey learned approaches that classify or regress the LiDAR–camera misalignment directly from the two inputs.

The second is radar. Target-based radar–camera calibration uses targets that both sensors can localize, most often a triangular corner reflector, sometimes combined with a styrofoam board that carries visual markers without disturbing the radar; targetless methods use radar velocity measurements of moving objects together with the camera pose (Yao et al., 2024).

How large an error matters can be reasoned from the projection. A rotation error $\delta$ shifts a distant point by about $f \tan\delta$ pixels whatever its range, while a translation error $e$ shifts a point at depth $z_c$ by about $f e / z_c$ pixels. Rotation errors therefore hurt small and far objects, whose image boxes are narrow, and translation errors hurt near objects most. The example measures both.

Time synchronization and ego-motion compensation

Sensors run at different rates. nuScenes records cameras at 12 Hz, the LiDAR at 20 Hz and radars at 13 Hz. Its cameras are triggered by the LiDAR’s rotation, as the LiDAR article describes; the image timestamp is the exposure trigger time and the LiDAR timestamp is the end of the rotation, and because 12 exposures are spread over 20 sweeps, not every sweep has an image (Caesar et al., 2020). Waymo reports a camera–LiDAR synchronization error bounded in [−6, 7] ms with 99.7% confidence (Sun et al., 2020). Offsets between sensors arise from different set-up times, oscillator frequencies and measurement latencies; Yao et al. (2024) survey methods that estimate the latency and drift periodically, predict them with a Kalman filter, or recover the offset from the same object seen by both sensors. Timestamps and clock domains in general are covered by real-time vision pipelines.

Ego-motion compensation moves each measurement from the ego pose at its own timestamp to the ego pose at a common reference time, using the vehicle’s localization. It also makes it possible to accumulate several sweeps of a sparse sensor. The nuScenes baseline moved the LiDAR sweeps of the last 0.5 s into the keyframe’s coordinate system and appended each point’s time offset as an extra feature; going from 1 to 10 sweeps raised a PointPillars baseline from 31.8 to 44.8 NDS on the validation set and cut its velocity error from 1.21 to 0.30 m/s (Caesar et al., 2020, Table 3). CenterFusion accumulates three radar sweeps, the detections of the past 0.25 s, with their radial velocities compensated for the ego motion (Nabati & Qi, 2021). Accumulation densifies radar data at the price of delay (Yao et al., 2024).

Ego-motion compensation corrects static objects exactly, given good localization, but a moving object is still displaced by its own velocity times the time offset; correcting that needs an estimate of the object’s velocity, which is what tracking provides. A time offset can also be accepted on purpose to cut latency: PointPainting can paint each LiDAR sweep with the segmentation of the previous image, after ego-motion compensation, so the LiDAR network need not wait for the image network. In the authors’ shortened-training ablation on the nuScenes validation set, this gave 46.4 NDS against 46.3 with the concurrent image, both at 33.9 mAP (Vora et al., 2020, Table 5), and painting with the previous image added 0.75 ms of latency over the unpainted detector.

Where the streams meet

Feng et al. (2021) use the early, middle and late positions of multimodal fusion and list their trade-offs for driving sensors. Early fusion of raw or preprocessed data is cheap, but must be retrained when a sensor changes and is sensitive to spatial and temporal misalignment. Late fusion of per-sensor outputs is modular, but discards intermediate features. Middle fusion combines intermediate features, once or at several depths. Driving papers use several other vocabularies for overlapping ideas, which makes comparisons confusing:

Source	Categories
Feng et al. (2021)	early, middle (including deep and short-cut fusion), late
Vora et al. (2020)	object-centric fusion, continuous feature fusion, explicit transform to the bird’s-eye view, detection seeding
Pang et al. (2020)	early, deep, late
Bai et al. (2022)	result-level, proposal-level, point-level
Liang et al. (2022)	point-level (image features attached to LiDAR points), feature-level (LiDAR features or proposals projected into the images to fetch image features)
Yao et al. (2024), radar–camera	data-level, feature-level, object-level, hybrid-level

The sources also weigh the costs differently. Feng et al. write that late fusion has high computation and memory costs, while Pang et al. call late fusion much simpler to build, because it reuses single-sensor detectors unchanged and needs jointly aligned data only for the final step. MV3D’s “early”, “late” and “deep” fusion describe how region features from three views are combined inside its second stage (Chen et al., 2017), not the stage at which raw sensor data meet.

How to Apply It

The example renders a road scene with four targets (three cars, one partly hidden behind another, and a pedestrian), a pole that is not a target, and flat ground. A camera with known $K$ and a 32-beam LiDAR mounted 0.6 m behind, 0.3 m to the right of and 0.4 m above it both observe it by ray casting (NumPy 2.3.5 and SciPy 1.16.3). The first part projects the LiDAR points into the image and paints each with the class of its pixel, taken from the rendered label image, which plays the role of a perfect segmentation; the rendered depth then explains every wrong label.

import numpy as np

rng = np.random.default_rng(0)

# Ego frame: x forward, y left, z up, origin on the ground under the LiDAR.
# Objects: name -> (centre x, y, length, width, height, yaw in degrees).
objects = {"car A": (14.0, 1.2, 4.5, 1.8, 1.5, 0.0),
           "car B": (24.0, 0.2, 4.5, 1.8, 1.5, 10.0),
           "ped P": (11.0, -3.2, 0.6, 0.6, 1.75, 0.0),
           "car C": (38.0, -6.0, 4.5, 1.8, 1.5, 90.0),
           "pole":  (17.0, -4.0, 0.15, 0.15, 2.5, 0.0)}
names = list(objects)

def hit_boxes(o, d, boxes):
    """Nearest hit of rays o + s*d (s > 0) with oriented boxes and the ground z = 0.
    Returns ray parameter s and label (0 = ground, i+1 = box i, -1 = nothing)."""
    s_best = np.full(d.shape[0], np.inf); label = np.full(d.shape[0], -1)
    with np.errstate(divide="ignore", invalid="ignore"):
        s = -o[2] / d[:, 2]
    ok = (s > 0) & (s < s_best); s_best[ok] = s[ok]; label[ok] = 0
    for i, (cx, cy, l, w, h, yaw) in enumerate(boxes):
        c, sn = np.cos(np.radians(yaw)), np.sin(np.radians(yaw))
        Rb = np.array([[c, sn, 0], [-sn, c, 0], [0, 0, 1]])        # ego -> box axes
        ob = Rb @ (o - np.array([cx, cy, h / 2])); db = d @ Rb.T
        half = np.array([l / 2, w / 2, h / 2])
        with np.errstate(divide="ignore", invalid="ignore"):
            t1, t2 = (-half - ob) / db, (half - ob) / db
        near = np.nanmax(np.minimum(t1, t2), 1); far = np.nanmin(np.maximum(t1, t2), 1)
        ok = (near <= far) & (near > 0) & (near < s_best)
        s_best[ok] = near[ok]; label[ok] = i + 1
    return s_best, label

# Camera: pinhole K, mounted 0.6 m ahead of, 0.3 m left of and 0.4 m below the LiDAR.
W, H = 640, 360
K = np.array([[500.0, 0, 320], [0, 500.0, 180], [0, 0, 1]])
R_cam_ego = np.array([[0, -1, 0], [0, 0, -1], [1, 0, 0]], float)  # ego axes -> camera axes
c_cam, c_lid = np.array([0.6, 0.3, 1.5]), np.array([0.0, 0.0, 1.9])
R = R_cam_ego                       # LiDAR axes are parallel to the ego axes
t = -R @ (c_cam - c_lid)            # extrinsics: p_cam = R @ p_lidar + t

# Render the camera image by ray casting: label image and depth (z along the optical axis).
u, v = np.meshgrid(np.arange(W) + 0.5, np.arange(H) + 0.5)
rays_cam = np.stack([u.ravel(), v.ravel(), np.ones(u.size)], 1) @ np.linalg.inv(K).T
s, lab = hit_boxes(c_cam, rays_cam @ R_cam_ego, objects.values())
depth_img = np.where(lab >= 0, s, np.inf).reshape(H, W)   # rays have z_cam = 1, so s = depth
label_img = lab.reshape(H, W)
for i, n in enumerate(names):
    print(f"{n}: {np.sum(label_img == i + 1):6d} pixels in the rendered image")

# LiDAR: 32 beams from -16 to +4 degrees, 0.2-degree azimuth steps over +-40 degrees.
el, az = np.meshgrid(np.radians(np.linspace(-16, 4, 32)), np.radians(np.arange(-40, 40, 0.2)))
d_lid = np.stack([np.cos(el) * np.cos(az), np.cos(el) * np.sin(az), np.sin(el)], -1).reshape(-1, 3)
r_lid, pt_label = hit_boxes(c_lid, d_lid, objects.values())
keep = (pt_label >= 0) & (r_lid < 80)
r_lid = r_lid[keep] + rng.normal(0, 0.02, keep.sum())          # 2 cm range noise
pts = d_lid[keep] * r_lid[:, None]                            # LiDAR frame
pt_label = pt_label[keep]
print(f"LiDAR points: {len(pts)}")

# Project into the image: p_cam = R p + t, then (u, v) = K p_cam / z.
p_cam = pts @ R.T + t
z = p_cam[:, 2]
uv = p_cam @ K.T
uv = uv[:, :2] / uv[:, 2:]
inside = (z > 0) & (uv[:, 0] >= 0) & (uv[:, 0] < W) & (uv[:, 1] >= 0) & (uv[:, 1] < H)
col, row = uv[inside, 0].astype(int), uv[inside, 1].astype(int)
z_in, lab_in = z[inside], pt_label[inside]

# Paint each point with the label of its pixel (a perfect segmentation), and use the
# rendered depth to explain every wrong label.
painted = label_img[row, col]
nearer = z_in > depth_img[row, col] + 0.3      # the camera sees a nearer surface here
print(f"points that project inside the image: {inside.sum()}")
print("object   points  painted correctly  wrong, occluded in camera  wrong, other")
for i, n in enumerate(names):
    m = lab_in == i + 1
    bad = painted[m] != i + 1
    print(f"{n:7s} {m.sum():6d} {1 - bad.mean():18.1%} {np.sum(bad & nearer[m]):26d}"
          f" {np.sum(bad & ~nearer[m]):13d}")

car A:   5427 pixels in the rendered image
car B:   1124 pixels in the rendered image
ped P:   3253 pixels in the rendered image
car C:   1342 pixels in the rendered image
pole:    461 pixels in the rendered image
LiDAR points: 9300
points that project inside the image: 7252
object   points  painted correctly  wrong, occluded in camera  wrong, other
car A      533              95.1%                          0            26
car B      131              77.9%                         23             6
ped P      263              95.1%                          0            13
car C      105             100.0%                          0             0
pole        36             100.0%                          0             0

Even with perfect segmentation and perfect calibration, painting is wrong for 22% of the points on car B. All 23 of its points counted as occluded receive car A’s label: the LiDAR, mounted higher, sees part of car B over car A, while along the camera’s ray to those points car A is in front. The remaining wrong labels sit on silhouettes, where a point and its pixel fall on different sides of an edge, and on car A’s roof, which the LiDAR sees from above and the camera, mounted at roof height, sees only edge-on. A real camera has no depth image, so an occlusion test has to be approximated from the projected LiDAR depths themselves.

The second part builds synthetic detections with known ground truth. The camera finds all four targets, plus a car printed on a roadside advertisement; the LiDAR detector finds all four targets, but gives the pedestrian a low score, as if confused by its pole-like shape, and reports the real pole as a pedestrian. Late fusion projects each 3D box into the image, matches the boxes by Hungarian assignment on $1 - \text{IoU}$, where IoU is the intersection over union of a projected box and a camera box, with a gate at an IoU of 0.5, combines the scores of matched pairs by a noisy-OR, keeps unmatched LiDAR boxes with their own score, drops unmatched camera boxes, which have no 3D position, and reports every 3D box whose final score is at least 0.6; the single-sensor baselines use the same 0.6 threshold. The fusion rule and the thresholds are choices made for the illustration. The same detections are then fused with a deliberately wrong extrinsic calibration, and with the LiDAR sweep taken later than the image while the ego vehicle drives at 15 m/s, the crossing car moves at 8 m/s and the pedestrian walks at 1.5 m/s.

from scipy.optimize import linear_sum_assignment

targets = ["car A", "car B", "ped P", "car C"]          # the pole is not a target
vel = {"car A": (0, 0), "car B": (0, 0), "ped P": (0, 1.5), "car C": (0, 8.0)}  # m/s, world
EGO_SPEED = 15.0                                           # m/s along x

def box_at(name, dt, ego_comp=0.0, obj_comp=False):
    """Box of an object dt seconds after the camera frame, in the ego frame at that time.
    ego_comp: seconds of ego motion to undo; obj_comp: also undo the object's own motion."""
    cx, cy, l, w, h, yaw = objects[name]
    vx, vy = vel.get(name, (0, 0))
    x, y = cx + (vx - EGO_SPEED) * dt, cy + vy * dt
    x += EGO_SPEED * ego_comp
    if obj_comp:
        x, y = x - vx * dt, y - vy * dt
    return x, y, l, w, h, yaw

def corners(b):
    x, y, l, w, h, yaw = b
    dx, dy, dz = np.meshgrid([-l / 2, l / 2], [-w / 2, w / 2], [0, h])
    c, s = np.cos(np.radians(yaw)), np.sin(np.radians(yaw))
    px, py = x + c * dx.ravel() - s * dy.ravel(), y + s * dx.ravel() + c * dy.ravel()
    return np.stack([px, py, dz.ravel()], 1) - c_lid       # LiDAR frame

def project_box(b, R_=R, t_=t):
    q = corners(b) @ R_.T + t_
    q = q @ K.T
    q = q[:, :2] / q[:, 2:]
    return np.array([q[:, 0].min(), q[:, 1].min(), q[:, 0].max(), q[:, 1].max()]).clip(0, [W, H, W, H])

def iou(a, b):
    w_ = max(0.0, min(a[2], b[2]) - max(a[0], b[0])); h_ = max(0.0, min(a[3], b[3]) - max(a[1], b[1]))
    inter = w_ * h_
    return inter / ((a[2] - a[0]) * (a[3] - a[1]) + (b[2] - b[0]) * (b[3] - b[1]) - inter)

# Synthetic detections. Camera: 2D boxes from the true boxes, 2 px noise, plus a false
# positive (a car printed on a roadside advertisement). LiDAR: 3D boxes with 0.1 m noise;
# the pedestrian gets a low score (pole-like shape) and the pole is a false positive.
cam_dets = [(n, project_box(box_at(n, 0)) + rng.normal(0, 2, 4), s)
            for n, s in [("car A", 0.95), ("car B", 0.70), ("ped P", 0.85), ("car C", 0.80)]]
cam_dets.append(("advert", np.array([430.0, 120.0, 490.0, 160.0]), 0.75))
lid_noise = {n: rng.normal(0, 0.1, 2) for n in names}
lid_scores = [("car A", 0.90), ("car B", 0.80), ("ped P", 0.45), ("car C", 0.85), ("pole", 0.50)]

def lidar_dets(dt=0.0, ego_comp=0.0, obj_comp=False):
    out = []
    for n, s in lid_scores:
        x, y, l, w, h, yaw = box_at(n, dt, ego_comp, obj_comp)
        out.append((n, (x + lid_noise[n][0], y + lid_noise[n][1], l, w, h, yaw), s))
    return out

def fuse(lid, R_=R, t_=t, gate=0.5, thr=0.6):
    """Late fusion: Hungarian matching on 1 - IoU of projected LiDAR boxes and camera boxes."""
    C = np.array([[1 - iou(project_box(b, R_, t_), cb) for _, cb, _ in cam_dets] for _, b, _ in lid])
    rows, cols = linear_sum_assignment(C)
    pairs = {r: c for r, c in zip(rows, cols) if 1 - C[r, c] >= gate}
    kept = []
    for r, (n, b, s_l) in enumerate(lid):
        if r in pairs:
            s_l = 1 - (1 - s_l) * (1 - cam_dets[pairs[r]][2])   # noisy-OR of the two scores
        if s_l >= thr:
            kept.append(n)
    true_iou = [1 - C[i, i] for i in range(len(targets))]       # same object in both lists
    n_right = sum(lid[r][0] == cam_dets[c][0] for r, c in pairs.items())
    return kept, true_iou, n_right, len(pairs) - n_right

def score(kept):
    tp = [n for n in kept if n in targets]
    return f"TP {len(tp)}, FP {len(kept) - len(tp)}, missed {len(targets) - len(tp)}"

lid = lidar_dets()
print("LiDAR alone:   ", score([n for n, _, s in lid if s >= 0.6]))
print("camera alone:  ", score([n for n, _, s in cam_dets if s >= 0.6]), "(2D boxes only)")
kept = fuse(lid)[0]
print("late fusion:   ", score(kept))

def painting_accuracy(R_, t_):
    q = pts @ R_.T + t_
    z_ = q[:, 2]; q = q @ K.T; q = q[:, :2] / q[:, 2:]
    ins = (z_ > 0) & (q[:, 0] >= 0) & (q[:, 0] < W) & (q[:, 1] >= 0) & (q[:, 1] < H)
    obj = ins & (pt_label > 0)
    return np.mean(label_img[q[obj, 1].astype(int), q[obj, 0].astype(int)] == pt_label[obj])

def row(kept, true_iou, right, wrong):
    return (f"{right:5d} {wrong:5d}   " + "  ".join(f"{v:.2f}" for v in true_iou)
            + f"   {score(kept)}")

print("\nCalibration error    painted  pairs right/wrong  IoU of true pairs (A, B, P, C)")
for yaw_err, lat_err in [(0, 0), (0.5, 0), (1, 0), (2, 0), (0, 0.2), (0, 0.5)]:
    c, sn = np.cos(np.radians(yaw_err)), np.sin(np.radians(yaw_err))
    R_err = R @ np.array([[c, -sn, 0], [sn, c, 0], [0, 0, 1]])
    t_err = t + R @ np.array([0, lat_err, 0])
    print(f"yaw {yaw_err:3.1f} deg, y {lat_err:3.1f} m {painting_accuracy(R_err, t_err):7.1%}"
          + row(*fuse(lid, R_err, t_err)))

print("\nLiDAR later than camera       pairs right/wrong  IoU of true pairs (A, B, P, C)")
for dt in (0.05, 0.1):
    for label, kw in [("no compensation", {}), ("ego motion", {"ego_comp": dt}),
                      ("ego + object motion", {"ego_comp": dt, "obj_comp": True})]:
        print(f"{dt * 1000:3.0f} ms {label:20s}" + row(*fuse(lidar_dets(dt, **kw))))

LiDAR alone:    TP 3, FP 0, missed 1
camera alone:   TP 4, FP 1, missed 0 (2D boxes only)
late fusion:    TP 4, FP 0, missed 0

Calibration error    painted  pairs right/wrong  IoU of true pairs (A, B, P, C)
yaw 0.0 deg, y 0.0 m   93.6%    4     0   0.85  0.78  0.93  0.84   TP 4, FP 0, missed 0
yaw 0.5 deg, y 0.0 m   85.2%    4     0   0.95  0.66  0.71  0.75   TP 4, FP 0, missed 0
yaw 1.0 deg, y 0.0 m   76.8%    4     0   0.88  0.56  0.54  0.65   TP 4, FP 0, missed 0
yaw 2.0 deg, y 0.0 m   60.7%    1     0   0.70  0.39  0.27  0.49   TP 3, FP 0, missed 1
yaw 0.0 deg, y 0.2 m   80.4%    4     0   0.89  0.67  0.55  0.79   TP 4, FP 0, missed 0
yaw 0.0 deg, y 0.5 m   59.8%    3     0   0.68  0.52  0.19  0.70   TP 3, FP 0, missed 1

LiDAR later than camera       pairs right/wrong  IoU of true pairs (A, B, P, C)
 50 ms no compensation         4     0   0.84  0.79  0.61  0.78   TP 4, FP 0, missed 0
 50 ms ego motion              4     0   0.85  0.78  0.77  0.73   TP 4, FP 0, missed 0
 50 ms ego + object motion     4     0   0.85  0.78  0.93  0.84   TP 4, FP 0, missed 0
100 ms no compensation         3     0   0.73  0.77  0.33  0.72   TP 3, FP 0, missed 1
100 ms ego motion              4     0   0.85  0.78  0.63  0.62   TP 4, FP 0, missed 0
100 ms ego + object motion     4     0   0.85  0.78  0.93  0.84   TP 4, FP 0, missed 0

With correct calibration and timing, fusion removes the camera’s false positive, which has no LiDAR box to match, and recovers the pedestrian, whose LiDAR score alone is below the threshold but whose camera detection confirms it. The pole stays rejected because the camera saw no pedestrian there. These outcomes follow from how the detections and the rule were written; the point is the mechanism, not the counts.

The calibration rows show the two error types predicted earlier. A rotation error of 1° shifts every projected box by about 9 pixels, a large fraction of the pedestrian’s 37-pixel-wide box and a small fraction of car A’s 78 pixels, so the pedestrian’s IoU falls fastest; at 2° only car A still passes the gate, and the pedestrian, kept until then by the camera’s confirmation, is lost. Car A’s IoU first rises at 0.5° because the rotation happens to cancel part of the position noise added to its LiDAR box. A 0.5 m lateral error shifts the pedestrian at about 10 m by about 24 pixels and car C at about 37 m by about 7, so the near pedestrian drops out while the far car keeps an IoU of 0.70. Painting degrades from the first step, losing 8 percentage points of correctly labelled object points at 0.5°, while late fusion still pairs every object at 1°. The comparison holds for this scene and these thresholds only.

The timing rows separate the two motions. Without compensation, the 0.75 m or 1.5 m the vehicle travels between image and sweep hurts the near pedestrian most, whose bearing changes fastest, and at 100 ms its pair fails the gate. Ego-motion compensation restores the parked cars A and B exactly, as expected for static objects, but leaves the moving pedestrian and the crossing car displaced by their own motion; at 50 ms the crossing car’s IoU even falls slightly, because without compensation its own motion and the ego motion had partly cancelled in the image. Compensating for object motion as well, with the true velocities that a tracker would have to estimate, restores the synchronized result.

Variants

Point-level fusion and detection seeding

PointPainting (Vora et al., 2020) runs an image semantic segmentation network, projects every LiDAR point into the segmentation output with the transform above, and appends the class scores of its pixel to the point’s coordinates and reflectance. The painted cloud then goes to an unchanged LiDAR detector, which only needs more input channels. Painting improved all three detectors the authors tried (PointPillars, VoxelNet and PointRCNN) on the KITTI validation set, by 1.71 to 3.37 points of bird’s-eye-view mAP at moderate difficulty (Table 1), and raised an improved PointPillars from 40.1 to 46.4 mAP and from 55.0 to 58.1 NDS on the nuScenes test set (Table 4). In their nuScenes validation ablations, detection improved with segmentation quality, and painting with ground-truth boxes as an oracle segmentation added 27 mAP, so the method is bounded by the segmentation network. Where camera fields of view overlap, a point projects into two images; the authors pick one of the two score vectors at random. Because it is sequential, it cannot always be optimized end to end; the authors answer that it was empirically more effective than the other fusion methods, that the segmentation is useful on its own, and that the two networks can be pipelined.

Detection seeding uses the camera to decide where the LiDAR is searched. Frustum PointNets (Qi et al., 2018) lift each 2D detection to a frustum of 3D space, collect the points inside it and estimate a 3D box from them with point networks. Its authors list the consequence: when the 2D detector misses an object, in darkness or under strong occlusion, no 3D box can be found. Vora et al. describe this as an upper bound on recall.

Feature-level fusion

MV3D (Chen et al., 2017) encodes the LiDAR cloud as a bird’s-eye-view image (height maps of several slices, intensity and point density, at 0.1 m resolution) and a front-view image, generates 3D proposals from the bird’s-eye view, projects each proposal into the bird’s-eye view, the front view and the camera image, and pools region features from all three. Its deep fusion averages the three region features after every layer. On the KITTI validation set for cars at moderate difficulty, deep fusion with auxiliary losses reached 89.05% 3D AP at an IoU of 0.5, against 87.60% for early and 87.70% for late fusion of the same features (Table 3), at about 0.36 s per image on a Titan X GPU.

AVOD (Ku et al., 2018) moved fusion into the proposal stage. Each 3D anchor is projected into full-resolution bird’s-eye-view and image feature maps, the two crops are resized to equal size and averaged, and both the region proposal network and the second stage work on the fused features; 1×1 convolutions shrink the feature maps so that tens of thousands of anchors fit in memory. With its feature-pyramid extractor it reported 71.88% 3D AP for cars at moderate difficulty on the KITTI test set (Table II), at 0.1 s per frame on a TITAN Xp GPU and with about 16% of MV3D’s parameters. Vora et al. (2020) later called such object-centric fusion slow and cumbersome, and described a “feature blurring” problem in “continuous feature fusion”, which shares features across the strides of both backbones: one bird’s-eye-view feature corresponds to many image pixels and vice versa.

TransFusion (Bai et al., 2022) replaced the fixed point-to-pixel association with attention. A transformer decoder first predicts boxes from LiDAR bird’s-eye-view features with a sparse set of object queries; its second layer lets each query attend to image features, with the cross-attention weighted by a Gaussian mask around the box’s centre projected into the image. The calibration is used only to place the queries, so the network can find the right image features even when the projection is off. It reported 68.9 mAP and 71.7 NDS on the nuScenes test set, against 65.5 and 70.2 for its LiDAR-only version (Table 1). On the validation set, with shortened training, a 1 m translation offset between the sensors cost it 0.49 mAP, against 2.33 and 2.85 for two point-level baselines built on the same detector (Section 5.2; the labels in Fig. 5 give 2.84 for the second).

Bird’s-eye-view fusion

Two papers released in May 2022 under the same name, BEVFusion, converted both sensors into a shared bird’s-eye-view grid and fused them there. Liu et al. (2023) lift camera features into the grid with a predicted depth distribution per pixel, following Lift-Splat-Shoot, concatenate them with LiDAR bird’s-eye-view features and pass the result through a convolutional encoder, which can absorb residual misalignment from inaccurate depth. The camera-to-grid pooling took more than 500 ms on an RTX 3090 until a precomputed, interval-parallel kernel cut it to 12 ms; the precomputation relies on the camera intrinsics and extrinsics staying fixed after calibration. They reported 70.2 mAP and 72.9 NDS on the nuScenes test set (Table I), and the shared grid also serves map segmentation. Liang et al. (2022) built two independent streams so that the camera branch never depends on LiDAR input. They reported 69.2 mAP and 71.8 NDS on the nuScenes test set (Table 2). Under LiDAR failures simulated on the validation set (a field of view cropped to ±60° or ±90°, or LiDAR points removed from objects), the fused models stayed well above their LiDAR-only counterparts; with points removed from objects and matching training augmentation, the gains were 15.7 to 28.9 mAP (Table 4). The bird’s-eye-view representation itself, and the camera-only methods that build it, are the subject of the bird’s-eye-view perception article.

Late fusion

Object-level fusion combines finished detections. Classical systems match the outputs of the two sensors by similarity of position, size and class, with Kalman filters, Bayesian methods or the Hungarian algorithm, or by projecting radar detections into the image with the calibration (Yao et al., 2024). CLOCs (Pang et al., 2020) learns the combination. It takes all 2D and 3D candidates before non-maximum suppression, since late fusion can prune but not create detections and the single-sensor detectors should therefore maximize recall. For every pair of a 2D candidate and a projected 3D candidate it builds a feature from their image-plane IoU, the two scores and the 3D box’s distance, and a small network of 1×1 convolutions over the non-empty pairs outputs a new score for each 3D candidate, adding less than 3 ms per frame. Paired with Cascade R-CNN, it raised SECOND from 72.55% to 78.45% 3D AP for cars at moderate difficulty on the KITTI test set (40 recall positions; Table I).

Radar–camera fusion

Automotive radars are millimetre-wave sensors working at 76–81 GHz. Yao et al. (2024) describe how they work and what limits them: range comes from the echo’s travel time, radial velocity from the Doppler shift, and azimuth from phase differences across an antenna array; conventional radars have no elevation, while newer “4D” radars add it with a vertical antenna arrangement. Their raw signal is processed into range–azimuth–Doppler tensors, and peak detection, usually with a constant-false-alarm-rate detector, reduces those to a sparse point cloud. The review lists the limitations: low angular resolution, a few points on a pedestrian and about a dozen on a car, only the radial component of velocity, noise from multipath and interference, and weak detection of stationary obstacles because strong static reflections are hard to separate. The nuScenes radars are 77 GHz FMCW units with up to 250 m range at 13 Hz (Caesar et al., 2020).

CenterFusion (Nabati & Qi, 2021) is a middle-fusion design. A monocular center-point detector proposes objects; each proposal’s 3D box defines a frustum, and the nearest radar detection inside the frustum, with each detection expanded into a fixed-size pillar to make up for the radar’s inaccurate height information, is associated with it; the associated depth and velocity become extra feature maps for regressing depth, rotation, velocity and attributes. On the nuScenes test set it reported 0.449 NDS against 0.400 for the camera-only CenterNet, and a mean velocity error of 0.614 m/s against 1.629, although the two used different backbones (Table 1).

Fusion over time and tracking

Tracking fuses measurements over time, and through a filter it can also fuse sensors that report at different instants: a Kalman filter predicts each track to the timestamp of the next measurement, whichever sensor produced it, and updates it there; measurement models that are nonlinear in the state, such as radar’s range, azimuth and radial velocity, can be handled with the extended Kalman filter. In learned systems the usual form is tracking by detection on the fused 3D boxes. AB3DMOT (Weng et al., 2020) is a minimal example: a constant-velocity 3D Kalman filter over box position, orientation, size and velocity, Hungarian matching on 3D IoU or centre distance, and no explicit ego-motion estimation, running at 207.4 frames per second on KITTI. TransFusion’s boxes, fed to an existing tracker, gave 71.8 AMOTA on the nuScenes tracking test set, against 68.6 for its LiDAR-only boxes (Bai et al., 2022, Table 3). The track’s velocity estimate is also what corrects time offsets for moving objects, as the example shows. Multi-object tracking covers association and track management in general.

Best Practices

• Resolve every measurement to one reference time and frame before fusing it. Carry a timestamp with each image, sweep and radar return, compensate the ego motion, and use track velocities for moving objects (example).
• Check visibility before transferring image information to LiDAR points, since parallax between the sensors mislabels points even with perfect calibration (example).
• Feed late fusion with high-recall candidates, taken before non-maximum suppression and at low thresholds (Pang et al., 2020).
• Monitor calibration and test with degraded inputs. Online checks of the kind Levinson and Thrun (2013) built, and protocols such as TransFusion’s dropped images and translation offsets or Liang et al.’s cropped LiDAR field of view, expose dependencies that a clean benchmark hides.

Pitfalls

• Coordinate conventions. Ego, sensor, camera and world frames differ in axis order and handedness between datasets, KITTI adds a rectifying rotation to its projection chain (see the LiDAR article), and a chain such as PointPainting’s mixes sensor-to-ego and ego-to-sensor factors.
• Mismatched class definitions. KITTI’s segmentation labels a cyclist’s rider and bicycle as separate classes, while its detection benchmark labels them together; PointPainting mapped bicycle points near a rider to the cyclist class, and the authors trace two false-positive cyclists to that heuristic (Vora et al., 2020).
• Radar ghosts and static clutter. Multipath returns create false detections, and strong reflections from static metal objects such as manhole covers complicate the detection of stationary obstacles (Yao et al., 2024).
• Comparing numbers across papers. Baselines copied between tables can drift: BEVFusion’s table lists TransFusion at 71.6 NDS on the nuScenes test set (Liu et al., 2023, Table I), while TransFusion’s own table gives 71.7 (Bai et al., 2022, Table 1), and TransFusion’s row for PointPillars (40.1 mAP, 55.0 NDS) carries the numbers PointPainting reports for its improved PointPillars+, not the 30.5 and 45.3 it reports for the original (Vora et al., 2020, Table 4). KITTI also changed its AP computation from 11 to 40 recall positions on 8 October 2019, as its benchmark page records, so MV3D’s and AVOD’s numbers are not comparable with CLOCs’.

Evidence and Impact

Three benchmarks dominate the evaluation of camera–LiDAR fusion. KITTI scores 3D and bird’s-eye-view detection by IoU-matched AP in three difficulty levels, as its article describes. nuScenes matches predictions by centre distance on the ground plane rather than IoU and combines mAP with five box-error measures, velocity among them, in the nuScenes detection score (NDS) (Caesar et al., 2020); the bird’s-eye-view perception article gives the definitions. The Waymo Open Dataset scores AP and APH, which weights each true positive by its heading accuracy, at an IoU of 0.7 for vehicles and 0.5 for pedestrians, with LEVEL 2 adding the examples a labeller marks as hard or that have at most five LiDAR points (Sun et al., 2020). The paper’s printed formula for heading accuracy, $\min(|\tilde\theta - \theta|, 2\pi - |\tilde\theta - \theta|)/\pi$, is the normalized heading error, which is zero for a perfect heading; the weight that matches the description is presumably one minus it (our reading).

Reported results, each from the paper’s own table. For AVOD, CLOCs, AB3DMOT and Liu et al.’s BEVFusion, table and figure numbers refer to the arXiv versions (v4, v1, v5 and v3), and the two reviews were read in their arXiv versions, since the cited IEEE versions are not openly available.

Method	Fusion	Benchmark and split	Result	Source
MV3D	region features, deep fusion	KITTI validation, cars, moderate	89.05% 3D AP at IoU 0.5	Chen et al. (2017), Table 3
AVOD (feature pyramid)	proposal and region features	KITTI test, cars, moderate	71.88% 3D AP	Ku et al. (2018), Table II
CLOCs (PV-RCNN + Cascade R-CNN)	late, learned	KITTI test, cars, moderate, 40 recall positions	80.67% 3D AP (PV-RCNN alone 80.28%)	Pang et al. (2020), Table I
Painted PointPillars+	point-level	nuScenes test	46.4 mAP, 58.1 NDS (unpainted 40.1, 55.0)	Vora et al. (2020), Table 4
CenterFusion	radar–camera, middle	nuScenes test	0.449 NDS (camera-only CenterNet, different backbone, 0.400)	Nabati & Qi (2021), Table 1
TransFusion	transformer, soft association	nuScenes test	68.9 mAP, 71.7 NDS (LiDAR-only 65.5, 70.2)	Bai et al. (2022), Table 1
BEVFusion (Liang et al.)	bird’s-eye view, independent streams	nuScenes test	69.2 mAP, 71.8 NDS	Liang et al. (2022), Table 2
BEVFusion (Liu et al.)	bird’s-eye view	nuScenes test	70.2 mAP, 72.9 NDS	Liu et al. (2023), Table I

The rows span five years and differ in backbones, training schedules and test-time augmentation, so they document the period rather than rank fusion positions against each other; the controlled comparisons within single papers (MV3D’s Table 3, PointPainting’s paired detectors, TransFusion against TransFusion-L) hold more of the setup fixed. The robustness results quoted above (dropped cameras, translation offsets, cropped LiDAR fields of view, night and rain subsets) were all measured on nuScenes, on the validation set where the papers state the split, with corruptions simulated by the authors or subsets selected from the scene descriptions. They are benchmark measurements, not measurements of on-road safety.