Lane Detection

Lane detection finds the lanes of a road in images from a vehicle’s forward-facing camera and returns each lane as a curve: the painted line between two lanes, a road edge, or, in some formulations, the lane’s centerline. Driver-assistance functions such as lane keeping and lane departure warning, and the planners of automated vehicles, use the result to place the vehicle within its lane (Neven et al., 2018; Chen et al., 2022). The task looks easy, since lane markings are bright, thin and nearly straight. It is hard because the markings are often not visible: other vehicles hide them, paint wears off, night, glare and shadows remove the contrast, and the road curves, splits and merges. Methods have moved from hand-built pipelines of edge detection, line voting and curve fitting to networks that infer lanes from the whole scene, as in semantic segmentation.

Problem Definition

Given an image $I$ from a camera looking along the road, predict a set of lanes $\{L_1, \dots, L_M\}$, where $M$ varies from image to image and each $L_j$ is a curve in the image. Most benchmarks parametrize a lane by its horizontal position as a function of image row, $x = f_j(y)$, over the rows where the lane exists. This assumes that a lane crosses each row at most once, which holds for a forward-looking camera; TuSimple’s guideline states it as the assumption that a lane “will not turn around”.

What counts as a lane differs between datasets. TuSimple annotates the markings of the current lane and the lanes to its left and right, up to four, with a fifth during lane changes. CULane annotates up to four markings, drawn as cubic splines, and annotates a lane even where a vehicle hides it or the paint is missing, inferring its position from context; lanes beyond a barrier are not annotated (Pan et al., 2018). Detectors are therefore trained to report the lanes a driver would use, not every painted line, and to fill in what is hidden.

Input and Output

The input is usually a single RGB frame, sometimes a short clip: each TuSimple sample is a one-second clip of 20 frames whose last frame is labeled. The output takes one of several representations, and the choice shapes the method.

• Masks. A per-pixel label map, either one “lane” class or one class per lane position (left-left, left, right, right-right). Segmentation networks produce it directly; curves are recovered afterwards by sampling the mask row by row and fitting.
• Point sets on fixed rows. The $x$-position of each lane at a fixed list of rows, with a code for “absent”. TuSimple’s labels have this form: $x$-coordinates at rows 10 pixels apart (rows 240 to 710 in the README’s example), with $-2$ where there is no lane. Row-anchor methods predict it directly; anchor-based methods use equally spaced rows with start and end indices.
• Polynomials and splines. A few coefficients per lane, such as a cubic $x = a_3 y^3 + a_2 y^2 + a_1 y + a_0$ over a vertical range, or the control points of a spline. Compact and smooth, but the right degree depends on the view: a circular arc on the road is not a low-degree polynomial in image coordinates.
• 3D curves. Lane points in vehicle or camera coordinates, including height. The image representations above say nothing about the road’s slope, which matters when lanes are projected onto a road that is not flat (Garnett et al., 2019).

A lane is a thin structure; image-space representations make errors far up the image, where a pixel spans many meters of road, look as small as errors near the car.

Variants

• Ego lanes against all lanes. Aly (2008) reports a mode for the two boundaries of the current lane and one for all visible lanes.
• Markings, boundaries and centerlines. Some datasets label painted markings, others lane boundaries including curbs, others centerlines. 3D-LaneNet outputs each curve as either a lane delimiter or a centerline (Garnett et al., 2019).
• Lane attributes. OpenLane labels each lane with one of 14 categories such as single white dashed or double yellow solid (Chen et al., 2022).
• 3D lane detection. Lanes in 3D from a single camera, without assuming a flat road.
• Video. Using the frames before the labeled one, or tracking lanes over time.

Challenges

The CULane test set is split by condition, and the split shows where detectors lose accuracy. SCNN, a segmentation model from 2018, reached an F1 score of 90.6 on normal images but 69.7 in crowded traffic, 66.1 at night, 66.9 in shadow, 64.4 on curves, 58.5 under dazzling light and 43.4 where no line is visible (CULane test set, IoU threshold 0.5; Pan et al., 2018, Table 5). For urban streets, Aly (2008) adds irregular lane shapes, emerging and merging lanes, varied pavement and changing slopes.

• Occlusion. Vehicles hide markings, and a detector must use what remains visible and the structure of the scene.
• Missing or worn markings. Faded paint, roads without markings, and dashed lines with long gaps. Qin et al. (2020) call this the “no-visual-clue” problem.
• Lighting. Night, low sun, headlight glare, tunnel exits and shadows change the contrast of markings and add strong edges that are not lanes.
• Curves. Most benchmark lanes are close to straight, which biases both models and metrics; PolyLaneNet’s authors conjecture that this imbalance explains their model’s weakness on strongly curved lanes (Tabelini et al., 2021a).
• Distractors. Arrows, stop lines, crosswalks, curbs, guard rails and the edges of vehicles look locally like lane markings. Aly (2008) reports false positives from stop lines, crosswalks, street writing, passing cars and curbs.
• Topology. Lanes split, merge, end and begin at junctions and ramps; a fixed number of lane slots cannot represent this, and 3D-LaneNet handles merges and splits explicitly.
• Non-flat roads. Methods that map lanes to a top view assume a flat road; on slopes a fixed top-view transform no longer fits (Neven et al., 2018), and the projection distorts the lanes’ height and curvature (Garnett et al., 2019).
• Speed. Lane detection runs on every frame, and Qin et al. (2020) note that vehicles with several cameras need a low cost per camera, so methods compete on frames per second as well as accuracy; the real-time pipeline around the detector sets the budget.

Approaches

Classical pipelines

Before deep learning, lane detectors were built from hand-designed stages. Neven et al. (2018) summarize the cues they used: color, the structure tensor, bar filters and ridge features, possibly combined with a Hough transform and with particle or Kalman filters. Aly’s (2008) detector for urban streets is a complete, documented example:

1. Inverse perspective mapping (IPM). The image is warped to a top view of the road plane using the camera’s intrinsic and extrinsic parameters (focal lengths, optical center, pitch, yaw and height), assuming a flat road. In the top view, lanes that converge at the horizon become parallel, near-vertical lines of fixed width. For a flat road the warp is a homography; bird’s-eye-view perception treats the mapping, its errors for points above the ground, and learned alternatives in depth.
2. Filtering. The top view is filtered with a separable kernel: a Gaussian along the lane and a second derivative of a Gaussian across it, sized to the expected marking width. Only the top 2.5% of responses are kept.
3. Line detection. A simplified Hough transform, the sum of each column of the top view, counts the lines; RANSAC then fits a line in a window around each.
4. Curve fitting. A RANSAC variant fits a cubic Bézier spline to each line’s window, scoring candidates by the filter response along the curve and penalizing short and strongly curved splines.
5. Post-processing. Points on each spline are re-localized along its normal, the spline is extended while the image supports it, and geometric checks replace splines that are too short or too curved by the line from step 3 and reject splines that are not near-vertical in the top view.

On four hand-labeled clips (1,224 frames, 4,172 lane boundaries, the Caltech Lanes set), Aly reports 96.3% correct detections with an 11.6% false-positive rate for the ego lane’s two boundaries, and 90.9% with 17.4% false positives for all lanes, at 50 Hz on 640 × 480 images, without tracking. False positives came from stop lines, crosswalks, street writing and passing cars, and mostly from curbs detected as the boundary of the rightmost lane.

The Hough transform behind step 3 is the standard tool for lines in an edge map. Each edge pixel $(x, y)$ votes for every line $\rho = x \cos\theta + y \sin\theta$ that passes through it, the votes are accumulated over a grid of $(\theta, \rho)$, and peaks of the accumulator are lines supported by many pixels. In the image itself, lanes converge towards a vanishing point, so a detector can restrict $\theta$ to the slanted directions that lanes take on each side of the image and discard horizontal and vertical edges.

Fitting turns the pixels assigned to a lane into a curve, usually by least squares with a robust wrapper such as RANSAC to resist outliers. In the top view, where lanes are near-vertical, $x$ is fitted as a second- or third-degree polynomial in $y$; Neven et al. (2018) fit in a top view because low-order polynomials fit curved lanes poorly in the image. Another way to collect a lane’s pixels in the top view is a sliding-window search: start at a peak of the column histogram near the bottom, move a small window upwards row band by row band, re-center it on the mean position of the marking pixels it contains, and fit a curve to all pixels found.

Tracking links detections over time. Lane parameters change slowly from frame to frame, so a filter such as the Kalman filter can predict where each lane will be, restrict the search, and smooth the output. Aly (2008) names tracking as the next step for removing false positives; Qin et al. (2020) describe it as a popular post-processing step when visual information is weak.

These pipelines are fast and interpretable, but every stage encodes an assumption: flat road, markings brighter than asphalt, lanes roughly parallel to the car, few distractors. When an assumption fails, nothing later in the pipeline can recover.

Example: a classical pipeline on rendered roads

The NumPy and scikit-image code renders 640 × 360 road images from a pinhole camera 1.5 m above a flat road (focal length 400 px, horizon at row 150). The lane is 3.6 m wide, with a solid line on the left and a dashed line on the right (4 m dashes, 4 m gaps), both 15 cm wide. Curved roads follow $x = x_0 + k z^2$ for lateral position $x$ at distance $z$, a curve of radius about $1/(2k)$ near the car: 125 m and 50 m. Some scenes contain a car, drawn as a dark box 1.8 m wide, 1.4 m tall and 4.5 m long. Pixel noise is seeded. Because the scene is rendered, the ground truth is exact: the projected $x$-position of each line at every fifth row from 37.5 m ahead to the bottom of the image, following the lane through the car where it is hidden, as CULane annotates occluded lanes.

The detector is the classical recipe in the image plane: Canny edges below the horizon, a Hough transform with 0.5° steps, the strongest peak on each side whose normal angle lies between 15° and 75°, then a quadratic $x = p(y)$ fitted by least squares to the edge pixels within 12 px of that line, reported over the rows those pixels cover. The detector’s settings were chosen on the straight road and are the same for every scene. Two scores follow the benchmarks’ definitions, scaled from their image widths to 640 px: TuSimple-style point accuracy (a point is correct within 10 px, widened for slanted lanes as the official code does, counted over the rows where the true lane exists; a lane is matched when at least 85% of its points are correct) and a CULane-style F1 (lanes drawn about 12 px wide, matched one to one by IoU, true positive above 0.5, counts summed over images). Each scene is rendered with seeds 0 to 9.

import numpy as np
from scipy.ndimage import binary_dilation
from scipy.optimize import linear_sum_assignment
from skimage.draw import line as draw_line, polygon
from skimage.feature import canny
from skimage.morphology import disk
from skimage.transform import hough_line, hough_line_peaks

W, H, F, U0, VH, CAM_H = 640, 360, 400.0, 320.0, 150.0, 1.5  # pinhole camera over a flat road
Z_FAR = 50.0                                                 # farthest annotated distance (m)
ROWS = np.arange(166, H, 5)                                  # sample rows, like TuSimple's h_samples


def lane_x(x0, k, z):                       # lateral position (m) of a lane line at distance z
    return x0 + k * z**2


def project(x0, k, rows):                   # exact image column of a lane line at each row
    z = F * CAM_H / (rows - VH)
    u = U0 + F * lane_x(x0, k, z) / z
    return np.where((z <= Z_FAR) & (u >= 0) & (u < W), u, np.nan)


def render(k, seed, car=None):
    rng = np.random.default_rng(seed)
    img = np.full((H, W), 0.75)                              # sky
    v, u = np.mgrid[0:H, 0:W].astype(float)
    road = v > VH + 1
    z = np.where(road, F * CAM_H / np.maximum(v - VH, 1e-6), 1e6)
    x = (u - U0) * z / F
    img[road] = 0.35
    for x0, dashed in [(-1.8, False), (1.8, True)]:          # solid left line, dashed right line
        paint = road & (np.abs(x - lane_x(x0, k, z)) < 0.075) & (z < 80)
        if dashed:
            paint &= (z % 8.0) < 4.0                         # 4 m dashes, 4 m gaps
        img[paint] = 0.9
    if car:                                                  # a box-shaped car: centre x (m), rear at z (m)
        xc, zc = car
        def uv(X, Y, Z):
            return U0 + F * X / Z, VH + F * (CAM_H - Y) / Z
        xs = xc - 0.9                                        # its left side, seen by the camera
        for face, shade in [([(xs, 0, zc), (xs, 0, zc + 4.5), (xs, 1.4, zc + 4.5), (xs, 1.4, zc)], 0.2),
                            ([(xc - 0.9, 0, zc), (xc + 0.9, 0, zc), (xc + 0.9, 1.4, zc), (xc - 0.9, 1.4, zc)], 0.12)]:
            pts = [uv(*p) for p in face]
            rr, cc = polygon([p[1] for p in pts], [p[0] for p in pts], img.shape)
            img[rr, cc] = shade
    img += rng.normal(0.0, 0.06, img.shape)                  # sensor noise
    return np.clip(img, 0, 1)


def detect(img):
    edges = canny(img, sigma=2.0, low_threshold=0.05, high_threshold=0.15)
    edges[: int(VH) + 12] = False                            # region of interest: below the horizon
    angles = np.deg2rad(np.arange(-89.5, 90, 0.5))
    acc, theta, dist = hough_line(edges, theta=angles)
    _, th, rh = hough_line_peaks(acc, theta, dist, num_peaks=20, min_distance=9, min_angle=5,
                                threshold=0.2 * acc.max())
    ys, xs = np.nonzero(edges)
    lanes = []
    for side in (+1, -1):                                    # theta > 0: left line, theta < 0: right line
        cand = [(t, r) for t, r in zip(th, rh) if side * t > np.deg2rad(15) and abs(t) < np.deg2rad(75)]
        if not cand:
            continue
        t, r = cand[0]                                       # strongest peak on this side
        near = np.abs(xs * np.cos(t) + ys * np.sin(t) - r) < 12
        if near.sum() < 20:
            continue
        coef = np.polyfit(ys[near], xs[near], 2)             # x = p(y), least squares
        u = np.polyval(coef, ROWS.astype(float))
        ok = (ROWS >= ys[near].min()) & (u >= 0) & (u < W)
        lanes.append(np.where(ok, u, np.nan))
    return lanes


def tusimple(pred, gt, thresh=10.0, min_acc=0.85):           # 20 px at 1280 wide -> 10 px at 640
    accs = []
    for g in gt:
        m = ~np.isnan(g)
        slope = np.polyfit(ROWS[m], g[m], 1)[0]
        t = thresh / np.cos(np.arctan(slope))                # official code widens the threshold for slanted lanes
        accs.append(max([np.mean(np.abs(np.nan_to_num(p[m], nan=-1e4) - g[m]) < t) for p in pred], default=0.0))
    matched = sum(a >= min_acc for a in accs)
    fp = (len(pred) - matched) / len(pred) if pred else 0.0
    return np.mean(accs), fp, (len(gt) - matched) / len(gt)


def mask(lane, width=12):                                    # 30 px at 1640 wide -> 12 px at 640
    m = np.zeros((H, W), bool)
    pts = [(int(r), int(round(c))) for r, c in zip(ROWS, lane) if not np.isnan(c)]
    for (r0, c0), (r1, c1) in zip(pts[:-1], pts[1:]):
        rr, cc = draw_line(r0, c0, r1, c1)
        m[rr, cc] = True
    return binary_dilation(m, disk(width // 2))


def culane_counts(pred, gt, iou_min=0.5):
    if not pred:
        return 0, 0, len(gt)
    iou = np.array([[(mask(g) & mask(p)).sum() / (mask(g) | mask(p)).sum() for p in pred] for g in gt])
    r, c = linear_sum_assignment(-iou)                       # one-to-one matching by IoU
    tp = int((iou[r, c] > iou_min).sum())
    return tp, len(pred) - tp, len(gt) - tp


scenes = {"straight": (0.0, None), "curve, R=125 m": (0.004, None), "curve, R=50 m": (0.01, None),
          "car cutting in, 4 m": (0.0, (2.2, 4.0)), "car next lane, 4 m": (0.0, (3.6, 4.0)),
          "car next lane, 8 m": (0.0, (3.6, 8.0))}
print(f"{'scene':<21}{'TuSimple acc':>13}{'(min-max)':>14}{'FP':>6}{'FN':>6}{'CULane F1':>11}  TP/FP/FN")
for name, (k, car) in scenes.items():
    res, counts = [], np.zeros(3, int)
    for seed in range(10):
        gt = [project(-1.8, k, ROWS.astype(float)), project(1.8, k, ROWS.astype(float))]
        pred = detect(render(k, seed, car))
        res.append(tusimple(pred, gt))
        counts += culane_counts(pred, gt)
    a = np.array(res)
    tp, fp, fn = counts
    f1 = 2 * tp / (2 * tp + fp + fn)
    print(f"{name:<21}{a[:, 0].mean():>13.3f}{f'({a[:, 0].min():.2f}-{a[:, 0].max():.2f})':>14}"
          f"{a[:, 1].mean():>6.2f}{a[:, 2].mean():>6.2f}{f1:>11.2f}  {tp}/{fp}/{fn}")

Output (NumPy 2.3.5, SciPy 1.16.3, scikit-image 0.25.2; two runs gave identical output):

scene                 TuSimple acc     (min-max)    FP    FN  CULane F1  TP/FP/FN
straight                     1.000   (1.00-1.00)  0.00  0.00       1.00  20/0/0
curve, R=125 m               0.914   (0.91-0.92)  0.00  0.00       1.00  20/0/0
curve, R=50 m                0.809   (0.81-0.82)  1.00  1.00       0.50  10/10/10
car cutting in, 4 m          0.885   (0.62-1.00)  0.15  0.15       0.85  17/3/3
car next lane, 4 m           0.564   (0.56-0.56)  0.50  0.50       0.50  10/10/10
car next lane, 8 m           1.000   (1.00-1.00)  0.00  0.00       1.00  20/0/0

On the straight road the pipeline is perfect for every seed: the scene matches its model of lanes as straight lines in the image. On curves it degrades, not because the edges are missing but because the Hough step looks for straight lines. The strongest peak follows the near part of each lane, the 12 px band around it loses the curved far part, and the quadratic fitted to the near pixels stops early or leaves the lane where it bends. In this camera geometry one image row spans about 1.4 cm of road at the bottom of the image and about 2.3 m at the top sample row, so most sample rows lie on the near part. On the gentle curve, about 9% of the points miss, yet both lanes pass both lane-level tests. On the sharp curve, the point accuracy still reads 0.81, while every lane falls below TuSimple’s 85% rule (FP and FN of 1.00) and the CULane criterion accepts 10 of the 20 lanes. The two benchmarks’ metrics disagree about the same lanes, and an averaged point accuracy hides lane-level failure.

The car shows two ways in which a vehicle breaks the pipeline. Cutting in across the dashed line at 4 m, it hides the near part of that line; in 3 of the 10 images, the remaining evidence lost the vote and the right lane was placed wrongly. In the next lane at 4 m, the car hides nothing, yet the right lane fails in every image: the lower edge of the car’s side lies on the ground along the direction of travel, so in the image it points to the same vanishing point as the lanes, and so close to the camera it is longer in the image than the visible dashes. At 8 m the same car is shorter in the image and does not disturb the result. The car positions were chosen to show these two mechanisms, with the 8 m scene as a check that the second depends on distance; they are not a sample of traffic. A clean, rendered image with no shadows, wear or glare is already enough to break a detector whose only knowledge of lanes is “long slanted edge”.

Segmentation-based methods

Deep networks first replaced the hand-made feature stages and then the whole pipeline. Qin et al. (2020) describe segmentation as the mainstream formulation of deep lane detection at the time of their paper.

SCNN (Pan et al., 2018) predicts one probability map per lane position (four classes plus background) and a second small head predicts whether each lane exists. Its contribution is the Spatial CNN layer: a feature map is cut into rows (or columns), and each slice is convolved and added to the next, in sequence, downwards, upwards, rightwards and leftwards. Information thus propagates across the whole image within one layer, which suits thin, long structures whose appearance is weak but whose shape is strong, such as a lane hidden behind a car. To produce curves, the probability map of each lane predicted to exist is searched every 20 rows for its maximum, and the points are joined by a cubic spline. On CULane, which the same paper introduced, SCNN raised the total F1 from 63.2 for its baseline to 71.6, above ResNet-101 (70.8) at IoU 0.5. The authors also report first place in the TuSimple challenge with an accuracy of 96.53%.

Assigning a fixed class to each lane position cannot handle a variable number of lanes, and lane changes swap the classes. LaneNet (Neven et al., 2018) casts lane detection as instance segmentation: one branch segments lane pixels against background, and a second outputs an embedding per pixel, trained with a clustering loss so that pixels of the same lane lie close together and different lanes lie far apart; clustering the embeddings separates the lanes. A small second network, H-Net, predicts a perspective transform for each image, constrained to keep horizontal lines horizontal, in which a third-order polynomial is fitted to each lane. A fixed top-view transform fails when the road slope changes; on the TuSimple validation set, the conditional transform lowered the fitting error and removed the missed points of the fixed one (arXiv version, Table III). LaneNet reached 96.4% accuracy on the TuSimple test set, at 52 frames per second for 512 × 256 images on a GTX 1080 Ti (Table IV; the text says 50).

Segmentation methods are accurate but dense: they classify every pixel and still need post-processing to get curves.

Row-anchor methods

Ultra Fast Lane Detection (Qin et al., 2020) replaces per-pixel classification with per-row selection. The image is cut into $h$ predefined rows (row anchors) and each row into $w$ cells; for each of $C$ lanes and each row anchor, the network classifies which of the $w$ cells contains the lane, with an extra class for “no lane in this row”. The output is a $C \times h \times (w+1)$ tensor computed from global features, against $H \times W \times (C+1)$ for segmentation; with CULane settings the authors estimate the cost at about $1.7 \times 10^4$ against $1.15 \times 10^6$ calculations. The classifier sees the whole image, which helps where the lane itself is invisible. Two structural losses encode lane shape: an L1 penalty on the difference between the distributions of adjacent rows, and a penalty on the second difference of the expected positions, which is zero for straight lanes. An auxiliary segmentation branch is used only during training. With a ResNet-18 backbone, the method reached a CULane F1 of 68.4 at 322.5 frames per second on 288 × 800 inputs; with ResNet-34, 72.3 at 175.4 (Table 6). Evaluated with TuSimple’s lane-level counts by Tabelini et al. (2021b), the ResNet-18 model has a false discovery rate of 19.05% despite 95.82% point accuracy, which shows the metric problem seen in the example above.

Anchor- and curve-based methods

Other methods treat each lane as an object with a parametric shape, as object detectors treat boxes.

PolyLaneNet (Tabelini et al., 2021a) regresses, from a whole-image feature vector, the coefficients of a polynomial for each of a fixed maximum number of lanes, together with each lane’s vertical start, a confidence score and a shared horizon line. It needs no post-processing and ran at 115 frames per second, but its TuSimple accuracy, 93.36%, was below that of the methods it was compared with. The authors argue that TuSimple’s accuracy is too permissive: lane markings in the data are mostly straight, and on the TuSimple validation set first-order polynomials (straight lines) scored 88.63% against 88.62% for third-order ones (arXiv version, Table II; the text says straight lines lowered accuracy by 0.35 points).

LaneATT (Tabelini et al., 2021b) brings anchors to lanes. An anchor is a straight line starting on the left, right or bottom border of the image with a given direction, following Line-CNN’s anchor set. Features are pooled from the backbone’s feature map along each anchor, an attention module lets each anchor’s features gather information from all other anchors, and for each anchor the network predicts a confidence, horizontal offsets of the lane from the anchor at equally spaced rows, and the lane’s length. Non-maximum suppression removes duplicates. On CULane, LaneATT reports F1 75.09 at 250 frames per second with ResNet-18 and 77.02 at 26 frames per second with a ResNet-122 backbone.

CLRNet (Zheng et al., 2022) refines learnable lane priors (a start point, an angle and offsets at equally spaced rows) across the levels of a feature pyramid, first with high-level features to find lanes, then with low-level features to place them precisely. A module called ROIGather gathers context for each prior from the whole feature map, and a Line IoU loss treats the lane as a whole: each point is widened into a short horizontal segment, and the overlaps of predicted and true segments, summed along the lane, are divided by their summed unions. CLRNet reports an F1 of 80.47 on CULane with a DLA-34 backbone. It also introduces mF1, the mean of the F1 scores at IoU thresholds 0.50, 0.55, …, 0.95, after COCO’s average precision; under it, SCNN scores 38.84 against 71.60 at IoU 0.5, and CLRNet 55.64.

Other formulations exist, such as sequential “drawing” of lanes and neural architecture search for curve lanes (Xu et al., 2020). Across families, the trend in these papers is the same: from dense per-pixel output with heavy post-processing to compact lane-level outputs that a network can predict in one pass, with lane shape built into the representation or the loss.

3D lanes

Image lanes become road geometry only through a projection, usually onto a flat ground plane. On hills the projection distorts both the height and the curvature of lanes (Garnett et al., 2019). 3D-LaneNet predicts lanes as 3D curves in camera coordinates from a single image. Its network projects feature maps to a top view inside the network, an inverse perspective mapping applied to features, and outputs, for each longitudinal column of the top view, whether a lane passes through it and its 3D curve; this handles merges and splits explicitly. PersFormer (Chen et al., 2022) replaces the fixed projection with a transformer that builds top-view features by attending to front-view regions, using the camera parameters as reference, and the same paper releases OpenLane, built on the Waymo Open Dataset: 200,000 frames, over 880,000 annotated lanes and 14 lane categories. OpenLane reports an F-score and the lateral and height errors near and far from the vehicle, following the metrics of Gen-LaneNet (Chen et al., 2022). These methods belong to the wider family of bird’s-eye-view perception, which lifts image features onto a ground grid for many tasks; the Bird’s-Eye-View Perception article covers the view transforms.

Datasets and Benchmarks

Dataset	Year	Images	Scenes	Labels	Metric
Caltech Lanes (Aly)	2008	1,224 frames, 4,172 lane boundaries	Urban streets, 4 clips	Splines, all visible lanes	Spline distance test
TuSimple	2017	3,626 train and 2,782 test clips, last frame labeled	Highways, good and medium weather	Points at fixed rows, up to 5 lanes	Accuracy, FP, FN
CULane	2018	133,235 frames (88,880 / 9,675 / 34,680)	Beijing urban, rural and highway, 9 test categories	Cubic splines, up to 4 lanes, occluded lanes inferred	F1 at IoU 0.5, 30 px lanes
LLAMAS	2019	100,042 images	Highways, about 350 km	Generated from maps	Segmentation AP; spline F1
CurveLanes	2020	150,000 images, 680,000 labels	Curved lanes	Lane curves	F1
OpenLane	2022	200,000 frames, 880,000+ lanes	Waymo Open Dataset scenes	3D lanes, 14 categories	F-score, near and far errors

TuSimple was released for a lane detection challenge at the CVPR 2017 Workshop on Autonomous Driving. Its README describes clips of 20 frames from highways in good and medium weather at different times of day, with the last frame of each clip labeled; it speaks of about 7,000 clips in its introduction, but its split lists 3,626 for training and 2,782 for testing. Images are 1280 × 720 (Qin et al., 2020). The scenes are comparatively easy, and results on it are close together: Tabelini et al. (2021b) describe them as saturated.

CULane (Pan et al., 2018) was extracted from more than 55 hours of video recorded by six vehicles in Beijing, at 1640 × 590. Its eight challenging test categories (crowded, night, no line, shadow, arrow, dazzle light, curve and crossroad) make up most of the test set; the crossroad images contain no lanes, so only false positives are counted there. As of October 2026, the dataset page licenses it for non-commercial use.

LLAMAS (Behrendt & Soussan, 2019) was labeled automatically by projecting lane markers from high-definition maps into the images and refining the projection, which made 100,042 images from about 350 km of highway driving possible without manual annotation. Its test labels are not public; methods are scored by the benchmark’s maintainers, using CULane’s F1 for lane curves (Tabelini et al., 2021b). CurveLanes (Xu et al., 2020) was collected to emphasize curved lanes. OpenLane (Chen et al., 2022) brings 3D labels and lane categories. The KITTI suite also has a road and ego-lane benchmark scored in the top view.

Evaluation Metrics

The two main benchmarks define their metrics differently, and the definitions matter as much as the numbers.

TuSimple accuracy. The official evaluation compares predicted and true $x$-positions at the labeled rows:

$$\text{accuracy} = \frac{\sum_{\text{clip}} C_{\text{clip}}}{\sum_{\text{clip}} S_{\text{clip}}},$$

where $C_{\text{clip}}$ is the number of correct points in the labeled frame and $S_{\text{clip}}$ the number of requested points. The official code makes the definition concrete. A point is correct when it lies within 20 px of the truth; the threshold is divided by the cosine of the lane’s angle, estimated by a line fit, so slanted lanes get a wider tolerance. Each true lane is compared with every prediction and keeps the best accuracy, and the image score averages these over the lanes, at most four, dropping the worst when there are five; the reported accuracy is the mean of these image scores, not the pooled ratio of the formula. Points absent in both truth and prediction count as correct. A true lane is matched when its best accuracy is at least 0.85; FN is the fraction of unmatched true lanes and FP the fraction of predicted lanes minus matched lanes over predicted lanes, both averaged over images. An image with more than two predicted lanes beyond the number of true lanes scores zero, and the code also zeroes images whose reported running time exceeds 200 ms; the README says that algorithms slower than 5 frames per second are treated as predicting no lanes. The challenge ranked by accuracy alone; the README says FP and FN were provided to help participants.

CULane F1. The official evaluation draws each lane, interpolated with a cubic spline, as a line 30 pixels wide on a 1640 × 590 canvas, and computes the IoU between the drawn masks of each true and predicted lane (Pan et al., 2018). True and predicted lanes are matched one to one by maximum total IoU (the code solves the assignment with a Hungarian-style matching), and a match counts as a true positive when its IoU exceeds the threshold, 0.5 in standard use (0.3 is the loose setting in the original paper). True positives, false positives and false negatives are summed over the whole test set before computing precision, recall and F1. Per-category results use the same F1, except for the crossroad category, where only false positives are reported.

The metrics differ in what they forgive. TuSimple accuracy averages points, which are dense near the car, so a lane that is right near the car and wrong far away can still score well; PolyLaneNet’s authors fitted polynomials to the test labels themselves and found that straight lines already reach 96.22% accuracy (arXiv version, Table III), 0.65 points below the best result in their comparison, and call the metric too permissive (Tabelini et al., 2021a). Accuracy also ignores false positives, which is why LaneATT reports F1, FDR and FNR on TuSimple too. CULane’s IoU at 0.5 is a coarse localization test with 30 px wide lanes; CLRNet’s mF1 asks for tighter overlap. Neither metric looks at the lane’s 3D position, which is what a controller uses; 3D benchmarks measure lateral and height errors in meters. Reported speeds depend on hardware and implementation: LaneATT remeasured its fastest competitor, and CLRNet the competitors with public code, on their own machines.

Applications

• Driver assistance. Lane keeping, automated lane centering and lane departure warning need the lane boundaries relative to the vehicle (Neven et al., 2018; Chen et al., 2022).
• Automated driving. Lanes feed trajectory planning (Neven et al., 2018) and help the vehicle localize itself (Zheng et al., 2022). 3D-LaneNet’s authors describe the alternative of loading pre-mapped lanes, which is geometrically accurate given precise localization but complex to deploy and maintain (Garnett et al., 2019); LLAMAS uses such maps in the other direction, to label images. Lane detection is one component of the autonomous-driving perception stack, alongside 3D detection, tracking and mapping.

Open Problems

• Hidden and missing markings. In the published CULane results, the no-line category remains far below normal images: CLRNet’s DLA-34 model reaches 54.58 there against 93.73 on normal images (Zheng et al., 2022, Table 1).
• Curves, splits and merges. Fixed lane slots and low-order curves fit highways; junctions, ramps and sharp urban curves need representations that handle topology, as 3D-LaneNet began to do.
• Metrics that match use. Image-space accuracy and IoU weight near-field pixels and say little about errors at distance; in our reading, mF1 and the 3D errors address only part of this.
• Generalization. The main benchmarks each come from one region: CULane from Beijing, TuSimple and LLAMAS from US highways. PolyLaneNet’s authors note that roads in developing countries are usually less well maintained, and also show results on a Brazilian dataset (Tabelini et al., 2021a).
• 3D and non-flat roads. Real-world 3D lane data is recent: PersFormer’s authors call OpenLane, released in 2022, the first real-world, large-scale 3D lane dataset; their abstract says one of the first (Chen et al., 2022).