Drivable Area Segmentation

Drivable area segmentation labels the pixels of a driving image, or the cells of a ground map, that a vehicle can drive on. Depending on the benchmark the target is the road surface, the lane the vehicle is in, the road area not occupied by other objects (free space), or a split of the road into the part the vehicle is driving on and parts it could move to by changing lanes. The output tells a driver-assistance function or a planner where the vehicle may go, including on roads without lane markings, where lane detection has nothing to find. The task is a special case of semantic segmentation with two or three classes, but its history runs through geometry as much as through appearance: early systems found the road as the flat surface in front of the car, from a ground-plane model or from stereo disparity, and only later learned its appearance. It is one output of the autonomous-driving perception stack, next to detection, tracking and mapping.

Problem Definition

Given an image $I$ from a vehicle’s forward-facing camera, sometimes with a second stereo view, a LiDAR scan or GPS, predict a label map $Y$ with $Y(p) = 1$ where pixel $p$ shows drivable surface and $0$ elsewhere, or a confidence map $C(p) \in [0, 1]$. What counts as drivable is a definition, and the benchmarks differ:

• Road area. The KITTI road benchmark labels “road”, which its page defines as the composition of all lanes, and, for one subset, the ego-lane, the lane the vehicle is driving on (Fritsch et al., 2013; KITTI road page).
• Directly and alternatively drivable area. BDD100K splits the drivable area into two categories. The directly drivable area is where the vehicle is currently driving, the region where it has priority or the right of way; the alternatively drivable area is a lane the vehicle is not in but could reach by changing lanes. The authors stress that the two are visually indistinguishable and differ only in function, so a model must use blocking objects and scene context to tell them apart, and that a lane occupied by another vehicle is not drivable (Yu et al., 2020, Section 3.4).
• Free space. In stereo systems, free space is the part of the road surface not covered by obstacles: Labayrade et al. (2002) first label every pixel that lies above the estimated road surface as a potential obstacle, then grow the free road area from the remaining pixels.
• Road-like classes in semantic segmentation. Cityscapes has no drivable class. Its class “road” belongs to the category “flat” together with sidewalk, parking and rail track, and parking and rail track are ignored in its benchmark (Cordts et al., 2016; cityscapesScripts labels.py). The India Driving Dataset (IDD) groups road, parking and a “drivable fallback” class under a top-level label “drivable”, against “non-drivable” sidewalk, rail track and a non-drivable fallback (AutoNUE anue_labels.py).

The definitions encode different assumptions. Road area and the Cityscapes road class describe surface type; BDD100K’s direct and alternative areas describe what the vehicle may do next, given lanes, traffic and the right of way; free space describes geometry. A model trained on one is not a model of the others.

Input and Output

The input is usually one RGB frame. KITTI’s road benchmark offers stereo images, Velodyne laser points and GPS/IMU data besides the left colour image, and its leaderboard marks methods that use stereo, laser points, GPS or additional training data (KITTI road page). BDD100K annotates one frame from each 40-second video, the frame at the 10th second (Yu et al., 2020).

The output is a binary or three-class mask in the image, or a confidence map that is later thresholded. KITTI evaluates in the bird’s-eye view (BEV): methods may submit perspective-image results, which the development kit maps into a metric grid on the road plane (Fritsch et al., 2013). BDD100K stores the drivable labels as masks with three values, 0 direct, 1 alternative and 2 background (BDD100K format.rst). Planners need the result in metric coordinates, so image masks are usually mapped onto the ground plane; bird’s-eye-view perception predicts drivable area directly as a layer of a ground grid, which is a related but separate formulation.

Variants

• Road area, ego-lane and lane-level areas. From the whole road (KITTI “road”) to the current lane (KITTI “lane”, only in the urban-marked subset) to direct and alternative areas (BDD100K). Fritsch et al. (2013) gave no ego-lane labels for the unmarked subset because they found no objective annotation guideline for it, and none for the multi-lane subset because lane changes near junctions with several turn lanes made it ambiguous.
• Free space from range data. Stereo or LiDAR methods find the ground surface and the obstacles on it and report the obstacle-free ground.
• Bird’s-eye-view drivable area. Drivable area as a static map layer rasterized from HD maps and predicted from multi-camera features, as described in the BEV perception article.
• Unstructured roads. IDD targets Indian roads where, according to its page, lanes are often not followed and roads are muddy, and it labels drivable surfaces beside the road (IDD details page).
• Multi-task perception. Drivable area segmented by one head of a network that also detects objects and lane lines (YOLOP, HybridNets, below).

Challenges

• Distance. A flat road seen from a camera compresses with distance: the far road occupies a few image rows, and an error of one row there covers metres of road. Fritsch et al. (2013) found that every method they evaluated scored higher in the perspective image than in the BEV, because the near range is more homogeneous, easier to classify and covers most of the pixels; they conclude that perspective-space evaluation is biased against the far range.
• Boundaries without edges. On unmarked roads the boundary between road and kerb, verge or gravel may have little contrast. Fritsch et al. (2013) describe unmarked or weakly marked roads in inner cities and rural areas as an unsolved problem because of the variability of scene layout and illumination.
• Ambiguity of the label. Annotators must decide about road visible beneath a car or covered by leaves (Fritsch et al., 2013), and BDD100K’s direct and alternative areas depend on traffic rules and right of way rather than appearance (Yu et al., 2020).
• Illumination. Strong sunlight and shadows make the road’s appearance vary within one image; Fritsch et al. (2013) name them as what the learned methods in their study had to cope with.
• Weather and time of day. Cityscapes deliberately did not record in heavy rain or snow, judging that such conditions need specialized techniques and datasets (Cordts et al., 2016). BDD100K does include such scenes; its image-tagging baseline (DLA-34) reached 93.4% weather-classification accuracy on clear scenes, 65.9% on rainy ones and 0.0% on foggy ones (Yu et al., 2020, Figure 4); in our reading, some conditions are hard even to recognize, let alone to segment in.
• Non-flat roads. Methods that assume a flat road misplace the road and obstacles on hills and when the vehicle pitches; Labayrade et al. (2002) show lane markings a few metres ahead flagged as obstacles under a flat-road assumption.

Approaches

Geometry: the flat-ground model

If the road is a plane, every road pixel is the image of a point on that plane, and the plane and the image are related by a homography; the BEV perception article derives it and shows what happens to points above the ground. For a camera at height $h$ with its optical axis parallel to the road, a road point at distance $Z$ appears in image row

$$v = v_h + \frac{f h}{Z},$$

where $f$ is the focal length in pixels and $v_h$ the horizon row. Two consequences matter for drivable area. First, the row of a pixel gives its distance, so near and far can be told apart without depth measurement as long as the pixel is on the road. Second, the distance covered by one image row grows as $Z^2/(f h)$, so the far road is seen at a much coarser resolution than the near road. Fritsch et al. (2013) map results to the BEV with a road plane fitted by RANSAC in each frame, rather than with one fixed plane; their Section IV fits it to the disparity maps, their Section V to the Velodyne laser points.

Fritsch et al. (2013) note that early road-area methods used the physical property of the road being flat, but that this requires the road to be bounded by sufficiently elevated structures and accurate depth, which stereo provides only at close range. A flat-ground model alone also cannot say which part of the plane is road and which is verge: it needs either depth, to find what rises above the plane, or appearance.

Stereo: v-disparity and stixels

Stereo matching gives a disparity $\Delta$ for each pixel, and for a rectified pair with baseline $b$ the road has a simple signature. Labayrade et al. (2002) build the v-disparity image: for each image row $v$, a histogram of the disparities found in that row, so that entry $(v, \Delta)$ counts the pixels of row $v$ with disparity $\Delta$. Their derivation shows that a plane in the scene maps to a straight line in the v-disparity image (their Eq. 9), so a road made of several planar pieces maps to a piecewise linear curve. With their mean values $b = 1$ m, $h = 1.4$ m and a camera pitch of 8.5°, the flat road appears as a line of slope 0.70.

The method then works in the v-disparity image rather than the full disparity map:

1. Extract the road profile with a Hough transform: the strongest line for a flat road, or for a non-flat road the upper or lower envelope of the $k$ strongest lines (they used $k$ between 5 and 12), choosing the envelope whose pixels accumulate the larger score.
2. Read the camera’s pitch $\theta$ and height $h$ from the line: with $c_r$ its slope and $v_{0r}$ the row where it reaches zero disparity, $\theta = \arctan\!\big((v_0 - v_{0r})/\alpha\big)$ and $h = b \cos\theta / c_r$. The row of zero disparity on the road line is the horizon.
3. For each row, the profile gives the road’s disparity $\Delta_i$. Pixels of that row with disparity $\Delta_i$ lie on the road; the others are potential obstacles. Free space on the road is grown from the road pixels with a region-growing algorithm.

Because a whole image row votes at once, the authors call the matching semi-global and report that the line extraction tolerates partial occlusion and matching errors. In their example, lane markings a few metres ahead were labeled as obstacles under a flat-road assumption but not with the estimated non-flat profile, and a vehicle at about 80 m was still detected; the horizon estimate stayed within 2 pixels of a manual estimate over the sequence they show (Labayrade et al., 2002, Figures 4 and 5).

The stixel world of Badino, Franke and Pfeiffer (2009) is a later representation, which its title calls a compact medium-level representation of the 3D world; its paper could not be read for this article, so its method is not described here.

Appearance: colour, texture and learned classifiers

Monocular methods must recognize the road by its look. In the review of Fritsch et al. (2013), many approaches use appearance cues such as the colour and texture of the road, either to estimate the road’s shape or to segment the whole road area, because geometric methods need depth that is reliable only nearby. A colour model can also be learned online: the Stanley vehicle trained a colour model of drivable ground on image patches that its lasers had just classified, as the autonomous-driving perception article describes.

The KITTI road paper evaluated four methods on its URBAN road-area task (Fritsch et al., 2013):

• Baseline (BL): the average of all training ground-truth masks, used as a confidence map, a scene prior that any method should beat.
• Geometric context (GC): Hoiem et al.’s superpixel classifier of surface orientation, whose “ground” probability serves as the road confidence.
• SPRAY: spatial ray features computed in the BEV from local confidence maps for road surface, boundary and lane markings, classified by boosting trained on KITTI road.
• CNN: a convolutional network by Alvarez et al. that learns a linear combination of colour planes for uniform road appearance and combines offline and online information; it was not retrained on KITTI road.

Method	AP (persp.)	MaxF (persp.)	AP (BEV)	MaxF (BEV)
BL	86.0	77.8	76.0	70.1
GC	66.9	68.8	65.3	64.2
SPRAY	92.7	88.5	90.9	86.3
CNN	82.4	83.4	78.9	78.9

(URBAN road area, results in %, Fritsch et al., 2013, Table II.) The scene prior beat geometric context and was close to the CNN that had not been retrained. The authors explain the strength of the baseline by the dataset: low traffic and mostly straight roads, so the average road layout fits many images. AP and MaxF fell for every method in the BEV, where far road weighs as much as near road.

Example: flat-ground colour model against a learned model

The code renders road scenes by casting a ray through each pixel of a 192 × 112 pinhole camera ($f = 96$ px) held 1.5 m above flat ground with its axis level, so the horizon is row 32 and the first equation above gives every ground pixel’s distance. Each scene has a road 5 to 8 m wide that curves as $x = x_0 + k z^2$, with a texture fixed to the ground (so it shrinks with distance), a dashed centre line, a verge of grass, dirt or grey paving like a sidewalk, tree shadows in half the scenes, and up to three box-shaped cars on the road between 6 and 30 m. The label is exact: a pixel is drivable when its ray first meets the ground inside the road, so the centre line and shadowed road are drivable and cars are not. Training uses 300 asphalt scenes; there are two test sets of 100 scenes, asphalt scenes from the same generator, and an “unpaved” set with a brown dirt road and no markings, a road appearance absent from training.

Three methods are compared. The prior map is the per-pixel mean of the training masks thresholded at 0.5, the BL baseline of Fritsch et al. (2013). The colour + homography baseline uses the ground-plane homography to find a patch of ground 4 to 7 m ahead and within 1 m of the vehicle’s path, assumes that patch is road (Stanley checked such a patch with its lasers; this baseline has nothing to check it with), fits a Gaussian to its RGB values, marks pixels below the horizon within the 99% Mahalanobis bound, fills holes and keeps the region connected to the patch. The FCN is a fully convolutional network of four $3 \times 3$ layers with dilations 1, 2, 4 and 8 (a 31-pixel receptive field), fed RGB plus the row index and trained with binary cross-entropy for 8 epochs, from three seeds. IoU of the drivable class is pooled over each test set and split by the homography’s distance into near (3 to 12 m, 36 rows) and far (12 to 40 m, 8 rows), once counting pixels and once weighting each pixel by the ground area it covers, proportional to $Z^3$, which approximates the BEV evaluation of KITTI. The script takes between 2 and 3 minutes on a laptop CPU, almost all of it training.

import numpy as np
import torch
import torch.nn as nn
from scipy.ndimage import binary_fill_holes, label as cc_label, map_coordinates

W, H, F, U0, VH, CAM_H = 192, 112, 96.0, 96.0, 32.0, 1.5    # pinhole camera 1.5 m above flat ground, no pitch

# Ground-plane homography: ground point (X, Z) -> pixel (u, v), and its inverse (pixel -> ground).
HG = np.array([[F, U0, 0.0], [0.0, VH, F * CAM_H], [0.0, 1.0, 0.0]])
v, u = np.mgrid[0:H, 0:W].astype(float) + 0.5
q = np.stack([u, v, np.ones_like(u)], -1) @ np.linalg.inv(HG).T
GROUND = v > VH                                              # pixels whose ray meets the ground ahead
XG = np.where(GROUND, q[..., 0] / q[..., 2], 0.0)            # lateral position (m)
ZG = np.where(GROUND, q[..., 1] / q[..., 2], 1e3)            # distance ahead (m)
NEAR = GROUND & (ZG >= 3) & (ZG < 12)
FAR = GROUND & (ZG >= 12) & (ZG <= 40)
BEV_W = ZG**3                                                # ground area covered by a pixel, up to a constant


def render(seed, unpaved=False):
    rng = np.random.default_rng(seed)
    x0, k, width = rng.uniform(-1.0, 1.0), rng.uniform(-0.004, 0.004), rng.uniform(5.0, 8.0)
    off = XG - (x0 + k * ZG**2)                              # lateral offset from the road centreline
    on_road = GROUND & (np.abs(off) < width / 2)
    grid = rng.random((400, 140))                            # texture on 0.2 m ground cells, Z 0-80 m, X -14-14 m
    tex = map_coordinates(grid, (np.clip(ZG, 0, 79.9) / 0.2, (np.clip(XG, -13.9, 13.9) + 14) / 0.2),
                          order=1, mode="nearest") - 0.5
    gain = rng.uniform(0.75, 1.25)
    road_col = (np.array([0.50, 0.42, 0.30]) * rng.uniform(0.85, 1.15) if unpaved
                else np.full(3, rng.uniform(0.30, 0.50)))   # brown dirt road or grey asphalt
    kind = rng.integers(3)
    off_col = [np.array([0.25, 0.42, 0.18]) * rng.uniform(0.8, 1.2),   # grass
               np.array([0.46, 0.39, 0.29]) * rng.uniform(0.8, 1.2),   # dirt
               np.full(3, rng.uniform(0.45, 0.65))][kind]               # grey verge, like a sidewalk
    col = np.where(on_road[..., None], road_col, off_col) + 0.12 * tex[..., None]
    if not unpaved:                                          # dashed white centre line: 3 m dashes, 6 m gaps
        col[on_road & (np.abs(off) < 0.08) & ((ZG % 9.0) < 3.0)] = 0.85
    shade = np.ones((H, W))
    if rng.random() < 0.5:                                   # shadows of roadside trees: dark ellipses
        for _ in range(rng.integers(1, 4)):
            cx, cz, rx, rz = rng.uniform(-6, 6), rng.uniform(5, 30), rng.uniform(1, 3), rng.uniform(1, 4)
            shade[((XG - cx) / rx) ** 2 + ((ZG - cz) / rz) ** 2 < 1] = 0.5
    img = np.where(GROUND[..., None], col * shade[..., None], np.array([0.60, 0.75, 0.90]))
    depth = np.where(GROUND, ZG, np.inf)
    d = np.stack([(u - U0) / F, (v - VH) / F, np.ones_like(u)], -1)   # ray directions (x right, y down, z ahead)
    for _ in range(rng.integers(0, 4)):                      # 0-3 cars: 1.8 x 1.4 x 4.0 m boxes, ray cast
        zc = rng.uniform(6, 30)
        xc = x0 + k * zc**2 + rng.uniform(-width / 2 + 1.0, width / 2 - 1.0)
        lo, hi = np.array([xc - 0.9, CAM_H - 1.4, zc]), np.array([xc + 0.9, CAM_H, zc + 4.0])
        with np.errstate(divide="ignore", invalid="ignore"):
            t1, t2 = lo / d, hi / d
        tn, tf = np.nanmax(np.minimum(t1, t2), -1), np.nanmin(np.maximum(t1, t2), -1)
        hit = (tn <= tf) & (tn > 0) & (tn < depth)
        depth = np.where(hit, tn, depth)
        img[hit] = rng.random(3) * 0.6 + 0.1
    img = np.clip(img * gain + rng.normal(0, 0.02, img.shape), 0, 1)
    car = depth < np.where(GROUND, ZG, np.inf)
    return img.astype(np.float32), on_road & ~car            # label: the road surface that is visible


def colour_baseline(img):
    seed = GROUND & (np.abs(XG) < 1.0) & (ZG > 4) & (ZG < 7)  # ground patch just ahead, via the homography
    x = img[seed]
    mu, cov = x.mean(0), np.cov(x.T) + 1e-5 * np.eye(3)
    diff = img - mu
    d2 = np.einsum("hwi,ij,hwj->hw", diff, np.linalg.inv(cov), diff)
    road = binary_fill_holes(GROUND & (d2 < 11.34))           # chi-square 99 % for 3 degrees of freedom
    lab, _ = cc_label(road)
    keep = np.unique(lab[seed & road])
    return np.isin(lab, keep[keep > 0])


class TinyFCN(nn.Module):                                    # receptive field 31 x 31 pixels
    def __init__(self, c=16):
        super().__init__()
        layers, cin = [], 4
        for dil in (1, 2, 4, 8):
            layers += [nn.Conv2d(cin, c, 3, padding=dil, dilation=dil), nn.ReLU()]
            cin = c
        self.body = nn.Sequential(*layers, nn.Conv2d(c, 1, 1))

    def forward(self, x):
        return self.body(x)[:, 0]


ROW = torch.tensor(v / H, dtype=torch.float32)


def to_input(imgs):
    x = torch.tensor(np.stack(imgs)).permute(0, 3, 1, 2) - 0.5
    return torch.cat([x, ROW.expand(len(imgs), 1, H, W)], 1)  # RGB plus the image row


def train_fcn(x, y, seed, epochs, batch=16):
    torch.manual_seed(seed)
    net = TinyFCN()
    opt = torch.optim.Adam(net.parameters(), lr=3e-3)
    lossf = nn.BCEWithLogitsLoss()
    g = torch.Generator().manual_seed(seed)
    for _ in range(epochs):
        for idx in torch.randperm(len(x), generator=g).split(batch):
            opt.zero_grad()
            lossf(net(x[idx]), y[idx]).backward()
            opt.step()
    return net


def iou(preds, labels):                                      # pooled over the set; near/far; pixels and BEV area
    out = []
    for w in (np.ones((H, W)), BEV_W):
        for region in (NEAR, FAR):
            tp = fp = fn = 0.0
            for p, t in zip(preds, labels):
                tp += (w * (p & t & region)).sum()
                fp += (w * (p & ~t & region)).sum()
                fn += (w * (~p & t & region)).sum()
            out.append(tp / (tp + fp + fn))
    return out


if __name__ == "__main__":
    torch.set_num_threads(4)
    train = [render(s) for s in range(300)]
    tests = {"paved": [render(s) for s in range(1000, 1100)],
             "unpaved": [render(s, unpaved=True) for s in range(2000, 2100)]}
    xtr = to_input([i for i, _ in train])
    ytr = torch.tensor(np.stack([m for _, m in train]), dtype=torch.float32)
    prior = ytr.numpy().mean(0) > 0.5                        # per-pixel mean of the training masks
    methods = {"prior map": lambda imgs: [prior] * len(imgs),
               "colour + homography": lambda imgs: [colour_baseline(i) for i in imgs]}
    for s in (0, 1, 2):
        net = train_fcn(xtr, ytr, seed=s, epochs=8)
        with torch.no_grad():
            methods[f"FCN, seed {s}"] = (lambda n: lambda imgs: list(n(to_input(imgs)).detach().numpy() > 0))(net)
    print(f"{'':21}{'image pixels':>26}{'BEV area':>24}")
    print(f"{'method':21}{'test':>9}{'near':>8}{'far':>8}{'near':>12}{'far':>8}")
    for name, f in methods.items():
        for tname, data in tests.items():
            r = iou(f([i for i, _ in data]), [m for _, m in data])
            print(f"{name:21}{tname:>9}{r[0]:8.3f}{r[1]:8.3f}{r[2]:12.3f}{r[3]:8.3f}")

Output (NumPy 2.5.4, SciPy 1.18.1, PyTorch 2.14.1 with 4 threads; two runs gave identical output; the first run stopped with an error before the FCN rows, because the prediction lacked a .detach(), and adding it was the only change):

                                   image pixels                BEV area
method                    test    near     far        near     far
prior map                paved   0.844   0.583       0.791   0.417
prior map              unpaved   0.846   0.557       0.790   0.400
colour + homography      paved   0.916   0.557       0.822   0.467
colour + homography    unpaved   0.863   0.355       0.703   0.265
FCN, seed 0              paved   0.851   0.141       0.669   0.055
FCN, seed 0            unpaved   0.006   0.000       0.003   0.000
FCN, seed 1              paved   0.867   0.358       0.753   0.183
FCN, seed 1            unpaved   0.000   0.000       0.000   0.000
FCN, seed 2              paved   0.864   0.382       0.737   0.230
FCN, seed 2            unpaved   0.000   0.000       0.000   0.000

Distance dominates every score. Every method that finds the road loses IoU from near to far, and loses more when pixels are weighted by ground area, the gap Fritsch et al. (2013) found between perspective and BEV scores on KITTI. The far band is 8 image rows for 28 m of road, against 36 rows for the 9 m of the near band, so errors there are cheap in pixels and expensive in metres.

The colour baseline, which re-estimates the road’s colour in every image from the patch the homography places in front of the car, is the best method near in image pixels on both test sets and the only one above the prior map in far BEV IoU on asphalt, though in far image pixels it is slightly below the prior. On the unpaved set its far BEV IoU falls from 0.467 to 0.265, and its near BEV IoU drops below the prior map’s. The plan had predicted that the FCN would drop on unpaved roads “much more than the colour baseline, which re-estimates the road colour in every image”; the FCN did drop more, but the colour baseline lost much of its far score too. In our reading, it fails where its assumption fails: a brown dirt road next to a dirt verge has nearly the same colour, and adapting to the road does not separate it from a verge of that colour.

The FCN did not do what the plan predicted, that on asphalt it “beats the colour baseline, mainly where shadows and road-coloured dirt shoulders fool the colour model, and far”. Trained for the 152 optimizer steps that fit the time budget, it learned the near road but not the far one: its far IoU on asphalt is 0.14 to 0.38 across seeds, below the prior map, and its seeds disagree most there; in our reading, because the far band gives it only a few rows per image to learn from. On dirt roads all three seeds mark almost nothing as drivable. Our reading is that the network learned “grey asphalt in a likely place”, so a road of a colour it never saw is not road to it: an extreme case of a learned model failing outside its training conditions. The example says nothing about large networks trained on large sets; on KITTI road the learned SPRAY classifier was the strongest method in the comparison above. It shows what each kind of method assumes, and that a near-field score in image pixels hides most of the difference between them.

Segmentation networks

Once large labeled sets existed, drivable area became an ordinary dense-prediction problem, solved with the fully convolutional encoder–decoder networks of semantic segmentation. BDD100K’s baseline treats it as three-way segmentation, background, directly and alternatively drivable, ignoring the region identities of the annotation (Yu et al., 2020, supplementary material). In the main paper’s multitask experiments, with a DLA-34 base network and a segmentation head of four $3 \times 3$ convolution blocks and a $1 \times 1$ convolution at a quarter of the input resolution, bilinearly upsampled, training on the full 70K training images gave 79.40 IoU for the direct area and 63.33 for the alternative area, 71.37 mean (Table 5; the paper does not state the evaluation split). Training jointly with the lane-marking task changed the drivable score little (72.21 mean with 70K images), which the authors attribute to the two tasks having a similar prediction structure. The supplementary material reports the same 79.4 and 63.3 for a DRN-D-22 model trained on the 70,000 training images, so the two documents name different networks for one pair of numbers. Its observation is the more general one: the same model reached much higher accuracy on road segmentation, which the authors take to mean that techniques beyond segmentation may be needed for drivable area. Distinguishing a lane the vehicle may use from one it may not is a question about traffic, not about pixels.

In the same paper, adding the lane-marking and drivable-area data to semantic segmentation training improved the IoU of road and sidewalk but lowered the overall mean IoU (56.9 to 53.3, Table 8).

Multi-task driving networks

Driving systems need detection, lanes and drivable area from the same frames, and running three networks costs three times the computation. Multi-task networks share one encoder among task heads.

YOLOP. The official repository describes YOLOP as a network for object detection, drivable area segmentation and lane detection, with one shared encoder and three decoders (Wu et al., code). In its model definition, the encoder is a stack of CSP bottleneck blocks with spatial pyramid pooling and an upsampling path that concatenates earlier feature maps, followed by an anchor-based YOLO-style detection head on three scales and two identical segmentation heads that upsample from the encoder’s stride-8 features to the input resolution and output two channels each, one head for drivable area and one for lane lines. The paper appeared in Machine Intelligence Research (2022) and was not read here; the description follows the code and README at commit 8d8f68d. The README reports a drivable-area mIoU of 91.5% for the end-to-end multi-task model at 41 frames per second, against 92.0% for a drivable-only model, and lists MultiNet (71.6), DLT-Net (71.3) and PSPNet (89.6) for comparison. The README does not name the split; the repository’s configuration evaluates on BDD100K val.

HybridNets. The HybridNets repository describes an end-to-end network for the same three tasks; its README reports a drivable mIoU of 90.5%, below YOLOP’s 91.5 in the same table, while reporting higher detection recall and mAP@0.5 (Vu et al., code). The two READMEs do not agree on every baseline: HybridNets lists Faster R-CNN at 77.2% recall and 55.6% mAP@0.5, YOLOP at 81.2% and 64.9%. The HybridNets paper is posted only on arXiv and was not read. The repository’s configuration names two segmentation classes, “road” and “lane”, and reads its drivable masks from a folder named as in YOLOP’s configuration; its test set is also BDD100K val.

These mIoU values are not on the BDD100K drivable-area scale of the previous section. They use a different class definition and a different average, as the section on evaluation metrics explains.

Datasets and Benchmarks

Benchmark	Year	Images	Classes	Evaluation
KITTI road	2013	289 train, 290 test (uu, um, umm)	road; ego-lane (um only)	BEV, MaxF, AP; corridor metric for lanes
Cityscapes	2016	5,000 fine, 20,000 coarse	road and other flat classes; no drivable class	IoU class and category
BDD100K drivable	2020	70K train, 10K val, 20K test	direct, alternative, background	mean IoU over direct and alternative
IDD	—	10,000 fine	road, parking, drivable fallback under “drivable”	four-level label hierarchy

KITTI road was built with Honda Research Institute Europe from KITTI recordings and has three urban categories: urban unmarked (uu, 98 training and 100 test images), urban marked two-way roads (um, 95 and 96) and urban roads with multiple marked lanes (umm, 96 and 94), 289 training and 290 test images in all, at 375 × 1242 pixels (KITTI road page; Fritsch et al., 2013, Section III and Table I). The paper’s abstract and text speak of 600 images while its Table I sums to 579; the benchmark page also lists 289 and 290. Frames were taken at least 20 m apart, from five days with low traffic, so the road is often fully visible. Annotation was done as polygons in the perspective image and then refined in the BEV, for example by keeping a constant road width at distances where a few pixels decide the boundary. Test labels are withheld, and on 9 February 2015 the ground truth was improved and all results recomputed (KITTI road page).

BDD100K collected 100,000 videos of 40 seconds each, crowd-sourced through Nexar from more than 50,000 rides in New York, the San Francisco Bay Area and other US regions, at 720p and 30 frames per second, with day and night in roughly equal numbers and image tags for six weather conditions, six scene types and three times of day (Yu et al., 2020). Drivable areas and lane markings share the same 70K training images. The paper notes that on highways and city streets drivable areas mostly stay within lanes, while in residential areas lanes are sparse and annotators infer the drivable area from the surroundings (Section 3.4). The licence quoted in the toolkit’s documentation, which covers “this software and/or data”, permits educational, research and not-for-profit use and grants commercial use to BDD and BAIR Commons members and their affiliates, referring other commercial users to UC Berkeley’s licensing office (BDD100K license.rst, at the cited commit).

Cityscapes does not define drivable area; its road class is the nearest equivalent. Thirty classes are grouped into eight categories; “flat” contains road, sidewalk, parking and rail track, and the benchmark evaluates 19 classes, ignoring parking and rail track, and reports both class and category scores, so category IoU for “flat” merges road with sidewalk (Cordts et al., 2016). The data come from 50 cities, mainly in Germany, in spring, summer and autumn, without heavy rain or snow.

IDD (India Driving Dataset) was collected around Hyderabad and Bangalore and their outskirts, with 10,000 images finely annotated with 34 classes from 182 drive sequences; its home page describes it as a dataset for unstructured environments, where lanes and the categories of traffic participants are less well defined than in earlier benchmarks (IDD pages, as of November 2026). Its label file maps each class to the Cityscapes label and train ids where one exists; “drivable fallback” has no Cityscapes counterpart, and “non-drivable fallback” has no Cityscapes label id but is mapped to the train id of Cityscapes “terrain” (AutoNUE anue_labels.py; cityscapesScripts labels.py). The dataset’s page says that models trained on it, unlike a Cityscapes-trained baseline, separate muddy drivable areas beside the road from the road itself. The paper introducing IDD was not read for this article, so the dataset is described only from its official pages and label file.

Evaluation Metrics

KITTI road. Fritsch et al. (2013) compute precision, recall and the F1-measure over pixels. For confidence maps the threshold is chosen to maximize F1, giving MaxF; AP is an 11-point average precision over recall levels, as in PASCAL VOC; accuracy, false positive rate and false negative rate are also reported at the MaxF threshold. All are computed in the BEV, over $-10$ to $10$ m laterally and 6 to 46 m ahead at 0.05 m per cell (an 800 × 400 grid). In BEV every square metre counts equally, so a missed strip of far road costs as much as a strip near the car.

For the ego-lane, the paper adds a behaviour-based measure. A single-track vehicle model generates driving-corridor hypotheses of a vehicle 2.2 m wide, short manoeuvres are chained and pruned by how much lane confidence each corridor covers, and the best corridor is compared with the ground truth: lateral precision, a longitudinal F1 and a hit rate (the fraction of corridors matched within 10% lateral mismatch) at 20, 30 and 40 m. The KITTI page lists the corridor metric for lane estimation and the pixel metrics for road estimation. In the paper’s experiment, the scene prior and SPRAY reached a similar longitudinal F1 near the vehicle (93.4% and 94.1% at 20 m), but the prior’s corridor ended at about 39 m, giving a 40 m hit rate of 0.0% against 47.4% (Table IV).

BDD100K. The documentation states that drivable area uses the same rule as semantic segmentation, the intersection over union per class pooled over the dataset and averaged over classes, with background excluded from the final mean. It also says that, unlike semantic segmentation, background pixels are taken into account when IoUs are computed (BDD100K evaluate.rst). The format page says instead that background “is not considered during evaluation” (BDD100K format.rst), and the evaluation code at the cited commit follows that narrower rule. Its confusion matrix keeps only pixels whose ground truth is direct or alternative, so ground-truth background pixels never enter it, and a prediction of drivable area on true background is not counted as a false positive (bdd100k/eval/seg.py, function fast_hist). We re-implemented that logic in a few lines rather than run the toolkit: for a mask whose drivable pixels are all direct, a prediction that marks the whole image as directly drivable scores a direct-area IoU of 1, the same as an exact prediction. The evaluation page disagrees with the format page and the code, the documentation does not say which rule the evaluation server follows, and anyone comparing numbers should check which rule produced them.

Multi-task repositories. YOLOP’s evaluation uses two classes, background and drivable, and its mIoU is the mean of the two IoUs, computed for each validation batch at the network’s input resolution and averaged over batches (lib/core/function.py, lib/core/evaluate.py). Its label-generation script draws BDD100K’s directly drivable polygons in white and its alternative polygons in black, the background colour (toolkits/datasetpre/gen_bdd_seglabel.py); the masks the README links for download were not inspected. HybridNets computes IoU for each image and reports, for the road class, the average over images of the mean of the background and road IoUs (val.py). Background is the largest and easiest class, so averaging it in raises the score: a mean of road and background IoUs above 90% and BDD100K’s mean over direct and alternative near 71% measure different things (our reasoning). The README tables are comparable across rows only if every method was re-evaluated with the repository’s own protocol, which the READMEs do not state.

Perspective against BEV. Pixel scores in the image are dominated by the near road. Fritsch et al. (2013) found every method’s scores lower in the BEV than in the perspective image, and the ego-lane scores of the scene prior and SPRAY similar in perspective but clearly different in BEV (Table III: MaxF 88.9 against 88.3 in perspective, 74.4 against 83.9 in BEV). They also note that accuracy is dominated by the more frequent class, so it looks high for the ego-lane, which covers a small part of the BEV.

Applications

• Driver assistance on unmarked roads. Fritsch et al. (2013) motivate road and ego-lane detection by lane keeping assistance, which in their description was limited to marked roads with smooth curvature, and by the need to support driving on unmarked rural and inner-city roads.
• Driving decisions. BDD100K’s drivable categories are defined around where the vehicle may drive given lanes, other vehicles and the right of way (Yu et al., 2020).
• Obstacle detection and lane finding. In stereo systems the road surface and free space are by-products of obstacle detection, and Labayrade et al. (2002) suggest using the free-space area to extract lane markings under better conditions.
• Multi-task perception. YOLOP and HybridNets produce drivable area alongside vehicle detections and lane lines from one network; the HybridNets repository also includes a ROS package for path planning (Vu et al., code).

All results above are measurements on recorded benchmark data. The sources cited make no claims about safety on the road, and none is made here.

Open Problems

• Function, not appearance. BDD100K’s alternative area scored about 16 IoU points below the direct area in the paper’s baseline, and the authors suggest that techniques beyond segmentation may be required (Yu et al., 2020); what a vehicle may drive on depends on rules and other road users.
• Far range. BEV evaluation in KITTI shows that methods that look good in the image lose accuracy far ahead, and the scene prior’s corridor ended at about 39 m (Fritsch et al., 2013).
• Unstructured and unmarked roads. IDD’s page describes roads where lane discipline is often not followed and drivable ground extends beside the road; the KITTI paper calls unmarked roads an unsolved problem in 2013.
• Weather and coverage. Cityscapes excluded heavy rain and snow, and BDD100K’s tagging baseline did not recognize a single foggy scene; most drivable-area benchmarks come from one country each.
• Comparable protocols. Drivable-area numbers on BDD100K are reported under at least three definitions (the toolkit’s documentation, its code, and the two-class mIoU of the multi-task repositories), so published comparisons require care.