Traffic Sign Recognition

Traffic sign recognition finds the traffic signs in an image taken from a road vehicle and says which sign each one is: a 60 km/h limit, a stop sign, a “keep right” arrow. It is also called road sign recognition or TSR, and it is usually split into two steps, detection (where are the signs?) and classification (which sign is this?), which benchmarks have often measured separately. Driver assistance systems that show the current speed limit, mapping services that keep an inventory of signs, and the perception stacks of automated vehicles all depend on it (Stallkamp et al., 2012; Ertler et al., 2020).

Signs are designed to be seen. They use a few colours and shapes, stand at predictable places and carry simple pictograms, so the task looks easy, and humans recognize cut-out sign images almost perfectly. The difficulty lies in the conditions: signs are small in the image, lit by sun, dusk or headlights, blurred by motion, faded, covered by stickers or branches, and drawn from a large number of classes whose members differ only in a digit or the direction of an arrow (Stallkamp et al., 2012). The task builds on image classification and object detection, and it is one of the camera tasks surveyed in autonomous driving perception.

Problem Definition

In the classification formulation, the input is a crop that contains exactly one sign, and the output is one of $K$ classes. The German Traffic Sign Recognition Benchmark (GTSRB) set this form: $K = 43$, crops with a margin of 10% (at least 5 pixels) around the sign, and the sign’s bounding box given in the annotation (Stallkamp et al., 2012).

In the detection formulation, the input is a full camera frame and the output is a set of boxes. The German Traffic Sign Detection Benchmark (GTSDB) asks for boxes of one sign category at a time (prohibitory, mandatory or danger signs), without the class within the category (Houben et al., 2013). Later benchmarks ask for both at once: each box must carry the specific class, among 45 classes in the Tsinghua-Tencent 100K evaluation and 400 in the Mapillary Traffic Sign Dataset (Zhu et al., 2016; Ertler et al., 2020).

The two formulations differ in what they assume. Zhu et al. (2016) argue that GTSRB is not representative of real tasks, because in GTSRB the sign fills most of the image and there are no negative samples, whereas in a real frame the sign often covers less than 1% of the image and the system must reject many sign-like objects.

Input and Output

The input is a colour image from a forward-facing camera, or a crop of one. GTSRB images come from a Prosilica GC 1380CH camera recording Bayer-pattern frames of 1360×1024 pixels at 25 frames per second; GTSDB uses the same camera with frames clipped to 1360×800, because the lower part shows mostly the bonnet (Stallkamp et al., 2012; Houben et al., 2013). Tsinghua-Tencent 100K uses 2048×2048 crops of street-view panoramas, and the Mapillary Traffic Sign Dataset uses street-level photographs from more than 200 camera makes, from about 1 to more than 16 megapixels (Zhu et al., 2016; Ertler et al., 2020).

The output is a class label per crop, or a list of bounding boxes with a class label and a confidence score each. Tsinghua-Tencent 100K also annotates a pixel mask for every sign (Zhu et al., 2016), and the Mapillary dataset adds attributes such as occluded, ambiguous (not classifiable at all), dummy (looks like a sign but is not, such as a car sticker), out-of-frame, included (part of a larger sign) and exterior (contains other signs) (Ertler et al., 2020).

Variants

• Crop classification. A classifier receives a crop from an upstream detector or an annotation, as in GTSRB.
• Category-level detection. A detector finds signs of one category, as in GTSDB’s prohibitory, mandatory and danger tracks.
• Class-agnostic detection. The Mapillary dataset defines a binary task, finding every traffic sign without its class (Ertler et al., 2020).
• Joint detection and fine-grained classification. Boxes with specific classes from a large taxonomy, as in Tsinghua-Tencent 100K and the Mapillary dataset. With hundreds of classes that differ in small details, this overlaps with fine-grained image classification.
• Recognition over tracks. A vehicle sees the same sign in many frames. GTSRB keeps each physical sign’s images grouped as a track in the training set, so that methods can exploit temporal information, but its test set is shuffled and has none (Stallkamp et al., 2012); Sermanet and LeCun (2011) built their classifier without temporal evidence accumulation for that reason.
• Signs outside the taxonomy. Tsinghua-Tencent 100K puts sign-like objects with other meanings into an “other” class of each category, and the Mapillary dataset labels every sign outside its 400 classes other-sign (Zhu et al., 2016; Ertler et al., 2020).

Challenges

The sources name the same difficulties from different data:

• Similar classes. Signs with the same general meaning share their appearance, such as the speed limits, which differ in their digits (Stallkamp et al., 2012). In the first stage of the GTSRB competition, the errors of humans and machines alike clustered in the speed-limit group, and the classifiers built on HOG features confused triangular danger signs with each other: the features captured the sign’s outline but not its pictogram (Stallkamp et al., 2011).
• Image quality. Low resolution, low contrast, motion blur, reflections, saturation, sun glare, fading, graffiti, stickers and physical damage (Sermanet & LeCun, 2011; Stallkamp et al., 2012). In GTSRB the smallest signs measure 15×15 pixels.
• Small objects in large images. A typical sign might be 80×80 pixels in a 2000×2000 image, about 0.2% of it, against an average of about 20% for objects in PASCAL VOC and ImageNet, by Zhu et al.’s (2016) count.
• Sign-like distractors. Advertisements, banners and billboards (Ertler et al., 2020), and sign-like objects that mean something else (Zhu et al., 2016).
• Imbalance and the long tail. In GTSRB a third of all images are speed limits, and the class frequencies are strongly unequal (Stallkamp et al., 2012). Zhu et al. (2016) found that some Chinese warning signs, such as those for mountain roads, appear rarely, and kept only the classes with at least 100 instances for training. The Mapillary dataset’s class distribution has a long tail over 400 classes (Ertler et al., 2020).
• Regional differences. Sign conventions differ between countries. Sermanet and LeCun (2011) note that colour and shape assumptions, such as the radial symmetry of round speed signs, do not hold in every country, for example for United States speed signs, and Ertler et al. (2020) describe appearance differences across regions and the lack of a complete public catalogue of signs for many countries.

Approaches

Colour and shape cues

Most detection approaches reviewed by Houben et al. (2013) used the two properties that sign designers chose for visibility: colour and shape. A typical classical pipeline ran cheap, hand-designed detection first, for example colour thresholding, and spent a more expensive classifier only on the candidates it returned (Sermanet & LeCun, 2011).

Colour reduces the search: pixels are mapped to a likelihood of carrying a sign colour, usually in a colour space that separates hue from brightness. Houben et al. (2013) describe the treatment of colour under natural lighting as difficult, with many heuristics in the literature. Shape then separates signs from other red or blue objects. Houben et al. (2013) distinguish two families of shape methods: model-based ones, which detect edges and connect them into regular polygons or circles, usually by Hough-like voting or template matching, and Viola–Jones-like ones, which train cascades of weak classifiers on fast features.

As the model-based baseline of GTSDB, Houben et al. (2013) implemented a scheme of this kind: a per-pixel colour likelihood from a mixture of Gaussians in YUV space, centred on the median colours of the training signs, followed by a Canny edge detector and a Hough-like vote that needs only two edge pixels to fix a circle and three to fix a triangle. They expected it to beat their general-purpose baselines. It did not: by area under the precision–recall curve, a Viola–Jones cascade scored higher in every category, and a HOG detector with a linear classifier trained by linear discriminant analysis (LDA) scored higher in two of the three (on mandatory signs its 12.01% was just below the model-based method’s 12.86%). The authors concluded that a classic general-purpose detector “clearly outperformed” the model-based approach.

For classification, the classical recipe was a hand-designed descriptor and a standard classifier. GTSRB shipped such descriptors with the images so that participants without an image-processing background could take part: three HOG variants with 1,568, 1,568 and 2,916 dimensions, 11,584 Haar-like features computed on 40×40 grayscale crops, and a 256-bin hue histogram (Stallkamp et al., 2012).

Learned features and the IJCNN 2011 competition

GTSRB was run as a competition at the International Joint Conference on Neural Networks (IJCNN) 2011, in two stages (Stallkamp et al., 2011; 2012). The first stage was online: the training set was published seven weeks before the evaluation, and participants then had three days to classify a test set and upload results. More than 20 teams took part, with support vector machines, LDA, subspace analysis, ensembles, slow feature analysis, k-d trees, random forests and several kinds of neural networks. Since the submissions were not limited at first, some teams flooded the leaderboard with variants, which the organizers saw as optimization against the test set; they introduced a limit of ten submissions during the stage, and wrote that they would set three to five in future (Stallkamp et al., 2011). The second, final stage took place at the conference on a test set that had not been published, partly to prevent cheating by hand-labelling the test images (Stallkamp et al., 2011; 2012). The benchmark overfitting article discusses the general problem.

In the first stage, the team from IDSIA (Cireşan, Meier, Masci and Schmidhuber) won with 98.98% correct with a committee of a convolutional neural network on raw pixels and a multilayer perceptron on the supplied HOG features; Sermanet and LeCun came second with 98.97% (Stallkamp et al., 2011). Stallkamp et al.’s (2011) results table prints 99.81% for human performance, measured with 36 people classifying chunks of 350 images, but places that row below the 98.97% entry, and their conclusion calls 98.98% comparable to human performance; Sermanet and LeCun (2011) give the human figure as 98.81%. The table’s value may be a misprint, which we could not confirm.

The two leading teams took different routes:

• Multi-scale CNN (Sermanet & LeCun, 2011). Their two-stage ConvNet fed the pooled output of the first stage to the classifier together with the second stage’s output, so that it could use local detail as well as global shape. It used a rectified sigmoid followed by subtractive and divisive local normalization, 32×32 inputs in YUV, and a training set enlarged fivefold with random shifts of up to 2 pixels, scalings of 0.9–1.1 and rotations of up to 15°. After the first stage they reported 99.17% with a larger network on grayscale input, better than with colour, and 97.33% even with random, untrained convolutional features.
• Multi-column deep neural network (Cireşan et al., 2012). For the final stage, IDSIA dropped the perceptrons and averaged 25 convolutional networks: five networks for each of five versions of the training data, the original crops and four contrast normalizations (intensity stretching, global and adaptive histogram equalization, and difference-of-Gaussians filtering). Each network saw the sign’s bounding box resized to 48×48 and was trained on images randomly translated by up to 10%, scaled by 0.9–1.1 and rotated by up to 5°, redrawn every epoch. The individual networks averaged 98.52% and the committee reached 99.46%.

The final-stage results, from Stallkamp et al. (2012), with the correct classification rate on the final test set:

Entry	Correct classification rate (%)
IDSIA, committee of CNNs	99.46
Human, best individual	99.22
Human, average	98.84
Sermanet, multi-scale CNN	98.31
CAOR, random forest on HOG	96.14
LDA on HOG set 2 (baseline)	95.68

The committee was significantly better than the best individual human (McNemar’s test, $p = 0.01366$), and the best human was significantly better than every other algorithm (Stallkamp et al., 2012). The Deep Learning article places this result in the history of GPU-trained networks. Two details of the sources are worth keeping in mind. The two reports of the IDSIA system disagree on training time: Cireşan et al. (2012) give 37 hours for the 25-column committee on four GPUs, while Stallkamp et al. (2012) give about 2 hours per network and about 50 hours in total. And the human figures were measured on isolated crops in an unfamiliar interface, which Stallkamp et al. (2012) point out does not reflect real traffic.

Cireşan et al. (2012) also report how confident their committee was: most predicted probabilities were close to zero or one, more than 80% of its 68 errors had the correct class as second choice, and rejecting the 1% of images with confidence below 0.51 lowered the error rate to 0.24%.

Why did learned features win? The competition reports give part of the answer from the errors. In the first stage, most entries used the supplied HOG features, which contain no colour, and they often mistook blue mandatory signs for speed limits; the multi-scale CNN, the only entry there to use colour, almost never did (Stallkamp et al., 2011). Together with the danger-sign confusions described under Challenges, this suggests, in our reading, that features learned from pixels could represent the pictograms as well as the outlines.

Detectors and two-stage systems

Once detection became the bottleneck, the field turned to learned detectors. GTSDB’s 2013 competition attracted 18 teams and more than 110 result files (Houben et al., 2013). The three teams the paper describes all used sliding-window or region-of-interest pipelines with learned or engineered features: a colour classification, shape matching and HOG-plus-colour-histogram SVM pipeline (LITS1); an integral channel features detector (VISICS); and a coarse-to-fine sliding window with HOG features classified by LDA and then an intersection-kernel SVM, followed by non-maximum suppression (wgy@HIT501).

On full-resolution street views, general-purpose CNN detectors had to be adapted to small objects. Zhu et al. (2016) found that the generic proposal methods of the time (Selective Search, Edge Boxes, BING) recalled under 70% of the signs in Tsinghua-Tencent 100K even with 10,000 proposals per image. They trained instead a fully convolutional network that branches after its sixth layer into three heads: a pixel head giving the probability that each 4×4 cell contains a sign, a box head, and a label head with the class. Because many classes were rare, they kept the 45 classes with at least 100 instances and augmented the rarer ones to 1,000 instances by pasting rotated, scaled and perspective-distorted sign templates into images without signs.

Ertler et al. (2020) report that a Faster R-CNN with a 401-way classification head (400 classes and other-sign) did not reach acceptable performance in preliminary experiments, and hypothesize two causes: the small size of signs before RoI pooling and to batches dominated by other-sign instances. Their baseline is therefore two-stage: a class-agnostic Faster R-CNN with a feature pyramid network and ResNet backbone, trained on 1000×1000 crops at full resolution so that small signs do not vanish, followed by a separate shallow classifier (seven 3×3 convolutions with batch normalization) on 40×40 crops, trained with batches of 100 classes × 4 samples plus 100 background crops. In the authors’ account, decoupling the detector from the classifier addresses both problems: the classifier sees every sign at the same resolution, and the class balance of its batches can be set directly.

Example: corruptions and mirror-image classes

The Data Augmentation article shows with a two-class arrow task that random horizontal flips destroy the labels of classes defined by direction. Sign sets contain such classes next to symmetric ones: GTSRB’s 43 classes include turn-left and turn-right signs, keep-left and keep-right signs, and left and right curves (Stallkamp et al., 2011, Fig. 3). The example below asks two other questions on synthetic signs. First, which image corruptions hurt a classifier trained on clean crops, which classes do they hurt, and where do the errors go? Second, in a multi-class problem, does flipping damage only the mirror classes, and does it become a valid augmentation when the flip also swaps the left and right labels?

The script renders 32×32 colour crops of eight classes in the style of European signs, each on a random cluttered background with random size, position, rotation, colour and brightness: four horizontally symmetric classes (an octagonal stop sign with a white bar, an inverted triangle, a red no-entry disc and a blue “ahead only” arrow) and two mirror pairs (turn left and turn right, keep left and keep right). Flipping a turn-left template gives exactly the turn-right template, so swapping labels under a flip is exact here. A small CNN with batch normalization and global average pooling trains for 10 epochs on 2,400 crops, then is tested on 1,600 new crops, clean and under five corruptions: Gaussian blur with $\sigma = 1$ and $2$ pixels, a grey square covering 40% of the sign’s bounding square at a random position, darkening to a quarter of the brightness with added noise, and overexposure by a gain of 3 with clipping. Three seeds are run, and for each the model is also trained with random horizontal flips, either keeping the label or swapping left and right labels. The script uses PyTorch 2.14.1 on the CPU with one thread, NumPy 2.5.4 and SciPy 1.18.1, and takes about 1.5 minutes.

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from scipy.ndimage import gaussian_filter

torch.set_num_threads(1)
S, UP = 32, 4                      # output size, supersampling factor
NAMES = ["stop", "yield", "no entry", "ahead only",
         "turn left", "turn right", "keep left", "keep right"]
SYM, MIR = [0, 1, 2, 3], [4, 5, 6, 7]
PARTNER = np.array([0, 1, 2, 3, 5, 4, 7, 6])   # label after a horizontal flip
RED, BLUE, WHITE = (0.8, 0.1, 0.1), (0.1, 0.25, 0.75), (0.95, 0.95, 0.95)

def seg(u, v, a, b, w):
    """Mask of points within w of the segment a-b (sign coordinates)."""
    p, d = np.stack([u - a[0], v - a[1]]), np.array(b) - np.array(a)
    t = np.clip((p[0] * d[0] + p[1] * d[1]) / (d @ d), 0, 1)
    return (p[0] - t * d[0]) ** 2 + (p[1] - t * d[1]) ** 2 <= w * w

def tri(u, v, p0, p1, p2):
    """Mask of points inside the triangle p0-p1-p2."""
    def side(a, b):
        return (b[0] - a[0]) * (v - a[1]) - (b[1] - a[1]) * (u - a[0])
    s = [side(p0, p1), side(p1, p2), side(p2, p0)]
    return ((s[0] >= 0) & (s[1] >= 0) & (s[2] >= 0)) | \
           ((s[0] <= 0) & (s[1] <= 0) & (s[2] <= 0))

def symbol(c, u, v):
    """White symbol mask for class c; u points right, v points down."""
    if c == 0: return seg(u, v, (-0.55, 0), (0.55, 0), 0.13)
    if c == 2: return seg(u, v, (-0.65, 0), (0.65, 0), 0.17)
    if c == 3: return seg(u, v, (0, 0.65), (0, -0.2), 0.12) | \
                      tri(u, v, (0, -0.7), (-0.33, -0.2), (0.33, -0.2))
    if c in (4, 5):                # shaft up, then a bend to the left (mirrored for right)
        s = -1 if c == 4 else 1
        return seg(u, v, (-0.2 * s, 0.65), (-0.2 * s, -0.1), 0.12) | \
               seg(u, v, (-0.2 * s, -0.1), (0.2 * s, -0.1), 0.12) | \
               tri(u, v, (0.62 * s, -0.1), (0.2 * s, -0.43), (0.2 * s, 0.23))
    if c in (6, 7):                # diagonal arrow pointing down-left (mirrored for right)
        s = -1 if c == 6 else 1
        tip, base = np.array([0.5 * s, 0.5]), np.array([-0.45 * s, -0.45])
        d = (tip - base) / np.linalg.norm(tip - base); n = np.array([-d[1], d[0]])
        h = tip - 0.42 * d
        return seg(u, v, base, h, 0.12) | tri(u, v, tip, h + 0.3 * n, h - 0.3 * n)
    return np.zeros_like(u, bool)

def render(c, rng, clean=False):
    """One S x S RGB crop of class c on clutter; returns image, centre and radius."""
    H = S * UP
    if clean:
        r, cx, cy, th, gain = 12.0, S / 2, S / 2, 0.0, 1.0
        bg = np.full((H, H, 3), 0.5)
        jit = np.zeros(3)
    else:
        r = rng.uniform(9, 14); cx, cy = S / 2 + rng.uniform(-2, 2, 2)
        th = np.deg2rad(rng.uniform(-8, 8)); gain = rng.uniform(0.8, 1.2)
        low = gaussian_filter(rng.random((S, S, 3)), (3, 3, 0))
        low = (low - low.min()) / (np.ptp(low) + 1e-9) * rng.uniform(0.3, 0.8) + rng.uniform(0, 0.3)
        bg = np.kron(low, np.ones((UP, UP, 1)))
        jit = rng.uniform(-0.07, 0.07, 3)
    y, x = (np.mgrid[0:H, 0:H] + 0.5) / UP
    dx, dy = (x - cx) / r, (y - cy) / r
    u = np.cos(th) * dx + np.sin(th) * dy
    v = -np.sin(th) * dx + np.cos(th) * dy
    img = bg.copy()
    red, blue, white = (np.clip(np.array(col) + jit, 0, 1) for col in (RED, BLUE, WHITE))
    if c == 0:
        oct_ = np.maximum(np.maximum(abs(u), abs(v)), (abs(u) + abs(v)) / np.sqrt(2)) <= 1
        img[oct_] = red
    elif c == 1:
        outer = tri(u, v, (-1, -0.7), (1, -0.7), (0, 1.0))
        inner = tri(u, v, (-0.62, -0.48), (0.62, -0.48), (0, 0.55))
        img[outer] = red; img[inner] = white
    else:
        disc = u * u + v * v <= 1
        img[disc] = red if c == 2 else blue
    img[symbol(c, u, v)] = white
    img = img.reshape(S, UP, S, UP, 3).mean((1, 3)) * gain
    if not clean:
        img = img + rng.normal(0, 0.02, img.shape)
    return np.clip(img, 0, 1).astype(np.float32), (cx, cy), r

def dataset(n_per_class, rng):
    X, y, meta = [], [], []
    for c in range(8):
        for _ in range(n_per_class):
            im, ctr, r = render(c, rng)
            X.append(im); y.append(c); meta.append((ctr[0], ctr[1], r))
    return np.stack(X), np.array(y), np.array(meta)

def corrupt(X, meta, kind, rng):
    X = X.copy()
    for i in range(len(X)):
        if kind == "clean":
            continue
        if kind.startswith("blur"):
            X[i] = gaussian_filter(X[i], (float(kind.split()[1]),) * 2 + (0,))
        elif kind == "occlusion 40%":
            cx, cy, r = meta[i]; side = 0.63 * 2 * r      # 40% of the sign's bounding square
            x0 = rng.uniform(cx - r, cx + r - side); y0 = rng.uniform(cy - r, cy + r - side)
            X[i, int(round(y0)):int(round(y0 + side)), int(round(x0)):int(round(x0 + side))] = 0.5
        elif kind == "dark x0.25":
            X[i] = X[i] * 0.25 + rng.normal(0, 0.02, X[i].shape)
        elif kind == "overexposed x3":
            X[i] = X[i] * 3.0
    return np.clip(X, 0, 1).astype(np.float32)

def net():
    return nn.Sequential(
        nn.Conv2d(3, 16, 3, padding=1), nn.BatchNorm2d(16), nn.ReLU(), nn.MaxPool2d(2),
        nn.Conv2d(16, 32, 3, padding=1), nn.BatchNorm2d(32), nn.ReLU(), nn.MaxPool2d(2),
        nn.Conv2d(32, 64, 3, padding=1), nn.BatchNorm2d(64), nn.ReLU(),
        nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(64, 8))

def train(X, y, flip, seed, epochs=10):
    torch.manual_seed(seed); gen = torch.Generator().manual_seed(seed)
    Xt = torch.tensor(X).permute(0, 3, 1, 2); yt = torch.tensor(y)
    model = net(); opt = torch.optim.Adam(model.parameters(), lr=3e-3)
    partner = torch.tensor(PARTNER)
    model.train()
    for _ in range(epochs):
        for idx in torch.randperm(len(Xt), generator=gen).split(64):
            xb, yb = Xt[idx].clone(), yt[idx].clone()
            if flip != "no flip":
                sel = torch.rand(len(xb), generator=gen) < 0.5
                xb[sel] = xb[sel].flip(-1)
                if flip == "flip + swap labels":
                    yb[sel] = partner[yb[sel]]
            loss = F.cross_entropy(model(xb), yb)
            opt.zero_grad(); loss.backward(); opt.step()
    return model

def predict(model, X):
    model.eval()
    with torch.no_grad():
        return model(torch.tensor(X).permute(0, 3, 1, 2)).argmax(1).numpy()

def scores(pred, y):
    sym, mir = np.isin(y, SYM), np.isin(y, MIR)
    wrong_mir = mir & (pred != y)
    to_partner = (pred[wrong_mir] == PARTNER[y[wrong_mir]]).mean() if wrong_mir.any() else np.nan
    return (pred == y).mean(), (pred[sym] == y[sym]).mean(), (pred[mir] == y[mir]).mean(), to_partner

if __name__ == "__main__":
    kinds = ["clean", "blur 1", "blur 2", "occlusion 40%", "dark x0.25", "overexposed x3"]
    flips = ["no flip", "flip, same label", "flip + swap labels"]
    seeds = [0, 1, 2]
    res = {k: [] for k in kinds}; conf = {k: np.zeros((8, 8), int) for k in kinds}
    fres = {f: [] for f in flips}
    for seed in seeds:
        rng = np.random.default_rng(seed)
        Xtr, ytr, _ = dataset(300, rng)
        Xte, yte, mte = dataset(200, rng)
        for f in flips:
            model = train(Xtr, ytr, f, seed)
            fres[f].append(scores(predict(model, Xte), yte)[1:3])
            if f == "no flip":
                for k in kinds:
                    pred = predict(model, corrupt(Xte, mte, k, rng))
                    res[k].append(scores(pred, yte))
                    np.add.at(conf[k], (yte, pred), 1)
    print(f"PyTorch {torch.__version__}; 2,400 training and 1,600 test crops per seed; seeds {seeds}")
    print("\nTrained without flips, tested under corruption (mean over seeds, min-max in brackets)")
    print(f"{'condition':<16}{'all':>18}{'symmetric':>18}{'mirror pairs':>18}"
          f"{'mirror errs->partner':>22}   most frequent error (share of errors)")
    for k in kinds:
        a = np.array(res[k])
        cells = ["-" if np.isnan(m) else f"{m:.3f} ({lo:.2f}-{hi:.2f})"
                 for m, lo, hi in zip(a.mean(0), a.min(0), a.max(0))]
        off = conf[k] * (1 - np.eye(8, dtype=int))
        if off.sum():
            t, p = np.unravel_index(off.argmax(), off.shape)
            top = f"{NAMES[t]} -> {NAMES[p]} ({off[t, p] / off.sum():.2f})"
        else:
            top = "-"
        print(f"{k:<16}" + "".join(f"{c:>18}" for c in cells[:3]) + f"{cells[3]:>22}   {top}")
    print("\nClean test accuracy by training regime (symmetric / mirror-pair classes), per seed")
    for f in flips:
        print(f"{f:<20}" + "".join(f"   seed {s}: {a:.3f} / {b:.3f}" for s, (a, b) in zip(seeds, fres[f])))

Output:

PyTorch 2.14.1; 2,400 training and 1,600 test crops per seed; seeds [0, 1, 2]

Trained without flips, tested under corruption (mean over seeds, min-max in brackets)
condition                      all         symmetric      mirror pairs  mirror errs->partner   most frequent error (share of errors)
clean            1.000 (1.00-1.00) 1.000 (1.00-1.00) 1.000 (1.00-1.00)                     -   -
blur 1           0.895 (0.78-1.00) 0.841 (0.69-1.00) 0.949 (0.86-1.00)     0.000 (0.00-0.00)   no entry -> stop (0.37)
blur 2           0.366 (0.27-0.46) 0.433 (0.27-0.53) 0.299 (0.26-0.38)     0.113 (0.01-0.32)   no entry -> stop (0.20)
occlusion 40%    0.545 (0.53-0.55) 0.755 (0.70-0.79) 0.334 (0.28-0.41)     0.115 (0.08-0.16)   keep left -> turn right (0.14)
dark x0.25       0.525 (0.31-0.68) 0.610 (0.46-0.72) 0.441 (0.17-0.71)     0.001 (0.00-0.00)   turn left -> ahead only (0.22)
overexposed x3   0.797 (0.73-0.85) 0.811 (0.76-0.84) 0.783 (0.63-0.87)     0.198 (0.00-0.37)   stop -> no entry (0.36)

Clean test accuracy by training regime (symmetric / mirror-pair classes), per seed
no flip                seed 0: 1.000 / 1.000   seed 1: 1.000 / 1.000   seed 2: 1.000 / 1.000
flip, same label       seed 0: 1.000 / 0.365   seed 1: 1.000 / 0.490   seed 2: 1.000 / 0.471
flip + swap labels     seed 0: 1.000 / 1.000   seed 1: 1.000 / 1.000   seed 2: 1.000 / 1.000

Two runs printed identical output. Every model trained without flips classified the clean test crops perfectly, so all the losses in the first table come from the corruptions, which the training data never showed.

The corruptions hurt different classes for different reasons. Mild blur cost more on the symmetric classes than on the arrows, and its most frequent error was no-entry read as stop. In our reading, the two red signs differ only in their outline, a circle against an octagon with corners, and in the size of the white bar, and blur removes the corners before it removes the arrows. Stronger blur cut accuracy below half for both groups. Occlusion behaved the other way round: symmetric classes kept 70–79% accuracy and the arrow classes fell to 28–41%; in our reading, colour and outline survive a covered patch, while the only cue separating the blue arrow classes is a thin white symbol that the square often covers. Darkening also hit hard; in our reading, a network trained only at normal brightness never learned that a dim blue disc with a faint arrow is still the same sign. Overexposure pushed the two red classes into each other again (stop read as no entry). The seed ranges are wide for blur and darkening although every model was perfect on its clean test crops: models that cannot be told apart on clean data (each seed renders its own training and test sets) differ in how they fail, so a clean test set says little about robustness to conditions it does not contain.

Errors on the mirror classes rarely went to the mirror partner (0–20% on average). In our reading, the networks lost the arrow itself rather than its direction: under occlusion and darkening the most frequent error named another blue sign. This matches the pattern in the competition reports, where errors stayed within groups of similar-looking signs (Stallkamp et al., 2012).

The flip experiment confirms the per-class picture. Random flips that keep the label left the symmetric classes at 100% and pushed the mirror classes to 37–49%, near the 50% of guessing within a pair: every training arrow appeared pointing both ways under one label. Swapping the left and right labels whenever an image is flipped restored 100% on all classes, because the flipped image is then a correctly labelled example of the partner class. Whether that holds for real signs is a separate check, since some mirror pairs are not exact mirror images of each other (text, for example, does not mirror).

The first run of this script used the same network without batch normalization, and it reached only 43–61% clean accuracy after 10 epochs, so its comparisons were meaningless. Diagnostic runs on clean data, not shown, found its training loss stalling in the early epochs and then tried batch normalization after each convolution, which was adopted with nothing else changed. A second change after that run added the last printed column, the most frequent confusion pooled over the three seeds, so that the text could say where errors go from printed output, and printed a dash in place of an undefined value in the clean row. The other printed numbers were unchanged by that addition. The image sizes, class designs, corruption strengths and network were chosen by hand for illustration.

Datasets and Benchmarks

Benchmark	Task	Images	Signs	Classes	Region
GTSRB (2011)	crop classification	51,840 crops	one per crop	43	Germany
GTSDB (2013)	detection by category	900	1,206	3 competition categories	Germany
Tsinghua-Tencent 100K (2016)	detection and classification	100,000 (10,000 with signs)	about 30,000	45 evaluated	China
Mapillary Traffic Sign Dataset (2020)	detection, detection and classification	52,453 fully annotated, 53,377 partially	257,541 boxes in the fully annotated set	400	worldwide

Counts are from the original papers (Stallkamp et al., 2012; Houben et al., 2013; Zhu et al., 2016; Ertler et al., 2020). Other tables count the public releases differently: Ertler et al. (2020) list 26,349 signs and 221 classes for Tsinghua-Tencent 100K and 852 signs for GTSDB.

GTSRB. The data come from about 10 hours of video recorded on German roads in daytime in March, October and November 2010 (Stallkamp et al., 2012). From 144,769 labelled images of 2,416 physical signs in 70 classes, the authors dropped tracks with fewer than 30 images and classes with fewer than 9 tracks, then sampled 30 images evenly from each remaining track, because slowly passed signs would otherwise contribute many near-identical images and favour classes common in slow traffic, such as stop, yield and low speed limits. The result is 51,840 images of more than 1,700 physical signs in 43 classes; the first-stage paper, written earlier, gives about 133,000 labelled images and about 50,000 selected ones (Stallkamp et al., 2011). The split is random but stratified by class and made at the level of tracks, so that all images of one physical sign fall into the same set: half for the basic training set and a quarter each for validation and test. For the final stage, the training set had 39,209 images and the test set 12,630 (Cireşan et al., 2012), together one image fewer than 51,840.

The track structure matters for any further split. Sermanet and LeCun (2011) note that each physical sign yields 30 images, usually at growing resolution as the camera approaches, so a validation set drawn by shuffling all images would contain near-copies of training images and would not predict test performance; they held out one track per class (1,290 images). The same rule applies to cross-validation on any data recorded as sequences. Stallkamp et al. (2012) make the point about earlier work too: for two of the earlier traffic sign studies they review, it is unclear whether images of the same physical sign were shared between training and test sets.

GTSDB. The 900 images were selected from drives near Bochum in spring and autumn 2010, in urban, rural and highway scenes, in daytime and at dusk and in various weather (Houben et al., 2013). Signs are 16 to 128 pixels on their longer side. The split is 600 training images with 846 signs and 300 evaluation images with 360 signs, with the evaluation ground truth kept on the server and images of the same physical sign kept in the same set. Signs outside the three competition categories are annotated but not evaluated.

Tsinghua-Tencent 100K. Zhu et al. (2016) took 100,000 panoramas from Tencent Street View in 10 regions of 5 Chinese cities, cut off the top and bottom quarter of each panorama and sliced the rest into four 2048×2048 images. About 10,000 images contain signs; these were split about 2:1 into training and test images, and the 90,000 images without signs were added to the test data, so a detector must also avoid false alarms on empty street scenes. Each sign has a box, a class and a pixel mask; the masks of occluded signs are the intersection of the box with a polygon or ellipse boundary.

Mapillary Traffic Sign Dataset. Ertler et al. (2020) sampled street-level images from the Mapillary platform per country, with a target distribution over continents (20% each for North America, Europe and Asia, 15% each for South America and Oceania, 10% for Africa), binned by location to cover rural and urban areas, and kept selected images far apart to avoid correlated samples. Annotators boxed every traffic-related sign whose shorter side exceeded 10 pixels, including direction and information signs, and chose each sign’s template from images harvested from Wikimedia Commons, helped by an embedding network trained with a contrastive loss that sorted the templates by similarity. A second annotator’s check of 5,000 images found that 0.5% of boxes needed correction and a false-negative rate of 0.89%. The fully annotated part has 36,589 training, 5,320 validation and 10,544 test images, with the test annotations withheld; a further 53,377 images received partial, automatically propagated labels, matched to the annotated signs through camera poses recovered by structure from motion. The paper’s counts of boxes differ slightly between sections (over 255,000 in the list of contributions, 257,541 in the statistics).

Evaluation Metrics

GTSRB scores the correct classification rate, the fraction of test images classified correctly under 0/1 loss, that is, accuracy. Every error costs the same although the test set is strongly unbalanced, which the authors justify by every sign being equally important; they also report accuracy per group of similar signs (speed limits, other prohibitory, derestriction, mandatory, danger and unique signs) and confusion matrices (Stallkamp et al., 2012).

GTSDB matches each submitted box $S$ to a ground-truth box $G$ by the Jaccard coefficient, the intersection over union

$$J(S, G) = \frac{|S \cap G|}{|S \cup G|},$$

counting a hit when $J$ reaches a threshold, given in the paper’s example as 0.6. If several boxes overlap one ground-truth box above the threshold, only the best counts and the others are ignored, neither hits nor false positives. A detection of a sign from another category counts as a false positive. Teams are ranked by the area under the precision–recall curve in each category (Houben et al., 2013).

Tsinghua-Tencent 100K reports recall and what Zhu et al. (2016) call accuracy, which the paper does not define and plots against recall where precision usually stands, at a Jaccard coefficient of 0.5, separately for signs of area below $32^2$ pixels, between $32^2$ and $96^2$, and above $96^2$, the size bins of COCO.

The Mapillary dataset defines average precision at IoU 0.5 for the binary task, with the same three size bins, and mean average precision over the 400 classes for the joint task. For the joint task, boxes are matched to ground truth ignoring the class, and only then are other-sign boxes removed (Ertler et al., 2020).

Reported results, each in its own benchmark’s protocol and therefore not comparable across rows:

• GTSDB 2013 competition (area under the precision–recall curve on the evaluation set): the winning team wgy@HIT501 scored 100% on prohibitory, 99.91% on danger and 100% on mandatory signs; the organizers’ Viola–Jones baseline scored 90.81%, 46.26% and 44.87% (Houben et al., 2013). The baselines’ weakness on mandatory and danger signs was attributed to rotation and perspective, since mandatory signs stand at varying heights and angles.
• Tsinghua-Tencent 100K (recall and “accuracy” at Jaccard 0.5, on the test images plus the 90,000 sign-free images): Zhu et al.’s network reached 0.91 recall and 0.88 accuracy overall, against 0.56 and 0.50 for Fast R-CNN with 10,000 proposals per image; for signs up to 32 pixels the figures were 0.87 and 0.82 against 0.24 and 0.45. Their sign-only detector reached 94% recall and 84% accuracy.
• Mapillary dataset (AP at IoU 0.5; the tables do not state the split): the binary FPN-101 detector reached 88.38 AP with inputs up to 4,000 pixels on the long side and 81.65 with inputs up to 2,048, the loss falling mostly on small signs (73.89 and 56.32 AP$_s$). The two-stage system reached 81.8 mAP over the 400 classes. Run on Tsinghua-Tencent 100K and scored with the Mapillary metrics, the two-stage system with an FPN-50 detector reached 89.9 mAP against 81.6 for Zhu et al.’s network, and 95.7 when its detector and classifier were pre-trained on the Mapillary data (Ertler et al., 2020).

Adversarial and Physical Attacks

Signs are an obvious target for adversarial examples in the physical world, because anyone can modify them and they matter for safety. Eykholt et al. (2018) proposed Robust Physical Perturbations (RP$_2$), an attack that computes a perturbation confined by a mask to the surface of the sign, shaped like graffiti or abstract art so that it looks unremarkable, and optimized over many images of the sign at different distances and angles, both photographed and synthetically transformed. A term in the objective penalizes colours a printer cannot reproduce. The attack is white-box: it needs the classifier’s weights.

They evaluated it on two classifiers they trained themselves, LISA-CNN on 17 United States sign classes (91% test accuracy) and GTSRB-CNN, a public implementation of the multi-scale CNN in which German stop signs were replaced by U.S. ones (95.7%), with a protocol of their own: stationary photographs at set distances and angles, and video from a car approaching the sign at 0 to 20 mph, sampling every tenth frame. An image counted as a success only if the clean sign was classified correctly in the same position. Black and white stickers on a real stop sign made LISA-CNN output “Speed Limit 45” in all stationary test conditions and in 84.8% of the drive-by frames; against GTSRB-CNN, stickers produced the targeted “Speed Limit 80” in 80% of stationary conditions and 87.5% of frames. The crops were placed by hand to match the clean images; with pseudo-random crops the targeted success fell to 70% for LISA-CNN and 60% for GTSRB-CNN, while the untargeted success, any wrong label, was 90% and 100%. The authors attacked classifiers on crops, not detectors, and cite earlier work by Lu et al. reporting that detectors could not be attacked in that way, with their own later extension to the YOLO detector (neither read here).

What these results establish is specific: white-box classifiers trained by the authors could be fooled by stickers under the tested viewing conditions. The authors’ own conclusion is that defences should not rely on physical noise, such as distance and angle, to destroy adversarial perturbations.

Applications

• Driver assistance. Traffic sign recognition is a long-standing driver assistance function; Stallkamp et al. (2012) and Houben et al. (2013) note commercial systems in vehicles, usually limited to a subset of signs such as speed limits.
• Automated driving. Signs belong to the scene description that the perception stack of an automated vehicle produces, alongside lane detection, and Ertler et al. (2020) note that accurate perception of signs supports localization and motion planning; Autonomous Driving Perception describes that stack.
• Mapping and inventory. Signs are map features: the Mapillary dataset was built from a street-level imagery platform whose own detections of signs seeded the annotation (Ertler et al., 2020), and Sermanet and LeCun (2011) list sign monitoring for maintenance among the uses.

Open Problems

The sources document limits that remain:

• Benchmark saturation versus real conditions. Classification on GTSRB reached and passed human accuracy in 2011, and Houben et al. (2013) wrote that the classification step might be regarded as solved. Zhu et al. (2016) answered with the objection given under Problem Definition.
• Rare and unknown classes. Hundreds of classes follow a long tail, many countries have no public sign catalogue, and real frames contain signs outside any taxonomy (Ertler et al., 2020). The Mapillary dataset’s partially annotated images raised mean class accuracy by 0.6 points overall and by 1–4 points on the rarest classes, which shows the problem but does not close it.
• Failure on unusual images. More than 15% of the winning committee’s GTSRB errors came from one physical sign, a diamond-shaped right-of-way sign, in the images where overexposure washed out its yellow centre, a sign humans recognize by its shape (Stallkamp et al., 2012). The algorithms were also worse on the largest images, where motion blur and ghosting were common, while humans improved with size. None of the competing methods handled inputs of varying size and aspect ratio except by rescaling to a fixed size (Stallkamp et al., 2012).
• Small signs. In Tsinghua-Tencent 100K and the Mapillary dataset the smallest signs score lowest (Zhu et al., 2016; Ertler et al., 2020), and down-scaling the input costs most on them (Ertler et al., 2020).
• Adversarial robustness. Physical perturbations fooled sign classifiers in the tests of Eykholt et al. (2018); robustness against such attacks, and against the corruptions that occur without an attacker, is not measured by the accuracy that benchmarks report.
• Regional transfer. Signs and their conventions differ by country, and transfer between datasets needs fine-tuning: pre-training on the Mapillary data improved Tsinghua-Tencent 100K results, but the models were still trained on the target dataset (Ertler et al., 2020).