ICDAR Robust Reading Benchmarks

The ICDAR Robust Reading Competition is a series of benchmarks for reading text in images, held with the International Conference on Document Analysis and Recognition (ICDAR). Simon Lucas and colleagues at the University of Essex started it in 2003 with competitions on locating and reading text in camera images of scenes. The reports of the 2013 and 2015 editions have Dimosthenis Karatzas of the Computer Vision Centre of the Universitat Autònoma de Barcelona as first author, and the Centre hosts the competition’s web portal; since 2013 the challenges have stayed open between editions, accepting submissions at any time and evaluating them online. Two of its datasets became the standard tests of scene text methods: Focused Scene Text (Challenge 2, in its definitive 2013 form), with 462 images in which the camera was aimed at the text, and Incidental Scene Text (Challenge 4, 2015), with 1,670 images taken with Google Glass without aiming at the text. Papers call them ICDAR 2013 (IC13) and ICDAR 2015 (IC15). Each comes with official rules for scoring text localization, word recognition and end-to-end reading, and the two localization rules differ: the 2013 one, Wolf and Jolion’s DetEval, credits split and merged detections, while the 2015 one requires an intersection over union above 0.5. Recognition methods are covered in optical character recognition.

Purpose

The 2013 report describes the series as addressing the need “to quantify and track progress” in text extraction (Karatzas et al., 2013). The competitions supply shared data with word-level ground truth, fix the tasks (find the text, read cropped words, do both), and fix the scoring, so that results from different groups can be compared.

The two main datasets were built for different use cases. In the Challenge 2 overview, focused text is the scenario of text reading and translation applications, where the user points the camera at the text of interest; the page adds that it is also the scenario most methods and datasets had addressed until then. Incidental text, the 2015 report explains, appears in images captured without the user taking any prior action to frame it, as with wearable cameras or large urban image collections; the report notes that perspective distortion can be significant.

Editions

Edition	Scene text data	Annotation	Localization protocol
2003 (Lucas et al.)	509 scene images: 258 TrialTrain, 251 TrialTest (official 2003 page)	word rectangles with text	ICDAR 2003 measure
2005 (Lucas)	text locating competition repeated
2011, Challenge 2 (Shahab et al.)	scene images; the 2013 set is a subset	word rectangles
2013, Challenge 2	462 images: 229 train, 233 test	word rectangles, character pixels, “don’t care”	Wolf and Jolion (DetEval)
2015, Challenge 4	1,670 images: 1,000 train, 500 test, 170 sequestered	word quadrilaterals, “do not care”	IoU > 0.5
2017, MLT	18,000 images, 9 languages	word quadrilaterals with script	IoU > 0.5
2019, MLT, ArT, LSVT, ReCTS	10,000 to 450,000 images	quadrilaterals or polygons	IoU-based

2003 and 2005. The first edition (Lucas et al., 2003; entries and results in Lucas et al., 2005) was announced on its official pages as a group of competitions on scene images: text locating, robust reading, robust word recognition on cropped words and robust character recognition. The official datasets page, as archived in October 2002, lists scene image sets of 258 TrialTrain and 251 TrialTest images, cropped word sets of 1,157 and 1,111 words and character sets of 6,185 and 5,430 characters, each with ground truth in XML, and states that separate competition datasets would be kept private until the conference. The 2013 and 2015 reports cite a 2005 text locating competition as the second edition.

2011. Two challenges ran under the Robust Reading name: born-digital images from web pages and email (Challenge 1) and scene images (Challenge 2, Shahab et al., 2011).

2013. The 2013 competition (Karatzas et al., 2013) unified the ground truth format, tasks and metrics of the born-digital and scene challenges and added a video challenge (Challenge 3: 28 videos, 13 for training and 15 for testing). For Challenge 2 the report describes the dataset as almost the same as in 2011: some ground truth texts were revised and a small number of images duplicated across the training and test sets were removed, so the 2013 set is a subset of the 2011 one, with 229 training and 233 test images. New in 2013 were pixel-level ground truth, in which every character was painted in its own color by operators using semi-automatic software and double-checked, and “don’t care” regions for illegible tiny text. To stay consistent with 2011, only text that the 2011 ground truth had not marked was made “don’t care”. The overview page calls the 2013 edition the definitive one for localization, segmentation and word recognition; these tasks stayed open in continuous mode, and the 2015 edition of Challenge 2 added an end-to-end task.

2015. The 2015 competition (Karatzas et al., 2015) ran from January to April 2015, accepted 44 of 76 submissions after removing near-duplicate entries of the same pipeline, and added end-to-end tasks to all four challenges. Its new Challenge 4 uses 1,670 images with 17,548 annotated regions, collected with Google Glass over a few months in Singapore. The report states that the focus was on Latin-script text; text in other scripts, present in the images, was marked “do not care”. The annotation was a collective effort of six institutions, using the competition’s web-based annotation tool: annotators placed a quadrilateral around each word while watching a rectified preview of it, and in a second pass the rectified words were shown in random order, so that readability was judged without context.

Later editions. The portal has since hosted many more challenges; its menu, in an archived copy of March 2026, lists 33. Those on scene text include COCO-Text (2017), the Multi-Lingual Text challenges of 2017 and 2019 (MLT), and in 2019 Arbitrary-Shaped Text (ArT), Large-scale Street View Text (LSVT) and Reading Chinese Text on Signboard (ReCTS). MLT 2017 has 18,000 images in 9 languages, split evenly between training and test and scored with the same IoU rule as Challenge 4 (Nayef et al., 2017); MLT 2019 extends it to 20,000 images in 10 languages and adds an end-to-end task (Nayef et al., 2019). ArT gathers 10,166 images of curved and multi-oriented text annotated with polygons (Chng et al., 2019), and LSVT has 450,000 street view images, 400,000 of them labeled only with the text of interest (Sun et al., 2019).

Annotation Types

Both datasets annotate words, each with its location and its transcription.

• Challenge 2 (focused). One text file per image, one line per word: left, top, right, bottom, "transcription", an axis-aligned rectangle in pixels, with backslash escapes for quotes and backslashes; the text files are ASCII. The 2013 release adds the pixel-level character masks described above, with a text file giving each character’s color, center and bounding box, and “don’t care” rectangles over illegible text. For word recognition, every word is also cut out as its own image, with a margin of 4 pixels around its box to keep some context, and all transcriptions are listed in one file.
• Challenge 4 (incidental). One UTF-8 file per image, one line per word: x1, y1, x2, y2, x3, y3, x4, y4, transcription, the four corners of a quadrilateral in clockwise order. Anything after the eighth comma belongs to the transcription, with no escape characters. A transcription of ### marks a “do not care” region. According to the 2015 report, “do not care” covers text in non-Latin scripts and text the annotator judged unreadable, and one- and two-character words are marked automatically. The word recognition set gives each word of 3 or more characters as a crop of its axis-aligned bounding rectangle, with the word’s quadrilateral in the crop’s coordinates.

Splits

Dataset	Training	Test	Source
Focused Scene Text 2013	229 images, 848 words	233 images; 1,095 cropped words for recognition	Karatzas et al. (2013); Challenge 2 tasks page
Incidental Scene Text 2015	1,000 images, about 4,500 readable words	500 images; about 2,000 cropped words	Karatzas et al. (2015); Challenge 4 tasks page

The 2015 report adds a third set of 170 images kept private as a sequestered set; the 1,500 others are public. The tasks page invited participants to submit an executable along with their results, to be run on a hidden test subset for further analysis. The word counts come from the tasks pages; the Challenge 4 page was written before the test phase (“will be provided”) and gives approximate counts. Neither dataset has an official validation split, so model selection is left to the user; the hazards of tuning on a public test set are discussed in benchmarking.

Tasks

• Text localization (Tasks 2.1 and 4.1). Return a box for each word in the image; Challenge 2 accepts rectangles, Challenge 4 quadrilaterals. The tasks pages say that boxes correspond to “parts of text (words or text lines)”, and the 2013 protocol is designed so that line-level boxes are not penalized.
• Text segmentation (Task 2.2, Challenge 2 only). Label every text pixel; scored at the level of “atoms”, the smallest units that can be given a transcription, with the framework of Clavelli and Karatzas.
• Word recognition (Tasks 2.3 and 4.3). Transcribe each cropped word image.
• End-to-end (Tasks 2.4 and 4.4, from 2015). Localize and read every word, optionally helped by a vocabulary.

Localization is object detection of words, with rectangles or quadrilaterals as bounding boxes.

Evaluation

All localization protocols count matches between ground-truth words and detections, then report recall, precision and their harmonic mean, the F-score (precision and recall). In both datasets these counts are pooled over the whole test set, not averaged per image: an image with many words weighs more. Ranking is by F-score.

Localization in 2013: DetEval

The 2013 competition scores localization with the framework of Wolf and Jolion (2006), whose software is called DetEval. For a ground-truth box $G_i$ and a detection $D_j$, it uses two area ratios instead of one overlap:

$$r_{ij} = \frac{|G_i \cap D_j|}{|G_i|} \;(\text{area recall}), \qquad p_{ij} = \frac{|G_i \cap D_j|}{|D_j|} \;(\text{area precision}).$$

A pair qualifies when $r_{ij} \ge t_r$ and $p_{ij} \ge t_p$. The 2013 report uses the defaults suggested by Wolf and Jolion, $t_r = 0.8$ and $t_p = 0.4$, so a detection must cover 80% of the word, and at least 40% of the detection must lie on the word. Matching then proceeds in three stages:

1. One-to-one. A word and a detection that qualify with each other and with nothing else are matched. DetEval adds a test on the box centers that, as its software page notes, is not described in the paper; in its code the distance between the two centers, divided by the mean of the two box diagonals, must be below 1 by default.
2. One-to-many (splits). A word not yet matched is matched to several detections if each lies mostly on it ($p_{ij} \ge t_p$) and together they cover it ($\sum_j r_{ij} \ge t_r$).
3. Many-to-one (merges). A detection is matched to several words if each word is mostly covered by it ($r_{ij} \ge t_r$) and together the words fill enough of it ($\sum_i p_{ij} \ge t_p$).

Each match adds a weight to a recall sum and a precision sum, and the final scores divide these by the numbers of ground-truth words and of detections:

$$R = \frac{\sum_i \mathrm{match}_G(G_i)}{|G|}, \qquad P = \frac{\sum_j \mathrm{match}_D(D_j)}{|D|}.$$

A one-to-one match adds 1 to both. For splits and merges the 2013 report chose the weights to suit word-level ground truth: methods that cut one word into several boxes should be penalized, but methods that return a whole text line, covering several words with one box, should not. The report therefore gives one-to-many matches the fixed weight 0.8 suggested by Wolf and Jolion and inflicts no penalty on many-to-one matches. In the third-party Python implementations of this protocol read for this article (in the DB and MMOCR repositories, described under Resources; the portal’s own script is available only after registration), a split adds 0.8 to the recall sum and 0.8 for each of its detections to the precision sum, and a merge adds 1 for each word to the recall sum and 1 to the precision sum. Wolf’s DetEval program, read here in a third-party copy of its C++ source, uses 0.8 for merges as well, so scores computed with the original tool and with the competition’s settings can differ.

Challenge 2 handles its “don’t care” regions in a pre-processing step: detections that fall inside a “don’t care” region are removed before matching, so marking illegible text neither helps nor hurts (Karatzas et al., 2013). The report does not give the overlap threshold for “inside”; the two Python implementations drop a detection when more than $t_p = 40\%$ of its area lies in a “don’t care” region.

The portal evaluates every Task 2.1 submission twice, and its FAQ (archived copy of February 2015) explains why. The “ICDAR 2013” protocol is the competition’s own implementation, built into the portal and following the order of stages in Wolf and Jolion’s paper. The “DetEval” protocol reproduces Wolf’s DetEval program, which relies on heuristics not documented in the paper: a pair that looks like a one-to-one match is still treated as a possible split or merge if either box overlaps other boxes, and the decision is taken at the end. The FAQ adds that DetEval looks for merges before splits; the copy of the C++ source read for this article searches for splits first. According to the FAQ the two give very similar numbers and rarely change the ranking. Papers often report both columns, and the gap can be large for detectors that output text lines: FOTS lists 82.15% (“IC13”) and 87.69% (“DetEval”) for CTPN, but 88.23% and 88.30% for its own word-level detector (Liu et al., 2018, Table 4). A May 2018 copy of the results page shows a third evaluation, “IoU”.

Localization in 2015: IoU above 0.5

Challenge 4 replaced DetEval with a rule from object detection: a detection matches a word if their intersection over union exceeds 0.5, and each word and each detection can be matched once. The tasks page presents this as similar to standard practice in object recognition and the PASCAL VOC challenge. The 2015 report adds that this framework penalizes granularity differences between the ground truth and the detections. As the standalone evaluation script computes it (read in two third-party copies whose matching code is identical, described under Resources; neither could be compared with the portal’s copy, which requires registration):

• Polygons are built from the integer parts of the submitted coordinates, and areas and intersections are exact polygon operations.
• “Do not care” first. A detection is discarded if more than half of its own area lies inside a “do not care” region (###), $|D \cap G_{\text{dnc}}| / |D| > 0.5$. Discarded detections and “do not care” words are left out of both counts.
• Greedy matching. The script visits the ground-truth words in file order, and for each one the detections in file order, and matches the first unmatched pair with IoU strictly above 0.5. It does not sort by confidence or search for the best assignment. In our reading this rarely matters: a detection with IoU above 0.5 against two words would need more than half of its area inside each, so when the ground-truth words do not overlap, each detection qualifies for at most one word, and the greedy pass finds as many matches as an optimal assignment. Which of two duplicate detections of the same word is counted as the match depends only on file order.
• Pooling. Recall is the number of matches over the number of cared-for words in all images, precision the number of matches over the number of kept detections, and the F-score their harmonic mean. An image without cared-for words scores recall 1, and precision 1 only if it has no kept detections, but these per-image values do not enter the pooled totals.
• Optional AP. If submissions carry confidences, the script can also compute a ranked average precision over the detections; by default it does not, and the competition ranked methods by F-score.

Splits and line-level boxes get no credit under this rule: the pieces and the line box usually stay at or below IoU 0.5, so the words are missed and the boxes count as false positives. A detector for this benchmark has to output words.

Example: the two localization rules on a rendered scene

The code below renders a synthetic scene of word quadrilaterals as binary images and scores one set of detections under both protocols. The ground truth has five words, one of them rotated by 15° and one by −8°, and a small “do not care” region; the detections include a tight box, a word split into two halves, one box over the two-word line BUS STOP, a loose box around EXIT, a box on the “do not care” region and a false alarm. Areas are pixel counts of the rendered masks. DetEval is run in two variants that mirror the portal’s two protocols: “rrc” follows the third-party Python implementations of the competition’s protocol (one-to-one first, merges unpenalized as in the 2013 report), and “wolf” follows the logic of Wolf’s C++ program, with the same “do not care” step added.

import numpy as np
from skimage.draw import polygon as fill_polygon

H, W = 240, 480                                     # synthetic "scene" in pixels

def quad(cx, cy, w, h, deg=0.0):
    """Corners of a w x h word box centred at (cx, cy), rotated by deg, clockwise from top-left."""
    t = np.deg2rad(deg)
    R = np.array([[np.cos(t), -np.sin(t)], [np.sin(t), np.cos(t)]])
    c = np.array([[-w, -h], [w, -h], [w, h], [-w, h]]) / 2
    return c @ R.T + [cx, cy]

def mask(q):
    """Render a quadrilateral as a binary image (pixel centres inside the polygon)."""
    m = np.zeros((H, W), bool)
    rr, cc = fill_polygon(q[:, 1], q[:, 0], (H, W))
    m[rr, cc] = True
    return m

# Ground truth: (corners, transcription); "###" marks a do-not-care region
GT = [(quad(90, 50, 120, 30, -8), "OPEN"),
      (quad(330, 50, 200, 34, 0), "PHARMACY"),
      (quad(130, 150, 80, 26, 0), "BUS"),      # BUS and STOP share one text line
      (quad(220, 150, 80, 26, 0), "STOP"),
      (quad(390, 160, 110, 28, 15), "EXIT"),
      (quad(250, 215, 40, 10, 0), "###")]       # tiny illegible text
# Detections: corners only (the localization task carries no text)
DET = [quad(92, 51, 124, 32, -8),               # OPEN: tight
       quad(281, 50, 96, 34, 0),                # PHARMACY split into two halves
       quad(379, 50, 96, 34, 0),
       quad(175, 150, 172, 28, 0),              # one box over the line BUS STOP
       quad(390, 160, 150, 45, 15),             # EXIT: loose box
       quad(250, 215, 44, 12, 0),               # on the do-not-care region
       quad(40, 200, 50, 20, 0)]                # false alarm

G = np.array([mask(q) for q, _ in GT]); D = np.array([mask(q) for q in DET])
care = np.array([t != "###" for _, t in GT])
inter = np.einsum("ghw,dhw->gd", G.astype(float), D.astype(float))   # pixel counts
aG, aD = G.sum((1, 2)).astype(float), D.sum((1, 2)).astype(float)
iou = inter / (aG[:, None] + aD[None, :] - inter)
area_recall = inter / aG[:, None]          # fraction of the GT box covered
area_precision = inter / aD[None, :]       # fraction of the detection covered

def icdar2015(iou_thr=0.5, dnc_thr=0.5):
    """Challenge 4 localization: drop detections inside do-not-care regions, then greedy one-to-one IoU > 0.5."""
    det_dnc = (area_precision[~care] > dnc_thr).any(0)
    g_used, d_used, pairs = np.zeros(len(GT), bool), np.zeros(len(DET), bool), []
    for g in range(len(GT)):                       # same loop order as the RRC script
        for d in range(len(DET)):
            if care[g] and not det_dnc[d] and not g_used[g] and not d_used[d] and iou[g, d] > iou_thr:
                g_used[g] = d_used[d] = True; pairs.append((g, d))
    n_g, n_d = care.sum(), (~det_dnc).sum()
    return len(pairs) / n_g, len(pairs) / n_d, pairs, det_dnc

def center_ok(g, d):
    """DetEval's extra one-to-one test: centre distance below the mean of the two box diagonals."""
    diag = lambda q: np.hypot(*(q.max(0) - q.min(0)))
    return 2 * np.hypot(*(GT[g][0].mean(0) - DET[d].mean(0))) / (diag(GT[g][0]) + diag(DET[d])) < 1

def deteval(variant, tr=0.8, tp=0.4, w_split=0.8):
    """DetEval: one-to-one, then splits (one GT, many detections), then merges (many GTs, one detection).
    variant "rrc":  as third-party Python versions of the RRC Challenge 2 script (merges unpenalized, a split
                    pass may match a GT to a single detection);
    variant "wolf": as Wolf's DetEval C++ (one-to-one only for boxes that overlap nothing else,
                    split/merge only for boxes with several overlaps, merges also weighted 0.8)."""
    w_merge = 1.0 if variant == "rrc" else 0.8
    det_dnc = (area_precision[~care] > tp).any(0)      # those versions drop these with tp
    ok = (area_recall >= tr) & (area_precision >= tp)
    touch = inter > 0
    g_used, d_used = ~care.copy(), det_dnc.copy()
    R = P = 0.0; log = []
    for g in range(len(GT)):                            # one-to-one
        for d in range(len(DET)):
            if g_used[g] or d_used[d] or not ok[g, d] or not center_ok(g, d):
                continue
            single = (ok[g].sum() == 1 and ok[:, d].sum() == 1) if variant == "rrc" else \
                     (touch[g, ~det_dnc].sum() == 1 and touch[care, d].sum() == 1)
            if single:
                g_used[g] = d_used[d] = True; R += 1; P += 1; log.append(f"one-to-one {names[g]}-D{d}")
    for g in range(len(GT)):                            # one GT, many detections (split)
        if g_used[g] or (variant == "wolf" and touch[g, ~det_dnc].sum() <= 1): continue
        ds = [d for d in range(len(DET)) if not d_used[d] and area_precision[g, d] >= tp]
        if len(ds) == 1 and variant == "wolf":          # degrades to one-to-one
            if ok[g, ds[0]]: g_used[g] = d_used[ds[0]] = True; R += 1; P += 1; log.append(f"one-to-one {names[g]}-D{ds[0]}")
        elif ds and round(area_recall[g, ds].sum(), 4) >= tr:
            g_used[g] = True; d_used[ds] = True
            R += 1.0 if len(ds) == 1 else w_split; P += 1.0 if len(ds) == 1 else w_split * len(ds)
            log.append(("one-to-one" if len(ds) == 1 else "split") + f" {names[g]}-D{ds}")
    for d in range(len(DET)):                           # many GTs, one detection (merge)
        if d_used[d] or (variant == "wolf" and touch[care, d].sum() <= 1): continue
        gs = [g for g in range(len(GT)) if not g_used[g] and area_recall[g, d] >= tr]
        if len(gs) == 1 and variant == "wolf":
            if ok[gs[0], d]: g_used[gs[0]] = d_used[d] = True; R += 1; P += 1; log.append(f"one-to-one {names[gs[0]]}-D{d}")
        elif gs and round(area_precision[gs, d].sum(), 4) >= tp:
            d_used[d] = True; g_used[gs] = True
            R += 1.0 if len(gs) == 1 else w_merge * len(gs); P += 1.0 if len(gs) == 1 else w_merge
            log.append(f"merge {[names[g] for g in gs]}-D{d}")
    return R / care.sum(), P / (~det_dnc).sum(), log

names = [t for _, t in GT]
f1 = lambda r, p: 0 if r + p == 0 else 2 * r * p / (r + p)
print("IoU and area recall/precision of the best detection for each GT word")
for g in range(len(GT)):
    d = int(np.argmax(iou[g]))
    print(f"  {names[g]:9s} D{d}  IoU {iou[g, d]:.2f}  area recall {area_recall[g, d]:.2f}  area precision {area_precision[g, d]:.2f}")
r, p, pairs, dnc = icdar2015()
print(f"\nICDAR 2015 (IoU > 0.5, one-to-one): matched {[(names[g], d) for g, d in pairs]}")
print(f"  detections dropped as do-not-care: {[int(d) for d in np.flatnonzero(dnc)]}")
print(f"  recall {r:.3f}  precision {p:.3f}  F {f1(r, p):.3f}")
for variant in ["rrc", "wolf"]:
    r, p, log = deteval(variant)
    print(f"\nDetEval, {variant} (tr 0.8, tp 0.4): " + "; ".join(log))
    print(f"  recall {r:.3f}  precision {p:.3f}  F {f1(r, p):.3f}")

Output:

IoU and area recall/precision of the best detection for each GT word
  OPEN      D0  IoU 0.89  area recall 0.99  area precision 0.90
  PHARMACY  D1  IoU 0.48  area recall 0.48  area precision 1.00
  BUS       D3  IoU 0.44  area recall 1.00  area precision 0.44
  STOP      D3  IoU 0.44  area recall 1.00  area precision 0.44
  EXIT      D4  IoU 0.46  area recall 1.00  area precision 0.46
  ###       D5  IoU 0.77  area recall 1.00  area precision 0.77

ICDAR 2015 (IoU > 0.5, one-to-one): matched [('OPEN', 0)]
  detections dropped as do-not-care: [5]
  recall 0.200  precision 0.167  F 0.182

DetEval, rrc (tr 0.8, tp 0.4): one-to-one OPEN-D0; one-to-one EXIT-D4; split PHARMACY-D[1, 2]; one-to-one BUS-D[3]
  recall 0.760  precision 0.767  F 0.763

DetEval, wolf (tr 0.8, tp 0.4): one-to-one OPEN-D0; one-to-one EXIT-D4; split PHARMACY-D[1, 2]; merge ['BUS', 'STOP']-D3
  recall 0.880  precision 0.733  F 0.800

The 2015 rule finds only OPEN. Every other word fails in a different way: the PHARMACY halves each cover less than half of the word, the BUS STOP box is too long for either word, and the loose EXIT box covers the word completely but has more than twice its area, so IoU stays below 0.5. With six kept detections and five cared-for words, recall is 1 in 5 and precision 1 in 6. The box on the “do not care” region is removed before matching and costs nothing.

Under DetEval’s area thresholds every one of these detections qualifies except the false alarm, because a detection need only cover 80% of a word and lie 40% on it. The split counts 0.8 for recall and 0.8 for each half on the precision side. The two DetEval variants differ only on the line box. In the Python implementations, the one-to-many stage looks at BUS first, finds one detection that lies 44% on it and covers it fully, and matches the two as if one-to-one; STOP is then left without a detection and counts as missed. Wolf’s program sends BUS to the many-to-one stage instead, because BUS overlaps a detection that also overlaps another word, and there the box is matched to both words, each weighted 0.8. On this scene the protocol, not the detector, moves the F-score between 0.18 and 0.80, and the two DetEval implementations differ by about 0.04. In our reading of the code, their behaviour depends on the order of the words in the file: had STOP come first, it would have been the word matched. These numbers come from one hand-built scene and only show which configurations each rule rewards.

Both Python reimplementations were cross-checked by running third-party code on the same scene, with a small stand-in for the polygon libraries that computes exact areas of convex polygons. The Challenge 4 localization and end-to-end scripts (corners rounded to integers) matched the same single pair and returned the same recall, precision and F-score; the DB repository’s DetEval implementation, run on the unrounded corners after defining a name that one of its log lines uses but never defines, made the same four matches as the “rrc” variant and returned the same scores. The “wolf” variant follows Wolf’s C++ source as read line by line; it was not compiled or run.

Word recognition

Word recognition is scored by edit distance (Levenshtein distance) with equal costs for insertion, deletion and substitution, the measure defined in optical character recognition. For 2013 the report specifies the details: each word’s edit distance is normalized by the length of its ground-truth transcription, the normalized distances are summed over all test words to give the total edit distance used for ranking, the comparison is case sensitive, and the percentage of correctly recognized words is reported alongside (Karatzas et al., 2013). With 1,095 test words, a total of 122.7, the 2013 winner’s score, is an average normalized distance of about 0.11 per word. The 2015 report defines Task 4.3 the same way, including the case-sensitive comparison (Karatzas et al., 2015).

Because each distance is normalized, a one-letter error costs 1/3 in a three-letter word but 0.1 in a ten-letter word, and an empty answer costs 1 whatever the word’s length. Short words therefore weigh heavily in the total, and the Challenge 2 tasks page notes that the focused set has many short words and even single letters. Recognition papers usually report word accuracy under their own conventions instead, so their numbers are not the competition’s (optical character recognition compares the two).

End-to-end recognition and word spotting

The end-to-end tasks, added in 2015 to both Challenge 2 and Challenge 4, use a protocol that the tasks pages attribute to Wang, Babenko and Belongie (2011): a detection counts as correct if it overlaps a ground-truth word by more than 50% IoU, as in localization, and its transcription matches the word, ignoring case. Detections that lie more than 50% inside a “do not care” region are discarded first, and “do not care” words, including all words of one or two characters, neither count when found nor when missed. Wang et al. in turn describe their rule as following the guidelines of the ICDAR 2003 competition (Lucas et al., 2003), which they call essentially those of object recognition competitions such as PASCAL VOC.

Each test image comes with three vocabularies, the per-image one again following the setup of Wang et al.:

• Strongly contextualized: 100 words for the image, all words of the image plus distractors drawn at random from the test set’s words. Submitting with this list is obligatory.
• Weakly contextualized: all words of the whole test set.
• Generic: any vocabulary; a list of about 90,000 words is provided.

All three lists contain only words of three or more letters, after stripping leading and trailing punctuation, and drop any word that still contains symbols other than hyphens. Prices, URLs, times, dates and e-mail addresses are therefore absent from the lists, although they are annotated and an end-to-end system should still read them. Participants who use other vocabularies or text corpora in training are asked to say so.

A third-party copy of the Challenge 4 end-to-end script (described under Resources, not compared with the portal’s) shows how “the words match” is computed:

• Both strings are upper-cased before comparison.
• The ground-truth word may differ from the detection by one character from the set !?.:,*"()·[]/' at its start, at its end, or both; such characters inside a word must be reproduced. The 2015 report states the rule more loosely, saying that punctuation marks at the beginning or end of the word are not taken into account.
• Geometry decides first. The first detection in file order with IoU above 0.5 claims the word even when its transcription is wrong; it then counts as a false positive, the word as missed, and a later detection with the right text cannot take its place.

The results page of Task 4.4 (archived copy of September 2021) offers a second evaluation of each submission, word spotting, which in the script sets aside words that do not suit a dictionary. A ground-truth word loses a final ’s and leading or trailing hyphens, and its punctuation marks are turned into spaces; if what remains contains a space (punctuation inside the word), is shorter than three characters, or contains characters outside the Latin and Greek letter ranges and the hyphen, the word becomes “do not care”. The remaining words, stripped in the same way, must match the detection exactly, ignoring case. Papers report both variants for each vocabulary, for example as end-to-end and word spotting F-scores with strong, weak and generic lexicons (FOTS, Table 2).

The evaluation server and the scripts

Results are scored by the portal: participants upload them as a zip archive of text files, one per image, and the server evaluates them online and offers visualizations of the results. Both competitions ran in open mode, the reports explain: the test images were public, participants submitted results rather than executables, and the organizers relied on their scientific integrity to follow the rules (Karatzas et al., 2013; 2015). Between editions the tasks stayed open, which the 2015 report credits for the framework’s acceptance as a de facto standard. The procedure page of the 2013 competition, as kept on the portal, released the test images without ground truth some weeks after the training set, and asked each participant for a short description of the method for the report. For each task the portal also offers a “standalone” Python version of the evaluation script, which scores a submission against a ground-truth zip file offline; the instructions kept with one third-party copy (YanShuang17, under Resources) say that in most cases the ground truth is included in the script’s archive as gt.zip. Since the scripts score test submissions, this means in our reading that the test labels are available to registered users, and a score computed offline need not have passed through the server. Downloads require registration, which the downloads page describes as a one-step expression of interest that does not imply taking part in any task. The copies of the scripts read for this article were written for Python 2.7, use the Polygon library for areas, and accept a JSON string that overrides default parameters such as the IoU threshold.

License

As of October 2026, no license for the images or the annotations appears on the public pages of the Robust Reading Competition read for this article: archived copies, from 2025 and 2026, of the Challenge 2 overview page, the tasks pages of Challenges 2 and 4 and the Challenge 4 downloads page, and the 2015 copies of the FAQ and procedure pages. The data can be downloaded only after registering, and any terms the portal shows to registered users could not be read, since no account was created for this article, so the terms of use are unclear. The 2015 report calls the Incidental Scene Text dataset one of the largest “public domain” fully ground-truthed datasets, but names no license; in our reading the phrase refers to public availability, not to a public-domain dedication. Neither report says who holds the copyright of the images or under what terms they may be reused, and the 2003 datasets page, in its 2002 archived copy, states no license either. The copies of the standalone evaluation scripts read for this article carry no license notice. The 2013 procedure page says only that participants may use the training set freely to train and validate their algorithms.

Limitations and Bias

• Small test sets. The 2013 test set has 233 images and the 2015 test set 500. A difference of a fraction of a point in F-score corresponds to a handful of words, and methods are rarely compared with significance tests (statistical significance in evaluation).
• One name, several protocols. “ICDAR 2013” localization scores have been computed with the competition’s weights, with the portal’s other evaluations and with third-party code, which do not always agree (see Evaluation and the example). Scores are comparable only when protocol and implementation are the same.
• IoU at 0.5 for thin objects. In our reading of the definition, a word box shifted vertically by a third of its height falls to IoU 0.5 and is missed, while the same shift in pixels along a long word costs little; for small, blurred incidental words the annotation itself is uncertain at that scale.
• Word-level conventions. What counts as one word, and whether punctuation belongs to it, is decided by the annotators; the special-character rules in the end-to-end script exist for this reason.
• Language and script. The 2013 dataset’s text is, in the words of its overview page, reasonably well focused and in most cases horizontal, and the Total-Text authors describe it as English (Ch’ng and Chan, 2017); the 2015 dataset scores only Latin-script text. Multi-lingual and Chinese text were addressed only in later challenges.
• Vocabularies leak information. The strong vocabulary contains every word of the image, and in 2015 the only two localization entries with precision above 70% had used the vocabularies to filter their detections (Karatzas et al., 2015). In our reading, end-to-end results with the strong vocabulary therefore measure a constrained matching problem, and only the generic setting comes close to open reading.
• Saturation and test-set reuse. Localization F-scores on the 2015 test set rose from about 0.50 in the competition to 0.87–0.88 in 2018–2020 (Historical Milestones), all on the same public test images; the general risks of such reuse are covered in benchmark overfitting. Benchmarks of curved and multi-lingual text such as Total-Text, ArT and MLT became the harder tests.
• People in the images. The 2015 images are street scenes from a head-worn camera. Neither competition report nor the challenge pages read for this article say how bystanders’ faces, vehicle licence plates or consent were handled, or under what terms the images were captured.

Common Uses

• Benchmarks for scene text detection: IC13 for horizontal, focused text and IC15 for multi-oriented, incidental text, reported by many detection papers, among them EAST, FOTS and DB, often together with MSRA-TD500, Total-Text or MLT.
• Fine-tuning data. The 229 and 1,000 training images are too few to train modern detectors alone; they are mostly used to fine-tune models pre-trained on other data, as FOTS did with 9,000 MLT 2017 images followed by the IC15 and IC13 training images.
• Recognition benchmarks: the cropped words of IC13 and IC15 are standard test sets for word recognition (optical character recognition).
• End-to-end text spotting, with the strong, weak and generic vocabularies.

Historical Milestones

Results as reported by the cited authors, with the protocol named.

• 2011 to 2013, focused text. In 2013 the winning localization method, USTB TexStar, reached recall 66.45%, precision 88.47% and F-score 75.89% under the competition’s implementation of Wolf and Jolion’s framework, and the report notes that only one method had passed 70% in 2011. In word recognition, Google’s PhotoOCR recognized 82.83% of the test words correctly, with a total edit distance of 122.7, which the report calls double the 2011 winner’s 41% (Karatzas et al., 2013, Tables V and VIII). An off-the-shelf commercial OCR package, ABBYY OCR SDK 10, served as the baseline, at 44.21% localization F-score and 45.30% correct words.
• 2015, incidental text. The new benchmark was much harder. The best localization F-score was 49.84% (Stradvision-2, precision 77.46%, recall 36.74%); the report notes that the two methods with precision above 70% had used the task’s vocabularies to filter their detections, which the rules had not forbidden. The best word recognition entry, MAPS, recognized 32.93% of the words, with a total edit distance of 1,128.0, and end-to-end recognition with the strong vocabulary reached an F-score of 43.7% (Stradvision-2). On focused text, the end-to-end task introduced the same year was won by VGGMaxBBNet with 86.18% using the strong vocabulary (Karatzas et al., 2015, Tables III, IV, V and IX). The report observes that almost all submitted methods started from MSER regions and that the top recognizers used commercial OCR engines.
• 2017, deep detectors. EAST, a fully convolutional network predicting rotated boxes or quadrilaterals per pixel, reported an F-score of 0.7820 on the Challenge 4 localization test at a single scale and 0.8072 with multi-scale testing (Zhou et al., 2017, Table 3), against 0.4984 for the 2015 winner in the same table.
• 2018, joint detection and recognition. FOTS shared convolutional features between detection and recognition and reported, on Challenge 4, a localization F-score of 87.99% and end-to-end F-scores of 81.09%, 75.90% and 60.80% with the strong, weak and generic vocabularies at a single scale (Liu et al., 2018, Table 2), after pre-training on the 9,000 MLT 2017 training and validation images.
• 2020, real-time detection. DB, a segmentation network with a differentiable binarization step, reported an F-score of 87.3 on the Challenge 4 test set at 12 frames per second with a ResNet-50 backbone and input height 1,152 (Liao et al., 2020; Table 5 of the arXiv version, 1911.08947v2).

Related Datasets

• Street View Text (SVT). 350 Google Street View images from 20 cities with 725 labeled words (100 training and 250 test images), in which only the words of nearby businesses are boxed, and each image comes with a lexicon of about 50 words built from local business names (Wang and Belongie, 2010). Wang, Babenko and Belongie (2011) evaluated end-to-end reading on SVT and on the ICDAR 2003 data with the match rule and per-image lexicons that the ICDAR end-to-end tasks adopted.
• Total-Text. 1,555 images (1,255 training, 300 test) in which text is curved as well as horizontal and multi-oriented, annotated with word polygons and scored with a version of DetEval modified for polygons (Ch’ng and Chan, 2017; read in its extended arXiv version).
• MSRA-TD500: 300 training and 200 test images of multi-oriented text in Chinese and English, annotated at the level of text lines, often reported alongside IC15 (Ch’ng and Chan, 2017; Zhou et al., 2017).
• COCO-Text: text annotations on COCO images, released in 2016 and run as a challenge on the portal in 2017.
• MLT, ArT, LSVT and ReCTS: the larger multi-lingual, curved and Chinese-text challenges of the portal, described under Editions.

Resources

• Robust Reading Competition portal: tasks, downloads after registration, submission, results and the standalone scripts for every challenge. In October 2026 the live site’s certificate could not be verified from the client used for this article, and its pages were read in Internet Archive copies.
• Focused Scene Text tasks and Incidental Scene Text tasks: the official task definitions, ground-truth formats and evaluation rules (archived copies).
• Open copies of the reports: the 2013 report in the Computer Vision Centre’s repository and the 2015 report on the competition site (archived copies). The portal’s Challenge 2 overview and FAQ give the 2013 report’s pages as 1115–1124; the IEEE proceedings metadata give 1484–1493.
• DetEval software page (archived): Wolf’s documentation of the tool and its parameters; the page releases it under the GNU General Public License, version 2 or later.
• Public copies of the standalone scripts, read for this article: the Challenge 4 localization script in the DB repository (MhLiao/DB, commit 65ca77a), identical in its matching code to a second third-party copy (YanShuang17/icdar_2015_evaluation_script, commit 5bde598); the Challenge 4 end-to-end script in the Mask TextSpotter repository (MhLiao/MaskTextSpotter, commit 7109132), which adds a result-preparation step of its own; and Python implementations of the Challenge 2 protocol in the DB repository (concern/icdar2015_eval/detection/icdar2013.py, same commit; a log line in its one-to-many stage uses an undefined name, so it fails at the first match of that stage unless the name is defined) and in MMOCR v0.6.3 (hmean_ic13.py). None is hosted by the competition, none could be compared with the portal’s scripts, and each may differ from them. A copy of Wolf’s DetEval C++ source, modified for text input, is in Lisaflyz/Deteval.