Scene Text Recognition

Scene text recognition (STR) is the task of reading the characters in a cropped image of text from a photograph: given the image region around one word, typically supplied by a scene text detection stage, output the string it shows. Earlier sources call it cropped word recognition or word recognition (the name of the ICDAR tasks). It is the recognition half of reading text in the wild, a branch of optical character recognition whose inputs are signs, shop fronts, packaging and posters rather than scanned pages, so the text can be curved, seen at an angle, blurred, partly occluded or written in decorative fonts. Modern recognizers read the word as a sequence without segmenting it into characters, decode it with connectionist temporal classification (CTC) or an attention decoder, often rectify distorted text first with a learned spatial transformation, and increasingly use a language model to restore characters the image does not show clearly. Almost all of them have been trained on synthetic word images and compared on a handful of small real benchmarks whose versions and conventions differ from paper to paper, a problem that Baek et al. (2019) made the subject of a unified re-evaluation.

Problem Definition

The input is an image $X$ that contains one word, or a short run of characters, and the output is a string $Y = y_1 y_2 \dots y_L$ over a character set $\mathcal{C}$, of a length $L$ that is not known in advance. Most methods are trained by maximizing the likelihood of the ground-truth transcription, minimizing

$$\mathcal{O} = -\sum_{(X_i, Y_i)} \log p(Y_i \mid X_i),$$

over a training set of word images and their strings, which is the objective Baek et al. write for all the models they compare. What differs between methods is how $p(Y \mid X)$ is factorized and computed.

Two settings are distinguished by the vocabulary. In lexicon-based (constrained) recognition each test image comes with a list of candidate words, such as the 50-word lists of the older benchmarks, and the recognizer picks the most probable one; in lexicon-free (unconstrained) recognition any string over $\mathcal{C}$ may be output. Lexicon-free reading is the standard setting of recent papers. The character set is part of the task definition: the benchmarks were long scored on the 36 case-insensitive alphanumeric characters, and recent work also reports mixed case and punctuation (see Evaluation Metrics).

Benchmarks are grouped by geometry. Following RARE (Shi et al., 2016), papers call text regular when the characters lie horizontally and frontally with even spacing, and irregular when it is curved, rotated, seen in perspective or loosely cropped; Baek et al. classify IIIT5K, SVT, IC03 and IC13 as regular and IC15, SVT-Perspective and CUTE80 as irregular.

Input and Output

• Input: a crop of a scene image around one word, axis-aligned or cut from a rotated box or polygon. Recognizers usually resize it to a fixed size: 100 × 32 pixels in CRNN, RARE and the framework of Baek et al., 128 × 32 in ABINet and PARSeq, 256 × 64 in the ASTER configuration. Grayscale input was common in early models; PARSeq uses RGB.
• Output: a string, usually with a confidence score per character or per word. CTC models output one distribution per image column and collapse it to a string; attention and transformer decoders output one character per step until an end-of-sequence token.
• Optional side input: a lexicon of candidate words.
• Optional side output: the rectified image and the predicted control points of a rectification module, which ASTER’s code returns alongside the text.

Variants

• Lexicon-based and lexicon-free recognition, as above. CRNN’s decoders for both are described in the CRNN article.
• Case-sensitive recognition and recognition with punctuation, which re-annotated benchmarks support (PARSeq uses the case-sensitive annotations of Long and Yao for IIIT5K, CUTE80, SVT and SVT-Perspective).
• Text lines and other scripts. The same kinds of model read longer lines and non-Latin text; SRN reports results on a non-Latin long-text set, and the ICDAR multilingual challenges are described in ICDAR.
• Vertical and arbitrarily rotated text, which the classic benchmarks hardly contain; PARSeq evaluates on rotated copies of the benchmarks and on Uber-Text, which includes vertical text.
• End-to-end text spotting, which detects and recognizes text in a full image with one model (outlined under Approaches).
• Handwriting, which shares the sequence models but has its own data and benchmarks (handwriting recognition).

Challenges

• Geometry. Curved text, perspective from side views and rotation break the assumption, built into column-wise sequence models, that characters run from left to right along the image rows.
• Appearance. Decorative and calligraphic fonts, cluttered backgrounds, uneven lighting, low resolution and blur. Among the 8,539 benchmark images that Baek et al. used, 644 (7.5%) were misread by every one of the 24 models they trained; they grouped these failures into calligraphic fonts, vertical text, special characters, heavy occlusion, low resolution and label noise.
• Occlusion and missing characters, which vision alone cannot resolve and which motivated language models.
• Labels. Real labeled data was scarce for years, so training relied on synthetic images. Benchmark labels are noisy: Baek et al. found mislabeled examples in all the benchmarks, 1.3% when special characters are ignored, 6.1% when they are taken into account and 24.1% when case is also taken into account.
• Evaluation. Small test sets in several versions, different training data and different character sets make published numbers hard to compare (see Datasets and Benchmarks).

Approaches

Before sequence models

RARE’s review divides early methods into bottom-up methods, which detect individual characters and assemble them into words, and top-down methods, which recognize the whole word image, among them Jaderberg et al.’s convolutional network with one class for each of 90,000 English words; the CRNN article describes both families. The sequence formulation won out because it needs neither character positions nor a closed vocabulary.

A four-stage framework

Baek et al. (2019) observed that most recognizers published from 2015 to 2018 fit one pipeline of four consecutive stages, and named each by its module options:

1. Transformation normalizes the input, either not at all (None) or with a thin-plate-spline (TPS) spatial transformer.
2. Feature extraction maps the image to a feature map with a convolutional network: a VGG-style network, a recurrent convolutional network (RCNN) or a ResNet. Each column of the final map has its own receptive field along the horizontal line of the image and becomes one frame of a sequence.
3. Sequence modeling adds context across frames with a bidirectional LSTM, or is omitted (None).
4. Prediction turns the frames into a string with CTC or with an attention decoder (Attn).

The previously published models become combinations: CRNN is None-VGG-BiLSTM-CTC, RARE is TPS-VGG-BiLSTM-Attn, R2AM is None-RCNN-None-Attn, GRCNN is None-RCNN-BiLSTM-CTC, Rosetta is None-ResNet-None-CTC and STAR-Net is TPS-ResNet-BiLSTM-CTC. Baek et al. trained all $2 \times 3 \times 2 \times 2 = 24$ combinations under one protocol; their results are summarized under Datasets and Benchmarks. They released their code as clovaai/deep-text-recognition-benchmark, which Baek, Matsui and Aizawa (2021) build on, and later papers, PARSeq among them, call the TPS-ResNet-BiLSTM-Attn model TRBA.

CTC decoders

A CTC recognizer predicts a distribution over the characters and a blank symbol for every frame and sums over all frame-level paths that collapse to the target string. CRNN (Shi, Bai and Yao) established the design for scene text: convolutional features, a deep bidirectional LSTM and a CTC output, trained only on synthetic word images; its architecture, decoding and results are in the CRNN article, and the loss, the forward–backward algorithm and the decoders in connectionist temporal classification. CTC prediction is fast and needs no autoregressive loop, but it labels each frame independently given the image, it requires characters to follow the frames monotonically, and the number of frames bounds the length of the output. In Baek et al.’s module study CTC prediction was far faster than attention but less accurate on average; the CTC article gives their numbers.

Attention decoders

Attention decoders generate the string one character at a time, the encoder-decoder pattern of machine translation and image captioning. The decoder of Baek et al.’s framework, the one-layer LSTM used in FAN, AON and Edit Probability, computes at step $t$ attention weights over the $I$ encoder frames $h_1, \dots, h_I$,

$$e_{ti} = v^\top \tanh(W s_{t-1} + V h_i + b), \qquad \alpha_{ti} = \frac{\exp(e_{ti})}{\sum_{k=1}^{I} \exp(e_{tk})},$$

a context vector $c_t = \sum_i \alpha_{ti} h_i$, a new state $s_t = \mathrm{LSTM}(y_{t-1}, c_t, s_{t-1})$ and the output $y_t = \mathrm{softmax}(W_o s_t + b_o)$, stopping when the end-of-sequence token is emitted. The attention mechanism is the additive attention of Bahdanau et al.. Because $y_t$ depends on $y_{t-1}$, the decoder learns dependencies between characters, an implicit character-level language model; Baek et al. credit it with recovering occluded or missing characters, such as the “S” in “HOUSE” in their qualitative examples.

R2AM (Lee and Osindero, 2016), published at the same conference as RARE, reads text lexicon-free by combining a recursive convolutional feature extractor with a recurrent attention decoder that, in their words, learns a character-level language model implicitly and so avoids N-grams. RARE (Shi et al., 2016) decoded its rectified images with a GRU attention decoder over CRNN-style features, with 37 outputs (26 letters, 10 digits and the end token), and searched large lexicons with a prefix tree. Attention needs no monotonic alignment, which suits rectified or irregular text, but it can lose its place: Cheng et al. (2017) named the misalignment between attention regions and the characters on complicated or low-quality images “attention drift” and added a focusing network (FAN) that checks where the decoder attends.

Rectification

Rectification undoes the geometric distortion of irregular text before recognition, so that the feature extractor sees horizontal, frontal characters.

Spatial transformer networks. Jaderberg et al. (2015) introduced the spatial transformer, a differentiable module made of a localization network that predicts the parameters of a transformation from the input, a grid generator that maps every output pixel to a sampling position in the input, and a sampler that interpolates the input there, typically bilinearly (the backward warping of image warping). Since the sampler is differentiable, the whole module is trained by backpropagation from the downstream loss without any annotation of the transformation. Their experiments used affine, projective and 16-point thin-plate-spline transformations on distorted MNIST digits, where the thin-plate spline, which they call the most powerful, gave the lowest or equal-lowest error on every distortion, and affine transformers on Street View House Numbers, where a model with four transformers in its convolutional stack reduced the sequence error on 64 × 64 crops from 4.0% to 3.6%.

RARE: thin-plate splines from fiducial points. Shi et al. (2016) made the transformation a thin-plate spline (TPS) defined by $K$ fiducial points. A convolutional localization network regresses the coordinates $C = [c_1, \dots, c_K] \in \mathbb{R}^{2 \times K}$ of $K$ points on the input image, in normalized coordinates, through a $\tanh$ output. A second, fixed set of base fiducial points $C'$ lies evenly along the top and bottom edges of the rectified output. The TPS that maps $C'$ onto $C$ has parameters

$$T = \left( \Delta_{C'}^{-1} \begin{bmatrix} C^\top \\ \mathbf{0}^{3 \times 2} \end{bmatrix} \right)^{\!\top} \in \mathbb{R}^{2 \times (K+3)}, \qquad \Delta_{C'} = \begin{bmatrix} \mathbf{1}^{K \times 1} & C'^\top & R \\ 0 & 0 & \mathbf{1}^{1 \times K} \\ 0 & 0 & C' \end{bmatrix},$$

where $R$ has entries $r_{ij} = d_{ij}^2 \ln d_{ij}^2$ for the distance $d_{ij}$ between base points $c'_i$ and $c'_j$ (ASTER’s code uses $d^2 \ln d$, half this value, which gives the same spline because the radial weights absorb the factor; Baek et al.’s supplement prints $d_{ij} \ln d_{ij}^2$ in $\Delta$ but $d^2 \ln d$ in the mapping below, two different functions, which in our reading is a misprint). Because $\Delta_{C'}$ depends only on the fixed base points, its inverse is a constant. An output pixel $p'$ is then mapped to the input point

$$p = T \, [1, x', y', r'_1, \dots, r'_K]^\top, \qquad r'_k = d_k^2 \ln d_k^2,$$

with $d_k$ its distance to the base point $c'_k$: an affine part plus a radial-basis correction that bends the grid through every fiducial. RARE used $K = 20$, a localization network of four convolutional layers and two fully connected layers on the $100 \times 32$ input, and an output of the same size. Its training needs no fiducial labels; the authors report that random initialization of the localization network failed to converge, so they set the final layer’s weights to zero and its biases to an initial pattern of points along the two edges. Their figures show the predicted points settling along the upper and lower edges of the text. Baek et al. later counted the TPS module at 1.7 million parameters and 3.6 ms per image, raising average accuracy by 1.1 points on regular and 3.4 points on irregular benchmarks.

ASTER. The same group extended RARE into ASTER (Shi, Yang, Wang, Lyu, Yao and Bai, TPAMI 2019, online 2018). The journal paper could not be read for this article, so its design is described from the official TensorFlow code (bgshih/aster). Its demo configuration predicts 20 control points from a copy of the input downsampled to 64 × 32, which makes the localization cheap, and samples the rectified image at 100 × 32 from the input at its full size (256 × 64 in the configuration); the initial control points follow a sine pattern rather than straight edges. The recognizer behind it uses a ResNet feature extractor (RESNET_50 in the configuration) with two bidirectional LSTM layers, and two attention decoders, one reading left to right and one right to left, each with weight 0.5 in the loss; at inference the code keeps the prediction of whichever decoder scores higher. Training draws equal halves of each batch from MJSynth and from SynthText crops. A correction in the README (October 2018) states that a bug had caused only part of SVT to be tested, and gives the corrected lexicon-free SVT accuracy as 89.5%.

MORAN. Luo, Jin and Sun’s multi-object rectified attention network (Pattern Recognition, 2019) rectifies with a dense offset map instead of a parametric transformation. The paper was not read; in the official version 2 code (MORAN_v2), the rectification network MORN downsamples the input, predicts a one-channel offset map with a small convolutional network, pools its positive and negative parts, and adds the interpolated offsets to the vertical coordinate of a regular sampling grid, so each point of the image can be moved up or down by its own amount; at test time the step can be repeated on the rectified image. During training the code skips rectification for a random half of the batches. The README lists MORAN v2 at 93.4% on IIIT5K, 79.7% on SVT-Perspective and 81.9% on CUTE80, with a ResNet backbone and a bidirectional decoder that it describes as borrowed from ASTER.

How much rectification helps. Rectification targets the irregular sets, and the gains are largest there in Baek et al.’s module study. Later evidence is mixed: PARSeq’s authors trained the TRBA model and its CTC counterpart TRBC on arbitrarily oriented text and found that the CTC model with TPS was still poor on rotated benchmarks, concluding that rectification contributes little to robustness against orientation and that attention is the main contributor. ABINet, which has no rectification module, reported the second-best CUTE80 accuracy among the methods it compared, which its authors attribute to the language model.

Language models

Language knowledge helps when the image alone is ambiguous. Attention decoders learn it implicitly; a series of methods made it explicit.

SRN. Yu et al. (2020) argued that recurrent decoders transmit semantic context only serially and in one direction. Their semantic reasoning network aligns one visual feature to each character position in parallel, takes the most likely character at each position (an argmax), embeds it, and passes the embeddings through transformer units, a global semantic reasoning module that sees all positions at once; a gated unit then fuses the visual and semantic features. ABINet’s authors describe SRN’s module as an ensemble of a left-to-right and a right-to-left transformer, and note that the argmax already stops gradients between the visual and language parts.

ABINet. Fang et al. (2021) proposed three principles for the language model, which they call autonomous, bidirectional and iterative. The vision model, a ResNet with transformer layers and a position-attention module, predicts character probabilities in parallel for up to $T$ positions. The language model is a bidirectional cloze network (BCN): a transformer decoder whose keys and values are the character probabilities and whose attention mask forbids each position from attending to itself, so every character is predicted from both sides of its context in one pass rather than by masking one character at a time as in BERT. The language model is autonomous because the gradient is blocked between the two models; it can therefore be pre-trained on text alone, such as WikiText-103. A gated fusion combines visual and linguistic features, and the language model is run again on the fused prediction, iteratively. In their ablations, allowing the gradient to flow lowered average accuracy by 0.9 points, which they attribute to the language model overfitting when the vision model “helps to cheat”; pre-training it on WikiText-103 helped while pre-training on the words of the training sets made little difference; and three iterations added 0.3–0.4 points on average, mostly on the harder sets. As a spelling corrector on 20,000 words with simulated spelling errors, BCN reached a top-5 word accuracy of 41.9%, against 27.6% for SRN’s module. Fang et al. also added ensemble self-training on unlabeled Uber-Text images.

PARSeq. Bautista and Atienza (2022) criticized the external language models of SRN and ABINet as conditionally independent of the image: such a model can “correct” a reading that is right but looks misspelled. Their appendix measured that ABINet’s language model, given the ground truth itself as input, returns the correct word only 50.44% of the time on the six benchmarks (36 characters). PARSeq instead learns an internal language model with permutation language modeling. A vision transformer (12 layers, 8 × 4 patches) encodes the image; a single decoder layer takes position queries, the context of already known characters and the image tokens. Training averages the cross-entropy over $K$ factorization orders of the likelihood,

$$\log p(\mathbf{y} \mid \mathbf{x}) = \mathbb{E}_{\mathbf{z} \sim \mathcal{Z}_T}\left[ \sum_{t=1}^{T} \log p_\theta\big(y_{z_t} \mid \mathbf{y}_{\mathbf{z}_{<t}}, \mathbf{x}\big) \right],$$

each order implemented by an attention mask rather than by permuting the labels ($K = 6$: the left-to-right order, two random orders and their reverses). The same weights then support autoregressive decoding, parallel non-autoregressive decoding and iterative refinement with a cloze mask in which every character sees all the others but itself, always together with the image. With $K = 1$, plain left-to-right training, non-autoregressive decoding failed completely (0.01%), which the authors take as evidence that permutation training is what unifies the decoding modes.

Other approaches named in these papers include SEED, which uses a pre-trained FastText model to guide the training of a recurrent decoder, and VisionLAN, which learns language by masking the visual features of individual characters during training.

Transformers and real training data

Two shifts mark recent work. Transformer encoders and decoders replaced recurrent ones, as in PARSeq and in the vision transformer recognizer ViTSTR that PARSeq compares with; general text recognizers such as TrOCR, described in optical character recognition, use the same encoder-decoder design. And labeled real data became large enough to train on. Baek, Matsui and Aizawa (2021) consolidated 11 public real datasets, 276,000 labeled words after preprocessing, about 1.7% of the size of the synthetic sets, and found that TRBA trained on them alone (86.6% over the six benchmarks) slightly surpassed TRBA trained on MJSynth and SynthText (85.7%), while CRNN came close (74.8% against 75.8%); with pseudo-labels and a rotation-prediction pretext task on unlabeled images TRBA reached 89.3%. An equal amount of synthetic data, 277,000 images, gave about 10 points less than the real data. PARSeq trained on 3.3 million real images, adding TextOCR and OpenVINO annotations of Open Images.

End-to-end text spotting

Text spotting joins detection and recognition in one network, so that the two share features and are trained together. FOTS (Liu et al., 2018) runs a fully convolutional oriented-text detector on shared convolutional features, extracts the features of each detected rotated box with RoIRotate, a bilinear-interpolation operator that turns an oriented region into an axis-aligned feature map of fixed height and variable width, related to RoI pooling, and reads them with a recurrent encoder and CTC; its authors report a real-time system (22.6 frames per second) that improved on earlier ICDAR 2015 spotting results by more than five points. Mask TextSpotter (Lyu et al., 2018) builds on Mask R-CNN: its mask branch predicts a text-instance map and 36 character segmentation maps (26 letters and 10 digits) for each proposal, and reads the word from the character maps, which handles curved text without a sequence decoder. End-to-end results are scored with the ICDAR end-to-end and word-spotting protocols, which require both an intersection over union match and a correct transcription and come with vocabularies of three sizes; the ICDAR article describes them.

Datasets and Benchmarks

Synthetic training data

MJSynth (MJ), also called Synth90k, is a synthetic set of word images made by Jaderberg et al. in 2014. Baek et al. describe it as 8.9 million word images rendered in seven steps: font rendering, border and shadow rendering, background coloring, composition of font, border and background, projective distortion, blending with real images and noise. RARE describes the released set as 8 million samples, and CRNN’s official configuration names its training database synth90k. The original paper and the official data page could not be read for this article.

SynthText (ST) was made by Gupta, Vedaldi and Zisserman (2016) for training text detectors: 800,000 scene images with text rendered onto 8,000 background images that contain no text of their own. Their engine segments each background into regions of uniform color and texture, estimates a dense depth map with a CNN, fits planar facets with RANSAC to obtain surface normals, renders words, lines or paragraphs from the Newsgroup20 corpus in a random font and a color learned from IIIT5K crops, warps the text to the local surface orientation and blends it with Poisson image editing. Recognizers use cropped word boxes; Baek et al. count 5.5 million crops after filtering out non-alphanumeric characters.

The counts differ between papers because of filtering: PARSeq’s appendix lists 7,224,586 MJSynth training images (plus 802,731 validation and 891,924 test images, which PARSeq also trains on) and 6,975,301 SynthText crops after filtering for its 94-character set, and Baek, Matsui and Aizawa use MJ (9 million) and ST (7 million). Training data matters: with their best model, Baek et al. (2019) obtained 80.0% total accuracy with MJSynth alone, 75.6% with SynthText alone and 84.1% with both (84.0% in their table of all combinations, a five-seed mean), and 81.3% with 20% of each (2.9 million images), which they read as diversity mattering more than size. PARSeq’s character-set results (below) suggest that the synthetic sets lack diversity of letter case.

Evaluation benchmarks

Seven sets of real cropped words became the standard tests. Their sizes as Baek et al. (2019) give them, with the versions in use:

Benchmark	Source of images	Test words	Versions in use	Group
IIIT5K	Google image search	3,000	3,000	regular
SVT	Google Street View	647	647	regular
IC03	ICDAR 2003 Robust Reading	1,110	867 and 860	regular
IC13	ICDAR 2013 Focused Scene Text	1,095	1,015 and 857	regular
IC15	ICDAR 2015 Incidental Scene Text (Google Glass)	2,077	2,077 and 1,811	irregular
SVT-Perspective	Google Street View, side views	645	645 (639 in its paper)	irregular
CUTE80	curved text, 80 images	288	288	irregular

• IIIT 5K-word (Mishra, Alahari and Jawahar, 2012) has 5,000 cropped words from scene and born-digital images found with Google image search queries such as billboards, signboards, house numbers and movie posters, split into 2,000 training and 3,000 test words; the official page also provides a lexicon of more than half a million dictionary words. RARE describes a 50-word and a 1,000-word lexicon for each test image, made of the true word and randomly chosen words.
• Street View Text (Wang, Babenko and Belongie) has 257 training and 647 test words from Google Street View, many noisy, blurred or of low resolution, each with a 50-word lexicon.
• ICDAR 2003 has 1,156 training and 1,110 test words by Baek et al.’s count (the competition’s 2002 data page listed 1,157 and 1,111; see ICDAR). Removing words shorter than three characters or with non-alphanumeric characters leaves 867, but many papers used 860; Baek et al. identified the seven missing words and estimated that they alone shift accuracy by 0.8 points. The protocol of discarding these words goes back to Wang et al. (2011), as RARE notes.
• ICDAR 2013 inherits most of its images from IC03; its 1,095 test words become 1,015 after removing non-alphanumeric words and 857 when short words are also removed. IC03’s training images overlap IC13’s test set (34 scene images with 215 words by Baek et al.’s count), so Baek et al. left IC03’s training data out of their validation set.
• ICDAR 2015 has 4,468 training and 2,077 test words; earlier papers used a 1,811-word subset that drops non-alphanumeric words and some extremely rotated, perspective and curved ones, without an unambiguous rule, so Baek et al. obtained the subset from an author of one of those papers and released it with their code. The competitions’ own protocols and annotation are in ICDAR.
• SVT-Perspective (Phan et al., 2013) was photographed at the addresses of the SVT test images from side views chosen so that the text stays readable, and reuses SVT’s lexicons. Its paper gives 238 images and 639 cropped words, with heights from 9 to 330 pixels; RARE also uses 639, while Baek et al. and PARSeq use 645.
• CUTE80 (Risnumawan et al., 2014) consists of 80 images of curved text lines with complex backgrounds, perspective distortion and low resolution, made for detection; its official page states that it is publicly available for non-commercial use. RARE cropped its words into the 288 test images used since.

Baek et al.’s unified test set is the union of all versions, 8,539 images (IIIT5K 3,000, SVT 647, IC03 867, IC13 1,015, IC15 2,077, SVT-Perspective 645, CUTE80 288). Later papers such as PARSeq and Baek, Matsui and Aizawa use six benchmarks with IC13 at 1,015 and IC15 at 2,077 words, 7,672 images in total, and PARSeq notes how small that is against the test sets of other vision tasks; it also reports on the test sets of ArT (35,149 words, curved and rotated text), COCO-Text (9,825, low resolution and occlusion) and Uber-Text (80,551, vertical and rotated text).

Reported results

Results below are word accuracies as the papers print them, on the 36-character case-insensitive protocol unless a column states otherwise; numbers from different training data or test versions are not comparable.

The unified comparison (Baek et al., 2019). All models trained on MJSynth plus SynthText (14.4 million images) with the same optimizer and schedule, each averaged over five random seeds, timed on one NVIDIA Tesla P40. Along the accuracy–time frontier, each step changes one module, and the order is ResNet, BiLSTM, TPS and attention:

Combination	Total accuracy, %	Time, ms/image	Parameters, millions
None-VGG-None-CTC	69.5	1.3	5.6
None-ResNet-None-CTC	80.0	4.7	46.0
None-ResNet-BiLSTM-CTC	81.9	7.8	48.7
TPS-ResNet-BiLSTM-CTC	82.9	10.9	49.6
TPS-ResNet-BiLSTM-Attn	84.0	27.6	49.6

The first three changes raise accuracy by 13.4 points for 9.6 ms; attention adds 1.1 points for 16.7 ms more. For memory the most efficient order is the reverse, and the ResNet alone accounts for most of the parameters, so the authors advise memory-constrained users to add transformation, sequence and prediction modules freely but to avoid heavy feature extractors. Averaged over the combinations that contain each module, the irregular sets gained about twice as much as the regular sets from every upgrade. Their best model scored 87.9% on IIIT5K, 87.5% on SVT, 92.3% on IC13 (1,015), 71.8% on IC15 (2,077), 79.2% on SVT-Perspective and 74.0% on CUTE80; training it with special characters raised IIIT5K accuracy from 87.9% to 90.3%. The CRNN rows of this comparison are discussed in the CRNN article.

Later models. ABINet’s large vision model with BCN, trained on MJSynth and SynthText, reported 97.4% on IC13, 93.5% on SVT, 96.2% on IIIT5K, 86.0% on IC15, 89.3% on SVT-Perspective and 89.2% on CUTE80 (ABINet’s Table 6, which does not state the IC13 and IC15 versions; PARSeq’s table lists these numbers under the 857- and 1,811-word versions), and 97.7%, 95.5%, 97.2%, 86.9%, 89.9% and 94.1% with ensemble self-training on unlabeled Uber-Text. ASTER’s reported results, as Baek, Matsui and Aizawa tabulate them, are 93.4% on IIIT5K, 89.5% on SVT, 91.8% on IC13 (1,015), 76.1% on IC15 (2,077), 78.5% on SVT-Perspective and 79.5% on CUTE80. PARSeq retrained CRNN, ViTSTR, TRBA and ABINet under its own protocol (four replicates each, compared with t-tests). Its word accuracies on the union of the six benchmarks (7,672 words), means of four replicates:

Method	Synthetic, 36 chars	Synthetic, 94 chars	Real, 36 chars	Real, 94 chars
CRNN	83.2	54.8	88.5	85.8
TRBA	90.6	69.9	95.2	92.5
ABINet	89.8	66.4	95.2	92.4
PARSeq (autoregressive)	91.9	73.0	96.0	93.3

Two observations follow from the table, as the authors draw them: real training data raises every model, and with synthetic training, accuracy collapses when case and punctuation are scored. On the three large test sets combined (125,525 words), PARSeq reached 51.8% with synthetic and 84.1% with real training data, against 41.1% and 56.3% for CRNN. Their reproduction of ABINet scored about 1.4 points below its published results, which they attribute to ABINet’s much longer original schedule and its WikiText-103 pre-training. Current results are tracked by the competitions’ leaderboards, linked from the ICDAR article.

Evaluation Metrics

Word accuracy is the primary metric: the percentage of test images whose predicted string equals the ground truth after normalization; PARSeq states the rule as a match at every character position. It is an accuracy over words, so one wrong character fails the whole word. The conventions that change it:

• Character set and case. The traditional protocol compares only letters and digits and ignores case. ASTER’s evaluation code lowercases both strings and deletes every character that is not an ASCII letter or digit before comparing; Baek et al. evaluate “only on alphabets and digits”, and their code does the same for case-sensitive models (lowercasing and keeping [0-9a-z]). PARSeq evaluates on three nested character sets, the first 36, 62 and 94 characters of Python’s string.printable (lowercase alphanumerics; mixed-case alphanumerics; mixed case with punctuation). The table above shows how much the choice matters.
• Test-set version, as in the benchmark table.
• Training data: synthetic, real, or synthetic plus benchmark training sets; PARSeq marks which.
• Lexicons: lexicon-based results (50, 1,000, full or 50,000 words) are not comparable with lexicon-free ones.
• Aggregation: per-benchmark accuracies, their mean, or the accuracy over the union of the test sets (“total” or “combined” accuracy), which weights large sets more.
• Run-to-run variation: Baek et al. average five seeds, Baek, Matsui and Aizawa three, and PARSeq reports means and standard deviations over four replicates with significance tests.

Edit-distance measures count partial errors: the ICDAR word recognition tasks rank methods by the total normalized edit distance, case-sensitive (ICDAR), and PARSeq’s appendix uses one minus the normalized edit distance in its analysis of ABINet’s language model. Character and word error rates are defined in optical character recognition.

Cost is reported as inference time per image on stated hardware, parameter count and FLOPs; Baek et al. plot accuracy against time and parameters, and PARSeq also against FLOPs.

Example: rectifying curved and perspective words with a thin-plate spline

The script below asks how much a thin-plate-spline rectification step helps a recognizer that has only seen straight text, when the fiducial points are known. It renders words of three to seven random capitals in the $5 \times 7$ bitmap font of the optical character recognition and CRNN examples (glyphs of $21 \times 10$ pixels) on a flat strip 32 pixels high, and places each word on a $96 \times 192$ canvas scaled by a random factor of 1 to 1.3, in one of three geometries: straight, perspective (a homography that makes one end 0.45–0.7 times as tall as the other, then a rotation of up to ±15°) or curved (wrapped on a circular arc subtending 60°–120°, bending up or down). Each canvas is blurred with a $3 \times 3$ box filter, given a random contrast and polarity and Gaussian noise of standard deviation 0.08. The ground-truth fiducials are 10 points along the top edge and 10 along the bottom edge of the strip, mapped through the same warp, the configuration of RARE and ASTER ($K = 20$).

A small CTC recognizer (convolutional features, 25 columns, a linear classifier per column, the None-VGG-None-CTC pattern) is trained for 800 steps on 6,000 axis-aligned crops, whose boxes are jittered by up to 2 pixels, in two versions: model S on straight words only, model A on equal shares of the three geometries, without rectification. Every test word (500 per geometry) is turned into a $32 \times 100$ input in up to four ways: the axis-aligned bounding box of the fiducials, cropped and resized; a TPS fitted to the 4 corner points only; a TPS fitted to the 20 exact fiducials; and a TPS fitted to the 20 fiducials perturbed by Gaussian noise of 1.5 pixels, standing in for the errors of a learned localization network. Model S is scored with all four, model A with the crop and the exact TPS. The TPS follows RARE’s equations in pixel coordinates, solved with NumPy, and sampling is bilinear. Before training, two checks use no model: the correlation between the exact TPS rectification of each clean rendering and the flat strip resized to $32 \times 100$, which tests the rectification itself, and the vertical spread of the text’s centre line inside the axis-aligned crop, in units of the text height, which measures how far each geometry departs from a horizontal line.

The script takes the random seed as an argument. It was run with PyTorch 2.14.1, NumPy 2.5.4 and SciPy 1.18.1 on four threads, with seeds 0, 1 and 2, seed 0 twice with identical output; one run took 140 to 210 seconds on our machine, longer than the “about 60 s per seed” estimated in our plan.

import sys
import numpy as np
import torch
import torch.nn as nn
from scipy import ndimage

seed = int(sys.argv[1]) if len(sys.argv) > 1 else 0
torch.set_num_threads(4)

# ---- 5x7 bitmap font (OCR and CRNN articles); each cell 3 px high, 2 px wide -> glyphs of 21 x 10 px ----
FONT = """
A .###. #...# #...# ##### #...# #...# #...#
B ####. #...# #...# ####. #...# #...# ####.
C .###. #...# #.... #.... #.... #...# .###.
D ####. #...# #...# #...# #...# #...# ####.
E ##### #.... #.... ####. #.... #.... #####
F ##### #.... #.... ####. #.... #.... #....
G .###. #...# #.... #.### #...# #...# .###.
H #...# #...# #...# ##### #...# #...# #...#
I .###. ..#.. ..#.. ..#.. ..#.. ..#.. .###.
J ..### ...#. ...#. ...#. ...#. #..#. .##..
K #...# #..#. #.#.. ##... #.#.. #..#. #...#
L #.... #.... #.... #.... #.... #.... #####
M #...# ##.## #.#.# #.#.# #...# #...# #...#
N #...# #...# ##..# #.#.# #..## #...# #...#
O .###. #...# #...# #...# #...# #...# .###.
P ####. #...# #...# ####. #.... #.... #....
Q .###. #...# #...# #...# #.#.# #..#. .##.#
R ####. #...# #...# ####. #.#.. #..#. #...#
S .#### #.... #.... .###. ....# ....# ####.
T ##### ..#.. ..#.. ..#.. ..#.. ..#.. ..#..
U #...# #...# #...# #...# #...# #...# .###.
V #...# #...# #...# #...# #...# .#.#. ..#..
W #...# #...# #...# #.#.# #.#.# #.#.# .#.#.
X #...# #...# .#.#. ..#.. .#.#. #...# #...#
Y #...# #...# .#.#. ..#.. ..#.. ..#.. ..#..
Z ##### ....# ...#. ..#.. .#... #.... #####
"""
GLYPH = {}
for row in FONT.strip().splitlines():
    ch, *rows = row.split()
    GLYPH[ch] = np.kron(np.array([[c == "#" for c in r] for r in rows], float), np.ones((3, 2)))
ALPHA = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
H0, CH, CW = 32, 96, 192            # flat source height; canvas height and width
OH, OW = 32, 100                    # recognizer input
K = 20                              # fiducial points (10 on the top edge, 10 on the bottom edge)

def flat_source(word, rng):
    gaps = rng.integers(0, 4, size=len(word) - 1)
    W0 = 10 * len(word) + gaps.sum() + 12
    src = np.zeros((H0, W0))
    x = 6
    for i, ch in enumerate(word):
        src[5:26, x:x + 10] = np.maximum(src[5:26, x:x + 10], GLYPH[ch])
        x += 10 + (gaps[i] if i < len(word) - 1 else 0)
    return src

# ---- geometries: forward map (u, v) on the flat source -> (x, y) on the canvas, and its inverse ----
def homography(src_pts, dst_pts):
    A = []
    for (u, v), (x, y) in zip(src_pts, dst_pts):
        A += [[u, v, 1, 0, 0, 0, -x * u, -x * v], [0, 0, 0, u, v, 1, -y * u, -y * v]]
    h = np.linalg.solve(np.array(A, float), np.array(dst_pts, float).ravel())
    return np.append(h, 1).reshape(3, 3)

def apply_h(Hm, u, v):
    w = Hm[2, 0] * u + Hm[2, 1] * v + Hm[2, 2]
    return (Hm[0, 0] * u + Hm[0, 1] * v + Hm[0, 2]) / w, (Hm[1, 0] * u + Hm[1, 1] * v + Hm[1, 2]) / w

def make_geometry(kind, W0, rng):
    sc = rng.uniform(1.0, 1.3)
    if kind in ("straight", "perspective"):
        if kind == "straight":
            s, ang = 1.0, 0.0
        else:
            s, ang = rng.uniform(0.45, 0.7), np.deg2rad(rng.uniform(-15, 15))
        hl, hr = H0 * sc, H0 * sc * s
        if rng.integers(2):
            hl, hr = hr, hl
        w = W0 * sc
        quad = np.array([[-w / 2, -hl / 2], [w / 2, -hr / 2], [w / 2, hr / 2], [-w / 2, hl / 2]])
        c, sn = np.cos(ang), np.sin(ang)
        quad = quad @ np.array([[c, sn], [-sn, c]])
        Hm = homography([(0, 0), (W0, 0), (W0, H0), (0, H0)], quad)
        Hi = np.linalg.inv(Hm)
        fwd = lambda u, v: apply_h(Hm, u, v)
        inv = lambda x, y: apply_h(Hi, x, y)
    else:                                                     # curved: wrapped on a circular arc
        phi = np.deg2rad(rng.uniform(60, 120))
        R = W0 * sc / phi
        d = 1 if rng.integers(2) else -1                      # +1 arch (bends down at the ends), -1 bowl
        def fwd(u, v):
            th = (u - W0 / 2) * sc / R
            r = R + d * (H0 / 2 - v) * sc
            return r * np.sin(th), -d * r * np.cos(th)
        def inv(x, y):
            dx, dy = x, -d * y
            r, th = np.hypot(dx, dy), np.arctan2(dx, dy)
            return W0 / 2 + th * R / sc, H0 / 2 - (r - R) / (d * sc)
    # centre the word's outline on the canvas, with a small random shift
    t = np.linspace(0, 1, 60)
    bu = np.concatenate([t * W0, t * W0, np.zeros(60), np.full(60, W0)])
    bv = np.concatenate([np.zeros(60), np.full(60, H0), t * H0, t * H0])
    bx, by = fwd(bu, bv)
    ox = CW / 2 - (bx.min() + bx.max()) / 2 + rng.uniform(-5, 5)
    oy = CH / 2 - (by.min() + by.max()) / 2 + rng.uniform(-5, 5)
    F = lambda u, v: tuple(a + o for a, o in zip(fwd(u, v), (ox, oy)))
    G = lambda x, y: inv(x - ox, y - oy)
    return F, G

def fiducials(F, W0, k):
    u = np.linspace(0, W0, k // 2)
    xt, yt = F(u, np.zeros_like(u))
    xb, yb = F(u, np.full_like(u, H0))
    return np.concatenate([np.stack([xt, yt], 1), np.stack([xb, yb], 1)])

def render(word, kind, rng):
    src = flat_source(word, rng)
    W0 = src.shape[1]
    F, G = make_geometry(kind, W0, rng)
    yy, xx = np.mgrid[0:CH, 0:CW] + 0.5                       # canvas pixel centres
    u, v = G(xx, yy)
    clean = ndimage.map_coordinates(src, [v - 0.5, u - 0.5], order=1, cval=0.0)
    p = np.pad(clean, 1, mode="edge")
    img = sum(p[i:i + CH, j:j + CW] for i in range(3) for j in range(3)) / 9
    img = rng.uniform(0.5, 1.0) * img
    if rng.integers(2):
        img = 1 - img
    img = img + rng.normal(0, 0.08, img.shape)
    return img, clean, src, F, fiducials(F, W0, K)

# ---- the four test-time pipelines, each giving a 32 x 100 image ----
def sample(img, x, y):
    return ndimage.map_coordinates(img, [y - 0.5, x - 0.5], order=1, mode="nearest")

def bbox_crop(img, pts, jitter=None):
    x0, y0 = pts.min(0)
    x1, y1 = pts.max(0)
    if jitter is not None:
        x0, y0, x1, y1 = np.array([x0, y0, x1, y1]) + jitter
    yy, xx = np.mgrid[0:OH, 0:OW] + 0.5
    return sample(img, x0 + xx / OW * (x1 - x0), y0 + yy / OH * (y1 - y0))

def tps_rectify(img, C):
    """RARE eq. 1-4: thin-plate spline from base points C' on the output edges to the fiducials C on the input."""
    k = len(C)
    xs = np.linspace(0, OW, k // 2)
    Cp = np.concatenate([np.stack([xs, np.zeros(k // 2)], 1), np.stack([xs, np.full(k // 2, OH)], 1)])
    def U(d2):
        return np.where(d2 > 0, d2 * np.log(np.maximum(d2, 1e-12)), 0.0)   # r^2 log r^2
    Rm = U(((Cp[:, None, :] - Cp[None, :, :]) ** 2).sum(2))
    delta = np.zeros((k + 3, k + 3))
    delta[:k, 0], delta[:k, 1:3], delta[:k, 3:] = 1, Cp, Rm
    delta[k, 3:] = 1
    delta[k + 1:, 3:] = Cp.T
    T = np.linalg.solve(delta, np.vstack([C, np.zeros((3, 2))]))           # (k+3) x 2
    yy, xx = np.mgrid[0:OH, 0:OW] + 0.5
    P = np.stack([xx.ravel(), yy.ravel()], 1)
    lifted = np.hstack([np.ones((len(P), 1)), P, U(((P[:, None, :] - Cp[None, :, :]) ** 2).sum(2))])
    q = lifted @ T
    return sample(img, q[:, 0], q[:, 1]).reshape(OH, OW)

def corners(C):
    h = len(C) // 2
    return C[[0, h - 1, h, 2 * h - 1]]

# ---- data ----
def random_word(rng):
    return "".join(rng.choice(list(ALPHA), size=rng.integers(3, 8)))

def train_set(kinds, n, rng):
    X, words = [], []
    for i in range(n):
        w = random_word(rng)
        img, _, _, _, C = render(w, kinds[i % len(kinds)], rng)
        X.append(bbox_crop(img, C, jitter=rng.uniform(-2, 2, 4)))
        words.append(w)
    return np.stack(X)[:, None].astype(np.float32) * 2 - 1, words

KINDS = ["straight", "perspective", "curved"]
rng_test = np.random.default_rng([seed, 2])
test = {k: [] for k in KINDS}
for kind in KINDS:
    for _ in range(500):
        w = random_word(rng_test)
        test[kind].append((w,) + render(w, kind, rng_test))

# ---- label-free checks ----
print("Label-free checks (no model)")
for kind in KINDS:
    corr, spread = [], []
    for w, img, clean, src, F, C in test[kind]:
        flat = ndimage.zoom(src, (OH / src.shape[0], OW / src.shape[1]), order=1)
        rect = tps_rectify(clean, C)
        corr.append(np.corrcoef(flat.ravel(), rect.ravel())[0, 1])
        W0 = src.shape[1]
        u = np.linspace(0, W0, 10)
        xc, yc = F(u, np.full_like(u, H0 / 2))
        hgt = np.hypot(*(C[10:] - C[:10]).T).mean()
        spread.append((yc.max() - yc.min()) / hgt)
    print(f"  {kind:<11} corr(TPS-20 rectified clean, flat source) = {np.mean(corr):.3f}   "
          f"centreline vertical spread = {np.mean(spread):.2f} text heights")

# ---- recognizer: conv features -> 25 columns -> per-column classifier, CTC ----
class Recognizer(nn.Module):
    def __init__(self):
        super().__init__()
        self.cnn = nn.Sequential(
            nn.Conv2d(1, 32, 3, 1, 1), nn.ReLU(), nn.MaxPool2d(2, 2),
            nn.Conv2d(32, 64, 3, 1, 1), nn.ReLU(), nn.MaxPool2d(2, 2),
            nn.Conv2d(64, 96, 3, 1, 1), nn.BatchNorm2d(96), nn.ReLU(), nn.MaxPool2d((2, 1)),
            nn.Conv2d(96, 96, 3, 1, 1), nn.ReLU(), nn.MaxPool2d((2, 1)),
            nn.Conv2d(96, 128, (2, 3), 1, (0, 1)), nn.BatchNorm2d(128), nn.ReLU())
        self.out = nn.Linear(128, 27)
    def forward(self, x):
        return self.out(self.cnn(x).squeeze(2).permute(2, 0, 1)).log_softmax(2)   # (T=25, B, 27)

def encode(words):
    return (torch.tensor([ALPHA.index(c) + 1 for w in words for c in w]),
            torch.tensor([len(w) for w in words]))

def decode(logp):
    out = []
    for r in logp.argmax(2).T.numpy():
        out.append("".join(ALPHA[k - 1] for t, k in enumerate(r) if k != 0 and (t == 0 or k != r[t - 1])))
    return out

def train(X, words, steps=800):
    torch.manual_seed(seed)
    model = Recognizer()
    opt = torch.optim.Adam(model.parameters(), lr=1e-3)
    ctc = nn.CTCLoss(blank=0, zero_infinity=True)
    rng_b = np.random.default_rng([seed, 4])
    model.train()
    for step in range(steps):
        idx = rng_b.integers(0, len(X), 32)
        tgt, tl = encode([words[i] for i in idx])
        logp = model(torch.from_numpy(X[idx]))
        loss = ctc(logp, tgt, torch.full((32,), 25, dtype=torch.long), tl)
        opt.zero_grad(); loss.backward()
        nn.utils.clip_grad_norm_(model.parameters(), 5.0)
        opt.step()
    model.eval()
    return model, loss.item()

def accuracy(model, imgs, words):
    X = np.stack(imgs)[:, None].astype(np.float32) * 2 - 1
    with torch.no_grad():
        pred = decode(model(torch.from_numpy(X)))
    return 100 * np.mean([p == w for p, w in zip(pred, words)])

rng_noise = np.random.default_rng([seed, 5])
pipelines = {
    "no rectification (axis-aligned crop)": lambda img, C: bbox_crop(img, C),
    "TPS, 4 corner points": lambda img, C: tps_rectify(img, corners(C)),
    "TPS, 20 points, exact": lambda img, C: tps_rectify(img, C),
    "TPS, 20 points, noise 1.5 px": lambda img, C: tps_rectify(img, C + rng_noise.normal(0, 1.5, C.shape)),
}
inputs = {(p, k): [f(img, C) for w, img, clean, src, F, C in test[k]] for p, f in pipelines.items() for k in KINDS}
truth = {k: [t[0] for t in test[k]] for k in KINDS}

models = {}
Xs, ws = train_set(["straight"], 6000, np.random.default_rng([seed, 1]))
models["S (trained on straight words)"] = train(Xs, ws)
Xa, wa = train_set(KINDS, 6000, np.random.default_rng([seed, 3]))
models["A (trained on all three geometries)"] = train(Xa, wa)

print(f"\nWord accuracy (%) on 500 test words per geometry, seed {seed}")
print(f"  {'model / pipeline':<58}" + "".join(f"{k:>13}" for k in KINDS))
for mname, (model, last_loss) in models.items():
    print(f"  model {mname}, final training CTC loss {last_loss:.3f}")
    use = list(pipelines) if mname.startswith("S") else ["no rectification (axis-aligned crop)", "TPS, 20 points, exact"]
    for p in use:
        print(f"    {p:<56}" + "".join(f"{accuracy(model, inputs[(p, k)], truth[k]):13.1f}" for k in KINDS))

Output of python str_rectification.py 0:

Label-free checks (no model)
  straight    corr(TPS-20 rectified clean, flat source) = 0.980   centreline vertical spread = 0.00 text heights
  perspective corr(TPS-20 rectified clean, flat source) = 0.918   centreline vertical spread = 0.36 text heights
  curved      corr(TPS-20 rectified clean, flat source) = 0.937   centreline vertical spread = 0.38 text heights

Word accuracy (%) on 500 test words per geometry, seed 0
  model / pipeline                                               straight  perspective       curved
  model S (trained on straight words), final training CTC loss 0.006
    no rectification (axis-aligned crop)                            100.0          3.6          0.8
    TPS, 4 corner points                                            100.0         64.8          0.0
    TPS, 20 points, exact                                           100.0         87.0         99.6
    TPS, 20 points, noise 1.5 px                                     96.4         57.0         93.6
  model A (trained on all three geometries), final training CTC loss 0.150
    no rectification (axis-aligned crop)                             93.2         52.4         49.4
    TPS, 20 points, exact                                            93.2         51.2         92.4

The checks without a model show that the warps are substantial, the centre line drifting vertically by more than a third of the text height inside the axis-aligned crop of a perspective or curved word, and that the exact thin-plate spline undoes them well but not perfectly, with lower correlations for perspective and curved words than for straight ones.

The recognizer trained only on straight words reads straight crops almost perfectly, but it fails almost completely on the axis-aligned crops of distorted words (under 4% for perspective and under 1% for curved words in all three seeds), a larger loss for perspective text than our plan predicted (“to roughly 30-70%”). The axis-aligned crop squeezes a slanted or bent word into a short, tilted band that the column-wise model has never seen. A thin-plate spline through the 20 exact fiducials brings the curved words back to 99–100%, as planned, but the perspective words only to 85–87%, about 13–15 points short of the straight set, where the plan expected both distorted sets “within about 5 points of the straight set”. A likely reason, which the example does not test, is resolution: the far end of a perspective word is rendered with as little as 45% of the height of the near end and is then stretched back to full height, so the rectified glyphs there are blurred, the effect the lower correlation also shows. Rectification can restore geometry but not detail that the camera did not capture.

The other two pipelines separate what the fiducials contribute. Four corner points recover a large part of the perspective loss (62–65%) and nothing of the curvature (0–0.2%): with four points, the three side conditions on the radial weights leave a spline that is an affine map plus one fixed combination of the four radial functions with a single free coefficient per coordinate, which cannot follow a bend, which is why RARE and ASTER place ten points along each edge. Noise of 1.5 pixels on the 20 points costs a few points on straight and curved words (96–97% and 93–97%) but 21–30 points on perspective words (57–63%), so the accuracy of the localization network matters most where the rectification already stretches the image; how accurate a learned localizer is depends on the model and its training, which the example does not settle, so both cases are printed.

Model A, trained on all three geometries without rectification, does far better than model S on unrectified distorted words (45–52% on perspective and 40–55% on curved words, against under 4%) and worse on straight ones (88–96%; the plan expected “a little below model S”, and the gap is 4–12 points), but stays far below model S with exact rectification on curved words. Given the exact fiducials, rectification helps model A on curved words too (88–94%), but on perspective words it moves the accuracy only to 51–64%, far below model S, and in seed 0 it lowers it slightly. Model A’s results vary much more across seeds than model S’s, consistent with its higher final training loss: 800 steps are few for a classifier that must cover three geometries with no stage to normalize them. In this small setting, a rectification stage with good fiducials is worth more than training on the distortions, but the example gives the rectifier the true fiducials; a learned localization network, as in RARE and ASTER, must predict them from the image and is trained only through the recognition loss. Read with the noisy fiducials instead, the comparison still holds in every seed: model S reads 57–63% of the perspective and 93–97% of the curved words, against 45–52% and 40–55% for model A without rectification.

Applications

Scene text recognition is the reading stage of systems that extract text from photographs and video. The papers cited here name reading road signs, billboards, product labels and logos, self-driving cars, augmented reality, retail, education and aids for visually impaired people (Bautista and Atienza); business-name search on online maps from street-level imagery, sign translation, driving assistance and robot navigation (Phan et al.); and translation, street-sign reading for autonomous driving and reading cards to enter personal information (Baek, Matsui and Aizawa). Recognizers also score candidate regions for detectors: TextBoxes rescored detections with a CRNN recognizer and a lexicon (scene text detection). Reading text in images is also part of visual question answering benchmarks such as TextVQA, and large vision-language models now read text as part of answering questions about images.

Open Problems

• Vertical, rotated and heavily curved text. Baek et al. found vertical text among the failures of all 24 models, and in PARSeq’s orientation tests (94-character accuracy of models trained on real data) CRNN’s accuracy fell from 85.8% to a mean of 9.6% on the benchmarks rotated by 90°, 180° and 270°, against means of 69.6–86.2% for ViTSTR, TRBA, ABINet and PARSeq.
• Occlusion and language. External language models can overcorrect, as PARSeq’s analysis of ABINet’s language model shows, and internal ones are limited by the text seen in training.
• Case, punctuation and other scripts. Synthetic training data lacks case diversity; the 36-character protocol hides it, and most benchmarks are English.
• The synthetic-to-real gap and data efficiency. Real labels now outperform synthetic ones in the studies above, but they are limited for artistic, handwritten and many non-English texts, the cases Baek, Matsui and Aizawa name as motivation for training with fewer labels.
• Benchmark saturation and noise. The six classic test sets total 7,672 words, contain label errors and are dominated by horizontal text, so, in our reading, differences of a point or less between methods can lie within the noise of training runs and labels; larger, harder test sets such as ArT, COCO-Text and Uber-Text show wider gaps.
• Low resolution and calligraphic fonts, which Baek et al. list among the remaining failures and for which they suggest super-resolution modules, image pyramids and regularization.