CRNN

CRNN, the convolutional recurrent neural network, is a word recognizer for images of text that reads a cropped word as a sequence instead of segmenting it into characters. It was introduced by Baoguang Shi, Xiang Bai and Cong Yao, of the Huazhong University of Science and Technology group that also published RARE, in a 2015 preprint, published in 2017 in IEEE Transactions on Pattern Analysis and Machine Intelligence. The model stacks three parts trained together: the convolutional layers of a convolutional neural network, whose feature map is cut into columns that form a left-to-right sequence of feature vectors; a deep bidirectional LSTM, a recurrent network that labels each column using context from both sides; and a transcription layer trained with connectionist temporal classification (CTC), which turns the per-column label distributions into a string. Its main application is scene text recognition, the reading of words cropped from photographs, a part of optical character recognition; the paper’s title presents it more generally as a network for image-based sequence recognition. Baek et al. (2019) describe CRNN as the first combined application of CNNs and RNNs to scene text recognition, and their framework writes it as None-VGG-BiLSTM-CTC: no rectification stage, a VGG-style feature extractor, a bidirectional LSTM and CTC prediction.

The journal paper could not be read for this article: its open copy is on arXiv and the publisher’s copy on IEEE Xplore, neither of which the site uses, OpenAlex lists no repository copy, and Xiang Bai’s publications page links the paper to arXiv only. The architecture, training settings and decoding procedures below are taken from the authors’ released code, bgshih/crnn (commit f5d41e3, the repository’s last commit, of May 2018); the results come from later papers that tabulate them, chiefly RARE (Shi et al., 2016), by the same group, which tested the CRNN model on further benchmarks, and the re-evaluation of Baek et al. (2019). The paper is listed among the references only to identify the work. Two parts of the paper are therefore known only from indirect pointers: its search procedure for large lexicons, which the released code does not contain, and its experiment on a second application, which the publications page suggests was the recognition of music scores (see Variants).

Problem Addressed

Text in natural images is read word by word from crops supplied by a scene text detection stage. RARE’s review of the earlier work divides it into two families. Bottom-up methods first detect individual characters, by sliding windows, connected components or Hough voting, and then assemble them into words by dynamic programming or lexicon search; in our reading, they inherit every error of character detection, which is hard when characters touch or are degraded. Top-down methods recognize the whole word image: Jaderberg et al. trained a convolutional network with 90,000 classes, one per English word, which RARE notes cannot recognize words outside its dictionary, such as telephone numbers; other work predicted label embeddings, or extracted a sequence of HOG descriptors along the word and read it with a recurrent network (Su and Lu).

Treating a word as a sequence suits the problem, since words vary in length and the reading order is fixed. CRNN combines the two ingredients that were available: convolutional features learned from pixels, instead of hand-designed descriptors, and a recurrent network with a CTC output, which needs only the transcription of each training image, not character positions. Baek et al. summarize its contribution in these terms: a recurrent network handles the varying number of characters and CTC identifies how many there are.

Architecture

The official configuration (model/crnn_demo/config.lua) takes a grayscale image scaled to $100 \times 32$ pixels, rescales the pixel values from $[0, 255]$ to about $[-1, 1]$, and outputs, for each of 26 columns, a log-probability distribution over 37 labels: the 10 digits, the 26 letters (case is not distinguished) and the CTC blank. Baek et al. call the three parts feature extraction, sequence modelling and prediction.

Convolutional Layers

The feature extractor is a VGG-style stack of seven convolutional layers with ReLU activations and no fully connected layers: six $3 \times 3$ convolutions (stride 1, padding 1) with 64, 128, 256, 256, 512 and 512 channels, and a final $2 \times 2$ convolution without padding with 512 channels. Batch normalization follows the third, fifth and seventh convolutions. Max pooling follows the first, second, fourth and sixth convolutions; the first two pool $2 \times 2$ with stride 2, and the last two use a $2 \times 2$ window with stride 2 in height but 1 in width, padded by one column on each side, so they halve the height while the width grows by one column each time. The comments in the configuration track the shapes: a $1 \times 32 \times 100$ input becomes $64 \times 16 \times 50$, $128 \times 8 \times 25$, then $512 \times 2 \times 26$, and the last convolution gives a $512 \times 1 \times 26$ map. The final shape is right, but the map before the last convolution is in fact 27 columns wide (each padded pooling turns a width $w$ into $w + 2 - 2 + 1 = w + 1$, so 25 becomes 26 and then 27); the unpadded $2 \times 2$ convolution removes one column. Text is wide and short, so the network keeps horizontal resolution, which becomes the length of the sequence, while it collapses the height to one row.

Re-implementations differ in these details. Baek et al.’s supplementary Table C, the VGG module they say is used in CRNN and RARE, puts batch normalization after the fifth and sixth convolutions and pools the last two times with a window one column wide and two rows high (written $1 \times 2$, width first), which gives 24 columns rather than 26. RARE’s own encoder, which follows CRNN, is described with $2 \times 2$ pooling after the same four layers.

Map-to-Sequence

The $512 \times 1 \times 26$ map is reshaped into a sequence of 26 vectors of 512 dimensions, one per column, ordered from left to right (nn.View and nn.Transpose in the configuration). Each column is a feature vector for a vertical strip of the image; Baek et al. describe each column as having its own receptive field along the horizontal line of the input, used to estimate the character in that field. Computed layer by layer for the official configuration, as in the example below, each column sees a window 54 pixels wide that covers the full height, and neighbouring windows are 4 pixels apart, so a character a few columns wide falls in several windows. In our reading, the sequence plays the role of a classifier scanned along the word at a stride of 4 pixels, as windows are scanned in sliding window detection, with the convolutions shared between overlapping windows. RARE gives this as its reason for following CRNN in its own encoder: describing patches cut from left to right with a separate CNN would not share computation between overlapping patches and would ignore the dependencies between them.

Deep Bidirectional LSTM

Two bidirectional LSTM layers, each with 256 hidden units per direction, read the sequence. In the official code each layer runs a forward and a backward LSTM over the 26 frames and joins them with BiRnnJoin, which projects each direction’s output with its own linear layer and adds the two: to 256 dimensions after the first layer and to the 37 output labels after the second. A log-softmax at every frame then gives the per-frame label distributions. The LSTM unit is the standard one with input, forget and output gates, adapted, according to the code, from Karpathy’s char-rnn. Baek et al. implement the same sequence stage as a two-layer BiLSTM with a linear layer between the layers that merges the two directions, and they describe its purpose as giving each frame the contextual information that a single column lacks. In their qualitative examples, adding the BiLSTM let models ignore characters partly cut off at the edge of a crop, such as an “I” at the end of “EXIT”.

The configuration fixes the number of frames: maxT = 26, and the recurrent layers are unrolled for that many steps. RARE describes its CRNN-style encoder as accepting an image of arbitrary size, but the released CRNN code resizes every image to $100 \times 32$ before recognition (loadAndResizeImage), so long words are compressed horizontally.

Transcription Layer

The transcription layer is CTC: the network’s 26 per-frame distributions define, for any label sequence $\mathbf{l}$, the probability

$$p(\mathbf{l} \mid \mathbf{y}) = \sum_{\pi:\, \mathcal{B}(\pi) = \mathbf{l}} \; \prod_{t=1}^{26} y^{t}_{\pi_t},$$

where $\pi$ runs over all frame-level label sequences, including blanks, and $\mathcal{B}$ merges repeated labels and then removes blanks. The official code computes the sum and its gradient with the forward–backward algorithm in log space (src/cpp/ctc.cpp, with blank as label 0). The connectionist temporal classification article derives the loss, the recursion and the decoders; what matters here is that the 26 frames bound the output, since each pair of equal adjacent characters needs a blank between them, so a word fits only if its length plus its number of repeated pairs is at most 26.

Training

Objective. With a training set $\mathcal{X} = \{(I_i, \mathbf{l}_i)\}$ of word images and their transcriptions, training minimizes the negative log-likelihood

$$\mathcal{O} = -\sum_{(I_i, \mathbf{l}_i) \in \mathcal{X}} \log p(\mathbf{l}_i \mid \mathbf{y}_i),$$

where $\mathbf{y}_i$ is the network’s output for $I_i$; Baek et al. write the same objective for all their models. The gradient flows from the CTC loss through the recurrent layers into the convolutional layers, so the whole network is trained from images and strings alone. The official training loop divides the summed loss and the gradient by the batch size.

Data. The official configuration reads its training and validation sets from LMDB databases named synth90k_train_lmdb and synth90k_val_lmdb, the synthetic word images of Jaderberg et al. RARE states that its own model was trained only on the 8 million synthetic samples released by Jaderberg et al. and that CRNN used the same training set; Baek et al.’s Table 1 lists CRNN’s reported training data as MJSynth, which they describe as 8.9 million word images rendered with fonts, borders and shadows, background colouring, projective distortion, blending with real images and noise. Labels are mapped to 36 classes, upper- and lower-case letters to the same class (str2label), and the validation accuracy compares predictions with lower-cased ground truth.

Procedure. The configuration uses Adadelta with default settings, batches of 64 images, validation every 1,000 iterations and at most 2,000,000 iterations. The data loader samples each batch at random from the whole training set and scales every image to $100 \times 32$; it applies no data augmentation beyond the synthetic rendering itself. For their re-implementation Baek et al. used Adadelta with decay rate $\rho = 0.95$, batches of 192, 300,000 iterations, gradient clipping at magnitude 5 and He initialization, trained on MJSynth and SynthText together (14.4 million images).

Example: a Small CRNN Trained with CTC

The script below has two parts. Part A builds the official configuration in PyTorch, layer for layer from config.lua including the BiRnnJoin projections, and checks the sequence length and the parameter count; it also computes the receptive field of each frame. Part B trains a small CRNN on synthetic word images: words of three to seven random capital letters from the $5 \times 7$ bitmap font used in the optical character recognition article, each font cell drawn 3 pixels high and 2 wide (glyphs of $21 \times 10$ pixels), with gaps of −1 to 3 pixels between letters (−1 overlaps neighbouring glyphs by a column), a random position on a $32 \times 100$ canvas, a $3 \times 3$ box blur, random contrast and polarity, and Gaussian noise with standard deviation 0.1. The CNN reduces the image to 25 columns of 128 features. Two sequence stages are compared, the choice Baek et al. call BiLSTM against None: a two-layer bidirectional LSTM with 64 units per direction followed by a linear layer, and the linear layer alone applied to each column. Both are trained with torch.nn.CTCLoss for a fixed number of steps (Adam, learning rate 0.001, batches of 32, gradient clipping at 5) on 10,000 images and tested on 1,000 others, with best-path decoding and with a lexicon of 50 words per test image (the true word and 49 random words), decoded as the official code does by choosing the lexicon word with the highest CTC probability. Before training, the script runs a check of the task that uses no model: the smallest number of font cells in which two glyphs differ, and how often counting connected components of the clean rendering gives the wrong number of letters, which is how often a segmentation-based reader would fail at the first step.

The script takes the random seed and the number of training steps as arguments. It was run with PyTorch 2.14.1, NumPy 2.5.4 and SciPy 1.18.1 on four threads, with seeds 0, 1 and 2 and with 1,000 and 3,000 steps, each run twice with identical output. On our machine a run takes about two minutes with the default 1,000 steps and about six minutes with 3,000. The script was planned with 1,000 steps, which was the only setting of its first runs; the steps argument was added afterwards, as described below, and leaves the default output unchanged.

import sys
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from scipy import ndimage

seed = int(sys.argv[1]) if len(sys.argv) > 1 else 0
steps = int(sys.argv[2]) if len(sys.argv) > 2 else 1000
torch.set_num_threads(4)
torch.manual_seed(seed)

# ---- Part A: the official configuration (bgshih/crnn, model/crnn_demo/config.lua) in PyTorch ----
class BLSTMJoin(nn.Module):
    """One bidirectional LSTM layer; forward and backward outputs are each projected and summed (BiRnnJoin.lua)."""
    def __init__(self, n_in, n_hidden, n_out):
        super().__init__()
        self.lstm = nn.LSTM(n_in, n_hidden, bidirectional=True)
        self.fwd, self.bwd = nn.Linear(n_hidden, n_out), nn.Linear(n_hidden, n_out)
    def forward(self, x):                                   # x: (T, B, n_in)
        h, _ = self.lstm(x)
        f, b = h.chunk(2, dim=2)
        return self.fwd(f) + self.bwd(b)

def official_crnn(n_classes=36):
    nm, ks, ps = [64, 128, 256, 256, 512, 512, 512], [3, 3, 3, 3, 3, 3, 2], [1, 1, 1, 1, 1, 1, 0]
    bn = {2, 4, 6}                                          # BN after conv3, conv5, conv7 (0-based 2, 4, 6)
    pool_after = {0: (2, 2, 0), 1: (2, 2, 0), 3: ((2, 1), (0, 1)), 5: ((2, 1), (0, 1))}
    layers, c_in = [], 1
    for i in range(7):
        layers.append(nn.Conv2d(c_in, nm[i], ks[i], 1, ps[i]))
        if i in bn:
            layers.append(nn.BatchNorm2d(nm[i]))
        layers.append(nn.ReLU())
        if i in (0, 1):
            layers.append(nn.MaxPool2d(2, 2))
        elif i in (3, 5):                                   # 2x2 window, stride 2 in height, 1 in width, pad 1 in width
            layers.append(nn.MaxPool2d((2, 2), stride=(2, 1), padding=(0, 1)))
        c_in = nm[i]
    cnn = nn.Sequential(*layers)
    rnn = nn.Sequential(BLSTMJoin(512, 256, 256), BLSTMJoin(256, 256, n_classes + 1))
    return cnn, rnn

cnn, rnn = official_crnn()
with torch.no_grad():
    fmap = cnn(torch.zeros(1, 1, 32, 100))
    seq = fmap.squeeze(2).permute(2, 0, 1)                  # map-to-sequence: (W, B, C)
    out = rnn(seq)
n_params = sum(p.numel() for p in list(cnn.parameters()) + list(rnn.parameters()))
print("Part A: official configuration, input 1 x 32 x 100")
print(f"  feature map {tuple(fmap.shape[1:])} -> sequence of {seq.shape[0]} frames x {seq.shape[2]} -> "
      f"output {out.shape[0]} x {out.shape[2]} (36 classes + blank)")
print(f"  trainable parameters: {n_params:,}")

def receptive_field(layers):
    """layers: (kernel, stride, pad) along one axis; returns size, stride and centre of the first output unit."""
    r, j, start = 1, 1, 0.0
    for k, s, p in layers:
        start += ((k - 1) / 2 - p) * j
        r += (k - 1) * j
        j *= s
    return r, j, start

conv, conv2 = (3, 1, 1), (2, 1, 0)
width_layers = [conv, (2, 2, 0), conv, (2, 2, 0), conv, conv, (2, 1, 1), conv, conv, (2, 1, 1), conv2]
height_layers = [conv, (2, 2, 0), conv, (2, 2, 0), conv, conv, (2, 2, 0), conv, conv, (2, 2, 0), conv2]
rw, jw, sw = receptive_field(width_layers)
rh, jh, sh = receptive_field(height_layers)
print(f"  each frame sees a window {rw} px wide x {rh} px high; neighbouring frames are {jw} px apart;")
print(f"  window centres run from x = {sw:.1f} (frame 1) to x = {sw + 25 * jw:.1f} (frame 26) on a 100-px-wide image")

# ---- Part B: synthetic word images (5x7 font of the OCR article; each font cell 3 px high, 2 px wide) ----
FONT = """
A .###. #...# #...# ##### #...# #...# #...#
B ####. #...# #...# ####. #...# #...# ####.
C .###. #...# #.... #.... #.... #...# .###.
D ####. #...# #...# #...# #...# #...# ####.
E ##### #.... #.... ####. #.... #.... #####
F ##### #.... #.... ####. #.... #.... #....
G .###. #...# #.... #.### #...# #...# .###.
H #...# #...# #...# ##### #...# #...# #...#
I .###. ..#.. ..#.. ..#.. ..#.. ..#.. .###.
J ..### ...#. ...#. ...#. ...#. #..#. .##..
K #...# #..#. #.#.. ##... #.#.. #..#. #...#
L #.... #.... #.... #.... #.... #.... #####
M #...# ##.## #.#.# #.#.# #...# #...# #...#
N #...# #...# ##..# #.#.# #..## #...# #...#
O .###. #...# #...# #...# #...# #...# .###.
P ####. #...# #...# ####. #.... #.... #....
Q .###. #...# #...# #...# #.#.# #..#. .##.#
R ####. #...# #...# ####. #.#.. #..#. #...#
S .#### #.... #.... .###. ....# ....# ####.
T ##### ..#.. ..#.. ..#.. ..#.. ..#.. ..#..
U #...# #...# #...# #...# #...# #...# .###.
V #...# #...# #...# #...# #...# .#.#. ..#..
W #...# #...# #...# #.#.# #.#.# #.#.# .#.#.
X #...# #...# .#.#. ..#.. .#.#. #...# #...#
Y #...# #...# .#.#. ..#.. ..#.. ..#.. ..#..
Z ##### ....# ...#. ..#.. .#... #.... #####
"""
BITS, GLYPH = {}, {}
for row in FONT.strip().splitlines():
    ch, *rows = row.split()
    BITS[ch] = np.array([[c == "#" for c in r] for r in rows], float)
    GLYPH[ch] = np.kron(BITS[ch], np.ones((3, 2)))          # 21 x 10 pixels, ink = 1
ALPHA = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"                         # class k + 1; class 0 is the blank
H, W = 32, 100

def random_word(rng):
    return "".join(rng.choice(list(ALPHA), size=rng.integers(3, 8)))

def render(word, rng):
    n = len(word)
    gaps = rng.choice([-1, 0, 1, 2, 3], size=n - 1)          # -1: neighbouring glyphs overlap by one column
    total = 10 * n + gaps.sum()
    x, y = rng.integers(0, W - total + 1), rng.integers(0, H - 21 + 1)
    clean = np.zeros((H, W))
    for i, ch in enumerate(word):
        clean[y:y + 21, x:x + 10] = np.maximum(clean[y:y + 21, x:x + 10], GLYPH[ch])
        x += 10 + (gaps[i] if i < n - 1 else 0)
    p = np.pad(clean, 1, mode="edge")
    blur = sum(p[i:i + H, j:j + W] for i in range(3) for j in range(3)) / 9
    img = rng.uniform(0.5, 1.0) * blur                      # contrast between ink and background
    if rng.integers(2):
        img = 1 - img                                       # dark text on a light background
    img = img + rng.normal(0, 0.1, img.shape)
    return ((img - 0.5) / 0.5).astype(np.float32), clean

def make_set(n, rng):
    words = [random_word(rng) for _ in range(n)]
    pairs = [render(w, rng) for w in words]
    return words, np.stack([p[0] for p in pairs])[:, None], [p[1] for p in pairs]

train_words, Xtr, _ = make_set(10000, np.random.default_rng([seed, 1]))
test_words, Xte, test_clean = make_set(1000, np.random.default_rng([seed, 2]))
rng_lex = np.random.default_rng([seed, 3])
lexicons = [[w] + [random_word(rng_lex) for _ in range(49)] for w in test_words]

# ---- Label-free difficulty check ----
names = list(BITS)
d = {(a, b): int((BITS[a] != BITS[b]).sum()) for i, a in enumerate(names) for b in names[i + 1:]}
dmin = min(d.values())
print(f"\nPart B data: min Hamming distance between 5x7 glyphs = {dmin} cells, for "
      + ", ".join(f"{a}/{b}" for (a, b), v in d.items() if v == dmin))
n_cc = [ndimage.label(c > 0.5, structure=np.ones((3, 3)))[1] for c in test_clean]
wrong = np.mean([k != len(w) for k, w in zip(n_cc, test_words)])
print(f"  connected components != number of letters in {100 * wrong:.1f}% of the 1,000 clean test words")

# ---- A small CRNN ----
class SmallCRNN(nn.Module):
    def __init__(self, use_lstm, n_classes=27):
        super().__init__()
        self.cnn = nn.Sequential(
            nn.Conv2d(1, 32, 3, 1, 1), nn.ReLU(), nn.MaxPool2d(2, 2),               # 16 x 50
            nn.Conv2d(32, 64, 3, 1, 1), nn.ReLU(), nn.MaxPool2d(2, 2),              # 8 x 25
            nn.Conv2d(64, 96, 3, 1, 1), nn.BatchNorm2d(96), nn.ReLU(), nn.MaxPool2d((2, 1)),  # 4 x 25
            nn.Conv2d(96, 96, 3, 1, 1), nn.ReLU(), nn.MaxPool2d((2, 1)),            # 2 x 25
            nn.Conv2d(96, 128, (2, 3), 1, (0, 1)), nn.BatchNorm2d(128), nn.ReLU())  # 1 x 25
        self.lstm = nn.LSTM(128, 64, num_layers=2, bidirectional=True) if use_lstm else None
        self.out = nn.Linear(128, n_classes)
    def forward(self, x):
        f = self.cnn(x).squeeze(2).permute(2, 0, 1)          # map-to-sequence: (T=25, B, 128)
        if self.lstm is not None:
            f, _ = self.lstm(f)
        return self.out(f).log_softmax(2)                    # (T, B, 27)

def encode(words):
    tgt = torch.tensor([ALPHA.index(c) + 1 for w in words for c in w])
    return tgt, torch.tensor([len(w) for w in words])

def best_path(logp):                                         # logp: (T, B, C)
    raw = logp.argmax(2).T.numpy()
    words, raws = [], []
    for r in raw:
        words.append("".join(ALPHA[k - 1] for t, k in enumerate(r) if k != 0 and (t == 0 or k != r[t - 1])))
        raws.append("".join("-" if k == 0 else ALPHA[k - 1] for k in r))
    return words, raws, raw

ctc = nn.CTCLoss(blank=0, reduction="mean")
ctc_each = nn.CTCLoss(blank=0, reduction="none")
results = {}
for name, use_lstm in [("BiLSTM", True), ("None", False)]:
    torch.manual_seed(seed)
    model = SmallCRNN(use_lstm)
    opt = torch.optim.Adam(model.parameters(), lr=1e-3)
    rng_b = np.random.default_rng([seed, 4])
    order, pos, block, curve = rng_b.permutation(len(Xtr)), 0, [], []
    model.train()
    for step in range(steps):
        if pos + 32 > len(order):
            order, pos = rng_b.permutation(len(Xtr)), 0
        idx = order[pos:pos + 32]; pos += 32
        tgt, tl = encode([train_words[i] for i in idx])
        logp = model(torch.from_numpy(Xtr[idx]))
        loss = ctc(logp, tgt, torch.full((32,), logp.shape[0], dtype=torch.long), tl)
        opt.zero_grad(); loss.backward()
        nn.utils.clip_grad_norm_(model.parameters(), 5.0)
        opt.step()
        block.append(loss.item())
        if len(block) == steps // 10:
            curve.append(np.mean(block)); block = []
    model.eval()
    with torch.no_grad():
        logp = model(torch.from_numpy(Xte))
        pred, raws, raw = best_path(logp)
        tgt, tl = encode(test_words)
        test_loss = ctc(logp, tgt, torch.full((1000,), 25, dtype=torch.long), tl).item()
        lex_pred = []
        for i0 in range(0, 1000, 100):                       # score every lexicon word by its CTC loss
            lp = logp[:, i0:i0 + 100].repeat_interleave(50, dim=1)
            words = [w for lex in lexicons[i0:i0 + 100] for w in lex]
            t, l = encode(words)
            nll = ctc_each(lp, t, torch.full((len(words),), 25, dtype=torch.long), l).view(100, 50)
            lex_pred += [lexicons[i0 + i][j] for i, j in enumerate(nll.argmin(1).tolist())]
    n_par = sum(p.numel() for p in model.parameters())
    acc = np.mean([p == w for p, w in zip(pred, test_words)])
    acc_lex = np.mean([p == w for p, w in zip(lex_pred, test_words)])
    print(f"\nsequence stage {name}: {n_par:,} parameters")
    print(f"  mean training CTC loss per {steps // 10} steps: " + " ".join(f"{c:.2f}" for c in curve))
    print(f"  test CTC loss {test_loss:.3f} | word accuracy: lexicon-free {100 * acc:.1f}%, "
          f"50-word lexicon {100 * acc_lex:.1f}% | frames predicted blank {100 * np.mean(raw == 0):.1f}%")
    for i in range(3):
        print(f"    truth {test_words[i]:<8} raw {raws[i]}  ->  {pred[i]}")

Output of python crnn_example.py 0:

Part A: official configuration, input 1 x 32 x 100
  feature map (512, 1, 26) -> sequence of 26 frames x 512 -> output 26 x 37 (36 classes + blank)
  trainable parameters: 8,331,594
  each frame sees a window 54 px wide x 86 px high; neighbouring frames are 4 px apart;
  window centres run from x = -0.5 (frame 1) to x = 99.5 (frame 26) on a 100-px-wide image

Part B data: min Hamming distance between 5x7 glyphs = 2 cells, for D/O
  connected components != number of letters in 80.8% of the 1,000 clean test words

sequence stage BiLSTM: 433,691 parameters
  mean training CTC loss per 100 steps: 4.48 3.62 3.61 3.59 3.56 3.41 3.12 2.50 1.66 0.97
  test CTC loss 0.787 | word accuracy: lexicon-free 32.8%, 50-word lexicon 99.9% | frames predicted blank 80.0%
    truth CHKFPXD  raw -D--H--K--P-PP-Y-------O-  ->  DHKPPYO
    truth CBIYD    raw C--B--I-----------------O  ->  CBIO
    truth JBMKGKM  raw --J-B--M--K--B-K-------X-  ->  JBMKBKX

sequence stage None: 235,035 parameters
  mean training CTC loss per 100 steps: 5.45 2.82 0.77 0.15 0.07 0.04 0.03 0.02 0.02 0.02
  test CTC loss 0.052 | word accuracy: lexicon-free 94.0%, 50-word lexicon 100.0% | frames predicted blank 77.7%
    truth CHKFPXD  raw ----C-H--K--F-PP--X--D---  ->  CHKFPXD
    truth CBIYD    raw --C-B--I-Y--D------------  ->  CBIYD
    truth JBMKGKM  raw --J---B--M--K--G--K--M---  ->  JBMKGKM

Part A confirms the shapes in the official configuration’s comments: 26 frames of 512 features, and 8,331,594 parameters, the 8.3 million that Baek et al. list. Each frame’s window, 54 pixels wide and taller than the image, overlaps its neighbours by 50 pixels, so a frame is not a slice of one character but a view of a character with its neighbours; at the borders the windows extend into the padding. The label-free check shows what the synthetic task demands. The closest glyphs, D and O, differ in only two of 35 font cells, so the classes are learnable but not trivially separable under blur and noise; and counting connected components gives the wrong number of letters in 80.8% of the clean words, because overlapping and touching letters merge, so a reader that segments first would fail on most of the test set before classifying anything.

The two trained models show what CTC learns from transcriptions alone. Their raw frame labels are mostly blanks, 77.7–80.0% of the frames once trained, with each letter emitted as a spike of one or two frames, the same pattern as the official README’s a-----v--a-i-l-a-bb-l-e---; no frame was ever labelled during training. The per-column model without a recurrent layer, a classifier applied to each of the 25 columns, leaves the initial plateau of the loss within 300–400 steps and reads 94.0% of the test words with seed 0 (98.0% and 98.3% with seeds 1 and 2). The plan, written before the first run, expected that the BiLSTM model “reaches >= 90% lexicon-free word accuracy” and that the per-column model would be “lower than BiLSTM”; the reverse happened. With seed 0 its loss stays near 3.6 for about 600 steps before it falls, and at 1,000 steps it reads 32.8% of the words, and in the three printed examples it misreads the last letters; with seeds 1 and 2 it never leaves the plateau within 1,000 steps and reads no word at all; in their printed examples it emits one to three letters, mostly near the start of the image. The plateau value, about 3.5, is per character, since PyTorch’s mean reduction divides each word’s loss by its length; it is a little above $\ln 26 \approx 3.26$, the cost of a uniform guess among the 26 letters, so in our reading a network on the plateau has learned to emit mostly blanks but little about which letters are present, which the chance-level lexicon results below also suggest.

Because the BiLSTM model was still on or just past the plateau, a second question was added after these results: does it catch up when trained three times longer? The steps argument was added for this, and each printed loss average was made to cover a tenth of the run (100 steps by default, as before), with the settings fixed before the runs (3,000 steps, everything else unchanged, both models retrained because the script trains both), and the expectation, in the plan’s words, that the BiLSTM “leaves the plateau for all three seeds within 3000 steps and ends within ~3 points of the None model”. The output of python crnn_example.py 0 3000, from the first sequence stage on (Part A and the data check print the same lines as above):

sequence stage BiLSTM: 433,691 parameters
  mean training CTC loss per 300 steps: 3.90 3.52 2.43 0.61 0.15 0.07 0.03 0.03 0.02 0.02
  test CTC loss 0.025 | word accuracy: lexicon-free 96.7%, 50-word lexicon 100.0% | frames predicted blank 79.4%
    truth CHKFPXD  raw --C-H--K--F--P--X------D-  ->  CHKFPXD
    truth CBIYD    raw C--B--I-Y---------------D  ->  CBIYD
    truth JBMKGKM  raw --J-B--M--K--G--K------M-  ->  JBMKGKM

sequence stage None: 235,035 parameters
  mean training CTC loss per 300 steps: 3.01 0.09 0.02 0.02 0.01 0.01 0.01 0.01 0.00 0.00
  test CTC loss 0.005 | word accuracy: lexicon-free 99.2%, 50-word lexicon 100.0% | frames predicted blank 77.6%
    truth CHKFPXD  raw ----C-H--K--F-PP--X--D---  ->  CHKFPXD
    truth CBIYD    raw --C-B--I-Y--D------------  ->  CBIYD
    truth JBMKGKM  raw --J---B--M--K--G--K--M---  ->  JBMKGKM

With seed 0 the BiLSTM model now leaves the plateau after about 600 steps and reads 96.7% of the words, 2.5 points below the per-column model’s 99.2%. With seed 2 it leaves the plateau only after 1,000–1,500 steps and reaches 84.4%; with seed 1 its loss starts to fall only in the last 300 steps, and it still reads no word (the per-column model reaches 98.4% and 98.7% with these seeds). The expectation therefore held for one seed of three. In this small setting, the recurrent layers mainly delay learning. In our reading, the per-column model’s results show that the convolutional features alone suffice to identify a letter of this font, so context adds little, while the recurrent path must first learn to carry each column’s evidence to the output before the CTC alignment can form. This is our reading of the example, not a finding of the CRNN paper or of Baek et al., who trained for 300,000 iterations on real-world-like synthetic text and found that adding the BiLSTM to the VGG-CTC model raised total accuracy from 69.5% to 78.4%. The example shows that the benefit of the sequence stage is not automatic and that with a short schedule the random seed, which here sets the initialization, the training data and the batch order, decided whether this small recognizer trained at all.

The lexicon results separate reading from choosing. Wherever a model has learned anything, the 50-word lexicon brings accuracy to 99.9–100%, even for the seed-0 BiLSTM at 1,000 steps, which reads only a third of the words without it: the true word needs only a higher CTC probability than 49 random words, which partial evidence about a few letters and the length usually gives. For the models stuck on the plateau, lexicon decoding is at chance (1.6% and 2.2% against 2% for one word in 50), and the seed-1 model after 3,000 steps, which has begun to place some letters, chooses correctly in 29.5% of cases. The lexicons of IIIT5K are built in the same way, from the true word and random words, which helps explain why CRNN’s lexicon-based results in the Performance section are so much higher than its lexicon-free ones.

Inference

Lexicon-free decoding. The released code decodes without a lexicon by best path, which its source calls naive decoding: the most probable label at each of the 26 frames, then the CTC collapse (merge repeats, drop blanks). The README shows the raw frame labels next to the result, for example a-----v--a-i-l-a-bb-l-e--- read as “available”: each character occupies one or two frames between blanks, and the double “b” collapses to one because no blank separates it. Best path returns the most probable frame sequence, not necessarily the most probable word; the CTC article describes the prefix search alternatives.

Lexicon-based decoding. When a test image comes with a lexicon, the released code (decodingWithLexicon) computes $p(w \mid \mathbf{y})$ for every word $w$ in the lexicon with the CTC forward algorithm and returns the most probable word. Its cost grows linearly with the lexicon size, which is negligible for the 50-word and 1,000-word lexicons of the benchmarks but not for a 50,000-word dictionary. The released code has no faster search for large lexicons, and the paper’s own procedure for them could not be read. Two later papers by or about the same group describe other speed-ups: RARE, whose attention decoder outputs one character per step, searches a prefix tree built over the lexicon with a beam, and uses exhaustive search for lexicons of up to 1,000 words; Bai et al. (2018) write that Shi et al. used a prefix tree (trie) to accelerate lexicon search, citing RARE and the original description of the trie, not CRNN.

Speed and use in other systems. In Baek et al.’s re-implementation on an NVIDIA Tesla P40, CRNN took 4.4 ms per image, against 1.3 ms for the same model without the BiLSTM; their Table 1 also lists 160 ms per image as the time reported by the original paper, a figure not measured on their platform and so not comparable with theirs. Because CTC gives a probability for any string, the recognizer can score candidates as well as read them: TextBoxes (Liao et al., 2017) used the released CRNN model to rescore detected word boxes by $\max_{w \in W} p(w \mid I)$ over a lexicon $W$, or over a generic 90,000-word lexicon when none is given, which removed false detections that do not read as words.

Performance

Benchmarks. The results that RARE tabulates for CRNN, cited from the 2015 arXiv version, are word accuracies on four benchmarks of cropped words, as RARE describes them. IIIT 5K-Words (IIIT5K) has 3,000 test images collected from the Internet, each with a 50-word and a 1,000-word lexicon made of the true word and randomly picked words. Street View Text (SVT) has 647 test word images from Google Street View, many noisy, blurred or of low resolution, each with a 50-word lexicon. ICDAR 2003 (IC03) has 860 test words after discarding words with non-alphanumeric characters or fewer than three characters, each with a 50-word lexicon defined by Wang et al., plus a “full” lexicon of all the per-image lexicon words and the 50,000-word Hunspell lexicon. ICDAR 2013 (IC13) inherits most of its images from IC03 and has, by RARE’s count, 857 words after the same filtering. The ICDAR article describes the competitions these sets come from.

Benchmark (lexicon)	Word accuracy, %
IIIT5K (50 / 1k / none)	97.6 / 94.4 / 78.2
SVT (50 / none)	96.4 / 80.8
IC03 (50 / full / 50k / none)	98.7 / 97.6 / 95.5 / 89.4
IC13 (none)	86.7

Among the earlier methods in RARE’s Table 1, CRNN is the only one with a lexicon-free result on IIIT5K and the best without a lexicon on SVT; on IC03 it is below the 90,000-word classifier of Jaderberg et al., which recognizes only dictionary words (93.1), and, by 0.2 points, below the same group’s network with a structured output layer (89.6), and on IC13 below that classifier (90.8) and Bissacco et al. (87.6). With lexicons, CRNN matched or exceeded every earlier method in the table except that classifier’s 98.6 on IC03 with the full lexicon. Baek et al. list the model at 8.3 million parameters, and the official configuration built in the example has 8,331,594.

Irregular text. RARE tested the CRNN model on two sets of distorted text, alongside its own synthetic-trained model, which it used without fine-tuning. On SVT-Perspective, words photographed from side views in Google Street View (639 test images by RARE’s count, 645 by Baek et al.’s), it scored 92.6% with the 50-word lexicons, 72.6% with the full lexicon and 66.8% without a lexicon, against 91.2%, 77.4% and 71.8% for RARE; on CUTE80, 288 word images that RARE cropped from 80 high-resolution photographs of mostly curved text, it scored 54.9% without a lexicon against RARE’s 59.2%. RARE attributes the gap to CRNN having no mechanism for perspective or curved text.

Re-evaluation under a common protocol. Baek et al. (2019) found that published scene text results were not comparable: papers trained on different synthetic sets and evaluated on different versions of the benchmarks (IC03 with 860 or 867 words, IC13 with 857 or 1,015, IC15 with 1,811 or 2,077), and they estimated that seven missing IC03 words alone shift accuracy by 0.8 points. Their Table 1 places CRNN’s reported 86.7% on IC13 in the 1,015-word column, whereas RARE, which reports the same number, describes its IC13 set as 857 words. They re-trained CRNN and five later models on MJSynth plus SynthText and evaluated each on all versions, averaging five runs with different random seeds. CRNN scored 82.9% on IIIT5K, 81.6% on SVT, 93.1% and 92.6% on the two IC03 versions, 91.1% and 89.2% on the two IC13 versions, 69.4% and 64.2% on the two ICDAR 2015 versions, 70.0% on SVT-Perspective and 65.5% on CUTE80, or 78.4% over all 8,539 test images; their best model, which adds a thin-plate-spline rectification stage, a ResNet feature extractor and an attention decoder, reached 84.0% at 27.6 ms per image. As a check of their platform they also trained CRNN on MJSynth alone, following the original pipeline, and obtained 81.2% on IIIT5K, 81.8% on SVT, 91.1% on IC03 (860) and 87.6% on IC13 (1,015), slightly above the originally reported numbers.

These are results as published; current results on the benchmarks are tracked by the competitions’ own leaderboards, linked from the ICDAR article.

Strengths

• No character segmentation or character labels. The network is trained from word images and their transcriptions alone, and reads touching or overlapping characters that defeat segmentation-based methods.
• Open vocabulary. Lexicon-free decoding outputs any string of the alphabet, unlike whole-word classifiers; RARE makes this point in contrasting CRNN with the 90,000-class classifier of Jaderberg et al.
• One model for constrained and unconstrained reading. The same per-frame distributions give a free reading by best path and a constrained one by scoring lexicon words, and the scores can rank detections, as in TextBoxes.
• Small and fast. About 8.3 million parameters, most of them in the convolutional layers (Baek et al. count 5.6 million for the model without its BiLSTM, None-VGG-None-CTC), and 4.4 ms per image in Baek et al.’s measurement.
• A simple, decomposable design. Its stages are the reference point of Baek et al.’s four-stage framework, in which each stage can be swapped independently.

Limitations

• Horizontal, roughly straight text. The map-to-sequence step assumes that characters follow the image columns from left to right. RARE’s results on SVT-Perspective and CUTE80 show the loss on perspective and curved text, and Baek et al. list vertical text among the failures shared by all 24 models they tested.
• Fixed input size and sequence length. The released code resizes every crop to $100 \times 32$ and unrolls the recurrent layers for 26 frames, so long words are squeezed and at most 26 frames are available for a word’s characters and the blanks between repeats.
• CTC’s conditional independence. Given the image, the frame labels are predicted independently, so the model learns no explicit character-level language model; Baek et al. attribute part of the advantage of attention decoders to their implicit language model, which recovered occluded characters in their examples. The CTC article discusses this assumption and the alternatives.
• Restricted alphabet in the released model. The official configuration recognizes 36 case-insensitive alphanumeric classes. Baek et al. note that benchmarks do not evaluate special characters, so models trained without them mistake them for alphanumerics.
• Training dynamics. In the example above, the bidirectional LSTM made training slower to leave CTC’s initial plateau than a per-column classifier, at least for a small model and a short schedule; this is an observation on that example, not a result reported for CRNN.

Historical Importance

CRNN appeared as an arXiv preprint in 2015 and in TPAMI in 2017. In RARE’s account of the field it belongs to the top-down methods that read a word as a character sequence, and it replaced the hand-designed descriptors of earlier sequence readers with convolutional features trained jointly with the recurrent layers and the CTC output. Its combination was taken up by many later recognizers. RARE, by the same group a year later, built its sequence recognition network on a CRNN-style encoder, “following” CRNN in its words, and replaced CTC by an attention decoder after a learned rectification. Baek et al. named CRNN as the first in a line of variants that added rectification modules, stronger feature extractors and attention decoders, and their framework, which decomposes a recognizer into transformation, feature extraction, sequence modelling and prediction, is in effect CRNN’s pipeline with a rectification stage added in front. CRNN remained a standard row in the comparison tables of later papers, from RARE (2016) to AON and Edit Probability (both 2018), and is one of the six earlier models that Baek et al. re-trained under one protocol.

Variants

• RARE (Shi et al., CVPR 2016) adds a spatial transformer network that predicts a thin-plate-spline transformation from 20 fiducial points to rectify the input, and decodes the CRNN-style sequence with a GRU attention decoder instead of CTC. Trained on the same synthetic data, it raised lexicon-free accuracy on IIIT5K from CRNN’s 78.2% to 81.9%.
• ASTER (Shi, Yang, Wang, Lyu, Yao and Bai, TPAMI 2019, published online in 2018), the same group’s attentional recognizer with flexible rectification, continues RARE’s line; the scene text recognition article describes it from its official code.
• Other CTC recognizers in Baek et al.’s framework replace one stage of CRNN: GRCNN uses a gated recurrent convolutional feature extractor, Rosetta removes the BiLSTM to save computation, and STAR-Net adds rectification and a ResNet feature extractor. Among the 24 combinations Baek et al. evaluated, removing CRNN’s BiLSTM (None-VGG-None-CTC) lowered total accuracy from 78.4% to 69.5%, and the combination with rectification and ResNet but still CTC prediction reached 82.9%.
• Other image sequences. The paper’s title presents scene text as one application of image-based sequence recognition. Xiang Bai’s publications page lists, next to the paper and its code, a music score recognition dataset (as of November 2026), which points to a second application in the paper; that experiment could not be read for this article.

Later recognizers moved from CTC to attention and then to transformer decoders, and from synthetic to real training data; the scene text recognition article follows that line and compares CRNN with its successors under common protocols. For lines of handwriting, systems in handwriting recognition use the same design of a convolutional front end, a BLSTM and CTC.

Implementations

• Official code (bgshih/crnn, MIT licence): Torch7 with a C++ CTC implementation, tested only on Ubuntu 14.04 with CUDA GPUs according to its README, with a demo, a pretrained model hosted on Dropbox, functions for lexicon-free and lexicon-based recognition, a Dockerfile and a script that builds LMDB training sets. The last commit is of May 2018 (as of November 2026).
• PyTorch port. The official README points to a PyTorch port by a third party, meijieru/crnn.pytorch, added in May 2017.
• Re-implementation in a benchmark framework. Baek et al. released their framework (clovaai/deep-text-recognition-benchmark), in which CRNN is the None-VGG-BiLSTM-CTC combination.
• Building blocks. The parts are standard layers in current libraries: PyTorch provides torch.nn.LSTM with bidirectional=True and torch.nn.CTCLoss, as used in the example.