Handwriting Recognition

Handwriting recognition turns handwritten text into machine-readable characters. The input is either the movement of a pen, recorded as it writes, or an image of finished writing, such as a scanned letter, a form or a page from an archive. The output is the text, from a single digit or word up to whole lines, paragraphs and pages. The task is also called handwritten text recognition (HTR), the name used by the research competitions on historical documents, and it is the handwritten branch of optical character recognition, which covers the general OCR pipeline, the evaluation metrics and printed and scene text.

Handwriting is harder than print because no two writers form letters alike and cursive letters run into each other, so the characters cannot be reliably cut apart before they are recognized. Graves et al. (2009) describe the circular dependency, segmentation needing recognition and recognition needing segmentation, as what is sometimes called Sayre’s paradox, and the history of the task is largely a history of ways around it. Hidden Markov models, applied to handwriting after their success in speech recognition, segment and recognize at the same time by decoding sequences of frame features. Recurrent networks trained with connectionist temporal classification (CTC) replaced them, and transformer decoders and attention models later took on whole paragraphs and the language model as well. Throughout, lexicons and language models decide much of the result, and benchmark protocols decide how far published numbers can be compared.

Problem Definition

Given a recording of handwriting, the system outputs the sequence of characters that was written. Graves et al. (2009) divide the task in three ways: online against offline, by what is recorded; isolated characters or words against whole lines of text; and constrained writing, for example hand-printed characters only, against unconstrained writing. They add that the results obtained for digits and isolated characters “have never been matched” for complete lines.

The second part of the definition is the vocabulary. Vinciarelli, Bengio and Bunke (2003; read in their IDIAP research report, which appeared in IEEE TPAMI in 2004) contrast unconstrained text with the applications on which offline cursive recognition research had focused. Legal amounts on bank checks use a closed lexicon of 20–30 words and postal address lines one of 10 to 1,000 words, and in both every word that can occur is in the lexicon. In unconstrained text the only safe assumption is the language. A lexicon must then be built without looking at the test data and will never contain every word that occurs, so some words are out of vocabulary (OOV). Their report treats recognition as a search for the most probable word sequence $\hat{W}$ given the observation sequence $O$ extracted from the image,

$$\hat{W} = \arg\max_{W} p(O \mid W)\, p(W),$$

where $p(O \mid W)$ comes from the recognizer’s model of the writing and $p(W)$ from a statistical language model; for single words with a uniform prior over the lexicon, the second factor drops out. The same split into an optical model and a language model runs through HMM and CTC systems, and the CTC article gives the version used for CTC networks.

Input and Output

Online handwriting is a time series. In the IAM On-Line Handwriting Database used by Graves et al. (2009), text was written on a whiteboard with the eBeam system: a pen in a special casing sends infrared signals to a receiver in a corner of the board, which records the $(x, y)$ position of the pen tip with a time stamp, at 30 to 70 frames per second, while the pen touches the board. Each period of contact is a stroke. After preprocessing, each time step of their recognizer had 25 input features.

Offline handwriting is an image. The IAM and RIMES pages are gray-scale scans at 300 dpi, and the READ 2016 pages are in color (Coquenet et al., 2023). Most recognizers are given a cropped line image; paragraph and page recognizers receive the whole text block or page.

Output. The output is the transcription, usually with case, digits, punctuation and spaces: the IAM transcriptions contain 81 distinct symbols, including a garbage symbol (Graves et al., 2009). Page-level systems also output where each line is, and the historical datasets of the READ competitions store pages in the PAGE format, which gives each line a baseline, a bounding polygon and its transcript (Sánchez et al., 2016).

Variants

• Online and offline recognition. Graves et al. (2009) note that online recognition generally gives better results because relevant features are easier to extract, and Graves and Schmidhuber (2008) put it the other way round: offline recognition is observed to be harder, since online systems can use both the trajectory and the image while offline systems see only the image. The two settings use the same recognition machinery after feature extraction.
• Characters, words, lines, paragraphs and pages. Isolated characters are a classification problem, as on MNIST and EMNIST. Word recognition chooses from a lexicon, often a closed one, as in the IFN/ENIT town-name task below. Line recognition must also find the word boundaries, and paragraph and page recognition must find and order the lines.
• Writer-dependent and writer-independent recognition. A system can be trained and tested on the same writers or tested on writers it has never seen. Vinciarelli et al. (2003) ran experiments on both single-writer and multiple-writer data, and the IAM line benchmark keeps each writer in one set only.
• Modern and historical documents. The ICFHR and ICDAR competitions of the tranScriptorium and READ projects use manuscripts written centuries ago, with several hands per collection, marginal notes, crossed-out text and hyphenated words.
• Scripts and languages. Most benchmarks below are in English, French or German; the IFN/ENIT words are Arabic, written right to left with 120 distinct character labels in the transcriptions used by Graves and Schmidhuber (2008).

Challenges

• Writer variation. Writing styles differ greatly in skew, slant, height and width of characters (Graves et al., 2009), and on a whiteboard the baseline is usually not straight and the characters shrink as the pen moves right.
• Segmentation. Cursive letters are joined, so characters, and often words, cannot be cut apart before recognition. In the READ 2016 competition, a line read with a character error rate of 4.2% had a word error rate of 50% because two words were run together, and the organizers attribute the large gaps between CER and WER to wrong word segmentation (Sánchez et al., 2016).
• Delayed strokes. The dot of an i or the bar of a t is often added after the word, in an order that varies between writers, which is a known problem for online recognition (Graves et al., 2009).
• Open vocabulary. A lexicon helps only for the words it contains. In READ 2016, about 15% of the running words of the test set were out of vocabulary, and the organizers estimated that this alone accounts for about 30% WER under a word-based language model, because each OOV word costs about two errors: the word itself and the next one, through the n-gram context (Sánchez et al., 2016).
• Line segmentation and layout. Line recognizers need cropped lines, and automatic line segmentation adds errors: Bluche (2016) found that applying line recognizers to automatically segmented IAM and RIMES paragraphs raised the character error rate over ground-truth lines in every case, by 0.9 to 7.2 points in his Table 2 (the text says by an absolute 1% in the best case).
• Small marks and degraded pages. Punctuation is much smaller than letters and is easily missed (Bluche, 2016); historical pages add bleed-through, faded writing, skewed images and slanted lines (Sánchez et al., 2016).

Approaches

Normalization and features

Before learned features, recognizers removed as much writer variation as possible. Graves et al. (2009) describe the online preprocessing used for both their HMM and their network. Whiteboard lines are split at gaps wider than the median gap, and each part is corrected for skew by a linear regression through its points. Slant is estimated from a histogram of the angles between successive trajectory points, weighted by a Gaussian centred on the vertical and smoothed. Delayed strokes, identified as strokes above already written parts followed by a pen movement to the right, are removed and replaced by a feature that marks their removal. The points are resampled to equal spacing to remove differences in writing speed, and the text is scaled between its baseline and corpus line, two regressions through the minima and the maxima of the trajectory. The offline counterpart (deskewing, deslanting, normalizing the heights of the ascender, body and descender zones, and a sliding window of geometric features) is described in the optical character recognition article.

Hidden Markov models

HMM recognizers model a line as a sequence of frames. Vinciarelli et al. (2003) describe a typical offline system. After normalization, a window shifts column by column from left to right; at each position the inked part of the window is divided into a $4 \times 4$ grid, and the fraction of the window’s foreground pixels that falls in each cell gives a 16-dimensional feature vector. Each character is modelled by a continuous-density HMM whose emission probabilities are mixtures of Gaussians, words are concatenations of character models, and all letter models share the same number of states and of Gaussians per state, chosen on validation data. Upper- and lower-case versions of a letter share one model, and punctuation and digits are treated as noise. The models are trained with the Baum–Welch algorithm and decoded with the Viterbi algorithm, which returns the best state sequence and with it the word segmentation, so the system never has to cut the line into words beforehand. The language model, unigrams, bigrams or trigrams, is applied during decoding rather than to the finished transcription, which the report presents as an important novelty compared with earlier systems that recognized segmented words and then applied a syntactic post-processing step. With a 50,000-word lexicon, the language model reduced the error rate by about 50% on single-writer data and about 25% on multiple-writer data (abstract).

The HMM baseline of Graves et al. (2009) used one linear HMM per character, with 8 states per character for online data and a number of states chosen per character for offline data, and mixtures of 32 and 12 diagonal Gaussians; the language model weight and a word insertion penalty were tuned on the validation set. HMMs remained the baselines of the competitions: the organizers of the 2014 and 2016 HTR competitions supplied systems built with the HTK and SRILM toolkits, and the READ 2016 baseline had a word error rate of 56.1% (Sánchez et al., 2014, 2016). Graves and Schmidhuber (2008) name the drawback they set out to remove: the input features must suit the independence assumptions of HMMs, so in practice they are redesigned for every alphabet and, to a lesser extent, every language.

Recurrent networks with CTC: BLSTM and MDLSTM

BLSTM on feature sequences. Graves et al. (2009) fed the same online and offline feature sequences as the HMM to a bidirectional LSTM network with a CTC output layer, and decoded its outputs with a dictionary and a bigram language model by CTC token passing. Their network had 100 LSTM memory blocks in each direction and an output layer of 81 units (80 characters and the blank), and it outperformed the HMM on both the online and the offline IAM data; the CTC article gives these results with the versions they come from. The same paper measured how the dictionary changes the result, on the online data and, for the network, without the language model (Table III). With open dictionaries of the 5,000 to 30,000 most frequent words of three text corpora, the share of test words covered rose from 86.1% to 95.7%, and the network’s word accuracy from 66.2% to 75.0%, while the HMM’s rose from 47.7% at 5,000 words to 65.0% at 25,000 and then fell to 61.5% at 30,000. A closed dictionary of the 5,597 test-set words gave 85.3% for the network and 70.1% for the HMM, and padding it to 20,000 words lowered these to 81.5% and 68.8% (Table IV). The ranking of the two systems did not change with the dictionary, but every number depends on it.

MDLSTM on raw pixels. Graves and Schmidhuber (2008) removed the hand-designed features. A multidimensional recurrent network has one recurrent connection per dimension of the data, so in an image the hidden state at a pixel depends on the states one step up and one step to the left, together with the input; four such layers, each scanning from a different corner, give every position context from all directions, the two-dimensional analogue of a bidirectional network. In multidimensional LSTM (MDLSTM), each memory cell has one forget gate per dimension. To turn the image into the one-dimensional sequence that CTC needs, the network is a hierarchy: the image is cut into small blocks of pixels (3 wide by 4 high in the paper’s figure), four MDLSTM layers scan the blocks, their activations are gathered into blocks and passed to a feedforward layer, and the process repeats, three stages in most tasks. The final activations are summed over each column, giving one vector per horizontal position for the CTC layer. The authors present the system as needing no alphabet-specific preprocessing, so the same network could be applied unchanged to any language.

They tested it on the ICDAR 2007 Arabic handwriting recognition competition, too late to enter but evaluated by the organizers on the unpublished test sets. The task was to identify the postcode of a handwritten Tunisian town or village name from the IFN/ENIT database: 32,492 public images (30,000 used for training and 2,492 for validation) from forms filled in by more than 400 people, 120 character labels, and a dictionary of 937 names with 1,518 entries including spelling variants; the CTC outputs were constrained to single dictionary words, and up to ten answers were ranked. On the main test set “f”, the network identified 91.43% of the postcodes with its first answer, against 87.22% for the best competition entry, SIEMENS-2; on set “s”, collected in the United Arab Emirates to test robustness to regional writing styles, it scored 78.83% against 73.94% (Table 1). The network had 159,369 weights. A footnote records that the authors first presented the Arabic transcriptions backwards, left to right, and that the network still reached a character error rate of 17.8%, against 10.7% with the correct order.

Convolutional front ends. Later systems interleaved MDLSTM layers with convolutions (Bluche, 2016) or replaced the recurrent front end by a convolutional neural network followed by a BLSTM and CTC, the design of CRNN. In the READ 2016 competition the organizers remark that all entrants used “CNN/CTC techniques” to train their optical models: the winner, from RWTH Aachen, combined 16 multidirectional LSTM networks with a 10-gram character language model, and A2iA used a multidirectional LSTM network with a hybrid word and character language model (Sánchez et al., 2016).

Paragraph and page recognition

Line recognizers need line segmentation, for training as well as for reading. Bluche (2016) replaced the final collapse of an MDLSTM recognizer, the sum over each column, by a weighted sum whose weights an attention network recomputes at every step. Each step focuses on one text line, so the network transcribes a paragraph line by line and performs a soft line segmentation that is trained only to minimize the CTC loss of the whole paragraph’s transcription; a BLSTM decoder then reads the concatenated steps. On the validation sets at 150 dpi, line models had character error rates of 8.4% (IAM) and 4.8% (RIMES) on ground-truth lines but 9.3% to 15.5% and 5.9% to 8.2% on lines from three automatic segmentation methods, while the paragraph model reached 6.8% and 2.8% without any segmentation (Table 2; Table 1, described as the same setting, gives 2.5% for the RIMES paragraph model and 4.9% for the RIMES line model). The author attributes the gain over ground-truth lines mainly to the BLSTM operating across lines, and notes that the attention misses punctuation more often and that the method needs paragraphs to be found first.

The Vertical Attention Network of Coquenet et al. (2023) follows the same line-by-line idea with a fully convolutional encoder, a vertical attention over feature rows that yields one line representation at a time, a learned decision when to stop, and CTC on each line. Without any external language model or lexicon, it reported test-set character and word error rates at paragraph level of 1.91% and 6.72% on RIMES, 4.45% and 14.55% on IAM, and 3.59% and 13.94% on READ 2016, with 2.7 million parameters (Tables 2–4).

Attention and transformer recognizers

Attention decoders generate the transcription one symbol at a time, conditioned on the symbols already produced, the encoder–decoder pattern of image captioning and machine translation; the CTC article contrasts them with CTC, and scene text recognition uses the same decoders for words cropped from photographs. TrOCR (Li et al., 2023) builds both parts from pretrained transformers: an image transformer initialized from BEiT or DeiT encodes patches of the line image, as in a vision transformer, and a text transformer initialized from RoBERTa or MiniLM generates wordpieces, with no convolutional network and no external language model. For handwriting, the second pre-training stage used 17.9 million synthetic text lines rendered in 5,427 handwritten fonts from Wikipedia text, including the IIIT-HWS dataset, and training images were augmented by rotation, blur, dilation, erosion, downscaling and underlining. On IAM (Aachen partition, case-sensitive CER) the small, base and large models, of 62, 334 and 558 million parameters, reached 4.22%, 3.42% and 2.89%; transcribing the 2,915 test lines took 348 s for the small model and 667 s for the large one (Tables 4 and 5). The authors read the result as showing that a transformer decoder can supply the language modelling that CTC systems take from an external model.

Datasets and Benchmarks

Benchmark	Content	Protocol used in the sources	Metric
IAM Handwriting Database (offline)	English text from the LOB corpus, gray-scale scans	writer-independent line task; Aachen partition 6,161 / 966 / 2,915 lines	CER, WER
IAM On-Line (IAM-OnDB)	whiteboard trajectories, LOB corpus prompts	IAM-OnDB-t2: 5,364 / 1,438 / 1,518 / 3,859 lines	word and character accuracy
RIMES	French handwritten letters (mail)	1,500 training and 100 test paragraphs	CER, WER
Bentham (HTRtS 2014)	English manuscripts of Jeremy Bentham and his secretaries	350 / 50 / 33 pages, line images given	WER (ranking), CER
READ 2016 (Ratsprotokolle)	Early Modern German council minutes, 1470–1805	350 / 50 / 50 pages, line regions given	mean of WER and CER
READ 2017 (ICDAR 2017 HTR)	historical pages, partly with page-level transcripts	Train-A 50 pages, Train-B 10,000 pages	not stated in the dataset record
IFN/ENIT (ICDAR 2007)	Arabic town names, isolated words	32,492 public images; hidden test sets f and s	top-1/5/10 recognition rate
MNIST, EMNIST	isolated digits and letters, $28 \times 28$	fixed training and test splits	error rate

IAM. The IAM Handwriting Database consists of scanned forms on which writers copied texts from the Lancaster-Oslo/Bergen (LOB) corpus. Graves et al. (2009) give 13,040 fully transcribed lines with 86,272 word instances of 11,050 distinct words. The Large Writer Independent Text Line Recognition Task, defined on the database’s official page, keeps each writer in one set only; that split, from the official page, and the other partitions in use are summarized in the optical character recognition article. The partitions disagree even on the number of training lines: Graves et al. (2009) describe 6,161 training lines from 283 writers, 920 validation lines from 56 writers and 2,781 test lines from 161 writers under the official task’s name; TrOCR’s Aachen partition has 6,161 lines from 747 forms, 966 from 115 and 2,915 from 336; Bluche (2016) and Coquenet et al. (2023) use 747 training documents, but count 6,482 training lines in them, with 976 validation and 2,915 test lines. Coquenet et al. call their split “commonly used but unofficial”. Error rates on IAM are therefore comparable only when the partition, the case handling and the use of external text are the same.

IAM On-Line. The online database was recorded on a whiteboard, with different prompts from the offline database. The IAM-OnDB-t2 task divides it into a training set, a first validation set, a second validation set that can be used to tune a language model, and a test set (Graves et al., 2009).

RIMES. The RIMES campaign collected letters written by individuals to companies and administrations. The 2008 description reports more than 5,600 annotated mails and secondary databases of isolated characters, about 300,000 handwritten word snippets and logos (Grosicki et al., 2008, abstract). Line and paragraph recognition papers use 1,500 training and 100 test paragraphs and hold out the last 100 training paragraphs for validation, with 11,333 training and 778 test lines in Bluche’s count (2016) and 10,532, 801 and 778 lines after the hold-out in that of Coquenet et al. (2023). Bluche found the language model quite important on IAM, probably, in his words, because there is more variability in the language; a footnote cites a perplexity of 18 for a simple language model on RIMES, against 298 for his IAM language model on the IAM validation set.

Historical documents: HTRtS and READ. The ICFHR 2014 HTRtS competition used 433 pages from the Bentham collection, written by Bentham and his secretaries, with carefully segmented line images supplied; pages were shuffled before splitting because the writers could not be identified, so some writers appear in both training and test sets. The text gives 11,537 lines, while the dataset table sums to 11,473. Methods were ranked by WER in a restricted track (provided data only) and an unrestricted track; the best results were 14.6% WER and 5.0% CER (CITlab) in the restricted track and 8.6% and 2.9% (A2iA) in the unrestricted one (Sánchez et al., 2014, Table III). The ICFHR 2016 competition on the READ dataset used 450 pages from the Ratsprotokolle collection, 10,550 lines with a vocabulary of over 8,000 words in Early Modern German, where compound words make language modelling harder than in English. Participants received page images and PAGE files with line regions, were ranked by the mean of WER and CER, and had to enter the restricted track. The best restricted result was 20.9 ± 1.2% WER and 4.8 ± 0.3% CER (RWTH), and A2iA’s unrestricted entry, trained with about 500 additional modern German letters, reached 21.0% and 5.1% (Table II). The ICDAR 2017 READ competition added page-level material: its dataset record lists a training set of 50 pages with manually revised baselines, a second training set of 10,000 pages with transcripts given only at page level, and test sets, two of which give only region geometry, so that lines must also be detected, released under CC BY 4.0 (Zenodo record, as of November 2026).

Characters: MNIST and EMNIST. MNIST is the standard benchmark of isolated handwritten digits. EMNIST, by Cohen, Afshar, Tapson and van Schaik, converts the digits and letters of NIST Special Database 19 to the $28 \times 28$ format of MNIST in six splits, the largest with 814,255 characters in 62 classes (NIST, as of November 2026); the MNIST article describes the splits. These are classification benchmarks, not tests of reading lines.

Evaluation Metrics

Results are reported as character and word error rates: the edit distance between the recognized text and the reference, counted in characters or in words, divided by the length of the reference. The optical character recognition article gives the definitions; Graves et al. (2009) report word and character accuracy, one minus these rates. Several conventions change the numbers:

• Pooling. Coquenet et al. (2023) divide the summed distances by the total length of all references, so that errors in short lines do not weigh more than errors in long ones.
• Tokenization. In READ 2016 and in Coquenet et al., punctuation marks count as words, and transcripts had to be tokenized as in the training data; IAM results in TrOCR are case-sensitive.
• Ranking measure. HTRtS ranked by WER alone; READ 2016 ranked by the mean of WER and CER, which, as its organizers note, favours systems whose character errors are concentrated in few words.
• Word recognition. Isolated-word tasks report the share of words recognized, often allowing several ranked answers, as in the top-1, top-5 and top-10 results of the ICDAR 2007 Arabic competition.

Lexicons and language models. Lexicons and language models change the word error rate substantially. Without a language model, Bluche’s paragraph model had a test WER of 29.5% on IAM at 150 dpi, and 16.6% with a trigram language model over a 50,000-word lexicon trained on the LOB, Brown and Wellington corpora, while on RIMES the model without a language model was already competitive (13.6% WER). Several systems for historical collections used character language models or hybrid models that fall back to characters for OOV words, as in A2iA’s entry to HTRtS and the RWTH and A2iA entries of READ 2016; the READ 2016 organizers note that character language models avoid the OOV problem of word-based ones. The mechanics of combining CTC outputs with a dictionary and a language model are in the CTC article.

Example: Line Recognition of Synthetic Cursive, With and Without a Lexicon

The PyTorch script below trains a small CTC line recognizer on synthetic handwriting and scores it by CER and WER with three decoders. Each of 12 lowercase letters is a pen trajectory, a few control points in units of the x-height; a word is drawn as one continuous, smoothed polyline, so its letters are joined as in cursive, and i-dots and t-bars are separate delayed strokes. Forty synthetic writers differ in slant, letter width, stroke width, letter spacing, x-height, word gap, point jitter and baseline wobble. Lines of three words are rendered into $32 \times 256$ images with Gaussian noise. Training uses 6,000 lines from writers 0–29, with each word taken from a 48-word vocabulary or, with probability one half, a random string of three to six letters. Three test sets of 500 lines use (a) the training writers with vocabulary words only, (b) the ten held-out writers with vocabulary words only, a writer-independent split in the manner of the IAM line task, and (c) the held-out writers with one word per line replaced by a random string that is not in the vocabulary.

The recognizer has three convolution and pooling blocks, which turn the image into 64 columns, a bidirectional LSTM and a per-column output over the 12 letters, the space and the blank, trained with torch.nn.CTCLoss for 1,200 steps. Its outputs are decoded three ways. Best path takes the most probable class per column and collapses the result, without a lexicon. Post-hoc correction replaces each word of the best-path output by the nearest vocabulary word in edit distance. The lexicon-constrained decoder searches for the best path through a graph in which every vocabulary word is a CTC-style chain of label and blank states and words are separated by a space state, which is the token-passing idea of Graves et al. (2009) without a language model. The sources describe both kinds of lexicon use, so both are printed. Before training, the script runs a check of the task that needs no model: how often the ink of a rendered word forms a number of connected components different from its number of letters, which is how often a segmentation-based reader would fail at the first step, and how much two renderings of a word differ within and across writers.

The data are fixed; the seed argument sets the network’s initialization and the batch order. The script was run with PyTorch 2.14.1, NumPy 2.5.4 and SciPy 1.18.1 on four threads with seeds 0, 1 and 2, each twice, with identical output apart from the printed runtime. One run took 238 s on our machine alone and 274–306 s with three runs in parallel; the plan had estimated about 90 s, and the example was kept at its planned size. The plan’s hypotheses, written before the first run, were that the character error rate would be higher for unseen than for seen writers, “by a factor of roughly 1.5-2”; that WER would be “about 2.5-4 times the CER”; that both lexicon methods would cut WER “by half or more”, with the constrained decoder “at least as good as post-hoc correction”; and that the lexicon decoders would read none of the out-of-vocabulary words. The first run printed sets (a) and (b) and then stopped with an error in set (c), because a random string happened to be a vocabulary word. The string is now redrawn until it is not one, which changes set (c) and the random draws of the task check rendered after it, whose printed values did not change; the rerun reproduced the first run’s numbers for (a) and (b). Two lines of printing were changed afterwards: a condition meant to print example lines for set (a) also matched set (b) and was corrected, and, after the first run showed unseen writers scoring better than seen ones, the script was made to print three of the writers’ eight style parameters for the training and held-out groups, chosen from a diagnostic run, kept with the script, that printed all eight. No computed value changed.

import sys, time
import numpy as np
import scipy
from scipy import ndimage
from scipy.interpolate import make_interp_spline
import torch
import torch.nn as nn

seed = int(sys.argv[1]) if len(sys.argv) > 1 else 0
torch.set_num_threads(4)
torch.manual_seed(seed)
t_start = time.time()

# ---- Synthetic cursive: each letter is a pen trajectory in x-height units (baseline y=0) ----
LETTERS = {
    "a": [(0, .2), (.5, .9), (.2, .9), (.05, .5), (.2, .05), (.5, .3), (.55, .9), (.55, .2), (.7, 0), (.8, .2)],
    "c": [(0, .2), (.55, .85), (.3, .95), (.05, .6), (.1, .1), (.4, 0), (.65, .25)],
    "d": [(0, .2), (.5, .9), (.2, .9), (.05, .5), (.2, .05), (.5, .3), (.55, 2.0), (.55, .2), (.7, 0), (.8, .2)],
    "e": [(0, .2), (.5, .5), (.45, .85), (.2, .85), (.05, .4), (.2, 0), (.5, .05), (.65, .25)],
    "i": [(0, .2), (.2, .9), (.2, .2), (.3, 0), (.45, .2)],
    "l": [(0, .2), (.4, 1.4), (.35, 2.0), (.15, 1.6), (.2, .3), (.35, 0), (.55, .2)],
    "m": [(0, .2), (.1, .9), (.1, 0), (.15, .7), (.35, .9), (.45, .6), (.45, 0), (.5, .7), (.7, .9),
          (.8, .6), (.8, .1), (.9, 0), (1.0, .2)],
    "n": [(0, .2), (.1, .9), (.1, 0), (.15, .7), (.35, .9), (.45, .6), (.45, .1), (.55, 0), (.65, .2)],
    "o": [(0, .2), (.35, .95), (.1, .8), (.05, .3), (.3, 0), (.5, .4), (.4, .9), (.5, .85), (.7, .8)],
    "r": [(0, .2), (.2, .9), (.35, .8), (.3, 0), (.4, .6), (.55, .8), (.7, .6)],
    "s": [(0, .2), (.35, .95), (.2, .6), (.45, .35), (.4, .05), (.1, .05), (.4, 0), (.6, .2)],
    "t": [(0, .2), (.25, 1.7), (.25, .2), (.4, 0), (.55, .2)],
}
DELAYED = {"i": [[(.17, 1.35), (.23, 1.45)]], "t": [[(.05, 1.0), (.5, 1.05)]]}   # i-dot, t-bar
ALPHABET = " " + "".join(sorted(LETTERS))            # class 0 = CTC blank, 1 = space, 2.. = letters
VOCAB = ("act aim air and arc art cat cot dam den dim ear ice lid lot mad man mat net not oil one red "
         "rim rod sat sea set sit son tan tea ten tie tin toe ton cream dream lemon melon metal model "
         "noise ocean radio salon scale smile stone train").split()
H, W, T = 32, 256, 64

def make_writer(rng):
    return dict(slant=rng.uniform(-.15, .45), wscale=rng.uniform(.8, 1.25), sigma=rng.uniform(.6, 1.1),
                spacing=rng.uniform(0, .25), xh=rng.uniform(7.5, 10), jitter=rng.uniform(.03, .08),
                wobble=rng.uniform(0, 1.5), gap=rng.uniform(.9, 1.4))

def smooth(points, step=.25):
    p = np.asarray(points, float)
    if len(p) < 4:
        n = max(2, int(np.linalg.norm(p[-1] - p[0]) / step) + 2)
        return np.linspace(p[0], p[-1], n)
    d = np.r_[0, np.cumsum(np.linalg.norm(np.diff(p, axis=0), axis=1))]
    keep = np.r_[True, np.diff(d) > 1e-6]
    p, d = p[keep], d[keep]
    spl = make_interp_spline(d, p, k=3)
    return spl(np.linspace(0, d[-1], max(4, int(d[-1] / step))))

def render_line(words, wr, rng):
    """Render words as cursive strokes; returns a 32 x 256 image in [0, 1] (ink = 1)."""
    xh, strokes, x = wr["xh"], [], 4.0
    period, phase = rng.uniform(60, 150), rng.uniform(0, 2 * np.pi)
    for w in words:
        main = []
        for ch in w:
            pts = np.array(LETTERS[ch], float) + rng.normal(0, wr["jitter"], (len(LETTERS[ch]), 2))
            main.append(np.c_[x / xh + pts[:, 0] * wr["wscale"], pts[:, 1]])
            for d in DELAYED.get(ch, []):
                dd = np.array(d) + rng.normal(0, wr["jitter"], (len(d), 2))
                strokes.append(np.c_[x / xh + dd[:, 0] * wr["wscale"], dd[:, 1]])
            x += (max(p[0] for p in LETTERS[ch]) * wr["wscale"] + wr["spacing"]) * xh
        strokes.append(np.vstack(main))
        x += (wr["gap"] + rng.uniform(-.2, .2)) * xh
    acc = np.zeros((H, W))
    paths = [smooth(s * xh) for s in strokes]                # to pixels, then smooth the pen path
    right = max((p[:, 0] + wr["slant"] * p[:, 1]).max() for p in paths)
    shrink = min(1.0, (W - 8 - 4) / (right - 4))             # squeeze rare over-long lines to fit
    for s in paths:
        px = 4 + (s[:, 0] - 4) * shrink + wr["slant"] * s[:, 1]
        base = 21 + wr["wobble"] * np.sin(2 * np.pi * px / period + phase)
        py = base - s[:, 1]
        ok = (px >= 0) & (px < W - 1) & (py >= 0) & (py < H - 1)
        np.add.at(acc, (np.round(py[ok]).astype(int), np.round(px[ok]).astype(int)), 1.0)
    img = 1 - np.exp(-1.5 * ndimage.gaussian_filter(acc, wr["sigma"]))
    return img + rng.normal(0, .05, img.shape)

def random_string(rng):
    return "".join(rng.choice(list(LETTERS), rng.integers(3, 7)))

def make_set(n, writer_ids, writers, rng, mode):
    X, Y = np.zeros((n, 1, H, W), np.float32), []
    for i in range(n):
        wid = writer_ids[i % len(writer_ids)]
        if mode == "train":
            words = [VOCAB[rng.integers(len(VOCAB))] if rng.random() < .5 else random_string(rng) for _ in range(3)]
        else:
            words = [VOCAB[rng.integers(len(VOCAB))] for _ in range(3)]
            if mode == "oov":
                oov = random_string(rng)
                while oov in VOCAB:                            # an out-of-vocabulary word must not be a word
                    oov = random_string(rng)
                words[rng.integers(3)] = oov
        X[i, 0] = render_line(words, writers[wid], rng)
        Y.append(" ".join(words))
    return X, Y

# ---- Data (fixed; the seed argument changes only initialization and batch order) ----------------
rng = np.random.default_rng(1234)
writers = [make_writer(rng) for _ in range(40)]
train_w, test_w = list(range(30)), list(range(30, 40))
Xtr, Ytr = make_set(6000, train_w, writers, rng, "train")
sets = {"(a) seen writers, in-vocabulary": make_set(500, train_w, writers, rng, "vocab"),
        "(b) unseen writers, in-vocabulary": make_set(500, test_w, writers, rng, "vocab"),
        "(c) unseen writers, 1 OOV word/line": make_set(500, test_w, writers, rng, "oov")}

# ---- Label-free check of the task -----------------------------------------------------------------
comp_diff, ncomp, nlet = 0, [], []
for i in range(200):
    w = VOCAB[i % len(VOCAB)]
    img = render_line([w], writers[i % 40], rng)
    n = ndimage.label(ndimage.gaussian_filter(img, 1) > .4)[1]
    ncomp.append(n); nlet.append(len(w)); comp_diff += n != len(w)
same, other = [], []
for i in range(100):
    w = VOCAB[i % len(VOCAB)]
    a = render_line([w], writers[i % 30], rng)
    same.append(np.abs(a - render_line([w], writers[i % 30], rng)).mean())
    other.append(np.abs(a - render_line([w], writers[30 + i % 10], rng)).mean())
print(f"NumPy {np.__version__}, SciPy {scipy.__version__}, PyTorch {torch.__version__}; seed {seed}")
print(f"label-free check: connected components != letters in {comp_diff / 2:.1f}% of 200 rendered words "
      f"(mean {np.mean(ncomp):.2f} components for {np.mean(nlet):.2f} letters)")
print(f"  mean |pixel difference| same word: same writer {np.mean(same):.3f}, other writer {np.mean(other):.3f}")
for k in ("wscale", "spacing", "sigma"):
    a, b = [writers[i][k] for i in train_w], [writers[i][k] for i in test_w]
    print(f"  writer {k:<8} training writers mean {np.mean(a):.3f} (range {min(a):.2f}-{max(a):.2f}), "
          f"held-out mean {np.mean(b):.3f} (range {min(b):.2f}-{max(b):.2f})")

# ---- Model: CNN -> columns -> BiLSTM -> per-frame class scores, trained with CTC ----------------------
class Recognizer(nn.Module):
    def __init__(self, n_cls):
        super().__init__()
        self.cnn = nn.Sequential(
            nn.Conv2d(1, 32, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),          # 16 x 128
            nn.Conv2d(32, 64, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),         # 8 x 64
            nn.Conv2d(64, 96, 3, padding=1), nn.ReLU(), nn.MaxPool2d((2, 1)))    # 4 x 64
        self.proj = nn.Linear(96 * 4, 128)
        self.rnn = nn.LSTM(128, 96, bidirectional=True)
        self.out = nn.Linear(192, n_cls)
    def forward(self, x):
        f = self.cnn(x)                                        # (B, 96, 4, 64)
        f = f.permute(3, 0, 1, 2).flatten(2)                   # (T=64, B, 384)
        h, _ = self.rnn(torch.relu(self.proj(f)))
        return self.out(h).log_softmax(-1)                     # (T, B, classes)

def encode(texts):
    tg = [torch.tensor([ALPHABET.index(c) + 1 for c in t]) for t in texts]
    return torch.cat(tg), torch.tensor([len(t) for t in tg])

model = Recognizer(len(ALPHABET) + 1)
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
ctc = nn.CTCLoss(blank=0, zero_infinity=True)
bat = np.random.default_rng(seed)
losses = []
for step in range(1200):
    idx = bat.integers(0, len(Xtr), 32)
    tgt, tl = encode([Ytr[i] for i in idx])
    logp = model(torch.from_numpy(Xtr[idx]))
    loss = ctc(logp, tgt, torch.full((32,), T, dtype=torch.long), tl)
    opt.zero_grad(); loss.backward()
    nn.utils.clip_grad_norm_(model.parameters(), 5.0); opt.step()
    losses.append(loss.item())
print("mean training CTC loss per 120 steps: " + " ".join(f"{np.mean(losses[i:i+120]):.2f}" for i in range(0, 1200, 120)))

# ---- Decoders ---------------------------------------------------------------------------------------
def edit_distance(a, b):
    d = list(range(len(b) + 1))
    for i in range(1, len(a) + 1):
        prev, d[0] = d[:], i
        for j in range(1, len(b) + 1):
            d[j] = min(prev[j] + 1, d[j - 1] + 1, prev[j - 1] + (a[i - 1] != b[j - 1]))
    return d[-1]

def best_path(lp):                                             # lp: (T, C) log-probs
    k = lp.argmax(1); out, prev = [], 0
    for c in k:
        if c != prev and c != 0:
            out.append(ALPHABET[c - 1])
        prev = c
    return " ".join("".join(out).split())

def posthoc(text):
    return " ".join(min(VOCAB, key=lambda v: edit_distance(w, v)) for w in text.split())

# word-loop graph for lexicon-constrained CTC Viterbi: each word's blank-extended states, plus one space state
lab, starts, ends, word_of = [], [], [], []
for wi, w in enumerate(VOCAB):
    base = len(lab)
    for c in w:
        lab += [0, ALPHABET.index(c) + 1]
    lab.append(0)
    starts += [base, base + 1]; ends += [len(lab) - 2, len(lab) - 1]
    word_of += [wi] * (2 * len(w) + 1)
SP = len(lab); lab.append(ALPHABET.index(" ") + 1); word_of.append(-1)
lab, starts, ends = np.array(lab), np.array(starts), np.array(ends)
S = len(lab)
wstart = np.zeros(S, bool); wstart[starts] = True
pos = np.zeros(S, int)
for wi, w in enumerate(VOCAB):
    b = np.flatnonzero(np.array(word_of) == wi)[0]
    pos[b:b + 2 * len(w) + 1] = np.arange(2 * len(w) + 1)
skip_ok = (pos >= 2) & (lab != 0)
skip_ok[skip_ok] &= lab[np.flatnonzero(skip_ok) - 2] != lab[skip_ok]
prev1_ok = pos >= 1

def lexicon_viterbi(lp):
    NEG = -1e30
    e = lp[:, lab]                                             # (T, S) emission log-probs
    delta = np.full(S, NEG); delta[starts] = e[0, starts]
    back = np.zeros((len(lp), S), int)
    idx = np.arange(S)
    for t in range(1, len(lp)):
        cand = np.stack([delta,
                         np.where(prev1_ok, np.roll(delta, 1), NEG),
                         np.where(skip_ok, np.roll(delta, 2), NEG),
                         np.where(wstart, delta[SP], NEG)])
        arg = cand.argmax(0)
        best = cand[arg, idx]
        src = np.choose(arg, [idx, idx - 1, idx - 2, np.full(S, SP)])
        e_end = ends[np.argmax(delta[ends])]                   # space state: stay or come from a word end
        if delta[e_end] > delta[SP]:
            best[SP], src[SP] = delta[e_end], e_end
        else:
            best[SP], src[SP] = delta[SP], SP
        delta, back[t] = best + e[t], src
    s = ends[np.argmax(delta[ends])]
    path = [s]
    for t in range(len(lp) - 1, 0, -1):
        s = back[t, s]; path.append(s)
    words, last = [], None
    for s in path[::-1]:                                       # read off the word entered at each start
        if s == SP:
            last = None
        elif last is None:
            words.append(VOCAB[word_of[s]]); last = word_of[s]
    return " ".join(words)

def rates(hyps, refs):
    cer = sum(edit_distance(h, r) for h, r in zip(hyps, refs)) / sum(len(r) for r in refs)
    wer = sum(edit_distance(h.split(), r.split()) for h, r in zip(hyps, refs)) / sum(len(r.split()) for r in refs)
    return 100 * cer, 100 * wer

model.eval()
print("\nset                                   decoder              CER %   WER %   OOV words read")
for name, (X, Y) in sets.items():
    with torch.no_grad():
        lp = np.concatenate([model(torch.from_numpy(X[i:i + 100])).numpy() for i in range(0, len(X), 100)], 1)
    greedy = [best_path(lp[:, i]) for i in range(len(Y))]
    outs = {"best path (no lexicon)": greedy, "post-hoc nearest word": [posthoc(g) for g in greedy],
            "lexicon CTC Viterbi": [lexicon_viterbi(lp[:, i]) for i in range(len(Y))]}
    for dec, hyp in outs.items():
        c, w = rates(hyp, Y)
        oov = ""
        if "OOV" in name:
            o = [next(t for t in r.split() if t not in VOCAB) for r in Y]
            oov = f"{100 * np.mean([t in h.split() for t, h in zip(o, hyp)]):.1f}%"
        print(f"{name:<38}{dec:<21}{c:6.1f}  {w:6.1f}   {oov}")
    if name.startswith("(a)"):
        for i in range(2):
            print(f"    truth '{Y[i]}' | best path '{greedy[i]}' | Viterbi '{outs['lexicon CTC Viterbi'][i]}'")
print(f"\nruntime {time.time() - t_start:.0f} s")

Output of python htr_example.py 0:

NumPy 2.5.4, SciPy 1.18.1, PyTorch 2.14.1; seed 0
label-free check: connected components != letters in 100.0% of 200 rendered words (mean 1.10 components for 3.52 letters)
  mean |pixel difference| same word: same writer 0.063, other writer 0.073
  writer wscale   training writers mean 1.016 (range 0.80-1.24), held-out mean 1.069 (range 0.89-1.23)
  writer spacing  training writers mean 0.118 (range 0.01-0.25), held-out mean 0.151 (range 0.02-0.24)
  writer sigma    training writers mean 0.881 (range 0.60-1.09), held-out mean 0.824 (range 0.61-1.09)
mean training CTC loss per 120 steps: 3.03 2.66 2.63 2.38 1.93 1.62 1.32 1.11 0.97 0.87

set                                   decoder              CER %   WER %   OOV words read
(a) seen writers, in-vocabulary       best path (no lexicon)  21.6    63.7   
(a) seen writers, in-vocabulary       post-hoc nearest word  15.7    31.7   
(a) seen writers, in-vocabulary       lexicon CTC Viterbi     4.5    11.5   
    truth 'cot sea oil' | best path 'cot sao oil' | Viterbi 'cot ear oil'
    truth 'train set melon' | best path 'train set maalrn' | Viterbi 'train set melon'
(b) unseen writers, in-vocabulary     best path (no lexicon)  18.7    56.9   
(b) unseen writers, in-vocabulary     post-hoc nearest word  11.1    22.8   
(b) unseen writers, in-vocabulary     lexicon CTC Viterbi     3.3     8.3   
(c) unseen writers, 1 OOV word/line   best path (no lexicon)  22.5    64.1   17.0%
(c) unseen writers, 1 OOV word/line   post-hoc nearest word  29.1    48.7   0.0%
(c) unseen writers, 1 OOV word/line   lexicon CTC Viterbi    24.1    39.6   0.0%

runtime 306 s

The task check confirms that this is cursive in the sense that matters: the rendered words have on average 1.10 connected components for 3.52 letters, and no word has as many components as letters, so a reader that first cuts the ink into characters cannot even start. The recognizer learns to read without any character positions, but it is far from converged: the training loss is still falling after 1,200 steps, and best-path decoding misreads about a fifth of the characters. The word error rate is about three times the character error rate (63.7% against 21.6% on set a), as expected, since a single wrong character spoils a whole word. Across the three seeds, best-path CER ranges over 21.6–24.1% on set (a) and 18.5–21.4% on set (b), and WER over 63.7–68.3% and 56.7–63.8%.

The writer-independent comparison went against the plan: in all three seeds the unseen writers were read better than the seen ones, by 2.7 to 4.2 points of CER. The printed style parameters suggest why, in our reading. The held-out writers were drawn from the same distribution as the training writers, the three printed parameters lie inside the training range, and on average they write wider letters with more space between them and thinner strokes, which plausibly makes their lines easier to read. The pixel differences point the same way: two renderings of a word by different writers differ only slightly more (0.073) than two renderings by the same writer (0.063). Synthetic writers that vary in eight parameters do not reproduce the differences between real writers, which is why the IAM line task keeps each writer in one set; this example cannot show the writer-independence gap and is reported as printed.

The lexicon results came out as planned, and the choice of decoder matters as much as the lexicon. Constraining the CTC search to vocabulary words cut WER on the in-vocabulary sets from 56.7–68.3% to 8.3–16.3% over the three seeds, while correcting the best-path words afterwards reached only 22.8–38.2%. In our reading the constrained search has two advantages: it scores every vocabulary word against the full per-column distributions, including the probability that best path throws away, and it chooses the word boundaries itself, whereas post-hoc correction inherits every merged or split word of the best-path output. A lexicon also changes the kind of error. In the first example line, best path reads “sea” as “sao”, two wrong characters; the constrained decoder replaces it by “ear”, a real word that differs in every position.

Set (c) shows the cost of a lexicon that is not closed. Best path reads 14.8–17.4% of the out-of-vocabulary strings exactly, and both lexicon decoders, which can output only vocabulary words, read none. The constrained decoder still has the lowest WER on this set (39.6–41.3%, against 63.1–69.4% for best path), because it repairs the two in-vocabulary words of each line, but its CER, 24.1–25.3%, is now higher than that of best path, 22.4–24.7%, because every OOV word is replaced by a different word. With one OOV word in three, the lexicon still pays off here at the word level; the balance depends on the OOV rate and on how good the optical model is, which is the trade-off behind the dictionary-size results of Graves et al. (2009) and the character and hybrid language models of the HTR competitions. All the numbers depend on the synthetic writing, the small vocabulary and the short training, and none of them is a property of real handwriting.

Applications

• Postal and banking automation. Reading address lines and the legal amounts on checks, the applications on which offline cursive recognition research focused, with small closed lexicons (Vinciarelli et al., 2003).
• Mail and form processing. The RIMES campaign targeted handwritten letters sent by post or fax by individuals to companies and administrations (Grosicki et al., 2008).
• Archives. The tranScriptorium and READ projects aimed at indexing, searching and transcribing historical manuscripts, and the READ 2016 organizers note that even the worst-read of the example lines they show could be useful for reading and searching (Sánchez et al., 2014, 2016). Transcriptions of archive pages are then laid out and indexed with methods from document image analysis.
• Pen input. Online recognition reads writing captured on whiteboards and tablets; Graves et al. (2009) compare their whiteboard recognizer with the Microsoft Tablet PC handwriting recognizer, which reached 71.32% word accuracy on their online test set with a different training set and language model.

Open Problems

• Full pages and reading order. Bluche (2016) notes that his paragraph model cannot handle arbitrary reading orders and needs the paragraphs to be found first, and that in complex documents the reading order of text blocks often cannot be inferred exactly.
• Out-of-vocabulary words and languages with compounds, as the READ 2016 results show, and languages and scripts with little training text.
• Less labelled data. Historical collections are written by many hands, and transcribing training lines is costly; the READ 2016 organizers name training with less ground truth as a next challenge.
• Small marks and diplomatic transcription: punctuation, hyphenation and crossed-out text, which the competition reports list among the main difficulties.
• Comparable evaluation. Different IAM partitions, case handling, external language models and extra training data make numbers from different papers hard to compare, as the tables of TrOCR and Coquenet et al. (2023) both point out.