Optical Character Recognition

Optical character recognition (OCR) converts an image of text into the characters it shows, so that a scanned page, a photographed receipt or a street sign becomes text a program can search, edit or translate. The term covers machine-printed and handwritten text, single characters and whole pages, and documents as well as text that appears incidentally in natural scenes; the last is usually called scene text recognition, and finding the text in the first place is scene text detection. Other names include text recognition, document digitization and, for handwriting, handwritten text recognition.

OCR shows clearly how a vision task changes shape as methods improve. Early systems cut a page into isolated characters and classified each one, which made OCR a special case of image classification. Segmentation turned out to be the weak link, and modern recognizers read whole lines or words at once, either by labeling a sequence of image columns, an idea related to sliding window detection, or by generating the text with a sequence decoder. This article surveys the task from printed pages to scene text; the LeNet article covers the digit and character classifiers of the 1990s, and thresholding the binarization step in detail.

Problem Definition

Given an image that contains text, an OCR system outputs the text as a sequence of character codes. A complete page reader also outputs where each line, word or character lies, groups lines into blocks, and puts the blocks in reading order. Recognition benchmarks often isolate a sub-problem: classify one pre-cut character, transcribe a cropped word or text line, or find and read every word in an image (end-to-end recognition, also called text spotting).

The problem is defined relative to a character set and usually a language. The set can be small, such as the ten digits of a postal code, or very large: Tesseract’s documentation notes that for scripts such as Han and Hangul it encodes each character as a short sequence of codes from a smaller set, because its recognizer learns long sequences of few classes more easily than short sequences of many. Most systems also assume a reading direction and a script, and treat everything else as background.

Input and Output

Inputs range from bilevel scans to camera images:

• Scanned documents. The UNLV accuracy tests that evaluated early Tesseract used 300 DPI binary page images (Smith, 2007), and the IAM handwriting forms were scanned at 300 dpi with 256 gray levels. Tesseract’s documentation recommends at least 300 dpi (as of October 2026).
• Camera and scene images: photographs of documents, signs, packaging and storefronts, including frames from wearable cameras, in which text can be small, blurred, rotated, curved or seen in perspective.
• Cropped text images: a single character, word or line, as given to a recognizer by a detector or by a benchmark.

Outputs are strings with confidences, often with geometry. Word boxes are axis-aligned rectangles for well-captured text and quadrilaterals for incidental scene text; the ICDAR 2015 incidental text ground truth gives the four corners of each word in clockwise order, followed by its transcription. Page readers commonly write their output in a structured format: The LayoutLM authors, for example, stored Tesseract’s output for their pre-training documents in hOCR, a format that stores the recognized text with a hierarchy of page elements and their positions (Xu et al., 2020).

Variants

• Printed versus handwritten text. Printed text varies by font and print quality; handwriting varies by writer and is often cursive, so characters touch and overlap.
• Offline versus online handwriting. Offline recognition sees only an image; online recognition receives the pen trajectory as a time series of coordinates, which Graves et al. (2009) note generally gives better results because relevant features are easier to extract.
• Isolated characters, words and lines. Recognizing isolated digits or characters is a classification problem, solved to error rates below 1% on MNIST by the late 1990s (see LeNet); Graves et al. (2009) observe that results for complete lines of handwriting have never matched those for isolated characters.
• Lexicon-constrained and lexicon-free recognition. A recognizer may choose among the words of a given list (a 50-word list per image, a 50,000-word dictionary) or output any string; Shi et al. (2017) report both settings.
• Scene text detection and end-to-end recognition. Finding text in natural images is a form of object detection, with long, thin, often rotated targets; end-to-end systems detect and read.
• Document understanding. Downstream tasks consume OCR output together with layout: extracting fields from receipts and forms, classifying document images, or answering questions about documents (visual question answering covers DocVQA and TextVQA).

Challenges

• Segmentation. Characters touch, break or overlap, and spacing differs between and within words. LeCun et al. (1998, §V) call segmenting naturally written character strings into well-formed characters almost impossible to do infallibly by image analysis alone, and Smith (2007) gives printed examples: a gap inside “11.9%” about as large as a word space, and no horizontal gap at all between the bounding boxes of two adjacent words.
• Degradation. Noise, blur, low resolution, uneven illumination, bleed-through and broken strokes. Uneven background darkness is a known failure case for global binarization, as Tesseract’s documentation notes for its built-in Otsu threshold.
• Geometry. Skewed scans, curved baselines near book bindings, slanted handwriting, and in scenes perspective distortion, curved placement and arbitrary orientation. Tesseract’s documentation states that line segmentation quality drops significantly when a page is too skewed.
• Variety. Thousands of fonts, many writing styles and scripts, and large character sets.
• Ambiguity. Some glyphs are identical or nearly so (O and 0, l, 1 and I), and others become confusable when degraded, so context from neighbouring characters, words and language is needed.
• Layout. Columns, tables, captions, headers and figures must be separated and ordered before lines can be read in sequence.

Approaches

The classical document pipeline

Classical systems, such as Tesseract in the form developed at HP from 1984 to 1994 (Smith, 2007), processed a page in stages, each a separate algorithm with its own failure modes.

Binarization. The gray or color page is reduced to ink and background. Tesseract, as described by Smith (2007), assumes a binary input image with optional polygonal text regions; its documentation (as of October 2026) states that it binarizes internally with Otsu’s method and that version 5.0.0 added adaptive Otsu and Sauvola thresholding. Global and local threshold selection are covered in thresholding.

Skew estimation and correction. A page scanned at a small angle is rotated back so that text lines are horizontal. A common estimate, used in the example below, is the angle at which the horizontal projection profile (ink per row) is most peaked, because text lines then fall on distinct rows. Graves et al. (2009) normalize whole handwritten lines before feature extraction: they rotate each line to remove its skew, shear the image to remove the mean slant, and scale the body, ascender and descender zones to fixed heights. Rotating the image resamples it, and Smith (2007) designed Tesseract’s line finding to work on skewed pages without de-skewing, to avoid that loss of image quality: blobs are assigned to lines while tracking the slope, baselines are estimated with a least median of squares fit, and then refined with a quadratic spline, which also handles curved baselines.

Layout analysis. The page is divided into text regions, which are ordered and split into lines. Tesseract never needed its own layout analysis, Smith (2007) explains, because HP had separate, proprietary technology for it. For handwritten forms, Graves et al. (2009) found text lines from a histogram of horizontal black-white transitions, splitting at its minima and assigning strokes that cross a boundary to the line containing their centre of gravity.

Character segmentation. Casey and Lecolinet (1996) group segmentation strategies into four families: classical dissection of the image into character-like pieces before classification; methods that classify fixed windows or subsets of features across the whole image without dissecting it; hybrid methods that generate candidate cuts and let the classifier choose among the admissible segmentations; and holistic methods that recognize whole words without segmenting them. The heuristic over-segmentation of LeCun et al. (1998, §V) is a hybrid strategy in these terms: candidate cuts are placed at minima of the vertical projection profile or of the distance between the upper and lower contours of a word, deliberately more than needed, and the alternatives form a directed acyclic graph whose paths are scored by the character recognizer. Tesseract (Smith, 2007) chops fixed-pitch text directly into character cells; for proportional text it chops the blob with the worst classifier confidence at concave vertices of its outline, keeping chops that improve the result, then, if the word is still unsatisfactory, runs an A* search over combinations of the fragments to associate broken characters.

Character classification. Each segment is size-normalized and classified. Tesseract’s static classifier matches short fixed-length outline features of the unknown character many-to-one against clustered prototype segments learned from polygonal approximations of training characters, after a class pruner shortlists candidates; it was trained on only 60,160 samples (20 samples of 94 characters from 8 fonts in 4 styles) yet copes with broken characters (Smith, 2007). An adaptive classifier, trained on the words recognized confidently earlier on the same page, then exploits the small number of fonts within one document, and a second pass re-reads words that were not recognized well enough. Classifiers learned from pixels are the alternative to designed features: LeNet-5 classifies size-normalized characters with a convolutional neural network trained end to end on pixels, and the LeNet article gives its MNIST results and its deployment in check reading. LeCun et al. (1998, §III.E) also describe why classifying segments is fragile: segmenters leave stray marks or cut characters too much, such pieces cannot be reliably size-normalized, and an enlarged stray mark can look like a genuine 1.

Language models and dictionaries. Word context resolves many character errors. Tesseract’s linguistic module (Smith, 2007) picks, for each candidate segmentation, the best word in several categories (frequent word, dictionary word, number, upper case, lower case, top classifier choice), weights each category by a constant, and keeps the word with the lowest weighted distance. Line recognizers combine character probabilities with a dictionary and an n-gram model: in Graves et al. (2009), adding a bigram language model raised online word accuracy with a 20,000-word open dictionary from 74.0% to 79.7%.

Example: a classical pipeline on rendered text lines

The NumPy/SciPy code below renders lines of six words from a $5 \times 7$ bitmap font at $4 \times 4$ pixels per font cell, with one blank cell between letters and four between words, as ink 1 on paper 0. It degrades each line with optional rotation, Gaussian blur ($\sigma = 1$ pixel) and additive Gaussian noise with standard deviation noise; the clean image is in $[0, 1]$, and the noise carries values outside that range. Recognition follows the classical pipeline: an Otsu threshold with removal of connected components under 8 pixels, optional skew correction by the projection-profile criterion on a $0.5^\circ$ grid, segmentation at near-empty columns (fewer than two ink pixels in the text band) of the vertical projection, and a nearest-neighbour classifier against one clean template per letter. To separate segmentation errors from classification errors, the same classifier is also run on segments cut at the true glyph positions. A closed-lexicon step replaces each output word by the nearest of the 33 vocabulary words from which the lines are drawn. Errors are reported as character and word error rates (defined under Evaluation Metrics), over five seeds of 20 lines each.

import numpy as np
from scipy import ndimage as ndi

# ---- A 5x7 bitmap font (uppercase letters), rendered at S x S pixels per font cell --------
FONT = """
A .###. #...# #...# ##### #...# #...# #...#
B ####. #...# #...# ####. #...# #...# ####.
C .###. #...# #.... #.... #.... #...# .###.
D ####. #...# #...# #...# #...# #...# ####.
E ##### #.... #.... ####. #.... #.... #####
F ##### #.... #.... ####. #.... #.... #....
G .###. #...# #.... #.### #...# #...# .###.
H #...# #...# #...# ##### #...# #...# #...#
I .###. ..#.. ..#.. ..#.. ..#.. ..#.. .###.
J ..### ...#. ...#. ...#. ...#. #..#. .##..
K #...# #..#. #.#.. ##... #.#.. #..#. #...#
L #.... #.... #.... #.... #.... #.... #####
M #...# ##.## #.#.# #.#.# #...# #...# #...#
N #...# #...# ##..# #.#.# #..## #...# #...#
O .###. #...# #...# #...# #...# #...# .###.
P ####. #...# #...# ####. #.... #.... #....
Q .###. #...# #...# #...# #.#.# #..#. .##.#
R ####. #...# #...# ####. #.#.. #..#. #...#
S .#### #.... #.... .###. ....# ....# ####.
T ##### ..#.. ..#.. ..#.. ..#.. ..#.. ..#..
U #...# #...# #...# #...# #...# #...# .###.
V #...# #...# #...# #...# #...# .#.#. ..#..
W #...# #...# #...# #.#.# #.#.# #.#.# .#.#.
X #...# #...# .#.#. ..#.. .#.#. #...# #...#
Y #...# #...# .#.#. ..#.. ..#.. ..#.. ..#..
Z ##### ....# ...#. ..#.. .#... #.... #####
"""
S = 4                                              # pixels per font cell
GLYPH = {}
for row in FONT.strip().splitlines():
    ch, *rows = row.split()
    bits = np.array([[c == "#" for c in r] for r in rows], float)
    GLYPH[ch] = np.kron(bits, np.ones((S, S)))     # 28 x 20 pixels, ink = 1
VOCAB = ("THE AND IMAGE PIXEL TEXT LINE PAGE READ WORD FONT SCAN EDGE BLUR NOISE GLYPH PRINT "
         "PAPER INK CODE DATA MODEL LIGHT SHAPE COLUMN ROW LETTER QUICK ZERO JUMP BOX KEY VIEW WAX").split()

def render(words):
    """Clean line image in [0, 1] (ink 1, paper 0): 1-cell gap between letters, 4 cells between words."""
    cols = []
    for w in words:
        for ch in w:
            cols += [GLYPH[ch], np.zeros((7 * S, S))]
        cols.append(np.zeros((7 * S, 3 * S)))
    return np.pad(np.hstack(cols[:-2]), 12)

def degrade(img, rng, noise, blur, skew):
    img = ndi.rotate(img, skew, reshape=True, order=1) if skew else img
    img = ndi.gaussian_filter(img, blur)
    return img + rng.normal(0, noise, img.shape)

# ---- Recognition pipeline: binarize, deskew, segment by projection, classify ----------------
def otsu(img):
    """Otsu's threshold: maximize the between-class variance of the 256-bin histogram."""
    hist, edges = np.histogram(img, 256)
    p = hist / hist.sum(); w = np.cumsum(p); mu = np.cumsum(p * (edges[:-1] + edges[1:]) / 2)
    between = (mu[-1] * w - mu) ** 2 / (w * (1 - w) + 1e-12)
    return edges[np.argmax(between) + 1]

def binarize(img, min_size=8):
    """Otsu threshold, then drop connected components smaller than min_size pixels (specks)."""
    lab, n = ndi.label(img > otsu(img))
    sizes = ndi.sum(np.ones_like(img), lab, index=np.arange(1, n + 1))
    return np.isin(lab, 1 + np.flatnonzero(sizes >= min_size)).astype(float)

def skew_angle(binary):
    """Angle (0.5 deg grid) whose rotation makes the row-projection profile peakiest."""
    angles = np.arange(-8, 8.01, 0.5)
    score = [np.sum(ndi.rotate(binary, -a, reshape=False, order=0).sum(1) ** 2) for a in angles]
    return angles[int(np.argmax(score))]

def segment(binary, min_ink=2, space=2 * S + 1):
    """Cut the line at empty columns of the vertical projection; wide gaps are word spaces."""
    prof = binary.sum(1)
    rows = np.flatnonzero(prof >= 0.2 * prof.max())          # text band: rows with enough ink
    band = binary[rows.min():rows.max() + 1]
    ink = band.sum(0) >= min_ink
    segs, start, gap = [], None, 0
    for x, on in enumerate(np.append(ink, False)):
        if on and start is None:
            if segs and gap > space: segs.append(" ")
            start = x
        elif not on and start is not None:
            segs.append(band[:, start:x]); start, gap = None, 0
        elif not on:
            gap += 1
    return segs

def normalize(patch):
    rows = np.flatnonzero(patch.sum(1) > 0)                  # trim to the inked rows, then resize
    patch = patch[rows.min():rows.max() + 1] if rows.size else patch
    return ndi.zoom(patch, (7 * S / patch.shape[0], 5 * S / patch.shape[1]), order=1)

TEMPLATES = {ch: normalize(segment(np.pad(g, 4))[0]) for ch, g in GLYPH.items()}  # one per class
KEYS = list(TEMPLATES); TMAT = np.stack([TEMPLATES[k].ravel() for k in KEYS])

def classify(patch):
    d = ((TMAT - normalize(patch).ravel()) ** 2).sum(1)      # nearest neighbour, Euclidean
    return KEYS[int(np.argmin(d))]

def recognize(img, fix_skew=False):
    binary = binarize(img)
    if fix_skew:                                             # rotate the gray image back, re-binarize
        binary = binarize(ndi.rotate(img, -skew_angle(binary), reshape=False, order=1))
    return "".join(s if isinstance(s, str) else classify(s) for s in segment(binary))

def recognize_true_cuts(img, words):
    """Same classifier, but cut at the true glyph positions (unskewed lines only)."""
    binary = binarize(img)
    out, x = [], 12
    for w in words:
        for ch in w:
            patch = binary[12:12 + 7 * S, x:x + 5 * S]
            cols = np.flatnonzero(patch.sum(0) > 0)          # trim to the glyph's inked columns
            out.append(classify(patch[:, cols.min():cols.max() + 1]) if cols.size else "?")
            x += 6 * S
        out.append(" "); x += 3 * S
    return "".join(out[:-1])

# ---- Metrics: edit distance, CER and WER ------------------------------------------------------
def edit_distance(a, b):
    d = np.arange(len(b) + 1)
    for i in range(1, len(a) + 1):
        prev, d[0] = d.copy(), i
        for j in range(1, len(b) + 1):
            d[j] = min(prev[j] + 1, d[j - 1] + 1, prev[j - 1] + (a[i - 1] != b[j - 1]))
    return d[-1]

def lexicon(text):
    """Replace each recognized word by the nearest vocabulary word (edit distance)."""
    return " ".join(min(VOCAB, key=lambda v: edit_distance(w, v)) for w in text.split())

def rates(pairs):
    """CER and WER in %: total edit distance over total reference length (characters / words)."""
    cer = sum(edit_distance(h, r) for h, r in pairs) / sum(len(r) for _, r in pairs)
    wer = sum(edit_distance(h.split(), r.split()) for h, r in pairs) / sum(len(r.split()) for _, r in pairs)
    return 100 * cer, 100 * wer

# ---- Experiments ----------------------------------------------------------------------------
SEEDS, LINES, BLUR = range(5), 20, 1.0
print(f"seeds {list(SEEDS)} (A), {[100 + s for s in SEEDS]} (B), 7 and 100 (examples); "
      f"{LINES} lines of 6 words per seed; blur sigma {BLUR} px; intensities in [0, 1]")
print("\nA. Noise, no skew. CER % as mean over seeds (min-max); WER %; segments minus characters per line")
print("noise   true cuts         projection cuts   WER   WER+lexicon   segments-chars")
for noise in [0.1, 0.2, 0.25, 0.3, 0.4, 0.5]:
    res = []
    for seed in SEEDS:
        rng = np.random.default_rng(seed)
        t, p, diff = [], [], []
        for _ in range(LINES):
            words = list(rng.choice(VOCAB, 6)); ref = " ".join(words)
            img = degrade(render(words), rng, noise, BLUR, 0)
            t.append((recognize_true_cuts(img, words), ref))
            hyp = recognize(img); p.append((hyp, ref)); diff.append(len(hyp) - len(ref))
        res.append([rates(t)[0], *rates(p), rates([(lexicon(h), r) for h, r in p])[1], np.mean(diff)])
    r = np.array(res); m = r.mean(0)
    print(f"{noise:4.2f}   {m[0]:5.1f} ({r[:,0].min():4.1f}-{r[:,0].max():4.1f})"
          f"   {m[1]:5.1f} ({r[:,1].min():4.1f}-{r[:,1].max():4.1f})  {m[2]:5.1f}   {m[3]:5.1f}"
          f"        {m[4]:+6.1f}")

print("\nB. Skew, noise 0.1. CER % as mean over seeds (min-max); segments minus characters; angle error")
print("skew    no deskew        segments-chars   deskewed          mean |angle error|")
for skew in [0, 1, 2, 3, 5, 7]:
    res = []
    for seed in SEEDS:
        rng = np.random.default_rng(100 + seed)
        a, b, err, diff = [], [], [], []
        for _ in range(LINES):
            words = list(rng.choice(VOCAB, 6)); ref = " ".join(words)
            img = degrade(render(words), rng, 0.1, BLUR, skew)
            hyp = recognize(img); a.append((hyp, ref)); diff.append(len(hyp) - len(ref))
            b.append((recognize(img, fix_skew=True), ref))
            err.append(abs(skew_angle(binarize(img)) - skew))
        res.append([rates(a)[0], np.mean(diff), rates(b)[0], np.mean(err)])
    r = np.array(res); m = r.mean(0)
    print(f"{skew:3d} deg  {m[0]:5.1f} ({r[:,0].min():4.1f}-{r[:,0].max():4.1f})     {m[1]:+5.1f}"
          f"       {m[2]:4.1f} ({r[:,2].min():3.1f}-{r[:,2].max():3.1f})     {m[3]:.2f} deg")

rng = np.random.default_rng(7)
words = list(rng.choice(VOCAB, 6))
img = degrade(render(words), rng, 0.3, BLUR, 0)
hyp = recognize(img)
print(f"\nexample line, noise 0.3\n  truth      : {' '.join(words)}\n  true cuts  : {recognize_true_cuts(img, words)}"
      f"\n  projection : {hyp}\n  + lexicon  : {lexicon(hyp)}")
rng = np.random.default_rng(100)
words = list(rng.choice(VOCAB, 6))
img = degrade(render(words), rng, 0.1, BLUR, 3)
print(f"example line, skew 3 deg\n  truth      : {' '.join(words)}\n  no deskew  : {recognize(img)}"
      f"\n  deskewed   : {recognize(img, fix_skew=True)}")

Output:

seeds [0, 1, 2, 3, 4] (A), [100, 101, 102, 103, 104] (B), 7 and 100 (examples); 20 lines of 6 words per seed; blur sigma 1.0 px; intensities in [0, 1]

A. Noise, no skew. CER % as mean over seeds (min-max); WER %; segments minus characters per line
noise   true cuts         projection cuts   WER   WER+lexicon   segments-chars
0.10     0.0 ( 0.0- 0.0)     0.1 ( 0.0- 0.3)    0.2     0.0          +0.0
0.20     0.0 ( 0.0- 0.0)     1.6 ( 1.0- 2.5)    5.5     1.3          -0.1
0.25     0.0 ( 0.0- 0.0)    19.0 (17.2-20.7)   48.7    16.7          -3.0
0.30     0.0 ( 0.0- 0.2)    46.9 (39.8-51.3)   87.8    57.3          -7.9
0.40     0.5 ( 0.3- 0.7)    80.4 (77.7-83.3)  100.0    91.3         -12.4
0.50     0.9 ( 0.7- 1.3)    91.4 (90.8-92.1)  100.0    96.0         -19.0

B. Skew, noise 0.1. CER % as mean over seeds (min-max); segments minus characters; angle error
skew    no deskew        segments-chars   deskewed          mean |angle error|
  0 deg    0.0 ( 0.0- 0.0)      +0.0        0.0 (0.0-0.0)     0.00 deg
  1 deg    0.3 ( 0.0- 0.6)      +0.0        0.0 (0.0-0.0)     0.00 deg
  2 deg    3.6 ( 1.9- 4.8)      +0.3        0.0 (0.0-0.0)     0.00 deg
  3 deg    7.1 ( 5.5- 9.5)      +0.6        0.0 (0.0-0.0)     0.00 deg
  5 deg    8.4 ( 7.5- 9.5)      +0.1        0.0 (0.0-0.0)     0.00 deg
  7 deg   36.5 (35.3-38.5)      -5.5        0.0 (0.0-0.0)     0.00 deg

example line, noise 0.3
  truth      : VIEW MODEL SHAPE BOX DATA LETTER
  true cuts  : VIEW MODEL SHAPE BOX DATA LETTER
  projection : VIEW IPSHAF WX WTA LETBTER
  + lexicon  : VIEW SCAN WAX DATA LETTER
example line, skew 3 deg
  truth      : LETTER ZERO TEXT DATA IMAGE FONT
  no deskew  : LETTER ZERO TEXT DATA IMAGE FONBI
  deskewed   : LETTER ZERO TEXT DATA IMAGE FONT

With the true cuts, the template classifier stays at or below 1.3% character error for every seed, even at noise 0.5, where the line is barely legible. With cuts found from the projection profile, the character error rate climbs from 1.6% at noise 0.2 to about 47% at 0.3, and the count of segments falls well below the count of characters. Our reading of the failures: noise pixels that touch strokes survive the speck filter and fill the one-cell gaps, so neighbouring letters merge into one segment, which is then forced into a single class. Some segments also split, as in LETBTER. In this synthetic setup the classifier is not the bottleneck; the cuts are, which is the problem the segmentation-free methods below remove.

The word error rate rises faster than the character error rate, because one wrong character spoils a word. The lexicon repairs words with one or two character errors (DATA from WTA), but can choose the wrong word (WAX for BOX), and it cannot repair merges across a word gap or words that lost most of their letters, so at high noise it barely helps. Here the lexicon is closed, containing every word that occurs; an open one could also replace correct words. Skew of one degree costs almost nothing. From 2° to 5° there are slightly more segments than characters; in our reading, the line drifts out of the horizontal text band at its ends, and end letters are cut or split (FONT read as FONBI at 3°). At 7° segments merge, and the error rate jumps. The projection-profile estimate recovers each angle exactly here, since the true angles lie on its search grid and the lines are clean, and rotating the gray image back before binarizing removes the skew errors. The numbers depend entirely on the synthetic font and noise model, and the finding that segmentation causes most errors is shown only for this setup; Casey and Lecolinet (1996) report a similar observation by researchers of the 1960s and 1970s, that segmentation caused more errors than shape distortions in reading unconstrained characters.

Segmentation-free line recognition

The alternative to cutting characters is to recognize a line as a sequence, and to let the model decide where characters are.

Hidden Markov models. HMM recognizers, carried over from speech recognition, slide a narrow window across a normalized text line and model the resulting feature sequence with one linear HMM per character, chained into the words of a dictionary. The HMM baseline of Graves et al. (2009) illustrates the design: for offline handwriting, a window one pixel wide computes nine geometric features per column (mean gray value, centre of gravity, positions of the uppermost and lowermost ink pixels, and others), each character has its own number of states, and observation probabilities are mixtures of diagonal Gaussians. Decoding finds the best path through these models, so the character boundaries are decided together with the characters. Smith (2007) named an HMM-based character n-gram model as a likely way to improve Tesseract.

Recurrent networks with connectionist temporal classification. Graves et al. (2006) introduced connectionist temporal classification (CTC), a training objective that lets a recurrent network label an unsegmented sequence directly. The network’s softmax output at each time step $t$ has one unit per label plus a blank, giving probabilities $y_k^t$. A path $\pi$ is one label or blank per time step, and a many-to-one map $\mathcal{B}$ removes repeated labels and then blanks, so that, writing - for the blank, the paths a-ab- and -aa--abb both map to aab. The probability of a labelling $\mathbf{l}$ for input $\mathbf{x}$ sums over all paths that collapse to it:

$$p(\mathbf{l} \mid \mathbf{x}) = \sum_{\pi \in \mathcal{B}^{-1}(\mathbf{l})} \prod_{t=1}^{T} y_{\pi_t}^{t}.$$

A forward-backward recursion like the one for HMMs computes this sum and its gradient, and training maximizes the log-probability of the correct transcriptions. Decoding is approximate: best path decoding takes the most active output at each step and collapses it; prefix search is slower but generally better. The 2006 experiments were on speech, phonetic labelling of the TIMIT corpus, where CTC with prefix search reached a mean label error rate of 30.51% over five runs, against 31.57% for the best hybrid of an HMM and the same recurrent network (trained with a weighted error signal) and 35.21% for a context-dependent HMM. Graves et al. (2009) applied bidirectional LSTM networks with CTC to handwriting lines and reported, with a 20,000-word open dictionary and a bigram language model, 74.1% word accuracy on offline IAM handwriting against 64.5% for an HMM system, and 79.7% against 65.0% on online data.

Tesseract 4. Tesseract, released as open source by HP in 2005 (Smith, 2007), gained an LSTM line recognizer in version 4.00. Its documentation describes the recognizer as originating in OCRopus’s Python LSTM implementation, redesigned in C++, and integrated as a line recognizer that works with the existing layout analysis or on a single text line; it became the default engine, while the legacy engine remains selectable with --oem 0 when the model files include the legacy models. Training uses CTC (the trainer displays CTC outputs and targets), and the Latin-script models were trained on about 400,000 text lines spanning about 4,500 fonts. The documentation puts the compute cost on the order of ten times the CPU of the legacy engine. Tesseract 4.0.0 was released on 29 October 2018, and the newest release is 5.5.3, of July 2026 (as of October 2026).

Convolutional recurrent networks. Shi et al. (2017) replaced hand-designed features with convolutional ones in CRNN: the convolutional layers of a CNN, without fully connected layers, turn a word image scaled to a height of 32 pixels into feature maps, each column of the maps becomes one feature vector of a left-to-right sequence, a deep bidirectional LSTM predicts a label distribution per column, and a CTC transcription layer gives the string. Each column corresponds to a receptive field in the image, so in our reading the network scans a classifier along the word, the role windows play in sliding window detection. CRNN was trained only on 8 million synthetic word images from Jaderberg et al.; Table 2 of the arXiv version (v1, 2015) reports lexicon-free word accuracies of 78.2% on IIIT5k, 80.8% on SVT, 89.4% on ICDAR 2003 and 86.7% on ICDAR 2013, and its Table 3 gives the model size as 8.3 million parameters.

Scene text

Text in natural images breaks the assumptions of the document pipeline: there is no page, no clean background to threshold, and words can be at any angle, scale and curvature. Systems split into detection and recognition, or train both together.

Detection. Scene text detectors adapt object detectors to long, thin and rotated boxes. EAST (Zhou et al., 2017) uses a fully convolutional network that predicts, at every pixel, a text score and the geometry of a rotated rectangle or a quadrangle, followed by a locality-aware variant of non-maximum suppression. On the ICDAR 2015 incidental scene text localization task (Challenge 4) the authors report recall 0.7347, precision 0.8357 and F-score 0.7820 at single scale with a PVANet2x backbone, 0.8072 with multi-scale testing, and 13.2 frames per second at 720p on a Titan X GPU (Tables 3 and 6 of the CVPR version). Oriented boxes are covered in bounding box.

Recognition with attention. Instead of CTC, attention-based recognizers generate the string one character at a time with a recurrent decoder that attends to a different part of the encoded image at each step, the same encoder-decoder pattern as in image captioning. RARE (Shi et al., 2016) adds a spatial transformer network in front that predicts a thin-plate-spline transformation to rectify perspective and curved text before an attention-based sequence recognizer reads it, trained end to end from images and their transcriptions only.

Transformers. TrOCR (Li et al., 2023) uses the transformer encoder-decoder for both parts: an image transformer initialized from BEiT or DeiT encodes image patches, as in a vision transformer, and a text transformer initialized from RoBERTa or MiniLM generates wordpieces. Its first pre-training stage used 684 million printed text lines cropped from two million PDF pages, and a second stage used task-specific data: synthetic handwritten lines, receipt text lines and synthetic scene text. On the IAM handwriting database it reports a case-sensitive character error rate of 2.89% for TrOCR-Large (558 million parameters, trained on synthetic data and IAM, no external language model, Aachen partition; Table 4), against 3.2% for an earlier CTC system with an external language model and 2.75% for a system trained with additional internal labelled data and a language model. Large vision-language models now read text in images as part of answering questions about them.

Document understanding

Recognized text is often only the input to a further task. LayoutLM (Xu et al., 2020) feeds each word from OCR together with its 2D bounding box, and optionally image features, to a pre-trained transformer. The arXiv version (v5, Tables 1, 4 and 5) reports F1 scores of 79.27 on FUNSD form understanding (base model with image features) and 95.24 on SROIE Task 3 receipt information extraction (large model, text and layout only), and 94.42% accuracy on RVL-CDIP document image classification (base model with image features). For SROIE the authors trained on the ground-truth OCR to remove the effect of recognition errors, a reminder that OCR mistakes propagate downstream.

Datasets and Benchmarks

Benchmark	Content	Split or protocol	Typical metric
MNIST	isolated handwritten digits, $28 \times 28$	60,000 train, 10,000 test	error rate
UNLV OCR test (1995)	300 DPI binary page images	four test sets (bus, doe, mag, news)	character and word errors
IAM Handwriting Database 3.0	657 writers, 1,539 pages, 13,353 lines, 115,320 words	Large Writer Independent Text Line Recognition Task: 6,161 / 900 / 940 / 1,861 lines	CER, WER
ICDAR 2013 Focused Scene Text	462 well-captured scene images focused on the text	229 train, 233 test	localization F-score, total edit distance, word accuracy
ICDAR 2015 Incidental Scene Text	1,670 Google Glass images, 17,548 annotated regions	1,000 train, 500 test, 170 sequestered	IoU-based F-score, normalized edit distance, end-to-end F-score
Cropped scene words	IIIT5k, SVT, IC03, IC13, IC15, SVT-Perspective, CUTE80	test crops, often with lexicons	word accuracy

MNIST is the standard isolated-digit benchmark, described in image classification and LeNet.

UNLV. Tesseract, then an HP research prototype, was entered in the fourth annual test of OCR accuracy run at the University of Nevada, Las Vegas, in 1995. Smith (2007, Table 1) compares the 1995 results with Tesseract 2.0 on the four 300 DPI binary sets: character error rates were 1.31% to 2.48% in 1995 and 1.61% to 2.22% for version 2.0, whose total number of character errors was 7.31% lower than in 1995.

IAM. The IAM Handwriting Database (Marti and Bunke, 2002) contains English handwriting of sentences from the LOB corpus, scanned at 300 dpi; its official page defines the Large Writer Independent Text Line Recognition Task as 9,862 lines in four sets, with each writer in one set only: 6,161 training lines from 283 writers, validation sets of 900 and 940 lines, and 1,861 test lines from 128 writers. Published results use other partitions too. Graves et al. (2009) describe a test set of 2,781 lines from 161 writers under the same task name, and TrOCR uses the Aachen partition of 6,161, 966 and 2,915 lines, so IAM error rates from different papers are not always comparable.

ICDAR Robust Reading. The ICDAR Robust Reading Competition series, which its 2015 report dates back to 2003, defines the scene text benchmarks most papers use. The 2013 Focused Scene Text dataset (Challenge 2) has 462 images, 229 for training and 233 for testing (Karatzas et al., 2013), well-captured images focused on the text content, as the 2015 report contrasts them with incidental text. The 2015 Incidental Scene Text dataset (Challenge 4) has 1,670 images taken with Google Glass in Singapore without aiming at the text, with 17,548 annotated regions; 1,000 training and 500 test images are public and 170 are kept private, words are annotated as quadrilaterals, and text in scripts other than Latin is marked “do not care” (Karatzas et al., 2015). Incidental text proved much harder. In word recognition the best 2013 entry recognized 82.83% of the focused words correctly, the 2013 report calling this double the 2011 winner’s 41%, while the best 2015 entry recognized 32.93% of the incidental words.

Cropped word sets and synthetic training data. Recognition papers such as CRNN and TrOCR report word accuracy on cropped word images from IIIT5k (3,000 test words), SVT, ICDAR 2003, 2013 and 2015, and the irregular-text sets SVT-Perspective and CUTE80, often training only on synthetic word images.

Evaluation Metrics

Most OCR metrics rest on the edit distance, or Levenshtein distance, $\mathrm{ED}(h, r)$: the minimum number of insertions, deletions and substitutions that turn the hypothesis $h$ into the reference $r$. Over a test set, the character error rate (CER) and word error rate (WER) divide the summed distances by the total reference length, counted in characters or in words:

$$\mathrm{CER} = \frac{\sum_i \mathrm{ED}_{\text{char}}(h_i, r_i)}{\sum_i |r_i|_{\text{char}}}, \qquad \mathrm{WER} = \frac{\sum_i \mathrm{ED}_{\text{word}}(h_i, r_i)}{\sum_i |r_i|_{\text{word}}}.$$

Graves et al. (2006) define their label error rate in exactly this way, and Graves et al. (2009) report word and character accuracy as one minus these rates. Because insertions count, the rates can exceed 100%. Several conventions change the numbers:

• Case and character set. ICDAR 2015 word recognition compares case-sensitively, while its end-to-end task ignores case; TrOCR’s scene text results filter outputs to 36 lowercase alphanumeric characters, but its IAM error rates are case-sensitive.
• How distances are pooled. The ICDAR 2013 and 2015 word recognition tasks rank methods by total edit distance with equal costs for all operations, which the competition reports specify as each word’s distance normalized by the length of its ground truth and summed over the test words; the percentage of correctly recognized words is reported alongside. CER instead pools raw distances over all characters.
• Lexicons and language models. CRNN’s table separates results with 50-word, 1,000-word, 50,000-word and full lexicons from lexicon-free ones, and TrOCR’s IAM table marks which systems used an external language model.
• Data split and training data, as for IAM above.

Detection is scored like object detection, with precision, recall and F-score. ICDAR 2015 Challenge 4 uses a single intersection over union criterion with a threshold of 50%, as in PASCAL VOC, while the 2013 focused text task used the evaluation method of Wolf and Jolion. End-to-end recognition in both challenges follows the protocol that the competition pages attribute to Wang et al. (2011): a detection counts if it overlaps a ground-truth word by more than 50% and the words match, ignoring case, with “do not care” regions excluded. Three vocabulary settings are reported: a strongly contextualized list of 100 words per image, a weakly contextualized list of all test-set words, and a generic vocabulary of about 90,000 words.

These protocols make published numbers hard to compare. For instance, TrOCR-Large trained on synthetic data reports 84.0% word accuracy on the 1,811-word and 78.0% on the 2,077-word versions of the ICDAR 2015 recognition test (Li et al., 2023, Table 6), with outputs filtered to 36 lowercase alphanumeric characters, while the competition’s case-sensitive 32.93% came from 2015 entries; the gap mixes years of progress with differences in protocol.

Applications

• Digitizing documents and archives: books, newspapers and records made searchable.
• Postal and banking automation: reading zip codes and check amounts, where the work behind LeNet was deployed in check readers.
• Forms, receipts and invoices: extracting fields from scanned business documents, the target of SROIE and FUNSD.
• Scene text: translation on demand, which the ICDAR 2015 report names as an application of focused text, wearable cameras and massive urban captures, which it names for incidental text, and the geo-location, driverless car and image-based translation uses cited by Shi et al. (2016).
• Question answering and accessibility: answering questions that require reading text in photos, as in TextVQA and VizWiz (visual question answering).

Open Problems

• Handwriting and degraded documents. Lines of unconstrained handwriting remain harder than isolated characters, as Graves et al. (2009) note, and in our reading degraded historical material adds damage and unusual scripts with little labelled data.
• Irregular scene text: curved, perspective-distorted, occluded or very small text, the subject of rectification models such as RARE.
• Many scripts. The ICDAR 2015 incidental text benchmark treated non-Latin text as “do not care”, and scripts with large character sets need special encodings, as Tesseract’s recoding shows.
• Comparable evaluation: differences in case handling, character filtering, lexicons and splits often matter as much as the method, as the sections above show.
• Error propagation: document understanding and question answering systems inherit OCR mistakes, as LayoutLM’s use of ground-truth OCR for SROIE illustrates, and models that answer directly from pixels still have to be evaluated on how well they read.