Character Error Rate

The character error rate (CER) counts how many single-character edits, substitutions, deletions and insertions, turn a recognizer’s output into the reference text, and divides that count by the length of the reference. Line and page recognition in optical character recognition and handwriting recognition is reported with it, since an output is rarely all right or all wrong. The count itself is the edit distance, also called the Levenshtein distance; computed over words instead of characters, the same ratio is the word error rate (WER), and over the labels of a sequence model it is the label error rate of connectionist temporal classification. A CER of 0 means a perfect transcription; there is no upper limit, because a recognizer can insert any number of characters. The definition looks fixed, but the reported number depends on choices that are easy to leave unstated: how distances are averaged over lines, how they are normalized, which of several optimal alignments supplies the error breakdown, and how the text is normalized before it is compared.

Formula

Edit distance. For a reference string $r = r_1 \dots r_n$ and a hypothesis $h = h_1 \dots h_m$, let $D[i, j]$ be the minimum number of edits that turn the first $i$ characters of $r$ into the first $j$ characters of $h$. The dynamic programming recurrence in Jurafsky and Martin’s textbook (§2.9, Eq. 2.19), with unit costs, is

$$D[i, 0] = i, \qquad D[0, j] = j, \qquad D[i, j] = \min \begin{cases} D[i-1, j] + 1 & \text{(deletion of } r_i\text{)} \\ D[i, j-1] + 1 & \text{(insertion of } h_j\text{)} \\ D[i-1, j-1] + [\, r_i \ne h_j \,] & \text{(substitution, or match at cost 0)} \end{cases}$$

and $\mathrm{ED}(r, h) = D[n, m]$. The table has $(n+1)(m+1)$ cells, each filled from three neighbours, so the cost is $O(nm)$ time; the value alone needs only two rows of memory.

Jurafsky and Martin attribute the unit-cost distance to Levenshtein (1966; their reference list gives the 1965 original in Doklady Akademii Nauk SSSR) and report that he also proposed a version without substitutions, equivalent to a substitution cost of 2: under it the distance from intention to execution is 8 instead of 5. They write that the minimum edit distance algorithm was named by Wagner and Fischer (1974) and discovered independently by many people. Neither original was read for this article.

Error counts. A path through the table from $D[n, m]$ back to $D[0, 0]$ along cells that achieve the minimum, the backtrace, is an optimal alignment. Each diagonal step is a hit $H$ (equal characters) or a substitution $S$, each vertical step a deletion $D$ (a reference character missing from the output) and each horizontal step an insertion $I$ (an extra output character). Along any optimal path $S + D + I = \mathrm{ED}$ and $H + S + D = n$.

Character and word error rate. For one line,

$$\mathrm{CER} = \frac{S + D + I}{N} = \frac{S + D + I}{H + S + D},$$

where $N = n$ is the number of reference characters. WER is the same ratio with words as the symbols. Over a test set of lines $i = 1, \dots, K$, the usual form pools the counts; this is the definition in the Optical Character Recognition article, which reports that Graves et al. (2006) define their label error rate the same way:

$$\mathrm{CER}_{\text{pooled}} = \frac{\sum_i \mathrm{ED}(r_i, h_i)}{\sum_i |r_i|}, \qquad \mathrm{CER}_{\text{line mean}} = \frac{1}{K} \sum_i \frac{\mathrm{ED}(r_i, h_i)}{|r_i|}.$$

The second form, a mean of per-line rates, is also in use, and the two can rank systems differently (see Example).

Range. $\mathrm{CER} \ge 0$, with 0 only for an exact match. Deletions and substitutions are bounded by $N$, but insertions are not, so CER can exceed 1: a reference m read as rn has one substitution and one insertion, a CER of 200%. Character accuracy, $1 - \mathrm{CER}$, is then negative.

Intuition

The edit distance is the number of keystrokes a proofreader needs to correct the output, if each keystroke replaces, deletes or types one character, and CER expresses those keystrokes per character of correct text. A CER of 2% means about one correction every fifty characters. Because the reference length is the denominator, the measure asks how much of the true text was damaged, not how much text the system wrote, which is why insertions can push it past 100%.

WER asks the same question about words. One wrong character spoils a whole word, so in our reading WER is the larger of the two on running text; the Handwriting Recognition article gives a READ 2016 line with a CER of 4.2% and a WER of 50%, caused by two words run together.

Interpretation

• 0 is a perfect transcription. Small values compare well only within one protocol: the same character set, case handling, normalization and averaging rule.
• Values above 100% occur when insertions dominate, as for garbage output on a short line. In our reading they matter mostly for single short items, where a few insertions outnumber the reference characters.
• Character accuracy in the ISRI tradition, still computed by the ocreval port of the ISRI Analytic Tools, is $(n - \text{errors})/n$ as a percentage (accrpt.c), so it is the complement of pooled CER when the errors are counted by edit distance. ocreval counts errors from its own synchronization of the two texts rather than from a Levenshtein table (accuracy.c, at commit 873a0de); in each unmatched segment it counts the larger of the two segment lengths as errors.
• Different breakdowns, same total. Two optimal alignments can split the same distance into different $S$, $D$ and $I$. The total, and therefore CER and WER, is unique; the breakdown is not.
• Naming of insertions. In jiwer and in this article an insertion is an extra output character. In our reading of ocreval’s accuracy.c, ISRI names the operations as corrections of the OCR output: its count of insertions is the number of reference characters missing from the output (ins = max(0, characters - genchars)), what jiwer calls deletions. A breakdown copied from one tool to another can therefore swap $D$ and $I$.

When to Use It

CER suits any task whose output is a character sequence of variable length and whose errors are partial: printed and handwritten line and page recognition, receipts and forms, licence plates, and the transcription stage of document pipelines. WER is reported beside it when words matter downstream, for search or language understanding. The measures are not the primary score everywhere. Scene text recognition takes word accuracy as its primary metric, the share of word crops read exactly; the ICDAR word recognition tasks rank by a total of per-word normalized distances; and document question answering uses ANLS, a thresholded similarity built on the normalized Levenshtein distance (document understanding).

Failure Cases

• The averaging rule changes the ranking. Pooling weights each line by its length, the per-line mean weights every line equally. On a test set with many short lines, a system that makes one mistake on every line looks bad under the per-line mean and good under pooling, and a system that fails only on long lines the reverse. Handwriting Recognition notes that Coquenet et al. (2023) pool so that errors in short lines do not weigh more than errors in long ones; the Example shows the two rules ranking two systems in opposite orders.
• Short items dominate normalized totals. Any score that normalizes per item, the per-line mean, ICDAR’s total edit distance and NED, gives a one-character error in a three-character word the weight of three such errors in a nine-character word.
• Insertions are unbounded. A system that emits long garbage on a few items can exceed 100% CER on them and dominate a per-line mean; NED, which divides by the longer string, caps each item at 1.
• Normalization is part of the metric. Case, whitespace, punctuation, Unicode form and the character set all change the count. Python’s len counts Unicode code points (Jurafsky and Martin, §2.3), and the same accented letter can be one code point (é, composed) or two (e followed by a combining accent, decomposed); comparing a composed reference with a decomposed output costs two edits per accented letter. The Example’s normalization chain shows the effect.
• Transpositions count twice. Swapped neighbours, form read as from, cost two edits under Levenshtein’s operations. NLTK’s edit_distance can count a swap as one edit with transpositions=True, a Damerau-style variant that gives a different number for the same output.
• Tie-breaks change the breakdown and the measures built on it. Tools that report $S$, $D$ and $I$, and measures that use them separately, depend on which optimal alignment the backtrace picks. jiwer takes its counts from rapidfuzz.distance.Levenshtein.opcodes, one particular alignment. MER and WIL (Variants) use the counts individually, so in principle they can change with the tie-break even when WER cannot; in the Example they did not.
• Word tokenization. WER depends on how the text is cut into words: whether punctuation marks are words, as in the READ 2016 protocol described in Handwriting Recognition, and how runs of spaces are treated.
• Empty references. CER is undefined when the reference is empty. jiwer returns the number of insertions as the error rate when the references contain no words at all, and PARSeq’s NED divides by the longer length, which is zero only when both strings are empty.

Variants

Measure	Per item	Aggregation	Range	Source of the definition here
CER, WER (pooled)	$\mathrm{ED}$	$\sum \mathrm{ED} / \sum \lvert r \rvert$	$[0, \infty)$	OCR article (Graves et al., 2006, as it reports); jiwer
Per-line mean CER	$\mathrm{ED}/\lvert r \rvert$	mean	$[0, \infty)$	—
ICDAR 2013 total edit distance	$\mathrm{ED}/\lvert r \rvert$ per word	sum	$[0, \infty)$	Karatzas et al. (2013), via ICDAR article
Normalized edit distance (NED)	$\mathrm{ED}/\max(\lvert h \rvert, \lvert r \rvert)$	mean, reported as $1 - \text{NED}$	$[0, 1]$	PARSeq code
Character accuracy	—	$(n - \text{errors})/n$	$(-\infty, 1]$	ocreval (ISRI)
MER	—	$(S+D+I)/(H+S+D+I)$	$[0, 1]$	jiwer
WIL	—	$1 - \frac{H}{N_{\text{ref}}}\frac{H}{N_{\text{hyp}}}$	$[0, 1]$	jiwer

Normalized edit distance. PARSeq’s evaluation code, with a comment that it follows the ICDAR 2019 definition of N.E.D., divides each word’s distance by the length of the longer of prediction and ground truth, using NLTK’s edit_distance, averages over the test words and reports $100(1 - \overline{\mathrm{NED}})$; its “Combined” row weights benchmarks by their number of samples (base.py, test.py at v1.0.0). The ICDAR 2019 reports were not read. Because $\max(|h|, |r|) \ge |r|$, each item’s NED is at most its CER, and it is at most 1. ANLS uses the same normalized distance (Document Understanding).

ICDAR 2013. The 2013 Robust Reading report normalizes each word’s distance by the length of its ground-truth transcription and sums over all test words, case-sensitively (ICDAR). Over $K$ items it is $K$ times the per-item mean CER.

MER and WIL. jiwer (v4.0.0, process.py) pools $H$, $S$, $D$ and $I$ over sentences and computes, besides WER, the match error rate $\mathrm{MER} = (S+D+I)/(H+S+D+I)$ and the word information lost $\mathrm{WIL} = 1 - \mathrm{WIP}$, with word information preserved $\mathrm{WIP} = (H/N_{\text{ref}})(H/N_{\text{hyp}})$, where $N_{\text{ref}} = H+S+D$ and $N_{\text{hyp}} = H+S+I$. MER has the insertions in its denominator as well, so it lies in $[0, 1]$ and equals WER when $I = 0$; WIL is 0 only for a perfect output. jiwer computes CER with the same code on lists of characters. Its default character transform only strips leading and trailing whitespace, so inner spaces count as characters and case and Unicode form are left alone; its default word transform also collapses runs of spaces before splitting (transformations.py). Its docstrings describe the counts as edits that transform the hypothesis into the reference, while the code takes them from opcodes of the reference against the hypothesis, whose “insert” is an extra hypothesis word; in our reading the docstring’s direction is reversed relative to the code, and the counts follow the convention used in this article.

Weighted and extended costs. The recurrence accepts any costs: Levenshtein’s substitution-free version (cost 2), letter-specific costs, or the transposition step of NLTK’s option. Each defines a different metric, so a reported CER should say which; NLTK’s default, which PARSeq’s code uses, is unit cost without transpositions.

Normalization and character sets. Scene text recognition evaluates on nested character sets (36, 62 or 94 characters in PARSeq’s test.py), and its benchmarks were long scored on the 36 case-insensitive alphanumeric characters (Scene Text Recognition article); handwriting benchmarks such as IAM are scored case-sensitively in TrOCR (Optical Character Recognition article). A score is comparable to another only under the same normalization.

Example

The script below scores one fixed set of outputs from two simulated recognizers in many ways. The images are glyphs of Pillow 12.0.0’s bundled bitmap font (ImageFont.load_default_imagefont(), courB08), each drawn into an $8 \times 12$ cell, and a line image is its cells side by side. A reader flips each pixel with probability $p$ and assigns the nearest of 27 templates (26 lowercase letters and a blank for the space) by Hamming distance. The test set has 40 one-word lines of 3–5 letters and 20 lines of eight words. The two systems err in known ways:

• A reads with $p = 0.05$ and splits the first glyph of every line into its left and right halves, each read as a character: one insertion, plus a substitution when neither half reads as the true letter.
• B reads with $p = 0.05$ up to character 20 and $p = 0.20$ beyond it, merges a letter pair at or after characters 24, 32 and 40 (the pixels of two cells combined and read as one character: a deletion, plus a substitution when the result is neither letter), and misses the first word gap at or after character 20.

So A’s injected error comes once per line whatever its length, and B’s extra noise and injected errors come only where long lines extend; at $p = 0.05$ either system can misread any character. The construction gives each line an edit count, which bounds the true distance from above. The code computes the distance with its own dynamic programme, checked against NLTK’s edit_distance and against the definition written as a plain recursion; it extracts $H$, $S$, $D$ and $I$ with two tie-break orders for the backtrace, “diagonal first” (prefer a match or substitution when it is optimal, then a deletion, then an insertion) and “gaps first” (deletion, insertion, then diagonal); and it ports jiwer’s MER and WIL and PARSeq’s NED with their licence notices. A loop over seeds 0–9 regenerates text and noise, and a last block runs the edge cases discussed above. Characters are Python code points. The output came from Python 3.13.9 with NumPy 2.3.5, Pillow 12.0.0 and NLTK 3.9.2; two runs printed identical output, and the script runs in under 2 seconds on our machine (approximate).

The parameters, the noise levels, the event positions and the line lengths were chosen at planning so that A errs on short lines and B on long ones, and a planning script that used only the construction’s edit counts printed the ranking flip of the first hypothesis, and its persistence over the ten seeds, before the script below was run. The example is therefore an illustration of that construction, not a test of it. One change was made at planning, before any run of this script: the first planning version cut 6-pixel cells from lines drawn by Pillow as whole strings, and a check showed that glyph ink spilled into neighbouring cells, so the cells were widened to 8 pixels and lines typeset from single-glyph cells.

import os
os.environ["OPENBLAS_NUM_THREADS"] = "1"; os.environ["OMP_NUM_THREADS"] = "1"
import string, unicodedata
import numpy as np
from PIL import Image, ImageDraw, ImageFont
from nltk.metrics.distance import edit_distance as nltk_edit_distance

# ---- Glyph images and two simulated recognizers ----------------------------------------------------
F = ImageFont.load_default_imagefont()            # Pillow's bundled bitmap font (courB08)
W, H = 8, 12                                       # one glyph per 8 x 12 cell, drawn at x = 1
CLASSES = list(string.ascii_lowercase) + [" "]

def glyph(c, canvas=(W, H), x=1):
    im = Image.new("L", canvas, 0); ImageDraw.Draw(im).text((x, 0), c, fill=255, font=F)
    return np.array(im) > 127

TEMPL = np.stack([glyph(c) for c in CLASSES])      # 27 templates; the space is a blank cell

def render(text):                                  # a line image is its glyph cells side by side
    return np.concatenate([TEMPL[CLASSES.index(c)] for c in text], 1)

def read_cell(cell, p, rng):                       # flip each pixel with probability p, nearest template
    noisy = cell ^ (rng.random(cell.shape) < p)
    return CLASSES[int((noisy[None] != TEMPL).sum((1, 2)).argmin())]

VOCAB = ("box hue ink dark blur zoom crop grid edge mask code band text page scan font word line chart "
         "depth noise shape frame color point label pixel glyph light focus").split()

def make_lines(seed):                              # 40 one-word lines and 20 eight-word lines
    rng = np.random.default_rng(seed)
    short = [str(rng.choice([w for w in VOCAB if len(w) <= 5])) for _ in range(40)]
    return short + [" ".join(rng.choice(VOCAB, 8)) for _ in range(20)]

def system_A(ref, rng):
    """Errs on short lines: the first glyph of every line is split in two halves, each read as a character."""
    img, out, cost = render(ref), [], 0
    for k, r in enumerate(ref):
        cell = img[:, W * k:W * (k + 1)]
        if k == 0:
            left, right = cell.copy(), cell.copy(); left[:, W // 2:] = False; right[:, :W // 2] = False
            o = read_cell(left, 0.05, rng) + read_cell(right, 0.05, rng); cost += 1 + (r not in o)
        else:
            o = read_cell(cell, 0.05, rng); cost += (o != r)
        out.append(o)
    return "".join(out), cost, (1, 0, 0)

def system_B(ref, rng):
    """Errs on long lines: more noise from character 20 on, letter pairs merged at or after 24, 32 and 40,
    and the first space at or after 20 dropped."""
    merges, k = [], 24
    for start in (24, 32, 40):
        k = max(k, start)
        while k + 1 < len(ref) and not (ref[k] != " " and ref[k + 1] != " "):
            k += 1
        if k + 1 < len(ref):
            merges.append(k); k += 2
    spaces = [i for i, c in enumerate(ref) if c == " " and i >= 20]
    drop = spaces[0] if spaces else None
    img, out, cost, k = render(ref), [], 0, 0
    while k < len(ref):
        p = 0.05 if k < 20 else 0.20
        if k in merges:                            # two touching glyphs read as one character
            o = read_cell(img[:, W * k:W * (k + 1)] | img[:, W * (k + 1):W * (k + 2)], p, rng)
            out.append(o); cost += 1 + (o not in ref[k:k + 2]); k += 2
        elif k == drop:                            # the word gap is missed
            cost += 1; k += 1
        else:
            o = read_cell(img[:, W * k:W * (k + 1)], p, rng); out.append(o); cost += (o != ref[k]); k += 1
    return "".join(out), cost, (0, len(merges), int(drop is not None))

def outputs(seed):
    refs = make_lines(seed); res = {}
    for name, system in (("A", system_A), ("B", system_B)):
        rng = np.random.default_rng([seed, ord(name)])
        res[name] = [system(r, rng) for r in refs]
    return refs, res

# ---- Edit distance, alignments and the measures ---------------------------------------------------
def dp_table(ref, hyp, sub=1):
    D = [[i + j if i * j == 0 else 0 for j in range(len(hyp) + 1)] for i in range(len(ref) + 1)]
    for i in range(1, len(ref) + 1):
        for j in range(1, len(hyp) + 1):
            D[i][j] = min(D[i - 1][j] + 1, D[i][j - 1] + 1, D[i - 1][j - 1] + (sub if ref[i - 1] != hyp[j - 1] else 0))
    return D

def ed(ref, hyp, sub=1):
    return dp_table(ref, hyp, sub)[-1][-1]

def backtrace(ref, hyp, order):
    """One optimal alignment. order: preference among optimal moves, e.g. ("diag", "del", "ins").
    Returns (H, S, D, I) and the hypothesis rebuilt by applying the edit script to the reference."""
    T = dp_table(ref, hyp); i, j = len(ref), len(hyp); n = dict(H=0, S=0, D=0, I=0); rebuilt = []
    while i or j:
        ok = {"diag": i and j and T[i - 1][j - 1] + (ref[i - 1] != hyp[j - 1]) == T[i][j],
              "del": i and T[i - 1][j] + 1 == T[i][j], "ins": j and T[i][j - 1] + 1 == T[i][j]}
        move = next(m for m in order if ok[m])
        if move == "diag":
            n["H" if ref[i - 1] == hyp[j - 1] else "S"] += 1; rebuilt.append(hyp[j - 1]); i -= 1; j -= 1
        elif move == "del":                        # reference symbol missing from the hypothesis
            n["D"] += 1; i -= 1
        else:                                      # extra hypothesis symbol
            n["I"] += 1; rebuilt.append(hyp[j - 1]); j -= 1
    return n, rebuilt[::-1]

ORDERS = {"diagonal first": ("diag", "del", "ins"), "gaps first": ("del", "ins", "diag")}

# MER and WIL as computed in jiwer v4.0.0, src/jiwer/process.py (process_words), pooled over sentences.
# Copyright @ 2018 - present 8x8, Inc. Licensed under the Apache License, Version 2.0.
# Modified (ported): rewritten for hit/substitution/deletion/insertion counts from the backtrace above.
# https://github.com/jitsi/jiwer/blob/v4.0.0/LICENSE
def mer_wil(n):
    N_ref, N_hyp = n["H"] + n["S"] + n["D"], n["H"] + n["S"] + n["I"]
    mer = (n["S"] + n["D"] + n["I"]) / (n["H"] + n["S"] + n["D"] + n["I"])
    wip = (n["H"] / N_ref) * (n["H"] / N_hyp) if N_hyp else 0.0
    return mer, 1 - wip

# Normalized edit distance as in PARSeq v1.0.0, strhub/models/base.py ("Follow ICDAR 2019 definition of N.E.D.").
# Copyright 2022 Darwin Bautista. Licensed under the Apache License, Version 2.0.
# Modified (ported): our edit distance in place of NLTK's, one line at a time.
# https://github.com/baudm/parseq/blob/v1.0.0/LICENSE
def ned(ref, hyp):
    return ed(ref, hyp) / max(len(hyp), len(ref))

def measures(refs, hyps):
    E = np.array([ed(r, h) for r, h in zip(refs, hyps)]); L = np.array([len(r) for r in refs])
    m = {"pooled CER": E.sum() / L.sum(), "per-line mean CER": np.mean(E / L),
         "ICDAR-style total": np.sum(E / L), "1 - mean NED": 1 - np.mean([ned(r, h) for r, h in zip(refs, hyps)]),
         "character accuracy": (L.sum() - E.sum()) / L.sum()}
    return m, E, L

def total(counts):
    return {k: sum(c[k] for c in counts) for k in "HSDI"}

# ---- 1. One set of outputs, many error rates (seed 0) ---------------------------------------------
refs, res = outputs(0)
L = np.array([len(r) for r in refs])
print(f"data: {len(refs)} lines; short {L[:40].min()}-{L[:40].max()} chars, words {sorted({len(r.split()) for r in refs[:40]})};"
      f" long {L[40:].min()}-{L[40:].max()} chars, words {sorted({len(r.split()) for r in refs[40:]})}; {L.sum()} reference chars")
lost = sum(int(glyph(c, (4 * W, 2 * H), W + 1).sum() - glyph(c).sum()) for c in CLASSES[:26])
clean = all(read_cell(TEMPL[i], 0.0, np.random.default_rng(0)) == c for i, c in enumerate(CLASSES))
print(f"ink lost by the 8-px cells: {lost}; noise-free reader reads all 27 templates: {clean}")
for name in "AB":
    ev = np.sum([e for _, _, e in res[name]], 0)
    print(f"system {name}: injected splits {ev[0]}, merges {ev[1]}, dropped spaces {ev[2]}; example: "
          f"{refs[0]!r} -> {res[name][0][0]!r}, {refs[40][:30]!r}... -> {res[name][40][0][:30]!r}...")

rows, info = {}, {}
for name in "AB":
    hyps = [h for h, _, _ in res[name]]
    m, E, _ = measures(refs, hyps)
    cons = np.array([c for _, c, _ in res[name]])
    m["pooled CER, strip"] = sum(ed(r.strip(), h.strip()) for r, h in zip(refs, hyps)) / L.sum()
    m["pooled CER, strip+collapse"] = sum(ed(" ".join(r.split()), " ".join(h.split())) for r, h in zip(refs, hyps)) / L.sum()
    Wr = [r.split() for r in refs]; Wh = [h.split() for h in hyps]
    m["pooled WER"] = sum(ed(r, h) for r, h in zip(Wr, Wh)) / sum(len(r) for r in Wr)
    bt = {}
    for oname, order in ORDERS.items():
        ch = [backtrace(r, h, order) for r, h in zip(refs, hyps)]
        wd = [backtrace(r, h, order) for r, h in zip(Wr, Wh)]
        valid = all(rb == list(h) for (_, rb), h in zip(ch, hyps)) and all(rb == h for (_, rb), h in zip(wd, Wh))
        cost_ok = all(n["S"] + n["D"] + n["I"] == e for (n, _), e in zip(ch, E)) and \
                  all(n["S"] + n["D"] + n["I"] == ed(r, h) for (n, _), r, h in zip(wd, Wr, Wh))
        wt = total([n for n, _ in wd]); mer, wil = mer_wil(wt)
        m[f"MER ({oname})"], m[f"WIL ({oname})"] = mer, wil
        m[f"WER from counts ({oname})"] = (wt["S"] + wt["D"] + wt["I"]) / (wt["H"] + wt["S"] + wt["D"])
        bt[oname] = (total([n for n, _ in ch]), wt, valid, cost_ok)
    rows[name] = m
    info[name] = dict(E=E, cons=cons, bt=bt, hyps=hyps, Wr=Wr, Wh=Wh,
                      nltk=sum(nltk_edit_distance(r, h) != e for r, h, e in zip(refs, hyps, E)) +
                           sum(nltk_edit_distance(r, h) != ed(r, h) for r, h in zip(Wr, Wh)))

higher_better = {"1 - mean NED", "character accuracy"}
print(f"\n{'measure':34s} {'A':>9s} {'B':>9s}   better")
for key in ["pooled CER", "pooled CER, strip", "pooled CER, strip+collapse", "per-line mean CER", "ICDAR-style total",
            "1 - mean NED", "character accuracy", "pooled WER", "WER from counts (diagonal first)",
            "WER from counts (gaps first)", "MER (diagonal first)", "MER (gaps first)", "WIL (diagonal first)", "WIL (gaps first)"]:
    a, b = rows["A"][key], rows["B"][key]
    better = ("A" if a > b else "B") if key in higher_better else ("A" if a < b else "B")
    fmt = (lambda v: f"{v:9.4f}") if key in ("ICDAR-style total", "1 - mean NED") else (lambda v: f"{100 * v:8.2f}%")
    print(f"{key:34s} {fmt(a)} {fmt(b)}   {better if a != b else 'tie'}")

print("\nedit operations (H, S, D, I) summed over the lines; characters, then words")
for name in "AB":
    I_ = info[name]
    print(f"system {name}: edit distance total {I_['E'].sum()}, construction total {I_['cons'].sum()}, "
          f"lines with ED < construction {int((I_['E'] < I_['cons']).sum())}, lines with ED > construction {int((I_['E'] > I_['cons']).sum())}")
    for oname, (ct, wt, valid, cost_ok) in I_["bt"].items():
        print(f"  {oname:15s} chars H {ct['H']:4d} S {ct['S']:3d} D {ct['D']:3d} I {ct['I']:3d} | "
              f"words H {wt['H']:3d} S {wt['S']:3d} D {wt['D']:3d} I {wt['I']:3d} | script rebuilds hypotheses {valid}, S+D+I = ED {cost_ok}")
    print(f"  hand DP vs NLTK edit_distance (chars and words): {I_['nltk']} mismatches")

print("\nidentities and bounds")
for name in "AB":
    m = rows[name]
    print(f"system {name}: |ICDAR total - N x per-line mean| = {abs(m['ICDAR-style total'] - len(refs) * m['per-line mean CER']):.1e};"
          f" |char accuracy - (1 - pooled CER)| = {abs(m['character accuracy'] - (1 - m['pooled CER'])):.1e};"
          f" mean NED <= per-line mean CER {1 - m['1 - mean NED'] <= m['per-line mean CER']};"
          f" MER <= WER {all(m[f'MER ({o})'] <= m['pooled WER'] for o in ORDERS)}")

# ---- 2. Seeds 0-9: does the ranking flip survive another draw? ------------------------------------
print("\nseed  pooled CER A/B     per-line mean A/B   1 - mean NED A/B   pooled ranks A better, per-line ranks B better")
flips, vals = 0, []
for seed in range(10):
    rs, out = outputs(seed)
    mA, _, _ = measures(rs, [h for h, _, _ in out["A"]]); mB, _, _ = measures(rs, [h for h, _, _ in out["B"]])
    flip = mA["pooled CER"] < mB["pooled CER"] and mA["per-line mean CER"] > mB["per-line mean CER"]; flips += flip
    vals.append([mA["pooled CER"], mB["pooled CER"], mA["per-line mean CER"], mB["per-line mean CER"], mA["1 - mean NED"], mB["1 - mean NED"]])
    print(f"{seed:4d}  {100*vals[-1][0]:6.2f} {100*vals[-1][1]:6.2f}     {100*vals[-1][2]:6.2f} {100*vals[-1][3]:6.2f}"
          f"       {vals[-1][4]:.4f} {vals[-1][5]:.4f}     {flip}")
v = np.array(vals)
print(f"flip in {flips} of 10 seeds; ranges: pooled A {100*v[:,0].min():.2f}-{100*v[:,0].max():.2f}, B {100*v[:,1].min():.2f}-{100*v[:,1].max():.2f};"
      f" per-line A {100*v[:,2].min():.2f}-{100*v[:,2].max():.2f}, B {100*v[:,3].min():.2f}-{100*v[:,3].max():.2f};"
      f" 1 - NED A {v[:,4].min():.4f}-{v[:,4].max():.4f}, B {v[:,5].min():.4f}-{v[:,5].max():.4f}")

# ---- 3. Edge cases ---------------------------------------------------------------------------------
print("\nedge cases")
r, h = "m", "rn"
print(f"m read as rn: ED {ed(r, h)}, CER {100 * ed(r, h) / len(r):.2f}%, NED {ned(r, h):.4f}, "
      f"character accuracy {100 * (len(r) - ed(r, h)) / len(r):.2f}%")
r, h = "form", "from"
print(f"form read as from: ED {ed(r, h)}, NLTK {nltk_edit_distance(r, h)}, NLTK with transpositions "
      f"{nltk_edit_distance(r, h, transpositions=True)}; " +
      "; ".join(f"{o}: S {n['S']} D {n['D']} I {n['I']}" for o, (n, _) in ((o, backtrace(r, h, od)) for o, od in ORDERS.items())))
print(f"intention -> execution: ED {ed('intention', 'execution')} (NLTK {nltk_edit_distance('intention', 'execution')}),"
      f" substitution cost 2: {ed('intention', 'execution', 2)} (NLTK {nltk_edit_distance('intention', 'execution', substitution_cost=2)})")
ref, hyp = "Café Noir", unicodedata.normalize("NFD", "café  noir ")
steps = [("raw", ref, hyp)]
steps.append(("NFC", unicodedata.normalize("NFC", steps[-1][1]), unicodedata.normalize("NFC", steps[-1][2])))
steps.append(("NFC + casefold", steps[-1][1].casefold(), steps[-1][2].casefold()))
steps.append(("NFC + casefold + whitespace", " ".join(steps[-1][1].split()), " ".join(steps[-1][2].split())))
for nm, r, h in steps:
    print(f"  {nm:28s} |ref| {len(r)} |hyp| {len(h):2d}  ED {ed(r, h)}  CER {100 * ed(r, h) / len(r):6.2f}%")

# ---- 4. Checks of the edit distance itself ---------------------------------------------------------
def brute(a, b):                                   # the definition as a plain recursion (exponential)
    if not a or not b:
        return len(a) + len(b)
    return min(brute(a[1:], b) + 1, brute(a, b[1:]) + 1, brute(a[1:], b[1:]) + (a[0] != b[0]))

rng = np.random.default_rng(1)
pairs = [("".join(rng.choice(list("bhco "), rng.integers(0, 6))), "".join(rng.choice(list("bhco "), rng.integers(0, 6))))
         for _ in range(200)]
edge = [("m", "rn"), ("form", "from"), ("intention", "execution")] + [(r, h) for _, r, h in steps]
print(f"\nhand DP vs brute-force recursion, 200 random pairs of length 0-5: {sum(ed(a, b) != brute(a, b) for a, b in pairs)} mismatches;"
      f" hand DP vs NLTK on the edge cases: {sum(ed(a, b) != nltk_edit_distance(a, b) for a, b in edge)} mismatches")

Output:

data: 60 lines; short 3-5 chars, words [1]; long 38-45 chars, words [8]; 1008 reference chars
ink lost by the 8-px cells: 0; noise-free reader reads all 27 templates: True
system A: injected splits 60, merges 0, dropped spaces 0; example: 'label' -> 'liabel', 'text box box dark box noise fo'... -> 't ext bcx box dark box noise f'...
system B: injected splits 0, merges 53, dropped spaces 20; example: 'label' -> 'label', 'text box box dark box noise fo'... -> 'text box box dark boxnoia fent'...

measure                                    A         B   better
pooled CER                             8.13%    14.98%   A
pooled CER, strip                      7.94%    14.98%   A
pooled CER, strip+collapse             7.94%    14.98%   A
per-line mean CER                     21.67%     7.01%   B
ICDAR-style total                    13.0047    4.2089   B
1 - mean NED                          0.8247    0.9299   B
character accuracy                    91.87%    85.02%   A
pooled WER                            56.00%    47.00%   B
WER from counts (diagonal first)      56.00%    47.00%   B
WER from counts (gaps first)          56.00%    47.00%   B
MER (diagonal first)                  45.90%    47.00%   A
MER (gaps first)                      45.90%    47.00%   A
WIL (diagonal first)                  64.30%    68.79%   A
WIL (gaps first)                      64.30%    68.79%   A

edit operations (H, S, D, I) summed over the lines; characters, then words
system A: edit distance total 82, construction total 82, lines with ED < construction 0, lines with ED > construction 0
  diagonal first  chars H  986 S  22 D   0 I  60 | words H 132 S  68 D   0 I  44 | script rebuilds hypotheses True, S+D+I = ED True
  gaps first      chars H  986 S  22 D   0 I  60 | words H 132 S  68 D   0 I  44 | script rebuilds hypotheses True, S+D+I = ED True
  hand DP vs NLTK edit_distance (chars and words): 0 mismatches
system B: edit distance total 151, construction total 152, lines with ED < construction 1, lines with ED > construction 0
  diagonal first  chars H  857 S  78 D  73 I   0 | words H 106 S  74 D  20 I   0 | script rebuilds hypotheses True, S+D+I = ED True
  gaps first      chars H  859 S  74 D  75 I   2 | words H 106 S  74 D  20 I   0 | script rebuilds hypotheses True, S+D+I = ED True
  hand DP vs NLTK edit_distance (chars and words): 0 mismatches

identities and bounds
system A: |ICDAR total - N x per-line mean| = 0.0e+00; |char accuracy - (1 - pooled CER)| = 0.0e+00; mean NED <= per-line mean CER True; MER <= WER True
system B: |ICDAR total - N x per-line mean| = 0.0e+00; |char accuracy - (1 - pooled CER)| = 0.0e+00; mean NED <= per-line mean CER True; MER <= WER True

seed  pooled CER A/B     per-line mean A/B   1 - mean NED A/B   pooled ranks A better, per-line ranks B better
   0    8.13  14.98      21.67   7.01       0.8247 0.9299     True
   1    7.94  14.90      19.84   5.97       0.8383 0.9403     True
   2    8.53  15.96      20.21   7.40       0.8343 0.9260     True
   3    8.06  14.23      20.83   5.69       0.8309 0.9431     True
   4    8.13  14.06      19.23   6.35       0.8416 0.9365     True
   5    8.46  16.12      22.50   7.78       0.8184 0.9222     True
   6    7.39  14.17      18.20   6.02       0.8513 0.9398     True
   7    7.81  12.51      18.48   5.34       0.8483 0.9466     True
   8    7.22  14.34      18.69   6.93       0.8484 0.9307     True
   9    9.12  13.78      22.12   6.11       0.8184 0.9389     True
flip in 10 of 10 seeds; ranges: pooled A 7.22-9.12, B 12.51-16.12; per-line A 18.20-22.50, B 5.34-7.78; 1 - NED A 0.8184-0.8513, B 0.9222-0.9466

edge cases
m read as rn: ED 2, CER 200.00%, NED 1.0000, character accuracy -100.00%
form read as from: ED 2, NLTK 2, NLTK with transpositions 1; diagonal first: S 2 D 0 I 0; gaps first: S 0 D 1 I 1
intention -> execution: ED 5 (NLTK 5), substitution cost 2: 8 (NLTK 8)
  raw                          |ref| 9 |hyp| 12  ED 6  CER  66.67%
  NFC                          |ref| 9 |hyp| 11  ED 4  CER  44.44%
  NFC + casefold               |ref| 9 |hyp| 11  ED 2  CER  22.22%
  NFC + casefold + whitespace  |ref| 9 |hyp|  9  ED 0  CER   0.00%

hand DP vs brute-force recursion, 200 random pairs of length 0-5: 0 mismatches; hand DP vs NLTK on the edge cases: 0 mismatches

The averaging rule decides the ranking. Pooled CER puts A first, 8.13% against 14.98%, and the per-line mean puts B first, 21.67% against 7.01%; the first hypothesis, that the two rules rank the systems in opposite orders, held. A’s errors cost about one or two edits on every line, which is a large fraction of a five-letter word and a small fraction of a 40-character line, so the per-line mean, which gives each line the same weight, is dominated by the 40 short lines. B’s merges, missed gap and higher noise sit at or beyond character 20, which only the long lines reach, so they carry the pooled count. The second hypothesis held under each of its three measures: $1 - \overline{\mathrm{NED}}$ (0.8247 against 0.9299) and the ICDAR-style total (13.0047 against 4.2089) side with the per-line mean, and character accuracy (91.87% against 85.02%) sides with pooled CER. That follows from their definitions: the ICDAR total is the sum of the per-line rates (13.0047 is 60 times 21.67%), NED is a per-line rate with a larger denominator, and character accuracy is $1 - \mathrm{CER}_{\text{pooled}}$, as the identity lines confirm to zero difference. The third hypothesis held too: the flip appears in all ten seeds, with pooled CER at 7.22–9.12% for A against 12.51–16.12% for B and per-line means of 18.20–22.50% against 5.34–7.78%. The ICDAR total is applied here to lines, the items of this test set, rather than to word crops as in the competition.

Words tell a third story. WER, which was not part of any hypothesis, ranks B first (47.00% against 56.00%), the opposite of pooled CER. A’s word counts include 44 insertions; in our reading these come from split halves read as the blank template, which cut the first word of a line in two and add a word. MER, whose denominator includes the insertions, puts A back in front (45.90% against 47.00%), and so does WIL; B has no word insertions, so its MER equals its WER, as the formula requires. Stripping leading and trailing spaces lowers A’s pooled CER from 8.13% to 7.94%, two edits fewer of the 1,008 reference characters, from leading or trailing spaces in the output, and collapsing inner runs of spaces removes nothing further; B is unchanged.

Alignments. For A the dynamic programme found exactly the construction’s 82 edits; for B it found 151 against 152, one line where the true distance is shorter than the injected script. No line exceeded its construction count, and every backtrace, applied to its reference, rebuilt the hypothesis, with $S + D + I$ equal to the distance. The fourth hypothesis had two parts. H4a, that the character-level breakdown differs between the two tie-break orders for at least one system, held: B’s 151 edits are 78 substitutions and 73 deletions with the diagonal preferred, and 74, 75 and 2 with gaps preferred, four substitutions traded for two deletions and two insertions, while A’s breakdown is the same under both. H4b, that word-level MER or WIL differs between the orders for at least one system, failed: the word-level counts came out identical under both orders for both systems, so MER and WIL did too, while WER was equal under both orders, as it must be.

Edge cases and checks. m read as rn gives 200% CER, an NED of exactly 1 and a character accuracy of −100%. form read as from costs 2, or 1 with NLTK’s transposition option, and its two backtraces give two substitutions or one deletion and one insertion. The unit-cost and substitution-cost-2 distances from intention to execution are 5 and 8, the values Jurafsky and Martin give, in both implementations. The normalization chain takes a 9-character reference and a decomposed, lowercased, double-spaced output from 6 edits (66.67%) to 4 after NFC composition, 2 after case folding and 0 after whitespace normalization: the letters were all read correctly, and the whole 66.67% came from Unicode form, case and spacing, two edits each. All planned checks held: zero mismatches between the dynamic programme and NLTK on every line and edge case, and against the brute-force recursion on 200 random pairs; no line above its construction count; the identities and bounds (zero differences, mean NED at most the per-line mean CER, MER at most WER); the data had the planned lengths, word counts and event counts (60 splits, 53 merges, 20 missed gaps); the cells lost no ink; and the noise-free reader read all 27 templates.

Related Metrics

• Accuracy over whole strings, the word accuracy of scene text recognition, counts an output as right only when every character matches; CER gives partial credit.
• ANLS turns the normalized Levenshtein distance into a thresholded similarity for answers (document understanding).
• Label error rate is CER over the output labels of a sequence model (connectionist temporal classification).
• Segmental edit score applies an edit distance to sequences of action segments rather than characters (temporal action segmentation).
• Total edit distance of the ICDAR word recognition tasks is the per-word normalized form (ICDAR).