Document Image Analysis

Document image analysis is the part of computer vision that turns images of documents, such as scanned book pages, faxed forms, photographed receipts, invoices, scientific articles and historical manuscripts, into structured information: which pixels are ink, where the text lines, paragraphs, figures and tables are, in what order they are read, what each block is, and which values a form or table holds. It is also called document analysis and recognition, as in the name of the International Conference on Document Analysis and Recognition (ICDAR), and its later, model-centred form is often called document understanding or document AI. Reading the characters themselves is the task of optical character recognition (OCR), which keeps the OCR pipeline, its line recognizers and its metrics; this article covers what surrounds OCR on a page: image clean-up, geometry, layout, tables and forms, and models that combine text, layout and pixels.

The field shows a pattern common in applied vision. Its classical methods are short, interpretable procedures, a threshold per pixel or a cut along an empty row, tuned to the regularities of printed pages. Its current methods are detectors and transformers trained on hundreds of thousands of pages whose labels were generated automatically from digital-born documents. Both kinds are still used, and the benchmarks reflect both: small hand-labelled contest sets scored pixel by pixel, and large weakly labelled sets scored like object detection.

Problem

A document image is a picture of something that was designed to be read: text set in lines and columns, with headings, captions, figures, tables and form fields placed according to conventions. Document image analysis recovers that design from the pixels. A complete system usually answers several questions in sequence:

• Which pixels are ink? Binarization separates foreground from paper, despite stains, uneven light and bleed-through.
• Is the page geometrically straight? Skew and warp correction undo a rotated scan or a curved, folded or photographed page.
• What is the layout? Page segmentation splits the page into homogeneous regions, and region classification labels them (text, title, list, table, figure, separator); reading order links the text regions.
• What do the tables and forms say? Table recognition recovers rows, columns and cells; form understanding finds keys, values and the links between them.
• What is the document? Classification assigns a page to a type (letter, invoice, scientific report), and information extraction and question answering pull out specific facts.

Antonacopoulos et al. (2009) describe layout analysis as the first major step of a document analysis workflow after image enhancement, and note that the correctness of page segmentation is crucial because all later analysis and recognition builds on it.

Visual Data

• Flatbed scans. Bilevel, grey or colour images at fixed resolution, usually of flat pages; degradations come from the original (faded ink, stains, show-through) and from the scanner (noise, skew). The RVL-CDIP collection, for example, consists of grayscale scans sized so that their largest side is at most 1,000 pixels (official dataset page).
• Camera captures. Phone photographs of receipts, forms and book pages add perspective, curvature, folds and uneven illumination. The DocUNet benchmark consists of 130 phone photographs of 65 documents (receipts, letters, fliers, magazines, papers, books), each deformed by hand into two shapes and photographed under sunlight, indoor light or flash; 94.6% of the images contain curving and 8.5% are crumpled paper (Ma et al., 2018).
• Historical documents. Manuscripts and early prints, with degradations such as smear, low contrast, bleed-through and shadows; the DIBCO test images were chosen to contain such degradations (Gatos et al., 2009).
• Digital-born documents rendered to images. PDFs of scientific articles and office documents, which can be rendered without degradation and come with their own text and structure. Most large layout and table datasets are built this way, because the source files supply labels for free (PubLayNet, DocBank, TableBank, PubTabNet, PubTables-1M).

Vision Tasks Involved

• Pixel classification: binarization, and segmentation of text against non-text, a form of image segmentation.
• Geometric estimation: skew angle, page boundary and the dense warp of a curved page, related to image warping.
• Detection and segmentation of regions: layout blocks, tables, figures and form fields, with detectors and instance segmentation models such as Mask R-CNN.
• Text detection and recognition: finding and reading lines and words, the subject of optical character recognition and, for photographs, scene text detection.
• Structure recognition: turning a table image into rows, columns and spanning cells, or a page into a reading-order graph.
• Classification and extraction: image classification of whole pages, sequence labelling of words, and visual question answering on documents.

Common Methods

Binarization

Binarization assigns each pixel to ink or paper. The DIBCO organizers call it an important step in the document image analysis and recognition pipeline (Gatos et al., 2009); it is a preprocessing step for many systems and an object of study in its own right, because historical and photographed pages break the assumption that ink and paper each have one grey level.

Global thresholds. Otsu’s method (Otsu, 1979) picks one threshold from the grey-level histogram by maximizing the between-class variance. It works when the page is evenly lit and the histogram has two well-separated modes, and it fails when the paper in one part of the page is darker than the ink in another, which happens with shadows, a book’s gutter or a lamp to one side.

Local thresholds. Local methods compute a threshold $T(x, y)$ for every pixel from the mean $m$ and standard deviation $s$ of the grey levels in a window around it. Niblack’s rule, as the thresholding article gives it from Sezgin and Sankur’s survey, is $T = m + k s$ with a negative $k$ for dark text. Sauvola and Pietikäinen (2000) proposed a rule designed for documents; in the form implemented in scikit-image (threshold_sauvola, version 0.25.2), it reads

$$T(x, y) = m(x, y)\left[1 + k\left(\frac{s(x, y)}{R} - 1\right)\right],$$

where $R$ is the dynamic range of the standard deviation, half the range of the image’s data type by default (127.5 for 8-bit images), and $k$ is positive. In a window of empty paper $s$ is small, so the threshold falls well below the local mean and noise in the paper does not cross it; Niblack’s threshold sits just below the local mean there, so noise in empty paper is labelled ink. The thresholding article derives these rules, discusses window size and $k$, and compares them on a synthetic page.

Contest methods. The Document Image Binarization Contest, DIBCO, has measured progress since 2009. The first edition, run by Gatos, Ntirogiannis and Pratikakis at ICDAR 2009, received 43 algorithms from 35 research groups. Their short descriptions in the report show the range of ideas: estimating and removing the background before thresholding, illumination compensation followed by a locally adaptive variant of Otsu’s method, high-pass filtering followed by Otsu’s threshold, morphological operators such as the ultimate attribute closing, Markov random field regularization of a modified Sauvola threshold, and Gaussian models of text and background fitted with the EM algorithm. The series continued with DIBCO editions at ICDAR and handwritten-document editions, H-DIBCO, at the handwriting conference ICFHR; the DIBCO 2019 page lists the earlier editions from 2009 to 2018; the protocol is described under Evaluation.

Skew and geometric correction

A page scanned at a small angle is rotated back so that text lines are horizontal. Skew estimation and correction, with the projection-profile criterion and Tesseract’s alternative of tracking sloped baselines without rotating the image, are described and run in optical character recognition.

Photographed pages also curve and fold, and the distortion is no longer a single rotation but a dense displacement field. DocUNet (Ma et al., 2018) presented itself as the first learning-based method for flattening such images: a stacked U-Net with intermediate supervision predicts, for every pixel of the distorted image, where it belongs in the flat page. Because real pairs with known deformation are hard to collect, the authors trained on about 100,000 synthetic images made by warping flat document images in 2D, each with up to 19 synthetic distortions, 30% of them curves and 70% folds; 90,000 were used for training. They also built the benchmark of 130 photographs described above, with flatbed scans of the documents before folding as ground truth. They chose image similarity over OCR accuracy as the measure, because OCR accuracy depends on the OCR engine and cannot score pages that are mostly figures: multi-scale SSIM (SSIM on a five-level pyramid) and a local distortion score from dense SIFT flow. On that benchmark DocUNet scored 0.41 MS-SSIM and 14.08 local distortion against 0.13 and 33.69 for the non-learning method of Tian and Narasimhan, and took 4 seconds per image against 212 (Table 1).

Layout analysis

Smith (2009) separates physical layout analysis, which divides a page into text and non-text areas and splits multi-column text into columns, from logical layout analysis, which detects headers, footers, body text, lists and articles. He divides physical methods roughly into two categories and adds a third type:

• Top-down methods cut the image recursively, vertically and horizontally, along white spaces expected to be column or paragraph boundaries. Smith’s reference for them is Nagy and Seth (1984), whose X-Y cut, in the DocBank paper’s description, uses projection profiles to determine where to cut. Top-down methods start from the largest structures but, Smith notes, cannot handle many magazine formats, such as non-rectangular regions and headings that span several columns and blend into the columns below.
• Bottom-up methods classify small parts of the image (pixels, groups of pixels or connected components) and merge parts of the same type into regions. Smith counts them as both the oldest and among the most recent methods, citing the block segmentation of Wahl, Wong and Casey (1982) as the oldest; they handle arbitrarily shaped regions but struggle with higher-level structures such as columns, which often leaves regions over-fragmented. The run-length smearing algorithm, which the DocBank paper attributes to Wahl et al. but lists among top-down methods, converts background pixels to foreground wherever fewer than a threshold of them separate two foreground pixels. O’Gorman’s document spectrum (Docstrum, 1993) is another widely cited method of page layout analysis.
• Whitespace analysis looks explicitly for white rectangles between columns, which fixes some flaws of recursive cutting but still has difficulty with non-rectangular regions.

Smith’s own method, the page layout analysis of the Tesseract OCR engine, is a hybrid: bottom-up connected-component analysis finds the tab-stops, the x-positions at which margins, column edges, indents and table columns were aligned when the page was formatted, and the column layout deduced from them is then imposed top-down to group text into regions and fix their reading order. In the ICDAR 2009 page segmentation competition, the Fraunhofer Newspaper Segmenter, another hybrid that combines black and white separator detection with a bottom-up grouping guided by the column layout, had an overall advantage over the other three entries, over ABBYY FineReader Engine 8.1 and OCRopus 0.3.1, and over the methods of the 2007 competition (Antonacopoulos et al., 2009).

Deep detectors. Once large labelled sets existed, layout analysis was recast as object detection: each block is a box or mask with a class. PubLayNet (Zhong et al., 2019), built by matching the PDF and XML versions of articles in the PubMed Central Open Access Subset, annotates five classes (text, title, list, table, figure) with bounding boxes and polygons in the COCO annotation format (official README; Jimeno Yepes et al., 2021). The repository released pre-trained Faster R-CNN and Mask R-CNN models, and LayoutLMv2 initialized its visual backbone from a Mask R-CNN trained on PubLayNet (Xu et al., 2021). In the layout task of the ICDAR 2021 Scientific Literature Parsing competition, which used PubLayNet, the top systems were ensembles of Mask R-CNN variants (Cascade Mask R-CNN, Hybrid Task Cascade) with large backbones, deformable convolutions and multi-scale testing. The report notes that the top systems compared their detections with information extracted from the PDF or from specialized classifiers, mostly for the text and title classes; one refined them with text-line coordinates parsed from the PDF, and the winning system fused the image with a second stream of text embeddings parsed from the PDF (Jimeno Yepes et al., 2021).

Token-level layout. DocBank (Li et al., 2020) frames layout analysis as labelling every word of a page with one of 12 semantic classes (abstract, author, caption, equation, figure, footer, list, paragraph, reference, section, table, title). Its labels come from recompiling the LaTeX sources of arXiv papers with each structure printed in its own font colour, then reading words, positions and colours from the PDF. Because the labels are per token, the same set trains sequence-labelling text models and, after grouping tokens into boxes, detectors.

Tables and forms

Tables pack data into a grid whose logical structure, which cell belongs to which row, column and header, is not explicit in the image. Smock et al. (2022) divide table extraction into three subtasks: table detection, which locates the table; table structure recognition, which recognizes rows, columns and cells; and functional analysis, which recognizes keys and values.

Large table datasets are again generated from digital-born documents. TableBank (Li et al., 2020) collected 417,234 labelled tables from Word and LaTeX documents on the internet, using weak supervision from the documents’ markup. PubTabNet (Zhong et al., 2020) contains about 568,000 table images from PubMed Central articles, each paired with the table’s HTML, located with the same PDF-to-XML matching algorithm that produced PubLayNet. With it the authors trained an encoder–dual–decoder model, a ResNet-18 encoder with one attention-based LSTM decoder for the structure tokens and another for the cell contents, that writes the HTML of a table directly from its image. PubTables-1M (Smock et al., 2022) contains 947,642 tables from PubMed Central for structure recognition, 52.7% of them complex (with at least one spanning cell). Its authors identified oversegmentation, extra blank cells in headers, as a source of inconsistent ground truth in earlier crowd-sourced datasets and corrected it with a canonicalization procedure that adjusted 34.7% of the tables. They modelled all three subtasks as object detection, with rows, columns, column headers, projected row headers and spanning cells as object classes, and trained DETR and Faster R-CNN baselines, announcing data and code in a repository named table-transformer.

Forms add a different structure: labels and the values written next to them. FUNSD (Jaume et al., 2019) contains 199 fully annotated scanned forms with 31,485 words, 9,707 semantic entities and 5,304 relations between them (official page). LayoutLMv2 uses its semantic entity labelling task, which assigns each word one of four labels (question, answer, header, other), with 149 forms for training and 50 for testing (Xu et al., 2021, appendix). Receipt datasets such as SROIE and CORD pose the same kind of key–value extraction for photographed or scanned receipts.

Document understanding models

The LayoutLMv2 authors describe pre-training on text and layout as having proved effective in a variety of visually rich document understanding tasks. In the descriptions of DocBank and LayoutLMv2, LayoutLM (Xu et al., 2020) adds a 2D position embedding of each word’s bounding box to a BERT-style transformer encoder and is pre-trained on the IIT-CDIP collection of scanned pages with a masked visual-language model objective (predicting masked words from the remaining words and all positions) and multi-label document classification; image features can be added for downstream tasks. Its words and boxes come from an OCR engine, so OCR errors enter the model, as the optical character recognition article notes for LayoutLM’s receipt results.

LayoutLMv2 (Xu et al., 2021) feeds image tokens from a ResNeXt-101-FPN visual backbone into the same encoder as the words, adds spatial-aware self-attention with relative 2D positions, and adds two pre-training tasks: text–image alignment, which predicts whether a word’s image region has been covered, and text–image matching, which predicts whether page image and text belong together. Its base and large models have about 200 and 426 million parameters. A third version, LayoutLMv3 (2022), followed.

Donut (Kim et al., 2022) removes the OCR engine. A Swin Transformer encoder reads the page image (input resolution $2560 \times 1920$ in the main experiments), and a BART-style decoder (the first four layers) generates a token sequence that converts one-to-one into JSON: a class label for classification, keys and values for extraction, an answer for a question. Pre-training teaches the model to read: it predicts the text of a page in reading order, a pseudo-OCR objective, on 11 million IIT-CDIP pages with text from a commercial OCR service and on synthetic pages from the authors’ generator SynthDoG in Chinese, Japanese, Korean and English. The authors motivate the design with three costs of OCR-based pipelines: the computation of OCR itself, its inflexibility across languages and document types, and the propagation of OCR errors. General vision-language models also read text in document images as part of answering questions about them.

Example: binarization decides whether a simple layout analysis works

The code below renders a synthetic two-column page ($700 \times 500$ pixels, 8-bit) with a known ink mask and six known blocks: a title in double-size pseudo-glyphs, four paragraphs and a line-art figure. Glyphs are random combinations of 2-pixel strokes, 14 pixels tall including an ascender zone; text lines have a pitch of 20 pixels, paragraphs are separated by 30 pixels and the columns by a 30-pixel gutter. The page is blurred ($\sigma = 0.7$ pixels), multiplied by an illumination field and given Gaussian noise (standard deviation 4 grey levels). Under flat light the illumination is 1 everywhere; under uneven light it falls linearly from 1.0 at the top right to 0.3 at the bottom left and is darkened further by a band along the left edge, as next to a book’s gutter.

Four binarizations are compared: Otsu’s global threshold, Niblack’s rule (scikit-image’s threshold_niblack, which computes $m - k s$, with $k = 0.2$), and Sauvola’s rule with $k = 0.2$ (scikit-image’s default) and $k = 0.5$, all with 25-pixel windows. Since the sources leave $k$ open, both values are printed. Each binarization is scored against the true ink mask with the DIBCO 2009 F-measure and PSNR, and then passed to a recursive XY-cut, which trims each region to its ink, finds runs of at least 16 empty rows or columns, cuts at all such runs in the direction of the widest one, and recurses until no such run is left. A row or column counts as empty if it has at most tol ink pixels; tol is a second open choice, so 0 and 2 are both printed. The leaves are matched one-to-one to the six true blocks at an intersection over union of at least 0.5. The last column gives the share of paper pixels in the darkest quarter of the page (bottom left) labelled as ink. Part B applies the same binarizations to scikit-image’s bundled page image, which its docstring calls a scanned page and which is already much darker on the left, as it is and with an added shadow at the bottom right, and counts the text-line bands in the horizontal projection profile. The page image has no pixel ground truth; counted by eye, it shows seven text lines (a heading, five paragraph lines and a line of code), two thin horizontal rules and a cut-off fragment at the bottom left.

import numpy as np
import scipy, skimage
from scipy import ndimage as ndi
from skimage import data
from skimage.filters import threshold_otsu, threshold_niblack, threshold_sauvola

H, W = 700, 500

# ---------- Synthetic page with known ink mask and known blocks ----------
STROKES = {  # 14 x 6 glyph cell: rows 0-3 ascender zone, rows 4-13 x-height; strokes 2 px wide
    "left": (slice(4, 14), slice(0, 2)), "mid": (slice(4, 14), slice(2, 4)),
    "right": (slice(4, 14), slice(4, 6)), "top": (slice(4, 6), slice(0, 6)),
    "bar": (slice(8, 10), slice(0, 6)), "bottom": (slice(12, 14), slice(0, 6)),
}

def glyph(rng):
    g = np.zeros((14, 6), bool)
    for name in rng.choice(list(STROKES), size=rng.integers(2, 4), replace=False):
        g[STROKES[name]] = True
    if rng.random() < 0.25:
        g[0:4, 0:2] = True                                   # ascender
    return g

def text_line(ink, rng, top, left, width, scale=1, fill=1.0):
    """Words of 2-9 pseudo-glyphs, 2 px between glyphs, 8 px between words."""
    x, end = left, left + int(width * fill)
    while True:
        n = rng.integers(2, 10)
        wlen = (n * 8 - 2) * scale
        if x + wlen > end:
            break
        for _ in range(n):
            g = np.kron(glyph(rng), np.ones((scale, scale), bool))
            ink[top:top + g.shape[0], x:x + g.shape[1]] |= g
            x += 8 * scale
        x += 6 * scale

def paragraph(ink, rng, top, left, width, n_lines, pitch=20):
    for i in range(n_lines):
        fill = rng.uniform(0.4, 0.8) if i == n_lines - 1 else 1.0   # short last line
        text_line(ink, rng, top + i * pitch, left, width, fill=fill)
    return top + (n_lines - 1) * pitch + 14                        # bottom of last line

def figure(ink, top, left, width, height):
    ink[top:top + 2, left:left + width] = ink[top + height - 2:top + height, left:left + width] = True
    ink[top:top + height, left:left + 2] = ink[top:top + height, left + width - 2:left + width] = True
    x = np.arange(left + 10, left + width - 10)
    y = (top + height / 2 + 0.3 * height * np.sin((x - left) / width * 4 * np.pi)).astype(int)
    for dy in (0, 1):
        ink[y + dy, x] = True
    return top + height

def make_page(rng):
    ink = np.zeros((H, W), bool)
    regions = []                                   # (block name, row range, col range)
    text_line(ink, rng, 40, 130, 240, scale=2)     # title, taller glyphs, centred
    regions.append(("title", (40, 68), (0, W)))
    L, R, CW = 40, 265, 195                        # two columns, 30 px gutter
    b = paragraph(ink, rng, 98, L, CW, 10); regions.append(("left para 1", (98, b), (L, L + CW)))
    t = b + 30; b = paragraph(ink, rng, t, L, CW, 12); regions.append(("left para 2", (t, b), (L, L + CW)))
    b = paragraph(ink, rng, 98, R, CW, 6); regions.append(("right para 1", (98, b), (R, R + CW)))
    t = b + 30; b = figure(ink, t, R, CW, 120); regions.append(("figure", (t, b), (R, R + CW)))
    t = b + 30; b = paragraph(ink, rng, t, R, CW, 8); regions.append(("right para 2", (t, b), (R, R + CW)))
    blocks = []
    for name, (r0, r1), (c0, c1) in regions:      # GT box = bounding box of the block's ink
        rr, cc = np.nonzero(ink[r0:r1, c0:c1])
        blocks.append((r0 + rr.min(), c0 + cc.min(), r0 + rr.max() + 1, c0 + cc.max() + 1))
    return ink, blocks

def render(ink, light, noise):
    refl = np.where(ink, 30.0, 235.0)                # ink and paper grey levels
    img = ndi.gaussian_filter(refl, 0.7) * light + noise
    return np.clip(np.round(img), 0, 255).astype(np.uint8)

yy, xx = np.mgrid[0:H, 0:W]
FLAT = np.ones((H, W))
t = ((W - 1 - xx) / (W - 1) + yy / (H - 1)) / 2     # 0 at top right, 1 at bottom left
UNEVEN = (1 - 0.7 * t) * (1 - 0.5 * np.exp(-xx / 40))

# ---------- Binarization (ink = True) ----------
METHODS = {
    "Otsu (global)":    lambda im: im <= threshold_otsu(im),
    "Niblack w25 k0.2": lambda im: im <= threshold_niblack(im, window_size=25, k=0.2),
    "Sauvola w25 k0.2": lambda im: im <= threshold_sauvola(im, window_size=25, k=0.2),
    "Sauvola w25 k0.5": lambda im: im <= threshold_sauvola(im, window_size=25, k=0.5),
}

def dibco_scores(pred, gt):
    """DIBCO 2009 F-measure (%) and PSNR (C = 1, foreground = ink)."""
    tp = np.sum(pred & gt); fp = np.sum(pred & ~gt); fn = np.sum(~pred & gt)
    rc, pr = tp / (tp + fn), tp / max(tp + fp, 1)
    f = 100 * 2 * rc * pr / max(rc + pr, 1e-12)
    mse = np.mean(pred != gt)
    return f, 10 * np.log10(1 / mse)

# ---------- Recursive XY-cut ----------
def gaps(occupied, min_gap):
    """Runs of unoccupied positions at least min_gap long: list of (start, stop)."""
    out, start = [], None
    for i, occ in enumerate(np.append(occupied, True)):
        if not occ and start is None:
            start = i
        elif occ and start is not None:
            if i - start >= min_gap:
                out.append((start, i))
            start = None
    return out

def xy_cut(ink, box, tol, min_gap, leaves):
    r0, c0, r1, c1 = box
    sub = ink[r0:r1, c0:c1]
    rows, cols = np.flatnonzero(sub.sum(1) > tol), np.flatnonzero(sub.sum(0) > tol)
    if rows.size == 0 or cols.size == 0:
        return
    r0, r1, c0, c1 = r0 + rows[0], r0 + rows[-1] + 1, c0 + cols[0], c0 + cols[-1] + 1
    sub = ink[r0:r1, c0:c1]
    g_r, g_c = gaps(sub.sum(1) > tol, min_gap), gaps(sub.sum(0) > tol, min_gap)
    wid = lambda g: max((b - a for a, b in g), default=0)
    if not g_r and not g_c:
        if sub.sum() >= 30:
            leaves.append((r0, c0, r1, c1))
        return
    horizontal = wid(g_r) >= wid(g_c)                # cut across rows (horizontal cuts)?
    cuts = g_r if horizontal else g_c
    edges = [0] + [x for g in cuts for x in g] + [(r1 - r0) if horizontal else (c1 - c0)]
    for a, b in zip(edges[0::2], edges[1::2]):
        child = (r0 + a, c0, r0 + b, c1) if horizontal else (r0, c0 + a, r1, c0 + b)
        xy_cut(ink, child, tol, min_gap, leaves)

def iou(a, b):
    ih = max(0, min(a[2], b[2]) - max(a[0], b[0])); iw = max(0, min(a[3], b[3]) - max(a[1], b[1]))
    inter = ih * iw
    area = lambda r: (r[2] - r[0]) * (r[3] - r[1])
    return inter / (area(a) + area(b) - inter)

def matched_blocks(pred, gt, thr=0.5):
    """Greedy one-to-one matching by IoU >= thr."""
    m = np.array([[iou(p, g) for g in gt] for p in pred]) if pred else np.zeros((0, len(gt)))
    n = 0
    while m.size and m.max() >= thr:
        i, j = np.unravel_index(np.argmax(m), m.shape)
        m[i, :] = -1; m[:, j] = -1; n += 1
    return n

def blocks_found(ink, gt_blocks, tol):
    leaves = []
    xy_cut(ink, (0, 0, H, W), tol, 16, leaves)
    return len(leaves), matched_blocks(leaves, gt_blocks)

# ---------- Part A ----------
print(f"NumPy {np.__version__}, SciPy {scipy.__version__}, scikit-image {skimage.__version__}")
SEEDS = range(5)
DARK = (yy >= H // 2) & (xx < W // 2)              # bottom-left quarter, the darkest part
res = {}
for seed in SEEDS:
    rng = np.random.default_rng(seed)
    ink, gt_blocks = make_page(rng)
    noise = np.random.default_rng(100 + seed).normal(0, 4, (H, W))
    for light_name, light in (("flat", FLAT), ("uneven", UNEVEN)):
        img = render(ink, light, noise)
        res.setdefault((light_name, "ground-truth ink"), []).append(
            (100.0, np.inf, *blocks_found(ink, gt_blocks, 0), *blocks_found(ink, gt_blocks, 2), 0.0))
        for name, fn in METHODS.items():
            b = fn(img)
            f, psnr = dibco_scores(b, ink)
            false_dark = 100 * np.mean(b[DARK & ~ink])
            res.setdefault((light_name, name), []).append(
                (f, psnr, *blocks_found(b, gt_blocks, 0), *blocks_found(b, gt_blocks, 2), false_dark))

print(f"\nA. Synthetic page {H}x{W}, 6 ground-truth blocks, seeds {list(SEEDS)}: mean (min-max)")
print("light   method              F-measure %          PSNR dB   blocks matched/6 (leaves)       paper->ink %")
print("                                                          tol 0          tol 2           dark quarter")
for (light_name, name), rows in res.items():
    r = np.array(rows, float)
    rng_str = lambda k, fmt: f"{r[:, k].mean():{fmt}} ({r[:, k].min():{fmt}}-{r[:, k].max():{fmt}})"
    psnr = "    -  " if np.isinf(r[0, 1]) else f"{r[:, 1].mean():6.2f} "
    print(f"{light_name:7s} {name:18s} {rng_str(0, '5.1f'):20s} {psnr}  "
          f"{rng_str(3, '.1f')} ({r[:, 2].mean():4.1f})  {rng_str(5, '.1f')} ({r[:, 4].mean():4.1f})  "
          f"{r[:, 6].mean():5.1f}")

# ---------- Part B: the bundled page image ----------
page = data.page()
ph, pw = page.shape
py, px = np.mgrid[0:ph, 0:pw]
shadow = 1 - 0.45 * np.exp(-((px - pw) ** 2 + (py - ph) ** 2) / (2 * 120.0 ** 2))
shaded = np.clip(np.round(page * shadow), 0, 255).astype(np.uint8)

def line_bands(b, tol=2, min_h=3):
    occupied = b.sum(1) > tol
    bands, start = [], None
    for i, occ in enumerate(np.append(occupied, False)):
        if occ and start is None:
            start = i
        elif not occ and start is not None:
            if i - start >= min_h:
                bands.append((start, i))
            start = None
    return bands

print(f"\nB. skimage data.page() {ph}x{pw}; hand count: 7 text lines, 2 rules, 1 cut-off fragment")
for img_name, img in (("original", page), ("with shadow", shaded)):
    print(f"  {img_name}: mean grey left 40 cols {img[:, :40].mean():.0f}, right 40 cols {img[:, -40:].mean():.0f}")
    for name, fn in METHODS.items():
        b = fn(img)
        bands = line_bands(b)
        print(f"    {name:18s} ink in left 40 cols {100 * b[:, :40].mean():5.1f}%  "
              f"row bands {len(bands):2d}: {' '.join(f'{a}-{e}' for a, e in bands)}")

Output (NumPy 2.3.5, SciPy 1.16.3, scikit-image 0.25.2; about 2 seconds; two runs gave identical output):

NumPy 2.3.5, SciPy 1.16.3, scikit-image 0.25.2

A. Synthetic page 700x500, 6 ground-truth blocks, seeds [0, 1, 2, 3, 4]: mean (min-max)
light   method              F-measure %          PSNR dB   blocks matched/6 (leaves)       paper->ink %
                                                          tol 0          tol 2           dark quarter
flat    ground-truth ink   100.0 (100.0-100.0)      -    5.8 (5.0-6.0) ( 6.8)  5.8 (5.0-6.0) ( 6.8)    0.0
flat    Otsu (global)       99.6 ( 99.4- 99.6)   31.53   5.8 (5.0-6.0) ( 6.8)  5.8 (5.0-6.0) ( 6.8)    0.1
flat    Niblack w25 k0.2    40.7 ( 40.2- 41.4)    6.34   0.0 (0.0-0.0) ( 1.0)  0.0 (0.0-0.0) ( 1.0)   27.4
flat    Sauvola w25 k0.2    96.1 ( 96.0- 96.2)   21.88   5.8 (5.0-6.0) ( 6.4)  5.8 (5.0-6.0) ( 6.6)    0.6
flat    Sauvola w25 k0.5    99.8 ( 99.8- 99.8)   35.75   5.8 (5.0-6.0) ( 6.8)  5.8 (5.0-6.0) ( 6.8)    0.0
uneven  ground-truth ink   100.0 (100.0-100.0)      -    5.8 (5.0-6.0) ( 6.8)  5.8 (5.0-6.0) ( 6.8)    0.0
uneven  Otsu (global)       34.0 ( 33.3- 34.4)    5.10   0.0 (0.0-0.0) ( 1.0)  0.0 (0.0-0.0) ( 1.0)   86.6
uneven  Niblack w25 k0.2    41.6 ( 41.1- 42.2)    6.51   0.0 (0.0-0.0) ( 1.0)  0.0 (0.0-0.0) ( 1.0)   26.7
uneven  Sauvola w25 k0.2    96.3 ( 96.3- 96.4)   22.17   0.0 (0.0-0.0) ( 1.0)  5.8 (5.0-6.0) ( 6.8)    0.8
uneven  Sauvola w25 k0.5    99.4 ( 99.3- 99.4)   29.92   5.8 (5.0-6.0) ( 6.8)  5.8 (5.0-6.0) ( 6.8)    0.0

B. skimage data.page() 191x384; hand count: 7 text lines, 2 rules, 1 cut-off fragment
  original: mean grey left 40 cols 97, right 40 cols 226
    Otsu (global)      ink in left 40 cols 100.0%  row bands  1: 0-191
    Niblack w25 k0.2   ink in left 40 cols  25.1%  row bands  4: 0-41 49-63 67-81 85-191
    Sauvola w25 k0.2   ink in left 40 cols  15.2%  row bands  6: 13-34 50-63 67-81 86-135 157-160 170-191
    Sauvola w25 k0.5   ink in left 40 cols   8.4%  row bands  7: 13-34 51-63 68-81 86-102 103-119 120-135 172-191
  with shadow: mean grey left 40 cols 97, right 40 cols 156
    Otsu (global)      ink in left 40 cols  79.2%  row bands  1: 13-191
    Niblack w25 k0.2   ink in left 40 cols  25.0%  row bands  5: 0-8 12-42 49-63 67-81 85-191
    Sauvola w25 k0.2   ink in left 40 cols  15.1%  row bands  6: 13-34 50-63 67-81 86-135 157-160 170-191
    Sauvola w25 k0.5   ink in left 40 cols   8.3%  row bands  7: 13-34 51-63 68-81 86-102 103-119 120-135 172-191

The rows for the true ink mask set the ceiling: even on perfect binarizations XY-cut returns 6.8 leaves on average and matches 5.8 of the 6 blocks. In our reading of the code, the cause is the title: at double scale its word gaps are $2 \times (6 + 2) = 16$ pixels, exactly the minimum gap, so XY-cut cuts every title of more than one word into pieces, which would explain the extra leaves, and on the seed with only five matches the missed block is presumably the title, whose pieces are each too small to match its box. A fixed gap threshold cannot tell a large heading’s word spaces from a gutter; in our reading, this is why such thresholds have to be set for each kind of document, for example relative to the font size.

Under flat light Otsu’s threshold and both Sauvola settings come close to the true mask, and XY-cut does as well on them as on the mask itself. Niblack’s rule labels 27% of the paper pixels in the bottom-left quarter as ink, its F-measure falls to about 41%, and its noise fills every gap, so XY-cut finds no cut at all and returns the whole page as one block. Under uneven light Otsu’s single threshold labels 87% of the paper in the dark quarter as ink: F-measure 34% and, again, a single block. Sauvola’s rule is almost unaffected at pixel level (F-measure 96.3% and 99.4%), as the thresholding article also found on its synthetic page.

The layout results add something the pixel scores hide. Under uneven light, Sauvola with $k = 0.2$ labels 0.8% of the dark paper as ink, an error that costs little F-measure, but those scattered specks leave no row or column of the page completely empty, so with tol = 0 XY-cut finds no gap and returns one block. Allowing two ink pixels per empty row or column restores 5.8 of 6 blocks. With $k = 0.5$ the dark paper stays clean and both tolerances work. A downstream method can thus depend on a detail of the binarization, the density of isolated false pixels in the background, that the F-measure weighs very little. In our reading, DIBCO’s misclassification penalty metric, which weights each error by its distance from the true ink (see Evaluation), would weigh such specks more heavily.

On the bundled page image Otsu’s threshold marks the whole dark left strip as ink, every row then contains ink, and the profile shows a single band. Sauvola with $k = 0.5$ finds seven bands: the heading, the five paragraph lines and one band for the code line merged with the cut-off fragment below it. With $k = 0.2$ the last three paragraph lines merge into one band (rows 86 to 135), and the thin rule above the code line appears as a band of its own (rows 157 to 160); neither setting finds the rule under the heading, presumably because it stays under the two-pixel tolerance or the three-row minimum. The output does not show why the paragraph lines merge, but the lines of this page are visibly tilted and curved in the image, which narrows the empty rows between them, and presumably the extra ink pixels of $k = 0.2$ are enough to close them. The added shadow darkens the right side of the page, yet leaves both Sauvola results unchanged, while Otsu’s still fails; in our reading this is because Sauvola’s threshold follows the local mean, so a smooth multiplicative shading lowers threshold and pixel together. All numbers depend on the synthetic page, the window size and the gap thresholds chosen here, and a real system would also correct skew before cutting.

Challenges

• Degradation. Variable background intensity, shadows, smear and smudge, low contrast, bleed-through and show-through, the degradations the DIBCO organizers selected for (Gatos et al., 2009), and uneven illumination in camera captures.
• Geometry. Skew, page curl near a binding, folds and crumples, and perspective in photographs.
• Layout variety. Magazine pages mix columns, non-rectangular regions and headings that span several columns (Smith, 2009); in forms and receipts the fields of interest sit at positions set by each document type’s own style, as the LayoutLMv2 authors put it.
• Ambiguous ground truth. Whether two paragraphs form one region or two, or whether a blank header cell is a cell at all, is a matter of convention. The ICDAR page segmentation competitions distinguish allowable from non-allowable merges and splits (Antonacopoulos et al., 2009), and Smock et al. (2022) found oversegmented header cells in crowd-sourced table datasets.
• Domain shift. The largest training sets are generated from digital-born scientific articles (PubLayNet, DocBank, PubTabNet, PubTables-1M); scanned business documents, handwriting and historical material look different.
• Many scripts and languages. Donut’s authors built a synthetic generator partly because pre-training corpora such as IIT-CDIP are English and comparable data are not always available for other languages (Kim et al., 2022).

Evaluation

Each stage has its own benchmarks, and their protocols differ more than their names suggest.

Benchmark	Content	Split or protocol	Metric
DIBCO 2009	10 scanned images from library collections (5 machine-printed, 5 handwritten) with binary ground truth	sample images at registration, then the test set; executables submitted	F-measure, PSNR, NRM, MPM; accumulated rank
DIBCO 2017	20 images (10 printed, 10 handwritten)	test set only	measures of the DIBCO series, with weight files
ICDAR 2009 page segmentation	realistic contemporary pages (magazines, technical journals), PAGE XML ground truth	test set; executables and results submitted	PRImA measure (merge, split, miss, false detection, weighted by scenario), F-measure
PubLayNet	PubMed Central article pages, 5 classes	335,703 / 11,245 / 11,405 (training / development / test, ICDAR 2021)	COCO box mAP at IoU 0.50:0.95
DocBank	500,000 arXiv pages, 12 token classes	400,000 / 50,000 / 50,000	area-weighted token precision, recall, F1 per class
FUNSD	199 scanned forms	149 / 50	entity-level F1 (labelling)
RVL-CDIP	400,000 grayscale pages, 16 classes	320,000 / 40,000 / 40,000	accuracy
PubTabNet	568,000 table images with HTML	training / development / test (test balanced: 5,000 tables with and 5,000 without spanning cells); ICDAR 2021 final evaluation set of 9,064	TEDS
PubTables-1M	947,642 tables	758,849 / 94,959 / 93,834 (structure recognition)	AP, GriTS, exact match
DocUNet benchmark	130 phone photographs of warped documents	test only	MS-SSIM, local distortion

Binarization (DIBCO). The 2009 contest scored each binarization pixel by pixel against binary ground truth made semi-automatically, with ink as the foreground. Its four measures (Gatos et al., 2009, Section 3) are the F-measure,

$$\mathrm{FM} = \frac{2 \cdot \mathrm{Recall} \cdot \mathrm{Precision}}{\mathrm{Recall} + \mathrm{Precision}},$$

with recall and precision counted over ink pixels; PSNR, $10 \log_{10}(C^2/\mathrm{MSE})$, where $C$ is the difference between foreground and background values; the negative rate metric, the mean of the false negative and false positive rates; and the misclassification penalty metric, which weights each wrong pixel by its distance from the contour of the ground-truth object. The ranking was computed by sorting the accumulated rank of each method over all four measures, so it differs from an F-measure ranking: the fourth-ranked method has a lower F-measure (87.03%) than the fifth (87.89%). The contest’s evaluation page, last updated in December 2008, before the contest, had announced the F-measure as the ranking measure. The winner, by Lu and Tan of the Institute for Infocomm Research in Singapore, reached an F-measure of 91.24% and a PSNR of 18.66 dB (Table 1). After the contest the test images, ground truth and evaluation software were released, a policy the later editions kept. Later contests used larger test sets, 10 handwritten and 10 printed images in 2017, and an evaluation program that needs per-image weight files produced by a separate executable (DIBCO 2017 ReadMe); the DIBCO 2017 and 2019 pages state that their methodology follows DIBCO 2013, H-DIBCO 2014 and H-DIBCO 2016.

Page segmentation (ICDAR competitions). The ICDAR page segmentation competitions have run since 2001. For the 2009 edition, the fifth, Antonacopoulos et al. (2009) replaced the earlier pixel-based precision and recall with a scheme that represents ground-truth and detected regions as intervals, establishes their correspondence, and classifies the errors: wrongly detected regions, split regions, merged regions, and partially or wholly missed regions. Errors are then weighted by context and scenario. A merger of two paragraphs in the same column is allowable, because it keeps the reading order, while a merger across columns, of a caption with body text, or of text with an image is not; a missed separator matters little for text recognition, a missed heading a great deal for indexing. Regions are isothetic polygons stored in the PAGE XML format, with text, graphics, line art, separator and noise as region types. The authors report that the F-measures of the entries were close, while the new measure separated them more clearly.

Layout detection (PubLayNet, DocBank). In the ICDAR 2021 Scientific Literature Parsing competition, layout results on PubLayNet were scored with the COCO detection protocol: average precision over IoU thresholds from 0.50 to 0.95 in steps of 0.05, averaged over the five classes. The validation and test sets were manually verified, while the training labels are generated automatically. The top three systems exceeded 0.97 overall; the winner reached 0.9733, with title the hardest class (0.9607) and text the easiest (0.9838) (Jimeno Yepes et al., 2021, Table 2). The report gives two counts for participation, 281 evaluation-phase submissions from 78 teams in the introduction and more than 200 submissions from over 80 teams in the results section. DocBank’s Table 3 lists PubLayNet as 364,232 pages, more than the 358,353 of the three splits used in the competition; the sources read for this article do not explain the difference. DocBank scores token labels instead of boxes: for each class, precision is the area of correctly labelled tokens over the area of all tokens labelled with that class, and recall the same area over the area of all ground-truth tokens of the class. On its test set the macro-averaged F1 was 0.9051 for a Faster R-CNN with a ResNeXt-101 backbone, 0.9350 for LayoutLM-large on text and positions, and 0.9488 for an ensemble of the two (DocBank paper, Table 4). The DocBank paper describes its splits in two ways, as manually split in the abstract and as randomly drawn pages in Section 3.5.

Forms and document classification (FUNSD, RVL-CDIP). FUNSD’s labelling task is scored by entity-level F1. On it LayoutLMv2 reported 0.8420 for the large model against 0.7895 for LayoutLM-large (Xu et al., 2021, Table 2); its appendix states that the dataset’s official OCR annotation was used. RVL-CDIP is a subset of the IIT-CDIP Test Collection, itself drawn from the Legacy Tobacco Document Library, with 25,000 images in each of 16 classes, among them letter, memo, email, form, invoice, scientific publication and resume (official page); it is scored by overall accuracy (Harley et al., 2015, introduced the dataset). LayoutLMv2’s Table 3 lists 90.97% for an image-only VGG-16, 93.07% for the best earlier ensemble in the table (Dauphinee et al., 2019), 94.42% for LayoutLM-base with image features, and 95.25% and 95.64% for LayoutLMv2 base and large. That paper’s dataset table (Table 1) gives the RVL-CDIP split as 320K/4K/4K, that is, validation and test sets of 4,000 images each, while its appendix states an 8:1:1 split of 400,000 images, as do the official page (40,000 each) and Donut’s paper. Donut reported 95.30% without OCR, with 143 million parameters and 752 ms per image on a P40 GPU, against 1,489 ms and 200 million parameters plus an OCR engine for LayoutLMv2-base (Kim et al., 2022, Table 1); the comparison is with the base model’s 95.25%, not the large model’s 95.64%.

Tables (TEDS, GriTS). Zhong et al. (2020) proposed the tree-edit-distance-based similarity, TEDS, because the adjacency-relation metric then common in table recognition papers and competitions, which they attribute to Hurst (2003), checks only immediate neighbours among non-empty cells and requires exact matches of cell content. TEDS compares the HTML trees of predicted and true tables,

$$\mathrm{TEDS}(T_a, T_b) = 1 - \frac{\mathrm{EditDist}(T_a, T_b)}{\max(|T_a|, |T_b|)},$$

where $|T|$ is the number of nodes, insertions and deletions cost 1, and substituting one cell for another costs 1 if their row or column spans differ and otherwise a normalized Levenshtein measure of their contents in $[0, 1]$ (Zhong et al., 2020; Jimeno Yepes et al., 2021). Both papers call this cost a “normalized Levenshtein similarity”; in our reading, as a cost it must act as a distance. On the balanced PubTabNet test set, the encoder–dual–decoder reached 88.3% TEDS (91.2% on simple tables, 85.4% on tables with spanning cells), against 78.6% for an earlier image-to-markup model trained on the same data and 73.0% for the best off-the-shelf PDF tool, Camelot, which reads text from the PDF (Zhong et al., 2020, Table 2). In the ICDAR 2021 table task, the best systems reached 96.36% TEDS on the final evaluation set; the report notes that bold tags were left out of the evaluation because of a problem with the final data. PubTables-1M reports detection AP for its object classes and the GriTS family of grid-cell similarity measures; DETR trained on canonicalized annotations recognized the full content of 69.44% of complex test tables exactly, against 53.60% for DETR trained and tested on the uncanonicalized annotations (Smock et al., 2022, Table 5).

Earlier table competitions were much smaller. The Scientific Literature Parsing report describes the ICDAR 2013 table competition as the first competition on table detection and recognition, with 156 tables for evaluation and no training data, and the ICDAR 2019 competition on table detection and recognition (cTDaR) as offering 3,600 samples of historical handwritten and modern documents in three tasks: table region detection, structure recognition with given regions, and structure recognition without them, with cell boxes but no cell content in the ground truth.

Geometric correction. The DocUNet benchmark scores flattened images against flatbed scans with MS-SSIM and local distortion, as described under Common Methods. The scene text benchmarks of the conference’s Robust Reading Competition are described in ICDAR.

Failure Modes and Risks

• Errors propagate. A binarization error can disable a layout analyser, as the example shows; a layout error splits or merges text that OCR then reads in the wrong order; OCR errors enter layout-aware language models.
• Merges and splits that break reading order. Merging two columns or a caption with body text interleaves unrelated lines.
• Benchmark artefacts. Automatically generated labels carry systematic errors: the Simo team in the ICDAR 2021 layout task found unannotated text regions in the PubLayNet training set and added annotations heuristically. Scores near 0.97 mAP on PubLayNet measure agreement with such labels on one kind of document.
• Inconsistent comparisons. Results change with the OCR engine used for text-based models (LayoutLMv2 used Microsoft’s OCR; Donut’s baselines used the Microsoft and CLOVA engines), and with the model size compared; even split sizes are not always reported consistently, as the RVL-CDIP split sizes above show.
• Structured outputs that fail silently. Donut converts its token sequence to JSON and treats a field as lost if its end token never appears (Kim et al., 2022), so a malformed output shows up as a missing value rather than an error.
• Personal and confidential data. Forms, receipts, invoices and letters carry names, addresses and financial details; RVL-CDIP and IIT-CDIP are drawn from the Legacy Tobacco Document Library, and the RVL-CDIP page refers users to that library for licence information.

Deployment Considerations

• Choose the input path. Digital-born PDFs already contain text and positions, which the leading ICDAR 2021 layout systems used alongside the image; scans and photographs need the full image pipeline.
• Preprocess for the capture device. Flatbed scans mainly need binarization and skew correction; phone captures also need page detection, dewarping and illumination correction.
• Match training data to the documents. Models pre-trained on IIT-CDIP or trained on PubLayNet and DocBank see mostly English scanned business documents or scientific articles; other domains and languages need their own fine-tuning or synthetic data.
• Count the whole pipeline’s cost. OCR-based models need an OCR engine whose parameters and runtime come on top of the model’s; Donut’s comparison counts them as an unknown extra for every OCR-based baseline (Kim et al., 2022, Table 1).
• Evaluate at the level that matters. Pixel F-measure, region mAP, entity F1 and TEDS measure different things; the example and the ICDAR page segmentation scheme both show that a score at one level can hide failures at the next, so a system is best judged on the output its users consume, such as extracted fields or correctly ordered text.