Document Understanding

Document understanding is the task of extracting meaning from images of documents: reading the fields of a form or receipt into a database, answering a question about a letter, a report or an infographic, deciding whether a page is an invoice or a memo, or reading a value off a chart. Its sources call it visually rich document understanding (VrDU; LayoutLMv2), visual document understanding (VDU; Donut) or document AI (LayoutLMv3), and treat it as the model-centred successor of document image analysis, which keeps binarization, geometry, page layout analysis and the field’s history. Reading the characters is the task of optical character recognition (OCR), and text in photographs is scene text recognition. Document understanding starts where those end: the words are known, or can be read, and the question is what they mean in their place on the page.

That place is what distinguishes the task from reading comprehension in natural language processing. A form says “Due date:” and puts a date to its right or under it; a receipt lists prices in a column; an infographic puts a number next to a pictogram. The text alone, flattened into a string, loses which value belongs to which label. The methods in this article differ in how they put the layout back: by adding the 2D positions of OCR words to a text encoder (the LayoutLM family), by generating the answer directly from pixels without OCR (Donut, Pix2Struct), or by giving a general vision–language model enough resolution to read the page.

Problem Definition

Let $I$ be a page image. Most methods first run an OCR engine, which returns words $w_1, \dots, w_n$ with axis-aligned boxes $b_i = (x_0, y_0, x_1, y_1)$. The tasks then differ in their output:

• Key information extraction (KIE) assigns each word a field label, or produces a structured record. As sequence labelling, the words are serialized in some reading order and tagged in the IOB2 scheme (B-field for the first word of a value, I-field for the following words, O for everything else); LayoutLMv2 treats FUNSD, CORD, SROIE and Kleister-NDA this way, with a token classifier predicting the tags (BIO tags, in its wording). As generation, a decoder writes the record as a token sequence that converts to JSON (Donut).
• Document visual question answering returns an answer $a$ to a question $q$ about $I$. DocVQA’s answers are spans of text copied verbatim from the page, so a model can predict start and end positions in the serialized OCR text, as in extractive question answering; InfographicVQA adds answers composed of several spans, answers taken from the question and numerical answers that must be computed (Mathew et al., 2021; 2022).
• Document classification assigns one of a fixed set of classes to a page, such as the 16 RVL-CDIP classes (letter, form, invoice, resume and others).
• Chart question answering answers questions about a plot, often requiring arithmetic or comparisons over the plotted values (ChartQA).

The general question-answering task, its history and its consensus metric belong to visual question answering; here the images are pages, and the answer is usually text printed on them.

Input and Output

OCR-based models take three inputs: the token sequence, the boxes and, from LayoutLMv2 on, the image. LayoutLMv2 normalizes every coordinate to an integer in $[0, 1000]$ relative to the page size and embeds $x$ and $y$ features with separate tables, so a word’s position is part of its input embedding. The OCR is not part of the model: LayoutLMv2 extracted text and word boxes with the Microsoft Read API for pre-training and for DocVQA, and used the datasets’ official OCR annotations for FUNSD, CORD and SROIE (Xu et al., 2021, appendix).

OCR-free models take only pixels. Donut’s encoder reads a $2560 \times 1920$ image and its decoder writes a token sequence: a class token for classification, field tokens such as [START name] … [END name] for extraction, an answer after a question prompt for VQA (Kim et al., 2022). Pix2Struct goes further and renders the question itself as a header at the top of the image, so text and image enter through one channel (Lee et al., 2023). The outputs of both are text, so the evaluation must decide how close a generated string is to the gold one; that is why ANLS exists (see Evaluation Metrics).

Variants

• Forms. FUNSD’s scanned forms, whose counts and split Document Image Analysis gives, are labelled word by word as question, answer, header or other in LayoutLMv2’s semantic entity labelling task (Xu et al., 2021, appendix).
• Receipts and invoices. CORD’s public release has 1,000 Indonesian receipts with fields such as menu item name, count and price, subtotal and total, organized in nested groups (an item with its name, count and price); SROIE asks for four keys (company, date, address, total) on scanned receipts.
• Long documents. Kleister-NDA extracts four keys from long non-disclosure agreements with complex layout, with values normalized (dates as YYYY-MM-DD), so the answer is not always a span of the text (Xu et al., 2021, appendix).
• Document and infographic VQA. DocVQA and InfographicVQA.
• Charts. ChartQA questions are human-written or generated from chart summaries, and many need arithmetic.
• Classification and retrieval. RVL-CDIP (Harley et al., 2015), which the Document Image Analysis article describes with its split.

Challenges

• Reading order. OCR returns words with boxes, not a sequence. Text models need one, and the usual choice is a raster order: the DocVQA baselines serialize OCR tokens “in top-left to bottom-right order”, and Donut is pre-trained to read text “from top-left to bottom-right, basically” (Kim et al., 2022). Multi-column forms, tables and infographics break this order, and the example below shows what that costs.
• OCR errors. A wrong character in the OCR output cannot be fixed by the model reading it, which is one of the reasons Donut’s authors give for removing OCR (Kim et al., 2022). LayoutLMv2’s best SROIE score, 0.9781 entity-level F1, is the one computed “excluding OCR mismatch”; including those cases the same model scores 0.9661 (Xu et al., 2021, Table 8).
• Resolution. Documents carry small text on large pages. Donut’s accuracy on DocVQA grows rapidly with input size, and its authors note that it misses tiny text in large images (Kim et al., 2022); Pix2Struct’s authors see clearly diminishing returns on DocVQA only after about one million pixels, while standard ViTs are pre-trained at $224 \times 224$ (Lee et al., 2023, Appendix A).
• Aspect ratio. Infographics are long and narrow. Pix2Struct’s authors speculate that Donut’s fixed aspect ratio causes its InfographicVQA score of 11.6 ANLS (Lee et al., 2023).
• Structure. CORD’s nested groups and tables need outputs with hierarchy, which a flat field-level F1 does not score (Kim et al., 2022).
• Numbers. InfographicVQA and ChartQA ask for counts, sorting and arithmetic, which copying a span cannot answer.

Approaches

Text-only readers on serialized OCR

The first DocVQA baselines show where the field started. Heuristics such as picking a random OCR token score near zero, but the answer is a substring of the serialized OCR text for 87.00% of test questions (a loose bound, since “2” also matches inside “2020”; 77.00% are subsequences of the OCR tokens), so an extractive reader has room to work. BERT-large, fine-tuned first on SQuAD 1.1 and then on DocVQA with answer spans found as the first match in the serialized OCR string, reached 0.665 ANLS on the test set, against 0.391 for M4C, a scene-text VQA model, and 0.981 for humans. Leaving out the object-detection features that natural-image VQA models use made M4C slightly better, not worse (Mathew et al., 2021, Tables 1–3). The text-only readers already beat the image-based VQA models; what they lacked was the page.

Layout-aware encoders: the LayoutLM family

LayoutLM (Xu et al., 2020) added absolute 2D position embeddings of each word’s box to a BERT-style transformer encoder; Document Image Analysis outlines its pre-training. Its KDD paper was not read for this article; the LayoutLMv2 paper adds that LayoutLM combines image features with the text only at fine-tuning. The official repository lists base and large models with 113 and 343 million parameters, pre-trained on IIT-CDIP Test Collection 1.0 with the overlap with downstream datasets removed.

LayoutLMv2 (Xu et al., 2021) moves the image into pre-training. The page is resized to $224 \times 224$, passed through a ResNeXt-101-FPN backbone (initialized from a Mask R-CNN trained on PubLayNet) and average-pooled to a $7 \times 7$ grid, giving 49 visual tokens that are concatenated with up to 512 text tokens. Every token, text or visual, gets a layout embedding from its box, $(x_{\min}, x_{\max}, y_{\min}, y_{\max}, \text{width}, \text{height})$, with the visual tokens taking the boxes of their grid cells. The encoder adds learnable relative position biases to the attention scores, which the authors call spatial-aware self-attention:

$$\alpha'_{ij} = \alpha_{ij} + b^{(1D)}_{j-i} + b^{(2D_x)}_{x_j - x_i} + b^{(2D_y)}_{y_j - y_i},$$

where $\alpha_{ij}$ is the usual scaled dot-product score, $(x_i, y_i)$ is the top-left corner of token $i$’s box and the biases differ between heads but are shared across layers. This is the 2D analogue of the relative 1D biases of text transformers (see positional encoding): attention can learn that the word directly to the left of a value, or directly above it, is its label.

Pre-training uses three objectives on 11 million IIT-CDIP pages. Masked visual-language modelling masks 15% of text tokens and blanks their image regions, so the model must recover words from their positions and context. Text–image alignment covers the image regions of 15% of text lines and asks, for each token, whether its region is covered. Text–image matching asks whether page image and text belong together, with 15% of images replaced and 5% dropped. In the ablation on the DocVQA validation set (base models, one epoch of pre-training), adding the visual module to LayoutLM raised ANLS from 0.6841 to 0.6915, alignment and matching together to 0.7124, the spatial-aware attention to 0.7217, and initializing the text side from UniLMv2 instead of BERT to 0.7421 (Table 5).

LayoutLMv3 (Huang et al., 2022) was published at ACM Multimedia, which was not readable for this article; what follows comes from its official repository, model card and code. Its README describes unified text and image masking and a word–patch alignment objective, which predicts whether the image patch of a text word is masked. In the official code, the image enters as $16 \times 16$ patches through a linear (strided convolution) patch embedding instead of a CNN backbone, at an input size of 224 in the FUNSD example. The FUNSD loader in the same code gives every word of an annotated entity the box enclosing all of the entity’s words (“segment-level layout”, on by default), and maps the “other” class to O. In our reading, this means the model is told which words form one entity, so its FUNSD score is not directly comparable with models given word boxes; the README reports entity-level F1 of 0.9059 (base) and 0.9215 (large) for its fine-tuned FUNSD models. Pix2Struct’s appendix lists 83.4 ANLS on DocVQA for LayoutLMv3 among OCR-based single-task methods (Lee et al., 2023, Table 3).

OCR-free generators

Donut (Kim et al., 2022), whose encoder–decoder design and pseudo-OCR pre-training Document Image Analysis outlines, uses Swin-B with layer numbers $\{2, 2, 14, 2\}$ and window size 10 as its encoder and the first four layers of BART, initialized from a multilingual model, as its decoder. Its pre-training text for the 11 million IIT-CDIP pages comes from the CLOVA OCR API, and SynthDoG adds 2 million synthetic pages, 0.5 million per language. Fine-tuning turns every task into generating a token sequence that maps one-to-one to JSON; if a field’s end token never appears, the field counts as lost. On CORD, Donut reported a field-level F1 of 91.6 and a tree-edit-distance accuracy of 93.5, against 88.9 and 87.0 for the multilingual extension of LayoutLMv2 with OCR; on DocVQA 67.5 ANLS on the test set, below the 78.1 of LayoutLMv2-base, but 72.1 on handwritten documents (Tables 2–3). The parameter counts of the OCR-based baselines in these tables leave out the OCR engine they need.

Pix2Struct (Lee et al., 2023) changes the pre-training signal. It parses masked screenshots of web pages into simplified HTML, using 80 million screenshots crawled from the URLs in the C4 corpus, after a short warm-up in which the model learns to read rendered text. The authors argue that this objective contains OCR (recovering visible text), masked language modelling (recovering masked text) and captioning (recovering image alt-text). Its ViT encoder takes variable-resolution input: the image is scaled, keeping its aspect ratio, so that the largest number of $16 \times 16$ patches fits the sequence length, with 2D absolute position embeddings for the patches. The base and large models have 282 million and 1.3 billion parameters. Pix2Struct-Large reported 76.6 ANLS on DocVQA, 40.0 on InfographicVQA and 58.6 relaxed accuracy on ChartQA (without the gold data table), against 67.5, 11.6 and 41.8 for Donut in the same table, for which the Pix2Struct authors fine-tuned Donut themselves where no purely visual baseline existed (Table 1). The authors state plainly that they lag behind OCR-based pipelines on documents: a text-only T5 model with 2D positional biases reached 81.0 on DocVQA and 46.1 on InfographicVQA. Ablations on the DocVQA validation set (base model, 100,000 pre-training steps) show the screenshot parsing stage matters most: removing it drops ANLS from 67.8 to 12.2 (Table 2).

Vision–language models on documents

Large vision–language models treat a page as one more image, and the VLM article covers the tiling schemes, such as LLaVA-1.5-HD’s grid of crops, that raise their resolution. Document benchmarks were their weak point at first. The authors of mPLUG-DocOwl 1.5 write that earlier multimodal LLMs with more than 7 billion parameters underperformed domain-specific models with less than 1 billion; their Table 1 lists, for example, DocVQA scores of 65.1 for Qwen-VL, 65.4 for UReader and 66.5 for Monkey, against 76.6 for Pix2Struct-Large (Hu et al., 2024; the table does not name the split or metric, and the scores for the specialist models match those reported with ANLS). DocOwl 1.5 crops a high-resolution page into up to nine $448 \times 448$ sub-images plus a downsampled global view, merges four horizontally adjacent visual features with a $1 \times 4$ convolution (H-Reducer) because text runs left to right, and trains on DocStruct4M, a set of parsing and text-localization tasks over documents, web pages, tables, charts and natural images. It lists 81.6 on DocVQA, 50.4 on InfographicVQA and 70.5 on ChartQA with 8.1 billion parameters (Table 1). These systems inherit the failure modes of VLMs, including hallucinated content, and their scores depend on how answers are extracted from free text.

Example: what 2D positions add to a key–value reader

The example below renders 900 synthetic invoice-like forms with eight fields chosen so that values cannot be told apart by their content: three dates (invoice, due, ship), three amounts (subtotal, tax, total) and two names (bill to, ship to). Each value is left blank with probability 0.15, and each form uses one of four layouts: label and value on one line in one column, value under its label in one column, and the same two arrangements in two columns whose rows do not line up. Word boxes are recovered from the pixels by dilating the ink and labelling connected components; each box is then given the text of the word it covers, which stands for a perfect OCR engine, as evaluations on FUNSD and CORD use the datasets’ own OCR annotations. Values are tagged in IOB2 by field.

Two readers share a classifier, multinomial logistic regression on each token, and see the same words: every word on the page, crossed with its position relative to the token being tagged. They differ only in that position. The text-only reader gets the offset in the serialized order (up to ±8, with two buckets for “further”), as a 1D sequence model would. The layout reader gets the offset of the word’s left edge and vertical centre on a fixed pixel grid, in the spirit of LayoutLMv2’s bucketed 2D relative biases. Two serializations are compared, because “top-left to bottom-right” can mean either: grouping words into text lines by their vertical centres, or sorting by the top edge and then the left edge. Rows labelled “gold tags” decode the true tags in each order, the ceiling that order allows. Scores are entity-level F1 computed as seqeval does (below). The code uses NumPy 2.3.5, SciPy 1.16.3, Pillow 12.0.0 and scikit-learn 1.7.2, and took about 40 seconds on the machine used for drafting.

import re
import numpy as np
from PIL import Image, ImageDraw, ImageFont
from scipy import ndimage, sparse
from sklearn.linear_model import LogisticRegression

# ---------- synthetic forms, rendered and read back from the pixels ----------
FONT = ImageFont.load_default(size=14)          # Pillow's bundled FreeType font
W, H, GAP = 640, 760, 14
FIELDS = [("INVOICE_DATE", ["Invoice", "date:"], "date"), ("DUE_DATE", ["Due", "date:"], "date"),
          ("SHIP_DATE", ["Ship", "date:"], "date"), ("SUBTOTAL", ["Subtotal:"], "amount"),
          ("TAX", ["Tax:"], "amount"), ("TOTAL", ["Total:"], "amount"),
          ("BILL_TO", ["Bill", "to:"], "name"), ("SHIP_TO", ["Ship", "to:"], "name")]
FIRST = ["Anna", "Ben", "Carla", "David", "Elena", "Frank", "Grace", "Hugo", "Ines", "Jonas"]
LAST = ["Smith", "Novak", "Rossi", "Weber", "Moreau", "Silva", "Berg", "Costa", "Lund", "Kato"]
COMPANY = [["Acme", "Supplies", "Ltd"], ["Northwind", "Traders"], ["Blue", "Harbor", "Inc"]]
FOOTER = ["Thank", "you", "for", "your", "business."]
LAYOUTS = ["inline-1col", "stacked-1col", "inline-2col", "stacked-2col"]

def value_words(kind, rng):
    if kind == "date":
        return [f"{rng.integers(1, 29):02d}/{rng.integers(1, 13):02d}/{rng.integers(2019, 2026)}"]
    if kind == "amount":
        return [f"{rng.uniform(5, 5000):,.2f}"]
    return [FIRST[rng.integers(10)], LAST[rng.integers(10)]]

def phrase(words, x, y, out, ent=None):
    for k, w in enumerate(words):
        out.append(dict(text=w, x=x, y=y, ent=ent, first=k == 0))
        x += int(np.ceil(FONT.getlength(w))) + GAP
    return x

def make_form(rng):
    layout = LAYOUTS[rng.integers(4)]
    words, x0 = [], int(rng.integers(20, 60))
    phrase(["INVOICE"], x0, 15, words)
    phrase(COMPANY[rng.integers(3)], x0, 38, words)
    order = rng.permutation(len(FIELDS))
    cols = [order] if layout.endswith("1col") else [order[:4], order[4:]]
    bottom = 0
    for c, fields in enumerate(cols):
        cx = x0 if c == 0 else int(rng.integers(330, 360))
        y, ls = int(rng.integers(75, 110)), int(rng.integers(22, 31))  # each column its own grid
        for f in fields:
            ent, label, kind = FIELDS[f]
            end = phrase(label, cx, y, words)
            blank = rng.random() < 0.15                                  # unfilled field
            if layout.startswith("inline"):
                if not blank:
                    phrase(value_words(kind, rng), end + int(rng.integers(14, 40)), y, words, ent)
                y += ls
            else:
                if not blank:
                    phrase(value_words(kind, rng), cx, y + ls, words, ent)
                y += 2 * ls + int(rng.integers(4, 12))
        bottom = max(bottom, y)
    phrase(FOOTER, x0, bottom + 20, words)
    img = Image.new("L", (W, H), 255)
    d = ImageDraw.Draw(img)
    for w in words:
        d.text((w["x"], w["y"]), w["text"], font=FONT, fill=0)
        w["drawbox"] = d.textbbox((w["x"], w["y"]), w["text"], font=FONT)
    return layout, words, np.asarray(img)

def read_words(img, words):
    """Word boxes from ink (dilate, label components); each box gets the text of the word it
    overlaps, i.e. a detector from the pixels plus a perfect recognizer."""
    ink = img < 128
    lab, n = ndimage.label(ndimage.binary_dilation(ink, np.ones((5, 9), bool)))
    toks = []
    for k, sl in enumerate(ndimage.find_objects(lab), start=1):
        ys, xs = np.nonzero((lab[sl] == k) & ink[sl])
        box = (sl[1].start + xs.min(), sl[0].start + ys.min(),
               sl[1].start + xs.max() + 1, sl[0].start + ys.max() + 1)
        ov = [max(0, min(box[2], b[2]) - max(box[0], b[0])) * max(0, min(box[3], b[3]) - max(box[1], b[1]))
              for b in (w["drawbox"] for w in words)]
        toks.append(dict(words[int(np.argmax(ov))], box=box, wid=int(np.argmax(ov))))
    return toks

# ---------- tokens, serialization, tags ----------
CLOSED = sorted({w for _, l, _ in FIELDS for w in l} | {"INVOICE"} |
                {w for c in COMPANY for w in c} | set(FOOTER))
def shape(w):  # unseen values -> character shape, e.g. 12/03/2021 -> d/d/d, Anna -> Xx
    return re.sub(r"(.)\1+", r"\1", re.sub("[0-9]", "d", re.sub("[A-Z]", "X", re.sub("[a-z]", "x", w))))
SHAPES = ["d/d/d", "d,d.d", "d.d", "Xx"]
VOCAB = {w: i for i, w in enumerate(CLOSED + SHAPES + ["<unk>"])}
def tok_id(w):
    return VOCAB.get(w, VOCAB.get(shape(w), VOCAB["<unk>"]))

def order_lines(boxes):          # group into text lines by vertical centre, then left to right
    cy = (boxes[:, 1] + boxes[:, 3]) / 2
    h = np.median(boxes[:, 3] - boxes[:, 1])
    lines, cur = [], []
    for j in np.argsort(cy, kind="stable"):
        if cur and abs(cy[j] - cy[cur].mean()) >= 0.5 * h:
            lines.append(cur); cur = []
        cur.append(j)
    lines.append(cur)
    return np.array([j for ln in lines for j in sorted(ln, key=lambda j: boxes[j, 0])])

def order_yx(boxes):              # sort by top edge, then left edge
    return np.lexsort((boxes[:, 0], boxes[:, 1]))

TAGS = ["O"] + [p + e for e, _, _ in FIELDS for p in ("B-", "I-")]
def tags_in_order(toks, order):
    tags, seen = [0] * len(toks), set()
    for j in order:
        e = toks[j]["ent"]
        if e is not None:
            tags[j] = TAGS.index(("I-" if e in seen else "B-") + e)
            seen.add(e)
    return tags

# ---------- features: every token on the page, crossed with its position relative to token i ----------
DX = [-250, -120, -60, -20, -3, 3, 20, 60, 120, 250]   # pixel bucket edges, 11 x-buckets
DY = [-70, -38, -12, 12, 38, 70]                        # 7 y-buckets
def features(toks, order, mode):
    n, ids = len(toks), np.array([tok_id(t["text"]) for t in toks])
    I, J = np.meshgrid(np.arange(n), np.arange(n), indexing="ij")
    if mode == "1d":
        pos = np.empty(n, int); pos[order] = np.arange(n)
        bucket, nb = np.clip(pos[J] - pos[I], -9, 9) + 9, 19
    else:
        b = np.array([t["box"] for t in toks], float)
        x0, cy = b[:, 0], (b[:, 1] + b[:, 3]) / 2
        bucket = np.digitize(x0[J] - x0[I], DX) * 7 + np.digitize(cy[J] - cy[I], DY)
        nb = 11 * 7
    cols = (ids[J] * nb + bucket).ravel()
    return sparse.csr_matrix((np.ones(n * n), (I.ravel(), cols)), shape=(n, len(VOCAB) * nb))

# ---------- entity-level F1, chunking as seqeval 1.2.2 does for IOB2 (default mode) ----------
def chunks(tag_names):
    out, prev_tag, prev_type, start = [], "O", "", 0
    for i, t in enumerate(tag_names + ["O"]):
        tag, typ = t[0], t[2:]
        end = (prev_tag in "BI" and (tag in "BO" or prev_type != typ))
        begin = tag == "B" or (prev_tag == "O" and tag == "I") or (tag != "O" and prev_type != typ)
        if end:
            out.append((prev_type, start, i - 1))
        if begin:
            start = i
        prev_tag, prev_type = tag, typ
    return out

def entities_pred(tags, order):
    names = [TAGS[tags[j]] for j in order]
    return {(typ, frozenset(order[s:e + 1].tolist())) for typ, s, e in chunks(names)}

def entities_gold(toks):
    g = {}
    for j, t in enumerate(toks):
        if t["ent"] is not None:
            g.setdefault(t["ent"], set()).add(j)
    return {(e, frozenset(s)) for e, s in g.items()}

def dataset(n_forms, seed):
    rng = np.random.default_rng(seed)
    return [(layout, read_words(img, words)) for layout, words, img in
            (make_form(rng) for _ in range(n_forms))]

# ---------- train both readers under both serializations, score on held-out forms ----------
train, test = dataset(600, seed=0), dataset(300, seed=1)
ORDERS = {"lines": order_lines, "y-then-x": order_yx}
def run(reader, oname):
    def xy(data):
        X, y, ords = [], [], []
        for _, toks in data:
            o = ORDERS[oname](np.array([t["box"] for t in toks], float))
            X.append(features(toks, o, reader)); y += tags_in_order(toks, o); ords.append(o)
        return sparse.vstack(X).tocsr(), np.array(y), ords
    Xtr, ytr, _ = xy(train)
    Xte, yte, ords = xy(test)
    pred = LogisticRegression(C=1.0, max_iter=3000).fit(Xtr, ytr).predict(Xte)
    return pred, ords

def scores(pred_tags_per_form, ords):
    counts = {l: np.zeros(3) for l in LAYOUTS + ["all"]}      # tp, n_pred, n_gold
    for (layout, toks), tags, o in zip(test, pred_tags_per_form, ords):
        p, g = entities_pred(tags, o), entities_gold(toks)
        for k in (layout, "all"):
            counts[k] += (len(p & g), len(p), len(g))
    return [2 * c[0] / (c[1] + c[2]) for c in (counts[k] for k in ["all"] + LAYOUTS)]

sizes = np.cumsum([0] + [len(t) for _, t in test])
print(f"{'entity-level F1':24s}" + "".join(f"{k:>13s}" for k in ["all"] + LAYOUTS))
for oname in ORDERS:
    ords = [ORDERS[oname](np.array([t["box"] for t in toks], float)) for _, toks in test]
    gold = [tags_in_order(toks, o) for (_, toks), o in zip(test, ords)]
    print(f"{'gold tags, ' + oname:24s}" + "".join(f"{v:13.3f}" for v in scores(gold, ords)))
for reader in ["1d", "2d"]:
    for oname in ORDERS:
        pred, ords = run(reader, oname)
        per_form = [pred[a:b].tolist() for a, b in zip(sizes[:-1], sizes[1:])]
        name = ("text-only, " if reader == "1d" else "layout, ") + oname
        print(f"{name:24s}" + "".join(f"{v:13.3f}" for v in scores(per_form, ords)))

entity-level F1                   all  inline-1col stacked-1col  inline-2col stacked-2col
gold tags, lines                1.000        1.000        1.000        1.000        1.000
gold tags, y-then-x             0.915        0.820        1.000        0.845        0.997
text-only, lines                0.969        0.998        0.998        0.989        0.898
text-only, y-then-x             0.876        0.817        0.992        0.824        0.880
layout, lines                   0.999        1.000        1.000        0.997        1.000
layout, y-then-x                0.830        0.784        0.869        0.796        0.868

In line order, both readers are near perfect on single-column forms and on two-column forms with values beside their labels, because the label of each value is the phrase just before it in the sequence. The gap opens on two-column forms with values under their labels: the line order interleaves the two columns (“Due date: Total:” on one line, then the values), and with blank fields the same sequence can come from different layouts. The text-only reader still gets most of these right from order patterns, but only the layout reader, which sees that a value sits directly under its label, gets them all.

The sort by top edge damages every reader, and the gold rows show why: it splits entities before any model runs. In our reading, words on one line differ by a pixel or so in their top edges (capitals, digits and lower-case letters start at different heights), so the sort interleaves words of one line with each other and with words of the neighbouring column; a two-word name then stops being a contiguous span and cannot be decoded as one entity. The layout reader is hit harder than the text-only one, although its features do not depend on the order at all; its targets do, because the code tags B on whichever word of a value comes first in the serialization. Our reading, which the printed scores do not test, is that under this sort that choice turns on the one-pixel differences, which the layout reader’s coarse buckets cannot see and the text-only reader’s order offsets encode. If so, a sequence-labelling formulation ties the output to a serialization, and a model that understands the layout still has to predict the artefacts of that serialization. The example uses perfect OCR, a linear classifier and invented fields; it shows the mechanism, not the size of the effect on real forms.

Datasets and Benchmarks

Benchmark	Content	Split	Task	Metric
FUNSD	199 scanned forms; 31,485 words, 9,707 entities, 5,304 relations	149 / 50	word labelling (question, answer, header, other); relations	entity-level F1
CORD (public release)	1,000 Indonesian receipts, 30 fields in nested groups	800 / 100 / 100	field extraction or parsing	entity-level F1; field-level F1 and TED accuracy (Donut)
SROIE (Task 3)	scanned receipts, 4 keys	626 / 347	key extraction	entity-level F1
RVL-CDIP	400,000 grayscale pages, 16 classes	320,000 / 40,000 / 40,000	classification	accuracy
DocVQA	50,000 questions on 12,767 page images	39,463 / 5,349 / 5,188 questions	extractive VQA	ANLS (primary), accuracy
InfographicVQA	30,035 questions on 5,485 infographics	23,946 / 2,801 / 3,288 questions	VQA with multi-span and numerical answers	ANLS, accuracy
ChartQA	9,608 human-written and 23,111 generated questions	test: 1,250 + 1,250 questions	chart QA	relaxed accuracy

FUNSD (Jaume et al., 2019; the paper was not readable, so the counts come from the official page and the split from LayoutLMv2’s appendix) is small, so results move with a handful of entities. CORD (Park et al., 2019; not read) is described here from its official repository: the whole collection is given as over 11,000 receipts with five superclasses and 42 subclasses, but the released 1,000 receipts use 30 classes after the store, payment and miscellaneous fields were removed for Indonesian legal reasons; a 2022 version 2 corrected labels. RVL-CDIP was sampled at random from IIT-CDIP for Harley et al. (2015); their best model, a holistic CNN fine-tuned from ImageNet, reached 89.8% accuracy, above an ensemble of region-specific CNNs (89.3%).

DocVQA (Mathew et al., 2021) takes page images from 6,071 documents of the UCSF Industry Documents Library dated from 1900 to 2018, mostly 1960–2000, favouring pages with tables, forms, lists and figures and keeping binarized scans, whose quality had suffered, to a minimum. Remote workers wrote up to ten questions per page with answers copied verbatim; a second worker answered the same questions and assigned question types, and disagreements went to the authors for review. Each question is tagged with one or more of nine types (such as form, table/list, layout, handwritten, figure). The split is random, 80/10/10, and test results come from the evaluation site.

InfographicVQA (Mathew et al., 2022) collects infographics from 2,594 web domains. Answers may be a span of the image, a span of the question, several spans (entered comma-separated; every permutation counts as correct), or a number that does not appear anywhere, allowed only for numerical answers so that answers stay short and unique. Humans scored 0.980 ANLS on the test set; the best baseline, LayoutLM with in-domain pre-training, 0.272.

ChartQA (Masry et al., 2022) uses charts from Statista, Pew, Our World in Data and OECD, with human-written questions (ChartQA-H) and questions generated from human-written Statista summaries (ChartQA-M). Its baselines read the chart through an extracted data table; VisionTaPas scored 61.84% on the test set with the gold table and 45.52% without it (Table 5), and Pix2Struct’s 58.6 is reported without the gold table.

Reported DocVQA test results, all in ANLS, illustrate how much the set-ups differ:

Model	Inputs	Fine-tuning data	ANLS (test)	Source
BERT-large (SQuAD, then DocVQA)	OCR text	train	0.665	Mathew et al. (2021), Table 3
LayoutLM-large	OCR text and layout	train	0.7259	Xu et al. (2021), Table 4
LayoutLMv2-large	OCR, layout, image	train	0.8348	Xu et al. (2021), Table 4
LayoutLMv2-large + QG	OCR, layout, image	question generation data, then train + dev	0.8672	Xu et al. (2021), Table 4
Donut	pixels	train	67.5	Kim et al. (2022), Table 3
Pix2Struct-Large	pixels	train	76.6	Lee et al. (2023), Table 1

The LayoutLMv2 abstract gives 0.7295 for LayoutLM on DocVQA, while its Table 4 lists 0.7259 for LayoutLM-large; the table is reported here. Pix2Struct’s tables do not name the split; the Donut score they list equals Donut’s test-set score. The leaderboard’s best entry at the time, an ensemble of 30 models, scored 0.8506 (Table 4). Rows differ in OCR engine, in whether validation data or extra question-generation data were used, and in model size, so they show progress, not a controlled comparison.

Evaluation Metrics

Entity-level F1

KIE benchmarks are scored by F1 over entities, not words. With $\mathrm{TP}$ the number of predicted entities that match a gold entity,

$$P = \frac{\mathrm{TP}}{|\text{predicted}|}, \qquad R = \frac{\mathrm{TP}}{|\text{gold}|}, \qquad F_1 = \frac{2PR}{P + R} = \frac{2\,\mathrm{TP}}{|\text{predicted}| + |\text{gold}|}.$$

In the official LayoutLM and LayoutLMv3 fine-tuning code the computation is seqeval’s: predicted and gold IOB2 tags are cut into chunks (type, start, end), a predicted chunk is a true positive only if type and both ends match exactly, and the counts are pooled over all entity types (micro average; the Hugging Face wrapper used by LayoutLMv3’s script reports the “micro avg” row as overall F1). In seqeval’s default mode an I- tag after an O starts a new chunk, so a prediction that gets a value’s words right but tags its first word I- instead of B- still finds the entity; seqeval’s strict mode, which these scripts do not use, would not count it. Because FUNSD’s “other” words are mapped to O, its F1 covers only questions, answers and headers. See precision and recall for the underlying quantities.

Donut scores extraction differently: field-level F1 checks whether each extracted field value is in the ground truth, so one wrong character fails the field, and a tree-edit-distance accuracy, $\max(0, 1 - \mathrm{TED}(\text{pr}, \text{gt}) / \mathrm{TED}(\phi, \text{gt}))$ with $\phi$ the empty tree, also scores the nesting of groups (Kim et al., 2022). LayoutLMv2’s CORD F1 of 0.9601 (entity-level, on BIO tags) and the 88.9 field-level F1 that Donut reports for LayoutLMv2’s multilingual extension are therefore not the same measurement.

Average normalized Levenshtein similarity (ANLS)

ANLS was proposed for scene-text VQA (Biten et al., 2019) and adopted by DocVQA as its primary metric so that small OCR errors in an answer are not scored as wrong (Mathew et al., 2021); InfographicVQA uses the same evaluation. For question $i$ with gold answers $a_{ij}$ and prediction $o_i$,

$$\mathrm{ANLS} = \frac{1}{N} \sum_{i=1}^{N} \max_j \, s(a_{ij}, o_i), \qquad s(a, o) = \begin{cases} 1 - \mathrm{NL}(a, o) & \text{if } \mathrm{NL}(a, o) < \tau \\ 0 & \text{otherwise,} \end{cases}$$

where $\mathrm{NL}$ is the normalized Levenshtein distance, a value in $[0, 1]$ (both implementations below divide the distance by the length of the longer string), and $\tau = 0.5$. Biten et al. motivate the threshold by reading a distance above 0.5 as the wrong text having been selected, not a recognition error. The companion accuracy metric counts exact matches with any gold answer.

The official evaluation runs on the Robust Reading Competition site, whose code could not be read for this article (the site failed a TLS certificate check). The Pix2Struct code says of DocVQA: “There does not seem to be an official evaluation script.” Two public implementations by the metric’s users differ in small ways: Pix2Struct’s lower-cases both strings and keeps $1 - \mathrm{NL}$ when $\mathrm{NL} < 0.5$; the MP-DocVQA framework of Tito, one of the DocVQA organizers, lower-cases and strips surrounding whitespace, scores an empty prediction 0, and keeps the similarity when it is at least 0.5, that is when $\mathrm{NL} \le 0.5$. The script below implements both, together with Pix2Struct’s relaxed accuracy for ChartQA, and runs in well under a second.

def levenshtein(a, b):
    prev = list(range(len(b) + 1))
    for i, ca in enumerate(a, 1):
        cur = [i]
        for j, cb in enumerate(b, 1):
            cur.append(min(prev[j] + 1, cur[j - 1] + 1, prev[j - 1] + (ca != cb)))
        prev = cur
    return prev[-1]

def nl(gold, pred):                      # normalized Levenshtein distance, in [0, 1]
    return levenshtein(gold, pred) / max(len(gold), len(pred), 1)

def anls_pix2struct(golds, pred):        # pix2struct/metrics.py: lower-case, keep 1 - NL if NL < 0.5
    return max((1 - d) if (d := nl(g.lower(), pred.lower())) < 0.5 else 0.0 for g in golds)

def anls_mpdocvqa(golds, pred):          # MP-DocVQA evaluator: lower-case and strip, keep if similarity >= 0.5
    p = pred.lower().strip()
    if not p:
        return 0.0
    s = max(1 - nl(g.lower().strip(), p) for g in golds)
    return s if s >= 0.5 else 0.0

def relaxed_correct(target, pred, tol=0.05):   # pix2struct/metrics.py relaxed_correctness (ChartQA)
    def num(t):
        try:
            return float(t.rstrip("%")) / 100 if t.endswith("%") else float(t)
        except ValueError:
            return None
    p, t = num(pred), num(target)
    if p is not None and t:                     # a target of 0 falls through to string comparison
        return abs(p - t) / abs(t) <= tol
    return pred.lower() == target.lower()

cases = [(["80202"], "80202"), (["Great western sugar Co."], "great western sugar co."),
         (["23 sep 1970"], "23 sep 1970 "), (["(910) 741-0673"], "(910) 741-O673"),
         (["1970"], "1985"), (["540"], "943"),
         (["dr. william j. darby", "william j. darby"], "William J. Darby"),
         (["indications for implantation"], ""), (["total"], "the total is 2,000")]
print(f"{'gold answers':42s} {'prediction':22s} {'NL':>5s} {'pix2struct':>10s} {'MP-DocVQA':>10s}")
for golds, pred in cases:
    d = min(nl(g.lower(), pred.lower()) for g in golds)
    print(f"{' | '.join(golds):42s} {repr(pred):22s} {d:5.3f} {anls_pix2struct(golds, pred):10.3f} "
          f"{anls_mpdocvqa(golds, pred):10.3f}")
s1 = sum(anls_pix2struct(g, p) for g, p in cases) / len(cases)
s2 = sum(anls_mpdocvqa(g, p) for g, p in cases) / len(cases)
print(f"ANLS over the {len(cases)} questions: {s1:.3f} (pix2struct)  {s2:.3f} (MP-DocVQA)")
print()
for target, pred in [("96", "100"), ("94", "100"), ("96%", "0.96"), ("0", "0.0"), ("Yes", "yes"),
                     ("2015", "1950"), ("Brazil", "brazil ")]:
    print(f"relaxed accuracy  target {target!r:8s} prediction {pred!r:9s} -> {relaxed_correct(target, pred)}")

gold answers                               prediction                NL pix2struct  MP-DocVQA
80202                                      '80202'                0.000      1.000      1.000
Great western sugar Co.                    'great western sugar co.' 0.000      1.000      1.000
23 sep 1970                                '23 sep 1970 '         0.083      0.917      1.000
(910) 741-0673                             '(910) 741-O673'       0.071      0.929      0.929
1970                                       '1985'                 0.500      0.000      0.500
540                                        '943'                  0.667      0.000      0.000
dr. william j. darby | william j. darby    'William J. Darby'     0.000      1.000      1.000
indications for implantation               ''                     1.000      0.000      0.000
total                                      'the total is 2,000'   0.722      0.000      0.000
ANLS over the 9 questions: 0.538 (pix2struct)  0.603 (MP-DocVQA)

relaxed accuracy  target '96'     prediction '100'     -> True
relaxed accuracy  target '94'     prediction '100'     -> False
relaxed accuracy  target '96%'    prediction '0.96'    -> True
relaxed accuracy  target '0'      prediction '0.0'     -> False
relaxed accuracy  target 'Yes'    prediction 'yes'     -> True
relaxed accuracy  target '2015'   prediction '1950'    -> True
relaxed accuracy  target 'Brazil' prediction 'brazil ' -> False

The two implementations agree except in two cases: a trailing space, which only the MP-DocVQA version strips, and a distance of exactly 0.5 (“1970” against “1985”), which Biten et al.’s definition and Pix2Struct score 0 and the MP-DocVQA version scores 0.5. On nine questions the difference is 0.065 ANLS; on a real test set such ties are rare, but a whitespace convention applied to every answer is not, so scores from different scripts are best not compared to the third decimal. One misread character costs only $1/14$ of the score on a 14-character phone number, while the wrong number “943” for “540” (the Donut paper’s own example) scores 0, as the threshold intends. A correct answer wrapped in a sentence (“the total is 2,000”) also scores 0, which matters for generative models that do not answer tersely.

Relaxed accuracy

ChartQA follows PlotQA’s relaxed accuracy: a numeric answer is correct within 5% of the gold value, and other answers need an exact match (Masry et al., 2022). In Pix2Struct’s implementation, a percentage is divided by 100 before comparison, so “0.96” matches “96%”; strings are compared case-insensitively; and a gold value of 0 falls through to string comparison, so “0.0” fails against “0”. The rule also applies to years: “1950” is within 5% of “2015” and counts as correct. For other answers, a trailing space is enough to fail the exact match.

Accuracy

Document classification uses plain accuracy over the test pages.

Applications

• Business documents: extracting fields from invoices, receipts, forms and contracts into structured records, the setting of SROIE, CORD, Kleister-NDA and Donut’s in-service receipt and business-card data (Kim et al., 2022).
• Archives and search: classifying and querying large scanned collections such as IIT-CDIP and the UCSF Industry Documents Library, whose pages make up RVL-CDIP and DocVQA.
• Question answering over reports, infographics and charts: DocVQA’s authors describe their task as a “purpose-driven” view of document analysis, in which a question drives which parts of the page are read (Mathew et al., 2021).

Documents of these kinds carry names, addresses and financial details; the document image analysis article discusses the confidentiality and licensing of the collections.

Open Problems

• Reading order and output format. Text-only and sequence-labelling models depend on a serialization that real layouts do not define; generative models trade this for the problem of scoring free text.
• Resolution against cost. Pixel-only and VLM approaches need about a million pixels or more per page to read small print (Lee et al., 2023), and every visual token adds cost.
• OCR-free against OCR-based. Pix2Struct’s authors report that OCR-based and even text-only pipelines remained ahead on DocVQA, while their OCR-free model was ahead in what they call low-resource domains, such as illustrations and user interfaces.
• Comparable evaluation. Results depend on the OCR engine, on the input boxes (word, line or gold segment), on training with validation or extra data, and on the scoring script; the official ANLS code is not public in the sources read here.
• Reasoning over numbers and structure. InfographicVQA’s baselines stayed far below human performance (0.272 against 0.980 ANLS), and ChartQA’s best model without the gold data table answered 45.52% of test questions correctly.