MVTec AD

MVTec AD (the MVTec Anomaly Detection dataset) is a benchmark for finding defects in images when only defect-free examples are available for training. Paul Bergmann, Michael Fauser, David Sattlegger and Carsten Steger of MVTec Software GmbH, a machine vision company, released it with a paper at CVPR 2019. It holds 5,354 colour and grey-scale images of five textures and ten industrial objects. The training images are all defect-free; the test images include defect-free parts and parts with scratches, dents, contaminations, missing pieces and other faults, each defect outlined by a pixel-precise mask (Bergmann et al., 2019). An extended journal version followed in the International Journal of Computer Vision in 2021 (not read here), and MVTec distributes Python scripts that fix how results are scored (the version read here is dated October 2022). A successor, MVTec AD 2, is described in a 2026 journal paper.

This article covers the dataset as an object (acquisition, categories, splits, files, licence and versions) and the official evaluation as its code computes it, checked here against reimplementations in circulation. The methods benchmarked on MVTec AD, and the definitions of image AUROC, pixel AUROC and the per-region overlap, belong to Visual Anomaly Detection; the imaging and operating thresholds of inspection belong to Industrial Visual Inspection.

Purpose

Before MVTec AD, the few public datasets for segmenting anomalies all covered textured surfaces. Bergmann et al. (2019, Section 2.1.2) describe NanoTWICE, 45 grey-scale microscope images of one nanofibrous material, and the DAGM 2007 set of artificially generated textures with defects marked only by coarse ellipses. They argue that industrial inspection needs unsupervised anomaly detection because inspection tasks often lack defective samples, or it is unclear which defects may appear (Section 1), and they built MVTec AD to give such methods real objects and textures, many defect kinds and exact masks. Their stated aim was a comprehensive, multi-object, multi-defect dataset with pixel-accurate regions focused on real applications (Abstract).

Dataset Size

The dataset has 3,629 images for training and validation and 1,725 for testing (Bergmann et al., 2019, Section 3). Table 1 of the paper breaks them down by category:

Category	Train	Test (good)	Test (defective)	Defect groups	Defect regions	Image side (px)
Carpet	280	28	89	5	97	1024
Grid	264	21	57	5	170	1024
Leather	245	32	92	5	99	1024
Tile	230	33	84	5	86	840
Wood	247	19	60	5	168	1024
Bottle	209	20	63	3	68	900
Cable	224	58	92	8	151	1024
Capsule	219	23	109	5	114	1000
Hazelnut	391	40	70	4	136	1024
Metal nut	220	22	93	4	132	700
Pill	267	26	141	7	245	800
Screw	320	41	119	5	135	1024
Toothbrush	60	12	30	1	66	1024
Transistor	213	60	40	4	44	1024
Zipper	240	32	119	7	177	1024
Total	3,629	467	1,258	73	1,888

The table calls the defect types “defect groups”; the text rounds the region count to “almost 1900” and the abstract the defect types to “over 70”. Category sizes vary by a factor of more than six in training images (60 toothbrushes, 391 hazelnuts), and in transistor the good test images outnumber the defective ones.

Acquisition was uniform: a 2048 × 2048 industrial RGB sensor with two bilateral telecentric lenses of magnification 1:5 and 1:1, highly controlled illumination that was varied on purpose for some objects, and crops to square images between 700 and 1,024 pixels a side (Section 3). Industrial Visual Inspection explains why telecentric lenses suit measurement.

Classes

The 15 categories are five textures and ten objects (Section 3):

• Regular textures: carpet and grid. Random textures: leather, tile and wood.
• Rigid objects with a fixed appearance: bottle and metal nut. Deformable: cable. With natural variation: hazelnut. The others are capsule, pill, screw, toothbrush, transistor and zipper.
• Pose. Some objects were photographed roughly aligned (toothbrush, capsule and pill, for example), others placed with a random rotation (metal nut, screw and hazelnut). This matters for methods that model each pixel position separately, such as the variation model, which the paper could apply only to the objects it could align (Section 4.1.5).
• Grey-scale categories. Grid, screw and zipper are released as single-channel images only, because grey-scale images are also common in industrial inspection.

The 73 defect types average about five per category. They range from surface defects such as scratches and dents to structural ones such as distorted parts, and defects that show as the absence of a part. The defects were made by hand to resemble those of real production (Section 3). Each defect type has its own test folder, beside a good folder of defect-free test images (evaluation code, evaluate_experiment.py).

Annotation Types

• Image labels are implicit: an image in a good folder is defect-free, any other is defective.
• Pixel masks. Every defective test image has a binary ground-truth mask, ground_truth/<defect type>/<image id>_mask.png, marking the anomalous pixels; good images have none (evaluation code, evaluate_experiment.py). The paper calls the masks pixel-precise and manually annotated (Section 3).
• Defect regions. The masks are not labelled by region. The official code splits each mask into connected components, and these 1,888 components are the regions that region-based metrics average over. How a mask splits into regions depends on the connectivity used, which the example below shows.

Splits

Each category is a separate one-class problem: a training folder of defect-free images, and a test folder with good and defective images (Table 1). No validation split is fixed. The 2019 benchmark drew validation images at random from the training set to estimate thresholds (Section 4.3); Visual Anomaly Detection describes the fixed 10% split of the journal version and the later habit of reporting threshold-free metrics on the test set. The training set contains only defect-free images (Section 3), so methods are fitted without seeing any defect.

Tasks

• Image-level anomaly detection: score each test image so that defective images rank above good ones.
• Anomaly localization (segmentation): produce an anomaly map, one real score per pixel, that marks the defects; see image segmentation for the general problem.

Both are trained per category on good images only. Industrial Visual Inspection discusses the step that the benchmark leaves out, choosing an operating threshold.

Evaluation

The 2019 protocol

The conference paper estimated a threshold for each method except the GMM-based texture inspection model, which supplies its own, from defect-free validation images: it raised the threshold until the largest connected component of the thresholded validation maps fell just below a minimum defect area set per category (Section 4.3). At that threshold it reported, for classification, the fraction of correctly classified good and defective images (Table 2), and for segmentation the relative per-region overlap with the ground truth (Table 3). As a threshold-independent measure it added the pixel-level ROC AUC, with the true-positive rate the fraction of anomalous pixels flagged and the false-positive rate the fraction of good pixels flagged, over a category (Section 4.3). The evaluation code of 2022 instead computes the area under the per-region overlap curve up to a false-positive rate of 0.3, which Visual Anomaly Detection defines from the journal version.

The official evaluation code

MVTec’s page links a set of evaluation scripts, offered for a consistent comparison of methods on the dataset (dataset page, as of November 2026). The archive read here is version 1.0, its files dated 13 October 2022, under the BSD 3-Clause License with “Copyright 2022 MVTec Software GmbH” in LICENSE.txt; the headers of evaluate_experiment.py and print_metrics.py say 2021. The README asks for a Python 3.7 installation, and requirements.txt says “Please use Python 3.7 or newer” and pins NumPy 1.18.5 and SciPy 1.7.1. As evaluate_experiment.py, pro_curve_util.py, roc_curve_util.py and generic_util.py compute it:

• Input. One anomaly map per test image, a TIFF file at <anomaly_maps_dir>/<category>/test/<defect type>/<image id>.tiff, real-valued and of the size of the original image. Ground-truth masks are read with Pillow; a good image gets an all-zero mask of the map’s shape.
• Regions. compute_pro labels each mask with scipy.ndimage.label and the structuring element np.ones((3, 3)), so regions are 8-connected. SciPy’s own default is the cross-shaped 4-connected element (Connected Component Labeling compares the defaults of SciPy and scikit-image).
• The PRO curve. Every pixel of every test image of the category, good images included, is sorted by score in decreasing order. A good pixel adds 1 to a false-positive count; a pixel of region $k$ adds $1/|C_k|$ to the overlap sum. Cumulative sums divided by the number of good pixels and by the number of regions give the false-positive rate (FPR) and the per-region overlap (PRO) after each pixel. Runs of equal scores are merged so that only the last point of each run is kept, the values are clipped to 1, and $(0, 0)$ and $(1, 1)$ are added at the ends. Every distinct score is therefore a threshold, and the curve is exact. Two implementation details: the FPR is cast to 32-bit floats after the cumulative sum, and the false-positive counts are 32-bit unsigned integers, guarded by an assertion that the number of pixels stays below the largest such integer, $2^{32} - 1$.
• The area. trapezoid(x, y, x_max) integrates by the trapezoidal rule up to x_max, adding the sliver between the last point and x_max with the PRO interpolated linearly, and asserts that x_max lies inside the curve. evaluate_experiment.py divides the area by the limit, so a perfect map scores 1. The limit is an argument, --pro_integration_limit, with default 0.3 and allowed range $(0, 1]$.
• Image-level score. compute_classification_roc scores each image by np.max of its map, sorts, merges ties and returns the ROC curve; trapezoid without a limit gives the full area.
• Aggregation. Both measures are computed per category, and the script reports their means over the 15 categories (mean_au_pro, mean_classification_au_roc).

Three differences between the code and the published protocols stand out. The code computes no pixel-level ROC AUC, the threshold-free measure of the 2019 paper; anyone reporting pixel AUROC on MVTec AD computes it with their own code, and Visual Anomaly Detection notes that the journal version integrated its ROC areas only up to an FPR of 0.3 as well. The code computes no threshold-dependent measure, so the 2019 tables of accuracies and overlaps at an estimated threshold cannot be reproduced with it. And the FPR is computed over all good pixels of all test images, so adding good test images adds negatives to the localization measure; Visual Anomaly Detection discusses that choice for VisA.

The scripts import label from scipy.ndimage.measurements. In SciPy 1.16.3 that module is a deprecated alias: its source (scipy/ndimage/measurements.py) hands the name on to the private module that also supplies scipy.ndimage.label, with a warning that the namespace will be removed in SciPy 2.0.0. The scripts as distributed were not run here.

Implementations in circulation

Method repositories and libraries also compute AU-PRO with their own code. Three that we read:

• anomalib (third-party, Apache-2.0, release lib/v2.7.0, commit 9fb1337). Its AUPRO metric computes the exact curve in the same way as MVTec’s code, sorting all pixels and merging tied scores, but casts the scores to 32-bit floats before sorting, which can merge scores that differ in 64 bits, and divides the area by the last FPR of the clipped curve. Its connected components come from OpenCV on the CPU and from kornia on the GPU, both 8-connected. Its unit test compares the metric with a reference, aupro_reference.py, whose docstring says it is based on MVTec’s official 3D-AD evaluation code and whose licence file says it is based on code by Eliahu Horwitz and Yedid Hoshen (MIT), itself based on MVTec’s code (BSD-3). That reference samples the curve at a fixed number of thresholds (100 by default), taken at evenly spaced positions of the sorted good-pixel scores over the whole FPR range, so about 30 of them fall below 0.3. anomalib’s metric needs PyTorch, torchmetrics, kornia and OpenCV and was read, not run; the reference was ported below.
• CFLOW-AD (third-party, BSD-3, commit b2ebf9e of gudovskiy/cflow-ad, the repository named in the CFLOW-AD paper). train.py credits its AUPRO code to the DFR repository. It sets 1,000 thresholds evenly spaced in score from the maximum down to just above the minimum, labels regions with scikit-image at connectivity=2 (8-connected), measures each region’s overlap with regionprops’ filled_image (holes filled) divided by its area, keeps the points with FPR ≤ 0.3, rescales those FPRs linearly so that the smallest becomes 0 and the largest 1, and takes scikit-learn’s auc. The rescaling divides by the largest grid FPR not above 0.3 rather than by 0.3, and on a mask with holes the filled image counts predicted pixels inside a hole toward the overlap. Gudovskiy et al. (2022, Table 4) report 94.60% AUPRO on MVTec AD for CFLOW-AD with a WideResNet-50 encoder, in a comparison table that also lists other methods’ published values.
• Industrial Visual Inspection on this site. Its example defines au_pro, which samples the curve at 300 thresholds taken at evenly spaced positions of the good-pixel scores between FPR 0 and just below 0.3, stops at the last sample instead of interpolating to 0.3, and divides by 0.3. On a map whose good-pixel scores are all equal, for instance a binary map, all 300 thresholds coincide and the area is 0. How much either property changes that article’s printed results was not computed here; the example below measures both on its own maps.

We found no readable implementation that labels regions with 4-connectivity, and none that normalizes the FPR per image, so the second is left out of the example; the first, in our reading, needs only a call to scipy.ndimage.label without a structuring element.

Example: the official AU-PRO against the approximations

The example builds a synthetic test set with exact defect masks and scores the same anomaly maps with six computations of AU-PRO at the official limit of 0.3:

1. exact: our reimplementation of the official computation, in 64-bit floats, written independently (np.unique merges ties, np.bincount accumulates the changes);
2. official: MVTec’s compute_pro and trapezoid re-typed into the script with their licence notice, with comments and progress prints removed and label imported from scipy.ndimage, and run on the in-memory arrays. The code was written for NumPy 1.18. NumPy 2.0 adopted NEP 50’s promotion rules (NumPy 2.0.0 release notes), under which a 32-bit float scalar combined with a Python float stays in 32 bits, where NumPy 1 returned a 64-bit result (NEP 50, table of changed behaviours). A second copy of trapezoid emulates that older behaviour;
3. IVI-300: Industrial Visual Inspection’s au_pro, its body copied unchanged;
4. grid-100: anomalib’s test reference ported line by line, 100 thresholds;
5. CFLOW: CFLOW-AD’s computation, re-typed with the loop over thresholds vectorized (regions are labelled once and the counts above each threshold come from sorted scores); the script checks it against the literal loop body at three thresholds;
6. exact-4c: the exact computation with 4-connected regions. This is not an approximation in circulation but a sensitivity condition: what the official computation gives if its structure argument is dropped and SciPy’s default applies.

The data are 100 maps of 256 × 256 pixels: 40 good images, 30 with one to three one-pixel-wide scratches at 45°, 29 with one to four small discs and one with a single 80 × 60 “missing part”. Defects keep a two-pixel gap, so each one is one 8-connected region, while a 45° scratch of $L$ pixels is $L$ separate 4-connected regions (Connected Component Labeling counts the same case). The anomaly maps are error models built from the masks, not trained networks: A blurs the mask with a Gaussian of $\sigma = 1$ and adds white noise of standard deviation 0.35; B blurs it with $\sigma = 4$, doubles it and adds correlated noise (white noise blurred with $\sigma = 2$, scaled to standard deviation 0.25). By construction, both score a thin scratch lower than a disc or the large region, B more so. Three more map sets test the computations: oracle_bin (the map is the mask), oracle_sep (defect scores in [1, 1.5], good-pixel scores in [0, 0.5], so they never overlap) and random (uniform scores independent of the masks, for which the expected AU-PRO is $\int_0^{0.3} f\,df / 0.3 = 0.15$).

The hypotheses were fixed before the code ran, from the code and from derivation. The official port should agree with our exact version to within $10^{-6}$ and the NumPy 1 emulation within $10^{-9}$. On oracle_bin, every computation should give 1 except IVI-300, which should give 0, and CFLOW, which should give nan (all FPRs are 0, and the rescaling divides 0 by 0). The three grids should all fall below the exact value for A and B, IVI-300 by less than $10^{-3}$ and grid-100 by more than IVI-300, and none should reverse the order of A and B. 4-connectivity should lower AU-PRO for both models, by more than any of the grid errors, because it turns each scratch into many one-pixel regions whose mean overlap equals the scratch’s own overlap: the scratch’s weight in the average grows from one region to $L$. Image AUROC from the official function should match scikit-learn’s roc_auc_score to within $10^{-12}$.

import os
for v in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS", "VECLIB_MAXIMUM_THREADS"):
    os.environ[v] = "1"                        # one BLAS thread, set before NumPy is imported
from bisect import bisect
import numpy as np
from scipy import ndimage as ndi
from skimage.draw import disk
from skimage.measure import label as sk_label, regionprops
from sklearn.metrics import auc, roc_auc_score

# ---------------------------------------------------------------- synthetic test set
S, N_GOOD, N_SCR, N_SPOT, N_LARGE = 256, 40, 30, 29, 1
EIGHT, FOUR = np.ones((3, 3), int), ndi.generate_binary_structure(2, 1)

def make_masks(rng):
    """Good images, then images with 1-3 one-pixel 45-degree scratches, 1-4 small spots,
    or one large 80 x 60 missing-part region. Defects never touch (2-pixel margin)."""
    masks, placed = [], []
    kinds = ["good"] * N_GOOD + ["scratch"] * N_SCR + ["spot"] * N_SPOT + ["large"] * N_LARGE
    for i, kind in enumerate(kinds):
        m = np.zeros((S, S), bool)
        n = {"good": 0, "scratch": rng.integers(1, 4), "spot": rng.integers(1, 5), "large": 1}[kind]
        k = 0
        while k < n:
            d = np.zeros((S, S), bool)
            if kind == "scratch":
                L = int(rng.integers(20, 61)); sgn = rng.choice([-1, 1])
                r0 = int(rng.integers(8, S - 8 - L)); c0 = int(rng.integers(8 + L, S - 8 - L))
                ii = np.arange(L); d[r0 + ii, c0 + sgn * ii] = True
            elif kind == "spot":
                rr, cc = disk(tuple(rng.integers(12, S - 12, 2)), rng.uniform(2, 5), shape=(S, S)); d[rr, cc] = True
            else:
                r0, c0 = rng.integers(20, S - 100, 2); d[r0:r0 + 80, c0:c0 + 60] = True
            if (ndi.binary_dilation(d, np.ones((5, 5))) & m).any():
                continue
            m |= d; placed.append((i, kind, int(d.sum()))); k += 1
        masks.append(m)
    return np.stack(masks), placed

def make_maps(rng, masks):
    """Score-map models built from the ground truth (no network): A is sharp with
    white noise, B is blurred with correlated noise; two oracles and random scores."""
    mf = masks.astype(float)
    A = np.stack([ndi.gaussian_filter(x, 1) for x in mf]) + 0.35 * rng.standard_normal(mf.shape)
    nB = np.stack([ndi.gaussian_filter(z, 2) for z in rng.standard_normal(mf.shape)])
    B = np.stack([2 * ndi.gaussian_filter(x, 4) for x in mf]) + 0.25 * nB / nB.std()
    sep = mf + 0.5 * rng.uniform(size=mf.shape)        # defects in [1, 1.5], background in [0, 0.5]
    rnd = rng.uniform(size=mf.shape)
    return dict(oracle_bin=mf.copy(), oracle_sep=sep, random=rnd, A=A, B=B)

# ---------------------------------------------------------------- (1) our exact AU-PRO
def au_pro_exact(maps, masks, lim=0.3, structure=EIGHT):
    """Every distinct score is a threshold (ties merged); FPR over all good pixels of all
    images; PRO = mean over ground-truth regions of the detected fraction; area up to lim
    with the curve cut by linear interpolation, divided by lim. All in float64."""
    w = np.zeros(maps.shape); n_reg = 0
    for i, m in enumerate(masks):
        lab, n = ndi.label(m, structure); n_reg += n
        sizes = np.bincount(lab.ravel())
        w[i][lab > 0] = 1.0 / sizes[lab[lab > 0]]
    bg = (~masks).ravel().astype(float)
    _, inv = np.unique(-maps.ravel(), return_inverse=True)
    F = np.r_[0, np.cumsum(np.bincount(inv, weights=bg))] / bg.sum()
    P = np.r_[0, np.cumsum(np.bincount(inv, weights=w.ravel()))] / n_reg
    k = np.searchsorted(F, lim, side="right")
    a = np.trapezoid(P[:k], F[:k])
    if k < len(F) and F[k - 1] < lim:
        p_lim = P[k - 1] + (P[k] - P[k - 1]) * (lim - F[k - 1]) / (F[k] - F[k - 1])
        a += 0.5 * (P[k - 1] + p_lim) * (lim - F[k - 1])
    return a / lim

# ---------------------------------------------------------------- (2) official functions
# compute_pro, trapezoid and compute_classification_roc re-typed from the MVTec AD
# evaluation code (version 1.0, 13 Oct 2022); comments and progress prints removed,
# scipy.ndimage.label imported from its current namespace. Licence (LICENSE.txt of that version):
# Copyright 2022 MVTec Software GmbH
# Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
# 1. Redistributions of source code must retain the above copyright notice, this list of
# conditions and the following disclaimer. 2. Redistributions in binary form must reproduce
# the above copyright notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution. 3. Neither the name of
# the copyright holder nor the names of its contributors may be used to endorse or promote
# products derived from this software without specific prior written permission.
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS
# OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
# COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE
# GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
# OF THE POSSIBILITY OF SUCH DAMAGE.
def compute_pro(anomaly_maps, ground_truth_maps):
    structure = np.ones((3, 3), dtype=int)
    num_ok_pixels = 0
    num_gt_regions = 0
    shape = (len(anomaly_maps), anomaly_maps[0].shape[0], anomaly_maps[0].shape[1])
    fp_changes = np.zeros(shape, dtype=np.uint32)
    assert shape[0] * shape[1] * shape[2] < np.iinfo(fp_changes.dtype).max, \
        'Potential overflow when using np.cumsum(), consider using np.uint64.'
    pro_changes = np.zeros(shape, dtype=np.float64)
    for gt_ind, gt_map in enumerate(ground_truth_maps):
        labeled, n_components = ndi.label(gt_map, structure)
        num_gt_regions += n_components
        ok_mask = labeled == 0
        num_ok_pixels_in_map = np.sum(ok_mask)
        num_ok_pixels += num_ok_pixels_in_map
        fp_change = np.zeros_like(gt_map, dtype=fp_changes.dtype)
        fp_change[ok_mask] = 1
        pro_change = np.zeros_like(gt_map, dtype=np.float64)
        for k in range(n_components):
            region_mask = labeled == (k + 1)
            region_size = np.sum(region_mask)
            pro_change[region_mask] = 1. / region_size
        fp_changes[gt_ind, :, :] = fp_change
        pro_changes[gt_ind, :, :] = pro_change
    anomaly_scores_flat = np.array(anomaly_maps).ravel()
    fp_changes_flat = fp_changes.ravel()
    pro_changes_flat = pro_changes.ravel()
    sort_idxs = np.argsort(anomaly_scores_flat).astype(np.uint32)[::-1]
    np.take(anomaly_scores_flat, sort_idxs, out=anomaly_scores_flat)
    anomaly_scores_sorted = anomaly_scores_flat
    np.take(fp_changes_flat, sort_idxs, out=fp_changes_flat)
    fp_changes_sorted = fp_changes_flat
    np.take(pro_changes_flat, sort_idxs, out=pro_changes_flat)
    pro_changes_sorted = pro_changes_flat
    del sort_idxs
    np.cumsum(fp_changes_sorted, out=fp_changes_sorted)
    fp_changes_sorted = fp_changes_sorted.astype(np.float32, copy=False)
    np.divide(fp_changes_sorted, num_ok_pixels, out=fp_changes_sorted)
    fprs = fp_changes_sorted
    np.cumsum(pro_changes_sorted, out=pro_changes_sorted)
    np.divide(pro_changes_sorted, num_gt_regions, out=pro_changes_sorted)
    pros = pro_changes_sorted
    keep_mask = np.append(np.diff(anomaly_scores_sorted) != 0, np.True_)
    del anomaly_scores_sorted
    fprs = fprs[keep_mask]
    pros = pros[keep_mask]
    del keep_mask
    np.clip(fprs, a_min=None, a_max=1., out=fprs)
    np.clip(pros, a_min=None, a_max=1., out=pros)
    zero = np.array([0.])
    one = np.array([1.])
    return np.concatenate((zero, fprs, one)), np.concatenate((zero, pros, one))

def trapezoid(x, y, x_max=None):
    x = np.asarray(x)
    y = np.asarray(y)
    finite_mask = np.logical_and(np.isfinite(x), np.isfinite(y))
    if not finite_mask.all():
        print("WARNING: Not all x and y values passed to trapezoid(...)"
              " are finite. Will continue with only the finite values.")
    x = x[finite_mask]
    y = y[finite_mask]
    correction = 0.
    if x_max is not None:
        if x_max not in x:
            ins = bisect(x, x_max)
            assert 0 < ins < len(x)
            y_interp = y[ins - 1] + ((y[ins] - y[ins - 1]) *
                                     (x_max - x[ins - 1]) /
                                     (x[ins] - x[ins - 1]))
            correction = 0.5 * (y_interp + y[ins - 1]) * (x_max - x[ins - 1])
        mask = x <= x_max
        x = x[mask]
        y = y[mask]
    return np.sum(0.5 * (y[1:] + y[:-1]) * (x[1:] - x[:-1])) + correction

def compute_classification_roc(anomaly_maps, scoring_function, ground_truth_labels):
    assert len(anomaly_maps) == len(ground_truth_labels)
    anomaly_scores = map(scoring_function, anomaly_maps)
    num_scores = len(anomaly_maps)
    sorted_samples = sorted(zip(anomaly_scores, ground_truth_labels), key=lambda x: x[0])
    ground_truth_labels_np = np.array(ground_truth_labels)
    num_nok = ground_truth_labels_np[ground_truth_labels_np != 0].size
    num_ok = ground_truth_labels_np[ground_truth_labels_np == 0].size
    fprs = [1.0]
    tprs = [1.0]
    num_fp = num_ok
    num_tp = num_nok
    next_score = None
    for i, (current_score, label) in enumerate(sorted_samples):
        if label == 0:
            num_fp -= 1
        else:
            num_tp -= 1
        if i < num_scores - 1:
            next_score = sorted_samples[i + 1][0]
        else:
            next_score = None
        if (next_score != current_score) or (next_score is None):
            fprs.append(num_fp / num_ok)
            tprs.append(num_tp / num_nok)
    return fprs[::-1], tprs[::-1]

def trapezoid_np1(x, y, x_max):
    """The official trapezoid with NumPy 1's scalar promotion emulated (written for this check):
    under NumPy 1, np.float32 scalar op Python float ran in float64; under NumPy 2 it runs in float32."""
    x, y = np.asarray(x), np.asarray(y)
    x64 = x.astype(np.float64)
    correction = 0.
    if x_max not in x:
        ins = bisect(x64, x_max)
        assert 0 < ins < len(x)
        y_interp = y[ins - 1] + ((y[ins] - y[ins - 1]) * (x_max - x64[ins - 1]) /
                                 np.float32(x[ins] - x[ins - 1]))
        correction = 0.5 * (y_interp + y[ins - 1]) * (x_max - x64[ins - 1])
    mask = x <= x_max
    x, y = x[mask], y[mask]
    return np.sum(0.5 * (y[1:] + y[:-1]) * (x[1:] - x[:-1])) + correction

# ---------------------------------------------------------------- (3) the published IVI function
def au_pro_ivi(maps, masks, fpr_max=0.3):
    """Industrial Visual Inspection's au_pro; body copied unchanged from that article."""
    neg = np.sort(maps[~masks])
    ts = neg[::-1][np.linspace(0, int(fpr_max * len(neg)) - 1, 300).astype(int)]
    fpr = 1 - np.searchsorted(neg, ts, side="right") / len(neg)
    regions = []
    for mp, mk in zip(maps, masks):
        lab, n = ndi.label(mk, np.ones((3, 3)))
        regions += [np.sort(mp[lab == k]) for k in range(1, n + 1)]
    pro = np.mean([1 - np.searchsorted(r, ts, side="right") / len(r) for r in regions], 0)
    return np.trapezoid(pro, fpr) / fpr_max

# ---------------------------------------------------------------- (4) a 100-threshold FPR grid
# Ported from anomalib lib/v2.7.0 (commit 9fb1337), tests/unit/metrics/aupro/aupro_reference.py.
# Copyright (C) 2022-2025 Intel Corporation. SPDX-License-Identifier: Apache-2.0
# (https://github.com/open-edge-platform/anomalib/blob/9fb1337c0ac866797f4eedcaa0972d0f24a84f19/LICENSE).
# Modified: compute_pro renamed compute_pro_grid; comments, docstrings, type hints, logging and the
# length assertion removed; MVTec's trapezoid above used in place of the file's own copy of it;
# au_pro_grid100 added. The LICENSE file in tests/unit/metrics/aupro/ reads:
#   Copyright (c) 2022 Intel Corporation
#   SPDX-License-Identifier: Apache-2.0
#   The file `aupro_reference.py` in this folder is based on code from Eliahu Horwitz and Yedid
#   Hoshen, which itself is based on code from MVTec Software GmbH.
#   Original license:
#     MIT License
#     Copyright (c) 2022 Eliahu Horwitz and Yedid Hoshen
#     Permission is hereby granted, free of charge, to any person obtaining a copy
#     of this software and associated documentation files (the "Software"), to deal
#     in the Software without restriction, including without limitation the rights
#     to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
#     copies of the Software, and to permit persons to whom the Software is
#     furnished to do so, subject to the following conditions:
#     The above copyright notice and this permission notice shall be included in all
#     copies or substantial portions of the Software.
#     THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
#     IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
#     FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
#     AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
#     LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
#     OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
#     SOFTWARE.
#   Original license:
#     Copyright 2021 MVTec Software GmbH
#     Redistribution and use in source and binary forms, with or without modification, are
#     permitted provided that the following conditions are met:
#     1. Redistributions of source code must retain the above copyright notice, this list of
#     conditions and the following disclaimer.
#     2. Redistributions in binary form must reproduce the above copyright notice, this list of
#     conditions and the following disclaimer in the documentation and/or other materials
#     provided with the distribution.
#     3. Neither the name of the copyright holder nor the names of its contributors may be used
#     to endorse or promote products derived from this software without specific prior written
#     permission.
#     THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS
#     OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
#     MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
#     COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
#     EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
#     SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
#     HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR
#     TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
#     SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
class GroundTruthComponent:
    def __init__(self, anomaly_scores):
        self.anomaly_scores = anomaly_scores.copy()
        self.anomaly_scores.sort()
        self.index = 0
        self.last_threshold = None
    def compute_overlap(self, threshold):
        if self.last_threshold is not None:
            assert self.last_threshold <= threshold
        while self.index < len(self.anomaly_scores) and self.anomaly_scores[self.index] <= threshold:
            self.index += 1
        return 1.0 - self.index / len(self.anomaly_scores)

def collect_anomaly_scores(anomaly_maps, ground_truth_maps):
    ground_truth_components = []
    anomaly_scores_ok_pixels = np.zeros(len(ground_truth_maps) * ground_truth_maps[0].size)
    structure = np.ones((3, 3), dtype=int)
    ok_index = 0
    for gt_map, prediction in zip(ground_truth_maps, anomaly_maps, strict=True):
        labeled, n_components = ndi.label(gt_map, structure)
        num_ok_pixels = len(prediction[labeled == 0])
        anomaly_scores_ok_pixels[ok_index: ok_index + num_ok_pixels] = prediction[labeled == 0].copy()
        ok_index += num_ok_pixels
        for k in range(n_components):
            ground_truth_components.append(GroundTruthComponent(prediction[labeled == (k + 1)]))
    anomaly_scores_ok_pixels = np.resize(anomaly_scores_ok_pixels, ok_index)
    anomaly_scores_ok_pixels.sort()
    return ground_truth_components, anomaly_scores_ok_pixels

def compute_pro_grid(anomaly_maps, ground_truth_maps, num_thresholds):
    ground_truth_components, ok = collect_anomaly_scores(anomaly_maps, ground_truth_maps)
    threshold_positions = np.linspace(0, len(ok) - 1, num=num_thresholds, dtype=int)
    fprs, pros = [1.0], [1.0]
    for pos in threshold_positions:
        threshold = ok[pos]
        fpr = 1.0 - (pos + 1) / len(ok)
        pro = 0.0
        for component in ground_truth_components:
            pro += component.compute_overlap(threshold)
        pro /= len(ground_truth_components)
        fprs.append(fpr); pros.append(pro)
    return fprs[::-1], pros[::-1]

def au_pro_grid100(maps, masks, lim=0.3):
    f, p = compute_pro_grid(list(maps), list(masks), num_thresholds=100)
    return trapezoid(f, p, x_max=lim) / lim

# ---------------------------------------------------------------- (5) CFLOW-AD's AUPRO
# The AUPRO block of gudovskiy/cflow-ad train.py (commit b2ebf9e), which credits YoungGod/DFR.
# Re-typed with the loop over thresholds vectorized: labels are computed once and counts above
# each threshold come from sorted scores; check_cflow_literal() compares with the literal loop body.
# Licence (LICENSE at commit b2ebf9e):
# BSD 3-Clause License
# Copyright (c) 2021, Panasonic AI Lab of Panasonic Corporation of North America
# All rights reserved.
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
# 1. Redistributions of source code must retain the above copyright notice, this
#    list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright notice,
#    this list of conditions and the following disclaimer in the documentation
#    and/or other materials provided with the distribution.
# 3. Neither the name of the copyright holder nor the names of its
#    contributors may be used to endorse or promote products derived from
#    this software without specific prior written permission.
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
def cflow_curve(super_mask, gt_mask, max_step=1000):
    max_th, min_th = super_mask.max(), super_mask.min()
    delta = (max_th - min_th) / max_step
    threds = np.array([max_th - step * delta for step in range(max_step)])
    regions = []                                 # (sorted scores inside filled_image, area)
    for i in range(len(gt_mask)):
        for prop in regionprops(sk_label(gt_mask[i], connectivity=2)):
            x_min, y_min, x_max, y_max = prop.bbox
            sc = super_mask[i][x_min:x_max, y_min:y_max][prop.filled_image]
            regions.append((np.sort(sc), prop.area))
    pros_mean = np.mean([(len(sc) - np.searchsorted(sc, threds, side="right")) / a
                         for sc, a in regions], axis=0)
    neg = np.sort(super_mask[~gt_mask])
    fprs = (len(neg) - np.searchsorted(neg, threds, side="right")) / (~gt_mask).sum()
    return threds, fprs, pros_mean

def rescale(x):
    return (x - x.min()) / (x.max() - x.min())

def au_pro_cflow(maps, masks, expect_fpr=0.3):
    _, fprs, pros_mean = cflow_curve(maps, masks)
    idx = fprs <= expect_fpr
    with np.errstate(invalid="ignore"):
        fprs_selected = rescale(fprs[idx])
    return auc(fprs_selected, pros_mean[idx])

def check_cflow_literal(super_mask, gt_mask, steps=(0, 500, 999)):
    threds, fprs, pros_mean = cflow_curve(super_mask, gt_mask)
    worst = 0.0
    for step in steps:
        thred = threds[step]
        binary_score_maps = np.zeros_like(super_mask, dtype=bool)
        binary_score_maps[super_mask <= thred] = 0
        binary_score_maps[super_mask > thred] = 1
        pro = []
        for i in range(len(binary_score_maps)):
            for prop in regionprops(sk_label(gt_mask[i], connectivity=2)):
                x_min, y_min, x_max, y_max = prop.bbox
                cropped_pred_label = binary_score_maps[i][x_min:x_max, y_min:y_max]
                intersection = np.logical_and(cropped_pred_label, prop.filled_image).astype(np.float32).sum()
                pro.append(intersection / prop.area)
        gt_masks_neg = ~gt_mask
        fpr = np.logical_and(gt_masks_neg, binary_score_maps).sum() / gt_masks_neg.sum()
        worst = max(worst, abs(np.array(pro).mean() - pros_mean[step]), abs(fpr - fprs[step]))
    return worst

# ---------------------------------------------------------------- run
rng = np.random.default_rng(0)
masks, placed = make_masks(rng)
maps = make_maps(rng, masks)

scr = [p for p in placed if p[1] == "scratch"]
n8 = sum(ndi.label(m, EIGHT)[1] for m in masks)
n4 = sum(ndi.label(m, FOUR)[1] for m in masks)
holes = sum(int((ndi.binary_fill_holes(lab == k) != (lab == k)).any())
            for m in masks for lab, n in [ndi.label(m, EIGHT)] for k in range(1, n + 1))
print(f"data: {len(masks)} maps of {S}x{S}, {N_GOOD} good; defects placed {len(placed)} "
      f"(scratches {len(scr)}, spots {sum(p[1] == 'spot' for p in placed)}, large {N_LARGE})")
print(f"      8-connected regions {n8} (placed {len(placed)}), 4-connected regions {n4} "
      f"(expected {sum(p[2] for p in scr) + len(placed) - len(scr)}), regions with holes {holes}, "
      f"defect pixels in good images {int(masks[:N_GOOD].sum())}, anomalous pixels {masks.mean():.4%}")

ivi_bound = 2 / (0.3 * (~masks).sum())
rows = {}
print(f"\n{'map set':10s} {'exact':>9s} {'official':>9s} {'IVI-300':>9s} {'grid-100':>9s} "
      f"{'CFLOW':>9s} {'exact-4c':>9s} | {'off-exact':>9s} {'np1-np2':>9s}")
for name in ("oracle_bin", "oracle_sep", "random", "A", "B"):
    m = maps[name]
    ex = au_pro_exact(m, masks)
    f, p = compute_pro(list(m), list(masks))
    off = trapezoid(f, p, x_max=0.3) / 0.3
    off1 = trapezoid_np1(f, p, x_max=0.3) / 0.3
    r = dict(exact=ex, official=off, ivi=au_pro_ivi(m, masks), grid100=au_pro_grid100(m, masks),
             cflow=au_pro_cflow(m, masks), four=au_pro_exact(m, masks, structure=FOUR))
    rows[name] = r
    print(f"{name:10s} {r['exact']:9.6f} {r['official']:9.6f} {r['ivi']:9.6f} {r['grid100']:9.6f} "
          f"{r['cflow']:9.6f} {r['four']:9.6f} | {abs(off - ex):9.1e} {abs(off1 - off):9.1e}")

print("\ndifference from exact      IVI-300     grid-100    CFLOW       exact-4c")
for name in ("A", "B"):
    r = rows[name]
    print(f"{name:26s} " + " ".join(f"{r[k] - r['exact']:+.2e}" for k in ("ivi", "grid100", "cflow", "four")))
print(f"oracle_sep: 1 - IVI-300 = {1 - rows['oracle_sep']['ivi']:.2e} (bound 2/(0.3 n_ok) = {ivi_bound:.2e})")
print("oracle_sep: 1 - v  " + "  ".join(f"{k}: {1 - rows['oracle_sep'][k]:.1e}"
                                         for k in ("exact", "official", "ivi", "grid100", "cflow", "four")))
print("A above B?  " + "  ".join(f"{k}: {rows['A'][k] > rows['B'][k]}"
                                 for k in ("exact", "ivi", "grid100", "cflow", "four")))
print(f"CFLOW vectorized vs literal loop (A, steps 0/500/999): max diff {check_cflow_literal(maps['A'], masks):.1e}")

y = [0] * N_GOOD + [1] * (len(masks) - N_GOOD)
for name in ("A", "B"):
    f, t = compute_classification_roc(list(maps[name]), np.max, y)
    a_off = trapezoid(f, t)
    a_sk = roc_auc_score(y, maps[name].reshape(len(masks), -1).max(1))
    print(f"image AUROC {name}: official {a_off:.6f}  scikit-learn {a_sk:.6f}  |diff| {abs(a_off - a_sk):.1e}")

Output (NumPy 2.3.5, SciPy 1.16.3, scikit-image 0.25.2, scikit-learn 1.7.2; about 15 s; two runs gave identical output):

data: 100 maps of 256x256, 40 good; defects placed 142 (scratches 62, spots 79, large 1)
      8-connected regions 142 (placed 142), 4-connected regions 2575 (expected 2575), regions with holes 0, defect pixels in good images 0, anomalous pixels 0.1631%

map set        exact  official   IVI-300  grid-100     CFLOW  exact-4c | off-exact   np1-np2
oracle_bin  1.000000  1.000000  0.000000  1.000000       nan  1.000000 |   5.0e-13   0.0e+00
oracle_sep  1.000000  1.000000  0.999999  1.000000  1.000000  1.000000 |   5.2e-14   0.0e+00
random      0.149539  0.149539  0.149524  0.149630  0.149449  0.149730 |   5.6e-11   0.0e+00
A           0.620673  0.620673  0.620578  0.618068  0.619122  0.402302 |   7.2e-11   0.0e+00
B           0.572487  0.572487  0.572383  0.570172  0.569433  0.320405 |   6.2e-11   0.0e+00

difference from exact      IVI-300     grid-100    CFLOW       exact-4c
A                          -9.42e-05 -2.61e-03 -1.55e-03 -2.18e-01
B                          -1.04e-04 -2.31e-03 -3.05e-03 -2.52e-01
oracle_sep: 1 - IVI-300 = 6.11e-07 (bound 2/(0.3 n_ok) = 1.02e-06)
oracle_sep: 1 - v  exact: -5.2e-14  official: 0.0e+00  ivi: 6.1e-07  grid100: 0.0e+00  cflow: 0.0e+00  four: -6.6e-14
A above B?  exact: True  ivi: True  grid100: True  cflow: True  four: True
CFLOW vectorized vs literal loop (A, steps 0/500/999): max diff 2.2e-16
image AUROC A: official 0.673750  scikit-learn 0.673750  |diff| 1.1e-16
image AUROC B: official 0.655000  scikit-learn 0.655000  |diff| 1.1e-16

The line printing $1 - v$ on oracle_sep was added after a pilot run, with the editor’s approval, because six decimals sit at the digit of the $10^{-6}$ tolerance; the pilot printed the same values to six decimals, and nothing else changed between the pilot and this run.

What the output shows:

• The port reproduces the official computation, and the data are as built. The region counts match the construction under both connectivities. The re-typed official functions agree with our independent 64-bit version to within $7.2 \times 10^{-11}$, a difference that, in our reading, comes from the official code holding the FPR in 32-bit floats. Emulating NumPy 1’s scalar promotion changes nothing on these data, since it affects only the interpolated sliver at the limit. The official image-level ROC area equals scikit-learn’s. The random maps give 0.1495, within the planned 0.01 of the expected 0.15. Every planned check held.
• Base cases separate the computations. On perfectly separated maps all six give 1, IVI-300 short by $6.1 \times 10^{-7}$ because its last sample stops one or two good pixels’ worth of FPR below 0.3. On the binary oracle, where all good pixels tie at 0, the hypotheses held: the exact, official, grid-100 and 4-connected computations give 1, IVI-300 gives 0 and CFLOW gives nan. A binary map is an unusual input, but post-processed or quantized maps can tie many good pixels, and the official code handles ties by merging them.
• The grids are slightly low. For both models, all three grids fall below the exact value, with IVI-300 within $10^{-3}$ and grid-100 further off than IVI-300, so the three grid hypotheses held for A and for B: IVI-300 is low by about $1 \times 10^{-4}$, grid-100 by $2.3$–$2.6 \times 10^{-3}$ and CFLOW by $1.6$–$3.1 \times 10^{-3}$. These are third- and fourth-decimal effects, of the size of some differences reported on the benchmark, such as PatchCore’s 93.4% and 93.1% PRO with two memory-bank sizes, which Visual Anomaly Detection reports. No grid reverses the order of A and B, so the ranking hypothesis held. The output also shows two things the hypotheses did not address: on the random maps grid-100 lies above the exact value, and CFLOW errs less than grid-100 for A but more for B, so the size of a grid’s error depends on the map.
• Connectivity is not a detail. With 4-connected regions, AU-PRO falls from 0.621 to 0.402 for A and from 0.572 to 0.320 for B. The 62 scratches become 2,495 single-pixel regions, so the average is dominated by the defect the models find worst. On this test set the choice of structuring element moves the score more than 70 times as much as any grid, so the connectivity hypothesis held for A and for B.

The example tests the computations on synthetic maps; it says nothing about how often each implementation was used for published numbers. Its test set is also more lopsided than the real one: 0.16% of its pixels are anomalous, against the 2.7% Industrial Visual Inspection reports for MVTec AD, and scratches make up almost half its regions.

License

As of November 2026, the dataset page releases the data under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC BY-NC-SA 4.0) and adds that “it is not allowed to use the dataset for commercial purposes”, asking users who are unsure whether their application is commercial to contact MVTec. The data are obtained by filling in a form on that page; the form and any terms attached to the download were not read here. The evaluation code carries its own licence, the BSD 3-Clause License (copyright 2022 MVTec Software GmbH), which allows redistribution and modification with the copyright notice kept. The MVTec AD 2 page states the same CC BY-NC-SA 4.0 licence for that dataset (as of November 2026).

Limitations and Bias

• Saturation at image level. Published image AUROCs on MVTec AD are close to 100%, so differences between leading methods are fractions of a point; Visual Anomaly Detection gives the figures and reports that Zou et al. introduced VisA partly because they judged MVTec AD to be saturating.
• Made defects. The defects were produced by hand to resemble real ones (Section 3). How well their kinds and frequencies match those of a real line is unknown, and each category has one to eight defect types, all present in the test set.
• Controlled imaging. Highly controlled illumination and telecentric optics make the images cleaner and more uniform than many production settings; the 2026 successor adds test images under lighting conditions not necessarily seen in training (MVTec AD 2 page).
• Small test sets. Categories have 30 to 141 defective test images and 12 to 60 good ones (Table 1), so per-category AUROCs move in steps of 1/(good × defective images), and the category mean weights a category of 42 test images as much as one of 167.
• Protocol drift. Reported numbers mix the official AU-PRO with library and paper reimplementations, pixel AUROC with and without an FPR limit, and different resolutions and crops. The example shows that the AU-PRO computations alone differ in the third decimal, and the region connectivity far more.
• Region definition. Regions are connected components of the masks, so two touching defects count as one region and a defect broken by the annotation counts as several; the metric weights them accordingly.

Common Uses

• Benchmark for unsupervised anomaly detection and localization, the use for which most methods in Visual Anomaly Detection report results: reconstruction models such as autoencoders, memory banks of pretrained features, Gaussian models, normalizing flows and student–teacher networks.
• Limited-data studies, which train on a subset of the defect-free images of each category, such as the PatchCore and DifferNet experiments that Visual Anomaly Detection describes.
• Teaching and prototyping inspection systems, within the non-commercial licence; Industrial Visual Inspection uses its statistics to discuss deployment.

Historical Milestones

• 2019, CVPR. The dataset and an initial benchmark of AnoGAN, $\ell^2$ and SSIM convolutional autoencoders, a CNN feature dictionary on ResNet-18 features, a GMM-based texture inspection model and the variation model (Section 4.1). None performed well across all categories. A high pixel ROC AUC did not necessarily come with a high per-region overlap at the estimated threshold: the SSIM autoencoder reached a ROC AUC of 0.97 on hazelnut with an overlap of 0.00, and the CNN feature dictionary 0.93 on tile with an overlap of 0.14 (Table 3), which the authors attribute to poorly estimated thresholds (Section 4.4).
• 2021, IJCV. The extended journal version (Bergmann et al., 2021, not read here), the version on which Visual Anomaly Detection bases its account of the area under the PRO curve.
• 2022, evaluation code 1.0. The version of the scripts described above has files dated 13 October 2022, fixing AU-PRO up to an FPR of 0.3 and image-level ROC AUC as the measures they compute.
• 2020–2022, methods near the ceiling. Methods on pretrained features pushed image AUROC towards 100%; CFLOW-AD with a WideResNet-50 encoder, for example, reported 98.26% image AUROC, 98.62% pixel AUROC and 94.60% AUPRO (Gudovskiy et al., 2022, Table 4), and Visual Anomaly Detection follows the progression from SPADE and PaDiM to PatchCore.
• 2026, MVTec AD 2. Eight new scenarios with more than 8,000 images, with defect-free training and validation sets and a test set in two parts: annotations are public for the first, while the second can be scored only through MVTec’s evaluation server (MVTec AD 2 page, as of November 2026; Heckler-Kram et al., 2026, not read here).

Related Datasets

• MVTec AD 2 (2026): the successor described above, with harder scenarios, lighting changes and a private test set.
• MVTec LOCO AD: logical as well as structural anomalies, such as objects missing or in the wrong place; Visual Anomaly Detection summarizes it.
• MVTec 3D-AD: listed among MVTec’s datasets; its evaluation code is the stated origin of the anomalib reference used above.
• VisA: 10,821 images of 12 objects, including printed circuit boards and multi-instance scenes, with a one-class protocol that follows MVTec AD’s; see Visual Anomaly Detection and Industrial Visual Inspection.
• DAGM 2007 and NanoTWICE: the earlier texture sets that the 2019 paper contrasts with MVTec AD (Section 2.1.2).

Resources

• MVTec AD dataset page: description, download form, evaluation code and licence.
• MVTec AD 2 dataset page: the successor dataset, its code utilities and the link to its evaluation server.
• anomalib: a third-party library with an AUPRO metric in PyTorch.
• Visual Anomaly Detection for the methods and the metric definitions, and Industrial Visual Inspection for the application.