ROC Curve and AUC

The area under the ROC curve (AUC, or AUROC) scores a classifier that outputs a number $s(x)$ for each image, higher meaning “more likely positive”, without fixing a decision threshold. The receiver operating characteristic (ROC) curve, defined in Precision and Recall, plots the true positive rate (TPR) against the false positive rate (FPR) as the threshold sweeps over all scores; its area equals the probability that a randomly chosen positive image scores above a randomly chosen negative one. AUC therefore measures ranking, not decisions: it is unchanged by any increasing transformation of the scores, ignores whether they are calibrated probabilities, and says nothing about which threshold to use. It is a standard summary in medical image classification, out-of-distribution and anomaly detection, deepfake detection and face verification, where it is often reported next to a TPR or FPR at a fixed operating point.

Formula

Let a test set contain $P$ positive and $N$ negative images, with scores $s^+_1,\dots,s^+_P$ and $s^-_1,\dots,s^-_N$. Calling an image positive when $s \ge t$ gives a point $(\mathrm{FPR}(t), \mathrm{TPR}(t))$, and the ROC curve joins these points from $(0, 0)$ at $t = +\infty$ to $(1, 1)$ at $t = -\infty$. The area is

$$\mathrm{AUC} = \int_0^1 \mathrm{TPR} \; d\,\mathrm{FPR}.$$

Probabilistic form. With $X^+$ a random positive and $X^-$ a random negative,

$$\mathrm{AUC} = \Pr\big(s(X^+) > s(X^-)\big) + \tfrac{1}{2}\Pr\big(s(X^+) = s(X^-)\big),$$

and on a finite test set this is the fraction of the $PN$ positive–negative pairs that the scores order correctly, with ties counted as one half:

$$\widehat{\mathrm{AUC}} = \frac{1}{PN}\sum_{i=1}^{P}\sum_{j=1}^{N} \Big( \mathbb{1}\big[s^+_i > s^-_j\big] + \tfrac{1}{2}\,\mathbb{1}\big[s^+_i = s^-_j\big] \Big).$$

Hanley and McNeil (1982) showed that the area under an empirical ROC curve measures this probability of a correct ranking, and that it is the quantity estimated by the Wilcoxon (Mann–Whitney) rank statistic: their score for a pair is 1, ½ or 0, and the trapezoidal area of the curve equals the statistic. They credit the link to the Wilcoxon statistic to Bamber (1975). Computed from ranks, with $R^+$ the sum of the ranks of the positives among all $P + N$ scores (tied scores sharing the mean of their ranks),

$$\widehat{\mathrm{AUC}} = \frac{U}{PN}, \qquad U = R^+ - \frac{P(P+1)}{2},$$

where $U$ is the Mann–Whitney statistic for the positives. Sorting makes this $O\big((P+N)\log(P+N)\big)$ instead of $O(PN)$ for the double sum.

Computing the curve, and ties. Fawcett (2006, Algorithm 1; Algorithm 2 computes the area the same way) builds the curve by sorting images by decreasing score and walking down the list, adding one to the true-positive count for each positive and to the false-positive count for each negative, and emits a point only after the last of a run of equal scores. A run of $n$ positives and $m$ negatives with the same score then becomes one diagonal segment, and the trapezoidal rule gives it area $nm/(2PN)$ beyond the step below it: the half credit of the pair formula. If equal scores were instead processed one at a time, the curve would depend on their arbitrary order, anywhere between an optimistic step (positives first) and a pessimistic one (negatives first); Fawcett notes that for one group the error can reach $nm/(2PN)$. The diagonal is the expected curve under a random order. Ties are common when scores come from a rating scale (Hanley and McNeil’s readers rated CT images in five categories), from coarse score outputs, or from scores rounded before evaluation.

Intuition

The ROC curve compares two score distributions, that of the positive images and that of the negative ones, through every threshold at once; AUC asks only how often a positive outranks a negative. A detector of a bright disc in noisy images, for example, has an AUC of 0.8 if it gives the higher score to the image containing the disc in four out of five random (disc, no disc) pairs.

Only the order matters. Replacing $s$ by $g(s)$ for any strictly increasing $g$ leaves every pairwise comparison, and so the ROC curve and the AUC, unchanged. It follows that AUC does not care whether scores are probabilities, logits or raw filter responses, or whether a score of 0.9 means a 90% chance of being positive. Fawcett (2006, §4.1) makes the point that ROC curves measure the ability to produce good relative scores, and that a classifier need not produce calibrated probability estimates to have a good curve. A model can have a perfect AUC and poor accuracy at a threshold of 0.5 when its scores are not calibrated; how well scores match empirical frequencies is measured separately, for example by the expected calibration error. Scores from different models are not comparable at a common threshold for the same reason.

Choosing an operating point. Each point on the curve is a classifier with a fixed threshold. If a false positive costs $c_{\mathrm{FP}}$, a false negative $c_{\mathrm{FN}}$, and positives have prevalence $\pi$, all points on a line of slope

$$m = \frac{c_{\mathrm{FP}}\,(1 - \pi)}{c_{\mathrm{FN}}\,\pi}$$

in ROC space have the same expected cost, and the best operating point is where the highest such line touches the curve (Fawcett, 2006, §6, summarizing Provost and Fawcett). Only points on the ROC convex hull, the upper boundary of the convex hull of all points, can be optimal for some costs and prevalence; points below it never are (Fawcett, 2006, §6), and a point on a hull segment between two classifiers can be reached by choosing randomly between them in the right proportion (§10). With ten negatives per positive and equal costs, $m = 10$ and the optimum lies in the steep, low-FPR part of the curve; with balanced classes and a false negative ten times as costly, $m = 1/10$ and it moves to the high-TPR end. AUC averages over all these operating points, including many no application would use.

Interpretation

• Range. $0 \le \mathrm{AUC} \le 1$. A score that ignores the image, or random guessing, gives 0.5 in expectation (the diagonal); 1 means every positive outranks every negative. An AUC below 0.5 means the scores are informative but inverted: negating them gives $1 - \mathrm{AUC}$. The Gini coefficient, twice the area between the curve and the diagonal, is $2\,\mathrm{AUC} - 1$ (Fawcett, 2006, citing Hand and Till, 2001).
• No universal scale. Whether 0.8 is good depends on the task: in a hard task it may be near what the image data allow, while 0.99 in an industrial inspection task may still let through too many defects at the required false-alarm rate. Hendrycks and Gimpel (2017) give a verbal scale (90–100% excellent down to 50–60% fail) and call it debatable and imprecise.
• Prevalence. AUC is built from TPR and FPR, so it does not change when the ratio of positives to negatives changes; Precision and Recall explains what this hides when positives are rare, and when the PR curve is the better display.
• Uncertainty. An AUC from a finite test set is an estimate. Hanley and McNeil (1982) used the properties of the Wilcoxon statistic to give a closed-form standard error; for one reader’s ratings of 109 CT images (51 abnormal, 58 normal) they obtained $\widehat{\mathrm{AUC}} = 0.893$ with a standard error of 0.032. DeLong, DeLong and Clarke-Pearson (1988) gave a nonparametric method for comparing the AUCs of two or more classifiers scored on the same images, whose estimates are correlated. Confidence intervals and tests for comparing models are the subject of statistical significance in evaluation; differences in the third decimal between models on a test set of a few hundred images are usually within the noise.

When to Use It

Use AUC to compare scoring classifiers when the operating threshold is not yet fixed, or will be set later for different users, and when positives and negatives are both meaningful classes with a finite set of negatives. For an object detector, where true negatives cannot be counted, the precision–recall curve and average precision are used instead; PASCAL VOC replaced ROC area with AP for classification in 2007.

Vision uses AUC in several fields:

• Medical image classification. Wang et al. (2017) reported per-pathology ROC AUCs for multi-label classification of chest X-rays in ChestX-ray8, each pathology label scored as its own binary problem; with a ResNet-50 they ranged from 0.5609 for mass to 0.8141 for cardiomegaly (Table 3, which the text refers to as Table 4). The transfer learning article cites AUCs for diabetic retinopathy detection. Hanley and McNeil (1982) wrote for radiologists, among whom ROC curves had become popular for measuring the information content of imaging systems.
• Out-of-distribution detection. Hendrycks and Gimpel (2017) evaluated detectors of misclassified and out-of-distribution inputs, using the maximum softmax probability as the score, with AUROC and the area under the PR curve, reporting the PR area with each class in turn treated as positive. Liang, Li and Srikant (2018) also reported FPR at 95% TPR, the fraction of out-of-distribution images accepted when 95% of in-distribution images are (in-distribution counted as positive); for a DenseNet-BC trained on CIFAR-10 with cropped Tiny ImageNet images as the out-of-distribution set, their method lowered it from 34.7% for the maximum-softmax baseline to 4.3% while AUROC rose from 95.3% to 99.1% (Table 2), a gap in AUC that understates the change at the operating point.
• Anomaly detection. Bergmann et al. (2019) score anomaly segmentation on MVTec AD by pixel-level ROC AUC, with TPR the fraction of anomalous pixels flagged and FPR the fraction of normal pixels flagged, as a threshold-independent measure next to threshold-dependent ones. The autoencoder article uses AUROC of reconstruction error to score anomalies.
• Deepfake detection. Li et al. (2020) compare detectors by frame-level AUC on Celeb-DF and earlier deepfake datasets, noting that it avoids differences in how frame scores are aggregated per video and the need to calibrate each method’s outputs across datasets; they round scores to five decimals before evaluation, for robustness to numerical imprecision, which can create ties. Averaged over the compared methods, AUC was lowest on Celeb-DF, at 56.9%, of the seven datasets in their Figure 7.
• Face verification. A verification system compares two face images and accepts the pair when their similarity exceeds a threshold. The IARPA Janus Benchmark A defines the true accept rate (TAR) as the fraction of genuine (same-person) comparisons above the threshold and the false accept rate (FAR) as the fraction of impostor comparisons above it, and asks for TAR at FAR of 0.1, 0.01 and 0.001 together with the ROC plot of TAR against FAR (Klare et al., 2015, Figure 6; the text names only 0.01 and 0.001); TAR and FAR are TPR and FPR under other names.

Failure Cases

• It hides the operating point. Two classifiers with the same AUC can have very different TPR at the FPR an application can tolerate. Bergmann et al. (2019) found that a high ROC AUC on MVTec AD did not necessarily coincide with a good segmentation at the threshold their protocol estimated: the anomaly maps ranked pixels well, but the threshold chosen from anomaly-free images was poor. Report the operating point that matters alongside AUC.
• Crossing curves. AUC can rank a classifier higher even though it is worse over the region that matters: in Fawcett’s (2006, Fig. 8a) example, classifier B has the larger area but A is better at FPR above 0.6. When curves cross, compare them over the relevant range with a partial AUC or at a fixed operating point.
• Rare positives. A high AUC is compatible with poor precision when negatives vastly outnumber positives, as in screening (Precision and Recall explains why). In the example, the operating point with TPR 0.95 has precision 0.27 at the test set’s prevalence of 20% and 0.015 at 1%, with the AUC unchanged.
• Calibration is invisible. The example’s three sigmoid transforms of one score share an AUC but give accuracies from 0.61 to 0.85 at a threshold of 0.5.
• Ties handled inconsistently. Breaking ties in input order instead of drawing diagonals makes the AUC depend on that order; with ten distinct scores in the example, the optimistic and pessimistic orders differ by 0.2.
• Pooling across groups. An AUC pooled over subgroups, scanners or acquisition sites with different score distributions can differ from every per-group AUC, because it also rewards ranking whole groups apart. This follows from the pairwise definition.

Variants

Partial AUC. McClish (1989) proposed analyzing only the part of the curve over a range of FPR, such as $[0, a]$ for applications that tolerate few false positives. The raw partial area runs from $a^2/2$ for the diagonal to $a$ for a perfect classifier, so its values are hard to read; scikit-learn’s roc_auc_score(..., max_fpr=a) returns the standardized partial AUC it attributes to McClish,

$$\mathrm{pAUC}_{\mathrm{std}} = \frac{1}{2}\left(1 + \frac{\mathrm{pAUC} - a^2/2}{a - a^2/2}\right),$$

which is 0.5 for a non-discriminating classifier and 1 for a perfect one (scikit-learn 1.7 source and documentation).

TPR at fixed FPR, and FPR at fixed TPR. A single point of the curve: TAR at FAR $= 10^{-3}$ in face verification, FPR at 95% TPR in out-of-distribution detection. These are what a deployed system experiences, but they depend on few images when the fixed rate is small: TPR at FPR $= 10^{-3}$ is set by the highest-scoring 0.1% of negatives.

DET curve and equal error rate. Martin et al. (1997) introduced the detection error tradeoff (DET) curve, with examples from speaker and language recognition: it plots the miss rate ($1 - \mathrm{TPR}$) against the false alarm rate (FPR), both on a normal deviate scale. If both score distributions are Gaussian the curve is a straight line, with unit slope when their variances are equal, and the scale spreads out the low-error corner where good systems bunch together on a ROC plot. The equal error rate (EER) is the error rate at the point where the miss rate equals the false alarm rate, a one-number summary of the curve.

Multi-class AUC. For $K$ classes, two reductions are used. One-versus-rest computes, for each class $k$, the AUC of the score for $k$ with class $k$ positive and all other classes negative, then averages over classes; Fawcett (2006, §9.2) notes that this loses the insensitivity to class distribution, because the negative set mixes the other classes in their current proportions, and describes Provost and Domingos’s prevalence-weighted average. One-versus-one, Hand and Till’s (2001) measure $M$, averages over all pairs of classes the AUC of separating that pair, using only the images of those two classes:

$$M = \frac{2}{K(K-1)}\sum_{j<k} \frac{A(j \mid k) + A(k \mid j)}{2},$$

where $A(j \mid k)$ is the AUC of the score for class $j$ with class $j$ positive and class $k$ negative; in general $A(j \mid k) \ne A(k \mid j)$. scikit-learn 1.7.2’s roc_auc_score requires multi_class='ovr' or 'ovo' for multi-class labels (the default 'raise' is an error), takes class probabilities that sum to 1, and supports average='macro' (unweighted) and 'weighted' for both (by class prevalence for one-versus-rest, by the fraction of images in each pair for one-versus-one), plus 'micro' and per-class output (average=None) for one-versus-rest only; partial AUC is not supported for multi-class (scikit-learn 1.7 documentation). Multi-label tasks such as ChestX-ray8 are scored per label, as independent binary problems.

Example

The example scores a template-matching detector for “bright disc present” on synthetic images: 1,000 floating-point images of 32×32 pixels with zero-mean Gaussian noise of standard deviation 1, of which 200 contain a disc of radius 3 that adds 0.6 to the intensity. The score is the largest response of a disc-shaped averaging filter, a matched filter. The code computes AUC by the trapezoidal rule over the ROC curve, from ranks (Mann–Whitney), and from all positive–negative pairs, with continuous and with tied scores, and cross-checks scikit-learn 1.7.2’s roc_auc_score (NumPy 2.3.5, SciPy 1.16.3).

import numpy as np
from scipy import ndimage, stats
from sklearn.metrics import roc_auc_score
import sklearn


def make_images(n_pos, n_neg, rng, size=32, radius=3, contrast=0.6, sigma=1.0):
    """Noisy 32x32 images; positives contain a bright disc at a random place."""
    imgs = rng.normal(0.0, sigma, (n_pos + n_neg, size, size))
    yy, xx = np.mgrid[:size, :size]
    for k in range(n_pos):
        cy, cx = rng.integers(radius, size - radius, 2)
        imgs[k][(yy - cy) ** 2 + (xx - cx) ** 2 <= radius ** 2] += contrast
    labels = np.r_[np.ones(n_pos, bool), np.zeros(n_neg, bool)]
    return imgs, labels


def template_score(imgs, radius=3):
    """Matched filter: mean over a disc template, maximized over positions."""
    r = np.arange(-radius, radius + 1)
    disc = (r[:, None] ** 2 + r[None, :] ** 2 <= radius ** 2).astype(float)
    disc /= disc.sum()
    return np.array([ndimage.correlate(im, disc, mode="constant").max() for im in imgs])


def roc_points(s, y):
    """Fawcett (2006), Algorithm 1: one ROC point per distinct score."""
    order = np.argsort(-s, kind="stable")
    s, y = s[order], y[order]
    tp, fp = np.cumsum(y), np.cumsum(~y)
    last = np.r_[s[1:] != s[:-1], True]          # last index of each run of equal scores
    tpr = np.r_[0, tp[last]] / y.sum()
    fpr = np.r_[0, fp[last]] / (~y).sum()
    return fpr, tpr


def auc_trapezoid(s, y):
    fpr, tpr = roc_points(s, y)
    return np.sum(np.diff(fpr) * (tpr[1:] + tpr[:-1]) / 2)


def auc_rank(s, y):
    """Mann-Whitney U / (P N), with average ranks for ties."""
    r = stats.rankdata(s)                         # ties get the mean of their ranks
    P, N = y.sum(), (~y).sum()
    return (r[y].sum() - P * (P + 1) / 2) / (P * N)


def auc_pairs(s, y):
    """Fraction of (positive, negative) pairs ranked correctly; ties count 1/2."""
    d = s[y][:, None] - s[~y][None, :]
    return np.mean((d > 0) + 0.5 * (d == 0))


rng = np.random.default_rng(0)
imgs, y = make_images(200, 800, rng)              # 200 images with a disc, 800 without
s = template_score(imgs)

print(f"scikit-learn {sklearn.__version__}")
print("continuous scores:  trapezoid {:.4f}  rank {:.4f}  pairs {:.4f}  sklearn {:.4f}".format(
    auc_trapezoid(s, y), auc_rank(s, y), auc_pairs(s, y), roc_auc_score(y, s)))

# Ties: round the scores to one decimal, as a coarse score output would.
q = np.round(s, 1)
print(f"rounded scores: {len(np.unique(q))} distinct values for {len(q)} images")
print("rounded scores:     trapezoid {:.4f}  rank {:.4f}  pairs {:.4f}  sklearn {:.4f}".format(
    auc_trapezoid(q, y), auc_rank(q, y), auc_pairs(q, y), roc_auc_score(y, q)))
# Breaking ties by putting positives first or last instead of averaging them:
eps = 1e-9
print("ties broken: optimistic {:.4f}  pessimistic {:.4f}".format(
    auc_pairs(q + eps * y, y), auc_pairs(q - eps * y, y)))

# Monotone transforms of the score leave the ranking, and AUC, unchanged,
# but move every image's "probability" and so the decision at 0.5.
for c in [0.55, 0.65, 0.75]:
    p = 1 / (1 + np.exp(-20 * (s - c)))          # a sigmoid centred at score c
    pred = p >= 0.5
    print(f"sigmoid(20(s - {c})): AUC {roc_auc_score(y, p):.4f}  at p >= 0.5: "
          f"TPR {pred[y].mean():.3f}  FPR {pred[~y].mean():.3f}  "
          f"accuracy {np.mean(pred == y):.3f}")

# Spread of AUC over independently generated test sets.
aucs = [auc_rank(template_score(im), lab)
        for im, lab in (make_images(200, 800, np.random.default_rng(k)) for k in range(10))]
print(f"AUC over seeds 0-9: {min(aucs):.4f} to {max(aucs):.4f}")

Output:

scikit-learn 1.7.2
continuous scores:  trapezoid 0.8156  rank 0.8156  pairs 0.8156  sklearn 0.8156
rounded scores: 10 distinct values for 1000 images
rounded scores:     trapezoid 0.7967  rank 0.7967  pairs 0.7967  sklearn 0.7967
ties broken: optimistic 0.8951  pessimistic 0.6984
sigmoid(20(s - 0.55)): AUC 0.8156  at p >= 0.5: TPR 0.865  FPR 0.458  accuracy 0.607
sigmoid(20(s - 0.65)): AUC 0.8156  at p >= 0.5: TPR 0.525  FPR 0.111  accuracy 0.816
sigmoid(20(s - 0.75)): AUC 0.8156  at p >= 0.5: TPR 0.310  FPR 0.015  accuracy 0.850
AUC over seeds 0-9: 0.7827 to 0.8510

The three computations agree, with and without ties, because they are the same quantity: diagonal segments over tied runs give the trapezoid the pair count’s half credit, and average ranks give it to the rank formula. Rounding to one decimal leaves ten distinct scores, so many pairs are tied: the gap between the optimistic and pessimistic tie orders is exactly the fraction of tied pairs, about a fifth. The AUC drops because rounding discards ranking information, and the optimistic and pessimistic tie orders bracket it, with the true value exactly at their midpoint. The three sigmoids map the same score to very different “probabilities”: the AUC is identical, while the decision at 0.5 moves along the curve from a high-recall, high-FPR point to a cautious one. Accuracy rises at the cautious end only because 80% of the images are negative. Over ten independently drawn test sets of this size the AUC of the same detector ranges from 0.78 to 0.85, a reminder of how much a single test set’s AUC can move.

The second part, continuing from the first, reads operating-point summaries off the same curve and computes the standardized partial AUC over FPR in $[0, 0.1]$.

# Continues from the first block: uses s, y, roc_points and roc_auc_score.
fpr, tpr = roc_points(s, y)

# Operating-point summaries read off the empirical ROC curve.
tpr_at_1 = tpr[fpr <= 0.01].max()                 # best TPR with FPR <= 1%
j = np.flatnonzero(tpr >= 0.95)[0]                # first point with TPR >= 95%
fpr_at_95 = fpr[j]
i = np.argmin(np.abs(fpr - (1 - tpr)))            # where FPR and FNR = 1 - TPR meet
print(f"TPR at FPR <= 1%: {tpr_at_1:.3f}   FPR at TPR >= 95%: {fpr_at_95:.3f}   "
      f"EER ~ {(fpr[i] + 1 - tpr[i]) / 2:.3f}")

# Partial AUC over FPR in [0, 0.1], with linear interpolation at the cut,
# then McClish's standardization to [0.5, 1] (as scikit-learn implements it).
a = 0.1
f = np.r_[fpr[fpr < a], a]
t = np.r_[tpr[fpr < a], np.interp(a, fpr, tpr)]
pauc = np.sum(np.diff(f) * (t[1:] + t[:-1]) / 2)
std = 0.5 * (1 + (pauc - a**2 / 2) / (a - a**2 / 2))
print(f"partial AUC [0, {a}]: raw {pauc:.4f} (max {a})   standardized {std:.4f}   "
      f"sklearn max_fpr={a}: {roc_auc_score(y, s, max_fpr=a):.4f}")

# That operating point judged by precision, here (prevalence 0.2) and at 0.01.
for prev in [0.2, 0.01]:
    prec = tpr[j] * prev / (tpr[j] * prev + fpr[j] * (1 - prev))
    print(f"prevalence {prev:4.2f}: precision at TPR {tpr[j]:.3f} = {prec:.3f}")

Output:

TPR at FPR <= 1%: 0.290   FPR at TPR >= 95%: 0.639   EER ~ 0.274
partial AUC [0, 0.1]: raw 0.0399 (max 0.1)   standardized 0.6836   sklearn max_fpr=0.1: 0.6836
prevalence 0.20: precision at TPR 0.950 = 0.271
prevalence 0.01: precision at TPR 0.950 = 0.015

An AUC of 0.82 sounds respectable, but the operating points tell a harsher story: to find 95% of the discs the detector must flag nearly two-thirds of the empty images, and at a 1% false-alarm budget it finds fewer than a third of the discs. The standardized partial AUC, 0.68, is much closer to chance than the full AUC because the low-FPR region is where this detector is weakest; the hand-written McClish standardization matches scikit-learn. Precision at the high-recall point is poor even at the test set’s own prevalence and collapses at 1%, while the AUC stays where it was.

The third part checks the multi-class options on three classes of synthetic images, with the classes deliberately unbalanced: 600 noise images, 300 with a disc and 100 with a 5×5 square, both of contrast 0.6. Each image gets three scores (a constant for “empty”, the disc matched filter, and a 5×5 box filter’s maximum), turned into probabilities with a softmax.

# Continues from the first block: uses ndimage, template_score, auc_rank and roc_auc_score.
from itertools import combinations

# Three classes: 600 empty noise images, 300 with a disc, 100 with a square.
rng = np.random.default_rng(1)
n = {0: 600, 1: 300, 2: 100}
imgs = rng.normal(0.0, 1.0, (1000, 32, 32))
lab = np.repeat([0, 1, 2], [n[0], n[1], n[2]])
yy, xx = np.mgrid[:32, :32]
for k in np.flatnonzero(lab > 0):
    cy, cx = rng.integers(4, 28, 2)
    shape = ((yy - cy) ** 2 + (xx - cx) ** 2 <= 9) if lab[k] == 1 else \
            ((abs(yy - cy) <= 2) & (abs(xx - cx) <= 2))
    imgs[k][shape] += 0.6
box = lambda im: ndimage.uniform_filter(im, 5, mode="constant").max()
z = np.c_[np.full(1000, 0.62), template_score(imgs), [box(im) for im in imgs]]
p = np.exp(20 * z); p /= p.sum(1, keepdims=True)    # softmax "probabilities"

ovr = [auc_rank(p[:, k], lab == k) for k in range(3)]
pairs = []
for j, k in combinations(range(3), 2):
    m = (lab == j) | (lab == k)
    pairs.append((auc_rank(p[m, j], lab[m] == j) + auc_rank(p[m, k], lab[m] == k)) / 2)
print("one-vs-rest AUC per class:", np.round(ovr, 4))
print(f"OvR macro {np.mean(ovr):.4f} (sklearn {roc_auc_score(lab, p, multi_class='ovr'):.4f})"
      f"   OvR weighted {np.average(ovr, weights=list(n.values())):.4f} "
      f"(sklearn {roc_auc_score(lab, p, multi_class='ovr', average='weighted'):.4f})")
print("pairwise AUC (0,1) (0,2) (1,2):", np.round(pairs, 4))
print(f"Hand-Till OvO macro {np.mean(pairs):.4f} "
      f"(sklearn {roc_auc_score(lab, p, multi_class='ovo'):.4f})")

Output:

one-vs-rest AUC per class: [0.7865 0.6868 0.6482]
OvR macro 0.7072 (sklearn 0.7072)   OvR weighted 0.7428 (sklearn 0.7428)
pairwise AUC (0,1) (0,2) (1,2): [0.7515 0.7339 0.5455]
Hand-Till OvO macro 0.6770 (sklearn 0.6770)

The hand-written one-versus-rest and Hand–Till averages match scikit-learn’s. The pairwise AUCs show what the one-versus-rest numbers blur: both shapes are told apart from empty images reasonably well, but disc versus square is barely above chance, because a disc of radius 3 and a 5×5 square excite both filters almost equally. One-versus-rest folds that confusion into each class’s “rest” together with the many easy empty images, and weighting by prevalence raises the average further because the large empty class has the highest one-versus-rest AUC. The one-versus-one average is lower because it gives the hard pair a full third of the weight.

Related Metrics

• Precision and recall define TPR and FPR, the PR curve, and the comparison of ROC and PR curves under class imbalance.
• Average precision summarizes the PR curve and is the standard for detection, where true negatives are undefined.
• Accuracy scores one threshold’s decisions; AUC scores the ranking over all thresholds.
• Expected calibration error measures what AUC ignores: whether predicted probabilities match observed frequencies.
• Equal error rate and TAR at FAR summarize a single point of the ROC or DET curve.