Expected Calibration Error

Expected calibration error (ECE) measures whether a classifier’s confidence can be taken at face value: of the images it labels with 80% confidence, are about 80% labeled correctly? It groups predictions into bins by confidence, compares each bin’s accuracy with its mean confidence, and averages the gaps weighted by how many predictions fall in each bin. Zero means confidence and accuracy agree in every bin. ECE says nothing about how accurate the classifier is, so it is always read next to accuracy, and its value depends on how the bins are chosen.

Formula

A classifier maps an image $x$ to class probabilities, usually the softmax of its logits. Its prediction is the most probable class $\hat{y}$, and its confidence $\hat{p}$ is the probability of that class. Guo, Pleiss, Sun and Weinberger (2017, Eq. 1) call the classifier perfectly calibrated when

$$P\big(\hat{Y} = Y \mid \hat{P} = p\big) = p \qquad \text{for all } p \in [0, 1],$$

and measure miscalibration by the expected gap $\mathbb{E}_{\hat{P}}\big[\,\lvert P(\hat{Y} = Y \mid \hat{P}) - \hat{P} \rvert\,\big]$ (their Eq. 2). Because $\hat{P}$ is continuous, the conditional probability cannot be estimated from finitely many images, so the confidences are binned. With $M$ bins, $B_m$ is the set of test images whose confidence falls in the interval $I_m = \big(\tfrac{m-1}{M}, \tfrac{m}{M}\big]$, and

$$\operatorname{acc}(B_m) = \frac{1}{|B_m|} \sum_{i \in B_m} \mathbf{1}[\hat{y}_i = y_i], \qquad \operatorname{conf}(B_m) = \frac{1}{|B_m|} \sum_{i \in B_m} \hat{p}_i .$$

For $n$ test images, the expected and maximum calibration errors are (Guo et al., 2017, Eqs. 3 and 5)

$$\mathrm{ECE} = \sum_{m=1}^{M} \frac{|B_m|}{n} \,\big\lvert \operatorname{acc}(B_m) - \operatorname{conf}(B_m) \big\rvert, \qquad \mathrm{MCE} = \max_{m} \,\big\lvert \operatorname{acc}(B_m) - \operatorname{conf}(B_m) \big\rvert ,$$

where empty bins are skipped. Guo et al. report ECE with $M = 15$ equal-width bins; Nixon et al. (2019) note that 10 to 20 bins are usual in practice.

A reliability diagram plots $\operatorname{acc}(B_m)$ against confidence for each bin; a calibrated classifier lies on the diagonal, and the bars’ distance from it is the gap that ECE averages. Guo et al. credit the diagrams to DeGroot and Fienberg (1983) and Niculescu-Mizil and Caruana (2005), and note that a reliability diagram does not show how many images each bin holds; their Figure 1 places a confidence histogram above each diagram.

Guo et al. take both measures from Pakdaman Naeini, Cooper and Hauskrecht (2015), who defined ECE and MCE for binary classifiers: the predicted probabilities of the positive class are sorted and split into 10 bins, and each bin’s mean predicted probability is compared with its fraction of positives. The paper does not say whether the bins have equal width or equal counts. Guo et al. carried the definition over to multi-class networks by applying it to the top-label confidence with equal-width bins.

Two older measures score the whole probability vector rather than only the top class. For $K$ classes, with $\hat{p}_{ik}$ the probability given to class $k$ for image $i$,

$$\mathrm{Brier} = \frac{1}{n} \sum_{i=1}^{n} \sum_{k=1}^{K} \big(\hat{p}_{ik} - \mathbf{1}[y_i = k]\big)^2, \qquad \mathrm{NLL} = -\frac{1}{n} \sum_{i=1}^{n} \log \hat{p}_{i y_i} .$$

The first is Brier’s (1950) verification score, which ranges from 0 for certain and correct forecasts to 2 for certain and wrong ones; some papers divide it by $K$ (Ovadia et al., 2019). The second is the cross-entropy loss evaluated on test images (Guo et al. write it as a sum rather than a mean).

Intuition

Think of a weather forecaster who says “70% chance of rain” on many days. If it rains on about 70% of those days, the forecasts are calibrated, whatever happens on any single day. An image classifier makes the same kind of statement with each softmax output. ECE sorts the test images by stated confidence, checks the actual hit rate in each group, and reports how far the two disagree on average, in the same units as accuracy. A classifier with 77% accuracy and 92% mean confidence is overconfident; when its gaps all have the same sign, its ECE is the difference between the two, about 15 points.

Interpretation

• Range: $[0, 1]$, often reported as a percentage; lower is better, and 0 means every bin’s accuracy equals its mean confidence. MCE is also in $[0, 1]$.
• A lower bound. By the triangle inequality, ECE is at least $\lvert \text{accuracy} - \text{mean confidence} \rvert$, with equality when every bin errs in the same direction; this is what ECE reports with a single bin. For the same reason, merging bins can only lower the estimate on a given test set, so splitting bins never decreases it (our reasoning).
• What the numbers looked like. Guo et al. (2017, Table 1) reported ECE (15 bins) typically between 4% and 10% across their image and text classifiers, 16.53% for a 110-layer ResNet on CIFAR-100 and 5.48% for ResNet-152 on ImageNet 2012 (measured on the 25,000 test images of a 1.3 million/25,000/25,000 train/validation/test split; ImageNet’s labeled validation set has 50,000 images, so these are presumably its two halves, though the paper does not say so), falling to 1.26% and 1.86% after temperature scaling. An ECE is only comparable with another computed on the same data with the same binning.
• Over- or underconfidence. ECE is unsigned. The sign of $\operatorname{acc} - \operatorname{conf}$ in the reliability diagram says which way the classifier errs: below the diagonal it is overconfident, above it underconfident.

When to Use It

Use ECE when the confidence itself is used: to defer uncertain images to a human, to threshold detections, to combine a classifier with other probabilistic evidence, or to report risk. Guo et al. motivate calibration with such cases, from a car deciding how much to trust its pedestrian detector to medical diagnosis. ECE is widely used, which is why Minderer et al. (2021) chose it, but they and others pair it with the Brier score and NLL, which do not depend on a binning.

Report the number of bins, whether they have equal width or equal counts, whether the confidence is the top-label or all class probabilities, and the number of test images. Fit anything that changes the probabilities, such as a temperature, on a held-out validation set, never on the test images used for ECE.

Failure Cases

Calibrated without skill. A classifier that gives every image the class frequencies of the training data is calibrated: on balanced data each confidence is $1/K$, and its accuracy is also about $1/K$. Its ECE is near zero while it is useless. Kumar, Liang and Ma (2019) give the example of a dataset of 50% dogs and 50% cats and a model that outputs 0.5 for every image. Ovadia et al. (2019) state the general point: ECE is not a proper scoring rule, and returning the marginal class probabilities for every input yields perfectly calibrated but uninformative predictions.

Not a proper scoring rule. A scoring rule is proper when the forecaster does best in expectation by reporting the true distribution of the outcome, and strictly proper when that optimum is unique (Gneiting & Raftery, 2007). The quadratic (Brier) and logarithmic scores are strictly proper; Brier (1950) already noted that his score rewards honest probabilities and that a forecaster with no skill does best by forecasting the climatological frequencies. ECE has no such property, so minimizing it is not a training objective, and two classifiers with equal ECE can be very different. For binary outputs, the mean squared error decomposes into a squared calibration error plus a sharpness term (Kumar et al., 2019, citing earlier meteorological work), which is why the Brier score rewards both.

Gaps that cancel inside a bin. A bin can average an underconfident and an overconfident region into a perfect match. Nixon et al. (2019) construct a binary example with 45% positives in which negatives get probabilities in (0.41, 0.43) and positives in (0.47, 0.49): the predictions separate the classes perfectly (AUC 1.0) and have ECE 0, yet are not calibrated. (The zero needs a bin that spans both ranges, as 10 equal-width bins do; our note.) Kumar et al. (2019) proved that binning can only underestimate the calibration error of the model’s continuous outputs (their Proposition 3.3) and that for any fixed bins the binned error can be zero while the true error is large (their Example 3.2).

Bias in both directions. The binning bias is negative; the finite number of images per bin adds a positive bias, because fewer images per bin make each bin’s accuracy estimate noisier (Minderer et al., 2021, §5). In our reading, more bins therefore trade a smaller negative bias for a larger positive one. On ImageNet, Kumar et al. found that a Platt-scaled VGG-16 measured with 15 equal-count bins appeared to have an ($\ell_2$) calibration error of about 0.02, while finer bins showed it was at least twice that. Minderer et al. (Figure 9) found that for a set of temperature-scaled BiT models on ImageNet, the sign of the relationship between ECE and classification error flipped depending on whether they used 15, 100 or 5,000 bins, because the positive bias depends on accuracy.

Top label only. The standard ECE ignores the $K - 1$ probabilities that were not the maximum. Nixon et al. (2019) found that ECE tracked the true calibration error worse as the non-maximal probabilities mattered more. Kull et al. (2019, Figure 1) show a wide ResNet on CIFAR-10 that, after temperature scaling, passes a statistical test of top-label calibration at the 0.01 level (p = 0.017) yet fails the same test for classwise calibration (p < 0.0001); in its reliability diagram for class 2, the observed frequency lies below the predicted probability in every bin, so the model overestimates that class. (The figure and the word “over-estimates” in the text agree; the same sentence of the text describes the inequality the other way round.)

Calibration on one distribution. Calibration measured on test images like the training images says little about shifted images. Ovadia et al. (2019) evaluated a ResNet-20 on CIFAR-10 and a ResNet-50 on ImageNet under 16 corruption types at 5 intensities: accuracy and calibration degraded with shift for every method, temperature scaling gave low ECE at small shifts but rising ECE as the shift grew, and better calibration on the i.i.d. test set did not usually carry over to shifted data.

Variants

• Maximum calibration error (MCE) reports the worst bin instead of the weighted mean. It suits high-risk uses (Guo et al., 2017), but a bin holding a handful of images can set it.
• Equal-mass (adaptive) bins. Bins chosen by quantiles of the confidences, so each holds the same number of images, keep the per-bin estimation error roughly constant (Ovadia et al., 2019). Nixon et al. (2019) found that with equal-width bins on confident networks, one or two bins carry most of the ECE, and proposed the adaptive calibration error (ACE), which uses equal-mass ranges for every class probability, and a thresholded version (TACE) that ignores tiny probabilities. Minderer et al. (2021) used equal-mass binning with 100 bins throughout.
• Classwise ECE. A classifier is classwise-calibrated when, for every class $k$, $P(Y = k \mid \hat{p}_k = q) = q$. Kull et al. (2019, Eq. 4) average the binned gap over all $K$ one-versus-rest reliability diagrams; Nixon et al.’s static calibration error (SCE) is the same idea. The strongest notion, full or multiclass calibration, requires the whole probability vector to match the class frequencies among images given that vector (Kull et al., 2019, Definition 1); top-label and classwise calibration are weaker conditions that it implies.
• $\ell_2$ calibration error and debiased estimators. Kumar et al. (2019) work with the root-mean-square gap and use an estimator from the meteorological literature that subtracts an estimate of each bin’s sampling bias, and prove that, for a model whose outputs take one of $B$ values, the number of samples it needs grows with $\sqrt{B}$, against linearly in $B$ for the plain (plug-in) estimate.
• Overconfidence error. Thulasidasan et al. (2019) weight each bin’s gap by its confidence and count only bins where confidence exceeds accuracy.

Calibration of Modern Networks

Guo et al. (2017) found modern networks much less calibrated than older ones; the cross-entropy loss article describes their comparison, the overfitting of the NLL they connected it with, and their remedy, temperature scaling. In their CIFAR-100 experiments (Figure 2), ECE grew with depth and width, a 6-layer convolutional network became worse calibrated with batch normalization although its accuracy improved slightly, and calibration kept improving with more weight decay past the setting that gave the best accuracy.

Minderer et al. (2021) revisited the question for later ImageNet classifiers, measuring ECE with 100 equal-mass bins and fitting temperatures on 20% of the ImageNet validation set. The non-convolutional MLP-Mixer and vision transformer families were among the best calibrated, before and after temperature scaling. Within most families, larger models had slightly higher ECE on clean ImageNet images but lower error, and under distribution shift calibration improved with model size for most families, reversing the in-distribution trend. Neither model size nor amount of pretraining explained the differences between families, and the authors speculate that architecture, in particular its spatial inductive bias, plays an important role.

Training changes affect calibration too. Müller, Kornblith and Hinton (2019, Table 3) reported that label smoothing (defined in the cross-entropy article, whose $\varepsilon$ is their $\alpha$) lowered the ECE of a ResNet-56 on CIFAR-100 from 0.150 to 0.024 ($\alpha = 0.05$), close to the 0.021 that temperature scaling reached ($T = 1.9$); for Inception-v4 on ImageNet the values were 0.071, 0.035 ($\alpha = 0.1$) and 0.022 ($T = 1.4$). Thulasidasan et al. (2019) found that networks trained with mixup (see regularization) were significantly better calibrated than networks trained normally, that mixing images while keeping hard labels did not give the same benefit, and that large mixing parameters made the networks underconfident.

Example

The example builds a four-class problem from $24 \times 24$ images of a square, a disk, a triangle and a cross, shifted by up to 3 pixels and buried in Gaussian noise of standard deviation 1.8 (the clean pixels are 0 or 1). The classifier is template matching: each logit is the best normalized correlation of the image with one class’s template over all shifts, multiplied by a gain of 300, which is far too large, so the softmax is overconfident. A held-out set of 2,000 images is used only to fit a temperature; 10,000 test images measure everything. The code computes the reliability table and the calibration measures before and after temperature scaling and for a base-rate predictor that gives every image the held-out class frequencies, then varies the number of bins and measures ECE for a classifier that is calibrated by construction, because its labels are drawn from its own probabilities.

import numpy as np
from scipy.optimize import minimize_scalar

# Four classes of 24x24 binary shapes (clean pixels 0 or 1).
S = 24
r, c = np.mgrid[0:S, 0:S] - (S - 1) / 2
shapes = {
    "square":   (np.abs(r) <= 5) & (np.abs(c) <= 5),
    "disk":     r**2 + c**2 <= 6.2**2,
    "triangle": (r <= 5) & (r >= -6) & (np.abs(c) <= (r + 6) / 2),
    "cross":    (np.abs(r) <= 2) & (np.abs(c) <= 7) | (np.abs(c) <= 2) & (np.abs(r) <= 7),
}
shape_img = np.stack([s.astype(float) for s in shapes.values()])           # 4 x 24 x 24
K = len(shape_img)
templates = shape_img - shape_img.mean(axis=(1, 2), keepdims=True)
templates /= np.linalg.norm(templates, axis=(1, 2), keepdims=True)

def make_images(n, rng, sigma=1.8):
    """Shapes shifted by up to 3 px, plus Gaussian noise of std sigma."""
    y = rng.integers(0, K, n)
    dy, dx = rng.integers(-3, 4, (2, n))
    X = np.stack([np.roll(shape_img[k], (a, b), axis=(0, 1)) for k, a, b in zip(y, dy, dx)])
    return X + rng.normal(0, sigma, X.shape), y

def logits(X, gain=300.0):
    """Template matching: best normalized correlation over shifts, times a gain."""
    Xc = X - X.mean(axis=(1, 2), keepdims=True)
    Xc /= np.linalg.norm(Xc, axis=(1, 2), keepdims=True)
    best = np.full((len(X), K), -np.inf)
    for a in range(-3, 4):
        for b in range(-3, 4):
            sh = np.roll(templates, (a, b), axis=(1, 2)).reshape(K, -1)
            best = np.maximum(best, Xc.reshape(len(X), -1) @ sh.T)
    return gain * best

def softmax(z, T=1.0):
    e = np.exp((z - z.max(axis=1, keepdims=True)) / T)
    return e / e.sum(axis=1, keepdims=True)

def bins_of(conf, M, scheme):
    if scheme == "width":                                # (m-1)/M < conf <= m/M
        return np.clip(np.ceil(conf * M).astype(int) - 1, 0, M - 1)
    order = np.argsort(conf, kind="stable")              # equal mass: M groups by rank
    b = np.empty(len(conf), int)
    b[order] = np.arange(len(conf)) * M // len(conf)
    return b

def calibration(P, y, M=15, scheme="width"):
    conf, pred = P.max(axis=1), P.argmax(axis=1)
    correct = (pred == y).astype(float)
    b = bins_of(conf, M, scheme)
    rows, ece, mce = [], 0.0, 0.0
    for m in range(M):
        idx = b == m
        if idx.any():
            acc, cf = correct[idx].mean(), conf[idx].mean()
            rows.append((m, idx.sum(), cf, acc))
            ece += idx.mean() * abs(acc - cf)
            mce = max(mce, abs(acc - cf))
    return ece, mce, rows

def brier(P, y):            # Brier (1950): squared error summed over classes, in [0, 2]
    return ((P - np.eye(K)[y]) ** 2).sum(axis=1).mean()

def nll(P, y):
    return -np.log(np.clip(P[np.arange(len(y)), y], 1e-300, None)).mean()

rng = np.random.default_rng(0)
X_val, y_val = make_images(2000, rng)            # held out: used only to fit T
X_test, y_test = make_images(10000, rng)
Z_val, Z_test = logits(X_val), logits(X_test)
P = softmax(Z_test)

ece, mce, rows = calibration(P, y_test)
print(f"test images: {len(y_test)}, accuracy {np.mean(P.argmax(1) == y_test):.3f}, "
      f"mean confidence {P.max(1).mean():.3f}")
print("reliability table, 15 equal-width bins (empty bins omitted):")
print("  bin           n   conf    acc    gap")
for m, n, cf, acc in rows:
    print(f"  ({m/15:.2f},{(m+1)/15:.2f}] {n:5d}  {cf:.3f}  {acc:.3f}  {acc-cf:+.3f}")

def report(name, P, y):
    e_w, m_w, _ = calibration(P, y, 15, "width")
    e_m, m_m, _ = calibration(P, y, 15, "mass")
    print(f"{name:<20} acc {np.mean(P.argmax(1) == y):.3f}  ECE {e_w:.4f} (equal-mass {e_m:.4f})  "
          f"MCE {m_w:.3f} (equal-mass {m_m:.3f})  Brier {brier(P, y):.4f}  NLL {nll(P, y):.4f}")

print()
report("uncalibrated", P, y_test)

# Temperature scaling: one scalar T minimizing NLL on the held-out set.
T = minimize_scalar(lambda t: nll(softmax(Z_val, t), y_val),
                    bounds=(0.05, 20), method="bounded").x
print(f"fitted temperature T = {T:.3f}")
report("temperature-scaled", softmax(Z_test, T), y_test)

# Base-rate predictor: class frequencies of the held-out set for every image.
prior = np.bincount(y_val, minlength=K) / len(y_val)
report("base rate", np.tile(prior, (len(y_test), 1)), y_test)

print("\nECE of the uncalibrated model vs number of bins:")
for M in (5, 15, 50, 100):
    print(f"  M = {M:3d}: equal-width {calibration(P, y_test, M, 'width')[0]:.4f}, "
          f"equal-mass {calibration(P, y_test, M, 'mass')[0]:.4f}")

# A perfectly calibrated model by construction: labels drawn from its own probabilities.
# Its true calibration error is 0; whatever ECE reports is estimation error.
Ps = softmax(Z_test, T)
print("\nlabels sampled from the scaled probabilities (true ECE = 0), 20 draws each:")
for n in (1000, 10000):
    for M in (15, 100):
        e = []
        for s in range(20):
            g = np.random.default_rng(100 + s)
            i = g.choice(len(Ps), n, replace=False)
            ys = (g.random(n)[:, None] > np.cumsum(Ps[i], axis=1)).sum(axis=1)
            e.append(calibration(Ps[i], ys, M, "width")[0])
        print(f"  n = {n:5d}, M = {M:3d}: ECE mean {np.mean(e):.4f}, range {min(e):.4f}-{max(e):.4f}")

Output:

test images: 10000, accuracy 0.772, mean confidence 0.918
reliability table, 15 equal-width bins (empty bins omitted):
  bin           n   conf    acc    gap
  (0.27,0.33]     1  0.302  0.000  -0.302
  (0.33,0.40]    16  0.369  0.250  -0.119
  (0.40,0.47]    63  0.432  0.397  -0.036
  (0.47,0.53]   224  0.508  0.424  -0.084
  (0.53,0.60]   300  0.566  0.377  -0.189
  (0.60,0.67]   337  0.632  0.475  -0.157
  (0.67,0.73]   347  0.701  0.504  -0.197
  (0.73,0.80]   400  0.766  0.520  -0.246
  (0.80,0.87]   486  0.836  0.547  -0.288
  (0.87,0.93]   745  0.903  0.603  -0.301
  (0.93,1.00]  7081  0.992  0.879  -0.113

uncalibrated         acc 0.772  ECE 0.1463 (equal-mass 0.1463)  MCE 0.302 (equal-mass 0.296)  Brier 0.3640  NLL 0.9277
fitted temperature T = 2.881
temperature-scaled   acc 0.772  ECE 0.0092 (equal-mass 0.0120)  MCE 0.037 (equal-mass 0.040)  Brier 0.3142  NLL 0.5759
base rate            acc 0.252  ECE 0.0092 (equal-mass 0.0139)  MCE 0.009 (equal-mass 0.034)  Brier 0.7508  NLL 1.3879

ECE of the uncalibrated model vs number of bins:
  M =   5: equal-width 0.1463, equal-mass 0.1463
  M =  15: equal-width 0.1463, equal-mass 0.1463
  M =  50: equal-width 0.1469, equal-mass 0.1463
  M = 100: equal-width 0.1472, equal-mass 0.1463

labels sampled from the scaled probabilities (true ECE = 0), 20 draws each:
  n =  1000, M =  15: ECE mean 0.0289, range 0.0189-0.0416
  n =  1000, M = 100: ECE mean 0.0763, range 0.0590-0.0913
  n = 10000, M =  15: ECE mean 0.0091, range 0.0055-0.0116
  n = 10000, M = 100: ECE mean 0.0233, range 0.0182-0.0286

Every bin of the reliability table lies below the diagonal: the classifier is overconfident at every confidence level, and most images sit in the top bin, where 99% average confidence buys 88% accuracy. Because all the gaps share a sign, the ECE equals the difference between mean confidence and accuracy, and the bin count and scheme barely change it. The equal-width MCE comes from a bin holding a single image, though here a bin of 745 images has nearly the same gap.

Dividing the logits by the fitted temperature, close to 3, leaves every prediction and so the accuracy unchanged (the softmax article explains why), and removes almost all of the calibration error. The Brier score and NLL fall as well, but less in relative terms, because they also charge for the 23% of images the classifier gets wrong, which no rescaling can fix. The base-rate predictor matches the temperature-scaled classifier’s ECE with a third of its accuracy; only the Brier score and NLL, close to the no-skill values $1 - \sum_k p_k^2 = 0.75$ and $\ln 4 = 1.386$, expose it.

The last block is the floor below which an ECE estimate means nothing. A classifier that is calibrated by construction still shows an ECE of about 0.009 on 10,000 images with 15 bins, and more with fewer images or more bins. The temperature-scaled classifier’s 0.0092 lies inside the spread of the 20 draws at that size, so on this test set it cannot be told apart from a perfectly calibrated one. Comparing a measured ECE with ECEs computed against labels drawn from the predicted probabilities is also the idea behind the calibration test that Kull et al. (2019) apply.

Related Metrics

• Accuracy: what fraction of images is right; ECE asks whether the stated confidence matches that fraction, and the two must be reported together.
• Brier score (Brier, 1950): the squared error of the whole probability vector, a strictly proper scoring rule; it is the mean squared error between predicted probabilities and one-hot labels, summed over classes.
• Negative log-likelihood: the test-set cross-entropy, also strictly proper; it is unbounded, and Ovadia et al. (2019) note that it can over-emphasize tail probabilities.
• Adaptive, classwise and static calibration errors (Nixon et al., 2019; Kull et al., 2019): ECE variants that use equal-mass bins or all class probabilities.
• Ranking metrics such as the area under the ROC curve and average precision (precision and recall): they depend only on the order of the scores, so a monotone rescaling of a binary classifier’s score, such as Platt scaling, leaves them unchanged while it can change ECE completely.