Class Imbalance

Class imbalance means that the classes of a classification problem have very different numbers of training examples. In image classification the extreme and common case is a long-tailed distribution: a few head classes hold most of the images, and most classes, the tail, have only a handful each. A classifier trained by plain cross-entropy on such data favours the head: it predicts frequent classes more often and recognizes rare ones poorly, even when the test set treats all classes as equally important. Long-tailed recognition is the study of learning from such data so that every class is recognized well. Its remedies change what the model sees (resampling), how much each example counts (reweighting and rebalanced losses), how the network is trained (decoupling the features from the classifier), or the decisions of a trained model (post-hoc adjustment for the class prior).

Definition

A training set is imbalanced when its class frequencies $n_1, \dots, n_K$ differ widely. Two summaries are standard:

• Imbalance ratio (Cui, Jia, Lin, Song and Belongie call it the imbalance factor): the number of training images in the largest class divided by the number in the smallest, $\max_k n_k / \min_k n_k$. On their long-tailed versions of CIFAR it ranges from 10 to 200; for ILSVRC 2012 it is 1.78, for iNaturalist 2017 435.44 and for iNaturalist 2018 500 (Cui et al., 2019, Table 1).
• Head, middle and tail groups. Liu et al. (2019) split classes by their number of training images into many-shot, medium-shot and few-shot classes and report accuracy for each group. Their text puts the boundaries at “over 100”, “20∼100” and “under 20” training images; the header of their Table 3 gives $>100$, $\le 100$ and $>20$, and $<20$, so a class with exactly 20 images is medium-shot by the text and in no group by the table.

Long-tailed benchmarks keep the training set imbalanced and make the test set balanced, or score it per class, so that tail classes count as much as head classes.

Intuition

Picture a wildlife camera dataset with thousands of photographs of deer and a dozen of a rare fox. Two different things go wrong.

The first is the prior. A classifier that estimates how likely each class is given the image builds the class frequencies of its training data into that estimate. For an ambiguous photograph, “deer” is the better bet under the training frequencies, so the classifier says deer. That is correct behaviour if the test images come with the same frequencies, and the wrong behaviour if every species matters equally.

The second is estimation. Twelve fox photographs say little about the variety of foxes: poses, seasons, lighting. The model learns the head classes well and the tail classes badly, and may simply memorize the few tail examples. No reweighting can create the information the missing images would have carried.

The two problems need different remedies. The prior can be corrected exactly, after training, if the class frequencies are known. The estimation problem can only be eased, by sharing what is learned about the head with the tail and by not letting the head dominate training.

Formal Definition

The prior inside the posterior

Write $p(\mathbf{x} \mid y)$ for the distribution of images of class $y$ and $\pi_y = n_y / n$ for the training frequency of class $y$. By Bayes’ rule the posterior that a model of the training data approximates is

$$p_{\text{train}}(y \mid \mathbf{x}) = \frac{p(\mathbf{x} \mid y)\,\pi_y}{\sum_k p(\mathbf{x} \mid k)\,\pi_k}.$$

The decision that minimizes the error on a balanced test set, where every class has probability $1/K$, is $\arg\max_y p(\mathbf{x} \mid y)$, the same rule with the prior removed. If a classifier outputs logits $f_y(\mathbf{x})$ with $\operatorname{softmax}(\mathbf{f}) = p_{\text{train}}(y \mid \mathbf{x})$, then $f_y(\mathbf{x}) = \log p(\mathbf{x} \mid y) + \log \pi_y + c(\mathbf{x})$, and it follows that

$$\arg\max_y \big[ f_y(\mathbf{x}) - \log \pi_y \big]$$

is the balanced-test Bayes decision. More generally, for any target prior $q_y$, the posterior under $q$ is $\operatorname{softmax}\big(f_y(\mathbf{x}) - \log \pi_y + \log q_y\big)$. This post-hoc logit adjustment is a direct consequence of Bayes’ rule; we write it with a strength $\tau$, as $f_y - \tau \log \pi_y$, where $\tau = 1$ is the exact correction to a uniform prior.

Prior shift

The setting in which the training and test distributions differ only in the class frequencies, $p(\mathbf{x} \mid y)$ unchanged and $p(y)$ changed, is called prior probability shift in the vocabulary of dataset bias, and label shift by Lipton, Wang and Smola (2018). They motivate it with diagnosis and recognition, where the label causes the observation: objects cause the image, so a change in how often an object appears need not change what it looks like. When the test prior is known, the adjustment above corrects the classifier. When it is not, Lipton et al.’s black box shift estimation (BBSE) estimates the ratios $q_y / \pi_y$ from unlabelled test images. It solves a linear system whose matrix is the confusion matrix of any fixed classifier estimated on labelled data from the training distribution and whose right-hand side is the classifier’s average output on the test images, and it requires only that this confusion matrix be invertible; the classifier may be biased or uncalibrated. They then correct the model by importance-weighted training with the estimated ratios. Long-tailed recognition with a balanced test set is the special case where the shift is known in advance and the target is uniform.

Reweighting as a change of prior

Weighting the cross-entropy of each training image by a factor $w_y$ that depends on its class changes the prior the model learns. By our reasoning, for a model flexible enough to represent any posterior, the population minimizer of $-\mathbb{E}\,[w_y \log q(y \mid \mathbf{x})]$ is

$$q(y \mid \mathbf{x}) \propto w_y\, \pi_y\, p(\mathbf{x} \mid y),$$

so weights $w_y \propto 1/\pi_y$, inverse-frequency weighting, give the balanced posterior in the limit of infinite data. Resampling each class to the same size has the same expected loss. The logistic regression article shows the binary case: balanced class weights shift the fitted intercept by about the logarithm of the class ratio. With finite data the routes differ, because each changes how the model is fitted, not only its final offset.

Effective number of samples

Cui et al. (2019) argue that inverse-frequency weights are too extreme because extra images of a class increasingly overlap with what earlier images already showed. They model each image as covering a region of unit volume in the space of all possible images of its class, of volume $N$, and define the effective number of samples as the expected volume covered by $n$ images. If a new image either falls entirely inside the covered region or entirely outside it, the effective number is

$$E_n = \frac{1 - \beta^n}{1 - \beta}, \qquad \beta = \frac{N - 1}{N} \in [0, 1),$$

so the $j$-th image adds $\beta^{j-1}$, $E_n = 1$ when $\beta = 0$, and $E_n \to n$ as $\beta \to 1$. Their class-balanced loss weights the loss of an image of class $y$ by

$$\frac{1 - \beta}{1 - \beta^{n_y}} \propto \frac{1}{E_{n_y}},$$

normalized so that the weights over the $K$ classes sum to $K$. The hyperparameter $\beta$ interpolates between no reweighting ($\beta = 0$) and inverse frequency ($\beta \to 1$). The same factor can multiply softmax cross-entropy, sigmoid cross-entropy or the focal loss, where it plays the role of the class weight $\alpha_t$.

Properties

Why the tail suffers during training

Under softmax or sigmoid cross-entropy, every training image of class $c$ is a negative example for every other class $j$, and the gradient on $z_j$ is $p_j$, pushing class $j$’s score down. Tan et al. (2020) call these discouraging gradients: for a rare class they arrive far more often than the encouraging gradients from its own few examples, and over many iterations they suppress its scores, so that even its positive examples get low probability. Their measurement in a detector’s last classifier layer shows the average gradient norms of positive and negative samples per category, sorted by instance count (their Figure 1).

Reweighting and resampling

The two classic rebalancing families are resampling, which changes how often each image is drawn, and cost-sensitive reweighting, which changes how much each image’s loss counts (Cui et al., 2019). Oversampling repeats tail images, and Cui et al. note that the duplicates slow training and invite overfitting; undersampling discards head images that could have helped learn features. Byrd and Lipton (2019) studied weighting in deep networks and found that its effect appears early in training and diminishes over successive epochs as long as the network can separate the training data. They connect this to the result, which they summarize from Soudry et al., that a linear model trained by gradient descent on separable data converges in direction to the hard-margin solution, which depends only on where the points lie and not on how they are weighted or how often they are repeated. L2 regularization restored some of the weighting effect in their experiments, which they believe is because the penalty keeps the solution away from the large-norm, margin-like solutions; dropout did not. They also report that weighting the loss of deep networks did not correct for training-set class imbalance in their experiments, while subsampling a class in the training set clearly affected the network’s predictions, which they suggest may be an alternative to weighting on sufficiently large training sets.

Representation against classifier

Zhou, Cui, Wei and Chen (2020) separated the two stages of a network’s training on long-tailed CIFAR-100 with imbalance ratio 50. They trained the backbone with plain cross-entropy, reweighting or resampling, then froze it and retrained the classifier layer from scratch with each of the three; on long-tailed CIFAR-10 with the same ratio they repeated the classifier stage on the CIFAR-100 backbones. With the representation fixed, rebalanced classifier training gave lower error; with the classifier training fixed, the representation learned with plain cross-entropy gave lower error (their Figure 2). They conclude that rebalancing helps the classifier but damages the features to some extent. Their Bilateral-Branch Network trains one branch with a uniform sampler and one with a reversed sampler, whose class probabilities are proportional to the reciprocal of the class size, and a cumulative learning strategy shifts the weight from the first branch to the second over training. Kang et al. (ICLR 2020; not read for this article) proposed, as Alshammari, Wang, Ramanan and Kong (2022) describe it, to decouple the two into independent stages: features learned with the standard cross-entropy loss, then a classifier learned on those features with a class-balancing loss.

Measurement

Plain accuracy on an imbalanced test set rewards the head; balanced accuracy, the mean of per-class recalls, is plain accuracy on a balanced test set. Per-group accuracy shows the trade-off a single number hides: many methods raise few-shot accuracy at some cost in many-shot accuracy. The confusion matrix article shows how rows and columns respond to imbalance, and precision and recall depend on the test prevalence while recall per class does not. In detection, LVIS reports AP separately for rare, common and frequent categories (LVIS covers the protocol and how a per-image cap on detections interacts with rare categories).

Remedies

Resampling

• Class-balanced sampling draws each class with equal probability, so tail images are repeated many times per epoch. PyTorch’s WeightedRandomSampler draws indices with probabilities proportional to given per-image weights, with replacement by default (source, v2.14.1); per-image weights of $1/n_y$ give class-balanced sampling.
• Smoothed sampling. Cui et al. (2019) report that large-scale long-tailed work found inverse-frequency resampling to perform poorly and instead resampled in inverse proportion to the square root of the class frequency, citing word-embedding training and Mahajan et al.’s weakly supervised pretraining.
• Repeat factor sampling for detection repeats an image according to the rarest category it contains, with factor $\max(1, \sqrt{t / f_c})$ for a category in a fraction $f_c$ of images; the LVIS article reports its effect on rare-category AP.
• Undersampling removes head images; Medical Image Classification summarizes Buda et al.’s systematic study on natural-image benchmarks, which found oversampling the minority class the dominant remedy.
• Synthesizing tail examples with interpolation or generation is a third option that Cui et al. mention; data augmentation, mixup included, is the general form.

Reweighting

Inverse-frequency weights $w_y \propto 1/n_y$ are the simplest; scikit-learn’s class_weight='balanced' and PyTorch’s weight argument of the cross-entropy loss implement per-class weights (see the logistic regression and cross-entropy articles for their exact normalizations). Cui et al.’s class-balanced weights soften them with $\beta$.

On long-tailed CIFAR, Cui et al. trained ResNet-32 from scratch and chose the loss type, $\beta \in \{0.9, 0.99, 0.999, 0.9999\}$ and the focal $\gamma$ by cross-validation (their Table 2). At imbalance 100, test error on long-tailed CIFAR-10 fell from 29.64% with softmax cross-entropy (29.55% for the best unweighted loss, sigmoid) to 25.43% with class-balanced focal loss ($\beta = 0.9999$, $\gamma = 1$); on long-tailed CIFAR-100 from 61.68% (61.22% sigmoid) to 60.40% ($\beta = 0.9$). The best $\beta$ was 0.9999 for every CIFAR-10 setting, close to inverse frequency, but smaller for CIFAR-100, where large $\beta$ hurt; the authors attribute this to CIFAR-100’s finer classes having fewer distinct prototypes. On the validation sets of iNaturalist 2018 and ILSVRC 2012, a ResNet-50 at $224 \times 224$ with class-balanced focal loss ($\beta = 0.999$, $\gamma = 0.5$) had top-1 errors of 38.88% and 22.71%, against 42.86% and 23.92% with softmax cross-entropy (their Table 3).

Rebalanced losses

The focal loss down-weights well-classified examples rather than classes and was designed for the foreground–background imbalance of dense detection; that article covers it and its long-tailed variant, Equalized Focal Loss. Liu et al. (2019) found that it raised few-shot accuracy on ImageNet-LT at a cost in many-shot accuracy (below).

The equalization loss of Tan et al. (2020) removes the discouraging gradients from rare classes. For a region proposal $r$ and category $j$ of a sigmoid classifier, the loss $-\sum_j w_j \log \hat p_j$ uses

$$w_j = 1 - E(r)\, T_\lambda(f_j)\, (1 - y_j),$$

where $E(r) = 1$ for a foreground proposal and 0 for background, $f_j$ is the fraction of training images containing category $j$, and $T_\lambda(f) = 1$ when $f < \lambda$. A foreground object of another class thus sends no negative gradient to a rare category, while background proposals still do; without them the model would predict many false positives. With $\lambda = 1.76 \times 10^{-3}$, which covers all rare and common categories, a Mask R-CNN with a ResNet-50-FPN backbone on LVIS v0.5 val went from 20.1 to 22.8 mask AP, with $\mathrm{AP}_r$ from 7.2 to 11.3 and $\mathrm{AP}_c$ from 19.9 to 24.7, while $\mathrm{AP}_f$ moved from 25.4 to 25.1 (their Table 1). They also give a softmax version for image classification, which randomly keeps some of the discouraging gradients because there is no background class.

Decoupled and two-branch training

Following the evidence that rebalancing helps the classifier more than the features, two-stage schemes learn features with ordinary sampling and then rebalance only the classifier, and two-branch networks such as BBN do both at once (see Representation against classifier above).

Post-hoc logit adjustment

Subtracting $\tau \log \pi_y$ from the logits of a model trained on the long-tailed data changes no training at all. With $\tau = 1$ and a model whose softmax matches the training posterior it gives the balanced-test Bayes decision, as derived above; Menon et al. (ICLR 2021; not read for this article) studied logit adjustment for long-tailed learning. The correction removes the prior but not the estimation error of the tail, which the example below shows.

Benchmarks

• Long-tailed CIFAR. Cui et al. (2019) subsample CIFAR-10 and CIFAR-100 with $n_i = n_{\max} \mu^i$ for class index $i$ and $\mu \in (0, 1)$, giving imbalance factors 10 to 200, and keep the balanced test set.
• ImageNet-LT and Places-LT. Liu et al. (2019) sampled a subset of ImageNet (ILSVRC 2012) following a Pareto distribution with power $\alpha = 6$: 115.8K images of 1,000 classes, from 1,280 to 5 images per class, with a balanced test set and extra ImageNet-2010 classes as an open set. Places-LT has 184.5K images of 365 scene classes, from 4,980 to 5 per class. In the closed-set setting with a ResNet-10 trained from scratch, a plain model reached top-1 accuracies of 40.9% (many-shot), 10.7% (medium) and 0.4% (few), 20.9% overall; the focal loss gave 36.4%, 29.9%, 16% and 30.5%, and the authors’ OLTR method 43.2%, 35.1%, 18.5% and 35.6% (Table 3a).
• iNaturalist. iNat2017 has 579,184 training images of 5,089 species; the most populated 1% of classes contain over 16% of the training images, and the validation split has between about 4 and 30 images per class. Accuracy per class on the public test set rose with the number of training images but varied widely among classes with similar amounts of data (Van Horn et al., 2018, Figure 5). Fine-grained image classification covers the dataset as a fine-grained benchmark.
• LVIS for long-tailed detection and instance segmentation, with rare, common and frequent categories.

Examples

Long-tailed shapes with an exact answer

The script below makes a ten-class image problem in which the right answer is known. Each image is one of ten $16 \times 16$ shape templates (disc, square, triangle, ring, plus, X, horizontal bar, vertical bar, diamond, hollow square) with independent Gaussian noise of standard deviation 1.5 on every pixel, averaged over $2 \times 2$ blocks into 64 features. Averaged noise is still independent and Gaussian, with standard deviation 0.75, so every class is a Gaussian with the same isotropic covariance. The exact posterior is then a softmax of linear functions of the features, the Bayes rule can be computed exactly, and multinomial logistic regression is the correct model: any gap between it and the Bayes rule comes from imbalance and finite data, not from a wrong model.

The training set is Zipf-like, $n_k = \operatorname{round}(1000\, k^{-2})$ for $k = 1, \dots, 10$, from 1,000 discs down to 10 hollow squares, an imbalance ratio of 100. Classes are grouped by Liu et al.’s text: many-shot above 100 images (3 classes), medium 20–100 (4), few-shot below 20 (3). The test set is balanced, 500 images per class. Two reference rows use the exact Bayes rule, under the training prior and under a uniform prior; the second is the best any classifier can do, in expectation, on a balanced test set.

The compared methods are a plain fit; inverse-frequency weights; class-balanced weights with $\beta$ = 0.9, 0.99 and 0.999; oversampling every class with replacement to 1,000 images; and post-hoc adjustment of the plain fit by $\tau \log \pi_y$ with $\tau$ = 0.5, 1 and 2. Sample weights are normalized to a mean of 1, and the resampled set’s rows are weighted so their total equals the original set size, so that the L2 penalty has the same relative strength for every method. The penalty strength is a modelling choice the sources leave open and it matters for weighting (Byrd and Lipton), so everything is fitted at scikit-learn’s default $C = 1$ and at a weak penalty, $C = 100$.

The checks come first. The exact posterior under the training prior, adjusted by $-\log \pi_y$, must equal the exact uniform-prior posterior. A plain fit on balanced training data (155 images per class, about the same total) gives the base result; on balanced data every method should make the same predictions as the plain fit, since the weights are equal, nothing is oversampled and the adjustment is a constant. The script then fits every method on long-tailed data for seeds 0–4, prints seed 0 in full and the range over the five seeds. It runs in about a minute, single-threaded, with NumPy 2.3.5, SciPy 1.16.3, scikit-image 0.25.2 and scikit-learn 1.7.2.

import os, sys
for v in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS"):
    os.environ[v] = "1"                       # single-threaded BLAS: reproducible sums
import numpy as np
from scipy.special import softmax
from skimage.draw import disk, polygon
from sklearn.linear_model import LogisticRegression

# --- ten 16x16 shape templates; features are 2x2 block means (8x8 = 64 values) ---
T16 = np.zeros((10, 16, 16))
rr, cc = disk((7.5, 7.5), 5.5, shape=(16, 16)); T16[0][rr, cc] = 1            # disc
T16[1][3:13, 3:13] = 1                                                         # square
rr, cc = polygon([2, 13, 13], [7.5, 2, 13], (16, 16)); T16[2][rr, cc] = 1     # triangle
rr, cc = disk((7.5, 7.5), 6, shape=(16, 16)); T16[3][rr, cc] = 1
rr, cc = disk((7.5, 7.5), 3.5, shape=(16, 16)); T16[3][rr, cc] = 0             # ring
T16[4][6:10, 2:14] = 1; T16[4][2:14, 6:10] = 1                                 # plus
for i in range(2, 14):                                                         # X
    T16[5][i, max(i - 1, 0):i + 2] = 1; T16[5][i, 15 - i - 1:15 - i + 2] = 1
T16[6][5:11, 1:15] = 1                                                         # horizontal bar
T16[7][1:15, 5:11] = 1                                                         # vertical bar
rr, cc = polygon([1, 8, 14, 8], [7.5, 1, 7.5, 14], (16, 16)); T16[8][rr, cc] = 1  # diamond
T16[9][3:13, 3:13] = 1; T16[9][5:11, 5:11] = 0                                 # hollow square
pool = lambda A: A.reshape(-1, 8, 2, 8, 2).mean((2, 4)).reshape(len(A), -1)
T = pool(T16)
SIGMA = 1.5                                   # pixel noise; 0.75 after 2x2 averaging

counts = np.round(1000 * np.arange(1, 11) ** -2.0).astype(int)   # Zipf-like, ratio 100
prior = counts / counts.sum()
groups = {"many": counts > 100, "medium": (counts >= 20) & (counts <= 100), "few": counts < 20}

def sample(n_per_class, rng):
    y = np.repeat(np.arange(10), n_per_class)
    return pool(T16[y] + SIGMA * rng.normal(size=(len(y), 16, 16))), y

def bayes_logits(X, logprior):                # exact: Gaussian classes, shared covariance
    return (X @ T.T - 0.5 * (T ** 2).sum(1)) / (SIGMA / 2) ** 2 + logprior

def fit(X, y, C, w=None):
    m = LogisticRegression(C=C, max_iter=10000, tol=1e-8).fit(X, y, sample_weight=w)
    assert m.n_iter_[0] < 10000, "not converged"
    return m

def class_weights(y, per_class):             # sample weights, normalized to mean 1
    w = per_class[y]
    return w / w.mean()

def methods(X, y, C, rng):
    """Every compared method as (name, logits for a test set)."""
    n = np.bincount(y, minlength=10)
    plain = fit(X, y, C)
    out = [("plain", plain.decision_function)]
    m = fit(X, y, C, class_weights(y, 1.0 / n))
    out.append(("inverse frequency", m.decision_function))
    for beta in (0.9, 0.99, 0.999):           # Cui et al.: weight (1 - beta) / (1 - beta^n)
        m = fit(X, y, C, class_weights(y, (1 - beta) / (1 - beta ** n)))
        out.append((f"class-balanced b={beta}", m.decision_function))
    idx = [np.flatnonzero(y == k) for k in range(10)]   # oversample every class to the largest
    idx = np.concatenate([np.concatenate([i, rng.choice(i, n.max() - len(i))]) for i in idx])
    m = fit(X[idx], y[idx], C, np.full(len(idx), len(y) / len(idx)))
    out.append(("resampling", m.decision_function))
    logprior = np.log(n / n.sum())
    for tau in (0.5, 1.0, 2.0):               # post-hoc: subtract tau * log prior
        out.append((f"logit adjust t={tau}",
                    lambda X, tau=tau: plain.decision_function(X) - tau * logprior))
    return out

def recalls(pred, y):
    return np.array([(pred[y == k] == k).mean() for k in range(10)])

seeds = [int(s) for s in sys.argv[1:]] or [0, 1, 2, 3, 4]
# --- checks: data, exact Bayes identity, base result and code paths on balanced data ---
rng = np.random.default_rng(seeds[0])
Xtr, ytr = sample(counts, rng)
Xte, yte = sample(np.full(10, 500), rng)
print("train counts", np.bincount(ytr).tolist(), "ratio", counts.max() / counts.min(),
      "| groups", {g: int(m.sum()) for g, m in groups.items()}, "| test", np.bincount(yte).tolist())
lt, lu = bayes_logits(Xte, np.log(prior)), bayes_logits(Xte, 0.0)
print(f"exact posterior, training prior minus log prior vs uniform prior: max |diff| "
      f"{np.abs(softmax(lt - np.log(prior), axis=1) - softmax(lu, axis=1)).max():.1e}")
bayes = lu.argmax(1)
rb, rt = recalls(bayes, yte), recalls(lt.argmax(1), yte)
print(f"exact Bayes balanced accuracy: uniform prior {rb.mean():.3f}, training prior {rt.mean():.3f}"
      f" (few-shot recall {rt[groups['few']].mean():.3f})")
Xb, yb = sample(np.full(10, 155), np.random.default_rng(1))
for C in (1.0, 100.0):
    ms = methods(Xb, yb, C, np.random.default_rng(1))
    p0 = ms[0][1](Xte).argmax(1)
    same = min((f(Xte).argmax(1) == p0).mean() for _, f in ms[1:])
    print(f"balanced training data, C={C:g}: plain balanced accuracy {recalls(p0, yte).mean():.3f};"
          f" every method agrees with plain on at least {same:.1%} of test images")

# --- long-tailed training set, balanced test set ---
summary = {}
for s in seeds:
    rng = np.random.default_rng(s)
    Xtr, ytr = sample(counts, rng)
    Xte, yte = sample(np.full(10, 500), rng)
    lt, lu = bayes_logits(Xte, np.log(prior)), bayes_logits(Xte, 0.0)
    rows = [("Bayes, training prior", lt), ("Bayes, uniform prior", lu)]
    for C in (1.0, 100.0):
        rows += [(f"C={C:g} {name}", f(Xte)) for name, f in methods(Xtr, ytr, C, np.random.default_rng(100 + s))]
    if s == seeds[0]:
        print(f"\nseed {s}: recall by group, balanced accuracy, agreement with the Bayes rule")
        print(f"{'':34s} many  medium  few   bal acc  =Bayes")
    for name, L in rows:
        pred = L.argmax(1)
        r = recalls(pred, yte)
        summary.setdefault(name, []).append((r.mean(), r[groups["few"]].mean()))
        if s == seeds[0]:
            print(f"{name:34s} {r[groups['many']].mean():.3f} {r[groups['medium']].mean():.3f} "
                  f"{r[groups['few']].mean():.3f}  {r.mean():.3f}   {(pred == lu.argmax(1)).mean():.3f}")
print(f"\nseeds {seeds}: range of balanced accuracy | few-shot recall")
for name, v in summary.items():
    v = np.array(v)
    print(f"{name:34s} {v[:, 0].min():.3f}-{v[:, 0].max():.3f} | {v[:, 1].min():.3f}-{v[:, 1].max():.3f}")

Output:

train counts [1000, 250, 111, 62, 40, 28, 20, 16, 12, 10] ratio 100.0 | groups {'many': 3, 'medium': 4, 'few': 3} | test [500, 500, 500, 500, 500, 500, 500, 500, 500, 500]
exact posterior, training prior minus log prior vs uniform prior: max |diff| 8.9e-16
exact Bayes balanced accuracy: uniform prior 0.771, training prior 0.624 (few-shot recall 0.319)
balanced training data, C=1: plain balanced accuracy 0.716; every method agrees with plain on at least 100.0% of test images
balanced training data, C=100: plain balanced accuracy 0.683; every method agrees with plain on at least 100.0% of test images

seed 0: recall by group, balanced accuracy, agreement with the Bayes rule
                                   many  medium  few   bal acc  =Bayes
Bayes, training prior              0.692 0.803 0.319  0.624   0.652
Bayes, uniform prior               0.659 0.878 0.739  0.771   1.000
C=1 plain                          0.657 0.613 0.161  0.491   0.510
C=1 inverse frequency              0.696 0.720 0.286  0.583   0.642
C=1 class-balanced b=0.9           0.654 0.614 0.177  0.495   0.515
C=1 class-balanced b=0.99          0.649 0.666 0.239  0.533   0.558
C=1 class-balanced b=0.999         0.692 0.710 0.279  0.575   0.628
C=1 resampling                     0.700 0.718 0.283  0.582   0.638
C=1 logit adjust t=0.5             0.679 0.716 0.275  0.572   0.618
C=1 logit adjust t=1.0             0.655 0.766 0.391  0.620   0.691
C=1 logit adjust t=2.0             0.397 0.739 0.595  0.593   0.639
C=100 plain                        0.637 0.608 0.192  0.492   0.514
C=100 inverse frequency            0.679 0.646 0.231  0.531   0.581
C=100 class-balanced b=0.9         0.637 0.609 0.197  0.493   0.515
C=100 class-balanced b=0.99        0.637 0.622 0.214  0.504   0.528
C=100 class-balanced b=0.999       0.667 0.639 0.227  0.524   0.571
C=100 resampling                   0.679 0.647 0.234  0.533   0.581
C=100 logit adjust t=0.5           0.655 0.638 0.226  0.519   0.562
C=100 logit adjust t=1.0           0.673 0.662 0.268  0.547   0.598
C=100 logit adjust t=2.0           0.610 0.693 0.334  0.561   0.604

seeds [0, 1, 2, 3, 4]: range of balanced accuracy | few-shot recall
Bayes, training prior              0.609-0.625 | 0.295-0.325
Bayes, uniform prior               0.760-0.777 | 0.722-0.745
C=1 plain                          0.491-0.511 | 0.133-0.179
C=1 inverse frequency              0.574-0.602 | 0.239-0.318
C=1 class-balanced b=0.9           0.495-0.515 | 0.146-0.201
C=1 class-balanced b=0.99          0.524-0.549 | 0.192-0.269
C=1 class-balanced b=0.999         0.567-0.594 | 0.227-0.311
C=1 resampling                     0.572-0.602 | 0.237-0.311
C=1 logit adjust t=0.5             0.559-0.589 | 0.231-0.305
C=1 logit adjust t=1.0             0.613-0.630 | 0.361-0.437
C=1 logit adjust t=2.0             0.593-0.611 | 0.567-0.629
C=100 plain                        0.483-0.519 | 0.163-0.231
C=100 inverse frequency            0.518-0.556 | 0.195-0.268
C=100 class-balanced b=0.9         0.484-0.520 | 0.167-0.233
C=100 class-balanced b=0.99        0.490-0.528 | 0.175-0.250
C=100 class-balanced b=0.999       0.513-0.550 | 0.189-0.263
C=100 resampling                   0.516-0.558 | 0.195-0.267
C=100 logit adjust t=0.5           0.507-0.551 | 0.190-0.270
C=100 logit adjust t=1.0           0.526-0.573 | 0.213-0.311
C=100 logit adjust t=2.0           0.541-0.582 | 0.275-0.377

The checks hold. The adjusted exact posterior matches the uniform-prior posterior to rounding error, which confirms the derivation and the code. The ground truth carries the imbalance effect: under the training prior even the exact Bayes rule recognizes under a third of the few-shot images, against nearly three quarters under a uniform prior. On balanced data all methods agree with the plain fit on every test image, so the differences below come from imbalance alone; the plain fit there falls short of the Bayes rule by 0.055 ($C = 1$) and 0.088 ($C = 100$), the cost of estimating 650 parameters from 1,550 images.

The plain fit. Few-shot recall is about a quarter of many-shot recall. Its balanced accuracy is well below not only the uniform-prior Bayes row but also the training-prior Bayes row, the classifier an exact model of the training data would give. We expected it near the training-prior row, and that expectation did not hold: besides the prior, the tail classes are estimated badly from 10 to 16 images each, and the gap from the plain fit to the training-prior row is about as large as the gap between the two Bayes rows, which is the whole effect of the prior.

Weighting and resampling. At $C = 1$, inverse-frequency weighting and oversampling give nearly the same result, as their equal expected losses suggest, and both raise medium and few-shot recall. We had expected them to lower many-shot recall, as the move from the training-prior to the uniform-prior Bayes row does; on seed 0 it rose instead, so that part of the expectation also did not hold. Class-balanced weights move from the plain fit ($\beta = 0.9$, where the weights of the largest and smallest class differ by a factor of only about 1.5) to almost the inverse-frequency result ($\beta = 0.999$), as their formula implies.

Post-hoc adjustment. At $C = 1$, adjusting the plain fit with $\tau = 1$ gives the highest balanced accuracy of all methods for every seed (its lowest value is above every other method’s highest) and, on seed 0, the highest agreement with the Bayes rule; $\tau = 2$ over-corrects, buying more few-shot recall with a collapse of many-shot recall. Even the best method stays far below the uniform-prior Bayes row: the adjustment removes the prior exactly, but the poorly estimated tail remains.

The weak penalty. On seed 0, weighting and resampling at $C = 100$ change balanced accuracy by less than half as much as at $C = 1$. This is the direction Byrd and Lipton’s findings suggest, since in their experiments L2 regularization restored some of the effect that weighting lost in unregularized training; whether this fit separates its training data is not shown here. We expected post-hoc adjustment to keep correcting under the weak penalty; it gains less than at $C = 1$, so that expectation held only in part. We also expected $\tau = 2$ to over-correct here as at $C = 1$, and that expectation failed: on seed 0, $\tau = 2$ has higher balanced accuracy and higher agreement with the Bayes rule than $\tau = 1$, and its range over seeds lies higher. One reading, not tested here, is that under the weak penalty the fitted logits no longer carry the training prior as $\log \pi_y$, so subtracting exactly $\log \pi_y$ is no longer the right correction.

All settings, including both penalties and every printed value, were fixed before the first run and not changed after it.

Common Misconceptions

• “Balance the data and the problem is solved.” Rebalancing corrects the prior, but a class with ten images is still estimated from ten images. In the example the best method recovers only part of the gap to the Bayes rule.
• “Weighting and resampling always change the model.” For models that separate their training data, Byrd and Lipton found that the effect of weighting fades with training; in the example, both matter less under the weaker penalty.
• “High accuracy means the tail is handled.” On a long-tailed test set the head dominates plain accuracy; report balanced accuracy and per-group results (accuracy shows a model that never finds the rare class and still scores 90%).
• “Rebalanced scores are probabilities for the deployment population.” A model trained or adjusted toward a uniform prior produces scores for that prior. If the deployment prior differs, the same Bayes correction moves the scores to it; expected calibration error measures whether they are calibrated.

Where It Is Used

Imbalance is the rule in recognition data drawn from the world: species in iNaturalist, object categories in LVIS, expressions in facial expression recognition, findings in medical image classification, where the rare class is often the one that matters. In object detection the imbalance between foreground and background is handled by sampling ratios, hard negative mining and the focal loss, and the long tail of categories by repeat factor sampling and equalization losses. In semantic segmentation the imbalance is between pixel classes, and the Dice coefficient loss is one answer. Prior shift between a development set and a deployment site is a concern of clinical validation, whose example shows the calibration intercept that a change of prevalence alone produces.