Mixup and CutMix

Mixup and CutMix train an image classifier on images that are combinations of two training images, with a label that combines the two labels in the same proportion. Mixup (Zhang et al., 2018) blends the two images pixel by pixel. CutMix (Yun et al., 2019) cuts a rectangle out of one image and fills it with the same rectangle from the other, so every pixel still belongs to one real image. Manifold Mixup (Verma et al., 2019) blends hidden features inside the network instead of pixels. The family is related to two other ideas: data augmentation that hides part of the image (Cutout and Random Erasing, which keep the label unchanged) and training on soft targets, as label smoothing does. These methods are sometimes called mixed-sample augmentation or sample-mixing augmentation. Together with label smoothing they are common ingredients of ImageNet training recipes, including DeiT’s recipe for vision transformers.

This article draws on the CutMix, Manifold Mixup, Random Erasing, DeiT and Bag of Tricks papers and on the official code. The mixup paper itself (on openreview.net) and the Cutout paper (on arXiv) could not be read for this article; only the mixup abstract on the ICLR site was read. Their methods are described here as the readable papers restate them.

Purpose

A classifier trained with one-hot labels sees each training image only with its own label, and can fit the training set while behaving arbitrarily between training images. Training on combinations of images, with matching combinations of labels, gives the network targets at points that are not in the training set.

CutMix was motivated differently. Regional dropout methods, such as Cutout, which blacks out a square, and Random Erasing, which fills a rectangle with noise, make a network attend to less discriminative parts of an object (the paper’s example is a person’s leg rather than the head), which helps both generalization and localization (Yun et al., 2019, §1). But the erased pixels carry no information. CutMix fills the hole with a patch of another training image and gives that image a share of the label, so no pixel is wasted while the occlusion effect is kept. The authors also argue that mixup’s blended images are locally ambiguous and unnatural, which confuses the network when it has to localize objects (§2, §3.2).

When to Use It

• Image classification trained for many epochs. The classification results below come mostly from long schedules: 300 epochs for CutMix on ImageNet and CIFAR, 200 instead of 120 for mixup in Bag of Tricks, 300 for DeiT. Both CutMix’s and Bag of Tricks’ authors note that the mixed examples need longer training to converge.
• Data-hungry models. DeiT’s authors write that transformers, which build in fewer priors than convolutions, need extensive augmentation when trained on ImageNet alone, and their ablation shows mixup and CutMix to be among the components that matter (Evidence and Impact).
• Backbones whose features will be used for localization or detection. CutMix improved weakly supervised localization and transfer to detection and captioning in its paper’s experiments. With ResNet-50, mixup lowered localization accuracy and the detection results and raised the captioning scores only slightly.

It is a poor fit, or needs care, when:

• The label is not a property of the whole image. Pixel-wise and box-level tasks need the annotations mixed too. Depth Anything, for example, applies CutMix to unlabeled images and compares each region of the student’s prediction with the teacher’s pseudo-label for the image that region came from, rather than mixing labels (see Depth Anything).
• The backbone will be fine-tuned for dense prediction. In Bag of Tricks, adding mixup lowered the ADE20K segmentation result of an FCN built on the backbone (Pitfalls).
• Training is short. In our reading of the long schedules above, the mixed examples are a harder training task, and a short schedule may stop before the network has fitted them.

How It Works

Mixing images and labels

Let $(x_A, y_A)$ and $(x_B, y_B)$ be two training images with one-hot labels. All three methods sample a weight $\lambda \in [0, 1]$ from a Beta distribution $\mathrm{Beta}(\alpha, \alpha)$ and build

$$\tilde{y} = \lambda\, y_A + (1 - \lambda)\, y_B .$$

They differ in how the image is formed:

• Mixup: $\tilde{x} = \lambda\, x_A + (1 - \lambda)\, x_B$ (He, T. et al., 2019, Eqs. 7 and 8, restating Zhang et al.).
• CutMix: $\tilde{x} = M \odot x_A + (\mathbf{1} - M) \odot x_B$, where $M$ is a binary mask that is 0 inside a rectangle and 1 elsewhere (Yun et al., 2019, Eq. 1). The rectangle’s area is chosen so that $x_B$ fills a fraction $1 - \lambda$ of the image.
• Manifold Mixup: the network $f = f_k \circ g_k$ is split at a layer $k$ chosen at random from a set of eligible layers, and the hidden features are mixed: $f_k\big(\lambda\, g_k(x_A) + (1 - \lambda)\, g_k(x_B)\big)$ is trained toward $\tilde{y}$ (Verma et al., 2019, §2). When the only eligible layer is the input, this is mixup.

$\alpha$ sets how far from the endpoints $\lambda$ usually falls. With $\alpha = 1$ the distribution is uniform on $[0, 1]$, the value CutMix used in all its experiments (§3.1). Small values such as $\alpha = 0.2$, used by Bag of Tricks for mixup, put most of the mass near 0 and 1, so most mixed images are close to one of the originals. Large values concentrate $\lambda$ near 0.5.

In practice the partner images come from the same mini-batch. The official CutMix code pairs each image with the image at a randomly permuted position in the batch. Manifold Mixup mixes the mini-batch with a copy of itself with shuffled rows, and samples one layer and one $\lambda$ per mini-batch, which the authors report performs the same as sampling them per example (§2). The timm library, which DeiT uses, pairs each image with the one at the mirrored position in the batch and by default draws one $\lambda$ (and, for CutMix, one rectangle) per batch.

The loss with mixed labels

The mixed target is used with the ordinary loss. Because the cross-entropy loss is linear in the target distribution,

$$-\sum_k \tilde{y}_k \log p_k = \lambda \Big(-\sum_k y_{A,k} \log p_k\Big) + (1 - \lambda) \Big(-\sum_k y_{B,k} \log p_k\Big),$$

training on the mixed label is the same as mixing the two losses (our reasoning from the definition). The official CutMix code takes the second form, criterion(output, target_a) * lam + criterion(output, target_b) * (1. - lam), while timm builds the soft target, which DeiT’s code pairs with timm’s soft-target cross-entropy. timm also applies label smoothing to the targets before mixing, and DeiT’s code uses this combination with $\varepsilon = 0.1$. Since smoothing is an affine map of the target and the mixing weights sum to one, the order does not matter (our reasoning).

Not every recipe uses a softmax cross-entropy. The ResNet article reports that the “ResNet strikes back” procedures of Wightman, Touvron and Jégou (2021) use a binary cross-entropy loss that treats every class mixed in by mixup or CutMix as present.

CutMix’s rectangle and the label correction

The CutMix paper samples the rectangle’s centre uniformly over the whole image and sets its width and height in proportion to the image’s:

$$r_x \sim \mathrm{Unif}(0, W),\quad r_y \sim \mathrm{Unif}(0, H),\quad r_w = W\sqrt{1 - \lambda},\quad r_h = H\sqrt{1 - \lambda},$$

so that $r_w r_h / (WH) = 1 - \lambda$ (Yun et al., 2019, Eq. 2). A centre near the border puts part of the rectangle outside the image, and the rectangle is clipped to the image. The pasted area is then smaller than $1 - \lambda$, and a label weight of $1 - \lambda$ would overstate how much of $x_B$ the image contains. The supplement’s pseudo-code (Algorithm A1, line 15) and the official train.py therefore recompute the weight from the clipped rectangle, “to exactly match pixel ratio” (in the code’s comment):

$$\lambda \leftarrow 1 - \frac{(x_2 - x_1)(y_2 - y_1)}{WH}.$$

Two small differences between the versions: Algorithm A1 omits the factor $W$ (and $H$) in the box size and has a min where line 13 needs a max, while Eq. 2 and the code are consistent with each other. The official code rounds the side down to an integer and spans cut_w // 2 pixels on each side of the centre, so even an unclipped box can be one pixel narrower than the side computed from $\lambda$; the correction absorbs this too. timm applies the same correction (correct_lam=True by default) with slightly different rounding. The example below measures how large the correction is.

Cutout and Random Erasing

These two contemporary methods erase a rectangle without mixing labels; the data augmentation article gives their parameters and TorchVision’s defaults. Cutout zeroes a fixed-size square at a random position, and CutMix’s authors used $16 \times 16$ on CIFAR and $112 \times 112$ on ImageNet when they compared it (§4.1). Random Erasing is applied with probability $p$, draws the rectangle’s area and aspect ratio at random, and fills it with random values. Unlike CutMix, it keeps the whole rectangle inside the image: a candidate whose corner would put it past the border is rejected and drawn again (Zhong et al., 2020, Algorithm 1). For detection, its authors also erase regions inside each object’s box (object-aware erasing). Random Erasing’s own ablation on CIFAR-10 found random fill values and the ImageNet mean pixel better than filling with 0 or 255, and found that pixel-level dropout or noise at the input did not help at all (their Tables 3 and 4).

What Manifold Mixup changes in the features

Verma et al. (2019, §3.1) prove that, for a sufficiently expressive network, if the dimension of the mixed layer is at least the number of classes minus one, the Manifold Mixup loss can be driven to zero with a linear map after that layer, and that at such a minimizer the representations of each class lie on a subspace of dimension $\dim(\mathcal{H}) - d + 1$, where $d$ is the number of classes: the class representations are “flattened”. In an MNIST experiment with fully connected networks, the sum of the singular values after the largest one (first hidden layer) was 78.67 for the baseline, 73.36 with weight decay, 77.47 with dropout, 66.89 with input mixup and 40.98 with Manifold Mixup (§3.2). The text says that only Manifold Mixup reduced this sum, although its own numbers show smaller reductions for the others. The authors also report that the layers before the mixing layer matter: when they stopped gradients immediately after the mixing layer, test error on CIFAR-10 with a PreActResNet18 was 4.33%, worse than input mixup and Manifold Mixup but better than the baseline (§5.1). Their related-work section suggests that Manifold Mixup may work for different reasons than input mixup.

How to Apply It

Libraries implement the batch operation directly. TorchVision’s MixUp and CutMix transforms act on whole batches after the data loader, as the data augmentation article shows. timm’s Mixup class combines both: with both enabled, each batch uses CutMix with probability switch_prob and mixup otherwise, and the result is a soft target matrix. DeiT’s training script, at the commit read, defaults to mixup $\alpha = 0.8$, CutMix $\alpha = 1.0$, mixing applied to every batch, a switch probability of 0.5, one $\lambda$ per batch, label smoothing 0.1 applied through the mixed targets, and Random Erasing with probability 0.25. The official CutMix README trains ImageNet models with CutMix applied to every batch (--cutmix_prob 1.0) and CIFAR-100 models with probability 0.5, both with $\alpha = 1$.

Example: how much of the image does a CutMix rectangle cover?

The script ports rand_bbox from the official CutMix code. NumPy’s np.int, which the original uses, was removed in NumPy 1.24; it was an alias of Python’s int, so the port uses int truncation with the same behaviour. A seeded NumPy generator replaces the global np.random. The script draws 200,000 values of $\lambda \sim \mathrm{Beta}(1, 1)$ for square images of side 32 (as in CIFAR) and 224 (as in ImageNet training), and measures:

• the pasted fraction;
• how often the rectangle is clipped;
• how often a corner pixel and the centre pixel are pasted over.

It compares each Monte Carlo mean with the exact value, which it computes by enumerating every rectangle side and centre. The side $k = \lfloor W\sqrt{1-\lambda} \rfloor$ takes the value $k$ with probability $(2k+1)/W^2$ when $\lambda$ is uniform. It also compares an inside-the-image sampler: the same side, but placed entirely inside the image, as Random Erasing places its rectangles.

The script then pastes rectangles into real photographs: batches of eight images bundled with scikit-image (astronaut, cat, coffee, rocket, camera, coins, moon and page, resized to $224 \times 224$). Each image carries a fourth channel that is 0 for its own pixels and 1 for pixels taken from its partner, so the fraction of each image that came from the partner can be counted exactly and compared with the corrected and uncorrected label weights. The script’s checks were fixed before it was first run, as tolerances of four Monte Carlo standard errors around the exact values. It prints them at the end. The script uses NumPy 2.3.5 and scikit-image 0.25.2 and takes about 10 seconds.

"""PILOT, example 1: CutMix's box sampler, the clipped area and the label correction.

rand_bbox is ported from clovaai/CutMix-PyTorch train.py (commit 2d8eb68): np.int (removed in
NumPy 1.24; it was an alias of the built-in int) becomes int truncation, np.random.randint(W)
becomes Generator.integers(0, W). Box and label logic unchanged.
"""
import numpy as np
from skimage import data
from skimage.color import gray2rgb
from skimage.transform import resize


def official_boxes(W, lam, rng):
    """Vectorized rand_bbox for square W x W images: returns x1, x2, y1, y2 (half-open)."""
    k = (W * np.sqrt(1.0 - lam)).astype(int)        # cut_w = cut_h = int(W * sqrt(1 - lam))
    cx = rng.integers(0, W, lam.shape); cy = rng.integers(0, W, lam.shape)
    h = k // 2
    return (np.clip(cx - h, 0, W), np.clip(cx + h, 0, W),
            np.clip(cy - h, 0, W), np.clip(cy + h, 0, W), k)


def inside_boxes(W, lam, rng):
    """Same side k, but the whole box inside the image (top-left uniform), as Random Erasing keeps it."""
    k = (W * np.sqrt(1.0 - lam)).astype(int)
    tx = rng.integers(0, W - k + 1); ty = rng.integers(0, W - k + 1)
    return tx, tx + k, ty, ty + k, k


def exact(W):
    """Exact expectations for the official sampler and the inside sampler, lam ~ U(0, 1)."""
    p = np.arange(W); cx = np.arange(W)
    out = dict(pasted=0.0, clipped=0.0, corner=0.0, centre=0.0, in_pasted=0.0, in_corner=0.0, in_centre=0.0)
    for k in range(W):
        pk = (2 * k + 1) / W ** 2                    # P(int(W sqrt(1 - lam)) = k)
        h = k // 2
        x1, x2 = np.clip(cx - h, 0, W), np.clip(cx + h, 0, W)
        w = (x2 - x1) / W
        out["pasted"] += pk * w.mean() ** 2
        out["clipped"] += pk * (1 - np.mean(x2 - x1 == 2 * h) ** 2)
        cov = ((x1[None] <= p[:, None]) & (p[:, None] < x2[None])).mean(1)
        out["corner"] += pk * cov[0] ** 2; out["centre"] += pk * cov[W // 2] ** 2
        t = np.arange(W - k + 1)
        cin = ((t[None] <= p[:, None]) & (p[:, None] < t[None] + k)).mean(1)
        out["in_corner"] += pk * cin[0] ** 2; out["in_centre"] += pk * cin[W // 2] ** 2
        out["in_pasted"] += pk * (k / W) ** 2
    return out


checks = []
def check(name, ok):
    checks.append((name, bool(ok)))


rng = np.random.default_rng(0)
S = 200_000
for W in (32, 224):
    ex = exact(W)
    lam = rng.beta(1.0, 1.0, S)
    x1, x2, y1, y2, k = official_boxes(W, lam, rng)
    pasted = (x2 - x1) * (y2 - y1) / W ** 2
    lam_c = 1 - pasted                                   # the official correction
    clipped = ((x2 - x1) < 2 * (k // 2)) | ((y2 - y1) < 2 * (k // 2))
    corner = (x1 <= 0) & (0 < x2) & (y1 <= 0) & (0 < y2)
    c = W // 2
    centre = (x1 <= c) & (c < x2) & (y1 <= c) & (c < y2)
    ix1, ix2, iy1, iy2, _ = inside_boxes(W, lam, rng)
    in_pasted = (ix2 - ix1) * (iy2 - iy1) / W ** 2
    in_corner = (ix1 <= 0) & (0 < ix2) & (iy1 <= 0) & (0 < iy2)
    in_centre = (ix1 <= c) & (c < ix2) & (iy1 <= c) & (c < iy2)
    se = lambda v: v.std(ddof=1) / np.sqrt(len(v))
    print(f"W = {W}")
    rows = [("nominal 1 - lam", 1 - lam, 0.5), ("pasted fraction", pasted, ex["pasted"]),
            ("lam_c - lam", lam_c - lam, 1 - ex["pasted"] - 0.5), ("clipped", clipped.astype(float), ex["clipped"]),
            ("corner covered", corner.astype(float), ex["corner"]), ("centre covered", centre.astype(float), ex["centre"]),
            ("inside: pasted", in_pasted, ex["in_pasted"]), ("inside: corner", in_corner.astype(float), ex["in_corner"]),
            ("inside: centre", in_centre.astype(float), ex["in_centre"])]
    for name, v, e in rows:
        z = (v.mean() - e) / se(v)
        print(f"  {name:16s} MC {v.mean():.5f} +- {se(v):.5f}   exact {e:.5f}   z {z:+.2f}")
        check(f"W={W} {name} within 4 SE", abs(z) <= 4)
    print(f"  min(lam_c - lam) = {np.min(lam_c - lam):.2e}  (lam_c >= lam always)")
print(f"continuous Eq. (2) with clipping: 77/240 = {77/240:.5f}")

# Pixel check on real images at 224: an indicator channel records which pixels came from the second image.
names = ["astronaut", "chelsea", "coffee", "rocket", "camera", "coins", "moon", "page"]
imgs = []
for nm in names:
    im = getattr(data, nm)()
    im = gray2rgb(im) if im.ndim == 2 else im[..., :3]
    imgs.append(resize(im, (224, 224), anti_aliasing=True))
base = np.stack(imgs).transpose(0, 3, 1, 2)            # (8, 3, 224, 224), values in [0, 1]
base = np.concatenate([base, np.zeros((8, 1, 224, 224))], axis=1)
print("batch", base.shape, "image range", f"{base[:, :3].min():.3f}..{base[:, :3].max():.3f}")
check("batch shape (8,4,224,224)", base.shape == (8, 4, 224, 224))
check("images in [0,1]", base[:, :3].min() >= 0 and base[:, :3].max() <= 1)

max_err_corrected, err_nominal = 0.0, []
for b in range(500):
    x = base.copy()
    src = base.copy(); src[:, 3] = 1.0                  # pixels taken from the shuffled batch carry indicator 1
    lam = rng.beta(1.0, 1.0)
    rand_index = rng.permutation(8)
    bx1, bx2, by1, by2, _ = official_boxes(224, np.array(lam), rng)
    bx1, bx2, by1, by2 = int(bx1), int(bx2), int(by1), int(by2)
    x[:, :, bx1:bx2, by1:by2] = src[rand_index, :, bx1:bx2, by1:by2]
    lam_c = 1 - (bx2 - bx1) * (by2 - by1) / (224 * 224)
    frac = x[:, 3].mean(axis=(1, 2))                    # fraction of each image from the second image
    if b == 0:
        check("indicator in {0,1}", set(np.unique(x[:, 3])) <= {0.0, 1.0})
    max_err_corrected = max(max_err_corrected, np.abs(frac - (1 - lam_c)).max())
    err_nominal.append(np.abs(frac[0] - (1 - lam)))
err_nominal = np.array(err_nominal)
print(f"pixel check: max |pixel fraction - (1 - lam_c)| = {max_err_corrected:.1e}; "
      f"mean |pixel fraction - (1 - lam)| = {err_nominal.mean():.4f} +- {err_nominal.std(ddof=1)/np.sqrt(500):.4f}")
check("corrected label exact", max_err_corrected <= 1e-12)
z = (err_nominal.mean() - 0.182222) / (err_nominal.std(ddof=1) / np.sqrt(500))
check("uncorrected error within 4 SE of exact 0.182222", abs(z) <= 4)
print(f"  z vs exact 0.182222: {z:+.2f}")

print("\nPILOT CRITERIA")
for name, ok in checks:
    print(f"  {'PASS' if ok else 'FAIL'}  {name}")

Output:

W = 32
  nominal 1 - lam  MC 0.49981 +- 0.00065   exact 0.50000   z -0.30
  pasted fraction  MC 0.29926 +- 0.00041   exact 0.29925   z +0.03
  lam_c - lam      MC 0.20055 +- 0.00037   exact 0.20075   z -0.55
  clipped          MC 0.78678 +- 0.00092   exact 0.78760   z -0.89
  corner covered   MC 0.13647 +- 0.00077   exact 0.13540   z +1.38
  centre covered   MC 0.45886 +- 0.00111   exact 0.45837   z +0.44
  inside: pasted   MC 0.47898 +- 0.00063   exact 0.47917   z -0.30
  inside: corner   MC 0.03394 +- 0.00040   exact 0.03421   z -0.67
  inside: centre   MC 0.81737 +- 0.00086   exact 0.81792   z -0.64
  min(lam_c - lam) = 9.43e-07  (lam_c >= lam always)
W = 224
  nominal 1 - lam  MC 0.49929 +- 0.00065   exact 0.50000   z -1.10
  pasted fraction  MC 0.31759 +- 0.00042   exact 0.31778   z -0.44
  lam_c - lam      MC 0.18170 +- 0.00037   exact 0.18222   z -1.41
  clipped          MC 0.82671 +- 0.00085   exact 0.82730   z -0.70
  corner covered   MC 0.12537 +- 0.00074   exact 0.12649   z -1.50
  centre covered   MC 0.49238 +- 0.00112   exact 0.49405   z -1.49
  inside: pasted   MC 0.49632 +- 0.00064   exact 0.49702   z -1.10
  inside: corner   MC 0.00541 +- 0.00016   exact 0.00556   z -0.90
  inside: centre   MC 0.83754 +- 0.00082   exact 0.83795   z -0.50
  min(lam_c - lam) = 1.61e-07  (lam_c >= lam always)
continuous Eq. (2) with clipping: 77/240 = 0.32083
batch (8, 4, 224, 224) image range 0.000..1.000
pixel check: max |pixel fraction - (1 - lam_c)| = 5.6e-17; mean |pixel fraction - (1 - lam)| = 0.1838 +- 0.0073
  z vs exact 0.182222: +0.21

PILOT CRITERIA
  PASS  W=32 nominal 1 - lam within 4 SE
  PASS  W=32 pasted fraction within 4 SE
  PASS  W=32 lam_c - lam within 4 SE
  PASS  W=32 clipped within 4 SE
  PASS  W=32 corner covered within 4 SE
  PASS  W=32 centre covered within 4 SE
  PASS  W=32 inside: pasted within 4 SE
  PASS  W=32 inside: corner within 4 SE
  PASS  W=32 inside: centre within 4 SE
  PASS  W=224 nominal 1 - lam within 4 SE
  PASS  W=224 pasted fraction within 4 SE
  PASS  W=224 lam_c - lam within 4 SE
  PASS  W=224 clipped within 4 SE
  PASS  W=224 corner covered within 4 SE
  PASS  W=224 centre covered within 4 SE
  PASS  W=224 inside: pasted within 4 SE
  PASS  W=224 inside: corner within 4 SE
  PASS  W=224 inside: centre within 4 SE
  PASS  batch shape (8,4,224,224)
  PASS  images in [0,1]
  PASS  indicator in {0,1}
  PASS  corrected label exact
  PASS  uncorrected error within 4 SE of exact 0.182222

Two runs printed identical output, and every Monte Carlo value agrees with its exact value within its sampling error. The headline is the second row. Although $1 - \lambda$ averages 0.5, the rectangle actually covers only 0.32 of a $224 \times 224$ image on average, and 0.30 of a $32 \times 32$ one; the rest falls outside the image and is clipped away. Clipping is the rule, not an edge case: 83% of rectangles at 224 lose part of their extent. For a continuous image the numbers follow in closed form (our derivation). Write $r = \sqrt{1 - \lambda}$, so that $r^2$ is uniform. A centre uniform on $[0, W]$ loses on average $r^2/4$ of the rectangle’s relative width at the two ends. The mean pasted fraction is then $\mathbb{E}[(r - r^2/4)^2] = 77/240 \approx 0.321$, and the chance that a rectangle is clipped is $5/6$. The discrete values at 224 are close to both.

The pixel check shows what the correction buys. With the corrected weight, the label matches the fraction of pixels from the partner image to rounding error, $5.6 \times 10^{-17}$, in all 4,000 pasted images. With the nominal weight, the label is off by 0.18 on average. The error always has one sign: the uncorrected label gives the pasted image more weight than its pixels, and the printed minimum of $\lambda_c - \lambda$ is positive. Training without the correction would systematically overweight the smaller image.

The coverage rows show why the paper’s sampler accepts the clipping. With centres anywhere in the image, a corner pixel is pasted over about a quarter as often as the centre pixel; the continuous limits are $1/8$ and $1/2$ (our derivation). If the rectangle is kept inside the image, as Random Erasing does, its area matches the nominal value but a corner pixel is almost never covered (0.6% of rectangles at 224). In our reading, the clipped sampler spreads the pasted region over the whole image, at the cost of a label that must be corrected.

Example: mixup cannot regularize a linear least-squares model

The second example takes the smallest model for which the expected mixup loss can be computed exactly. The model is a linear map with a bias from the 64 pixels of 500 of scikit-learn’s $8 \times 8$ digit images to one-hot targets, fitted by squared error: a least-squares classifier and a small stand-in for a network. Because the loss is quadratic in $\lambda$, its expectation needs only the Beta moments $\mathbb{E}[\lambda] = 1/2$ and $\mathbb{E}[\lambda^2] = (\alpha + 1)/(2(2\alpha + 1))$, and the expectation over pairs is a finite sum over all $500^2$ ordered pairs (partners drawn independently, with replacement). The script computes this exact loss in two ways, from summary moments and from an explicit table of all pairs. It compares both with a Monte Carlo average over 200,000 sampled mixed pairs, at the least-squares weights and at random weights, for $\alpha = 0.2$ and $\alpha = 1$. It then minimizes the exact mixup objective and compares the result with ordinary least squares. The script uses NumPy 2.3.5 and scikit-learn 1.7.2 and takes about 5 to 15 seconds.

The plan for this example said that “the fitted weights show the shrinkage it adds”, and the script’s last check for each $\alpha$ tests for it. That expectation was wrong: the check failed in the pilot run, and it was kept unchanged and is reported as printed. Both scripts shown ran unchanged as the pilot and as the first run, which is why their labels say “PILOT”.

"""PILOT, example 2 as planned: mixup for a least-squares model on scikit-learn's 8x8 digits.
Exact expected mixup loss (Beta moments + all ordered pairs) vs a brute-force pair sum vs a
Monte Carlo average over sampled mixed pairs; minimizer of the exact objective vs ordinary least
squares (the planned shrinkage check D3)."""
import numpy as np
from sklearn.datasets import load_digits

X, y = load_digits(return_X_y=True)
rng = np.random.default_rng(0)
idx = rng.permutation(len(X))[:500]
n, K = 500, 10
Z = np.c_[X[idx] / 16.0, np.ones(n)]          # 64 pixels + bias
Y = np.eye(K)[y[idx]]
print(f"{n} digit images, Z {Z.shape}, rank {np.linalg.matrix_rank(Z)}, classes {np.bincount(y[idx]).tolist()}")

checks = []
def check(name, ok):
    checks.append((name, bool(ok)))


def moments(a):
    return 0.5, (a + 1) / (2 * (2 * a + 1))      # E[lam], E[lam^2] for Beta(a, a)


def exact_quadratic(a):
    """E over pairs (with replacement) and lam of z~ z~^T, z~ y~^T, ||y~||^2."""
    m1, m2 = moments(a)
    c_same = m2 + (1 - 2 * m1 + m2)              # E[lam^2] + E[(1-lam)^2]
    c_cross = 2 * (m1 - m2)                      # 2 E[lam (1-lam)]
    zb, yb = Z.mean(0), Y.mean(0)
    Gzz = c_same * Z.T @ Z / n + c_cross * np.outer(zb, zb)
    Gzy = c_same * Z.T @ Y / n + c_cross * np.outer(zb, yb)
    Gyy = c_same * (Y * Y).sum(1).mean() + c_cross * yb @ yb
    return Gzz, Gzy, Gyy


def exact_loss(Wt, a):
    Gzz, Gzy, Gyy = exact_quadratic(a)
    return np.trace(Wt.T @ Gzz @ Wt) - 2 * np.trace(Wt.T @ Gzy) + Gyy


def brute_loss(Wt, a):
    """Explicit n x n table of pair terms: E_lam ||lam r_i + (1-lam) r_j||^2."""
    m1, m2 = moments(a)
    R = Y - Z @ Wt
    sq = (R * R).sum(1)
    P = m2 * sq[:, None] + 2 * (m1 - m2) * (R @ R.T) + (1 - 2 * m1 + m2) * sq[None, :]
    return P.mean()


def mc_loss(Wt, a, M=200_000):
    lam = rng.beta(a, a, M); i = rng.integers(0, n, M); j = rng.integers(0, n, M)
    Zm = lam[:, None] * Z[i] + (1 - lam[:, None]) * Z[j]
    Ym = lam[:, None] * Y[i] + (1 - lam[:, None]) * Y[j]
    l = ((Ym - Zm @ Wt) ** 2).sum(1)
    return l.mean(), l.std(ddof=1) / np.sqrt(M)


W_ols = np.linalg.lstsq(Z, Y, rcond=None)[0]
W_rand = rng.normal(0, 0.1, W_ols.shape)
for a in (0.2, 1.0):
    print(f"alpha = {a}")
    for name, Wt in (("W_ols", W_ols), ("random W", W_rand)):
        e, b = exact_loss(Wt, a), brute_loss(Wt, a)
        m, se = mc_loss(Wt, a)
        print(f"  {name:9s} exact {e:.8f}  brute-force {b:.8f}  MC {m:.5f} +- {se:.5f}  z {(m-e)/se:+.2f}")
        check(f"alpha={a} {name}: exact == brute (rel 1e-9)", abs(e - b) <= 1e-9 * abs(b))
        check(f"alpha={a} {name}: MC within 4 SE", abs(m - e) <= 4 * se)
    Gzz, Gzy, _ = exact_quadratic(a)
    W_mix = np.linalg.lstsq(Gzz, Gzy, rcond=None)[0]
    ratio = np.linalg.norm(W_mix[:-1]) / np.linalg.norm(W_ols[:-1])
    print(f"  ||W_mix - W_ols|| / ||W_ols|| = {np.linalg.norm(W_mix - W_ols)/np.linalg.norm(W_ols):.1e}; "
          f"weight-norm ratio ||W_mix|| / ||W_ols|| = {ratio:.6f}")
    check(f"alpha={a}: D3 planned shrinkage, ratio < 1", ratio < 1 - 1e-6)

print("\nPILOT CRITERIA")
for name, ok in checks:
    print(f"  {'PASS' if ok else 'FAIL'}  {name}")

Output:

500 digit images, Z (500, 65), rank 60, classes [47, 51, 49, 57, 48, 52, 47, 52, 53, 44]
alpha = 0.2
  W_ols     exact 0.23611120  brute-force 0.23611120  MC 0.23672 +- 0.00040  z +1.54
  random W  exact 3.09545442  brute-force 3.09545442  MC 3.09743 +- 0.00221  z +0.90
  ||W_mix - W_ols|| / ||W_ols|| = 1.7e-11; weight-norm ratio ||W_mix|| / ||W_ols|| = 1.000000
alpha = 1.0
  W_ols     exact 0.18364204  brute-force 0.18364204  MC 0.18372 +- 0.00030  z +0.26
  random W  exact 2.83002706  brute-force 2.83002706  MC 2.82672 +- 0.00197  z -1.68
  ||W_mix - W_ols|| / ||W_ols|| = 1.9e-11; weight-norm ratio ||W_mix|| / ||W_ols|| = 1.000000

PILOT CRITERIA
  PASS  alpha=0.2 W_ols: exact == brute (rel 1e-9)
  PASS  alpha=0.2 W_ols: MC within 4 SE
  PASS  alpha=0.2 random W: exact == brute (rel 1e-9)
  PASS  alpha=0.2 random W: MC within 4 SE
  FAIL  alpha=0.2: D3 planned shrinkage, ratio < 1
  PASS  alpha=1.0 W_ols: exact == brute (rel 1e-9)
  PASS  alpha=1.0 W_ols: MC within 4 SE
  PASS  alpha=1.0 random W: exact == brute (rel 1e-9)
  PASS  alpha=1.0 random W: MC within 4 SE
  FAIL  alpha=1.0: D3 planned shrinkage, ratio < 1

Two runs printed identical output. The exact loss computed from moments matches the explicit pair table to all printed digits, and the Monte Carlo averages agree with it within their sampling error, at both weights and both values of $\alpha$. The shrinkage check fails: the minimizer of the exact mixup objective equals the least-squares solution to $2 \times 10^{-11}$, and its weight norm is unchanged.

The reason is a short derivation (ours). For a model linear in its input, $f(x) = W^\top x + b$, the residual of a mixed pair is the same mixture of the two residuals:

$$\tilde{y} - f(\tilde{x}) = \lambda\, r_A + (1 - \lambda)\, r_B, \qquad r_i = y_i - f(x_i).$$

Averaging its squared norm over $\lambda$ and over independent uniform partners gives

$$\mathbb{E}\,\big\lVert \tilde{y} - f(\tilde{x}) \big\rVert^2 = c_1 \cdot \frac{1}{n} \sum_i \lVert r_i - \bar{r} \rVert^2 + \lVert \bar{r} \rVert^2, \qquad c_1 = \mathbb{E}\big[\lambda^2 + (1 - \lambda)^2\big] = \frac{\alpha + 1}{2\alpha + 1},$$

where $\bar{r}$ is the mean residual. The bias can always make $\bar{r} = 0$, and then the objective is $c_1$ times the variance of the residuals, which ordinary least squares minimizes. Mixup only rescales the loss: with $c_1 = 2/3$ for $\alpha = 1$ and $6/7$ for $\alpha = 0.2$, the two exact losses at the least-squares weights stand in the ratio $(6/7)/(2/3) = 9/7 \approx 1.286$, which is the ratio of the printed 0.2361 and 0.1836. The same identity holds for any model that is linear in what is mixed. So mixup’s regularizing effect has to come from the model’s nonlinearity between the mixed input and the output, or from a loss that is not quadratic, such as the cross-entropy. This is our conclusion from the derivation, not a result of the papers.

Variants

• Feature CutMix. CutMix’s authors also applied the cut-and-paste to feature maps at a randomly chosen layer per mini-batch, as Manifold Mixup does for blending. On ImageNet it was slightly worse than CutMix at the input (Table 3). On CIFAR-100, input-level CutMix was best, and every feature level except the deepest still improved on the baseline (Figure 3).

• CutMix design ablations. On CIFAR-100 with PyramidNet-200 (Table 8, top-1 error), every variant was worse than the proposed 14.47%:

  • box centres drawn from a Gaussian around the image centre: 15.95%;
  • a fixed $16 \times 16$ box: 14.97%;
  • a one-hot label for the larger part: 15.89%;
  • a probability of applying CutMix that grows during training: 14.72%;
  • a fixed 50/50 label regardless of area: 15.17%.

  The one-hot and fixed 50/50 variants show that the area-proportional label matters.

• Combining erasing and mixing. On CIFAR-100, Cutout plus mixup (15.46%) and Cutout plus Manifold Mixup (15.09%) were better than either mixing method alone. The authors read this as evidence that mixing by cutting and pasting beats interpolation (Table 5).

• Min-max boxes. timm offers an alternative CutMix sampler that draws the box’s height and width independently between minimum and maximum fractions of the image and keeps the box inside it; the label weight then always comes from the box area.

• Object-aware erasing. Random Erasing’s detection variants erase inside object boxes, or both inside boxes and over the whole image (Zhong et al., 2020, §4.3).

Best Practices

• Start from published settings. CutMix used $\alpha = 1$ and found it the best of the values it tried on CIFAR-100 (0.1 to 4.0; Figure 3). Bag of Tricks used mixup with $\alpha = 0.2$ on ImageNet. DeiT combines mixup at 0.8 and CutMix at 1.0. Wightman et al.’s ResNet-50 procedures, as the data augmentation article tabulates them, use mixup 0.1 or 0.2 with CutMix 1.0.
• Correct CutMix’s label from the pasted area. The official code, the paper’s pseudo-code and timm all do. The example shows that without the correction the label overstates the pasted image by 0.18 on average.
• Train longer. CutMix’s comparisons used 300 epochs because the regularized methods needed more epochs to converge (§4.1.1), and Bag of Tricks raised mixup training from 120 to 200 epochs. In Bag of Tricks, training 200 epochs without mixup gained only about 0.15% top-1, which the authors take to show that mixup’s effect is still significant.
• Use soft-target losses. Mixed labels are distributions. Either mix the losses, as the official CutMix code does, or use a cross-entropy that accepts soft targets.
• Report the protocol. The strength of the effect depends on $\alpha$, on how often mixing is applied and on the schedule, so results should state them.

Pitfalls

• Too strong mixing underfits. In Manifold Mixup’s TinyImageNet table (PreActResNet18), input mixup at $\alpha = 0.2$ raised top-1 accuracy from 55.52% to 56.47%, but $\alpha = 1.0$ lowered it to 52.65% and $\alpha = 2.0$ to 44.18% (Table 3). The regularization article cites the mixup paper for the same effect on ImageNet.
• Mixup can hurt localization and detection transfer. In CutMix’s experiments with ResNet-50, mixup lowered weakly supervised localization accuracy and the Faster R-CNN mAP of a backbone fine-tuned on PASCAL VOC, while CutMix raised both (Evidence and Impact). The CutMix authors’ explanation is that ambiguous blended images make the classifier focus on even more discriminative parts of objects (§4.2).
• Dense prediction. He, T. et al. (2019, Table 9) found that adding mixup to an ImageNet-pretrained backbone lowered the ADE20K mIoU of an FCN from 38.75 (with label smoothing) to 37.99, while the same refinement raised ImageNet accuracy. They suggest that soft labels and mixing blur pixel-level information.
• Better accuracy is not better calibration. CutMix’s out-of-distribution table (below) reports that mixup and Cutout made the CIFAR-100 classifier’s over-confidence worse. In Manifold Mixup’s tables, input mixup increased the test negative log-likelihood on SVHN for all three networks and on CIFAR-10 for Wide-ResNet-28-10, although it lowered the error.
• Uncorrected CutMix labels. Code that pastes a clipped rectangle but keeps $\lambda$ as sampled trains on labels that are wrong by 0.18 on average in one direction (example above).
• Expecting mixup to regularize a linear model. For a model linear in the mixed quantity with a squared loss, mixup only rescales the objective (example above).

Evidence and Impact

ImageNet classification. In CutMix’s comparison (Yun et al., 2019, Table 3; ResNet-50, 300 epochs, top-1 validation error, best value during training), the baseline’s 23.68% fell to 22.93% with Cutout, 22.58% with mixup, 22.50% with Manifold Mixup, 21.80% with Feature CutMix and 21.40% with CutMix. The text gives Feature CutMix as 21.78%, against the table’s 21.80%. With ResNet-101 the errors were 21.87% (baseline), 20.72% (Cutout), 20.52% (mixup) and 20.17% (CutMix) (Table 4). In Bag of Tricks (He, T. et al., 2019, Table 6; mean of four runs for ResNet-50-D), adding mixup with $\alpha = 0.2$ after label smoothing raised top-1 accuracy from 78.37% to 79.01% for ResNet-50-D, from 78.40% to 78.77% for Inception-V3 and from 72.93% to 73.28% for MobileNet.

CIFAR. On CIFAR-100 with PyramidNet-200 (Table 5, mean of three runs, best during training), top-1 error was:

• 16.45% for the baseline;
• 16.53% with Cutout and 16.73% with label smoothing;
• 15.61% with Cutout and label smoothing together;
• 15.63% with mixup at $\alpha = 1$;
• 16.14% with Manifold Mixup;
• 14.47% with CutMix.

The CutMix README lists 14.23% for its released CutMix model, against the table’s 14.47%. The README does not say whether that model is one of the three runs averaged in the table. Manifold Mixup’s own experiments (Verma et al., 2019, Table 1; five runs each) found it better than input mixup on both datasets. On CIFAR-10 with PreActResNet18, test error was 4.83% without mixing, 3.82% with input mixup ($\alpha = 1$) and 2.95% with Manifold Mixup ($\alpha = 2$), with test NLL 0.190, 0.186 and 0.137. Random Erasing lowered CIFAR-10 error for every architecture it tried, for example from 3.80% to 3.08% for WRN-28-10, and ImageNet top-1 error for ResNet-50 from 23.39% to 22.75% (Zhong et al., 2020, Tables 1 and 2).

Transformers. DeiT’s ablation (Touvron et al., 2021, Table 7; DeiT-B, top-1 on ImageNet at $224 \times 224$, then fine-tuned at 384) shows how much the vision transformer recipe depends on mixing:

• the full recipe reached 81.8% (83.1% fine-tuned);
• without mixup, 78.7% (79.8%);
• without CutMix, 80.0% (80.6%);
• without both, 75.8% (76.7%).

Without random erasing the model did not train well (4.3%), which the authors suggest may be because the hyper-parameters were not adapted. The Vision Transformer article places DeiT in the history of training ViTs on ImageNet alone.

Localization and transfer. CutMix’s weakly supervised localization results (Table 9) use class activation maps thresholded at 15% of their maximum and count a hit when the box overlaps the ground truth with IoU above 0.5 and the class is correct (supplement, Appendix B). With ResNet-50, localization accuracy on CUB-200-2011 was 49.41% for the baseline, 49.30% with mixup, 52.78% with Cutout and 54.81% with CutMix; on ImageNet it was 46.30%, 45.84%, 46.69% and 47.25%. Used as pretrained backbones (Table 10, PASCAL VOC 2007 test mAP after fine-tuning on VOC 2007+2012 trainval), the CutMix model raised SSD from 76.7 to 77.6 and Faster R-CNN from 75.6 to 76.7. Mixup gave 76.6 and 73.9, and Cutout 76.8 and 75.0. The paper’s overview (Table 1), captioned as SSD fine-tuning, lists the Faster R-CNN values instead, with Cutout at 75.1. On MS-COCO image captioning with a Show-and-Tell-style model, BLEU-4 went from 22.9 to 23.2 (mixup), 24.0 (Cutout) and 24.9 (CutMix). Random Erasing improved Fast R-CNN trained on VOC 2007 trainval from 69.1 to 71.5 mAP on VOC 2007 test in its image-and-object-aware form (Zhong et al., 2020, Table 6). It also improved person re-identification baselines (Table 7), as the person re-identification article reports for a later strong baseline.

Robustness and uncertainty. In CutMix’s experiments on the ImageNet validation set (ResNet-50), top-1 accuracy after a white-box FGSM attack was 8.2% for the baseline, 24.4% with mixup, 11.5% with Cutout and 31.0% with CutMix (Table 11). Under occlusion by a central hole or a frame, Cutout and CutMix were much more robust than the baseline and mixup only slightly (Figure 5). For detecting out-of-distribution images by thresholding the classifier’s confidence (CIFAR-100, PyramidNet-200, averaged over seven out-of-distribution sets; Table 12), AUROC was 87.3% for the baseline, 49.3% with mixup, 68.7% with Cutout and 94.4% with CutMix. The authors conclude that mixup and Cutout aggravate over-confidence, while CutMix alleviates it. Calibration in the sense of expected calibration error is reported differently: that article summarizes Thulasidasan et al. (2019), who found mixup-trained networks better calibrated, found no such benefit from mixing images while keeping hard labels, and found large mixing parameters made networks underconfident. Random Erasing’s model had a 56.36% error on CIFAR-10 test images with half their area occluded, against 75.04% for the baseline (§5.2).

Recipes. Mixing became part of the standard ImageNet recipe after these papers. Mixup appears among Bag of Tricks’ refinements, DeiT uses mixup, CutMix and Random Erasing together, and the ResNet and data augmentation articles describe how Wightman et al.’s ResNet-50 procedures combine mixup and CutMix with RandAugment. The regularization article places these methods among the other regularizers, and Label Smoothing covers the other common soft-target change, which DeiT applies to the mixed targets.