Data Augmentation

Data augmentation trains a network on randomly transformed copies of its training images: a crop instead of the whole photograph, a mirror image, a darker or bluer version, a version with a patch erased. Each pass over the data sees different copies, so the network meets far more variation than the stored images contain. The transformations must not change the correct answer. A flipped photograph of a cat is still a cat, but a flipped arrow points the other way. The regularization article places augmentation among the other regularizers.

Purpose

A classifier should give the same answer when an object moves, changes size, or is lit differently. Such properties are invariances. A convolutional network gets some of them from its architecture: convolution is translation-equivariant and pooling adds a little tolerance to shifts. Most of the rest, such as tolerance to scale, viewpoint, colour, occlusion and deformation, must be learned from examples, and a training set of a few thousand images rarely shows enough of them.

Augmentation supplies those examples. Simard, Steinkraus and Platt (2003) put the argument plainly: generating transformed data is simple, while building invariance to the transformation into a model can be arbitrarily hard, so they let the learning algorithm infer the invariance from transformed examples. The invariance is then stored in the weights, and recognition costs nothing extra. Cubuk et al. (2019) make the same point for modern networks: encoding a candidate invariance through augmentation can be easier than hardcoding it into the architecture.

When to Use It

Augmentation is part of nearly every image-training pipeline. It matters most when labelled images are few relative to the network’s capacity, as in medical imaging, inspection, or fine-tuning on a small dataset (see transfer learning). It also helps when the test images will vary in ways the training images do not. It helps less, or hurts, when a transformation changes what the label means (orientation, colour or position carrying the class), when the training set already covers the variation, or, in our reading of the recipes under Evidence and Impact, when training is short.

How It Works

Let $\mathcal{T}$ be a distribution over transformations $t$ (a random crop, flip, colour change, or a composition of these). Training minimizes the expected loss over transformed images:

$$J(\theta) = \frac{1}{N}\sum_{i=1}^{N} \mathbb{E}_{t \sim \mathcal{T}}\; \ell\big(f_\theta(t(\mathbf{x}_i)),\, y_i\big).$$

In practice the expectation is estimated by drawing a fresh $t$ for every image each time it enters a mini-batch, so no augmented copy is stored. The training images keep their label $y_i$, which is correct only if $t$ is label-preserving: the true class of $t(\mathbf{x})$ equals the class of $\mathbf{x}$ for every $t$ that $\mathcal{T}$ can produce. For dense tasks the label is itself an image or a set of coordinates, and geometric transformations must be applied to it as well: the pair becomes $(t(\mathbf{x}), t(\mathbf{y}))$.

Geometric transformations

• Flips. A horizontal flip mirrors the image. It is the default for natural photographs; in our reading, these have no preferred left or right but do have a preferred up, which is why vertical flips are rarely used for them.
• Crops and translations. Taking a random crop, or padding the image and cropping it back to its size, shifts the content, as in AlexNet’s training.
• Scale. Rescaling before cropping changes the apparent object size. VGG drew the training scale at random for each image, which its authors called scale jittering.
• Random resized crop. A crop with random area and aspect ratio, resized to the network’s input size, combines translation, scale and stretch. It comes from GoogLeNet’s training (see Inception) and is TorchVision’s RandomResizedCrop. In version 0.29.0, the crop’s area is drawn uniformly from 8% to 100% of the image area and its aspect ratio log-uniformly from 3/4 to 4/3; after ten attempts that do not fit inside the image, it falls back to a central crop.
• Rotation, shear and perspective. Affine and projective warps. Like every geometric augmentation, they resample the image through backward mapping and interpolation (see image warping). Downscaling without low-pass filtering aliases (see sampling and aliasing); TorchVision’s RandomResizedCrop antialiases tensor images by default since version 0.17.
• Elastic deformation. Simard et al. (2003) drew a random displacement for every pixel, smoothed the displacement field with a Gaussian of standard deviation $\sigma$ (which they called the elasticity coefficient) and scaled it by a factor $\alpha$, to mimic the oscillations of a writing hand. Ronneberger et al. relied on the same idea to train U-Net from a few dozen microscopy images. TorchVision’s ElasticTransform (defaults $\alpha = 50$, $\sigma = 5$) notes that its transformation of bounding boxes is only approximate.

Photometric transformations

• Colour jitter. TorchVision’s ColorJitter draws brightness, contrast and saturation factors uniformly from $[\max(0, 1-b),\, 1+b]$ for a setting $b$, and a hue shift from $[-h, h]$, and applies the four adjustments in a random order. The hue shift goes through HSV (see colour spaces), so its documentation requires non-negative pixel values: jitter before normalization, not after.
• PCA lighting. AlexNet added random multiples of the principal components of the training set’s RGB values, to imitate changes in the colour and intensity of illumination; the paper article gives the details and its effect.
• Grayscale, noise and blur. Random conversion to grayscale removes colour cues. Additive noise (see image noise) and Gaussian blur imitate sensor noise and defocus.

Photometric transformations change pixel values but not positions, so masks, boxes and keypoints stay as they are.

Labels that constrain or transform

Whether a transformation preserves the label depends on the task, not on the transformation. Flipped or rotated characters change class, as the regularization article notes. In our reasoning, the same holds whenever the class is defined by orientation or handedness (arrows, traffic signs for left and right turns, text, which side of the body a medical image shows) or by colour (ripe and unripe fruit, some traffic lights), while overhead and microscope images have no preferred orientation, so all rotations and both flips are usually safe there.

For detection and segmentation, the geometric transformation must be applied to the annotations too. A horizontal flip of an image of width $W$ maps a box with corners $x_1 < x_2$ to $x_1' = W - x_2$ and $x_2' = W - x_1$, and mirrors masks; TorchVision’s detection reference code does exactly this. For human keypoints a flip must also swap the left and right labels, otherwise a left wrist becomes a right wrist: the same code reorders the 17 COCO keypoints with a fixed permutation before mirroring their $x$ coordinates. Crops can cut objects in half or remove them entirely, and TorchVision’s documentation of RandomIoUCrop requires a SanitizeBoundingBoxes step afterwards to remove the boxes that fall below its overlap threshold.

How to Apply It

TorchVision’s torchvision.transforms.v2 API, introduced in version 0.15 (March 2023), transforms images, videos, bounding boxes, masks and keypoints together, so one pipeline serves classification and detection. A typical ImageNet-style classification pipeline, following the version 0.29.0 API (not run here, since TorchVision is not installed in our environment):

import torch
from torchvision.transforms import v2

mean, std = [0.485, 0.456, 0.406], [0.229, 0.224, 0.225]
train_tf = v2.Compose([
    v2.ToImage(),                     # PIL image -> uint8 tensor image
    v2.RandomResizedCrop(224),        # area 8-100%, aspect ratio 3/4-4/3
    v2.RandomHorizontalFlip(),        # p = 0.5
    v2.RandAugment(),                 # num_ops=2, magnitude=9 of 31 bins; expects uint8
    v2.ToDtype(torch.float32, scale=True),
    v2.Normalize(mean, std),
    v2.RandomErasing(p=0.25),         # after normalization, as in its docstring example
])
test_tf = v2.Compose([                # no randomness at test time
    v2.ToImage(), v2.Resize(256), v2.CenterCrop(224),
    v2.ToDtype(torch.float32, scale=True),
    v2.Normalize(mean, std),
])

MixUp and CutMix act on whole batches of images and labels (class indices or class probabilities), so they are applied after the data loader, not in the per-image pipeline.

Example: an augmentation that preserves the label and one that destroys it

The script below makes 24×24 grayscale images of an arrow pointing left (class 0) or right (class 1), with random length, head size, position, brightness and noise. Because each arrow is symmetric about its horizontal axis, a vertical flip leaves its direction unchanged; so do small shifts and brightness or contrast changes. A horizontal flip reverses the direction while keeping the label, so it turns a right-pointing training example into a left-pointing image labelled “right”. A small CNN trains on only 40 images, under three settings: no augmentation, the label-preserving set (shifts of up to 3 pixels, contrast and brightness jitter, random vertical flips), and the same set plus random horizontal flips. Test accuracy is measured on 2,000 new images, for five seeds. The script uses PyTorch 2.14.1 on the CPU with one thread and takes about 16 seconds.

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.set_num_threads(1)
S = 24  # image size

def arrows(n, rng):
    """n grayscale S x S arrows; label 0 points left, 1 points right."""
    X = np.zeros((n, 1, S, S), np.float32)
    y = rng.integers(0, 2, n)
    for i in range(n):
        L = rng.integers(9, 15)                    # total length
        h = rng.integers(3, 5)                     # half-height of the head
        cy = rng.integers(h + 1, S - h - 1)
        x0 = rng.integers(1, S - L - 1)            # leftmost column
        img = np.zeros((S, S), np.float32)
        img[cy, x0:x0 + L] = 1.0                   # shaft
        for d in range(h + 1):                     # head: a triangle narrowing to the tip
            col = x0 + L - 1 - h + d if y[i] else x0 + h - d
            img[cy - (h - d):cy + (h - d) + 1, col] = 1.0
        fg, bg = rng.uniform(0.5, 1.0), rng.uniform(0.0, 0.3)
        X[i, 0] = bg + (fg - bg) * img + rng.normal(0, 0.05, (S, S))
    return torch.tensor(X), torch.tensor(y)

def shift(x, gen, m=3):
    """Random translation by up to m pixels: pad with the border, crop at a random offset."""
    xp = F.pad(x, (m, m, m, m), mode="replicate")
    out = torch.empty_like(x)
    off = torch.randint(0, 2 * m + 1, (len(x), 2), generator=gen)
    for i, (a, b) in enumerate(off.tolist()):
        out[i] = xp[i, :, a:a + S, b:b + S]
    return out

def photometric(x, gen):
    """Random contrast and brightness per image (a ColorJitter-like change)."""
    n = len(x)
    c = 0.6 + 0.8 * torch.rand(n, 1, 1, 1, generator=gen)
    b = 0.3 * (torch.rand(n, 1, 1, 1, generator=gen) - 0.5)
    m = x.mean((2, 3), keepdim=True)
    return (x - m) * c + m + b

def flip(x, gen, dim):
    """Flip each image along dim (-2 vertical, -1 horizontal) with probability 0.5."""
    sel = torch.rand(len(x), generator=gen) < 0.5
    x = x.clone()
    x[sel] = x[sel].flip(dim)
    return x

def augment(x, mode, gen):
    if mode == "none":
        return x
    x = flip(photometric(shift(x, gen), gen), gen, -2)   # label-preserving
    if mode == "+ horizontal flip":
        x = flip(x, gen, -1)                            # label-destroying
    return x

def run(seed, mode, n_train=40, steps=400):
    rng = np.random.default_rng(seed)
    Xtr, ytr = arrows(n_train, rng)
    Xte, yte = arrows(2000, rng)
    torch.manual_seed(seed)
    gen = torch.Generator().manual_seed(seed)
    net = nn.Sequential(
        nn.Conv2d(1, 8, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
        nn.Conv2d(8, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
        nn.Flatten(), nn.Linear(16 * 6 * 6, 2))
    opt = torch.optim.Adam(net.parameters(), lr=3e-3)
    for _ in range(steps):
        loss = F.cross_entropy(net(augment(Xtr, mode, gen)), ytr)
        opt.zero_grad(); loss.backward(); opt.step()
    with torch.no_grad():
        return (net(Xte).argmax(1) == yte).float().mean().item()

modes = ["none", "label-preserving", "+ horizontal flip"]
print(f"PyTorch {torch.__version__}; 40 training images, 2,000 test images")
print("test accuracy\nseed " + "".join(f"{m:>20}" for m in modes))
for seed in range(5):
    print(f"{seed:4d} " + "".join(f"{run(seed, m):20.3f}" for m in modes))

Output:

PyTorch 2.14.1; 40 training images, 2,000 test images
test accuracy
seed                 none    label-preserving   + horizontal flip
   0                0.831               0.996               0.512
   1                0.691               0.992               0.535
   2                0.796               1.000               0.512
   3                0.799               0.990               0.498
   4                0.823               0.997               0.522

Two runs printed identical output. Over the five seeds, the network trained without augmentation reached 69–83% test accuracy: in our reading, 40 images do not show the arrow at enough positions and brightness levels, and the flattened final layer ties features to positions. The label-preserving augmentations raised accuracy to 99.0–100% with the same 40 images. Adding horizontal flips brought it down to 50–54%, near chance: every training arrow now appears pointing both ways under one label, so direction no longer predicts the class, and whatever the network memorizes about individual training images says nothing about new arrows. The network, step count and augmentation ranges were chosen by hand for illustration.

Variants

Erasing and occlusion

Cutout (DeVries & Taylor, 2017) zeroes a fixed-size square at a random position in each training image; the authors found the size of the region mattered more than its shape. Random Erasing (Zhong et al., 2020) replaces a rectangle of random area and aspect ratio with random pixel values, applied with probability $p$; their base setting used $p = 0.5$, areas of 2% to 40% of the image and aspect ratios from 0.3 to 1/0.3. For detection they also erased regions inside each object’s box. Both make the network rely on more of the object than its most distinctive part. TorchVision’s RandomErasing defaults to $p = 0.5$, areas from 2% to 33%, aspect ratios from 0.3 to 3.3, and fills with zeros unless value="random" is given.

Mixing images

mixup (Zhang et al., 2018) and CutMix (Yun et al., 2019), defined in the regularization article, combine two training images and mix their labels. Neither preserves the label in the sense above; both replace it with a soft target. A dedicated article on mixing methods is planned.

Learned and random policies

AutoAugment (Cubuk et al., 2019) searched for augmentation policies with a reinforcement-learning controller. A policy had five sub-policies, each two operations from a list of 16 (shears, translations, rotation, colour and contrast operations, inversion, Cutout and others) with a probability and a magnitude; for each image one sub-policy was drawn at random. Each search trained small child networks on a reduced dataset (4,000 CIFAR-10 images, or 6,000 ImageNet images from 120 classes) and sampled about 15,000 policies, which the authors estimated at 5,000 GPU hours for CIFAR-10 and 15,000 for ImageNet. The policies differed by dataset: mostly colour operations on CIFAR-10, where inversion was almost never chosen, but inversion, shears and rotation on the house numbers of SVHN, where the authors note that the colour of the digits matters less than its contrast with the background.

RandAugment (Cubuk et al., 2020) dropped the search. It applies $N$ operations, each drawn uniformly from 14, all at one shared magnitude $M$, so only two numbers need tuning, by a small grid search on the target task itself. TorchVision’s RandAugment defaults to $N = 2$ and magnitude index 9 out of 31 bins. TrivialAugment (Müller & Hutter, 2021) has no tuned parameters at all: it applies a single operation per image, with the operation and its strength both drawn uniformly at random. TorchVision implements it as TrivialAugmentWide, with wider magnitude ranges (rotations up to 135°, for example).

Test-time augmentation

Augmented views can also be averaged at test time. AlexNet, VGG and ResNet all reported results averaged over crops, scales or flips; the AlexNet paper, VGG and ResNet articles describe the protocols. Each view costs a forward pass, so ten views cost ten times the computation. In our reasoning, for segmentation or detection the prediction for a flipped or rescaled view must be mapped back to the original geometry before averaging. Because the protocol changes the reported accuracy, results must state it.

Augmentation as the training signal

In self-supervised and contrastive learning, two augmented views of one unlabelled image form a positive pair, and the network is trained to give them similar embeddings. Augmentation then does more than regularize: it defines what the representation ignores. Those articles cover SimCLR’s augmentation pipeline and the ablations showing which transformations matter.

Best Practices

• Start from the invariances of the task: list what can vary in test images without changing the label, and augment exactly those. For natural-image classification the usual base is random resized crops with horizontal flips, which Wightman et al. (2021) describe as typically the base of all augmentations in the timm library.
• Apply augmentation to training batches only. Validation and test images go through a fixed resize and centre crop, so measurements are comparable across runs.
• Check the transformed data by looking at it, including the boxes, masks and keypoints drawn on the transformed images. A transformation that silently desynchronizes images and labels is easy to miss in the loss curve.
• Tune augmentation strength together with model size and training length, on a validation set from the target task (see Evidence and Impact).

Pitfalls

• Destroying the label. A transformation that changes the class turns augmentation into label noise, as horizontal flips did in the example above.
• Train–test mismatch. Random resized crops show objects larger, on average, than the centre crop used at test time. Touvron et al. (2019) estimated that under standard pre-processing an object’s apparent size at test time is on average about 0.80 of its size during training. Their ResNet-50 trained at 224×224 (with repeated augmentation) reached 77.0% top-1 accuracy on the ImageNet validation set when tested at 224 and 78.4% at 288, the best of the test sizes they listed, with no change to the weights; fine-tuning the classifier and the last batch normalization layer at the test resolution raised the best result to 79%, at 384.
• Magnitudes that do not transfer between implementations. “Magnitude 9” has different meanings in different code. In TorchVision’s RandAugment it indexes 31 bins; for rotation these span 0–30°, so the default rotates by 9° (our arithmetic). The RandAugment paper uses AutoAugment’s integer scale, on which 10 is an operation’s maximum, and the appendix of AutoAugment’s arXiv version (v3, Table 6) gives rotations up to 30°, so there magnitude 9 would mean 27° (our arithmetic). Wightman et al. point out that in the original implementation increasing $M$ did not strengthen every operation: for sharpness, magnitudes 0, 5 and 10 gave strong blurring, no change and strong sharpening. Their timm version adds an “increasing” mode to fix this.
• Assuming a gain transfers between tasks. Yun et al. (2019, Table 1) reported that mixup raised ResNet-50’s ImageNet classification accuracy but lowered weakly supervised localization accuracy and the PASCAL VOC detection mAP of a detector fine-tuned from it. Cubuk et al. (2020) wrote, citing other work, that Cutout improves CIFAR and SVHN but not ImageNet; CutMix’s authors, however, list a small ImageNet gain for Cutout in their own 300-epoch training (Evidence and Impact).

Evidence and Impact

Early results. On MNIST (training on the first 50,000 training images, test error on the 10,000 test images), Simard et al. (2003) reported 1.6% error for a two-layer fully connected network with no distortion, 1.1% with affine distortions and 0.7% with elastic ones; their simple convolutional network reached 0.6% with affine and 0.4% with elastic distortions, which they believed the best published MNIST result at the time. LeCun et al.’s earlier affine distortions for LeNet-5 are covered in LeNet.

Automated and mixing methods on ImageNet. AutoAugment raised ResNet-50 from 76.3% to 77.6% top-1 validation accuracy (Cubuk et al., 2019, Table 3; the baseline, with Inception-style pre-processing, was taken from another paper). RandAugment matched this on ResNet-50 and reported larger gains on larger networks, 84.7% against 83.7% for the baseline and 84.3% for AutoAugment on EfficientNet-B7 (Cubuk et al., 2020, Table 3). In CutMix’s comparison (Yun et al., 2019, Table 3), in the authors’ own runs of 300 epochs, ResNet-50’s top-1 validation error fell from 23.68% to 22.93% with Cutout, 22.58% with mixup and 21.40% with CutMix; the authors report the best value reached during training.

Strength depends on model size and data. Cubuk et al. (2020, Figure 1) trained Wide ResNets on CIFAR-10 with one tied magnitude for all operations, averaging 20 runs per point. The best magnitude grew monotonically with network width and with training set size, while smaller training sets gained more from augmentation (3.0% against 1.5% in their example). On CIFAR-100, the best setting was $N = 1$, $M = 2$ for Wide-ResNet-28-2 and $N = 2$, $M = 14$ for the wider Wide-ResNet-28-10. The authors concluded that policies tuned on a small proxy task, as in AutoAugment, are suboptimal, because the proxy’s model and data are smaller than the target’s. For vision transformers, Steiner et al. (2022) studied the trade-off between augmentation, regularization and data (see Vision Transformer).

Recipes. Wightman et al. (2021, Table 2) tabulated the augmentations of several ResNet-50 training procedures. In their own procedures, the shortest (A3) also used slightly weaker augmentation:

Procedure	Epochs	Augmentation
Original ResNet	90	flips, PCA lighting
PyTorch	90	random resized crop, flips, colour jitter
DeiT	300	random resized crop, flips, RandAugment 9/0.5, mixup 0.8, CutMix 1.0, erasing 0.25, repeated augmentation
A1 (Wightman et al.)	600	random resized crop, flips, RandAugment 7/0.5, mixup 0.2, CutMix 1.0, repeated augmentation
A2 (Wightman et al.)	300	as A1 with mixup 0.1
A3 (Wightman et al.)	100	as A2 with RandAugment 6/0.5, no repeated augmentation, training at 160×160

The table does not define its RandAugment notation; in our reading, each entry gives the magnitude and the standard deviation of the noise that timm adds to it (the MSTD parameter of the paper’s Appendix A). Mixup and CutMix entries give the parameter $\alpha$ of the Beta distribution for the mixing weight, and erasing the probability. The table has no row for the original ResNet’s scale augmentation. The accuracies of these procedures are compared in ResNet.