Transfer Learning

Transfer learning reuses what a model learned on a source task, usually with a large image collection, to solve a target task with less data. In vision it takes two stages: pretraining a network on the source, then adapting it, either by training a new classifier on its frozen features (a linear probe) or by continuing to train some or all of its weights (fine-tuning). A network pretrained on ImageNet long served as the starting point for image classification, object detection and semantic segmentation, a paradigm He, Girshick and Dollár (2019) call de facto in computer vision.

Purpose

A convolutional neural network has millions of parameters, and training one from random weights on a few hundred labelled images overfits. Zeiler and Fergus (2014) retrained only the softmax classifier of their ImageNet-trained network on Caltech-101, with 30 images per class, and reached 86.5% mean per-class accuracy; the same architecture trained from scratch on those images reached 46.5%. On Caltech-256, six images per class with the pretrained features beat the best earlier method, which used ten times as many. Transfer saves labels, compute and the effort of designing features for each new problem.

When to Use It

Transfer helps most when the target set is small and the target images resemble the source images. It helps less, or not at all, when:

• the target set is large: Mask R-CNN trained on COCO from random weights can match ImageNet initialization, given longer training (He et al., 2019);
• the domain differs, as for retinal photographs and chest X-rays (Raghu et al., 2019);
• the target classes are finer than the source’s, as for car models and aircraft variants (Kornblith et al., 2019).

When transfer makes the target model worse than training from scratch, the effect is called negative transfer.

How It Works

Features go from general to specific. Yosinski et al. (2014) split the ImageNet classes into two random halves A and B of about 645,000 images each and trained an eight-layer network on each. They then copied the first $n$ layers of the A network into a new network, froze them, and trained the rest on B. Layers 1 and 2 transferred almost perfectly; accuracy dropped slightly at layer 3 and more at layers 4 to 7.

Two causes of the drop. In a control that copied the first $n$ layers from the B network itself, accuracy still fell, most for $n = 4$ and $5$. The authors attributed this to fragile co-adaptation: neighboring middle layers had learned to work together in a way the upper layers alone could not relearn. Lost co-adaptation dominated the drop at layers 3 to 5 and feature specificity at layers 6 and 7; fine-tuning all layers removed the co-adaptation drop. Between man-made and natural halves of ImageNet, features transferred worse than between random halves but still beat random weights.

Pretraining as initialization. Networks initialized with transferred layers and then fine-tuned on B generalized better than networks trained on B alone, by 1.6% top-1 accuracy on average over transfer depths 1 to 7, an effect that persisted through 450,000 fine-tuning iterations. Pretrained weights thus serve both as features and as an initialization that acts as a prior (see regularization).

Formally, a network $f = h_w \circ \phi_\theta$ is a backbone $\phi_\theta$ followed by a head $h_w$. Pretraining finds backbone weights $\theta_S$ on the source data. On the target data $\{(x_i, y_i)\}_{i=1}^{n}$ with loss $\ell$, a linear probe solves

$$\min_{w} \; \frac{1}{n}\sum_{i=1}^{n} \ell\big(h_w(\phi_{\theta_S}(x_i)),\, y_i\big),$$

a convex problem when $h_w$ is linear and $\ell$ is the cross-entropy loss. Fine-tuning minimizes the same loss over $(\theta, w)$ jointly, starting from $\theta = \theta_S$ and a fresh head. The loss is not convex in $\theta$, so where gradient descent ends depends on where it starts, and a small learning rate and few steps keep it near $\theta_S$ (our reasoning).

How to Apply It

Four adaptation strategies, from fewest to most trainable parameters:

1. Feature extraction (linear probe). Freeze the backbone, compute features once, and train a linear classifier on them. With a linear SVM on the first fully connected layer of an ImageNet network (DeCAF$_6$, with dropout), Donahue et al. (2014) reported 86.9% on Caltech-101 with 30 images per class, 2.6 points above the previous best, which combined five hand-designed features. Razavian et al. (2014) reported that linear SVMs on 4,096-dimensional OverFeat features matched or beat highly tuned methods across object, scene, fine-grained and attribute recognition.
2. Fine-tuning the last layers. Replace the head, freeze the early layers, which Yosinski et al. found most general, and train the rest. He et al. (2019) note that it is common practice to freeze the convolutional filters of the first few layers when fine-tuning.
3. Full fine-tuning. Train every layer, with a learning rate below the pretraining rate. R-CNN (Girshick et al., 2014) fine-tuned its ImageNet network on PASCAL VOC region proposals at one tenth of the pretraining rate, raising VOC 2007 detection mAP by 8.0 points to 54.2%.
4. Discriminative learning rates. Give lower layers smaller rates than upper ones. ULMFiT, a method for language models, divided the rate by 2.6 per layer going down and unfroze layers gradually from the top (Howard & Ruder, 2018).

The PyTorch example below (PyTorch 2.14.1, about 30 seconds on a CPU) pretrains a small CNN on 4,000 noisy $32 \times 32$ images of eight shapes at random positions, sizes and contrasts. The target task has three shapes the network never saw (L, T and H), with 5 or 20 labelled images per class. It compares training from scratch, a linear probe, and full fine-tuning at a third of the from-scratch learning rate, each for 300 Adam steps, over five seeds.

import copy
import numpy as np
import torch
from torch import nn
F = nn.functional
torch.set_num_threads(1)                       # single thread: identical output on every run

def shape_mask(kind, s, rng):
    """32x32 binary mask of a shape of size s pixels at a random position."""
    yy, xx = np.mgrid[:32, :32]
    cy, cx = rng.uniform(s / 2 + 1, 31 - s / 2, size=2)
    dy, dx, r, t = yy - cy, xx - cx, s / 2, max(2.0, s / 6)       # t: stroke width
    ady, adx, side = np.abs(dy), np.abs(dx), np.maximum(np.abs(dy), np.abs(dx))
    m = {
        # source task: 8 shapes
        "disk":     dy**2 + dx**2 <= r**2,
        "square":   side <= r,
        "triangle": (ady <= r) & (adx <= (dy + r) / 2),
        "ring":     np.abs(np.hypot(dy, dx) - r + t / 2) <= t / 2,
        "frame":    (side <= r) & (side >= r - t),
        "plus":     ((ady <= t / 2) & (adx <= r)) | ((adx <= t / 2) & (ady <= r)),
        "x-cross":  ((np.abs(dy - dx) <= 0.7 * t) | (np.abs(dy + dx) <= 0.7 * t)) & (side <= r),
        "diamond":  ady + adx <= r,
        # target task: 3 shapes never seen in pretraining
        "L":        ((adx <= r) & (dy >= r - t) & (dy <= r)) | ((ady <= r) & (dx >= -r) & (dx <= -r + t)),
        "T":        ((adx <= r) & (dy >= -r) & (dy <= -r + t)) | ((adx <= t / 2) & (ady <= r)),
        "H":        ((ady <= r) & (np.abs(adx - r + t / 2) <= t / 2)) | ((adx <= r) & (ady <= t / 2)),
    }[kind]
    return m.astype(np.float32)

def make_images(classes, n_per_class, seed):
    rng, X, y = np.random.default_rng(seed), [], []
    for label, kind in enumerate(classes):
        for _ in range(n_per_class):
            m = shape_mask(kind, rng.uniform(12, 22), rng)
            fg, bg = rng.uniform(0.5, 1.0), rng.uniform(0.0, 0.3)        # random contrast
            X.append(bg + (fg - bg) * m + rng.normal(0, 0.25, m.shape))   # pixel noise
            y.append(label)
    return torch.tensor(np.array(X), dtype=torch.float32).unsqueeze(1), torch.tensor(y)

class SmallCNN(nn.Module):
    def __init__(self, n_classes):
        super().__init__()
        self.features = nn.Sequential(                # 1x32x32 image -> 64-d feature vector
            nn.Conv2d(1, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
            nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
            nn.Conv2d(32, 64, 3, padding=1), nn.ReLU(), nn.AdaptiveAvgPool2d(1), nn.Flatten())
        self.head = nn.Linear(64, n_classes)
    def forward(self, x):
        return self.head(self.features(x))

def fit(model, params, X, y, lr, steps, seed, batch=16):
    """Adam for a fixed number of steps on shuffled mini-batches."""
    g, opt = torch.Generator().manual_seed(seed), torch.optim.Adam(params, lr=lr)
    batches = []
    for _ in range(steps):
        if not batches:                                    # new epoch: reshuffle
            batches = list(torch.randperm(len(y), generator=g).split(batch))
        idx = batches.pop(0)
        loss = F.cross_entropy(model(X[idx]), y[idx])
        opt.zero_grad(); loss.backward(); opt.step()

def accuracy(model, X, y):
    with torch.no_grad():
        return (model(X).argmax(1) == y).float().mean().item()

SOURCE = ["disk", "square", "triangle", "ring", "frame", "plus", "x-cross", "diamond"]
TARGET = ["L", "T", "H"]

print("torch", torch.__version__)

# 1. Pretraining: 8 shape classes, 500 images each, 8 epochs.
torch.manual_seed(0)
Xs, ys = make_images(SOURCE, 500, seed=0)
pre = SmallCNN(8)
fit(pre, pre.parameters(), Xs, ys, lr=3e-3, steps=504, seed=0, batch=64)   # 8 epochs
print(f"source task: {len(ys)} images, 8 classes, "
      f"test accuracy {accuracy(pre, *make_images(SOURCE, 100, seed=1)):.3f}")

# 2. Adaptation to 3 new shapes, 300 steps per method, 5 seeds per label budget.
def adapt(method, X, y, seed):
    torch.manual_seed(seed)
    net = SmallCNN(3)                                      # new 3-way head
    if method == "scratch":                                # (a) random init, all layers
        fit(net, net.parameters(), X, y, lr=3e-3, steps=300, seed=seed)
    elif method == "linear probe":                         # (b) frozen pretrained features
        net.features = copy.deepcopy(pre.features).requires_grad_(False)
        with torch.no_grad():
            Z = net.features(X)                            # features computed once
        fit(net.head, net.head.parameters(), Z, y, lr=1e-2, steps=300, seed=seed)
    else:                                                  # (c) full fine-tuning, lower lr
        net.features = copy.deepcopy(pre.features)
        fit(net, net.parameters(), X, y, lr=1e-3, steps=300, seed=seed)
    return net

print("target task: 3 new classes, 600 test images; accuracy over 5 seeds")
for n in [5, 20]:
    acc = {m: [] for m in ["scratch", "linear probe", "fine-tune"]}
    for seed in range(5):
        Xt, yt = make_images(TARGET, n, seed=100 + seed)
        Xte, yte = make_images(TARGET, 200, seed=200 + seed)
        for m in acc:
            acc[m].append(accuracy(adapt(m, Xt, yt, seed), Xte, yte))
    print(f"  {n} labelled images per class")
    for m, a in acc.items():
        print(f"    {m:12s}  mean {np.mean(a):.3f}  range {min(a):.3f}-{max(a):.3f}")

Output:

torch 2.14.1
source task: 4000 images, 8 classes, test accuracy 0.934
target task: 3 new classes, 600 test images; accuracy over 5 seeds
  5 labelled images per class
    scratch       mean 0.489  range 0.437-0.523
    linear probe  mean 0.750  range 0.740-0.763
    fine-tune     mean 0.643  range 0.558-0.790
  20 labelled images per class
    scratch       mean 0.806  range 0.637-0.933
    linear probe  mean 0.834  range 0.823-0.843
    fine-tune     mean 0.872  range 0.842-0.893

With 15 training images in all, training from scratch reaches 48.9% (guessing gives one third) and the linear probe 75.0%, with little spread across seeds; fine-tuning all 23,491 weights on so few images is unstable (55.8–79.0%). With 20 images per class, fine-tuning is best on average (87.2%), and training from scratch varies most, from 63.7% to 93.3%. The pretrained features help although L, T and H were never source classes, presumably because their bars, corners and junctions occur in the source shapes (our interpretation). Kornblith et al. found the same pattern for ImageNet transfer (below).

Variants

Pretraining sources. Beyond ImageNet-1k, Big Transfer (BiT) pretrained on ImageNet-21k (14M images) and the 300M-image JFT dataset (Kolesnikov et al., 2020). Self-supervised learning pretrains on unlabelled images, and image–text models such as CLIP on captioned web images.

Parameter-efficient fine-tuning. Fine-tuning a large model for every task means storing a full copy per task. Parameter-efficient methods freeze the pretrained weights and train a few new ones:

• Adapters (Houlsby et al., 2019) insert small bottleneck modules into each layer of a transformer. On the GLUE language benchmark they came within 0.4% of full fine-tuning while adding 3.6% parameters per task.
• LoRA (Hu et al., 2022) adds a trainable low-rank update to a frozen weight matrix $W_0 \in \mathbb{R}^{d \times k}$:

$$W = W_0 + BA, \qquad B \in \mathbb{R}^{d \times r},\; A \in \mathbb{R}^{r \times k},\; r \ll \min(d, k).$$

$B$ starts at zero, so training begins exactly at the pretrained model, and $BA$ can be merged into $W_0$ after training, adding no inference latency. The authors developed and evaluated it on language models.

• Visual prompt tuning (Jia et al., 2022) adds a small set of learnable tokens to the input of a frozen vision transformer’s first layer (VPT-shallow) or of every layer (VPT-deep), under 1% of the model’s parameters; the authors reported that VPT-deep beat full fine-tuning on 20 of their 24 recognition tasks.

Domain adaptation keeps the task but changes the input distribution, for example from product photos to webcam images, often without target labels.

Best Practices

• Start with a linear probe: cheap, convex and stable, and with very few labels possibly the best option.
• Give pretrained layers a lower learning rate than the new head.
• Probe, then fine-tune. On ten distribution-shift benchmarks, Kumar et al. (2022) found full fine-tuning on average 2% better than a linear probe in-distribution but 7% worse out-of-distribution, which they explain by the features being distorted while a random head is still being learned. Training the head first, then everything (LP-FT), was 1% and 10% better than full fine-tuning.
• Mind batch normalization with small batches. Detectors trained with a few images per GPU can fix the pretrained batch normalization statistics, which He et al. (2019) note is impossible from scratch; they used group normalization instead, as did BiT, partly because updating running statistics hurt transfer (Kolesnikov et al., 2020).
• Use early stopping (see the training workflow); with few labels, fine-tuning overfits quickly.
• Match the pretraining preprocessing: input size, channel order and normalization constants.

Pitfalls

• Unequal comparisons. Short schedules understate training from scratch (He et al., 2019).
• Test-set overlap. The source may contain the target’s test images: the R-CNN tech report found near-duplicates of 38 of the 4,952 VOC 2007 test images in ILSVRC 2012 trainval, and Zeiler and Fergus removed the few Caltech images they found in their ImageNet training set and retrained.
• Judging features by ImageNet accuracy alone. In Kornblith et al.’s study, label smoothing, dropout, auxiliary heads and a missing batch-norm scale parameter changed ImageNet top-1 accuracy by under 1% but made the penultimate-layer features much worse for linear probes.

Evidence and Impact

Detection without pretraining. He, Girshick and Dollár (2019) trained Mask R-CNN (ResNet-50-FPN, group normalization) on COCO train2017. With the 2× schedule (180k iterations), random initialization reached 36.8 box AP on val2017 against 40.3 with ImageNet pretraining; at 6× it reached 41.3, against 41.1. Training from scratch stayed comparable down to 10k COCO training images; with 1k it reached 3.5 AP against 9.9 (3.4 in the Figure 8 caption), and a Faster R-CNN (ResNet-101-FPN) trained from scratch on PASCAL VOC reached 77.6 mAP against 82.7. The authors concluded that pretraining speeds up convergence but does not necessarily improve final accuracy or reduce overfitting unless target data are very scarce.

Better ImageNet models transfer better. Kornblith, Shlens and Le (2019) compared 16 ImageNet architectures on 12 classification datasets. For networks trained with identical settings, ImageNet top-1 accuracy correlated with transfer accuracy at $r = 0.99$ for logistic regression on fixed features and $r = 0.96$ for fine-tuning. Fine-tuning beat training from random initialization in 189 of 192 dataset–model combinations, but on Stanford Cars and FGVC Aircraft by little: 93.3% against 92.7% and 89.0% against 88.8% for Inception v4. Even so, fine-tuning Inception v4 reached 90% of the maximum odds of correct classification in about 17 times fewer steps than training from scratch (geometric mean over datasets). On subsets of the data, logistic regression matched or beat fine-tuning with 47 to 800 examples in total; with more data, fine-tuning won, and on fine-grained datasets training from scratch approached it.

Medical images. Raghu et al. (2019) found that ImageNet transfer did not significantly improve accuracy on retinal fundus photographs or chest X-rays: ResNet-50 detected diabetic retinopathy with an AUC of 96.4% from random initialization and 96.7% with transfer, and much smaller CNNs did about as well. With only 5,000 training images, transfer helped ResNet-50 (92.2% to 94.6%) but barely changed the small models. Feature reuse occurred mainly in the lowest layers, and random weights drawn with each pretrained layer’s mean and variance already sped up convergence.

In our summary, pretraining reliably saves training time, but its accuracy benefit shrinks with more target data, finer categories and larger domain shift.