Fine-Tuning

Fine-tuning adapts a pretrained network to a new task by continuing to train its weights on the new task’s data, starting from the pretrained values rather than from random ones. It is the adaptation stage of transfer learning, which covers why pretraining helps and when it does not. Fine-tuning is the set of decisions made once a pretrained model is in hand: which layers to train and which to freeze, how fast each should learn and on what schedule, how the new output head starts, and how much of what the model already knew should survive. Its two extremes are the linear probe, which freezes every pretrained layer and trains only a new linear head, and full fine-tuning, which trains everything; partial freezing, layer-wise learning rates and regularized fine-tuning lie between them.

Purpose

A pretrained backbone computes features that fit its source task. A linear probe cannot change them, so when the target task needs features the source did not reward, a probe falls short. Training every weight on a small target set from random values overfits. Fine-tuning sits between: it lets the weights move as far as the target data justify and, ideally, no further.

The difficulty is that “no further” is not built into gradient descent. Fine-tuning and training from scratch minimize the same loss and differ only in where they start (Kumar et al., 2022, §1). Everything that keeps fine-tuning from discarding the pretrained solution, such as a small learning rate, few steps, frozen layers or an explicit penalty, is a choice the practitioner makes. Those choices trade three things against one another: accuracy on the target task, what the network keeps of its source abilities and its robustness to distribution shift, and the compute and storage spent on each new task.

When to Use It

Yosinski et al. (2014) state the usual rule: whether to fine-tune copied layers depends on the size of the target set and the number of parameters in those layers. With a small target set and many parameters, fine-tuning may overfit, so the layers are often left frozen; with a large target set or few parameters, fine-tuning them improves performance.

Fine-tuning pays off most when the pretrained features are close to, but not exactly, what the target needs:

• Features that are not linearly separable but are good starting points. Masked autoencoder (MAE) features score much lower under a linear probe than under fine-tuning; tuning only the last transformer block already closes much of the gap (He et al., 2021, §4.3).
• A shifted input domain, such as a new camera, blur or corruption, where early layers may need to change (Lee et al., 2023).
• Enough target data to constrain the extra parameters.

It is the wrong tool, or needs care, when the target set has only a handful of labels per class and the features are already good, where a probe is cheaper and often as accurate (representation learning treats the probe as an evaluation protocol); when the model must keep performing its original task, where plain fine-tuning forgets (below); when robustness to inputs unlike the target training set matters, where fine-tuning can do worse than a probe (Kumar et al., 2022); and when one large model serves many tasks, where storing a full fine-tuned copy per task is expensive and parameter-efficient methods fit better.

How It Works

Write the network as a backbone $\phi_\theta$ followed by a head $h_w$. Pretraining gives backbone weights $\theta_0$; a new task with $n$ labelled images $(x_i, y_i)$ and loss $\ell$ gets a new head $w$. Fine-tuning minimizes

$$\mathcal{L}(\theta, w) = \frac{1}{n}\sum_{i=1}^{n} \ell\big(h_w(\phi_\theta(x_i)),\, y_i\big) + \Omega(\theta, w)$$

over a chosen subset of the parameters, starting from $\theta = \theta_0$. Freezing a layer removes its weights from the subset; the linear probe is the case where only $w$ is trained. The regularizer $\Omega$ is often plain weight decay, which pulls weights toward zero.

Staying near the starting point

Weight decay toward zero ignores where the weights started. Li, Grandvalet and Davoine (2018) argue that the pretrained values are a more sensible reference and propose L2-SP (“starting point”):

$$\Omega(\theta, w) = \frac{\alpha}{2}\,\lVert \theta - \theta_0 \rVert_2^2 + \frac{\beta}{2}\,\lVert w \rVert_2^2 ,$$

which penalizes distance from the pretrained backbone and keeps ordinary decay on the new head. Fine-tuning a ResNet-101 with L2-SP instead of plain L2 raised accuracy from 79.6% to 84.2% on MIT Indoors 67 (pretrained on Places 365) and from 81.4% to 85.1% on Stanford Dogs 120 (pretrained on ImageNet), means of five runs (Table 2). On the two datasets where they also tried freezing the first layers, freezing improved plain L2 fine-tuning considerably, but every freezing depth with L2 was beaten by L2-SP without freezing (Figure 3).

The explicit penalty is optional. Big Transfer (BiT) fine-tunes without weight decay toward zero, without weight decay toward the initial parameters, and without dropout; its authors found that a schedule whose length grows with the dataset provides sufficient regularization (Kolesnikov et al., 2020, §2.2). Regularization covers the general penalties.

Feature distortion and the head’s starting point

The new head usually starts random. Kumar et al. (2022) showed that this matters. When the backbone and a random head are trained together to fit the target data, the backbone changes substantially for the training inputs. In a two-layer linear model they proved that fine-tuning updates the features along the directions spanned by the training data, while features in other directions, which out-of-distribution inputs may occupy, change less. The head ends up fitted to the changed features and mismatched to the unchanged ones; the authors call this feature distortion, and showed that in their setting it cannot simply be fixed by early stopping. With MoCo v2 and CLIP features on ten distribution-shift benchmarks, fine-tuning beat a linear probe in distribution on average but lost to it out of distribution (transfer learning gives the averages).

Their remedy is LP-FT: train a linear probe first, then fine-tune everything starting from that head. Because the head is already good, the backbone needs to change less. In their measurements LP-FT moved the features about 20 times less than fine-tuning on Living-17 (§4.3), and between roughly 8 and 1,000 times less across the five datasets of their appendix tables (Tables 5 and 6 of the ICLR version, our arithmetic); the introduction summarizes this as 10 to 100 times. On average LP-FT beat both fine-tuning and probing, in and out of distribution (Tables 1 and 2). Kumar et al. note that earlier work had used the same two steps as a heuristic, without the robustness argument.

Which layers to free

Features in a CNN trained on ImageNet go from general in the first layers to specific in the last, which is the basis of the convention of freezing early layers and tuning late ones (transfer learning describes the Yosinski et al. experiments). Oquab et al. (2014) followed it: they kept all five convolutional layers and the first two fully connected layers of their ImageNet network frozen and trained two new fully connected layers on Pascal VOC.

The convention fits a change of classes but not every shift. Lee et al. (2023) tuned one block of a network at a time (surgical fine-tuning) on seven shifts of three kinds and found that on every one, the best single block beat tuning all parameters. Which block was best depended on the shift: the first block for input-level shifts such as image corruptions (on CIFAR-10-C, 82.8% accuracy against 79.9% for full fine-tuning in their Figure 1), a middle block for feature-level shifts such as new subpopulations of the same classes, and the last layer for output-level shifts such as spurious correlations between label and attribute. Their explanation is that the layers left frozen may already be close to optimal for the target, so updating them on a small target set only risks overfitting; they note that the benefit comes from limited target data, and that tuning more layers pays off when target data are plentiful.

Gradual unfreezing, from ULMFiT, makes freezing a schedule: train the last layer for one epoch, unfreeze the next layer down and train all unfrozen layers for another epoch, and continue until the whole network trains (Howard & Ruder, 2018, §3.3). The authors introduced it to avoid catastrophic forgetting.

Discriminative learning rates

Instead of a binary choice between frozen and free, every layer can learn at its own rate. ULMFiT’s discriminative fine-tuning splits the parameters by layer and gives each its own learning rate; the authors chose the last layer’s rate $\eta_L$ by fine-tuning only that layer, then used $\eta_{l-1} = \eta_l / 2.6$ for each layer below (Howard & Ruder, 2018, §3.2). The same idea applied to vision transformers is layer-wise learning-rate decay: with blocks numbered $l = 1, \dots, L$ from the input, block $l$ trains at

$$\eta_l = \eta \, \gamma^{\,L + 1 - l}, \qquad 0 < \gamma < 1 ,$$

the head at $\eta$ and the patch embedding, below block 1, at $\eta \gamma^{L+1}$. This is the rule implemented in the MAE code, which credits BEiT and ELECTRA for it; MAE fine-tunes its ViTs with $\gamma = 0.75$ (He et al., 2021, appendix Table 9). For ViT-B, with $L = 12$, the patch embedding thus learns at $0.75^{13} \approx 0.024$ of the head’s rate (our arithmetic). ULMFiT’s division by 2.6 per layer corresponds to $\gamma \approx 0.38$.

Both forms encode the same assumption as freezing early layers: that lower layers are the more general ones. Surgical fine-tuning shows where that assumption fails.

What fine-tuning forgets

A network trained on one task and then on another loses performance on the first. Kirkpatrick et al. (2017) describe this catastrophic forgetting as happening because weights important for the first task are changed to meet the objectives of the second, and cite connectionist work from 1989 onward. For transfer it means that the fine-tuned backbone no longer serves the source task, and, in Kumar et al.’s and Lee et al.’s analyses, that it may also lose the source knowledge that made it robust.

The size of the effect can be measured by putting the original source head back on the fine-tuned backbone. Li, Grandvalet and Davoine (2018, Table 3) did this: after fine-tuning with plain L2, ImageNet accuracy (originally 76.7%) dropped by 14.1 to 16.9 points and Places 365 accuracy (54.7%) by 24.1 points; with L2-SP the drops were 3.6 to 5.3 points. Li and Hoiem (2018, Table 1(c), read in the arXiv v3 preprint of the journal paper) report a similar comparison for a VGG network fine-tuned from ImageNet to Caltech-UCSD Birds: its ImageNet accuracy fell 18.1 points below that of the unchanged network (our arithmetic from the table’s differences), even though their fine-tuning first trained the new head alone, a warm-up step they found greatly reduces forgetting.

Remedies differ in what they anchor:

• Parameters. L2-SP pulls all weights toward $\theta_0$ equally. Elastic weight consolidation (EWC; Kirkpatrick et al., 2017) weights the pull by each parameter’s importance to the old task $A$, estimated by the diagonal of the Fisher information $F_i$: the loss for the new task $B$ becomes $\mathcal{L}_B(\theta) + \sum_i \frac{\lambda}{2} F_i (\theta_i - \theta^*_{A,i})^2$. L2-SP with Fisher weights reduced forgetting slightly further than plain L2-SP but did not improve target accuracy, so Li et al. recommend the simpler form.
• Outputs. Learning without Forgetting (LwF; Li & Hoiem, 2018, first at ECCV 2016) records the old head’s outputs on the new task’s images before training, then adds a distillation loss (temperature $T = 2$) that keeps those outputs unchanged while the new head learns. It needs no old training data. In their experiments it kept old-task accuracy much closer to the original than fine-tuning did and, unexpectedly, beat fine-tuning on the new task in most task pairs as well, which the authors read as the preserved outputs acting as a regularizer.
• Weights after training. WiSE-FT (Wortsman et al., 2022) interpolates between the pretrained and fine-tuned weights, $\theta_\alpha = (1-\alpha)\,\theta_0 + \alpha\,\theta_{\text{ft}}$. For CLIP ViT-L/14 at 336 pixels (their Table 1), end-to-end fine-tuning raised ImageNet top-1 accuracy from 76.6% (zero-shot) to 86.2% but lowered the average over five ImageNet-derived distribution shifts from 73.4% to 68.6%; WiSE-FT with $\alpha = 0.5$ reached 86.8% on ImageNet and 76.9% under shift, at no extra cost in fine-tuning or inference. The abstract summarizes this as 4 to 6 points under shift over prior work and 1.6 points on ImageNet; by our reading of the table, the 1.6 is the gain over the authors’ own fine-tuned linear classifier (85.2%).

How to Apply It

Learning rates. Pretrained layers train at a fraction of the pretraining rate; the R-CNN detector, for example, started fine-tuning at 0.001, a tenth of its initial pretraining rate, so that fine-tuning could make progress without destroying the initialization, as the authors explain (Girshick et al., 2014, tech report v5). A new head can learn faster than the layers below it, which is what discriminative rates formalize. Both Adam and stochastic gradient descent with momentum are used: Lee et al. (2023) fine-tuned with Adam, BiT with SGD.

Schedules. Fine-tuning usually runs for a short schedule with warmup and decay (learning-rate schedule). MAE fine-tunes ViT-L and ViT-H for 50 epochs with a 5-epoch warmup and cosine decay (appendix Table 9). ULMFiT’s slanted triangular learning rate rises linearly for a short first part of training and then decays linearly over the rest, by default over the first 10% of iterations to a peak of 0.01 (Howard & Ruder, 2018, §3.2). BiT replaces per-task tuning with a fixed rule, the BiT-HyperRule: SGD with momentum at an initial rate of 0.003; 500 steps for tasks with fewer than 20,000 labelled images, 10,000 steps below 500,000 and 20,000 steps above; the rate divided by 10 at 30%, 60% and 90% of training; and mixup only for the larger two regimes (Kolesnikov et al., 2020, §3.3).

Normalization. With small batches, the pretrained batch normalization statistics are often kept fixed rather than re-estimated from noisy batches.

Example: probing, freezing and fine-tuning on shifted shapes

The PyTorch script below (PyTorch 2.14.1; about 35 seconds on a laptop CPU) pretrains a small CNN with three convolutional blocks on six shapes drawn at random positions, sizes and contrasts with pixel noise. The target task has four shapes the network never saw (L, T, H and X), rendered in a shifted domain, blurred and with 30% less contrast, with 20 labelled images per class. Each of seven adaptation schemes gets a new four-way head and 200 Adam steps (LP-FT gets 200 probe steps first). New heads learn at $10^{-2}$ and pretrained layers at $10^{-3}$ at most, except in “full, one rate”, where the head also uses $10^{-3}$.

Besides target accuracy, the script measures forgetting as Li, Grandvalet and Davoine did: it puts the original six-way head back on the adapted backbone and tests it on source images. The loop runs seeds 0 to 4, each with its own target training and test sets, and prints the mean and range.

import copy
import numpy as np
import torch
from torch import nn
from scipy.ndimage import gaussian_filter
F = nn.functional
torch.set_num_threads(1)                      # fixed summation order: identical output every run

def render(kind, rng):
    """32x32 binary mask of one shape with random size and position."""
    s = rng.uniform(12, 22)
    yy, xx = np.mgrid[:32, :32]
    cy, cx = rng.uniform(s / 2 + 1, 31 - s / 2, size=2)
    dy, dx, r, t = yy - cy, xx - cx, s / 2, max(2.0, s / 6)       # t: stroke width
    ady, adx = np.abs(dy), np.abs(dx)
    m = {   # source classes
         "disk":     dy**2 + dx**2 <= r**2,
         "square":   np.maximum(ady, adx) <= r,
         "triangle": (ady <= r) & (adx <= (dy + r) / 2),
         "ring":     np.abs(np.hypot(dy, dx) - r + t / 2) <= t / 2,
         "plus":     ((ady <= t / 2) & (adx <= r)) | ((adx <= t / 2) & (ady <= r)),
         "diamond":  ady + adx <= r,
            # target classes, never seen in pretraining
         "L": ((adx <= r) & (dy >= r - t) & (dy <= r)) | ((ady <= r) & (dx >= -r) & (dx <= -r + t)),
         "T": ((adx <= r) & (dy >= -r) & (dy <= -r + t)) | ((adx <= t / 2) & (ady <= r)),
         "H": ((ady <= r) & (np.abs(adx - r + t / 2) <= t / 2)) | ((adx <= r) & (ady <= t / 2)),
         "X": ((np.abs(dy - dx) <= 0.7 * t) | (np.abs(dy + dx) <= 0.7 * t)) & (np.maximum(ady, adx) <= r),
    }[kind]
    return m.astype(np.float32)

def make_images(classes, n_per_class, seed, shifted=False):
    """Bright shapes on a dark background with noise; the shifted domain is blurred and lower in contrast."""
    rng, X, y = np.random.default_rng(seed), [], []
    for label, kind in enumerate(classes):
        for _ in range(n_per_class):
            m = render(kind, rng)
            fg, bg = rng.uniform(0.6, 1.0), rng.uniform(0.0, 0.3)
            if shifted:
                m, fg = gaussian_filter(m, 1.0), bg + 0.7 * (fg - bg)
            X.append(bg + (fg - bg) * m + rng.normal(0, 0.2, m.shape))
            y.append(label)
    return torch.tensor(np.array(X), dtype=torch.float32).unsqueeze(1), torch.tensor(y)

def small_cnn(n_classes):
    return nn.Sequential(
        nn.Sequential(nn.Conv2d(1, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2)),    # block 1
        nn.Sequential(nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2)),   # block 2
        nn.Sequential(nn.Conv2d(32, 64, 3, padding=1), nn.ReLU(),
                      nn.AdaptiveAvgPool2d(1), nn.Flatten()),                         # block 3
        nn.Linear(64, n_classes))                                                     # head

def train(net, groups, X, y, steps, seed, batch=16):
    """Adam on random mini-batches. `groups` lists (module, learning rate); all else is frozen."""
    net.requires_grad_(False)
    for module, _ in groups:
        module.requires_grad_(True)
    opt = torch.optim.Adam([{"params": m.parameters(), "lr": lr} for m, lr in groups])
    g = torch.Generator().manual_seed(seed)
    for _ in range(steps):
        idx = torch.randint(len(y), (batch,), generator=g)
        loss = F.cross_entropy(net(X[idx]), y[idx])
        opt.zero_grad(); loss.backward(); opt.step()
    return net

def accuracy(net, X, y):
    with torch.no_grad():
        return (net(X).argmax(1) == y).float().mean().item()

SOURCE = ["disk", "square", "triangle", "ring", "plus", "diamond"]
TARGET = ["L", "T", "H", "X"]
print("torch", torch.__version__)

# 1. Pretrain on the source task: 6 shapes, 400 images each.
torch.manual_seed(0)
pre = small_cnn(len(SOURCE))
train(pre, [(pre, 3e-3)], *make_images(SOURCE, 400, seed=0), steps=600, seed=0, batch=64)
Xs_te, ys_te = make_images(SOURCE, 100, seed=1)
print(f"source test accuracy after pretraining: {accuracy(pre, Xs_te, ys_te):.3f}")

# 2. Adapt to the target task: 4 new shapes, blurred and lower in contrast, 20 labelled images each.
#    Every method gets a new head and 200 Adam steps; pretrained layers train at 1e-3 at most.
def adapt(method, X, y, seed, lr=1e-3, head_lr=1e-2):
    torch.manual_seed(seed)
    net = copy.deepcopy(pre)
    net[3] = nn.Linear(64, len(TARGET))                       # new, randomly initialized head
    b1, b2, b3, head = net
    if method == "LP-FT":                                     # stage 1: linear probe
        train(net, [(head, head_lr)], X, y, 200, seed)
    groups = {
        "linear probe":       [(head, head_lr)],
        "freeze blocks 1-2":  [(b3, lr), (head, head_lr)],
        "freeze block 1":     [(b2, lr), (b3, lr), (head, head_lr)],
        "full, one rate":     [(net, lr)],
        "full, faster head":  [(b1, lr), (b2, lr), (b3, lr), (head, head_lr)],
        "layer-wise decay":   [(b1, lr / 8), (b2, lr / 4), (b3, lr / 2), (head, head_lr)],
        "LP-FT":              [(net, lr)],                    # stage 2: everything, one rate
    }[method]
    return train(net, groups, X, y, 200, seed)

def source_accuracy(net):
    """Forgetting: the original source head on the adapted backbone."""
    old = copy.deepcopy(net)
    old[3] = pre[3]
    return accuracy(old, Xs_te, ys_te)

methods = ["linear probe", "freeze blocks 1-2", "freeze block 1", "full, one rate",
           "full, faster head", "layer-wise decay", "LP-FT"]
res = {m: [] for m in methods}
for seed in range(5):
    Xt, yt = make_images(TARGET, 20, seed=100 + seed, shifted=True)
    Xt_te, yt_te = make_images(TARGET, 100, seed=200 + seed, shifted=True)
    for m in methods:
        net = adapt(m, Xt, yt, seed)
        res[m].append((accuracy(net, Xt_te, yt_te), source_accuracy(net)))

print("accuracy over seeds 0-4: mean (min-max)")
print(f"  {'method':18s} {'target task':>20s} {'source task, old head':>24s}")
for m in methods:
    t, s = np.array(res[m]).T
    print(f"  {m:18s} {t.mean():8.3f} ({t.min():.3f}-{t.max():.3f}) "
          f"{s.mean():11.3f} ({s.min():.3f}-{s.max():.3f})")

Output:

torch 2.14.1
source test accuracy after pretraining: 0.973
accuracy over seeds 0-4: mean (min-max)
  method                      target task    source task, old head
  linear probe          0.704 (0.680-0.733)       0.973 (0.973-0.973)
  freeze blocks 1-2     0.766 (0.695-0.825)       0.907 (0.878-0.948)
  freeze block 1        0.782 (0.730-0.848)       0.922 (0.862-0.968)
  full, one rate        0.694 (0.587-0.793)       0.780 (0.735-0.807)
  full, faster head     0.804 (0.760-0.858)       0.918 (0.828-0.962)
  layer-wise decay      0.767 (0.710-0.822)       0.969 (0.962-0.977)
  LP-FT                 0.794 (0.750-0.855)       0.934 (0.870-0.963)

The linear probe forgets nothing by construction: the backbone is untouched, so the old head scores exactly its pretraining accuracy on every seed. On the target it is limited by features that were never trained on blurred, low-contrast L, T, H and X shapes. Every scheme that lets the pretrained layers move while the new head learns fast, or starts trained, does better on the target, by 6 to 10 points on average. Among those schemes the target differences are smaller than the spread across seeds, so this small experiment cannot rank them on the target. They differ more clearly in what they keep. Layer-wise decay, which trains block 3 at half the pretrained-layer rate and block 1 at an eighth, keeps source accuracy within half a point of the pretrained network on average, and no seed loses more than 1.1 points; the two freezing schemes and “full, faster head” lose 5 to 7 points on average, and LP-FT about 4.

The row to compare with LP-FT is “full, one rate”. Both train every layer at $10^{-3}$; the only difference is the head at the start of that stage, random in one and probed in the other. With a random head that learns slowly, full fine-tuning has the worst target accuracy and the widest range of all schemes and forgets the most. In our interpretation this is Kumar et al.’s mechanism at small scale: while the head is still random, the backbone does the fitting and its features are pushed around. LP-FT also trains for 200 more steps in total, which accounts for part of its advantage. Giving the head its own higher rate (“full, faster head”) avoids most of the damage without a separate probing stage. The network has 23,556 parameters, of which the linear probe trains 260; with a backbone this small and synthetic data this simple, the ranking illustrates the mechanisms rather than predicting results on photographs.

Variants

• Partial fine-tuning: freezing a fixed set of early layers (the convention), or one chosen block (surgical fine-tuning).
• Gradual unfreezing and discriminative or layer-wise rates, which spread the decision over training time or over depth.
• LP-FT: a probe, then full fine-tuning from the probed head.
• Regularized fine-tuning: L2-SP and EWC anchor parameters; LwF anchors outputs; knowledge distillation from the original model is the general form of the latter.
• Weight-space ensembling: WiSE-FT interpolates the pretrained and fine-tuned weights after training.
• Fine-tuning at a higher resolution than pretraining, common for vision transformers, which requires resampling the position embeddings (vision transformer gives the details).
• Parameter-efficient fine-tuning freezes the pretrained weights and trains a small added set, so one stored backbone serves many tasks. Adapters insert small bottleneck modules in every layer (Houlsby et al., 2019); LoRA adds trainable low-rank updates to frozen weight matrices without adding inference latency (Hu et al., 2022); prompt tuning learns a few input embeddings for a frozen language model (Lester et al., 2021); and visual prompt tuning learns extra input tokens for a frozen vision transformer, under 1% of its parameters (Jia et al., 2022). Transfer learning gives their reported results.

Best Practices

• Probe first. A probe gives a cheap baseline, shows whether the features already suffice, and supplies the head for LP-FT.
• Give the new head a higher rate, or train it first. A random head on a slowly learning schedule lets the backbone absorb the early gradients.
• Pick what to free by the kind of shift. Changes in the labels favor late layers; changes in the input, such as a new sensor, blur or corruption, may favor early ones (Lee et al., 2023).
• Prefer a penalty toward the pretrained weights over decay toward zero when target data are scarce, or keep regularization light, as BiT does, when the schedule is short.
• Tune the fine-tuning learning rate separately from the probe’s and from pretraining’s; MAE’s authors found that linear probing needs a very different recipe from end-to-end fine-tuning (He et al., 2021, appendix A.1).
• Measure forgetting and shift robustness when either matters, not only target accuracy on the target distribution.

Pitfalls

• Comparing methods at one shared setting. A single learning rate for a probe, a partly frozen network and a fully fine-tuned one favors whichever method that rate suits; in the example the same full fine-tuning goes from the worst to the best target accuracy when the head gets its own rate.
• Assuming early layers never need to change. Freezing them, or giving them tiny rates, is wrong for input-level shifts (Lee et al., 2023).
• Relying on early stopping to preserve features. For good features and a large shift, Kumar et al. (2022) proved in their two-layer linear model that fine-tuning from a random head never passes through parameters that do as well out of distribution as the probe; in their experiments, even stopping fine-tuning at the point best on out-of-distribution test data left it below the probe on average (61.3% against 67.1%, §4.3).
• Silently losing the source ability. A fine-tuned backbone shared with other heads, or reused for its original task, may no longer work for them.

Evidence and Impact

Early CNN transfer. Oquab et al. (2014) trained only two new fully connected layers on top of a frozen ImageNet network and reported 77.7% mAP on Pascal VOC 2007 classification, against 70.5% for the best earlier method they compared with. On VOC 2012 action classification, retraining the two upper pretrained fully connected layers along with the new ones (as the CVPR version states; the technical report lists only the first of them) raised mAP from 68.4% to 70.2%, the best published result at the time to the authors’ knowledge.

Discriminative rates and unfreezing. In ULMFiT’s text-classification ablation, fine-tuning only the last layer gave 16.09% validation error on TREC-6, full fine-tuning 6.86%, full fine-tuning with discriminative rates 6.21%, and the complete method with gradual unfreezing, discriminative rates and slanted triangular rates 5.69% (Howard & Ruder, 2018, Table 7). These are language results; in vision, the related layer-wise decay is part of ViT fine-tuning recipes such as MAE’s.

Robustness. Kumar et al. (2022) and Wortsman et al. (2022) showed that fine-tuning can raise accuracy on the target distribution while lowering it under distribution shift, and that staying closer to the pretrained model, through a probed head or weight interpolation, improves accuracy under shift without giving up target accuracy, at no extra inference cost.

Which layers. Lee et al. (2023) found that tuning the single right block beat tuning everything on all seven of their shifts, and that the right block followed the kind of shift.

In our summary, fine-tuning works best when it changes the pretrained network only where the target task requires, whether through freezing, per-layer rates, a trained head or an explicit anchor, and evaluating it on the target task alone hides what it costs.