Parameter-Efficient Fine-Tuning

Parameter-efficient fine-tuning (PEFT), also called parameter-efficient transfer learning, adapts a pretrained network to a new task by training a small set of parameters while the pretrained weights stay frozen. The trained set is either new (small modules inserted into the network, learned tokens added to its input, a low-rank update of its weight matrices) or a chosen subset of the existing weights (the biases, the last block). Each task then needs only its small set and its head, and one stored copy of the backbone serves every task. The idea entered deep learning through language models, with adapters (Houlsby et al., 2019), prefix tuning (Li & Liang, 2021), prompt tuning (Lester et al., 2021) and BitFit (Ben-Zaken et al., 2022), and was carried to vision by visual prompt tuning (Jia et al., 2022) and the comparisons that followed on vision transformers. Fine-tuning and transfer learning place the family among the other ways to reuse a pretrained model; LoRA treats the low-rank member in depth.

Purpose

Full fine-tuning changes every weight, so it produces a complete copy of the model for every task. Houlsby et al. (2019) framed the problem for a cloud service that receives tasks one after another from customers: the goal was a model that solves each new task by adding a few parameters, without retraining or forgetting the earlier ones (§1). Jia et al. (2022) make the same point for vision, where adapting the latest large backbone to a recognition problem means storing and deploying a separate copy of its parameters for every task (§1).

Three costs can be run together under “efficiency”, and the methods differ on each:

• Storage per task. The number of parameters that must be saved for each task. This is what every method in the family reduces: by about two orders of magnitude for adapters on BERT (Houlsby et al., 2019, Fig. 1) and by over five for a short prompt on an 11-billion-parameter T5 (Lester et al., 2021, Fig. 2).
• Training memory and time. Weights, gradients, optimizer state and the activations kept for backpropagation. Freezing removes the gradients and optimizer state of the frozen weights, but the activations needed to propagate gradients through the network remain whenever a trained parameter sits early in it. The example below measures all of these.
• Inference cost. Methods that add modules or tokens make every forward pass longer; methods that change existing weights, or whose change can be folded into them, do not.

Freezing also has a statistical side. A small trained set restricts how far the model can move from its pretrained state. VPT’s experiments found parameter-efficient methods ahead of full fine-tuning when target data were scarce (§4.2, Fig. 3), and Houlsby et al. note that adapters avoid forgetting by construction, since the shared parameters never change (§1).

When to Use It

The family suits a large pretrained backbone that must serve several tasks, or tasks that arrive over time, and settings where storing or shipping a full copy per task is impractical. Typical vision cases are a vision transformer adapted to many small classification datasets, a vision–language model tuned for new tasks (visual instruction tuning), and a text-to-image diffusion model taught a subject or style (personalized image generation).

The sources point to conditions under which it does less well:

• Plenty of target data. In VPT’s data-scale experiment on five fine-grained datasets, the methods with few trainable parameters beat full fine-tuning with little data, but the trend reversed with more data for the linear probe and adapters; VPT-deep stayed ahead of full fine-tuning at every size tried (Jia et al., 2022, §4.2, Fig. 3).
• Small models. Lester et al. (2021) found that prompt tuning becomes more competitive as the model grows, closing the gap to model tuning as models exceed billions of parameters and matching even multi-task model tuning at the largest T5 size, with 11 billion parameters (abstract, §3.1). On ResNet-50, VPT’s advantage diminished and there was no clear winner across the 19 VTAB-1k tasks (Jia et al., 2022, §5, Table 5).
• Some pretraining objectives. With self-supervised MAE and MoCo v3 backbones, VPT found the comparisons among methods less conclusive than with a supervised backbone (§5, Table 4), and He et al. (2023) write that existing parameter-efficient methods were inferior to full fine-tuning on both (§4), although their Table 2 shows LoRA of rank 16 slightly above full fine-tuning on MAE (64.8 against 64.3 mean VTAB-1k accuracy).
• Training memory as the binding constraint. Methods with trained parameters in the early layers still backpropagate through the whole backbone, so their training memory stays near full fine-tuning’s (VPT’s measurements and the example below).

In our reading, the decision is mainly about storage and serving: when one task and one copy are all that is needed, the case for a parameter-efficient method rests on its accuracy with the available data, not on parameter counts.

How It Works

What trains and where

The methods can be sorted by where the trained parameters sit. VPT groups the tuning protocols it compares into head-oriented ones (a linear probe, the last $k$ blocks, an MLP head), backbone-oriented ones (biases, adapters, a side network), and its own input-space prompts (Jia et al., 2022, Fig. 1, §4.1). He et al. (2023) divide parameter-efficient methods into addition-based ones, which attach new trainable parameters (prompts, adapters), and reparameterization-based ones, which tune parameters already in the backbone or parameters that can be folded into it at inference, such as biases, the last layers or LoRA’s low-rank matrices (§2). DoRA’s authors separate adapters and prompt-based methods, which change the architecture or the input and so add inference latency, from LoRA and its variants, which add none (Liu et al., 2024, §2).

The main forms, for a transformer layer with width $d$:

Method	What trains	Per-task parameters	Inference cost
Linear probe	new head only	head	none added
Partial-$k$	last $k$ blocks	$k$ blocks	none added
BitFit	every bias	all bias vectors	none added
Adapters	bottleneck modules in each layer	$2md + m + d$ per module	extra layers
Prefix tuning	key and value vectors per layer	$2pd$ per layer	longer key sequence
Prompt tuning, VPT-shallow	input tokens	$pd$	longer sequence
VPT-deep	input tokens of every layer	$pd$ per layer	longer sequence
LoRA	low-rank update $BA$ of chosen matrices	$r(d_\text{in} + d_\text{out})$ per matrix	none once merged

Here $m$ is the adapter’s bottleneck width, $p$ the number of prompt or prefix vectors and $r$ the rank. A new task head is trained in every case.

Adapters

Houlsby et al. (2019) insert a small module after each of the two sublayers of every transformer layer: after the projection that follows multi-head attention and after the feed-forward sublayer, before the residual addition (§2.1, Fig. 2). Each module projects the $d$-dimensional features down to $m \ll d$ dimensions, applies a nonlinearity, projects back up and adds its input through an internal skip connection:

$$\mathrm{Adapter}(\mathbf{h}) = \mathbf{h} + W_\text{up}\, \sigma(W_\text{down} \mathbf{h} + \mathbf{b}_\text{down}) + \mathbf{b}_\text{up}, \qquad W_\text{down} \in \mathbb{R}^{m \times d},\; W_\text{up} \in \mathbb{R}^{d \times m}.$$

With biases this is $2md + d + m$ parameters per module. The paper’s text calls this the number “added per layer”; with two modules in each layer, as its Figure 2 shows, a layer gains twice that, and the example below counts it that way. The new layer normalization parameters of each layer are trained as well (§2.1). The projections are initialized near zero (a truncated Gaussian with standard deviation $10^{-2}$), so the module starts close to the identity and the pretrained network is nearly unchanged at the start of training; the authors found that larger initial scales degraded performance (§2, §3.6). Removing adapters from single layers changed accuracy little, removing all of them dropped it to the majority-class level, and adapters in the lower layers mattered less than those in the upper ones (§3.6). Training only the layer-normalization parameters, about 40,000 for BERT-base, performed clearly worse (§3.4).

Later work changes the placement. VPT’s implementation follows studies by Pfeiffer et al., which it reports found a single adapter after the feed-forward sublayer best, and sweeps the reduction rate $d/m \in \{8, 64, 256\}$ (Jia et al., 2022, supplementary §A.1). Sung et al. (2022) applied adapters to the language model of a CLIP-BART vision–language model and compared sharing one adapter across tasks, adapters per task, and the weight-sharing variants Hyperformer and Compacter (§3, Table 1).

Prefix and prompt tuning

Prefix tuning (Li & Liang, 2021) keeps a language model frozen and learns, for every layer, a sequence of continuous vectors that the real tokens attend to as if they were preceding “virtual tokens” (§1, §4). In GPT-2 each layer’s activation at a position is a key–value pair (§3, footnote 1), so a prefix of length $p$ holds $2pd$ numbers per layer. Optimizing the prefix directly was unstable, so the authors reparametrize it during training: a smaller matrix passes through a feed-forward network to produce the prefix, and only the prefix itself is kept after training (§4.3). Training therefore uses more parameters than are stored, which Lester et al. (2021) also note (Fig. 4). Tuning only the input embeddings of the virtual tokens (“embedding-only”) performed significantly worse than tuning every layer’s activations (Li & Liang, 2021, §7.2, Table 5).

Prompt tuning (Lester et al., 2021) is that embedding-only form: $p$ trainable vectors are prepended to the input embeddings of a frozen model and nothing else changes. The authors describe it as a simplification of prefix tuning (abstract). For T5-XXL with 11 billion parameters, a five-token prompt is 20,480 parameters per task (Fig. 2). Its quality depends strongly on scale: on SuperGLUE, prompt tuning trailed model tuning for small T5 models and matched even multi-task model tuning at the XXL size (§3.1), and the XXL model was the most robust to the choice of prompt length and initialization (Fig. 3).

Visual prompt tuning

VPT (Jia et al., 2022) applies the idea to a vision transformer. After the patch embedding, $p$ learnable $d$-dimensional prompt vectors are concatenated with the class token and the patch tokens (§3.2):

$$[\mathbf{x}_1, Z_1, E_1] = L_1([\mathbf{x}_0, P, E_0]), \qquad [\mathbf{x}_i, Z_i, E_i] = L_i([\mathbf{x}_{i-1}, Z_{i-1}, E_{i-1}]),\; i = 2, \dots, N,$$

where $\mathbf{x}$ is the class token, $E$ the patch tokens and $Z$ the prompts’ outputs. This is VPT-shallow, prompt tuning on images. VPT-deep gives every layer its own prompts $P_{i-1}$ and discards the prompts’ outputs, $[\mathbf{x}_i, \cdot\,, E_i] = L_i([\mathbf{x}_{i-1}, P_{i-1}, E_{i-1}])$. Only the prompts and a linear head on the final class token train. Because prompts are added after the positional encoding, their position in the sequence does not matter. The paper’s example is ViT-Base with $d = 768$: 50 shallow prompts add $50 \times 768 = 0.038$M parameters and deep prompts $N \times p \times d = 0.46$M, given as 0.04% and 0.53% of the model (prompt engineering quotes the same figures); the example below recomputes them.

The ablations shape how VPT is used. Prepending prompts beat adding them element-wise to the patch embeddings; prompts in pixel space, as extra padding or an extra channel, did worse than prompts in the latent space (§4.3, Fig. 5); the best prompt length varied by task (Fig. 6); inserting prompts at more layers generally helped, and earlier layers mattered more than later ones (Fig. 7); and pooling over the prompts’ outputs instead of reading the class token lost up to 8 points (Fig. 8). The supplementary material reports that keeping randomly initialized prompts fixed performed about like a linear probe, which the authors read as the gain coming mainly from learning the prompts rather than from the longer sequence (§B, Fig. 2). For Swin, prompts are attended within local windows and dropped at patch merging; for ConvNets, the “prompt” is a learnable border of pixels padded around the input (§5).

Selective methods

Selective methods train a subset of the pretrained weights and add nothing. The simplest are a linear probe and partial fine-tuning of the top layers, which fine-tuning covers in depth.

BitFit (Ben-Zaken et al., 2022) trains only the bias terms of a BERT encoder and the task classifier. Its biases are those of the query, key and value projections, the dense layers after attention and in the feed-forward block, and the layer normalizations; they make up 0.09% of BERT-base’s parameters and 0.08% of BERT-large’s (§3). On the GLUE validation sets with BERT-large, BitFit averaged 84.2 against 84.1 for the authors’ own full fine-tuning (Table 1); tuning a random subset of the same size did substantially worse (§4, Table 3). Measuring how far each bias moved, the authors found no change in the key biases and the largest changes in the query biases and those of the intermediate feed-forward layer (§4, Fig. 1). Training only the query and middle-of-MLP biases, half of all biases and 0.04% of the parameters, lost about 1.3 points on average for BERT-base (82.4 to 81.1, Table 3). VPT reports that bias-only tuning had earlier been applied to ConvNets by Cai et al. (2020), and evaluates it on ViTs as “Bias” (Jia et al., 2022, §2).

Sensitivity-aware fine-tuning (SPT; He et al., 2023) chooses where to spend a parameter budget for each task: before training, it scores each weight by the loss reduction expected from tuning it, approximated to first order from one forward and backward pass on the task’s data; it tunes the most sensitive individual weights, and for weight matrices with more sensitive weights than a threshold it uses a structured method such as LoRA or an adapter instead (§1, §3). He et al. also describe SSF, which trains scaling and shifting parameters that can be absorbed into the preceding linear layers (§2).

Low-rank updates

LoRA freezes each chosen weight matrix $W_0 \in \mathbb{R}^{d \times k}$ and learns $W_0 + BA$ with $B \in \mathbb{R}^{d \times r}$, $A \in \mathbb{R}^{r \times k}$ and $r \ll \min(d, k)$; $B$ starts at zero, so training starts at the pretrained model, and $BA$ can be added into $W_0$ after training so that inference costs nothing extra (Liu et al., 2024, §3.1). The LoRA article covers its initialization and scaling, which matrices receive it, merging, the evidence on rank, and variants including QLoRA and DoRA.

A unified view

The methods look different but act on the same computation. He et al. (2023) report that He, Zhou, Ma, Berg-Kirkpatrick and Neubig (2022), whose paper was not read for this article, connected prompt tuning and adapters and gave a unified view in which all parameter-efficient methods adjust the hidden representations (§2). Two connections can be checked directly.

VPT-deep is prefix tuning with tied keys and values. In VPT-deep, the prompts $P_{i-1}$ entering layer $i$ influence the other tokens only through the keys and values that attention computes from them, $\mathrm{LN}(P_{i-1}) W_K$ and $\mathrm{LN}(P_{i-1}) W_V$ (with biases), and their outputs are discarded before the next layer. Replacing the prompts by these keys and values, prepended inside attention, gives exactly the same class-token output; this is our derivation from VPT’s Eq. 7, and the example checks it numerically. VPT’s supplementary material uses the same equivalence as a cheaper test-time implementation (“VPT-prefix”), which it found reduced inference latency and memory, especially for large $p$ (§B, Fig. 10). Prefix tuning proper is the more general form: its keys and values are $2pd$ free numbers per layer, while VPT-deep’s are $pd$ numbers passed through the frozen normalization and projections.

Adapters, LoRA and prompts all add a learned term to a frozen computation. An adapter adds $W_\text{up}\sigma(W_\text{down}\mathbf{h})$ to a sublayer’s output; LoRA adds $BA\mathbf{x}$ to a linear layer’s output, a bottleneck without a nonlinearity whose input is the layer’s input rather than its output; a prefix adds terms to the attention’s weighted sum. In our reading, the differences that matter in practice are where the term enters, whether it is linear (and so can be merged into the frozen weights), and whether it lengthens the sequence.

Parameters, memory and compute

For a ViT-B/16 at $224 \times 224$ input, with $N = 12$ layers, $d = 768$ and $n = 197$ tokens, the counts follow from the module shapes. The backbone has 85,798,656 parameters (85.8M, the figure VPT and He et al. give), which agrees with Distributed Training’s count of 86,567,656 for the same model with a 1,000-class head. Its attention and MLP layers cost about $N(12nd^2 + 2n^2d)$ multiply-adds per image; adding $p$ prompt tokens raises $n$ to $n + p$ in every term, while a prefix adds $p$ only to the keys and values.

Training memory has three parts: the weights (all of them, frozen or not), the gradients and optimizer state (only for trained parameters; two moments per parameter for Adam), and the activations that backpropagation needs. Freezing removes the second part for the frozen weights and some of the third: an automatic differentiation system needs a linear layer’s input only to compute its weight gradient, so a frozen layer that only passes gradients backward can drop it. Activations of nonlinearities, normalizations and attention are still needed wherever gradients must flow through them. Distributed Training covers how the model states are sharded when even the frozen weights do not fit; mixed-precision training and quantization shrink them further.

How to Apply It

1. Start with a linear probe. It is the cheapest baseline and shows whether the frozen features already suffice.
2. Choose a method by what must be saved. If per-task storage matters most, prompts, BitFit and small adapters are the smallest; if inference must cost nothing extra, use selective methods or a low-rank update that is merged after training; if one batch must serve several tasks at once, methods whose task-specific part is separate from the backbone (prompts, prefixes, unmerged adapters or LoRA modules) allow it.
3. Tune the method’s size per task. VPT selected the prompt length per task from $\{1, 5, 10, 50, 100, 200\}$ on a validation split, and the best length differed widely between tasks (supplementary §A.1, Tables 8–9); Houlsby et al. selected the adapter size per task (§3.2).
4. Tune the learning rate separately for each method. VPT used AdamW with base rates between $10^{-4}$ and $5 \times 10^{-3}$ for full fine-tuning, biases and adapters, but SGD with base rates from 0.05 to 50 for the linear probe and prompts, and up to 1,000 for VPT-shallow on a few tasks (supplementary Table 1).
5. Train the head with the method, and save only the method’s parameters and the head per task.

Example: counting what each method trains, stores and costs

The example builds a ViT-B/16 in PyTorch with fixed random weights, standing in for a pretrained backbone (no weights are downloaded, and nothing is trained beyond one optimizer step), and applies fifteen configurations to it. Two synthetic $224 \times 224$ RGB images, a red disk and a blue square on a noisy gray background, are the batch. For each configuration it counts:

• the trainable parameters, compared with a closed-form count derived from the module shapes, and their share of the 85,798,656-parameter backbone;
• the AdamW state after one step, which should hold two numbers per trainable parameter, and whether gradients reach exactly the trainable parameters;
• the memory of float32 weights, gradients and AdamW moments (“train MB”, $4$ bytes for every parameter and $12$ more for every trainable one; 1 MB $= 10^6$ bytes);
• the bytes autograd keeps for the backward pass of the batch of two images (“saved MB”), counted with saved_tensors_hooks as unique storages that are not parameters;
• the multiply-adds of one image’s forward pass, counted with FlopCounterMode and compared with a closed form; for LoRA rows also after merging $BA$ into the frozen weights;
• how much the final class-token feature changes at initialization, relative to the frozen backbone ($\max|\Delta| / \max|f|$).

Conventions: the head has 100 classes (a choice; VPT’s tasks have 2 to 397); adapters follow VPT’s placement after the MLP at reduction rates 8, 64 and 256, and Houlsby’s two-per-block placement at size 64 with all layer normalizations trained; LoRA uses $\alpha = r$ on the query and value parts of the fused query–key–value projection, or on every linear layer; VPT-deep counts $N p d$ prompts, including the first layer’s; prefix rows count only the stored prefixes, not the reparametrization network used in training. Adapters are initialized as Houlsby et al. describe, prompts and prefixes with the Xavier-uniform scheme VPT uses (supplementary §A.1). The last check compares VPT-deep with the prefix form derived above. The script ran with PyTorch 2.14.1 on four CPU threads in about 8 to 30 seconds, depending on the load from other processes on the machine, and repeated runs gave identical output.

import copy, math
import torch, torch.nn as nn, torch.nn.functional as F
from torch.utils.flop_counter import FlopCounterMode

torch.set_num_threads(4)
D, L, H, MLP, P, C = 768, 12, 12, 3072, 16, 100       # ViT-B/16, 224 x 224 input, 100-class head

class LoRA(nn.Module):                                   # delta(x) = (alpha/r) B A x, B = 0 at start
    def __init__(s, d_in, d_out, r):
        super().__init__()
        s.A = nn.Parameter(torch.empty(r, d_in)); nn.init.kaiming_uniform_(s.A, a=math.sqrt(5))
        s.B = nn.Parameter(torch.zeros(d_out, r))
    def forward(s, x): return (x @ s.A.T) @ s.B.T     # alpha = r, so the scaling is 1

class Adapter(nn.Module):                                # h + up(GELU(down(h))), near-identity init
    def __init__(s, m):
        super().__init__(); s.down, s.up = nn.Linear(D, m), nn.Linear(m, D)
        for lin in (s.down, s.up):
            nn.init.trunc_normal_(lin.weight, std=1e-2, a=-2e-2, b=2e-2); nn.init.zeros_(lin.bias)
    def forward(s, h): return h + s.up(F.gelu(s.down(h)))

class Block(nn.Module):
    def __init__(s):
        super().__init__()
        s.n1, s.n2 = nn.LayerNorm(D, eps=1e-6), nn.LayerNorm(D, eps=1e-6)
        s.qkv, s.proj = nn.Linear(D, 3 * D), nn.Linear(D, D)
        s.fc1, s.fc2 = nn.Linear(D, MLP), nn.Linear(MLP, D)
        s.lora, s.ad_attn, s.ad_mlp = nn.ModuleDict(), None, None
    def forward(s, x, pk=None, pv=None):
        b, n, _ = x.shape
        h = s.n1(x)
        qkv = s.qkv(h)
        if "q" in s.lora:                                # LoRA on the query and value parts of the fused qkv
            qkv = qkv + torch.cat([s.lora["q"](h), torch.zeros_like(h), s.lora["v"](h)], -1)
        if "qkv" in s.lora: qkv = qkv + s.lora["qkv"](h)
        q, k, v = qkv.split(D, -1)
        if pk is not None:                               # prefix keys and values, shared by the batch
            k, v = torch.cat([pk.expand(b, -1, -1), k], 1), torch.cat([pv.expand(b, -1, -1), v], 1)
        heads = lambda t: t.reshape(b, t.shape[1], H, D // H).transpose(1, 2)
        att = (heads(q) @ heads(k).transpose(-1, -2) / math.sqrt(D // H)).softmax(-1)
        o = (att @ heads(v)).transpose(1, 2).reshape(b, n, D)
        o = s.proj(o) + (s.lora["proj"](o) if "proj" in s.lora else 0)
        if s.ad_attn is not None: o = s.ad_attn(o)
        x = x + o
        h = s.n2(x)
        u = s.fc1(h) + (s.lora["fc1"](h) if "fc1" in s.lora else 0)
        u = F.gelu(u)
        m = s.fc2(u) + (s.lora["fc2"](u) if "fc2" in s.lora else 0)
        if s.ad_mlp is not None: m = s.ad_mlp(m)
        return x + m

class ViT(nn.Module):
    def __init__(s):
        super().__init__()
        s.patch = nn.Conv2d(3, D, P, P)
        s.cls = nn.Parameter(torch.randn(1, 1, D) * .02); s.pos = nn.Parameter(torch.randn(1, 197, D) * .02)
        s.blocks = nn.ModuleList(Block() for _ in range(L))
        s.norm, s.head = nn.LayerNorm(D, eps=1e-6), nn.Linear(D, C)
        s.prompts, s.deep, s.prefix_k, s.prefix_v = None, False, None, None
    def features(s, img, prefix=None):
        t = s.patch(img).flatten(2).transpose(1, 2)
        t = torch.cat([s.cls.expand(len(t), -1, -1), t], 1) + s.pos
        p = 0 if s.prompts is None else s.prompts.shape[1]
        for i, blk in enumerate(s.blocks):
            if p and (i == 0 or s.deep):                 # VPT: insert (shallow) or replace (deep) the prompt tokens
                rest = t[:, 1:] if i == 0 else t[:, 1 + p:]
                t = torch.cat([t[:, :1], s.prompts[i if s.deep else 0].expand(len(t), -1, -1), rest], 1)
            kv = prefix[i] if prefix is not None else (
                (s.prefix_k[i], s.prefix_v[i]) if s.prefix_k is not None else (None, None))
            t = blk(t, *kv)
        return s.norm(t)[:, 0]
    def forward(s, img): return s.head(s.features(img))

def make_images():                                       # a disk and a square on a noisy grey background
    g = torch.Generator().manual_seed(0)
    yy, xx = torch.meshgrid(torch.arange(224.), torch.arange(224.), indexing="ij")
    disk = (yy - 100) ** 2 + (xx - 120) ** 2 <= 60 ** 2
    square = ((yy - 112).abs() <= 50) & ((xx - 100).abs() <= 50)
    imgs = 0.5 + 0.05 * torch.randn(2, 3, 224, 224, generator=g)
    for i, (m, col) in enumerate(((disk, (0.9, 0.2, 0.2)), (square, (0.2, 0.3, 0.9)))):
        for c in range(3): imgs[i, c][m] = col[c]
    return imgs.clamp(0, 1), (disk.float().mean().item(), square.float().mean().item())

def apply(model, method):
    """Freeze the backbone, add the method's modules (seed 1) and mark what trains."""
    torch.manual_seed(1)
    kind, arg = method
    for prm in model.parameters(): prm.requires_grad_(False)
    if kind == "adapter":                                # one adapter after the MLP (VPT's configuration)
        for b in model.blocks: b.ad_mlp = Adapter(D // arg)
    if kind == "houlsby":                                # two adapters per block, LayerNorms trained
        for b in model.blocks: b.ad_attn, b.ad_mlp = Adapter(arg), Adapter(arg)
        for mod in model.modules():
            if isinstance(mod, nn.LayerNorm): mod.requires_grad_(True)
    if kind == "lora_qv":
        for b in model.blocks: b.lora.update({"q": LoRA(D, D, arg), "v": LoRA(D, D, arg)})
    if kind == "lora_all":
        for b in model.blocks:
            b.lora.update({"qkv": LoRA(D, 3 * D, arg), "proj": LoRA(D, D, arg), "fc1": LoRA(D, MLP, arg), "fc2": LoRA(MLP, D, arg)})
    if kind == "prefix":
        model.prefix_k = nn.ParameterList(nn.init.xavier_uniform_(torch.empty(arg, D)) for _ in range(L))
        model.prefix_v = nn.ParameterList(nn.init.xavier_uniform_(torch.empty(arg, D)) for _ in range(L))
    if kind in ("vpt_shallow", "vpt_deep"):
        model.deep = kind == "vpt_deep"
        model.prompts = nn.Parameter(nn.init.xavier_uniform_(torch.empty(L if model.deep else 1, arg, D)))
    if kind == "full": model.requires_grad_(True)
    if kind == "partial": model.blocks[-1].requires_grad_(True); model.norm.requires_grad_(True)
    if kind == "bitfit":
        for name, prm in model.named_parameters():
            if name.endswith("bias"): prm.requires_grad_(True)
    model.head.requires_grad_(True)
    return model

# closed forms (our derivation from the module shapes)
BLOCK = 4 * D + (3 * D * D + 3 * D) + (D * D + D) + (2 * D * MLP + MLP + D)
BACKBONE = 3 * P * P * D + D + D + 197 * D + L * BLOCK + 2 * D
HEAD = D * C + C
def closed_params(kind, a):
    extra = {"full": lambda: BACKBONE, "linear": lambda: 0, "partial": lambda: BLOCK + 2 * D,
             "bitfit": lambda: D + L * (7 * D + MLP) + D,
             "adapter": lambda: L * (2 * D * (D // a) + D // a + D),
             "houlsby": lambda: L * 2 * (2 * D * a + a + D) + L * 4 * D + 2 * D,
             "lora_qv": lambda: L * 2 * a * 2 * D, "lora_all": lambda: L * a * (4 * D + 2 * D + 2 * (D + MLP)),
             "prefix": lambda: L * 2 * a * D, "vpt_shallow": lambda: a * D, "vpt_deep": lambda: L * a * D}[kind]()
    return extra + HEAD
def closed_macs(kind, a, merged=False):
    n = 197 + (a if kind.startswith("vpt") else 0); m = n + (a if kind == "prefix" else 0)
    blk = n * D * 3 * D + 2 * n * m * D + n * D * D + 2 * n * D * MLP
    if kind == "adapter": blk += n * 2 * D * (D // a)
    if kind == "houlsby": blk += 2 * n * 2 * D * a
    if kind == "lora_qv" and not merged: blk += 2 * n * 2 * D * a
    if kind == "lora_all" and not merged: blk += n * a * (4 * D + 2 * D + 2 * (D + MLP))
    return 196 * D * 3 * P * P + L * blk + D * C

def saved_bytes(model, x, y):
    """Bytes of the unique non-parameter storages autograd keeps for backward."""
    params = {p.untyped_storage().data_ptr() for p in model.parameters()}
    seen = {}
    def pack(t):
        st = t.untyped_storage()
        if st.data_ptr() not in params: seen[st.data_ptr()] = st.nbytes()
        return t
    with torch.autograd.graph.saved_tensors_hooks(pack, lambda t: t):
        loss = F.cross_entropy(model(x), y)
    return loss, sum(seen.values())

def macs(model, x):
    with torch.no_grad(), FlopCounterMode(display=False) as fc: model(x)
    return fc.get_total_flops() // 2

imgs, frac = make_images()
x, y = (imgs - 0.5) / 0.5, torch.tensor([0, 1])
print(f"data: {tuple(imgs.shape)}, values {imgs.min():.3f}..{imgs.max():.3f}, shape pixels {frac[0]:.3f} (disk), {frac[1]:.3f} (square)")
torch.manual_seed(0)
base = ViT().eval()                                      # stand-in for a pretrained backbone: fixed random weights
with torch.no_grad(): f_base = base.features(x)
base_macs = macs(base, x[:1])

METHODS = [("full", None), ("linear", None), ("partial", 1), ("bitfit", None),
           ("adapter", 8), ("adapter", 64), ("adapter", 256), ("houlsby", 64),
           ("lora_qv", 8), ("lora_qv", 64), ("lora_all", 8), ("prefix", 10),
           ("vpt_shallow", 50), ("vpt_deep", 50), ("vpt_deep", 200)]
print(f"\n{'method':15s} {'trainable':>11s} {'% of bb':>8s} {'train MB':>9s} {'saved MB':>9s} {'GMAC':>6s} {'init change':>12s}")
fails = {k: [] for k in ("C1", "C2", "C3", "C4")}
for kind, a in METHODS:
    name = kind + ("" if a is None else f" {a}")
    model = apply(copy.deepcopy(base), (kind, a))
    trainable = [p for p in model.parameters() if p.requires_grad]
    n_tr = sum(p.numel() for p in trainable); n_all = sum(p.numel() for p in model.parameters())
    if n_tr != closed_params(kind, a): fails["C1"].append(name)
    with torch.no_grad(): f0 = model.features(x)
    change = ((f0 - f_base).abs().max() / f_base.abs().max()).item()
    if kind in ("full", "linear", "partial", "bitfit", "lora_qv", "lora_all") and change != 0.0: fails["C4"].append(name)
    n_macs = macs(model, x[:1])
    if n_macs != closed_macs(kind, a): fails["C3"].append(name)
    if kind.startswith("lora"):                          # merge W + BA into the frozen weights, drop the LoRA modules
        merged = copy.deepcopy(model)
        with torch.no_grad():
            for b in merged.blocks:
                for key, lin, rows in (("q", b.qkv, slice(0, D)), ("v", b.qkv, slice(2 * D, 3 * D)), ("qkv", b.qkv, slice(None)),
                                       ("proj", b.proj, slice(None)), ("fc1", b.fc1, slice(None)), ("fc2", b.fc2, slice(None))):
                    if key in b.lora: lin.weight[rows] += b.lora[key].B @ b.lora[key].A
                b.lora = nn.ModuleDict()
        if macs(merged, x[:1]) != base_macs or base_macs != closed_macs(kind, a, merged=True): fails["C3"].append(name + " merged")
    model.train()
    opt = torch.optim.AdamW(trainable, lr=1e-3, weight_decay=0.05)
    loss, sb = saved_bytes(model, x, y)
    loss.backward(); opt.step()
    state = sum(st["exp_avg"].numel() + st["exp_avg_sq"].numel() for st in opt.state.values())
    grads_ok = all(p.grad is not None for p in trainable) and all(
        p.grad is None for p in model.parameters() if not p.requires_grad)
    if state != 2 * n_tr or not grads_ok: fails["C2"].append(name)
    train_mb = (4 * n_all + 12 * n_tr) / 1e6             # fp32 weights + gradients + two AdamW moments
    print(f"{name:15s} {n_tr:11,d} {100 * n_tr / BACKBONE:7.3f}% {train_mb:9.1f} {sb / 1e6:9.1f} "
          f"{n_macs / 1e9:6.2f} {change:12.2e}")

print(f"\nbackbone parameters {BACKBONE:,}; saved MB are for the batch of 2 images")
for k, desc in (("C1", "counted trainable = closed form"), ("C2", "AdamW state = 2 x trainable, grads only on trainable"),
                ("C3", "counted MACs = closed form (LoRA merged = base)"), ("C4", "zero init change where expected")):
    print(f"{k} {desc}: {'all rows pass' if not fails[k] else 'FAILED for ' + ', '.join(fails[k])}")

# C5: VPT-deep equals prefix tuning whose keys and values are the prompts' projections
model = apply(copy.deepcopy(base), ("vpt_deep", 50)).eval()
with torch.no_grad():
    f_vpt = model.features(x)
    pre = [b.qkv(b.n1(model.prompts[i])).split(D, -1)[1:] for i, b in enumerate(model.blocks)]
    model_p = copy.deepcopy(model); model_p.prompts = None
    f_pre = model_p.features(x, prefix=pre)
rel = ((f_vpt - f_pre).abs().max() / f_vpt.abs().max()).item()
print(f"C5 VPT-deep vs prefix form, relative max difference {rel:.1e} ({'pass' if rel <= 1e-5 else 'FAILED'}, tolerance 1e-5)")

data: (2, 3, 224, 224), values 0.200..0.900, shape pixels 0.225 (disk), 0.203 (square)

method            trainable  % of bb  train MB  saved MB   GMAC  init change
full             85,875,556 100.090%    1374.0     280.8  17.56     0.00e+00
linear               76,900   0.090%     344.4       1.2  17.56     0.00e+00
partial 1         7,166,308   8.352%     429.5      25.5  17.56     0.00e+00
bitfit              179,812   0.210%     345.7     179.1  17.56     0.00e+00
adapter 8         1,856,740   2.164%     372.9     181.5  17.91     1.34e-02
adapter 64          307,444   0.358%     348.1     178.3  17.61     5.83e-03
adapter 256         141,448   0.165%     345.5     177.9  17.57     2.84e-03
houlsby 64        2,494,564   2.907%     383.0     211.8  18.03     1.85e-02
lora_qv 8           371,812   0.433%     349.1     191.5  17.62     0.00e+00
lora_qv 64        2,436,196   2.839%     382.2     193.7  18.03     0.00e+00
lora_all 8        1,256,548   1.465%     363.3     279.0  17.80     0.00e+00
prefix 10           261,220   0.304%     347.4     179.2  17.60     7.92e-02
vpt_shallow 50      115,300   0.134%     345.0     237.3  22.22     1.89e-01
vpt_deep 50         537,700   0.627%     351.8     237.3  22.22     3.79e-01
vpt_deep 200      1,920,100   2.238%     373.9     450.0  36.74     8.29e-01

backbone parameters 85,798,656; saved MB are for the batch of 2 images
C1 counted trainable = closed form: all rows pass
C2 AdamW state = 2 x trainable, grads only on trainable: all rows pass
C3 counted MACs = closed form (LoRA merged = base): all rows pass
C4 zero init change where expected: all rows pass
C5 VPT-deep vs prefix form, relative max difference 1.1e-06 (pass, tolerance 1e-5)

All five checks pass: the counts follow from the shapes, AdamW keeps exactly two moments per trainable parameter and none for frozen ones, the multiply-add counts match the closed form, LoRA after merging costs the same as the backbone, and VPT-deep and its prefix form agree to float32 rounding. The parameter-efficient rows train between 0.09% and 2.9% of the backbone.

Storage and training state. The trainable fractions differ by a factor of about 30 across the parameter-efficient rows, but the training-state memory differs by little: every one of them needs between 344 and 383 MB, against 1,374 MB for full fine-tuning, because the 343 MB of frozen float32 weights dominate once the gradients and the two AdamW moments of 86 million parameters are gone. Partial fine-tuning of one block already trains 8.4% and costs 430 MB.

Activations kept for backward: hypothesis H4 failed in part. The plan written before the pilot stated H4 as “linear probe < 1% of full’s; VPT and prefix rows >= full’s x 0.9 (gradients must reach the input of block 1); BitFit, adapters, LoRA between 30% and 110% of full’s”. The first and third parts held; the second failed: prefix tuning kept 64% of full fine-tuning’s 280.8 MB and VPT with 50 prompts 85%, below the planned 90%. What the column shows follows from the code’s construction. The linear probe keeps only the final normalized tokens (the class-token feature is a slice of a $2 \times 197 \times 768$ float32 tensor, 1.2 MB); partial fine-tuning keeps one block’s activations. Every other method has a trained parameter in or before the first block and must send gradients through all twelve blocks. BitFit, the adapters, LoRA on query and value, and prefix tuning keep 178 to 212 MB, about two-thirds to three-quarters of full fine-tuning’s; in our reading of the counts, the difference is the inputs of the frozen linear layers, which are needed only for weight gradients, since LoRA on every linear layer, which needs each layer’s input for its own $A$ gradient, keeps almost as much as full fine-tuning (279.0 MB). The prompt rows run on 247 or 397 tokens instead of 197, so VPT-deep with 200 prompts keeps 1.6 times as much as full fine-tuning. VPT’s measured peak training memory on a GPU shows the same pattern at batch 64: 11.7 GB for full fine-tuning, 10.1 GB for biases, 9.8 to 9.9 GB for adapters, 10.3 GB for one prompt and 25.8 GB for 200 prompts, against 0.9 GB for the linear probe and 1.4 GB for partial-1 (supplementary Table 7).

Inference. Merged LoRA and the selective methods leave the 17.56 GMAC of the backbone unchanged. Adapters add 0.06% to 2.7%, unmerged LoRA 0.3% to 2.7%, a prefix of 10 adds 0.2%, and prompts, which lengthen every layer’s sequence, add 26.5% at $p = 50$ and more than double the cost at $p = 200$. VPT measured inference latency and memory roughly doubling at 200 prompts (supplementary §B, Table 7).

Initialization. LoRA (with $B = 0$) and the selective methods start exactly at the backbone’s function. Adapters with near-zero projections change the feature by 0.3% to 1.9% of its largest value; prefixes and prompts change it by 8% to 83%, since the real tokens attend to the new vectors from the first step.

The second script recomputes the parameter percentages the papers print. It uses the class counts and per-task prompt lengths of VPT’s supplementary Tables 2, 8 and 9, BERT’s dimensions from Hugging Face’s model table and default configuration (12 layers of width 768 for BERT-base, 24 of width 1,024 for BERT-large, a vocabulary of 30,522 and 512 positions, assuming BERT-large’s intermediate width is four times its hidden width, as BERT-base’s 3,072 is), and GPT-2 Medium’s 24 layers of width 1,024 from the same table. It runs in well under a second with Python 3 alone.

D, L, MLP = 768, 12, 3072
BLOCK = 4 * D + (3 * D * D + 3 * D) + (D * D + D) + (2 * D * MLP + MLP + D)
BB = 3 * 16 * 16 * D + D + D + 197 * D + L * BLOCK + 2 * D       # ViT-B/16 backbone, 85,798,656
BIAS = D + L * (7 * D + MLP) + D
head = lambda c: c * D + c

def agree(value, printed):                     # value shown at the printed precision equals the printed value
    dec = len(printed.split(".")[1]) if "." in printed else 0
    return f"{value:.{dec}f}" == printed

# 1. VPT, Sec. 3.2: 50 prompts on ViT-Base, "0.04%" (shallow) and "0.53%" (deep)
for name, n, printed in (("shallow", 50 * D, "0.04"), ("deep", L * 50 * D, "0.53")):
    v = 100 * n / BB
    print(f"VPT 3.2 {name}: {n:,} params = {v:.3f}% of {BB:,} ->", "agrees" if agree(v, printed) else f"differs from {printed}%")
print(f"         rounded inputs, 0.46M / 86M = {100 * 0.46 / 86:.3f}%")

# 2. VPT supplement, Tables 8 and 9: prompt length and "Tuned / Total (%)" per task (VTAB-1k, then FGVC)
tasks = [  # (task, classes, shallow p, shallow %, deep p, deep %)
    ("CIFAR-100", 100, 100, "0.18", 10, "0.20"), ("Caltech101", 102, 5, "0.10", 10, "0.20"),
    ("DTD", 47, 1, "0.04", 10, "0.15"), ("Flowers102", 102, 200, "0.27", 1, "0.10"),
    ("Pets", 37, 50, "0.08", 1, "0.04"), ("SVHN", 10, 200, "0.19", 50, "0.54"),
    ("Sun397", 397, 1, "0.36", 5, "0.41"), ("Patch Camelyon", 2, 5, "0.01", 100, "1.06"),
    ("EuroSAT", 10, 50, "0.05", 100, "1.07"), ("Resisc45", 45, 50, "0.09", 10, "0.15"),
    ("Retinopathy", 5, 10, "0.01", 1, "0.02"), ("Clevr/count", 8, 100, "0.10", 50, "0.54"),
    ("Clevr/distance", 6, 200, "0.18", 200, "2.11"), ("DMLab", 6, 100, "0.09", 100, "1.07"),
    ("KITTI/distance", 4, 100, "0.09", 50, "0.54"), ("dSprites/location", 16, 100, "0.10", 10, "0.12"),
    ("dSprites/orientation", 16, 100, "0.10", 50, "0.55"), ("SmallNORB/azimuth", 18, 200, "0.19", 200, "2.12"),
    ("SmallNORB/elevation", 9, 200, "0.19", 200, "2.11"),
    ("CUB-200-2011", 200, 100, "0.31", 10, "0.29"), ("NABirds", None, 50, "0.54", 50, "1.02"),
    ("Oxford Flowers", 102, 100, "0.23", 5, "0.14"), ("Stanford Dogs", 120, 100, "0.20", 100, "1.17"),
    ("Stanford Cars", 196, 100, "0.26", 200, "2.27")]
print("\nVPT Tables 8-9, 48 per-task percentages (24 tasks x shallow/deep):")
for nabirds in (55, 555):
    rows = [(t, nabirds if c is None else c, *rest) for t, c, *rest in tasks]
    for conv, denom in (("tuned / backbone", lambda t: BB), ("tuned / (backbone + tuned)", lambda t: BB + t)):
        miss = []
        for t, c, ps, vs, pd, vd in rows:
            for var, n, printed in (("shallow", ps * D, vs), ("deep", L * pd * D, vd)):
                tuned = n + head(c)
                v = 100 * tuned / denom(tuned)
                if not agree(v, printed): miss.append(f"{t} {var} {v:.3f} vs {printed}")
        print(f"  NABirds {nabirds:3d} classes, {conv:27s}: {48 - len(miss)}/48 agree" + ("; " + "; ".join(miss) if len(miss) <= 3 else ""))
    # Table 1, "Total params" for all 24 tasks, relative to one backbone
    hs = sum(head(c) for _, c, *_ in rows)
    sh = sum(r[2] * D for r in rows)
    dp = sum(L * r[4] * D for r in rows)
    mult = [("Full", (24 * BB + hs) / BB, "24.02"), ("Linear", (BB + hs) / BB, "1.02"),
            ("Partial-1", (BB + 24 * (BLOCK + 2 * D) + hs) / BB, "3.00"), ("Bias", (BB + 24 * BIAS + hs) / BB, "1.05"),
            ("VPT-shallow", (BB + sh + hs) / BB, "1.04"), ("VPT-deep", (BB + dp + hs) / BB, "1.18")]
    print(f"  Table 1 totals, NABirds {nabirds}: " + ", ".join(
        f"{k} {v:.3f}{'' if agree(v, p) else ' (printed ' + p + ')'}" for k, v, p in mult))

# 3. BitFit (BERT, HuggingFace layout: embeddings, encoder, pooler)
def bert(d, l, ffn):
    emb = 30522 * d + 512 * d + 2 * d + 2 * d
    layer = 4 * (d * d + d) + 2 * d + (d * ffn + ffn) + (ffn * d + d) + 2 * d
    return emb + l * layer + d * d + d, l * (7 * d + ffn), l * (7 * d + ffn) + 2 * d
print("\nBitFit:")
for name, d, l, ffn, printed in (("BERT-base", 768, 12, 3072, "0.09"), ("BERT-large", 1024, 24, 4096, "0.08")):
    tot, enc_b, all_b = bert(d, l, ffn)
    for which, b in (("encoder biases", enc_b), ("all biases", all_b)):
        v = 100 * b / tot
        print(f"  {name} {which}: {b:,} of {tot:,} = {v:.4f}% ->", "agrees" if agree(v, printed) else f"differs from {printed}%")
tot = bert(768, 12, 3072)[0]
for which, b, printed in (("query + intermediate", 12 * (D + MLP), "0.04"), ("intermediate", 12 * MLP, "0.03"), ("query", 12 * D, "0.01")):
    v = 100 * b / tot
    print(f"  BERT-base {which} biases: {b:,} = {v:.4f}% ->", "agrees" if agree(v, printed) else f"differs from {printed}%")

# 4. Houlsby et al.: adapters of size 64, two per layer, 2md + d + m parameters each, plus LayerNorms
print("\nAdapters (Houlsby et al.):")
for name, d, l, ffn in (("BERT-base", 768, 12, 3072), ("BERT-large", 1024, 24, 4096)):
    tot = bert(d, l, ffn)[0]
    ad, ln = l * 2 * (2 * 64 * d + d + 64), l * 4 * d + 2 * d
    vs330 = f" and {100 * (ad + ln) / 330e6:.2f}% of 330M" if name == "BERT-large" else ""
    print(f"  {name}: adapters {ad:,}, LayerNorms {ln:,}; with LayerNorms {100 * (ad + ln) / tot:.2f}% of {tot:,}"
          f"{vs330}; adapters alone {100 * ad / tot:.2f}%")
print("  printed: BERT-large size 64 '2.1%'; BERT-base LayerNorms '40k'; BERT-base size 64 'approximately 2M'")

# 5. Prefix-tuning, GPT-2 Medium (24 layers, width 1024): a key and a value per prefix position and layer
print("\nPrefix-tuning (GPT-2 Medium):")
for task, p, printed in (("E2E", 5, "250K"), ("WebNLG", 5, "250K"), ("DART", 10, "500K")):
    n = 24 * 2 * 1024 * p
    print(f"  {task}: prefix length {p}, {n:,} parameters (printed {printed}), {100 * n / 345e6:.3f}% of 345M (printed 0.1%)")

VPT 3.2 shallow: 38,400 params = 0.045% of 85,798,656 -> agrees
VPT 3.2 deep: 460,800 params = 0.537% of 85,798,656 -> differs from 0.53%
         rounded inputs, 0.46M / 86M = 0.535%

VPT Tables 8-9, 48 per-task percentages (24 tasks x shallow/deep):
  NABirds  55 classes, tuned / backbone           : 32/48 agree
  NABirds  55 classes, tuned / (backbone + tuned) : 44/48 agree
  Table 1 totals, NABirds 55: Full 24.014 (printed 24.02), Linear 1.014 (printed 1.02), Partial-1 2.998, Bias 1.043 (printed 1.05), VPT-shallow 1.034 (printed 1.04), VPT-deep 1.178
  NABirds 555 classes, tuned / backbone           : 33/48 agree
  NABirds 555 classes, tuned / (backbone + tuned) : 46/48 agree; CUB-200-2011 shallow 0.268 vs 0.31; Oxford Flowers shallow 0.181 vs 0.23
  Table 1 totals, NABirds 555: Full 24.019, Linear 1.019, Partial-1 3.002, Bias 1.048, VPT-shallow 1.039, VPT-deep 1.183

BitFit:
  BERT-base encoder biases: 101,376 of 109,482,240 = 0.0926% -> agrees
  BERT-base all biases: 102,912 of 109,482,240 = 0.0940% -> agrees
  BERT-large encoder biases: 270,336 of 335,141,888 = 0.0807% -> agrees
  BERT-large all biases: 272,384 of 335,141,888 = 0.0813% -> agrees
  BERT-base query + intermediate biases: 46,080 = 0.0421% -> agrees
  BERT-base intermediate biases: 36,864 = 0.0337% -> agrees
  BERT-base query biases: 9,216 = 0.0084% -> agrees

Adapters (Houlsby et al.):
  BERT-base: adapters 2,379,264, LayerNorms 38,400; with LayerNorms 2.21% of 109,482,240; adapters alone 2.17%
  BERT-large: adapters 6,343,680, LayerNorms 100,352; with LayerNorms 1.92% of 335,141,888 and 1.95% of 330M; adapters alone 1.89%
  printed: BERT-large size 64 '2.1%'; BERT-base LayerNorms '40k'; BERT-base size 64 'approximately 2M'

Prefix-tuning (GPT-2 Medium):
  E2E: prefix length 5, 245,760 parameters (printed 250K), 0.071% of 345M (printed 0.1%)
  WebNLG: prefix length 5, 245,760 parameters (printed 250K), 0.071% of 345M (printed 0.1%)
  DART: prefix length 10, 491,520 parameters (printed 500K), 0.142% of 345M (printed 0.1%)

The counting conventions explain most of the published percentages, and a few do not match:

• VPT’s “0.53%”. 460,800 deep prompts are 0.537% of the 85,798,656-parameter backbone, which rounds to 0.54%; the printed 0.53% is what the rounded inputs, 0.46M of “86 million”, give.
• VPT’s denominator. The per-task “Tuned / Total” percentages of the supplementary Tables 8 and 9 count the prompts and the head, and divide by the backbone plus the tuned parameters: that convention reproduces 46 of the 48 values, against 33 for dividing by the backbone alone. The two that do not match are VPT-shallow on CUB-200-2011 (0.268% computed, 0.31% printed) and Oxford Flowers (0.181% against 0.23%), both with 100 prompts.
• NABirds’ class count. The supplementary Table 2 lists NABirds with 55 classes. With 55, neither NABirds percentage and four of six “Total params” multiples of Table 1 come out as printed; with 555 classes, both NABirds percentages and all six multiples do. In our reading, the printed 55 is a typo for 555, the count with which the paper’s percentages and multiples come out.
• BitFit. All seven percentages agree, whether or not the embedding layer-normalization and pooler biases are counted.
• Houlsby et al.‘s “2.1%”. For BERT-large with adapters of size 64, two per layer, plus the layer normalizations, the count is 1.92% of the computed 335 million parameters or 1.95% of the paper’s 330M, not the 2.1% of its Table 1; reading “per layer” literally (one module per layer) would halve the adapters’ 1.89%. The “40k” layer-normalization parameters of BERT-base (38,400) and its “approximately 2M” adapter parameters at size 64 (2.4 million) agree at the precision printed.
• Prefix tuning’s “0.1%”. The 250K and 500K of footnote 7 agree with $2 \times 24 \times 1{,}024 \times p$ for the prefix lengths of the appendix, but “0.1%” of GPT-2 Medium holds only for length 10 (0.142%); length 5 gives 0.071%. The 345M is also the model’s original count, which OpenAI’s repository later called wrong (gpt-2 README), giving 355 million (gpt-2 model card).

Variants

• Adapter variants for vision transformers. He et al. (2023) compare adapters with bottlenecks 8 and 32, AdaptFormer and NOAH on VTAB-1k and FGVC (Table 1); the original papers of the last two were not read. Sung et al. (2022) compare Hyperformer, which generates adapter weights from task embeddings with a hypernetwork, and Compacter, which reduces adapter parameters with Kronecker products and a low-rank parameterization (§1).
• Prompt variants. VPT’s supplementary material tests prompts shared within or across layers, prompts initialized from class-mean features, and a test-time prefix implementation (§B), and VPT combined with bias tuning (Table 4). Prompt learning for CLIP’s text encoder, CoOp and CoCoOp, is covered in the Prompt Engineering article.
• Task-specific selection. SPT allocates a fixed budget of trainable parameters per task by sensitivity (He et al., 2023).
• Dense prediction. VPT applied to SETR on ADE20K segmentation tunes the decoder head and the prompts (Jia et al., 2022, §5, Table 3).

Best Practices

• Compare against a linear probe and full fine-tuning on the same data, each with its own learning rate; VPT’s tuned rates differed by orders of magnitude between methods (supplementary Table 1).
• Tune the method’s size per task, since the best prompt length and adapter size vary between tasks (Jia et al., 2022, Fig. 6; Houlsby et al., 2019, §3.2).
• Report the counting convention with any percentage: whether the head is included and what the denominator is. In the example, one convention reproduced 46 of VPT’s 48 per-task values and the other 33.
• Measure training memory rather than infer it from trainable parameters. In VPT’s measurements, biases, adapters and prompts needed most of full fine-tuning’s peak training memory, and in the example the methods with trained parameters in the first block kept two-thirds or more of its activations.
• Merge linear updates for single-task deployment, and keep task-specific parts separate when one backbone serves several tasks.
• Keep the start at the pretrained function where the method allows it (zero-initialized LoRA, near-zero adapters); Houlsby et al. found large adapter initializations harmful (§3.6).

Pitfalls

• Equating fewer trainable parameters with cheaper training. Gradients still pass through the frozen backbone; VPT-deep with 200 prompts needed more than twice the training memory of full fine-tuning in VPT’s measurements (supplementary Table 7).
• Ignoring inference cost. Prompts lengthen every layer’s sequence and adapters add layers; only merged linear updates and selective methods cost nothing extra at inference.
• Comparing percentages computed with different conventions. Whether the head and the tuned parameters enter the denominator changes the second decimal, as the example shows for VPT’s tables.
• Transferring results across backbones and pretraining objectives. Rankings obtained on a supervised ImageNet-21k ViT-B did not hold on MAE and MoCo v3 backbones (Jia et al., 2022, Table 4; He et al., 2023, Table 2) or on ResNet-50 (Jia et al., 2022, Table 5).
• Expecting prompt tuning to work on small models. Lester et al. (2021) found it competitive only at billions of parameters, and on CLIP-BART, prompts updating 2.00% of the parameters averaged 59.0 against 77.6 for full fine-tuning (Sung et al., 2022, Table 1).

Evidence and Impact

Language models. On GLUE with BERT-large, adapters came within 0.4 points of full fine-tuning (80.0 against 80.4) while adding 3.6% parameters per task, and needed 1.3 times the parameters of one model for all nine tasks instead of nine times (Houlsby et al., 2019, Table 1); on 17 further classification tasks with BERT-base, adapters averaged 73.3 against 73.7 with 1.14% new parameters per task (Table 2). Prefix tuning, with 0.1% of the parameters, was comparable to fine-tuning on table-to-text generation with full data, better in low-data settings and better on topics unseen in training (Li & Liang, 2021, abstract, §6). BitFit matched full fine-tuning on the GLUE validation sets with BERT-large on average (Ben-Zaken et al., 2022, Table 1). Prompt tuning closed the gap to model tuning only at the largest model sizes (Lester et al., 2021, §3.1).

Vision transformers. On ViT-B/16 pretrained with labels on ImageNet-21k, VPT-deep averaged 89.11% top-1 accuracy over five fine-grained datasets (FGVC) against 88.54% for full fine-tuning, and on the three VTAB-1k groups 78.48, 82.43 and 54.98 against 75.88, 83.36 and 47.64, beating full fine-tuning on 20 of the 24 tasks; biases reached 88.41% on FGVC and adapters 85.66%, with the linear probe at 79.32% (Jia et al., 2022, Table 1). On ViT-L and ViT-H, VPT-deep still clearly beat full fine-tuning on the Natural and Structured groups and was nearly equal on Specialized (Fig. 4, §4.2); on Swin-B, full fine-tuning was best overall, with VPT ahead of the other parameter-efficient methods (Table 2). He et al. (2023) collected later methods on the same backbone, by mean VTAB-1k accuracy: full fine-tuning 69.0, VPT-deep 72.0, adapters with bottleneck 8 73.9, AdaptFormer 74.8, LoRA at rank 8 74.9, NOAH 75.5 and their SPT-LoRA 76.4, with biases at 65.2 and the linear probe at 57.6 (Table 1).

Self-supervised backbones. With MAE and MoCo v3 pretraining, VPT found other methods, such as partial-1, ahead of VPT on MAE, and VPT no longer the best on MoCo v3 (Jia et al., 2022, §5, Table 4). He et al. write that earlier parameter-efficient methods were inferior to full fine-tuning on both, with inconsistent rankings between them; in their Table 2 all are below it except LoRA on MAE (rank 8 equal at 64.3, rank 16 at 64.8), and their SPT variants exceeded full fine-tuning on both (§4, Table 2).

Vision–language models. On CLIP-BART, trained jointly on VQAv2, GQA, NLVR2 and COCO captioning, a single adapter shared across tasks updated 4.18% of the parameters and averaged 77.4 against 77.6 for full fine-tuning, ahead of Hyperformer (76.4), Compacter (75.8), LoRA (76.5) and prompt tuning (59.0) (Sung et al., 2022, Table 1). The LoRA article reports results of low-rank updates on VL-BART and LLaVA-1.5, including DoRA’s.

Impact. In our reading, the family’s main effect in vision has been to make one frozen backbone serve many tasks; foundation models describes models released to be used as frozen backbones with prompts, probes or small adapters on top.