LoRA

LoRA, short for low-rank adaptation, adapts a pretrained network to a new task without changing its weights. For each chosen weight matrix $W_0$, it freezes $W_0$ and learns an update of low rank, written as the product $BA$ of two thin matrices, so that the adapted layer computes $(W_0 + BA)\mathbf{x}$. Only $B$ and $A$ are trained and stored per task. After training, $BA$ can be added into $W_0$, and the adapted network then runs exactly as fast as the original. LoRA is a widely used member of the family of parameter-efficient methods that fine-tuning and transfer learning summarize; it was introduced for large language models by Hu et al. (2022), whose paper was not read for this article. The method is described here from the authors’ official repository and code, from DoRA (Liu et al., 2024), which restates it, and from vision papers that use it.

Purpose

Full fine-tuning trains every weight and so stores a full copy of the model for every task. For a model with billions of parameters this is costly to train, because the optimizer keeps state for every parameter, and costly to serve, because each task needs its own copy. The repository’s README states the aim directly: LoRA reduces the number of trainable parameters by learning pairs of rank-decomposition matrices while freezing the original weights, which cuts the storage needed per task and allows switching between tasks during deployment without adding inference latency (Microsoft, LoRA README).

The README’s tables show the scale of the saving. On the GLUE language benchmark, LoRA on RoBERTa-base trained 0.8 million parameters against 125 million for full fine-tuning, with an average score of 87.24 against 86.40; on DeBERTa XXL it trained 4.7 million against 1.5 billion, averaging 91.32 against 91.06. The full fine-tuning numbers there are taken from the original RoBERTa and DeBERTa papers, not rerun. The released GPT-2 Medium LoRA checkpoints are 1.5 MB each.

Two other families do the same job. Adapters insert small trained modules into the network, and prompt-based methods add trained tokens to the input. DoRA’s authors group LoRA and its variants as a third family, distinguished by adding no inference cost, while the first two change the architecture or the input and so increase inference latency (Liu et al., 2024, §2). Parameter-efficient fine-tuning compares the families.

When to Use It

LoRA suits a pretrained model that must be adapted to one or many tasks when storing or training a full copy per task is impractical. Typical cases in vision are a large vision transformer adapted to many small classification datasets, a vision–language model instruction-tuned for new tasks, and a text-to-image diffusion model taught a subject or a style, where the shared base model is downloaded once and the per-subject files are small.

In our reading, it is less suited when:

• the task is far from what the pretrained model represents and plenty of labelled data exist, so that a full update can be learned and the rank restriction only limits it;
• the model is small enough that full fine-tuning and storing copies is cheap;
• inference must serve many tasks in one batch with merged weights, since a merged weight belongs to one task (unmerged serving keeps the low-rank path separate, at some extra cost).

The evidence on how well LoRA matches full fine-tuning is mixed and depends on the backbone and the task; it is summarized in Evidence and Impact below.

How It Works

The update

For a pretrained weight matrix $W_0 \in \mathbb{R}^{d \times k}$, which maps a $k$-dimensional input to a $d$-dimensional output, LoRA writes the fine-tuned weight as

$$W' = W_0 + \Delta W = W_0 + BA, \qquad B \in \mathbb{R}^{d \times r},\; A \in \mathbb{R}^{r \times k},\; r \ll \min(d, k),$$

with $W_0$ frozen and only $B$ and $A$ trained (Liu et al., 2024, Eq. 1). The rank of $\Delta W$ is at most $r$, and the pair holds $r(d + k)$ numbers instead of $dk$. For $d = k = 1{,}024$ and $r = 8$ that is 16,384 against 1,048,576, about 1.6%. The forward pass computes both paths and adds them:

$$\mathbf{y} = W_0 \mathbf{x} + \frac{\alpha}{r}\, B (A \mathbf{x}),$$

where $\alpha/r$ is the scaling the official code applies (scaling = lora_alpha / r in loralib/layers.py). In the official Linear layer the low-rank path multiplies by $A$ first, so it never forms the $d \times k$ matrix $BA$ during training.

The notation varies between sources. DoRA and the official code use $BA$ with $B$ of size $d \times r$, as here; HyperDreamBooth’s authors, as summarized in Personalized Image Generation, write $\Delta W = AB$ with $A$ of size $n \times r$. The roles of the two letters are swapped, not the method.

Initialization and scaling

In DoRA’s restatement of LoRA, $A$ is initialized with a Kaiming uniform distribution and $B$ with zeros, so that $\Delta W = BA$ is zero at the start of training (Liu et al., 2024, §3.1). The fine-tuned model therefore starts exactly at the pretrained one, as transfer learning notes. The official Linear layer does this (kaiming_uniform_ on lora_A, zeros_ on lora_B), although the comment above those two lines says the reverse and notes that the initialization differs from what the paper describes; the code is what runs. The official Embedding layer does it the other way round, with $A$ at zero and $B$ drawn from a normal distribution. Either way, one factor is zero and the product is zero.

The zero factor matters for the gradients. With $B = 0$, the gradient with respect to $A$ is zero at the first step (it passes through $B$), while the gradient with respect to $B$ is not, so $B$ moves first and $A$ follows. This follows from the product rule; it is our reasoning, not a statement from the sources.

The factor $\alpha/r$ scales the update. In the official code, lora_alpha defaults to 1, and the scaling is applied in the forward pass and when merging. Hugging Face’s PEFT library defaults to $r = 8$ and $\alpha = 8$, and offers a “rank-stabilized” option that uses $\alpha/\sqrt{r}$ instead of $\alpha/r$ (PEFT LoraConfig, v0.21.2). With plain gradient descent and $B$ starting at zero, multiplying $\alpha$ by $c$ is equivalent to keeping $\alpha$ and multiplying the learning rate of $B$ by $c^2$ while leaving that of $A$ unchanged (substitute $\tilde B = cB$; our derivation); with adaptive optimizers such as Adam the relation differs, which is our reading of why the scaling is exposed as a separate setting. DoRA’s hyperparameter tables use $\alpha = 2r$ for the LLaMA models and LLaVA-1.5 and $\alpha = r = 128$ for VL-BART (Liu et al., 2024, Tables 8–10).

PEFT also offers other initializations, documented in the same configuration class, among them a Gaussian one, PiSSA (principal singular values and singular vectors adaptation), and LoftQ, which quantizes the backbone and initializes the LoRA layers; with init_lora_weights=False both factors are random and the adapter is not a no-op before training, which the documentation says is meant for debugging.

Which layers receive it

LoRA can be applied to any subset of weight matrices. The official examples adapt only the query and value projections of the self-attention layers in a transformer, and the README says that LoRA can be applied to any subset of pretrained weights, suggests trying the embedding and MLP layers, and expects the best configuration to vary with the architecture and the task. Implementations that compute query, key and value with one fused linear layer need either three separate layers or the repository’s MergedLinear, which attaches low-rank updates to chosen parts of the fused output (enable_lora=[True, False, True] adapts query and value but not key).

Later work adapts more matrices. DoRA’s LLaMA experiments put the update on the query, key, value, up and down projections, and its LLaVA-1.5 experiments on all attention and MLP projections (Liu et al., 2024, Tables 8 and 10). PEFT accepts a list of module names or 'all-linear', and lets each layer have its own rank and $\alpha$ (rank_pattern, alpha_pattern).

Two practical points follow from the code. A newly added task head, for example a classifier for new classes, has no pretrained weight to adapt; PEFT trains and saves such modules in full (modules_to_save). Biases stay frozen by default; the README suggests that training them as well may be a cheap way to gain some accuracy, and provides bias='lora_only' and bias='all'.

The official code also adapts convolutions. Its ConvLoRA stores $A$ with shape $(r \cdot k_s) \times (C_\text{in} k_s)$ and $B$ with shape $(C_\text{out} k_s / g) \times (r \cdot k_s)$ for kernel size $k_s$ and $g$ groups, and reshapes their product to the kernel’s shape. The rank of that product, as a matrix, can be up to $r \cdot k_s$ rather than $r$ (our reading of the shapes).

Merging and serving

Because $W' = W_0 + BA$ has the same shape as $W_0$, the product can be added into the frozen weight after training. The official layers do this automatically: calling model.eval() adds $\frac{\alpha}{r}BA$ to the weight, and model.train() subtracts it again; merge_weights=False turns this off. A merged layer has the same shape as the original, so inference has no extra latency (Liu et al., 2024, §3.1). Saving stores only the LoRA parameters (lora.lora_state_dict), and loading needs the pretrained checkpoint first, then the small LoRA checkpoint on top (README).

Merging trades flexibility for speed. Unmerged, one copy of $W_0$ can serve several tasks, each adding its own low-rank path, and adapters can be swapped without touching the base weights. Merged, the weight belongs to one task, and switching means subtracting one product and adding another. In floating point, merging is exact only up to rounding, which the example below measures.

Why a low rank might suffice

The argument for a low rank is empirical. DoRA describes LoRA as building on the hypothesis that the updates made during fine-tuning have a low “intrinsic rank” (Liu et al., 2024, §3.1). A related finding is Aghajanyan, Gupta and Zettlemoyer’s (2021). They fine-tuned pretrained language models inside random low-dimensional subspaces of the full parameter space, $\theta = \theta_0 + P\theta_d$ with a fixed random projection $P$ and only $\theta_d$ trained from zero, and used as the intrinsic dimension $d_{90}$ the smallest $d$ that reached 90% of full fine-tuning’s metric. With their structure-aware method, RoBERTa-Large reached it on the MRPC paraphrase task with 207 parameters and on QQP with 774 (Table 1); they also found that larger pretrained models tended to have lower intrinsic dimension, and that pretraining itself lowered it (§5). A random subspace of dimension $d$ is not the same restriction as a rank-$r$ update of each matrix, so these results support the general idea that fine-tuning a pretrained model needs few degrees of freedom rather than LoRA’s particular form; the link is our reading.

Nothing guarantees a low-rank update for a given model and task. The example below shows a case where it does not hold.

How to Apply It

The usual workflow, as the official README and PEFT describe it:

1. Choose the matrices. The official examples adapt the query and value projections; the README expects the best set to vary with the architecture and the task, and DoRA’s configurations also adapt the MLP layers.
2. Choose $r$ and $\alpha$. The papers above use ranks from 1 (HyperDreamBooth) through 8 and 16 (SPT) to 16–128 (DoRA’s configurations), and DoRA’s configurations set $\alpha$ to $r$ or $2r$; tune the learning rate together with them, since $\alpha$ scales the update.
3. Freeze everything else. mark_only_lora_as_trainable sets requires_grad to false for every parameter without lora_ in its name; train new heads in full.
4. Train with the task loss as in ordinary fine-tuning.
5. Save only the adapter, and merge for deployment when one task per model is enough.

Example: the best rank-$r$ update for a new task

A full LoRA experiment needs a pretrained network and gradient descent. The example below keeps what can be computed exactly: a frozen linear layer on fixed features, adapted to a new task by least squares. Because the layer is linear, the best update of each kind has a closed form, and the example can ask how close a rank-$r$ update can come to the full update at all, independent of how it would be trained.

Set-up. Synthetic $24 \times 24$ grayscale images hold one to three disks, squares or bars of random size and brightness on a noisy background. Each image $x$ passes through fixed random features $\mathbf{h} = \mathrm{ReLU}(R\mathbf{x} + \mathbf{b})$, with $R$ a $256 \times 576$ Gaussian matrix; each feature’s bias is set so that it is active on half the “pretraining” images. These stand in for frozen lower layers. A linear layer $W_0 \in \mathbb{R}^{36 \times 256}$ maps the features to a $6 \times 6$ thumbnail of the image (4 × 4 block means): this is the source task, fitted by least squares on 4,000 images and then frozen. The convention is $\mathbf{y} = W\mathbf{h}$ for one image, so $d = 36$ outputs and $k = 256$ inputs, and $B$ and $A$ have the shapes of the official code. The two new tasks ask the same layer for a different thumbnail:

• blur: the thumbnail of the image blurred by a Gaussian of standard deviation 2 pixels;
• edges: the thumbnail of the Sobel gradient magnitude.

The update is fitted on $n = 2{,}000$ or $n = 300$ training images (just above the 256 features) and tested on 2,000 others. Three updates are compared:

• the full update $\Delta W$: least squares on the residual $Y - HW_0^\top$, where the rows of $H$ and $Y$ are the training images’ features and targets;
• reduced-rank regression (RRR): the update of rank $r$ with the smallest training error. With $H$ of full column rank, minimizing $\|Y - H(W_0 + \Delta W_r)^\top\|_F^2$ over rank-$r$ updates amounts to approximating the full fit’s predictions $H\Delta W^\top$ by a rank-$r$ matrix, so by the Eckart–Young theorem the answer projects $\Delta W$ onto the top $r$ right singular vectors of $H\Delta W^\top$ (our derivation). This is the lowest training error that a rank-$r$ LoRA could reach on these data. Whether gradient descent on $B$ and $A$ reaches it is not tested here, and the scaling $\alpha/r$ and the initialization, which only affect training, play no part;
• the truncated SVD of $\Delta W$: the best rank-$r$ approximation of the matrix $\Delta W$ itself, as used to compress a trained layer in model compression.

Errors are relative to predicting each output’s mean, $\|Y - \hat Y\|_F^2 / \|Y - \bar Y\|_F^2$; values above 1 are worse than the mean. Row $r = 0$ is the frozen layer alone. The script also checks that rank 36 reproduces the full update for both factorizations, that RRR’s training error is never above the truncated SVD’s, and that the merged weight $W_0 + BA$ gives the same outputs as the frozen path plus the low-rank path. The last block repeats the fits for seeds 0 to 4 (new images and new features each time). It ran with NumPy 2.3.5 and SciPy 1.16.3 on one BLAS thread, in about 3 to 4 seconds.

import os
for v in ("OPENBLAS_NUM_THREADS", "OMP_NUM_THREADS", "MKL_NUM_THREADS", "VECLIB_MAXIMUM_THREADS"):
    os.environ[v] = "1"                  # one BLAS thread: reproducible sums
import numpy as np
from scipy.ndimage import gaussian_filter, sobel

S, BLK, K = 24, 4, 256                    # image side, thumbnail block, random features
yy, xx = np.mgrid[0:S, 0:S]

def make_images(n, rng):
    """1-3 disks, squares or bars of random size and brightness on a noisy background."""
    imgs, kinds = np.empty((n, S, S)), np.zeros(3, int)
    for i in range(n):
        img = 0.1 + 0.05 * rng.standard_normal((S, S))
        for _ in range(rng.integers(1, 4)):
            kind = rng.integers(3)
            kinds[kind] += 1
            cy, cx = rng.uniform(3, S - 3, 2)
            s, b = rng.uniform(2.5, 5.5), rng.uniform(0.5, 1.0)
            if kind == 0:
                m = (yy - cy) ** 2 + (xx - cx) ** 2 <= s ** 2               # disk
            elif kind == 1:
                m = (np.abs(yy - cy) <= s) & (np.abs(xx - cx) <= s)        # square
            elif rng.random() < 0.5:
                m = (np.abs(yy - cy) <= 1) & (np.abs(xx - cx) <= s + 2)    # horizontal bar
            else:
                m = (np.abs(xx - cx) <= 1) & (np.abs(yy - cy) <= s + 2)    # vertical bar
            img[m] = b
        imgs[i] = np.clip(img, 0, 1)
    return imgs, kinds

def thumb(a):                             # 6x6 thumbnail of 4x4 block means -> 36 outputs
    return a.reshape(len(a), S // BLK, BLK, S // BLK, BLK).mean(axis=(2, 4)).reshape(len(a), -1)

TASKS = {
    "source": thumb,
    "blur":   lambda im: thumb(np.stack([gaussian_filter(a, 2.0, mode="nearest") for a in im])),
    "edges":  lambda im: thumb(np.stack([np.hypot(sobel(a, 0, mode="nearest"),
                                                  sobel(a, 1, mode="nearest")) for a in im])),
}

def rel_err(Y, P):                        # squared error relative to predicting the mean
    return ((Y - P) ** 2).sum() / ((Y - Y.mean(0)) ** 2).sum()

def setup(seed):
    rng = np.random.default_rng(seed)
    src, k1 = make_images(4000, rng)
    pool, k2 = make_images(2000, rng)
    test, k3 = make_images(2000, rng)
    R = np.random.default_rng(1000 + seed).standard_normal((K, S * S)) / S
    bias = -np.median(src.reshape(len(src), -1) @ R.T, axis=0)  # each feature on for half the images
    feat = lambda im: np.maximum(im.reshape(len(im), -1) @ R.T + bias, 0.0)
    W0 = np.linalg.lstsq(feat(src), TASKS["source"](src), rcond=None)[0].T   # 36 x 256, frozen
    return src, pool, test, feat, W0, k1 + k2 + k3

def adapt(H, Y, W0):
    """Full least-squares update, and the factors of the rank-r updates (RRR and SVD)."""
    dW = np.linalg.lstsq(H, Y - H @ W0.T, rcond=None)[0].T             # 36 x 256
    _, _, Vt = np.linalg.svd(H @ dW.T, full_matrices=False)            # fitted values
    U, s, VtW = np.linalg.svd(dW, full_matrices=False)
    rrr = lambda r: (Vt[:r].T, Vt[:r] @ dW)                            # B (d x r), A (r x k)
    svd = lambda r: (U[:, :r] * s[:r], VtW[:r])
    return dW, s, rrr, svd

d, RANKS = 36, (0, 1, 2, 4, 8, 16, 36)
src, pool, test, feat, W0, kinds = setup(0)
H_te = feat(test)
n_unique = len(np.unique(np.concatenate([src, pool, test]).reshape(8000, -1), axis=0))
print(f"shapes (disk, square, bar): {kinds}; distinct images: {n_unique} of 8000")
print(f"source task, frozen W0: test error {rel_err(TASKS['source'](test), H_te @ W0.T):.3f}")

checks = {"C1": 0.0, "C1err": 0.0, "C2": np.inf, "C3": 0.0}
for task in ("blur", "edges"):
    Y_te = TASKS[task](test)
    for n in (2000, 300):
        H, Y = feat(pool[:n]), TASKS[task](pool[:n])
        print(f"\n{task}, n = {n}: rank(H) = {np.linalg.matrix_rank(H)}, "
              f"least active feature on {(H > 0).mean(0).min():.0%} of images")
        dW, s, rrr, svd = adapt(H, Y, W0)
        print("  sigma_i / sigma_1 of dW, i = 2, 4, 8, 16, 36:",
              " ".join(f"{s[i - 1] / s[0]:.3f}" for i in (2, 4, 8, 16, 36)))
        print("     r  trainable   train(RRR)  test(RRR)  test(SVD of dW)")
        for r in RANKS:
            row = []
            for fac in (rrr, svd):
                B, A = fac(r)
                W = W0 + B @ A                                         # merged weight
                unmerged = H_te @ W0.T + (H_te @ A.T) @ B.T            # frozen path + low-rank path
                checks["C3"] = max(checks["C3"], np.abs(H_te @ W.T - unmerged).max())
                row.append((rel_err(Y, H @ W.T), rel_err(Y_te, H_te @ W.T), W))
            print(f"  {r:4d}  {r * (d + K):9,d}   {row[0][0]:9.4f}  {row[0][1]:9.4f}  {row[1][1]:9.4f}")
        full_tr, full_te = rel_err(Y, H @ (W0 + dW).T), rel_err(Y_te, H_te @ (W0 + dW).T)
        print(f"  full  {d * K:9,d}   {full_tr:9.4f}  {full_te:9.4f}")
        for fac in (rrr, svd):                                         # C1: rank d = full update
            B, A = fac(d)
            checks["C1"] = max(checks["C1"], np.abs(B @ A - dW).max())
            checks["C1err"] = max(checks["C1err"], abs(rel_err(Y_te, H_te @ (W0 + B @ A).T) - full_te))
        for r in range(1, d + 1):                                      # C2: RRR optimal on training data
            tr = [rel_err(Y, H @ (W0 + B @ A).T) for B, A in (rrr(r), svd(r))]
            checks["C2"] = min(checks["C2"], tr[1] - tr[0])

print(f"\nC1 rank 36 = full update: max |dW_r - dW| {checks['C1']:.1e}, "
      f"test-error difference {checks['C1err']:.1e}")
print(f"C2 min over r of train error (SVD) - train error (RRR): {checks['C2']:.1e}")
print(f"C3 merged vs unmerged outputs, max abs difference: {checks['C3']:.1e}")

print("\nseeds 0-4, test error range at r = 4 (RRR | SVD of dW) and full update:")
res = {}
for seed in range(5):
    src, pool, test, feat, W0, _ = setup(seed)
    H_te = feat(test)
    for task in ("blur", "edges"):
        Y_te = TASKS[task](test)
        for n in (2000, 300):
            H, Y = feat(pool[:n]), TASKS[task](pool[:n])
            dW, s, rrr, svd = adapt(H, Y, W0)
            e = [rel_err(Y_te, H_te @ (W0 + B @ A).T) for B, A in (rrr(4), svd(4))]
            e.append(rel_err(Y_te, H_te @ (W0 + dW).T))
            res.setdefault((task, n), []).append(e)
for (task, n), v in res.items():
    v = np.array(v)
    print(f"  {task:5s} n = {n:4d}: " + " | ".join(f"{v[:, j].min():.3f}-{v[:, j].max():.3f}" for j in range(3)))

shapes (disk, square, bar): [5242 5358 5427]; distinct images: 8000 of 8000
source task, frozen W0: test error 0.083

blur, n = 2000: rank(H) = 256, least active feature on 47% of images
  sigma_i / sigma_1 of dW, i = 2, 4, 8, 16, 36: 0.683 0.624 0.543 0.450 0.190
     r  trainable   train(RRR)  test(RRR)  test(SVD of dW)
     0          0      0.1921     0.1910     0.1910
     1        292      0.1800     0.1810     0.1921
     2        584      0.1689     0.1715     0.1836
     4      1,168      0.1490     0.1534     0.1691
     8      2,336      0.1149     0.1243     0.1425
    16      4,672      0.0733     0.0881     0.1053
    36     10,512      0.0492     0.0694     0.0694
  full      9,216      0.0492     0.0694

blur, n = 300: rank(H) = 256, least active feature on 43% of images
  sigma_i / sigma_1 of dW, i = 2, 4, 8, 16, 36: 0.467 0.380 0.243 0.153 0.048
     r  trainable   train(RRR)  test(RRR)  test(SVD of dW)
     0          0      0.1822     0.1910     0.1910
     1        292      0.1624     0.3559     0.3764
     2        584      0.1497     0.3552     0.4143
     4      1,168      0.1269     0.3472     0.4607
     8      2,336      0.0899     0.3591     0.5053
    16      4,672      0.0397     0.4026     0.5032
    36     10,512      0.0069     0.4631     0.4631
  full      9,216      0.0069     0.4631

edges, n = 2000: rank(H) = 256, least active feature on 47% of images
  sigma_i / sigma_1 of dW, i = 2, 4, 8, 16, 36: 0.969 0.900 0.750 0.542 0.184
     r  trainable   train(RRR)  test(RRR)  test(SVD of dW)
     0          0      1.2386     1.2371     1.2371
     1        292      0.7630     0.7737     1.1592
     2        584      0.7216     0.7426     0.8546
     4      1,168      0.6611     0.7028     0.8192
     8      2,336      0.5781     0.6473     0.7287
    16      4,672      0.4587     0.5616     0.6172
    36     10,512      0.3612     0.4939     0.4939
  full      9,216      0.3612     0.4939

edges, n = 300: rank(H) = 256, least active feature on 43% of images
  sigma_i / sigma_1 of dW, i = 2, 4, 8, 16, 36: 0.843 0.683 0.558 0.335 0.104
     r  trainable   train(RRR)  test(RRR)  test(SVD of dW)
     0          0      1.2386     1.2371     1.2371
     1        292      0.7424     0.9459     1.4697
     2        584      0.6782     1.0646     1.7267
     4      1,168      0.5759     1.4031     1.8359
     8      2,336      0.4209     1.7814     2.3261
    16      4,672      0.2249     2.2832     2.7213
    36     10,512      0.0530     2.8606     2.8606
  full      9,216      0.0530     2.8606

C1 rank 36 = full update: max |dW_r - dW| 1.5e-14, test-error difference 3.1e-15
C2 min over r of train error (SVD) - train error (RRR): -8.7e-19
C3 merged vs unmerged outputs, max abs difference: 9.3e-15

seeds 0-4, test error range at r = 4 (RRR | SVD of dW) and full update:
  blur  n = 2000: 0.147-0.153 | 0.163-0.169 | 0.065-0.069
  blur  n =  300: 0.290-0.347 | 0.385-0.461 | 0.390-0.463
  edges n = 2000: 0.697-0.711 | 0.788-0.970 | 0.494-0.516
  edges n =  300: 1.361-1.531 | 1.836-2.273 | 2.861-3.392

The planned hypothesis (an effectively low-rank update) failed. The plan written before the script ran expected that “the singular values of $\Delta W$ fall; for blur (n = 2000) RRR at r = 4 has test error closer to the full update’s than to the frozen layer’s”, and that edges would need more rank than blur. Neither happened: the check failed. For blur, the second singular value of $\Delta W$ is still 0.68 of the first and the 36th is 0.19, so the update is not effectively low rank: at $r = 4$ RRR’s test error (0.153) is closer to the frozen layer’s (0.191) than to the full update’s (0.069), and test error keeps falling all the way to rank 36. Edges behaved the other way: a single rank removes over a third of the frozen layer’s error (1.237 to 0.774), and $r = 4$ is already nearer the full update than the frozen layer. Apart from one direction of the predictions that matters for edges (visible in RRR’s rank-1 row, not in the singular values of $\Delta W$), the update in this toy has no low-rank structure to find, and nothing in its construction puts any there; in our reading, the low intrinsic dimension that Aghajanyan et al. measured is a property of large pretrained networks, which a random-feature layer does not share, and the example should not be read as evidence against it or for it.

What is best for the matrix is not best for the task. At every rank from 1 to 16, in all four settings, RRR has lower test error than the truncated SVD of $\Delta W$; on edges at $r = 1$ it is 0.774 against 1.159. The truncated SVD keeps the directions in which $\Delta W$ is largest, but the features are correlated, so some large directions of $\Delta W$ change the outputs little and some small ones change them a lot. LoRA trained on the task loss optimizes the task, like RRR, not the distance to some full update; compressing a full fine-tuning result by SVD afterwards is a different and, here, worse procedure. At rank 36 both reproduce the full update to about $10^{-14}$, and RRR is never worse on the training data, up to rounding, as the derivation requires. Over seeds 0 to 4 the gap at $r = 4$ persists: the printed ranges for RRR and the truncated SVD do not overlap in any setting.

With few images, a low rank regularizes. With 300 training images the full update fits the training set closely (training error 0.007 on blur) and generalizes badly: its test error is above the frozen layer’s on both tasks (0.463 against 0.191 on blur, 2.861 against 1.237 on edges). Small ranks do better than the full update (blur 0.347 at $r = 4$, edges 0.946 at $r = 1$), and in the seed loop RRR at $r = 4$ stays below the full update for all five seeds, since the two ranges do not overlap. On blur, however, no rank beats the frozen layer itself: with this little data, not adapting at all was best. On edges, ranks 1 and 2 do beat it.

Parameters and merging. The trainable count grows as $r(d + k) = 292r$; at $r = 36$ the two factors hold 10,512 numbers, more than the 9,216 of $\Delta W$, so LoRA saves parameters only while $r < dk/(d + k)$, here about 31.6. Merged and unmerged outputs agree to $9 \times 10^{-15}$, the rounding of double precision, so merging changes the outputs only by rounding.

Variants

QLoRA (Dettmers et al., 2023; not read here) is described by DoRA’s authors as a way to reduce memory further: it quantizes the pretrained model to 4 bits and trains LoRA on top of the frozen low-bit backbone, since even with few trainable parameters the full model’s weights must still be loaded onto the GPUs (Liu et al., 2024, §6.1). Quantization covers low-bit formats. DoRA’s §6.1 reports a project by others (QDoRA) that replaced LoRA in QLoRA by DoRA; on 500 held-out Orca-Math problems, scored by exact match, the figure shows QDoRA at 0.31 on LLaMA2-7B and 0.56 on LLaMA3-8B, against 0.12 and 0.32 for QLoRA and 0.26 and 0.51 for full fine-tuning (Fig. 5); the text gives the gaps to QLoRA as 0.19 and 0.23, while the plotted values differ by 0.19 and 0.24.

DoRA (Liu et al., 2024) splits each weight into a magnitude and a direction, following weight normalization. Writing $\|\cdot\|_c$ for the norm of each column,

$$W' = m\, \frac{W_0 + BA}{\|W_0 + BA\|_c},$$

with the magnitude vector $m \in \mathbb{R}^{1 \times k}$ initialized to $\|W_0\|_c$ and trained, and the direction updated by a LoRA pair initialized as in LoRA, so that $W' = W_0$ at the start (Eq. 5). It adds $k$ parameters per matrix and can be merged like LoRA. The motivation is an analysis on VL-BART: across layers and training steps, the changes in magnitude and in direction of the query matrices were positively correlated for LoRA (0.83) and negatively for full fine-tuning (−0.62) and DoRA (−0.31) (§3.2, §4.1). To save memory, the norm $\|W_0 + BA\|_c$ is treated as a constant in backpropagation, which reduced training memory by 24.4% on LLaMA-7B and 12.4% on VL-BART with little change in accuracy (§4.3, Table 7). PEFT supports DoRA for linear and 2D convolution layers and recommends merging for inference, since unmerged DoRA costs more than LoRA.

Other variants. DoRA’s related-work section lists LoRA variants that prune singular values of the update, use orthogonal factorizations to fine-tune diffusion models, tie weights, or, in VeRA, share one pair of frozen random matrices across layers and train only scaling vectors (Liu et al., 2024, §2); DoRA combined with VeRA (“DVoRA”) trained 0.04% of LLaMA-7B’s parameters (Table 5). PEFT implements alternative initializations (PiSSA, LoftQ and others) and the rank-stabilized scaling described above. In personalization, Lightweight DreamBooth factors a rank-1 LoRA further through frozen random bases, and SVDiff trains only shifts of the singular values of the pretrained weights (Personalized Image Generation describes both).

Best Practices

• Start from the zero product. Keep one factor at zero so training starts at the pretrained model; random initialization of both factors changes the model before training.
• Tune the learning rate with $r$ and $\alpha$. The update’s size depends on all three. Evidence on rank is not monotone (below), so a larger rank is not a safe default.
• Consider adapting more than query and value. The README expects the best set to vary with the architecture and the task, and DoRA’s configurations adapt attention and MLP projections.
• Train new heads in full and decide deliberately whether to train biases.
• Compare with a linear probe and full fine-tuning on the same data. In the example, with 300 images, the frozen layer beat every adapted one on one task.
• Merge for single-task deployment; keep adapters separate for multi-task serving.

Pitfalls

• Assuming rank is the only limit. DoRA’s authors found LoRA’s changes in magnitude and direction proportional, unlike full fine-tuning’s, and suspect that part of LoRA’s gap stems from having to learn both together, not only from its rank (Liu et al., 2024, §3.2).
• Assuming more rank is better. In DoRA’s rank sweep on LLaMA-7B, LoRA’s average commonsense accuracy was 39.5 at $r = 4$, 40.7 at 8, 70.9 at 16, 74.7 at 32 and 65.8 at 64 (Table 15); the text of §5.5 quotes 40.74 and 39.49. SPT’s LoRA-16 was slightly worse than LoRA-8 on FGVC (below).
• Compressing a full fine-tune by SVD and calling it LoRA. The best rank-$r$ approximation of an update is not the best rank-$r$ update for the task, as the example shows.
• Merging with the wrong scaling or layout. The merge must use the same $\alpha/r$ as training, and layers that store weights transposed (GPT-2’s Conv1D) need fan_in_fan_out=True in both the official code and PEFT.
• Forgetting that biases and heads changed. If biases or a head were trained, disabling the adapter does not restore the base model (PEFT LoraConfig documentation on bias).
• Loading order. The adapter checkpoint holds only the LoRA parameters; the README loads the pretrained checkpoint first and then the LoRA checkpoint, both with strict=False.

Evidence and Impact

LoRA was introduced and first evaluated on language models; its own results are known here only through the README, which reports the GLUE numbers above and says that on GPT-2 Medium and Large, evaluated on E2E, DART and WebNLG with 0.35 to 0.77 million trainable parameters, LoRA compares favorably with full fine-tuning, adapters and prefix tuning; its table shows LoRA ahead on most but not all of the scores. The vision evidence comes from later papers that use it as a baseline or a tool.

Vision transformers. He et al. (2023) compared parameter-efficient methods on ViT-B/16 (85.8 million parameters) on the 19 tasks of VTAB-1k and on five fine-grained classification datasets (FGVC: CUB-200-2011, NABirds, Oxford Flowers, Stanford Cars and Stanford Dogs), reporting mean top-1 accuracy. With a backbone pretrained with labels on ImageNet-21k, LoRA with rank 8 trained 0.23% of the parameters on VTAB-1k and reached a mean of 74.9, rank 16 reached 75.0, and full fine-tuning 69.0; on FGVC, LoRA-8 reached 86.0, LoRA-16 84.8 and full fine-tuning 88.5 (Table 1). With self-supervised backbones the picture changed (Table 2): on an MAE backbone LoRA-8 matched full fine-tuning on VTAB-1k (64.3 against 64.3), and LoRA-16 exceeded it (64.8), although the text says existing parameter-efficient methods were inferior to full fine-tuning on both self-supervised backbones; on a MoCo v3 backbone LoRA-8 reached 44.3 against full fine-tuning’s 69.6, and LoRA-16 42.9. The authors note that earlier parameter-efficient methods gave inconsistent results across pretraining strategies. Their own method, SPT, applies LoRA to the weight matrices it identifies as sensitive for the task, combined with tuning of individual sensitive weights, and improved on LoRA-8 by 3.3 points on FGVC at 0.41% of the parameters.

Vision–language models. Sung, Cho and Bansal (2022) applied LoRA to the BART language model of a CLIP-BART model, keeping the CLIP encoder frozen, trained jointly on VQAv2, GQA, NLVR2 and COCO captioning. One LoRA shared across the four tasks updated 5.93% of the parameters and averaged 76.5, against 77.6 for full fine-tuning and 77.4 for a single shared adapter with 4.18%; one LoRA per task (17.72%) averaged 74.2 (Table 1). The authors found LoRA about one point below the single adapter but better than the other parameter-efficient methods they tried. DoRA’s authors later report, on the same set-up, DoRA at 77.4 against LoRA at 76.5 (Liu et al., 2024, Table 2); they say they quote the LoRA and full fine-tuning rows from Sung et al., but their full fine-tuning row (average 77.3) differs from Sung et al.‘s Table 1 (77.6), while the LoRA row matches. On LLaVA-1.5-7B visual instruction tuning, averaged over seven benchmarks, LoRA at rank 128 updating 4.61% of the parameters scored 66.9, full fine-tuning 66.5 and DoRA 67.6 (Table 4, Table 10); the authors suggest full fine-tuning may have overfitted. Visual instruction tuning records that LLaVA-RLHF trained LoRA modules of rank 64 in its reinforcement-learning stage, and embodied AI reports that LoRA adaptation of the OpenVLA vision–language–action model to a new robot matched full fine-tuning in its authors’ tests.

Diffusion models. DoRA’s authors describe LoRA as a popular method for fine-tuning large Stable Diffusion models (Liu et al., 2024, §6.2). HyperDreamBooth’s authors describe it as usually applied to the cross- and self-attention layers in diffusion models, and report that a rank-1 LoRA DreamBooth of Stable Diffusion 1.5 has about 386,000 parameters, about 1.6 MB (Ruiz et al., 2024, §3); Personalized Image Generation compares it with full DreamBooth, Custom Diffusion and SVDiff. Published articles record other uses: DragDiffusion fine-tunes a rank-16 LoRA on the input image before editing (image editing), Stable Video Diffusion was given camera-motion LoRA modules in its temporal attention (video generation), and LCM-LoRA packages consistency distillation as a low-rank module that plugs into different checkpoints (diffusion distillation). DoRA’s authors fine-tuned SDXL with DreamBooth using LoRA and DoRA under the same settings and judged DoRA’s images to match the training subjects more closely, from qualitative examples (Liu et al., 2024, §6.2).

Language models, through DoRA. In DoRA’s tables, on eight commonsense reasoning tasks, LoRA (its row taken from an earlier benchmark study, as the caption says) averaged 74.7 on LLaMA-7B with 0.83% of the parameters, DoRA 78.4 with 0.84%, and DoRA at half the rank 77.5 (Table 1). As of November 2026, LoRA is supported by Hugging Face’s PEFT library, which the official README points to, alongside DoRA and many initializations.