Scaling Laws

Scaling laws are empirical relationships between a network’s error and the resources spent on it: the number of training examples, the number of parameters, and the training compute. Over a range of scales the error often falls as a power of each resource, a straight line on a log-log plot, and studies fit such lines in order to predict what a larger model or dataset will reach before paying for it, and to decide how a fixed compute budget should be split between model size and data. The term neural scaling laws is also used. In vision, the evidence comes from supervised pretraining on hundreds of millions to billions of weakly labelled images, from vision transformers, and from contrastive image–text models. The foundation models article summarizes the scaling results for those models; this article covers the laws themselves: their forms, the vision evidence, how far they transfer to downstream tasks, where they fail, and how they are fitted and extrapolated.

Definition

A scaling law is a function $E(x)$, fitted to measurements, that gives a model’s error $E$ (test loss, error rate, or one minus a downstream accuracy) as a function of a scale variable $x$: the training-set size $n$, the parameter count $N$, the compute $C$, or the number of examples seen during training. The law holds within a family of experiments that keeps everything else, architecture family, data distribution, training recipe and evaluation, fixed or tuned in a fixed way. Outside that family it is not a law at all: Cherti et al. (2023) found that two CLIP model series with the same architectures and similar recipes scaled differently, a difference they attribute to the different training data.

Hestness et al. (2017), as Cherti et al. (2023, §2) describe them, highlighted the power-law relation between scale and performance, and in their account Kaplan et al. (2020) and others fitted power laws for language models that connect model size, data and compute to performance; the vision studies below followed. The laws are empirical: fitted, not derived from theory. They are curves fitted to points, and the claims they support are claims about a range of measured scales and about what happens slightly beyond it.

Intuition

Each extra order of magnitude of data or model size buys a roughly constant fraction off the remaining error, not a constant amount. If ten times more images cut the error by a third, another ten times cut what is left by a third again. On a log-log plot such a curve is a straight line whose slope is the exponent. This is the regularity that makes extrapolation tempting: a line fitted to cheap small runs predicts expensive large ones.

Two things bend the line. At the top, error cannot fall below what the task allows (label noise, ambiguous images, an imperfect test set), so the curve flattens toward a floor. At the bottom, even a trivial model, one that predicts the most frequent class, has an error below the line’s value at tiny scales, so the curve flattens there too. And each resource only helps while the others are not the bottleneck: a small model cannot use more data, and a large model trained on few images cannot use its capacity. Scaling laws are therefore laws about growing the resources together.

Formal Definition

Power law. In its simplest form,

$$E(x) = a\,x^{-b}, \qquad \log E = \log a - b \log x ,$$

with a scale constant $a > 0$ and an exponent $b > 0$. Cherti et al. (2023) fitted this form, written $E = \beta C^{\alpha}$ with $\alpha < 0$, to the best models in each bin of total compute, by fitting a line in log-log space (supplementary §B.2.1).

Irreducible error. Zhai, Kolesnikov, Houlsby and Beyer (2022, §2.2) observed that their largest vision transformers did not tend toward zero error and added a constant:

$$E(x) = a\,x^{-b} + c ,$$

where $c$ is the error an infinitely large model would keep. They note that Henighan et al. (2020) had observed the same for generative models and called the residual the “irreducible entropy” of the task; for error rates, Zhai et al. write, the information-theoretic reading does not apply, but the observation supports the idea of a performance ceiling on ImageNet. Goyal et al. (2024, §4) write the same form for error after training on $n$ samples, $y = a n^{b} + d$ with $b < 0$, and read $d$ as the irreducible error.

Saturation at both ends. Because trivial solutions already achieve a non-zero accuracy, the smallest models do better than the power law predicts. Zhai et al. (2022, §2.2) model this by shifting the scale axis, giving the double-saturating power law

$$E(C) = a\,(C + d)^{-b} + c ,$$

in which $d$ reflects the non-zero accuracy that a zero-compute model still obtains. Fitted to the Pareto frontier of ImageNet error against pretraining compute $C$ in TPUv3 core-days, with error as a fraction, they obtained $E = 0.09 + 0.26\,(C + 0.01)^{-0.35}$ for fine-tuning and $E = 0.12 + 0.63\,(C + 0.52)^{-0.32}$ for 10-shot linear evaluation (Figure 2), and on the public ImageNet-21k, $E = 0.24 + 0.96\,(C + 1.57)^{-0.61}$ for 10-shot evaluation (Figure 4).

Logarithmic scaling. Some vision studies report a different shape: accuracy that rises by a fixed amount each time the data are multiplied by a fixed factor, $A(n) = \alpha + \gamma \log n$ (Sun et al., 2017; Mahajan et al., 2018). In our reading, this is a straight line on a semi-log plot of accuracy, not on a log-log plot of error. Over a short range the two descriptions are hard to tell apart; they differ in how they extrapolate, since a log-linear accuracy must eventually exceed 100%, which Mahajan et al. note themselves (below).

Compute and the frontier. Compute is a budget that can be spent on a bigger model, more data or longer training. Studies plot every run at its compute and fit the law to the frontier, the lowest error reached at each budget: Zhai et al. describe their frontier as containing the models with the best allocation of compute to model shape and training duration (§2.2), and Cherti et al. bin the compute axis and take the best model in each bin (supplementary §B.2.1).

Compute-optimal allocation. Kaplan et al. (2020), as Goyal et al. (2024, §2) describe them, observed for language models that for a fixed compute budget there is an optimal model size, training-data size and training time, and that model size, data size and batch size tend to scale roughly in lock-step; Hoffmann et al. (2022), as Cherti et al. (2023, §2) describe them, used scaling laws to choose the scale parameters that give the best performance for a fixed compute. The reasoning behind such allocations can be shown with an illustrative form of our own (not a law fitted by any of these sources). Suppose the error separates into a model-size term and a data term,

$$E(N, n) = c + A N^{-\alpha} + B n^{-\beta}, \qquad C = k\,N\,n ,$$

where the cost of training grows with the product of parameters and examples. Substituting $n = C/(kN)$ and setting $\partial E / \partial N = 0$ gives

$$N^\star \propto C^{\beta/(\alpha+\beta)}, \qquad n^\star \propto C^{\alpha/(\alpha+\beta)} ,$$

so under this form both the model and the data grow as powers of the budget, and neither should be held fixed while the other grows. The exponents decide the split: the resource whose error term falls more slowly gets the larger share of a growing budget.

Properties

Resources must grow together. Several of the vision studies cited here report bottlenecks. Zhai et al. (2022, §2.1) found that representation quality can be bottlenecked by model size: the Ti/16 model reached the same error whatever the dataset size, and the curves for Ti/16 overlapped, while the largest models still improved when the data grew from one to three billion images; L/16 on a 30-million-image subset did not improve past 27% 10-shot error, against 19% on larger sets. Big Transfer (BiT) found limited benefit from a large model on a small dataset and limited, or even negative, benefit from a small model on a large one: ResNet-50x1 pretrained on JFT-300M could do worse than the same network pretrained on ImageNet-21k, so that, the authors write, someone using only that network might conclude that more data brings no benefit (Kolesnikov et al., 2020, §4.1, Fig. 4). Cherti et al. (2023, §4.1, §5) found OpenCLIP ViT-B/32 and B/16 bottlenecked by the number of samples seen: more data helped only once the models saw 34 billion samples instead of 13 billion.

Bigger models are more sample-efficient. Zhai et al. (2022, §2.3, Fig. 3) found that for 10-shot ImageNet error, Ti/16 needed to see nearly 100 times more images than L/16 to reach the same error, and about 20 times more for fine-tuning, and concluded that with enough data, training a larger model for fewer steps is preferable. Their conclusion states the allocation rule directly: not increasing a model’s size when extra compute becomes available is suboptimal.

Exponents describe a dataset as well as an architecture. Fitting $E = \beta C^\alpha$ to zero-shot ImageNet error, Cherti et al. (2023, §4.1) measured $\alpha = -0.11$ for OpenCLIP models trained on LAION and $-0.16$ for OpenAI’s CLIP models trained on the private WIT-400M; for zero-shot retrieval the order reversed, with $-0.08$ against $-0.05$ on MS-COCO and $-0.19$ against $-0.10$ on Flickr30K (§4.2). They attribute the difference to the training data, and suggest that LAION’s construction, filtered with an OpenAI CLIP model’s image–text similarity, may have made it better suited to retrieval (§5).

Downstream and transfer error follow laws too, with caveats. The vision studies measure transfer, not the pretraining loss: Zhai et al. measure few-shot linear probes and fine-tuning, Cherti et al. zero-shot classification and retrieval, linear probing and fine-tuning, and Dehghani et al. (2023) linear probes of frozen features. Cherti et al. (2023, §2) note that upstream performance does not always correlate with downstream performance, and that in their own results the slope relating downstream accuracy to pretraining compute depended on the task and the pretraining data (§4.4). Robustness benchmarks moved with ImageNet: Cherti et al. found the OpenCLIP exponent for five ImageNet distribution-shift sets ($-0.13$) close to that for ImageNet ($-0.11$), and Dehghani et al. (2023, §4.2.3) found out-of-distribution accuracy rising with model size in line with ImageNet, while fine-tuning lowered effective robustness.

Scale changes other properties. Dehghani et al. (2023) report that scaling to 22 billion parameters improved the trade-off between fairness and performance and reduced disparities between subgroups, though scale by itself did not change demographic parity (§4.5.1), and that its fine-tuned models had the highest shape bias recorded for vision models, 87% for the 384-pixel model against 96% for humans (§4.5.2).

Evidence in Vision

The vision studies differ in data, models and measures, so their numbers are not comparable across rows; each is summarized with its own set-up.

Study	Scaled	Data	Models	Measure	Shape reported
Sun et al. (2017)	Data	JFT-300M subsets, 10M–300M	ResNet-50/101/152	COCO minival mAP@[.5,.95], VOC 2007 mAP@.5, VOC 2012 segmentation	Logarithmic in data
Mahajan et al. (2018)	Data, model	Instagram, 3.5M–3.5B images, 1.5k or 17k hashtags	ResNeXt-101 32×{4,8,16,32,48}d	ImageNet-1k/5k/9k and CUB linear probe; fine-tuned top-1	Near log-linear, breaking at the top
Kolesnikov et al. (2020)	Data, model	ImageNet-1k, ImageNet-21k, JFT-300M	ResNet-50x1 to ResNet-152x4	Transfer accuracy on ImageNet, Pets, CIFAR-100	Model and data must grow together
Zhai et al. (2022)	Compute, model, data	JFT-3B subsets, 30M–3B; ImageNet-21k	ViT, 5M–2B parameters	ImageNet 10-shot and fine-tuned error	Double-saturating power law
Dehghani et al. (2023)	Model	JFT, about 4B images	ViT up to 22B parameters	ImageNet linear probe and others	Gains continue; ImageNet saturating
Cherti et al. (2023)	Compute, model, data, samples seen	LAION-80M, 400M, 2B	CLIP ViT-B/32 to g/14	Zero-shot, retrieval, linear probe, fine-tuning	Power law; exponent depends on data
Goyal et al. (2024)	Compute, data quality	DataComp medium pool, 128M	CLIP ViT-B/32	Average zero-shot accuracy, 18 tasks	Law with utility decaying under repetition

Sun et al. (2017). Training ResNets on JFT-300M, a set of 300 million images with more than 375 million noisy labels and about 20% error in label precision, Sun, Shrivastava, Singh and Gupta found that performance on vision tasks increased logarithmically with the amount of pretraining data; their data-size experiments covered object detection and semantic segmentation. Their detection study pretrained on random subsets of 10, 30, 100 and 300 million images; the logarithmic trend was clearest when the feature layers were frozen (§5.2, Fig. 4). Capacity mattered: the COCO detection gain from JFT-300M was 1.87% for ResNet-50 against 3% for ResNet-152. With a ResNet-101 Faster R-CNN on COCO test-dev, pretraining on JFT-300M from an ImageNet initialization raised mAP@[.5,.95] from 34.3 to 37.4 (Table 2). The authors add that their training schedules were developed for a million images, so the results likely underestimate the effect of data (§6).

Mahajan et al. (2018). Mahajan and colleagues trained ResNeXt-101 networks to predict hashtags on up to 3.5 billion Instagram images and found near log-linear behaviour: each time the data were multiplied by a factor, linear-probe accuracy on ImageNet rose by a fixed amount, with steeper lines for higher-capacity networks, which they read as current architectures being prone to underfitting at that scale (§3.1.2, Fig. 2). They also report where the log-linear trend broke: accuracy is bounded, so the ceiling forces sub-log-linear scaling on ImageNet-1k and CUB, and the trend deviated between 1 billion and 3.5 billion images even on ImageNet-5k and 9k, where no ceiling was apparent. Their best fine-tuned model, ResNeXt-101 32×48d pretrained on 940 million images with 1.5k hashtags, reached 85.4% top-1 on ImageNet-1k, against about 79.6% for the 32×16d model trained from scratch, where larger models did not help (§3.1.5, Fig. 5).

Big Transfer (2020). Kolesnikov et al. trained ResNets of five sizes on ImageNet-1k (1.3M images), ImageNet-21k (14M) and JFT-300M and measured transfer. The benefit of larger models was small on ImageNet-1k and larger on the two bigger datasets, and the authors’ Fig. 4 caption warns that using only more data or only larger models may hurt (§4.1). They also found that enough compute is needed to see a dataset’s benefit: trained for ImageNet-1k’s standard budget of about 100 million images seen, the ImageNet-21k model was worse on ImageNet, and only longer training recovered and then exceeded it (§4.2, Fig. 6).

Zhai et al. (2022). This study fitted laws, rather than trends, to vision models. Zhai, Kolesnikov, Houlsby and Beyer trained vision transformers from 5 million to 2 billion parameters on 1 million to 3 billion images, with compute from under one to over 10,000 TPUv3 core-days (§1), and characterized the frontier of error against compute with the double-saturating law above. They called theirs the first study of scaling laws for discriminative image models (§4). Their two-billion-parameter ViT-G/14 reached 90.45% top-1 on ImageNet after fine-tuning and 84.86% with 10-shot linear evaluation (Table 1, Fig. 1). They checked that the law also held on the public ImageNet-21k (§2.4) and cautioned that their conclusions may not generalize beyond the scales studied or beyond the ViT family (§6). The vision transformer article covers the architecture changes behind the larger models.

ViT-22B (2023). Dehghani et al. scaled a ViT to 22 billion parameters, trained on a JFT dataset of about 4 billion images (§4.1). Linear probes on frozen features improved with size in each step of their Table 2, from 80.18% ImageNet top-1 for B/32 to 88.98% for G/14, 89.26% for e/14 and 89.51% for 22B, with the steps shrinking as accuracy approached saturation; on ObjectNet the gain from e/14 to 22B was larger, 71.54% to 74.30% (Table 2). They fitted no law; the evidence is a continued trend.

Cherti et al. (2023). Cherti et al. trained CLIP models with the open-source OpenCLIP code on LAION subsets of 80 million, 400 million and 2 billion image–text pairs, with 3, 13 and 34 billion samples seen, and found power laws for zero-shot classification, retrieval, linear probing and fine-tuning (§3–4). The CLIP article gives the resulting models; their ViT-H/14 reached 78.0% zero-shot top-1 on ImageNet (Table 1). From the fitted laws they predicted 79.1% for a ViT-g/14 trained on LAION-2B with 34 billion samples, and 81.9% for a ViT-G/14 with 68 billion (§4.1, supplementary Table 4). For the H/14 model at 68 billion samples, their supplementary Table 4 gives a predicted MS-COCO image-retrieval recall@5 of 75.03%, while the text of §B.2.1 gives 75.10%.

Where the Laws Break

Floors and ceilings. A pure power law predicts zero error at infinite scale, which no real task allows. Mahajan et al. (2018) saw the ceiling bend their ImageNet-1k curves, Zhai et al. (2022) built a floor $c$ into the law, and the ViT-22B linear probes gained 0.25 points of ImageNet accuracy between the two largest models (Dehghani et al., 2023, Table 2). Zhai et al. read their ReaL results, where ViT-G/14 beat earlier models by only a small margin, as a sign that ImageNet classification is likely reaching saturation (§2.5). A benchmark near its ceiling stops measuring scale; benchmark overfitting discusses the related problem of years of tuning on one test set.

Data quality and repetition. A law fitted to one data source does not transfer to another. Beyond the dataset-dependent exponents of Cherti et al., Goyal et al. (2024) showed that the value of filtering depends on the compute budget. Training a CLIP ViT-B/32 on the 128-million-pair DataComp medium pool, the LAION-style filter that keeps about 10% of the pairs beat no filtering at small budgets, by 7.5 points of average zero-shot accuracy at 128 million samples seen, but beyond 450 million samples seen, by which point each filtered pair had been seen about 32 times, the unfiltered pool was better (§3.2, Fig. 2). They explain this as the utility of an example decaying with each repetition and extend the law with a utility exponent that decays with the number of times data are seen (§4). DINOv2’s curated data, summarized in the foundation models article, is another case where what is in the data mattered as much as how much.

Bottlenecks hide gains. A law measured with one resource held too low can look flat. In Cherti et al. (2023), the benefit of LAION-2B over LAION-400M would have stayed hidden had the number of samples seen been kept at 13 billion (§5), and in BiT, ImageNet-21k looked worse than ImageNet-1k under the shorter schedule (Kolesnikov et al., 2020, §4.2).

More is sometimes worse. Several results run against “bigger is better”: BiT’s ResNet-50x1 doing worse with JFT-300M than with ImageNet-21k; Mahajan et al.’s networks pretrained on 1.5k hashtags matched to ImageNet’s classes beating those trained on 17k hashtags on ImageNet-1k, while the 17k-hashtag models were 7 points better on ImageNet-9k (§3.1.2); and Goyal et al.’s filtered data losing to unfiltered data at large budgets. Each is specific to its set-up, and none says that scale hurts in general. Overparameterized models also show a non-monotone error curve near the point where they first fit the training set exactly, the double descent that the regularization article describes; the example below shows how it breaks a fitted law.

Extrapolation range. Cherti et al. (2023, §5) write that extrapolation has its limits because saturation has been observed at both ends of the scale range, and that one can extrapolate to scales close to those measured. They also note that their scales were sampled sparsely and that hyperparameters were tuned only at small scales (§5). Zhai et al. (2022, §2.1) found that at the largest size their models started to fall behind the power-law frontier.

Fitting and Extrapolating in Practice

The sources suggest a protocol, stated here as practice rather than as a law:

• Fit the frontier, not every run. Smaller models, or models trained on fewer images, saturate and fall off the frontier when trained for longer (Zhai et al., 2022, Fig. 2); fitting all runs mixes allocation with scaling.
• Choose the form and say why. A pure power law cannot represent a floor or the shoulder at small scale. Adding $c$ and $d$ adds parameters that a short range of points determines poorly, as the example below shows.
• Choose the fit’s error measure. Fitting a line in log-log space, as Cherti et al. did, weights each scale equally in relative terms; fitting the error itself weights the small, high-error scales more. Zhai et al. do not say which they used; the example prints both.
• Hold out the largest scales. The only test of a law’s predictions is a run it was not fitted to; the example fits on small scales and scores the prediction at the largest.
• Report the spread. Small-scale points are noisy across seeds, and the fitted exponent inherits that noise.
• Keep the recipe fixed or re-tuned consistently. Schedules tuned for small scale can understate large-scale results (Sun et al., 2017, §6; Kolesnikov et al., 2020, §4.2).

Large runs need the systems side too: distributed training and mixed-precision training are what make the compute budgets of these studies affordable, and both the ViT and the CLIP studies report precision changes made at scale: Zhai et al. stored Adam’s momentum in half precision to save memory (§3.4), and Cherti et al. switched from float16 to bfloat16 to avoid loss spikes (§3.2).

Examples

Fitting and extrapolating learning curves of a closed-form model

No deep network is trained here. The stand-in is ridge regression on fixed random features, fitted in closed form, whose error can be measured on a grid of training-set sizes $n$ and feature counts $p$ in seconds. Each image is $16 \times 16$ with one antialiased disk of random radius $r \in [2, 6]$ pixels, position (the disk lies fully inside the frame) and brightness, plus pixel noise; the target is the radius, with Gaussian label noise of standard deviation 0.25 on both training and test targets. So no predictor can have an expected test mean squared error below $0.25^2 = 0.0625$, a known part of the irreducible error, and predicting the mean radius gives about 1.38. The model is ridge regression on $p$ random ReLU features $\max(0, Wx + b)$ of the pixels, the first $p$ of 2,048, with penalty $n\lambda\lVert w\rVert^2$ and $\lambda$ set to a fixed multiple (1e-2) of the mean squared feature. Training sets are nested (the first $n$ of 8,192 images), each of seeds 0–4 draws new features and new images, and 4,000 test images are shared. A second penalty, 1e-6, is nearly ridgeless.

Three laws are fitted: a pure power law, a power law with a floor $c$, and Zhai et al.’s double-saturating form. Whether to fit the error or its logarithm is a choice the sources leave open, so each law is fitted both ways (“log” and “linear”). Each fit uses only the smaller scales and is scored on the largest. The first block checks the ridge solver against numpy.linalg.lstsq and the fitting code on noise-free curves drawn from each law; the reference line is the seed-0 model with $p = 2{,}048$ fitted on 131,072 images, an estimate of where its curve in $n$ levels off. The last part treats $C = np$, the cost of one pass of gradient descent over $n$ images with $p$ features, as a stand-in for compute, and finds the best split of each budget. The script uses NumPy 2.3.5 and SciPy 1.16.3; a run took between about 40 seconds and 3 minutes on a CPU shared with other jobs, with identical output on every run.

import warnings
import numpy as np
from scipy.optimize import least_squares
warnings.filterwarnings("ignore", category=RuntimeWarning, module="scipy.optimize")

# Synthetic images: one antialiased disk per 16x16 image, random radius r in [2, 6],
# position (fully inside the frame) and brightness, plus pixel noise.
# Target: the radius, with Gaussian label noise (sd 0.25) on training AND test targets,
# so no predictor can have expected test MSE below 0.25**2 = 0.0625.
H = W = 16
SIGMA_Y, PIX_NOISE = 0.25, 0.1

def images(n, rng):
    r = rng.uniform(2.0, 6.0, n)
    cy, cx = rng.uniform(r, H - r), rng.uniform(r, W - r)
    br = rng.uniform(0.5, 1.0, n)
    yy, xx = np.mgrid[0:H, 0:W] + 0.5
    d = np.hypot(yy[None] - cy[:, None, None], xx[None] - cx[:, None, None])
    img = br[:, None, None] * np.clip(r[:, None, None] - d + 0.5, 0, 1)
    img += PIX_NOISE * rng.standard_normal(img.shape)
    margin = np.minimum.reduce([cy - r, H - cy - r, cx - r, W - cx - r]).min()
    return img.reshape(n, -1), r + SIGMA_Y * rng.standard_normal(n), r, margin

# Model: ridge regression on p fixed random ReLU features (the first p of 2,048),
# fitted in closed form on the first n training images (nested sets).
P_MAX, N_MAX = 2048, 8192
NS = 2 ** np.arange(5, 14)        # 32 ... 8192 training images
PS = 2 ** np.arange(4, 12)        # 16 ... 2048 features
LAMS = {"ridge 1e-2": 1e-2, "near-ridgeless 1e-6": 1e-6}   # relative penalty

def features(X, Wf, bf):
    return np.maximum(X @ Wf + bf, 0.0)

def ridge_grid(F, y, F_te, y_te, lam_rel):
    """Test MSE for every (n, p) on the grid; penalty n*lam*|w|^2, lam = lam_rel * mean F^2."""
    err = np.zeros((len(NS), len(PS)))
    for i, n in enumerate(NS):
        Fn, yn = F[:n], y[:n]
        for j, p in enumerate(PS):
            mu, ym = Fn[:, :p].mean(0), yn.mean()
            Fc = Fn[:, :p] - mu
            lam = lam_rel * (Fn[:, :p] ** 2).mean()
            w = np.linalg.solve(Fc.T @ Fc + n * lam * np.eye(p), Fc.T @ (yn - ym))
            err[i, j] = (((F_te[:, :p] - mu) @ w + ym - y_te) ** 2).mean()
    return err

# Three laws, each fitted by least squares on log error ("log") or on error ("linear").
def f_pow(x, a, b): return a * x ** (-b)
def f_pc(x, a, b, c): return a * x ** (-b) + c
def f_zhai(x, a, b, c, d): return a * (x + d) ** (-b) + c
FORMS = {"power  a x^-b": f_pow, "power+c  a x^-b + c": f_pc, "Zhai  a (x+d)^-b + c": f_zhai}

def fit(form, x, E, space):
    f = FORMS[form]
    sl, ic = np.polyfit(np.log(x), np.log(E), 1)
    a0, b0 = np.exp(ic), max(-sl, 1e-3)
    cs = (0.0, 0.5 * E.min(), 0.9 * E.min())
    starts = {f_pow: [[a0, b0]], f_pc: [[a0, b0, c] for c in cs],
              f_zhai: [[a0, b0, c, d] for c in cs for d in (0.0, x.min())]}[f]
    if space == "log":
        res = lambda q: np.log(f(x, *q)) - np.log(E)
    else:
        res = lambda q: (f(x, *q) - E) / E.mean()
    fits = [least_squares(res, s, bounds=(0, np.inf), x_scale="jac",
                          xtol=1e-15, ftol=1e-15, gtol=1e-15, max_nfev=20000) for s in starts]
    return min(fits, key=lambda o: o.cost).x

def report(title, x, E, n_fit):
    print(f"\n{title}: fit on the first {n_fit} points, extrapolate to x = {x[-1]}")
    print(f"  {'law':22s} {'fit on':6s}  {'pred at max':>11s} {'meas':>7s} {'rel err':>8s} {'c':>7s}")
    for form in FORMS:
        for space in ("log", "linear"):
            q = fit(form, x[:n_fit], E[:n_fit], space)
            pred = FORMS[form](x[-1], *q)
            c = f"{q[2]:.4f}" if len(q) > 2 else "   -"
            print(f"  {form:22s} {space:6s}  {pred:11.4f} {E[-1]:7.4f} {(pred - E[-1]) / E[-1]:+8.1%} {c:>7s}")

# ---- checks ----
rng = np.random.default_rng(0)
A = rng.standard_normal((500, 300)); t = rng.standard_normal(500)
Ac, tc = A - A.mean(0), t - t.mean()
w1 = np.linalg.solve(Ac.T @ Ac + 500 * 1e-2 * np.eye(300), Ac.T @ tc)
w2 = np.linalg.lstsq(np.vstack([Ac, np.sqrt(500 * 1e-2) * np.eye(300)]),
                     np.concatenate([tc, np.zeros(300)]), rcond=None)[0]
print(f"check, ridge solve vs lstsq: max |dw| = {np.abs(w1 - w2).max():.1e}")
TRUE = {f_pow: [3.0, 0.4], f_pc: [3.0, 0.5, 0.07], f_zhai: [3.0, 0.5, 0.07, 40.0]}
worst = 0.0
for form, f in FORMS.items():
    for space in ("log", "linear"):
        E = f(NS.astype(float), *TRUE[f])
        q = fit(form, NS[:6].astype(float), E[:6], space)
        worst = max(worst, abs(f(NS[-1], *q) - E[-1]) / E[-1])
print(f"check, noise-free curves of each law: worst rel. extrapolation error = {worst:.1e}")

X_te, y_te, r_te, m_te = images(4000, np.random.default_rng(1000))
print(f"check, test set: label-noise sd {np.std(y_te - r_te):.3f}, "
      f"min disk margin {m_te:.4f}, predict-the-mean MSE {((y_te - y_te.mean()) ** 2).mean():.3f}")

# ---- learning curves over seeds 0-4 (new features and new training images each) ----
SEEDS = range(5)
err = {k: np.zeros((len(SEEDS), len(NS), len(PS))) for k in LAMS}
for s in SEEDS:
    rng = np.random.default_rng(s)
    X, y, r, m = images(N_MAX, rng)
    scale = np.sqrt((X ** 2).sum(1).mean())
    Wf, bf = rng.standard_normal((H * W, P_MAX)) / scale, rng.standard_normal(P_MAX)
    F, F_te = features(X, Wf, bf), features(X_te, Wf, bf)
    for k, lam in LAMS.items():
        err[k][s] = ridge_grid(F, y, F_te, y_te, lam)
    if s == 0:   # reference: the same model (seed 0, p = 2048, ridge 1e-2) fitted on 131,072 images
        G, Fy, Fs, ys, n_ref = 0, 0, 0, 0, 0
        for chunk in range(16):
            Xc, yc, _, _ = images(8192, np.random.default_rng(10_000 + chunk))
            Fc = features(Xc, Wf, bf)
            G, Fy, Fs, ys, n_ref = G + Fc.T @ Fc, Fy + Fc.T @ yc, Fs + Fc.sum(0), ys + yc.sum(), n_ref + 8192
        mu, ym = Fs / n_ref, ys / n_ref
        Gc = G - n_ref * np.outer(mu, mu)
        lam = 1e-2 * np.trace(G) / (n_ref * P_MAX)
        w = np.linalg.solve(Gc + n_ref * lam * np.eye(P_MAX), Fy - n_ref * mu * ym)
        ref = (((F_te - mu) @ w + ym - y_te) ** 2).mean()
        print(f"check, seed 0 training set: label-noise sd {np.std(y - r):.3f}, radius "
              f"{r.min():.2f}-{r.max():.2f}, min disk margin {m:.4f}")

E = {k: v.mean(0) for k, v in err.items()}
print(f"\ntest MSE, p = 2048, mean over seeds 0-4 (min-max); noise floor sigma^2 = {SIGMA_Y**2:.4f}")
for k in LAMS:
    print(f"  {k}")
    for i, n in enumerate(NS):
        e = err[k][:, i, -1]
        print(f"    n = {n:5d}: {e.mean():.4f} ({e.min():.4f}-{e.max():.4f})")
print(f"reference: seed 0, p = 2048, ridge 1e-2, n = 131,072: test MSE {ref:.4f}")
print(f"check, smallest test MSE anywhere: {min(v.min() for v in err.values()):.4f}")

for k in LAMS:
    report(f"error vs training images n, p = 2048, {k}", NS, E[k][:, -1], 6)
print("\ntest MSE, n = 8192, ridge 1e-2, mean over seeds 0-4 (min-max)")
for j, p in enumerate(PS):
    e = err["ridge 1e-2"][:, -1, j]
    print(f"    p = {p:5d}: {e.mean():.4f} ({e.min():.4f}-{e.max():.4f})")
report("error vs features p, n = 8192, ridge 1e-2", PS, E["ridge 1e-2"][-1, :], 5)

# ---- compute frontier: C = n * p, the cost of one pass of gradient descent ----
Em = E["ridge 1e-2"]
C = np.outer(NS, PS)
print("\ncompute-optimal allocation, ridge 1e-2 (best mean test MSE for each C = n p)")
Cs, front = np.unique(C), []
for c in Cs:
    i, j = np.argwhere(C == c)[np.argmin(Em[C == c])]
    front.append(Em[i, j])
    print(f"  C = 2^{int(np.log2(c)):2d}: n = {NS[i]:5d}, p = {PS[j]:5d}, MSE {Em[i, j]:.4f}")
front = np.array(front)
k_fit = int(np.sum(Cs <= 2 ** 20))
report("frontier error vs compute C", Cs.astype(float), front, k_fit)

Output:

check, ridge solve vs lstsq: max |dw| = 9.0e-16
check, noise-free curves of each law: worst rel. extrapolation error = 1.2e-15
check, test set: label-noise sd 0.251, min disk margin 0.0000, predict-the-mean MSE 1.376
check, seed 0 training set: label-noise sd 0.249, radius 2.00-6.00, min disk margin 0.0002

test MSE, p = 2048, mean over seeds 0-4 (min-max); noise floor sigma^2 = 0.0625
  ridge 1e-2
    n =    32: 0.3379 (0.2977-0.3866)
    n =    64: 0.2915 (0.2391-0.3595)
    n =   128: 0.2054 (0.1950-0.2114)
    n =   256: 0.1907 (0.1783-0.1992)
    n =   512: 0.1648 (0.1529-0.1792)
    n =  1024: 0.1373 (0.1301-0.1411)
    n =  2048: 0.1132 (0.1097-0.1177)
    n =  4096: 0.1004 (0.0988-0.1022)
    n =  8192: 0.0927 (0.0918-0.0941)
  near-ridgeless 1e-6
    n =    32: 0.3380 (0.2977-0.3867)
    n =    64: 0.2917 (0.2391-0.3598)
    n =   128: 0.2064 (0.1959-0.2126)
    n =   256: 0.1998 (0.1855-0.2103)
    n =   512: 0.1995 (0.1862-0.2203)
    n =  1024: 0.2800 (0.2768-0.2852)
    n =  2048: 1.5321 (1.4272-1.6398)
    n =  4096: 0.1749 (0.1723-0.1773)
    n =  8192: 0.1114 (0.1094-0.1145)
reference: seed 0, p = 2048, ridge 1e-2, n = 131,072: test MSE 0.0854
check, smallest test MSE anywhere: 0.0918

error vs training images n, p = 2048, ridge 1e-2: fit on the first 6 points, extrapolate to x = 8192
  law                    fit on  pred at max    meas  rel err       c
  power  a x^-b          log          0.0784  0.0927   -15.4%       -
  power  a x^-b          linear       0.0744  0.0927   -19.7%       -
  power+c  a x^-b + c    log          0.1074  0.0927   +15.9%  0.0867
  power+c  a x^-b + c    linear       0.1047  0.0927   +13.0%  0.0811
  Zhai  a (x+d)^-b + c   log          0.1096  0.0927   +18.2%  0.0922
  Zhai  a (x+d)^-b + c   linear       0.1282  0.0927   +38.3%  0.1257

error vs training images n, p = 2048, near-ridgeless 1e-6: fit on the first 6 points, extrapolate to x = 8192
  law                    fit on  pred at max    meas  rel err       c
  power  a x^-b          log          0.1772  0.1114   +59.0%       -
  power  a x^-b          linear       0.1650  0.1114   +48.1%       -
  power+c  a x^-b + c    log          0.2199  0.1114   +97.3%  0.2199
  power+c  a x^-b + c    linear       0.2223  0.1114   +99.5%  0.2223
  Zhai  a (x+d)^-b + c   log          0.2191  0.1114   +96.6%  0.2191
  Zhai  a (x+d)^-b + c   linear       0.2215  0.1114   +98.7%  0.2215

test MSE, n = 8192, ridge 1e-2, mean over seeds 0-4 (min-max)
    p =    16: 0.5475 (0.4248-0.6977)
    p =    32: 0.3955 (0.3393-0.5468)
    p =    64: 0.2918 (0.2570-0.3928)
    p =   128: 0.2089 (0.1917-0.2398)
    p =   256: 0.1474 (0.1422-0.1525)
    p =   512: 0.1140 (0.1107-0.1181)
    p =  1024: 0.0983 (0.0973-0.0999)
    p =  2048: 0.0927 (0.0918-0.0941)

error vs features p, n = 8192, ridge 1e-2: fit on the first 5 points, extrapolate to x = 2048
  law                    fit on  pred at max    meas  rel err       c
  power  a x^-b          log          0.0562  0.0927   -39.4%       -
  power  a x^-b          linear       0.0574  0.0927   -38.0%       -
  power+c  a x^-b + c    log          0.0562  0.0927   -39.4%  0.0000
  power+c  a x^-b + c    linear       0.0574  0.0927   -38.0%  0.0000
  Zhai  a (x+d)^-b + c   log          0.0534  0.0927   -42.4%  0.0000
  Zhai  a (x+d)^-b + c   linear       0.0556  0.0927   -40.0%  0.0000

compute-optimal allocation, ridge 1e-2 (best mean test MSE for each C = n p)
  C = 2^ 9: n =    32, p =    16, MSE 0.7966
  C = 2^10: n =    64, p =    16, MSE 0.6624
  C = 2^11: n =   128, p =    16, MSE 0.6177
  C = 2^12: n =   128, p =    32, MSE 0.4850
  C = 2^13: n =    32, p =   256, MSE 0.3882
  C = 2^14: n =   256, p =    64, MSE 0.3455
  C = 2^15: n =   256, p =   128, MSE 0.2914
  C = 2^16: n =   512, p =   128, MSE 0.2456
  C = 2^17: n =   512, p =   256, MSE 0.1942
  C = 2^18: n =  1024, p =   256, MSE 0.1686
  C = 2^19: n =  1024, p =   512, MSE 0.1423
  C = 2^20: n =  2048, p =   512, MSE 0.1251
  C = 2^21: n =  2048, p =  1024, MSE 0.1133
  C = 2^22: n =  4096, p =  1024, MSE 0.1039
  C = 2^23: n =  8192, p =  1024, MSE 0.0983
  C = 2^24: n =  8192, p =  2048, MSE 0.0927

frontier error vs compute C: fit on the first 12 points, extrapolate to x = 16777216.0
  law                    fit on  pred at max    meas  rel err       c
  power  a x^-b          log          0.0606  0.0927   -34.6%       -
  power  a x^-b          linear       0.0620  0.0927   -33.1%       -
  power+c  a x^-b + c    log          0.0634  0.0927   -31.6%  0.0065
  power+c  a x^-b + c    linear       0.0620  0.0927   -33.1%  0.0000
  Zhai  a (x+d)^-b + c   log          0.0730  0.0927   -21.3%  0.0332
  Zhai  a (x+d)^-b + c   linear       0.0726  0.0927   -21.6%  0.0325

The checks pass: the ridge solver agrees with lstsq to rounding, the fitting code recovers each law exactly from noise-free points, the data have the stated label noise and margins, and no measured error falls below the 0.0625 noise floor. The largest model, at 0.093, is far below the 1.38 of predicting the mean, so the target is learnable, but it stays above the reference of 0.085: even 131,072 images leave this model about 0.023 above the label noise, which in our reading comes mostly from what 2,048 random features and this penalty can represent.

Data. With the ridge penalty, the error in $n$ falls steadily, but not as one straight line on a log-log plot: it drops sharply between 64 and 128 images, flattens to 256, then falls again at a slowly decreasing rate. Fitted on 32 to 1,024 images, the pure power law under-predicts the error at 8,192 by 15–20%, overstating the gain from eight times more data, because it cannot bend toward a floor. We expected the two forms with a floor to extrapolate better. That expectation did not hold: they over-predict the error by 13–38%, misses as large as the power law’s in the opposite direction, because six points spanning one and a half decades leave $c$ poorly determined. We also expected the fitted floor to land near the reference of 0.085. That held for the fits on log error (0.087 and 0.092) and for the fit of the power law with a floor on error itself (0.081), but not for the fit of Zhai et al.’s form on error itself, which put the floor at 0.126, above the error already measured at 8,192 images. The same law fitted to the same points with a different error measure moved the prediction by 20 percentage points.

Near the interpolation threshold. With an almost zero penalty, the curve stops falling at 256–512 images, rises at 1,024 and jumps to 1.53 at 2,048, where the number of training images equals the number of features and the fit nearly interpolates the noisy targets; at that point it is worse than predicting the mean. It then falls again to 0.111 at 8,192. This is the double descent that the regularization article explains. For scaling laws, the lesson is that no monotone law survives it: fitted on the rising points, every form predicts an error at 8,192 images that is 48–100% too high, and the forms with a floor put it at about 0.22, inside the range of the fitted points, leaving nothing to fall.

Model size. With 8,192 images, the error in $p$ looks like a clean power law up to 256 features, and the fits on those points find no floor at all: $c$ goes to its bound of zero in every form. From 512 features on the curve bends, and all six fits under-predict the error at 2,048 features by 38–42%. In our reading, this is the bottleneck that Zhai et al. and BiT describe, in miniature: at fixed data, more parameters stop helping, and nothing in the first five points says where that happens, since the bend is set by $n$, which those points do not vary.

Compute. For each budget $C = np$, the best split moves toward larger $p$ and larger $n$ together, with the ratio $n/p$ between 2 and 8 above $2^{13}$; at $2^{13}$ the best point is an outlier with only 32 images, the smallest $n$, where the printed seed spread at $p = 2{,}048$ is widest. Holding either resource fixed would leave the frontier, as Zhai et al. found for ViTs. Fitted to the frontier up to $C = 2^{20}$, every law under-predicts the error at $2^{24}$, the pure power law by about a third and Zhai et al.’s form by about a fifth, the smallest miss here. Its fitted floor, 0.033, is still below the 0.0625 that the label noise alone imposes, so the best extrapolation in this example rests on a floor that is known to be wrong.

Common Misconceptions

• “Scaling laws are laws of nature.” They are fits to experiments within one family of models, data and recipes. Cherti et al. (2023) found the exponent changing with the dataset, and Zhai et al. (2022) cautioned that their conclusions may not hold beyond the scales and model family studied.
• “A straight line on a log-log plot means error will keep falling.” Mahajan et al., Zhai et al. and Dehghani et al. report saturation or a ceiling, and the example shows a pure power law under-predicting the error at the next scale.
• “More data always helps.” Only while the model is large enough and trained long enough to use it: BiT’s smallest ResNet did worse with JFT-300M, and filtered data repeated many times lost to unfiltered data (Goyal et al., 2024).
• “The fitted floor is the task’s irreducible error.” It is a parameter fitted from a short range of points. In the example, fits of the same curve placed it anywhere from 0.08 to 0.13, and on the compute frontier below the known noise floor.
• “Scaling laws are about model size.” They are about allocation. Zhai et al. found that not growing the model when compute grows is suboptimal, and the illustrative derivation above shows why neither resource should be held fixed.

Where It Is Used

Scaling laws are used to plan pretraining runs for vision transformers, CLIP-style image–text models and other foundation models: to predict the gain from a larger model or dataset, to choose model size for a compute budget, and to compare datasets by how fast models trained on them improve (Cherti et al., 2023). They inform data curation, as in Goyal et al.’s compute-aware filtering, and the choice of pretraining data and schedules for transfer learning. Generative models have their own scaling results; the autoregressive image model article reports test loss falling as a power law of model size and compute for one such family. Model-design rules that scale one network, such as EfficientNet’s compound scaling of depth, width and resolution, are a different subject: they fix how to grow an architecture, not how error depends on resources.