Masked Image Modeling

Masked image modeling (MIM) pre-trains an image encoder without labels by hiding part of each image and training the network to predict what was hidden. After pre-training, the prediction head is thrown away and the encoder is fine-tuned for classification, detection or segmentation. The idea copies masked language modeling, in which a model predicts the missing words of a sentence. Images differ from text in ways that change almost every design choice. Nearby pixels are strongly correlated, pixel values are continuous rather than drawn from a vocabulary, and a pixel carries little meaning on its own.

The family took shape in 2021–2022 around the Vision Transformer, whose patch tokens can be dropped or replaced by a mask token one at a time. Its members differ in four choices, which SimMIM (Xie et al., 2022) names as the components of any MIM framework:

• the masking strategy;
• the encoder;
• the prediction head;
• the prediction target and its loss.

This article covers the family and the choice of target. Several methods have their own articles:

• The masked autoencoder (MAE) covers MAE’s asymmetric encoder, its ablations and its results.
• Self-supervised learning places MIM beside contrastive and self-distillation methods.
• VideoMAE and video foundation models cover masking in video.

Purpose

Masked modeling learns from unlabeled images with a single view of each image. Contrastive and self-distillation methods compare two augmented views. MaskFeat’s authors contrast the two families: contrastive methods need a Siamese structure and carefully designed augmentation, while MaskFeat uses one network, one view and minimal augmentation (Wei et al., 2022).

The task is simple to state. Its difficulty is set by how much of the image is hidden and how it is hidden.

The approach answers a practical problem. MAE’s authors note that large models can easily overfit the one million images of ImageNet-1K, and MaskFeat’s that ViTs are data-hungry and had relied on large-scale supervised pre-training. MaskFeat’s ViT-L, pre-trained on ImageNet-1K without labels, fine-tuned to 85.7%. The same ViT-L pre-trained with labels on ImageNet-21K, which has about ten times more images, reached 85.2% (Wei et al., 2022, Table 7). The two were not run at the same image size: $224^2$ for MaskFeat and $384^2$ for the supervised model.

When to Use It

MIM suits encoders that will be fine-tuned. The papers evaluate it mainly by end-to-end fine-tuning on ImageNet-1K.

• SimMIM states that it does not account for linear probing, because its goal is features that complement downstream tasks.
• With ViT-B, SimMIM fine-tuned to 83.8% but reached only 56.7% with a linear probe on its best intermediate layer, against 78.2% for DINO (Xie et al., 2022, Table 6).
• MAE’s frozen features are likewise weaker than contrastive ones (see masked autoencoder).

If the encoder will stay frozen, as in retrieval or a linear classifier, a method whose features are linearly separable is the safer starting point. Two examples are DINO and the self-distillation hybrids below.

MIM also needs an encoder that can hide a patch cleanly. A transformer can drop a token or replace it with a learned mask token. As the ConvNeXt article describes for ConvNeXt V2, a convolution slides across the boundary and leaks information from the masked region, and ConvNeXt V2 answers this with sparse convolutions.

How It Works

An image $\mathbf{x}$ is cut into $L$ non-overlapping patches. A mask $\mathbf{m} \in \{0,1\}^L$ marks a subset $\mathcal{M}$ of them as hidden. The encoder sees the image with those patches removed or replaced by a learned mask token, and a head predicts a target $\mathbf{t}_i$ for each hidden patch. The loss is averaged over hidden patches only:

$$\mathcal{L} = \frac{1}{|\mathcal{M}|} \sum_{i \in \mathcal{M}} \ell\big(\hat{\mathbf{t}}_i, \mathbf{t}_i\big), \qquad \hat{\mathbf{t}} = h\big(f(\mathbf{x} \odot (1 - \mathbf{m}))\big),$$

where $f$ is the encoder, $h$ the head and $\ell$ a per-patch loss. SimMIM’s own form of this loss is an $\ell_1$ loss averaged over masked pixels.

Restricting the loss to hidden patches matters. SimMIM computed it over the whole image as an alternative, which makes the task part prediction and part reconstruction of visible pixels. Fine-tuning accuracy fell from 82.8% to 81.7% with a Swin-B encoder (Table 4). The authors read this as evidence that predicting the invisible and reconstructing the visible are different tasks.

Masking Strategy

Three families of mask appear in the papers:

• Random patches. Patches are drawn uniformly at random without replacement. This is MAE’s and SimMIM’s choice.
• Block-wise masks. Blocks of adjacent patches are masked together. As data2vec describes BEiT’s scheme, each block contains at least 16 patches and has a random aspect ratio.
• A single region. SimMIM’s movable square relaxes the central region masking that, as SimMIM describes it, Context Encoders introduced (Pathak et al., 2016); image inpainting describes Context Encoders filling a central square.

Two quantities govern the difficulty: the masking ratio (the fraction of patches hidden) and the masked patch size (the side of each hidden unit, which need not equal the encoder’s token size).

SimMIM varied both with a Swin-B encoder at $192^2$, pre-trained for 100 epochs and fine-tuned on ImageNet-1K (Table 1):

• Patch size 32. Random masking held 82.7–83.0% from 10% to 70% masked, peaked at 83.0% at 50%, and fell to 82.4% at 80% and 90%.
• Smaller patches. At sizes 4, 8 and 16, accuracy rose as the ratio grew from 0.4 to 0.8: from 81.9% to 82.1%, 82.0% to 82.4% and 82.4% to 82.8%. It stayed below the best size-32 results.
• Patch size 64. It scored 82.6% at ratios 0.1 and 0.2, where size 32 gave 82.7% and 82.8%. The text calls this degraded accuracy, but the drop in the table is 0.1–0.2 points.
• Block-wise masks gave at most 82.7%.
• The square mask gave 82.5–82.6%.

To explain these results, SimMIM introduced AvgDist, the average distance from a masked pixel to the nearest visible pixel. It grows with the masking ratio and with the masked patch size. Plotting accuracy against it gives a ridge: the best runs lie roughly between AvgDist 10 and 20 pixels, and runs with smaller or larger values do worse. The authors suggest that short distances may let the network learn too many short connections, while long ones may be too difficult to learn.

MAE reached the same conclusion from the ratio side. Its optimum was 75%, far above BERT’s 15%, and it attributes the gap to images being spatially redundant (see masked autoencoder for its ratio and mask-sampling ablations). SimMIM likewise hypothesizes that differing redundancy explains why images and text prefer different ratios.

The masking ratio used across the family therefore varies:

Method	Mask	Ratio
BEiT	block-wise	about 40% (as Self-Supervised Learning and data2vec describe)
MaskFeat (images)	block-wise, following BEiT	40%
SimMIM	random, 32-pixel patches	60%
data2vec	block-wise, following BEiT	60%
MAE	random	75%
VideoMAE	tube	90–95% (see VideoMAE)

Encoder and Mask Tokens

SimMIM, MaskFeat and data2vec replace each hidden patch with a learned mask token and run the encoder over the full sequence. MAE removes hidden patches from the encoder entirely and adds mask tokens only in a small decoder. This makes the encoder cheaper and avoids showing it inputs that never occur at test time; its article gives the costs.

Prediction Head

SimMIM found that the head can be a single linear layer:

• Accuracy. A linear head gave 82.8%, the same as a two-layer MLP. Inverse Swin-T and Swin-B heads gave 82.4% and 82.5% at 1.7× and 2.3× the training cost (Table 2).
• Loss. Heavier heads reached slightly lower pre-training loss, 0.3722 against 0.3743 for the inverse Swin-B against the linear head.
• Reading. The authors conclude that better inpainting does not imply better transfer, because capacity spent in the head is discarded.

MAE reports the opposite trade-off for linear probing: a deeper decoder raises linear accuracy, because it takes over the reconstruction-specific layers. The two papers measure different protocols.

Prediction Target

The target is where the family differs most. MaskFeat sorts targets into one-stage targets, computed directly from the image, and two-stage targets, which need another trained network.

Raw pixels.

• SimMIM regresses RGB values with an $\ell_1$ loss. $\ell_1$, smooth-$\ell_1$ and $\ell_2$ performed alike (82.7–82.8%, Table 5).
• Predicting a downsampled target worked down to $12^2$ for a $192^2$ input, and dropped only at $6^2$ (82.3%, Table 3).
• MaskFeat’s pixel-colour target gave 82.5% on ImageNet-1K with ViT-B. Its authors suggest, citing DALL-E, that pixels are “too explicit” a target, and note a risk of over-fitting to illumination and contrast variations and high-frequency detail.

Normalized pixels. MAE normalizes each target patch by its own mean and standard deviation, which improved its representations. The authors observe that the normalization enhances local contrast and that a PCA target, which drops high frequencies, did worse. The example below takes this target apart in closed form.

Discrete classes. MIM can copy language’s classification over a vocabulary:

• Visual tokens. As MaskFeat and SimMIM describe BEiT, a discrete VAE from DALL-E maps each patch to one of 8,192 tokens, and the model predicts the token with a cross-entropy loss. The tokenizer needs its own pre-training stage on 250 million images. See VQ-VAE for discrete image codes.
• Colour classes. SimMIM tested iGPT’s 512 colour clusters (82.4% at a $48^2$ target) and per-channel 8-bin colour classes (82.7%). It also tested BEiT’s tokenizer target in the same framework, which gave 82.7% against 82.8% for its $\ell_1$ regression.
• Conclusion. Both SimMIM and MaskFeat conclude that discretization is unnecessary for vision.

Hand-crafted features.

• MaskFeat predicts histograms of oriented gradients of the masked regions. That article gives the HOG settings and the normalization ablation.
• In MaskFeat’s ImageNet-1K comparison (ViT-B, 300 pre-training epochs, 100 epochs of fine-tuning; Table 2), HOG reached 83.6%. This beat pixels (82.5%), the DALL-E tokens (82.8%) and training from scratch (81.8%).

Features of a trained network. MaskFeat’s two-stage targets in the same table:

• Unsupervised teachers did best: DINO ViT-B gave 84.0% and MoCo v3 ViT-B 83.9%. The teachers themselves took 1,535 and 600 effective epochs to train.
• Supervised teachers did worse: 82.6% from a ResNet-50 and 81.9% from a DeiT ViT-B.
• Per-patch class pseudo-labels gave 78.8%, below training from scratch.
• The authors saw significant over-fitting with class-derived targets during fine-tuning. They suggest that features trained on class labels discard the local shape and texture that masked prediction needs.

Contextualized features of the model itself. data2vec uses a single network in two modes:

• In student mode it encodes the masked image.
• In teacher mode, with weights that are an exponential moving average of the student’s, it encodes the full image.
• The target for each masked patch averages the outputs of the teacher’s top $K$ Transformer blocks, each passed through a parameter-free layer normalization. The vision models use $K = 6$, and the student regresses the target with a Smooth L1 loss.
• The teacher sees the whole image through self-attention, so its targets carry context from all of it. The authors contrast this with the local targets of BEiT, MAE, SimMIM and MaskFeat.
• They note that normalizing the targets helps prevent collapse to a constant representation. As Self-Supervised Learning notes, fixed targets such as pixels cannot collapse, because they do not come from the network being trained.

Online tokenizers. iBOT, as Self-Supervised Learning describes it, replaces BEiT’s fixed tokenizer with an online teacher and combines masked prediction with DINO-style self-distillation. The same article covers DINOv2, which builds on it.

How to Apply It

The example asks how hard the masked-prediction task is when the best possible predictor is used. Treating that predictor as a stand-in for the network’s ability is our reading, not a claim of the papers.

The image is a $32 \times 32$ stationary Gaussian random field with unit variance, plus pixel noise of variance $10^{-4}$, cut into 64 patches of $4 \times 4$. For a Gaussian, the predictor with the lowest expected squared error of the hidden pixels $\mathbf{x}_M$ given the visible pixels $\mathbf{x}_V$ is the conditional mean:

$$\hat{\mathbf{x}}_M = \Sigma_{MV}\Sigma_{VV}^{-1}\mathbf{x}_V, \qquad \Sigma_{M\mid V} = \Sigma_{MM} - \Sigma_{MV}\Sigma_{VV}^{-1}\Sigma_{VM}.$$

The script reports the relative masked error $R = \operatorname{tr}\Sigma_{M\mid V} / \operatorname{tr}\Sigma_{MM}$. This is the expected squared error of the optimal predictor divided by that of predicting zero, the field’s mean. $R = 0$ means the hidden pixels can be copied from the visible ones, and $R = 1$ means nothing about them can be inferred.

• Grid and distances. Distances are Euclidean between pixel centres, in pixels. Patches are numbered in raster order on the $8 \times 8$ grid.
• Covariances. Real images follow neither covariance, so the script prints both: a smooth field with squared-exponential covariance $e^{-d^2/2\ell^2}$ and a rough field with exponential covariance $e^{-d/\ell}$. The correlation length $\ell$ takes the values 1, 2, 4 and 8 pixels.
• Masks.
  • 10, 26, 38, 48 or 58 of the 64 patches are hidden, about the 15%, 40%, 60%, 75% and 90% used by BERT, BEiT, SimMIM, MAE and VideoMAE.
  • Random masks draw the patches uniformly.
  • Block masks take the patches whose centres lie nearest a random point, giving one compact region.
  • Ten masks of each kind are shared by all fields.
• AvgDist is SimMIM’s measure, computed with a Euclidean distance transform. A hidden pixel that shares an edge with a visible one scores 1.
• Checks. Before the sweep, the script runs three checks:
  • It samples 2,000 fields per setting and compares their variance and neighbour covariance with the model.
  • It compares the closed-form error with the empirical error of the conditional mean on those fields.
  • It compares the formula with scikit-learn’s Gaussian-process regression, which computes the same conditional mean and variance.

The script uses NumPy 2.3.5, SciPy 1.16.3 and scikit-learn 1.7.2 on one thread. It takes about 15–20 seconds, and two runs gave identical output.

import os
os.environ["OMP_NUM_THREADS"] = os.environ["OPENBLAS_NUM_THREADS"] = "1"   # before NumPy loads
import numpy as np
from scipy.linalg import cho_factor, cho_solve
from scipy.ndimage import distance_transform_edt
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import RBF, Matern

N, P, EPS = 32, 4, 1e-4            # 32x32 image, 4x4 patches (an 8x8 grid of 64 patches), pixel-noise variance
G = N // P
ELLS = (1, 2, 4, 8)                # correlation lengths in pixels
COUNTS = (10, 26, 38, 48, 58)      # masked patches out of 64: about 15%, 40%, 60%, 75% and 90%
N_MASKS, N_FIELDS = 10, 2000

yy, xx = np.mgrid[:N, :N]
coords = np.c_[yy.ravel(), xx.ravel()].astype(float)
D = np.sqrt(((coords[:, None] - coords[None]) ** 2).sum(-1))           # pixel-to-pixel distances
pid = ((coords[:, 0] // P) * G + coords[:, 1] // P).astype(int)        # patch index of every pixel

def covariance(kind, ell):
    """Stationary field: squared-exponential ("smooth") or exponential ("rough") covariance, unit variance."""
    k = np.exp(-0.5 * (D / ell) ** 2) if kind == "smooth" else np.exp(-D / ell)
    return k + EPS * np.eye(N * N)

def random_mask(m, rng):
    """m of the 64 patches, uniformly at random without replacement (MAE and SimMIM)."""
    mk = np.zeros(G * G, bool)
    mk[rng.permutation(G * G)[:m]] = True
    return mk

def block_mask(m, rng):
    """One compact block: the m patches whose centres lie nearest a random point of the image."""
    c = rng.uniform(0, G, 2)
    gy, gx = np.divmod(np.arange(G * G), G)
    d = (gy + 0.5 - c[0]) ** 2 + (gx + 0.5 - c[1]) ** 2
    mk = np.zeros(G * G, bool)
    mk[np.argsort(d, kind="stable")[:m]] = True
    return mk

def condition(K, M):
    """Gaussian conditioning on the visible pixels: weights and covariance of the masked pixels' error."""
    V = ~M
    Kmv = K[np.ix_(M, V)]
    cf = cho_factor(K[np.ix_(V, V)])
    return Kmv, cf, K[np.ix_(M, M)] - Kmv @ cho_solve(cf, Kmv.T)

def errors(K, mk):
    """Bayes-optimal masked error / variance for raw pixels, patch means (dc) and mean-removed patches."""
    M = mk[pid]
    _, _, S = condition(K, M)
    K0 = K[np.ix_(M, M)]
    same = pid[M][:, None] == pid[M][None]                 # pixel pairs inside one patch
    dc = lambda A: (A * same).sum() / P ** 2               # energy of the patch means, in pixel units
    raw = np.trace(S) / np.trace(K0)
    return raw, dc(S) / dc(K0), (np.trace(S) - dc(S)) / (np.trace(K0) - dc(K0))

def avg_dist(mk):
    """SimMIM's AvgDist: mean distance (pixels) from each masked pixel to the nearest visible pixel."""
    M = mk[pid].reshape(N, N)
    return distance_transform_edt(M)[M].mean()

KINDS = {"smooth": "squared-exponential", "rough": "exponential"}

# ---- checks on one mask (48 of 64 patches) ----
check_mk = random_mask(48, np.random.default_rng(0))
M = check_mk[pid]; V = ~M
right = np.arange(N * N).reshape(N, N)[:, :-1].ravel(); left = right + 1   # horizontal neighbours
mc_rng = np.random.default_rng(3)
print(f"checks on {N_FIELDS} sampled fields per row; mask: 48 of 64 patches")
print("field    ell | variance z | lag-1 cov z | masked error: closed form  Monte Carlo     z")
dmean = dvar = 0.0
for kind in KINDS:
    for ell in ELLS:
        K = covariance(kind, ell)
        L = np.linalg.cholesky(K)
        X = mc_rng.standard_normal((N_FIELDS, N * N)) @ L.T
        se_var = np.sqrt(2 * (K ** 2).sum() / (N * N) ** 2 / N_FIELDS)
        z_var = ((X ** 2).mean() - K.diagonal().mean()) / se_var
        Krr, Kll, Krl = K[np.ix_(right, right)], K[np.ix_(left, left)], K[np.ix_(right, left)]
        se_lag = np.sqrt((Krr * Kll + Krl * Krl.T).sum() / len(right) ** 2 / N_FIELDS)
        z_lag = ((X[:, right] * X[:, left]).mean() - K[right, left].mean()) / se_lag
        Kmv, cf, S = condition(K, M)
        tr0 = np.trace(K[np.ix_(M, M)])
        pred = X[:, V] @ cho_solve(cf, Kmv.T)                # conditional mean of every field's masked pixels
        r_cf = np.trace(S) / tr0
        r_mc = ((X[:, M] - pred) ** 2).sum(1).mean() / tr0
        z_mc = (r_mc - r_cf) / (np.sqrt(2 * (S ** 2).sum() / N_FIELDS) / tr0)
        print(f"{kind:6s} {ell:5d} | {z_var:10.2f} | {z_lag:11.2f} | {r_cf:24.4f} {r_mc:12.4f} {z_mc:6.2f}")
        # library check: scikit-learn's Gaussian-process regression with the same fixed kernel
        f = L @ np.random.default_rng(1).standard_normal(N * N)
        kern = RBF(ell, "fixed") if kind == "smooth" else Matern(ell, "fixed", nu=0.5)
        mu, sd = GaussianProcessRegressor(kern, alpha=EPS, optimizer=None).fit(coords[V], f[V]).predict(
            coords[M], return_std=True)
        dmean = max(dmean, np.abs(mu - Kmv @ cho_solve(cf, f[V])).max())
        dvar = max(dvar, np.abs(sd ** 2 + EPS - S.diagonal()).max())
print(f"scikit-learn GaussianProcessRegressor vs hand formula: max |mean diff| {dmean:.1e}, max |var diff| {dvar:.1e}")

# ---- the sweep: correlation length x masking ratio x mask type ----
rng = np.random.default_rng(2)
masks = {(t, m): [random_mask(m, rng) if t == "random" else block_mask(m, rng) for _ in range(N_MASKS)]
         for m in COUNTS for t in ("random", "block")}
print("every mask hides exactly its count of patches:",
      all(mk.sum() == m for (t, m), mks in masks.items() for mk in mks))

for kind, name in KINDS.items():
    res = {}
    for ell in ELLS:
        K = covariance(kind, ell)
        for key, mks in masks.items():
            res[key + (ell,)] = np.array([errors(K, mk) for mk in mks])        # (masks, 3)
    print(f"\n{kind} field ({name}): masked error of the optimal predictor / variance, "
          f"mean (min-max) over {N_MASKS} masks")
    print("masked        type   AvgDist " + "".join(f"{'ell = ' + str(e):>21s}" for e in ELLS))
    for m in COUNTS:
        for t in ("random", "block"):
            ad = np.mean([avg_dist(mk) for mk in masks[(t, m)]])
            cells = "".join(f"  {res[(t, m, e)][:, 0].mean():.4f} ({res[(t, m, e)][:, 0].min():.3f}-"
                            f"{res[(t, m, e)][:, 0].max():.3f})" for e in ELLS)
            print(f"{m:2d}/64 ({m / 64:4.0%})  {t:6s} {ad:7.2f}{cells}")
    mean = lambda t, m, e, j=0: res[(t, m, e)][:, j].mean()
    falls = all(mean(t, m, a) > mean(t, m, b) for t in ("random", "block") for m in COUNTS
                for a, b in zip(ELLS, ELLS[1:]))
    rises = all(mean(t, a, e) < mean(t, b, e) for t in ("random", "block") for e in ELLS
                for a, b in zip(COUNTS, COUNTS[1:]))
    not_above = [(m, e) for m in COUNTS for e in ELLS if not mean("block", m, e) > mean("random", m, e)]
    print(f"error falls as ell grows, every type and ratio: {falls}")
    print(f"error rises with the masking ratio, every type and ell: {rises}")
    print(f"block above random at every ratio and ell: {not not_above}"
          + (f"; exceptions (masked, ell): {not_above}" if not_above else ""))
    for t in ("random", "block"):
        first = [next((f"{m / 64:.0%}" for m in COUNTS if mean(t, m, e) >= 0.5), "none") for e in ELLS]
        print(f"smallest ratio with mean error >= 0.5, {t:6s}: " +
              ", ".join(f"ell {e}: {s}" for e, s in zip(ELLS, first)))
    print("random masks, patch mean / raw pixels / mean-removed patch (MAE's target without the division):")
    for m in COUNTS:
        print(f"{m:2d}/64  " + "".join(f"   ell {e}: {mean('random', m, e, 1):.4f}/{mean('random', m, e, 0):.4f}/"
                                      f"{mean('random', m, e, 2):.4f}" for e in ELLS))
    order = [(m, e) for m in COUNTS for e in ELLS
             if not mean("random", m, e, 1) < mean("random", m, e, 0) < mean("random", m, e, 2)]
    print(f"patch mean < raw < mean-removed everywhere: {not order}"
          + (f"; exceptions (masked, ell): {order}" if order else ""))

checks on 2000 sampled fields per row; mask: 48 of 64 patches
field    ell | variance z | lag-1 cov z | masked error: closed form  Monte Carlo     z
smooth     1 |      -0.66 |       -0.54 |                   0.8808       0.8793  -0.80
smooth     2 |       1.79 |        1.76 |                   0.4983       0.5007   1.00
smooth     4 |      -0.49 |       -0.53 |                   0.1742       0.1754   0.52
smooth     8 |      -0.47 |       -0.49 |                   0.0326       0.0328   0.19
rough      1 |      -1.96 |       -0.90 |                   0.9571       0.9543  -1.88
rough      2 |      -1.28 |       -1.50 |                   0.8442       0.8436  -0.31
rough      4 |      -0.03 |       -0.07 |                   0.6525       0.6546   0.76
rough      8 |       1.88 |        1.90 |                   0.4347       0.4350   0.10
scikit-learn GaussianProcessRegressor vs hand formula: max |mean diff| 3.4e-12, max |var diff| 1.4e-13
every mask hides exactly its count of patches: True

smooth field (squared-exponential): masked error of the optimal predictor / variance, mean (min-max) over 10 masks
masked        type   AvgDist               ell = 1              ell = 2              ell = 4              ell = 8
10/64 ( 16%)  random    1.50  0.6552 (0.586-0.732)  0.0672 (0.014-0.154)  0.0016 (0.000-0.005)  0.0002 (0.000-0.000)
10/64 ( 16%)  block     3.52  0.8792 (0.823-0.910)  0.5225 (0.349-0.626)  0.1455 (0.018-0.261)  0.0082 (0.000-0.026)
26/64 ( 41%)  random    1.78  0.7370 (0.713-0.760)  0.1446 (0.070-0.182)  0.0080 (0.001-0.025)  0.0005 (0.000-0.001)
26/64 ( 41%)  block     5.82  0.9303 (0.895-0.945)  0.7062 (0.572-0.767)  0.3771 (0.149-0.499)  0.0882 (0.002-0.168)
38/64 ( 59%)  random    2.13  0.8052 (0.778-0.835)  0.2490 (0.192-0.345)  0.0196 (0.006-0.052)  0.0011 (0.000-0.003)
38/64 ( 59%)  block     8.48  0.9553 (0.948-0.965)  0.8101 (0.770-0.848)  0.5750 (0.487-0.639)  0.2473 (0.156-0.273)
48/64 ( 75%)  random    3.25  0.8788 (0.858-0.901)  0.4679 (0.376-0.592)  0.1172 (0.033-0.317)  0.0179 (0.001-0.068)
48/64 ( 75%)  block     9.97  0.9632 (0.950-0.970)  0.8421 (0.785-0.868)  0.6463 (0.556-0.700)  0.3467 (0.231-0.412)
58/64 ( 91%)  random    6.17  0.9543 (0.942-0.974)  0.7712 (0.696-0.878)  0.4386 (0.277-0.681)  0.1179 (0.027-0.299)
58/64 ( 91%)  block    13.56  0.9833 (0.976-0.986)  0.9221 (0.886-0.937)  0.7863 (0.653-0.837)  0.5228 (0.159-0.598)
error falls as ell grows, every type and ratio: True
error rises with the masking ratio, every type and ell: True
block above random at every ratio and ell: True
smallest ratio with mean error >= 0.5, random: ell 1: 16%, ell 2: 91%, ell 4: none, ell 8: none
smallest ratio with mean error >= 0.5, block : ell 1: 16%, ell 2: 16%, ell 4: 59%, ell 8: 91%
random masks, patch mean / raw pixels / mean-removed patch (MAE's target without the division):
10/64     ell 1: 0.7380/0.6552/0.6257   ell 2: 0.0584/0.0672/0.0802   ell 4: 0.0009/0.0016/0.0058   ell 8: 0.0001/0.0002/0.0032
26/64     ell 1: 0.8025/0.7370/0.7137   ell 2: 0.1316/0.1446/0.1639   ell 4: 0.0060/0.0080/0.0204   ell 8: 0.0003/0.0005/0.0051
38/64     ell 1: 0.8557/0.8052/0.7872   ell 2: 0.2310/0.2490/0.2756   ell 4: 0.0160/0.0196/0.0416   ell 8: 0.0008/0.0011/0.0080
48/64     ell 1: 0.9113/0.8788/0.8672   ell 2: 0.4489/0.4679/0.4962   ell 4: 0.1111/0.1172/0.1554   ell 8: 0.0170/0.0179/0.0397
58/64     ell 1: 0.9669/0.9543/0.9497   ell 2: 0.7589/0.7712/0.7894   ell 4: 0.4303/0.4386/0.4906   ell 8: 0.1150/0.1179/0.1929
patch mean < raw < mean-removed everywhere: False; exceptions (masked, ell): [(10, 1), (26, 1), (38, 1), (48, 1), (58, 1)]

rough field (exponential): masked error of the optimal predictor / variance, mean (min-max) over 10 masks
masked        type   AvgDist               ell = 1              ell = 2              ell = 4              ell = 8
10/64 ( 16%)  random    1.50  0.8771 (0.852-0.904)  0.6358 (0.583-0.700)  0.3771 (0.327-0.442)  0.2006 (0.169-0.243)
10/64 ( 16%)  block     3.52  0.9566 (0.936-0.967)  0.8489 (0.782-0.884)  0.6590 (0.539-0.727)  0.4295 (0.306-0.512)
26/64 ( 41%)  random    1.78  0.9052 (0.897-0.914)  0.6970 (0.671-0.719)  0.4380 (0.403-0.466)  0.2411 (0.215-0.264)
26/64 ( 41%)  block     5.82  0.9748 (0.963-0.980)  0.9094 (0.868-0.928)  0.7790 (0.687-0.824)  0.5770 (0.442-0.653)
38/64 ( 59%)  random    2.13  0.9297 (0.920-0.941)  0.7571 (0.731-0.793)  0.5061 (0.472-0.563)  0.2891 (0.262-0.340)
38/64 ( 59%)  block     8.48  0.9838 (0.981-0.987)  0.9412 (0.931-0.954)  0.8534 (0.825-0.881)  0.6999 (0.647-0.744)
48/64 ( 75%)  random    3.25  0.9563 (0.949-0.964)  0.8382 (0.811-0.870)  0.6301 (0.580-0.712)  0.4015 (0.349-0.518)
48/64 ( 75%)  block     9.97  0.9867 (0.982-0.989)  0.9514 (0.934-0.960)  0.8783 (0.839-0.900)  0.7474 (0.680-0.787)
58/64 ( 91%)  random    6.17  0.9837 (0.979-0.991)  0.9348 (0.915-0.965)  0.8230 (0.769-0.905)  0.6335 (0.544-0.776)
58/64 ( 91%)  block    13.56  0.9940 (0.992-0.995)  0.9772 (0.967-0.982)  0.9372 (0.904-0.951)  0.8482 (0.748-0.882)
error falls as ell grows, every type and ratio: True
error rises with the masking ratio, every type and ell: True
block above random at every ratio and ell: True
smallest ratio with mean error >= 0.5, random: ell 1: 16%, ell 2: 16%, ell 4: 59%, ell 8: 91%
smallest ratio with mean error >= 0.5, block : ell 1: 16%, ell 2: 16%, ell 4: 16%, ell 8: 41%
random masks, patch mean / raw pixels / mean-removed patch (MAE's target without the division):
10/64     ell 1: 0.7754/0.8771/0.9050   ell 2: 0.4399/0.6358/0.7739   ell 4: 0.2021/0.3771/0.6672   ell 8: 0.0900/0.2006/0.6019
26/64     ell 1: 0.8256/0.9052/0.9271   ell 2: 0.5238/0.6970/0.8190   ell 4: 0.2661/0.4380/0.7232   ell 8: 0.1257/0.2411/0.6599
38/64     ell 1: 0.8698/0.9297/0.9462   ell 2: 0.6101/0.7571/0.8608   ell 4: 0.3416/0.5061/0.7789   ell 8: 0.1703/0.2891/0.7198
48/64     ell 1: 0.9186/0.9563/0.9667   ell 2: 0.7345/0.8382/0.9113   ell 4: 0.4950/0.6301/0.8542   ell 8: 0.2896/0.4015/0.8073
58/64     ell 1: 0.9694/0.9837/0.9876   ell 2: 0.8906/0.9348/0.9660   ell 4: 0.7509/0.8230/0.9425   ell 8: 0.5545/0.6335/0.9200
patch mean < raw < mean-removed everywhere: True

The checks pass. Every $z$-score of the sampled variance, neighbour covariance and Monte Carlo error lies within $\pm 2$, inside the planned limit of 4. The hand formula agrees with scikit-learn to about $10^{-12}$, and every mask hides exactly its planned count.

Redundancy lowers the error everywhere.

• For both fields, both mask types and every ratio, the mean error falls strictly as $\ell$ grows, and rises strictly with the masking ratio. The script prints True for all four of these tests.
• The smooth field shows how far redundancy can go. With 75% of patches hidden at random, the optimal predictor leaves 88% of the variance unexplained at $\ell = 1$ and 1.8% at $\ell = 8$.
• At $\ell = 8$, hiding 16% of the patches leaves an error of 0.0002. The hidden pixels are then recovered almost exactly from their neighbours, the situation MAE describes when it notes that a missing patch can be recovered from neighbouring patches.

The ratio that makes the task hard moves up with redundancy. The script finds the smallest printed ratio at which the mean error reaches 0.5. With random masks:

• the smooth field needs 16%, 91%, and then no printed ratio at $\ell$ = 1, 2, 4 and 8;
• the rough field needs 16%, 16%, 59% and 91%.

With block masks the thresholds are 16%, 16%, 59% and 91% for the smooth field and 16%, 16%, 16% and 41% for the rough one. In all four series the threshold never moves to a lower ratio as $\ell$ grows. This is the closed-form version of the argument that redundant signals need high masking ratios. The 0.5 level is arbitrary, and the example says nothing about which error level yields the best features; SimMIM’s ridge suggests that this depends on more than difficulty.

Block masks are harder at equal ratio.

• Block masks have the larger AvgDist at every ratio (3.52 against 1.50 pixels at 10 of 64, up to 13.56 against 6.17 at 58 of 64).
• Their mean error is higher than random masks’ at every ratio, $\ell$ and field.
• The printed ranges over the ten masks overlap in four cells, all at 58 of 64 hidden patches: smooth and rough fields at $\ell = 4$ and $\ell = 8$. In those cells some random masks score worse than some block masks.
• In a fifth cell, smooth field at $\ell = 8$ with 10 of 64 hidden, the ranges meet at the printed 0.000 (random 0.000–0.000, block 0.000–0.026), so three decimals cannot show whether they overlap.

This matches SimMIM’s Figure 3(a), where block-wise and square masks have higher AvgDist than random masks of small patches, and AvgDist grows with the ratio.

The choice of field matters as much as $\ell$.

• The same correlation length gives very different redundancy under the two covariances. At $\ell = 8$ with random masks, the rough field’s error is already 0.20 at 16% hidden, while the smooth field’s stays at 0.12 at 91% hidden.
• In our reading, the smooth field puts almost all its variance at low spatial frequencies, which a few visible pixels pin down, while in the rough field pixel-scale detail stays unpredictable at any ratio.
• So no single ratio is right for all images. The useful ratio depends on how an image’s correlations fall off with distance, which neither field captures for natural photographs.

Per-patch normalization removes the most predictable part. The last block of each table splits the random-mask error into each hidden patch’s mean and the patch with its mean removed. The mean-removed patch is MAE’s normalized target before division by the patch’s standard deviation; the division is nonlinear and has no closed form here.

• Rough field. The patch mean is always the easiest component and the mean-removed patch the hardest. At $\ell = 8$ with 48 of 64 patches hidden, the errors are 0.2896 for the mean, 0.4015 for raw pixels and 0.8073 for the mean-removed patch.
• Smooth field, $\ell$ = 2, 4 and 8. The same order holds; the four-decimal print shows it even at $\ell = 8$, for example 0.0001, 0.0002 and 0.0032 at 10 of 64.
• Smooth field, $\ell = 1$. The order reverses at every ratio, as the plan written before the first run predicted (“holds everywhere except smooth ℓ=1 (ℓ a quarter of the patch), where the order reverses”): 0.7380, 0.6552 and 0.6257 at 10 of 64, through 0.9669, 0.9543 and 0.9497 at 58 of 64. The script’s verdict therefore prints False, listing these five cells.
• Why $\ell = 1$ differs (our reading). There the correlation reaches only a pixel or two. The predictable part of a hidden patch is its edge pixels, which neighbour visible pixels. That signal lies in the pixel-to-pixel variation rather than in the patch mean, which averages mostly unpredictable interior pixels.

When neighbouring patches are correlated over several pixels, normalizing each patch strips out the low-frequency content that neighbours give away. In our reading, this is one way to see MAE’s report that per-patch normalization helps: the remaining target is less redundant.

Variants

• Pixel regression with a light head. SimMIM uses random 32-pixel masks, a linear head and an $\ell_1$ loss, and runs with ViT and Swin Transformer encoders. SwinV2-G was pre-trained this way before supervised training.
• Asymmetric encoder–decoder. MAE encodes visible patches only, uses a high ratio and a normalized pixel target, and has a small Transformer decoder.
• Tokenizer targets. BEiT predicts DALL-E visual tokens. iBOT replaces the fixed tokenizer with an online one.
• Feature targets. MaskFeat predicts HOG, and data2vec predicts a moving-average teacher’s averaged top-layer features. V-JEPA, as video foundation models describes it, predicts the representation of hidden parts of a clip in a learned latent space.
• Video. MaskFeat masks space-time “cubes”: in its video ablation, cube masking at 40% gave 82.2% on Kinetics-400, against 81.9% for tube and 81.0% for frame masking (Table 6). VideoMAE and MAE-ST use tube or random space-time masks at 90% and higher, as the video foundation models article compares.
• Convolutional encoders. ConvNeXt V2’s fully convolutional masked autoencoder treats the visible pixels as a sparse array (see ConvNeXt).
• Early forms. Context Encoders trained an encoder–decoder to fill a missing region, as image inpainting describes. The original ViT paper’s pilot predicted the mean colour of masked patches (see ViT paper). These and other pretext tasks predate MAE and SimMIM.

Best Practices

• Mask enough to defeat copying. Raise the ratio for small mask units, or enlarge the units. SimMIM’s results with small patches and MAE’s 75% both point this way.
• Compute the loss on hidden patches only. Adding visible-patch reconstruction cost SimMIM 1.1 points.
• Start with pixel targets, normalized per patch, or HOG. They need no second network. Tokenizer and teacher-feature targets add a pre-training stage, and in MaskFeat’s and MAE’s comparisons they did not beat the better one-stage targets by a clear margin, except unsupervised teacher features, which cost a separately trained model.
• Keep the head light when the goal is fine-tuning. Use a deeper decoder if frozen features matter, as MAE’s linear-probing results suggest.
• Use little augmentation. MaskFeat and MAE report that random cropping and flipping suffice. SimMIM uses a light crop range of 0.67–1.
• Evaluate by fine-tuning and by a frozen protocol before choosing a method for a frozen use. The two rankings can disagree (see representation learning).

Pitfalls

• Reading reconstructions as quality. SimMIM’s heavier heads and full-image loss produced lower loss or better-looking recoveries but worse fine-tuning. Reconstruction error is not a proxy for representation quality.
• Class-derived targets. MaskFeat saw significant over-fitting with supervised-feature and pseudo-label targets; the DeiT-feature target came within 0.1 point of training from scratch, and the pseudo-label target fell below it.
• Comparing across protocols. Fine-tuning schedules, epochs, image sizes and probes differ between papers. data2vec’s comparison table sets other methods’ numbers beside its own, with data2vec trained for 800 (ViT-B) and 1,600 (ViT-L) epochs. SimMIM’s Table 6 reports BEiT’s linear probe at 56.7%, the same value as its own.
• Leaking through the encoder. A convolution slides across masked regions and leaks information from them, as the ConvNeXt article describes for ConvNeXt V2. In our reading, mask units much smaller than an early layer’s receptive field fail in a related way, like a mask too small to defeat copying.
• Collapse with self-predicted targets. Targets produced by the network itself need a safeguard against a constant solution: data2vec normalizes its targets, and iBOT keeps DINO’s centering and sharpening, as Self-Supervised Learning describes.

Evidence and Impact

data2vec’s ImageNet-1K table (Table 1) lists top-1 validation accuracies after fine-tuning:

Method	Target	ViT-B	ViT-L
BEiT	DALL-E tokens	83.2	85.2
MAE	normalized pixels	83.6	85.9
SimMIM	raw pixels	83.8	–
iBOT	online tokens	83.8	–
MaskFeat	HOG	84.0	85.7
data2vec	teacher features	84.2	86.6

For comparison, the same table lists MoCo v3 at 83.2 and 84.1 and DINO at 82.8 for ViT-B. The MIM entries differ by about one point and come from different recipes and schedules, so the table does not isolate the effect of the target.

Larger results:

• SwinV2-H. SimMIM pre-training on ImageNet-1K alone gave 87.1% after fine-tuning at $512^2$, against 85.7% at $224^2$. Supervised SwinV2-H reached 83.3% at $224^2$ (Table 7).
• SwinV2-G. It was pre-trained with SimMIM before supervised training (see Swin Transformer).
• Video. MaskFeat pre-trained MViTv2-L on Kinetics-400 without labels or external data, and reached 86.7% top-1 on Kinetics-400 at $352^2$ with 40 frames (Table 3).

Masked prediction now appears well beyond these papers:

• Backbones. MAE-trained encoders initialize detection and segmentation backbones (see masked autoencoder).
• Hybrids. iBOT and DINOv2 combine masked prediction with self-distillation.
• Video. Two of the four objectives that recur among video foundation models are reconstructing masked pixels and predicting a frozen teacher’s masked features.