Pretext Tasks

A pretext task is a prediction problem built automatically from unlabeled images, so that a network trained to solve it learns features that are useful for something else. The label costs nothing because the task’s designer creates it: cut two patches from a photograph and ask where the second one sat relative to the first; shuffle nine tiles and ask for the permutation; erase a region and ask for its pixels; drop the colors and ask for them back; rotate the image and ask by how much. Nobody needs the answers. What is kept is the encoder, which is then fine-tuned or probed on labelled tasks such as object detection. Doersch, Gupta and Efros (2015), whose context prediction Kolesnikov et al. (2019) credit with sparking interest in self-supervised visual representation learning, put the word in quotation marks and borrowed it from word embeddings, where predicting a word’s context serves as a “pretext” for learning the embedding.

Pretext tasks were the first wave of self-supervised learning for convolutional networks; that article gives the overview and summarizes the five classic tasks with their headline numbers. This article covers how a pretext task is constructed, what can go wrong with one, and what the studies that compared them found; it runs a context-prediction task with no learning at all to show how much a low-level cue can solve.

Definition

A pretext task is a supervised learning problem whose inputs and targets are both computed from the same unlabeled image by a fixed, designer-chosen procedure, and which is solved only as a means of training a representation for other, downstream tasks. Kolesnikov, Zhai and Beyer (2019) state the design requirement directly: the task must be set up so that high-level image understanding is useful for solving it.

Two things separate a pretext task from an ordinary supervised task. Its labels carry no human judgement, so their only value lies in what solving them forces the network to compute. And its success is measured elsewhere: the pretext head is thrown away, and the representation is judged by a downstream protocol such as a linear probe or fine-tuning (evaluating representations describes the protocols).

Intuition

Doersch et al. argued that doing well on relative position requires recognizing objects and their parts; their first figure asks the reader to place two patches and remarks that the task is much easier once the object is recognized. A network pushed to guess well should therefore learn objects. That is the hope behind every pretext task: a question chosen so that the cheapest way to answer it runs through semantics.

The risk is the opposite. A network has no reason to take the intended route if a cheaper one exists. Lines that run straight across the boundary between two patches, a texture that continues, a brightness gradient, the color fringes a particular lens leaves near the image corners: each can answer the question without any notion of objects. A pretext task is only as good as the shortcuts its designers managed to close, and closing them is most of the craft. The example below measures one such shortcut directly.

Formal Definition

Let $x$ be an unlabeled image and $\omega$ a random variable drawn by the designer (a patch position, a permutation, a mask, an angle). A pretext task is a pair of functions that produce an input and a target,

$$\tilde{x} = T(x, \omega), \qquad y = L(x, \omega),$$

and training minimizes the expected loss of an encoder $f_\theta$ followed by a task head $h_\phi$:

$$\min_{\theta, \phi} \; \mathbb{E}_{x, \omega} \Big[ \ell\big(h_\phi(f_\theta(\tilde{x})),\, y\big) \Big].$$

Two kinds are common. In discriminative tasks $y$ is one of a few classes and $\ell$ is the cross-entropy loss: the relative position of two patches ($y \in \{1, \dots, 8\}$), a permutation index, a rotation index. In generative tasks $y$ is part of the image itself, the pixels under a mask or the color channels, and $\ell$ is a reconstruction loss. After training, $h_\phi$ is discarded and the downstream task uses $f_\theta$, frozen or fine-tuned.

For context prediction, the first patch $a$ is taken at a uniformly sampled position $u$, and the second at $u + (s + g)\,\delta_y + \epsilon$, where $\delta_y$ is one of the eight unit offsets to the neighbouring cells of a $3 \times 3$ grid, $s$ the patch size, $g$ a gap and $\epsilon$ a random jitter. In Doersch et al.’s implementation, $s = 96$ pixels, $g = 48$ and each patch is jittered by up to 7 pixels in each direction (§3.1).

Properties

The task must deny shortcuts, and they are hard to see in advance. Doersch et al. added the gap because “boundary patterns or textures continuing between patches” could solve the task, and the jitter because long lines spanning neighbouring patches could still give away the answer even with the gap (§3.1). Neither was enough. Some images were solved through chromatic aberration: in some cameras one color channel, commonly green, is shrunk toward the image centre relative to the others, so a network could locate a patch on the lens from the separation between green and magenta and read off relative positions from absolute ones. A network trained to regress a patch’s absolute $(x, y)$ coordinates reached a root-mean-square error of 0.255 on its best 10% of images, against 0.371 for always predicting the centre (§4.2). Two remedies were tried: projecting every pixel off the green–magenta color axis, which raised that error to 0.321, and randomly dropping two of the three color channels from each patch (§3.1, §4.2). Pathak et al. (2016) met the same problem in a generative task. A context encoder trained to fill a fixed central square learned low-level features that “latch on” to the mask’s boundary and generalized poorly to images without masks, so the authors also randomized the mask, with random blocks and with randomly shaped regions, covering up to a quarter of the image; the authors found that random blocks and random regions gave similarly general features that significantly outperformed the central region’s (§3.3). Image inpainting covers the context encoder’s architecture and loss.

The negatives must share the shortcut cues. Doersch et al. rejected an apparently natural task, telling real images from images with one patch replaced by a patch from another image, because low-level color statistics and lighting would solve it. Both patches in their task come from the same image, so they share lighting and color statistics, and those cues cannot separate the eight answers (§2).

High pretext accuracy is neither required nor a reliable signal. Doersch et al.’s network classified the relative position correctly 38.4% of the time on 500 PASCAL VOC 2007 images (256 pairs each; chance 12.5%), and 40.3% on ImageNet validation, against 39.5% on its own training set; they report human performance as similar (§4.5). Restricting pairs to large, unoccluded object boxes gave 39.2%, and to cars 45.6%, from which they concluded that the network is almost as sensitive to the layout of the rest of the image as to objects. Yet the features transferred. Across many networks, Kolesnikov et al. found pretext accuracy a good proxy for downstream accuracy only once the architecture is fixed: the widest VGG model scored best on the rotation task and poorly downstream (§4.4, Figure 4). The example below shows the other side of the coin, a rule with no features at all that scores far above chance.

The last layers specialize to the pretext. Kolesnikov et al. found that with a batch-normalized VGG19 the quality of the representation falls toward the end of the network, as earlier studies they cite had observed for AlexNet; they attribute this to the later layers specializing to the pretext task and discarding more general features. In residual networks it did not fall, and the pre-logits layer was always best, which they relate to residual units being invertible under some conditions (§4.5, Figure 5). Self-Supervised Learning gives Gidaris et al.’s numbers for this effect in a rotation network, as that article reports them.

The task and the architecture interact. In Kolesnikov et al.’s Table 1 (ImageNet linear logistic regression on their own validation split), the same relative-patch-location task gave 50.5% with a ResNet50 v1 at widening factor 8 but 29.4% with VGG19-BN at the same factor; neither the ranking of tasks nor the ranking of architectures was consistent across the other. Selecting an architecture per task and widening the networks let their models significantly outperform previously reported results: relative patch location rose from 36.2% to 51.4% and rotation from 38.7% to 55.4% top-1 on the official ImageNet validation set, against 72.7–75.8% for the supervised networks in the same table (Table 2). The authors concluded that pretext tasks should not be judged in isolation from the network that solves them. Representation learning adds their warning about slow convergence of the linear probe, and VTAB’s evidence that linear and fine-tuned rankings disagree.

Optimization can fail in a task-specific way. Doersch et al. found that plain SGD left their network stuck predicting a uniform distribution over the eight answers, with fc6 and fc7 activations collapsed to zero; adding batch normalization let it escape (§3.1). Their final network trained for about four weeks on one GPU.

Kinds of Pretext Task

The classic tasks fall into three groups. Self-Supervised Learning gives their sizes and headline results; the points below add what the readable studies say about how each was set up.

• Spatial context. Context prediction (relative patch location) classifies which of eight neighbours a second patch came from (Doersch et al., 2015). Its network was a pair of AlexNet-style stacks with weights tied up to fc6, after which the two are fused and only two layers receive input from both patches, so that most of the reasoning happens on each patch separately and each patch gets its own embedding (§3). Jigsaw puzzles (Noroozi and Favaro, 2016), as Kolesnikov et al. describe the task, shuffle nine patches and ask for the permutation: all nine patches pass through the same network, their representations are concatenated and an MLP predicts which of a fixed set of permutations was applied; in their reimplementation the set had 100 permutations, patches were separated by a random gap, and each was turned to grayscale with probability 2/3 and standardized, to block chromatic aberration and edge alignment (§3.2 and supplement).
• Reconstruction. Context encoders fill a masked region of the image from its surroundings (Pathak et al., 2016). Colorization (Zhang, Isola and Efros, 2016) predicts color from lightness, as Self-Supervised Learning describes it, and Kolesnikov et al. list it, with its split-brain variant, among the tasks with dense spatial outputs. The autoencoder is the older relative: Pathak et al. argue that an autoencoder’s bottleneck may only compress the image, and that a denoising autoencoder’s corruption is local and low-level, whereas a large hole cannot be filled from nearby pixels. Masked image modeling, as in the masked autoencoder, later revived reconstruction as a pretext for transformers.
• Transformations. Rotation prediction (Gidaris, Singh and Komodakis, 2018), as Kolesnikov et al. describe it, classifies which of four rotations, 0°, 90°, 180° or 270°, was applied to the whole image, which Kolesnikov et al. explain as requiring the network to recognize the canonical orientation of objects. Exemplar training, also in their study, treats each image as its own class and generates its examples by heavy augmentation, trained with a triplet loss so that it scales to all of ImageNet. Instance discrimination in contrastive learning also treats each image as its own class, with many negatives in place of the triplet.

On PASCAL VOC 2007, both readable papers that introduced a task reported transfer gains over random initialization that stopped short of ImageNet-supervised pretraining. Doersch et al.’s R-CNN, initialized by context prediction, reached 46.3% mAP with color dropping and 45.7% with projection, against 39.8% for the same architecture trained from scratch and 54.2% for ImageNet-supervised R-CNN (Table 1, VOC 2007 test, no bounding-box regression). In Pathak et al.’s comparison, which used Fast R-CNN, context encoders reached 44.5% detection mAP against 46.6% for Doersch et al.’s features, 43.4% for random initialization and 56.8% for ImageNet; on PASCAL VOC 2012 validation segmentation they reached 29.7%, against 19.8% random and 25.2% for an autoencoder (Table 2). The two tables used different detectors, so their numbers do not compare across papers.

Examples

Solving context prediction without learning

Doersch et al.’s gap and jitter target one shortcut: brightness and texture continuing from one patch into its neighbour. The script below measures how far that cue goes by itself. It uses no network and no training. A fixed rule compares the facing borders of the two patches under each of the candidate directions and answers with the direction across which the image looks most continuous.

Set-up. Five photographs bundled with scikit-image 0.25.2 (astronaut, coffee, chelsea, camera, rocket) are converted to grayscale. Patches are 32 pixels. The gap takes the values 0, 4, 8 and 16 pixels; 16 is half the patch width, Doersch et al.’s ratio. Jitter, where used, moves each patch by up to 2 pixels in each direction, Doersch et al.’s 7 of 96 scaled to 32 pixels. For each gap and seed, 400 pairs are drawn from each image, so 2,000 pairs per gap and seed, and the table reports the mean and the range over seeds 0–4.

The rule. Each patch is reduced to a $4 \times 4$ grid of $8 \times 8$ block means. For an edge neighbour (above, below, left, right) the score is the mean squared difference between the four facing block means of the first patch and the four facing block means of the second; for a diagonal neighbour it is the squared difference between the two facing corner blocks. The predicted direction minimizes the score. The rule assumes the nominal layout, so it knows neither the gap nor the jitter.

What is printed. The first line is a base check: the rule applied to random patches built so that their facing blocks continue each other exactly, where it must find the true direction every time. The columns are the 8-way task without jitter, the 8-way task with jitter, the 8-way task with jitter after standardizing each patch to zero mean and unit variance (part of the preprocessing Kolesnikov et al. used for relative patch location), the 4-way task with only the edge neighbours and jitter, and the second column split by whether the true neighbour was an edge or a diagonal one. Chance is 0.125 for eight answers and 0.25 for four. The script uses NumPy 2.3.5 and scikit-image 0.25.2 and runs in about four seconds.

import os
os.environ["OPENBLAS_NUM_THREADS"] = "1"   # one BLAS thread, set before NumPy loads
os.environ["OMP_NUM_THREADS"] = "1"
import numpy as np
from skimage import data, color

P, M, J = 32, 8, 2                  # patch size, block size of the rule, jitter (pixels)
GAPS, SEEDS, N = (0, 4, 8, 16), range(5), 400      # N pairs per image per gap and seed
OFFS = [(-1, -1), (-1, 0), (-1, 1), (0, -1), (0, 1), (1, -1), (1, 0), (1, 1)]
EDGE, DIAG = [1, 3, 4, 6], [0, 2, 5, 7]

ims = [color.rgb2gray(data.astronaut()), color.rgb2gray(data.coffee()),
       color.rgb2gray(data.chelsea()), data.camera() / 255.0, color.rgb2gray(data.rocket())]

def blocks(p):                      # P x P patch -> (P/M) x (P/M) grid of block means
    return p.reshape(P // M, M, P // M, M).mean((1, 3))

def score(A, B, d):
    """Discontinuity if B lies in direction d from A (block grids A, B; lower = smoother).
    Edge neighbours: mean squared difference of the P/M facing block pairs.
    Diagonal neighbours: squared difference of the two facing corner blocks."""
    dy, dx = d
    if dy == 0:
        ea, eb = (A[:, 0], B[:, -1]) if dx < 0 else (A[:, -1], B[:, 0])
    elif dx == 0:
        ea, eb = (A[0], B[-1]) if dy < 0 else (A[-1], B[0])
    else:
        ea = A[0 if dy < 0 else -1, 0 if dx < 0 else -1]
        eb = B[-1 if dy < 0 else 0, -1 if dx < 0 else 0]
    return np.mean((ea - eb) ** 2)

def predict(a, b, cand, std=False):
    if std:                         # per-patch standardization (zero mean, unit variance)
        a, b = (a - a.mean()) / (a.std() + 1e-8), (b - b.mean()) / (b.std() + 1e-8)
    A, B = blocks(a), blocks(b)
    return cand[int(np.argmin([score(A, B, OFFS[k]) for k in cand]))]

def base_check(rng, n=400):
    """B built so that its facing blocks continue A's exactly: the rule must find the direction."""
    ok = 0
    for t in range(n):
        k = t % 8; dy, dx = OFFS[k]
        a, b = rng.random((P, P)), rng.random((P, P))
        A, Bb = blocks(a), blocks(b)
        if dy == 0:   Bb[:, -1 if dx < 0 else 0] = A[:, 0 if dx < 0 else -1]
        elif dx == 0: Bb[-1 if dy < 0 else 0] = A[0 if dy < 0 else -1]
        else:         Bb[-1 if dy < 0 else 0, -1 if dx < 0 else 0] = A[0 if dy < 0 else -1, 0 if dx < 0 else -1]
        b = b + np.kron(Bb - blocks(b), np.ones((M, M)))
        ok += predict(a, b, list(range(8))) == k
    return ok / n

def crop(im, y, x):
    assert 0 <= y and 0 <= x and y + P <= im.shape[0] and x + P <= im.shape[1]
    return im[y:y + P, x:x + P]

def run(seed):
    rng = np.random.default_rng(seed)
    out = {}
    for gap in GAPS:
        s = P + gap
        hits = {k: [] for k in ("8", "8j", "8js", "4j")}; truth8 = []; labs = []
        for im in ims:
            H, W = im.shape
            r = rng.integers(s + J, H - P - s - J + 1, N)
            c = rng.integers(s + J, W - P - s - J + 1, N)
            lab8, lab4 = rng.integers(0, 8, N), rng.choice(EDGE, N)
            jit = rng.integers(-J, J + 1, (N, 2, 4))        # A and B jitter, for 8-way and 4-way
            for i in range(N):
                a0 = crop(im, r[i], c[i])
                dy, dx = OFFS[lab8[i]]
                b0 = crop(im, r[i] + dy * s, c[i] + dx * s)
                hits["8"].append(predict(a0, b0, list(range(8))) == lab8[i])
                ja, jb = jit[i, 0, :2], jit[i, 0, 2:]
                a = crop(im, r[i] + ja[0], c[i] + ja[1])
                b = crop(im, r[i] + dy * s + jb[0], c[i] + dx * s + jb[1])
                hits["8j"].append(predict(a, b, list(range(8))) == lab8[i])
                hits["8js"].append(predict(a, b, list(range(8)), std=True) == lab8[i])
                dy, dx = OFFS[lab4[i]]
                ja, jb = jit[i, 1, :2], jit[i, 1, 2:]
                a = crop(im, r[i] + ja[0], c[i] + ja[1])
                b = crop(im, r[i] + dy * s + jb[0], c[i] + dx * s + jb[1])
                hits["4j"].append(predict(a, b, EDGE) == lab4[i])
            truth8.append(lab8); labs.append(lab4)
        t8 = np.concatenate(truth8); h8j = np.array(hits["8j"])
        out[gap] = {k: np.mean(v) for k, v in hits.items()}
        out[gap]["8j_edge"] = h8j[np.isin(t8, EDGE)].mean()
        out[gap]["8j_diag"] = h8j[np.isin(t8, DIAG)].mean()
        out[gap]["counts"] = (np.bincount(t8, minlength=8).min(), np.bincount(t8, minlength=8).max(),
                              np.bincount(np.concatenate(labs), minlength=8)[EDGE].min())
    return out

print(f"base check (constructed continuations, 8-way): {base_check(np.random.default_rng(99)):.4f}")
res = [run(seed) for seed in SEEDS]
print(f"{len(ims)} images x {N} pairs per gap and seed; seeds {SEEDS.start}-{SEEDS.stop - 1}; "
      f"8-way label counts per seed in [{min(r[g]['counts'][0] for r in res for g in GAPS)}, "
      f"{max(r[g]['counts'][1] for r in res for g in GAPS)}]")
cols = [("8", "8-way"), ("8j", "8-way jit"), ("8js", "8-way jit std"), ("4j", "4-way jit"),
        ("8j_edge", "jit, edge truth"), ("8j_diag", "jit, diag truth")]
print("gap  " + "  ".join(f"{name:>22s}" for _, name in cols) + "   (mean [min, max] over seeds)")
for gap in GAPS:
    cells = []
    for key, _ in cols:
        v = np.array([r[gap][key] for r in res])
        cells.append(f"{v.mean():.4f} [{v.min():.4f},{v.max():.4f}]")
    print(f"{gap:3d}  " + "  ".join(f"{c:>22s}" for c in cells))
print("chance: 0.1250 (8-way), 0.2500 (4-way)")

base check (constructed continuations, 8-way): 1.0000
5 images x 400 pairs per gap and seed; seeds 0-4; 8-way label counts per seed in [216, 287]
gap                   8-way               8-way jit           8-way jit std               4-way jit         jit, edge truth         jit, diag truth   (mean [min, max] over seeds)
  0  0.3899 [0.3640,0.4100]  0.3883 [0.3730,0.4025]  0.1799 [0.1685,0.1890]  0.7113 [0.7005,0.7210]  0.2858 [0.2692,0.3034]  0.4905 [0.4764,0.5019]
  4  0.3270 [0.3160,0.3365]  0.3250 [0.3180,0.3340]  0.1663 [0.1605,0.1735]  0.6243 [0.6025,0.6510]  0.2088 [0.1955,0.2303]  0.4438 [0.4315,0.4562]
  8  0.2786 [0.2735,0.2850]  0.2806 [0.2680,0.2950]  0.1621 [0.1575,0.1695]  0.5679 [0.5540,0.5805]  0.1556 [0.1308,0.1752]  0.4081 [0.3925,0.4172]
 16  0.2354 [0.2190,0.2445]  0.2359 [0.2250,0.2450]  0.1511 [0.1400,0.1655]  0.4934 [0.4810,0.5085]  0.1075 [0.1002,0.1167]  0.3619 [0.3479,0.3794]
chance: 0.1250 (8-way), 0.2500 (4-way)

The base check is exact, so the scoring code assigns each direction to the right pair of borders. On the photographs, a rule that knows nothing about objects answers the 8-way question correctly about 39% of the time with abutting patches, in the range Doersch et al. report for their trained network on other images (38.4–40.3%), although the two numbers come from different images, patch sizes and gaps (Doersch et al. used a half-patch gap and jitter) and are not a comparison. The gap is the main defence, and it works only partly: accuracy falls at every step, yet with a gap of half the patch width the rule is still right in about 24% of 8-way pairs and 49% of 4-way pairs, roughly double chance in both cases. In our reading, brightness varies slowly enough in photographs that the facing borders of true neighbours stay closer than those of other pairs even across a gap. Jitter of 2 pixels barely changes the results; in our reading because 8-pixel block means average over more than the displacement, so the long lines Doersch et al. worried about would need a finer comparison than this rule makes. Standardizing each patch, which discards its mean brightness and contrast, lowers accuracy at every gap but leaves the 8-way mean above chance at every gap.

The split by true direction shows that the 8-way numbers are not a clean measure. At a gap of 16, pairs whose true neighbour is diagonal are answered correctly 36% of the time, while pairs whose true neighbour is above, below, left or right are answered correctly less often than chance, in every seed. In our reading, the rule’s eight scores are not on a common scale: a diagonal score rests on one pair of corner blocks and an edge score on the mean of four, so the noisier diagonal scores more often produce the smallest value, and the rule leans toward diagonal answers. We therefore take the 4-way column, where every candidate uses four block pairs, as the cleaner measure of the continuity cue; this is our judgement, not something the code tests.

Outcomes of the planned hypotheses. The plan, fixed before any run, made seven predictions; the thresholds for “above chance” are chance plus three binomial standard errors for 2,000 pairs (0.1472 for eight answers, 0.2790 for four).

• With no gap and no jitter, the plan predicted an 8-way mean of at least 0.40. This check failed: the printed mean is 0.3899 (range 0.3640–0.4100).
• That 8-way accuracy with jitter falls strictly as the gap grows held: 0.3883, 0.3250, 0.2806, 0.2359.
• That it stays above chance at a gap of 16 held: the lowest seed is 0.2250, above 0.1472.
• That standardization lowers the 8-way accuracy at every gap held (0.1799 against 0.3883, 0.1663 against 0.3250, 0.1621 against 0.2806, 0.1511 against 0.2359). That every seed stays above chance at a gap of 16 after standardization failed: the lowest seed is 0.1400, below 0.1472, although the mean, 0.1511, is above it.
• The plan predicted that pairs with a diagonal true neighbour would be answered correctly less often than pairs with an edge neighbour, at every gap. This check failed at every gap: 0.4905 against 0.2858, 0.4438 against 0.2088, 0.4081 against 0.1556 and 0.3619 against 0.1075. The plan also compared the lowest diagonal-truth seed at a gap of 16 with chance plus three standard errors for about 1,000 pairs (0.1564), to be reported either way: it is 0.3479, above it.
• That the 4-way task stays above chance at a gap of 16 held: the lowest seed is 0.4810, above 0.2790.
• That jitter lowers 8-way accuracy without a gap held, by 0.0016 (0.3883 against 0.3899). At the other gaps, where nothing was predicted, the jittered values are 0.3250 against 0.3270 at a gap of 4, 0.2806 against 0.2786 at 8 and 0.2359 against 0.2354 at 16.

In our reading, a pretext network sees the same cue and is rewarded for using it. Nothing in the loss tells it to prefer object parts over continuing brightness, which is why Doersch et al. needed the gap, the jitter and color projection or dropping on top of each other, and why the remaining low-level signal can still help a network to its pretext accuracy.

Common Misconceptions

• “A high pretext accuracy means good features.” The rule above has no features and still scores well above chance; Kolesnikov et al. found pretext accuracy predictive of downstream accuracy only within one architecture.
• “A low pretext accuracy means the task failed.” Doersch et al.’s network was right in fewer than half of its pairs, at about human level by their account, and its features still improved detection over training from scratch.
• “The gap removes the shortcut.” It weakens it. In the example a half-patch gap left the 4-way accuracy near 0.49 against 0.25 chance.
• “Shortcuts are only about pixels at the border.” Chromatic aberration solved context prediction through the absolute position on the lens, and the context encoder’s fixed central mask let features attach to the mask’s own edge.
• “The best pretext task can be chosen once.” Kolesnikov et al. found the ranking of tasks changed with the network architecture and width.

Where It Is Used

Later self-supervised methods moved from hand-designed labels to contrastive learning and masked image modeling, a sequence Self-Supervised Learning traces, but the idea of a task built from the data runs through both. Masked prediction, as in the masked autoencoder and VideoMAE, is a reconstruction pretext; video foundation models face the same shortcut problem in time, since a hidden patch can often be copied from a neighbouring frame. Domain-specific pretraining builds the data’s own structure into the task, as SatMAE does with spectral and temporal masking in remote sensing, and CroCo, which DUSt3R uses for initialization, reconstructs a masked view from a second view of the same scene. The lesson of this article carries over to all of them: before trusting a pretext, ask which low-level cue could answer it, and measure how far that cue alone goes. Transfer learning and fine-tuning cover what is done with the encoder afterwards, and invariance and equivariance covers the related question of what a representation keeps or discards under transformations.