I-JEPA

The Image-based Joint-Embedding Predictive Architecture (I-JEPA) is a self-supervised pre-training method for Vision Transformers, introduced by Assran, Duval, Misra, Bojanowski, Vincent, Rabbat, LeCun and Ballas of Meta AI (FAIR) at CVPR 2023. From a single visible block of an image, a predictor network estimates the representations of several hidden blocks of the same image, as computed by a second, slowly updated encoder. The loss is computed between representations, not pixels, and no second augmented view of the image is needed. The authors describe the method as non-generative and present it as an image instance of a joint-embedding predictive architecture (JEPA), a term for which they cite LeCun’s 2022 position paper.

I-JEPA belongs to the family of masked image modeling in the broad sense: part of an image is hidden and a network predicts what was hidden. What sets it apart is the target. Masked Image Modeling covers the choice of target across the family (pixels, normalized pixels, tokens, HOG, teacher features), and the masked autoencoder (MAE) covers the pixel-reconstruction method that I-JEPA compares itself with most. This article covers the image method, its ablations and results, and a closed-form example of what predicting in a learned representation space selects; the video successor V-JEPA is summarized under Variants.

Problem Addressed

The paper sorts self-supervised image methods into two families (§1–2):

• Invariance-based methods (joint-embedding architectures, such as contrastive learning and DINO) train an encoder to give similar embeddings to two or more views of an image made with hand-crafted augmentations: random scaling, cropping and colour jitter. The authors grant that these give representations of a high semantic level, but argue that the augmentations encode biases that may hurt tasks or data distributions that need different invariances, and that they do not carry over easily to other modalities such as audio.
• Generative methods remove or corrupt part of the input and reconstruct it, at the pixel or token level, as MAE and BEiT do. They need less prior knowledge, but in the authors’ account their representations are of a lower semantic level: they trail invariance-based methods under off-the-shelf evaluation such as linear probing and in low-label transfer, and need end-to-end fine-tuning to show their strength.

I-JEPA keeps the single view of masked modeling and drops the augmentations, but moves the prediction into representation space, where, in the authors’ words, unnecessary pixel-level details are “potentially eliminated” from the targets (§1). The goal stated in the paper is semantic representations that work without extensive fine-tuning.

The paper frames all three families with energy-based models (§2, Figure 2). A joint-embedding architecture compares the embeddings $s_x$ and $s_y$ of two inputs. A generative architecture decodes $y$ from $x$ and a conditioning variable $z$, such as the positions to reconstruct. A joint-embedding predictive architecture predicts the embedding $s_y$ from $s_x$ and $z$, so it looks like a generative architecture whose loss is applied in embedding space. Both embedding families face representational collapse, an encoder whose output does not depend on its input; I-JEPA, like several joint-embedding methods, uses an asymmetric design between the two encoders to avoid it (§2). For generative architectures, the paper notes, collapse is not a concern as long as the information capacity of $z$ is low compared with $y$.

Architecture

Three networks, all Vision Transformers, act on the patch tokens of one image (§3; supplement A.1):

• a context encoder $f_\theta$, which sees only the visible context patches;
• a target encoder $f_{\bar\theta}$, which sees the whole image and produces the targets;
• a predictor $g_\phi$, a narrow ViT that maps the context representation and the positions of a target block to predicted representations for that block.

The encoders are standard ViTs (ViT-B/16, ViT-L/16, ViT-H/14, ViT-H/16 at $448^2$ and ViT-G/16 in the paper). I-JEPA uses no [CLS] token. The paper notes that the encoder–predictor layout resembles MAE’s encoder–decoder; the difference is that the predictions are made in representation space.

Target Encoder and Targets

The image $y$ is cut into $N$ non-overlapping patches and passed through the target encoder, giving patch-level representations $s_y = \{s_{y_1}, \dots, s_{y_N}\}$. The method samples $M$ target blocks, possibly overlapping, with masks $B_1, \dots, B_M$; the target for block $i$ is $s_y(i) = \{s_{y_j}\}_{j \in B_i}$ (§3). By default $M = 4$, each block with a random scale in $(0.15, 0.2)$ of the image and a random aspect ratio in $(0.75, 1.5)$.

The targets are taken by masking the output of the target encoder, not its input: the target encoder sees the full image once, and each target patch’s representation is contextualized by the rest of the image. The supplement compares this with masking the input, which needs one forward pass per target block: with a ViT-H/16 pre-trained for 300 epochs, linear evaluation on 1% of ImageNet-1K labels gave 67.3% for output masking and 56.1% for input masking (Table 11).

The target encoder’s weights are not trained by the loss. They start equal to the context encoder’s and are updated after each step as an exponential moving average (EMA) of them, $\bar\theta \leftarrow m\bar\theta + (1-m)\theta$, with momentum $m$ rising linearly from 0.996 to 1.0 over pre-training (supplement A.1). The authors note that an EMA target encoder has proven essential for training joint-embedding architectures with ViTs, and report the same for I-JEPA (§3).

Context Encoder

The context is a single block with a random scale in $(0.85, 1.0)$ and unit aspect ratio, from which every region that overlaps a target block is removed, so that the prediction task is not trivial (§3). The context block is therefore large but has holes where the targets are; Figure 4 of the paper shows examples. The context encoder processes only the remaining visible patches, giving $s_x = \{s_{x_j}\}_{j \in B_x}$. Context and target masks are sampled independently for every image (supplement A.1).

Predictor

For each target block, the predictor takes the context representation $s_x$ and one mask token per target patch, $\{m_j\}_{j \in B_i}$, and outputs $\hat s_y(i) = g_\phi(s_x, \{m_j\}_{j \in B_i})$. The mask tokens are a shared learnable vector plus a positional embedding, so the position tells the predictor which block to predict; the predictor is run $M$ times, once per target block (§3).

The predictor is narrow: its embedding dimension is fixed at 384, with as many attention heads as the encoder, and its depth is 6 for ViT-B/16, 12 for ViT-L/16, ViT-H/16 and ViT-H/14, and 16 for ViT-G/16 (supplement A.1). Two supplementary ablations support the shape:

• Depth. A ViT-L/16 pre-trained for 500 epochs scored 64.0% with a 6-layer predictor and 66.9% with a 12-layer one, under linear evaluation on 1% of ImageNet-1K labels (Table 12).
• Width. With a ViT-L encoder of width 1,024, a 384-wide predictor gave 70.7% and a 1,024-wide one 68.4% on ImageNet-1K 1% with fine-tuning (Table 14).

Training

Objective. The loss is the squared $L_2$ distance between predicted and target patch representations, averaged over the $M$ target blocks (§3):

$$\mathcal{L} = \frac{1}{M}\sum_{i=1}^{M} D\big(\hat s_y(i), s_y(i)\big) = \frac{1}{M}\sum_{i=1}^{M}\sum_{j \in B_i} \big\lVert \hat s_{y_j} - s_{y_j} \big\rVert_2^2 .$$

The gradient updates $\theta$ and $\phi$ only; $\bar\theta$ follows by EMA. The loss contains no negatives, no variance or covariance term and no pixel reconstruction. In the paper’s account, collapse is avoided by an asymmetric architecture between the two encoders (§2), and the EMA target encoder proved essential (§3).

Data and schedule. Pre-training uses ImageNet-1K at $224 \times 224$ unless stated otherwise (§5), and for Table 5 also ImageNet-22K. The supplement (A.1) gives AdamW with batch size 2,048; a learning rate raised linearly from $10^{-4}$ to $10^{-3}$ over the first 15 epochs and then decayed to $10^{-6}$ on a cosine schedule; and weight decay raised linearly from 0.04 to 0.4. The authors chose the rising weight decay because it improved linear evaluation; a fixed 0.05 gave better fine-tuning on ImageNet-1K 1% but worse linear evaluation (Table 13). The ViT-H/14 model trained for 300 epochs on ImageNet-1K.

Masking strategy. The authors call the masking strategy a core design choice (abstract). With a ViT-B/16 pre-trained for 300 epochs and a linear probe on 1% of ImageNet-1K labels (Table 6):

Masking	Targets	Context	Context share	Top-1
multi-block	4 blocks, scale 0.15–0.2	block, scale 0.85–1.0, minus targets	0.25	54.2
rasterized	3 quadrants	1 quadrant	0.25	15.5
block	1 block, scale 0.6	complement	0.4	20.2
random	random patches, 0.6	complement	0.4	17.6

The supplementary ablations of the multi-block strategy, in the same set-up, show that it is narrow. Smaller targets ($0.075$–$0.2$) gave 19.2% and larger ones ($0.2$–$0.3$) 33.6% (Table 8); a smaller context ($0.40$–$1.0$) gave 31.2% (Table 9); and one, two or three target blocks gave 9.0%, 22.0% and 48.5% against 54.2% for four (Table 10). The authors conclude that the targets should be large enough to be semantic and the context informative and spatially distributed.

Representation space against pixels. Predicting pixels with the same framework gave 40.7% on ImageNet-1K 1% with a linear probe, against 66.9% for predicting the target encoder’s output, both with a ViT-L/16 (Table 7). The two rows were pre-trained for different lengths, 800 epochs for pixels and 500 for representations. The authors present it as their conjecture that the gain comes from the target encoder producing abstract targets from which irrelevant pixel-level details are eliminated (§9).

Paper and code. The rendered pages show several internal inconsistencies. Table 13 gives the default ViT-L/16 (600 epochs) 77.8% in ImageNet linear evaluation, where Table 1 gives the same configuration 77.5%. Table 14’s 384-wide row (70.7%) equals Table 13’s fixed-weight-decay row rather than its default row (69.4%), and the text gives the width as 386 where the table gives 384. Table 5’s caption speaks of a ViT-H/16 where its row is a ViT-G/16, Table 9’s caption says it ablates the target size where it ablates the context, the loss is called an “average $L_2$ distance” while the equation squares it, and one citation of DINO in §5 points to the reference for SwAV. The official code at commit 52c1ae9 differs from the paper in three places: src/train.py applies a layer normalization over the feature dimension to the target encoder’s outputs and uses PyTorch’s smooth_l1_loss rather than a squared $L_2$ loss; the ViT-H/14 config starts the learning rate at $2 \times 10^{-4}$ with 40 warm-up epochs rather than $10^{-4}$ and 15; and every training image goes through a RandomResizedCrop with scale $(0.3, 1.0)$, with colour jitter, blur and flips switched off. That crop is a single-view augmentation, not the second view that invariance-based methods compare, so it is consistent with the paper’s claim of no view augmentations, but it is an augmentation all the same.

Inference

After pre-training, the predictor is discarded. The target encoder is used for evaluation, and its patch outputs are average-pooled into one image representation, since there is no [CLS] token (supplement A.1). For linear evaluation in VISSL the authors report the better of the last layer’s pooled output and the concatenation of the last four layers’ pooled outputs, each with a linear head with or without batch normalization in front (A.2). The encoder can also be fine-tuned like any ViT. Pre-training cost is reported below; the paper gives no inference-speed figures.

The predictor is used once more, for visualization. The authors freeze the context encoder and predictor, train a diffusion decoder (the RCDM framework) to map the average-pooled predictor output back to pixels, and sample it with different seeds (§8, Figure 6). What stays the same across samples is read as what the prediction contains: object parts in the correct pose, such as the back of a bird or the top of a car; precise low-level details and background vary, which the authors read as information the predictor discards.

Performance

All numbers are top-1 accuracy as printed in the paper, with models pre-trained on ImageNet-1K unless stated. The paper groups baselines into methods without view augmentations (data2vec, MAE, CAE) and methods with them (SimCLR v2, DINO, iBOT, MSN, BYOL).

ImageNet-1K linear evaluation (Table 1; frozen weights, linear classifier on the full training set):

Method	Architecture	Epochs	Top-1
MAE	ViT-H/14	1,600	77.2
data2vec	ViT-L/16	1,600	77.3
CAE	ViT-L/16	1,600	78.1
I-JEPA	ViT-L/16	600	77.5
I-JEPA	ViT-H/14	300	79.3
I-JEPA	ViT-H/16 at $448^2$	300	81.1
DINO	ViT-B/8	300	80.1
iBOT	ViT-L/16	250	81.0

ImageNet-1K with 1% of the labels (Table 2; each method adapted by fine-tuning or linear probing, whichever worked best for it): I-JEPA reached 73.3% with ViT-H/14 and 77.3% with ViT-H/16 at $448^2$, against 71.5% for MAE ViT-H/14, 73.3% for data2vec ViT-L/16 and 75.7% for MSN ViT-B/4, the best of the augmentation-based methods listed. I-JEPA’s low-shot recipe fine-tunes for 50 epochs with RandAugment, without mixup, CutMix, random erasing or drop path (supplement A.2).

Linear transfer (Tables 3 and 4; ViT-H/14 unless stated):

Method	CIFAR-100	Places205	iNat18	Clevr/Count	Clevr/Dist
MAE ViT-H/14	77.3	55.0	32.9	90.5	72.4
I-JEPA ViT-H/14	87.5	58.4	47.6	86.7	72.4
DINO ViT-B/8	84.9	57.9	55.9	86.6	53.4
iBOT ViT-L/16	88.3	60.4	57.3	85.7	62.8

On the semantic tasks (CIFAR-100, Places205, iNaturalist 2018) I-JEPA beats MAE and data2vec and passes DINO on CIFAR-100 and Places205, but stays below iBOT on all three and below DINO on iNaturalist. On the CLEVR tasks, object counting and depth prediction, the authors report that I-JEPA outperforms the view-invariance methods, by a large margin on Clevr/Dist (72.4% against 53.4% and 62.8%); on Clevr/Count the margin over DINO is 0.1 point, and MAE is ahead of all of them at 90.5%.

Full fine-tuning (supplement Table 15): ViT-H/16 at $448^2$ reached 87.1% on ImageNet-1K, against 87.8% for MAE’s ViT-H/14 at $448^2$, which was pre-trained for 1,600 epochs against I-JEPA’s 300.

Compute. The abstract reports a ViT-H/14 trained on ImageNet with 16 A100 GPUs in under 72 hours. In the introduction, ViT-H/14 pre-training needs under 1,200 GPU hours, which the authors put at over 2.5 times faster than a ViT-S/16 trained with iBOT and over 10 times more efficient than a ViT-H/14 trained with MAE. Against MAE the authors explain the trade (§7): computing targets in representation space makes each iteration about 7% slower, but I-JEPA converges in roughly five times fewer iterations.

Scale (Table 5). Pre-training ViT-H/14 on ImageNet-22K instead of ImageNet-1K raised CIFAR-100 from 87.5% to 89.5%, iNaturalist from 47.6% to 50.5% and Clevr/Dist from 72.4% to 75.0%, and lowered Places205 from 58.4% to 57.8%. A ViT-G/16 on ImageNet-22K improved Places205 and iNaturalist further (59.1%, 55.3%) but not the low-level CLEVR tasks; the authors suggest its larger patches can hurt local prediction tasks.

How Latent Prediction Selects Features: a Linear Example

The paper’s argument is that a target encoder can learn targets that leave out unpredictable pixel detail. It also needs a safeguard, because a target encoder trained on the same loss could make its targets trivially predictable. Both points can be worked out exactly in a linear-Gaussian model of images. The model, the derivations below and the reading of them are ours, not the paper’s; the example illustrates the derivations and does not model I-JEPA’s EMA or its networks.

The model. An image $u \in \mathbb{R}^{256}$ ($16 \times 16$ pixels) is split into a target $y$, the bottom-right $8 \times 8$ block, and a context $x$, the other 192 pixels. Each image is

$$u = \sum_{k=1}^{3} z_k\, p_k + \sigma \epsilon, \qquad z \sim \mathcal{N}(0, I_3),\ \ \epsilon \sim \mathcal{N}(0, I_{256}),\ \ \sigma = 1,$$

so $x = A z + \sigma\epsilon_x$ and $y = B z + \sigma\epsilon_y$, where $A$ and $B$ hold the three patterns $p_k$ restricted to the context and to the target, and the white pixel detail $\sigma\epsilon$ is independent across pixels and therefore unpredictable. The three patterns, with their amplitude in the context and in the target, are:

• stripes, vertical, period 4 pixels, amplitude 0.5 everywhere: visible across the image, little energy in the target;
• pattern, horizontal bands of period 8, amplitude 0.15 in the context and 2.0 in the target: strong in the target, faint in the context;
• brightness, a constant of amplitude 2.0 in the target and 0 in the context: invisible from the context.

Each $8 \times 8$ block holds whole periods of both cosines, so the three patterns are orthogonal within the context and within the target, and every quantity below has a per-factor closed form. Writing $a_k^2 = \lVert A_{:,k} \rVert^2$ and $b_k^2 = \lVert B_{:,k} \rVert^2$, the amplitudes give $a^2 = (24, 2.16, 0)$ and $b^2 = (8, 128, 256)$. The amplitudes were a design choice, made from these closed forms before any pilot ran, so that the three criteria below rank the factors differently; the example therefore illustrates the derivations rather than testing whether such a ranking occurs.

D1: the ceiling of pixel prediction. The best predictor of the target pixels in mean squared error is $\mathbb{E}[y \mid x] = \Sigma_{yx}\Sigma_{xx}^{-1}x$, linear because everything is Gaussian. It explains the covariance $G = \Sigma_{yx}\Sigma_{xx}^{-1}\Sigma_{xy}$, and since $B\Sigma_z B^\top - G$ is $B$ times the posterior covariance of $z$ times $B^\top$, which is positive semi-definite,

$$R^2_\ast = \frac{\operatorname{tr} G}{\operatorname{tr}\Sigma_{yy}} \le \frac{\operatorname{tr}(B B^\top)}{\operatorname{tr}\Sigma_{yy}} < 1 .$$

With orthogonal patterns, $R^2_\ast = \sum_k b_k^2\, a_k^2/(a_k^2+\sigma^2)\,/\operatorname{tr}\Sigma_{yy}$. A pixel loss at its optimum is left with everything the context cannot see: here the brightness and the pixel detail. Masked Image Modeling’s example computes this kind of Bayes masked error over masking ratios and correlation lengths of a random field; here it serves only as the reference for the latent comparison.

D2: a learnable target encoder collapses, exactly. Take one target feature $w^\top y$ and a linear prediction $m^\top x$ of it (context encoder and predictor combined), with the loss $L(m, w) = \mathbb{E}(m^\top x - w^\top y)^2$. Since $m^\top x - w^\top y = (m^\top A - w^\top B)z + \sigma m^\top \epsilon_x - \sigma w^\top \epsilon_y$ is a sum of three independent terms, $L \ge \sigma^2(\lVert m\rVert^2 + \lVert w\rVert^2)$, so $L = 0$ only at $m = 0$, $w = 0$: if the target encoder is trained on the same loss without any safeguard, collapse is the unique global minimum. Alternating the two exact least-squares fits, $m$ given $w$ and then $w$ given $m$, gives

$$w^\top \leftarrow w^\top C, \qquad C = \Sigma_{yx}\Sigma_{xx}^{-1}\Sigma_{xy}\Sigma_{yy}^{-1},$$

and $C$ is similar to $KK^\top$ with $K = \Sigma_{yy}^{-1/2}\Sigma_{yx}\Sigma_{xx}^{-1/2}$, whose eigenvalues are the squared canonical correlations $\rho_k^2 < 1$ between context and target. So $\lVert w \rVert$ shrinks by $\rho_1^2$ per step in the limit, and the loss, which is quadratic in $w$, by $\rho_1^4$. A frozen target, $w$ fixed at its initial value as a stop-gradient would hold it, keeps a positive loss, but its size scales with $\lVert w\rVert^2$ and says nothing on its own.

D3: a variance constraint selects the most predictable directions. Requiring the target feature to have unit variance, $w^\top\Sigma_{yy}w = 1$, and fitting $m$ exactly gives $L = 1 - w^\top G w$. With $v = \Sigma_{yy}^{1/2}w$ this is $1 - v^\top KK^\top v$ over unit vectors, minimized by the top canonical direction of canonical correlation analysis (CCA), at $L = 1 - \rho_1^2$; for $r$ features the minimum is $\sum_{k \le r}(1-\rho_k^2)$. The alternating fits with rescaling become power iteration on $C$, converging at $(\rho_2/\rho_1)^2$ per step. Per factor, $\rho_k^2 = \frac{a_k^2}{a_k^2+\sigma^2}\cdot\frac{b_k^2}{b_k^2+\sigma^2}$, so the three criteria rank factors by different quantities:

• constrained latent prediction (CCA): by predictability alone, $\rho_k^2$: stripes 0.853, pattern 0.678, brightness 0;
• rank-1 pixel prediction (reduced-rank regression, the top eigenvector of $G$): by explained target energy $b_k^2 a_k^2/(a_k^2+\sigma^2)$: stripes 7.68, pattern 87.49, brightness 0;
• largest target variance (principal component analysis of $y$): by $b_k^2 + \sigma^2$: 9, 129, 257.

The script checks D1–D3 against closed forms, against fits on 20,000 sampled training images scored on 20,000 test images, and against scikit-learn’s LinearRegression and CCA. It uses NumPy 2.3.5 and scikit-learn 1.7.2 on one BLAS thread, takes about 25 seconds, and two runs gave identical output.

import os
os.environ["OMP_NUM_THREADS"] = os.environ["OPENBLAS_NUM_THREADS"] = "1"   # before NumPy loads
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.cross_decomposition import CCA

# 16x16 images; the target is the bottom-right 8x8 block, the context the other three blocks.
H, SD, N, STEPS = 16, 1.0, 20000, 200          # size, pixel-detail std, images per split, iterations
r, c = np.mgrid[:H, :H].astype(float)
tgt = ((r >= 8) & (c >= 8)).ravel()
ctx = ~tgt
NAMES = ("stripes", "pattern", "brightness")
shape = {"stripes": np.cos(2 * np.pi * c / 4),          # vertical stripes, period 4, whole image
         "pattern": np.cos(2 * np.pi * r / 8),          # horizontal bands, period 8
         "brightness": np.ones((H, H))}                 # uniform brightness
amp = {"stripes": (0.5, 0.5), "pattern": (0.15, 2.0), "brightness": (0.0, 2.0)}   # (in context, in target)
PAT = np.stack([(np.where(tgt.reshape(H, H), amp[n][1], amp[n][0]) * shape[n]).ravel() for n in NAMES])
A, B = PAT[:, ctx].T, PAT[:, tgt].T                     # 192x3 and 64x3 loadings of the three factors
# image = sum_i z_i * pattern_i + SD * white noise, with z ~ N(0, I): exact covariances
Sxx = A @ A.T + SD**2 * np.eye(ctx.sum())
Syy = B @ B.T + SD**2 * np.eye(tgt.sum())
Sxy = A @ B.T
G = Sxy.T @ np.linalg.solve(Sxx, Sxy)                  # covariance of the best prediction E[y|x]
Syx_res = Syy - G                                      # covariance of what x cannot predict

def canonical(Sxx, Syy, Sxy):
    """Squared canonical correlations and target-side directions (unit variance under Syy)."""
    Ly, Lx = np.linalg.cholesky(Syy), np.linalg.cholesky(Sxx)
    U, s, _ = np.linalg.svd(np.linalg.solve(Ly, Sxy.T) @ np.linalg.inv(Lx).T)
    return s**2, np.linalg.solve(Ly.T, U)

def describe(u):
    """Feature u.y: squared correlation with each factor, share of pixel detail, R^2 from the context."""
    v = u @ Syy @ u
    return [(u @ B[:, i])**2 / v for i in range(3)], SD**2 * (u @ u) / v, (u @ G @ u) / v

a2, b2 = (A**2).sum(0), (B**2).sum(0)
rho2, U = canonical(Sxx, Syy, Sxy)
rho2_f = a2 / (a2 + SD**2) * b2 / (b2 + SD**2)          # derived per factor (patterns are orthogonal)
print("factor loadings ||a||^2 (context), ||b||^2 (target):",
      ", ".join(f"{n} {x:.2f}/{y:.2f}" for n, x, y in zip(NAMES, a2, b2)))
print(f"patterns orthogonal in context and target: max off-diagonal "
      f"{max(np.abs(A.T @ A - np.diag(a2)).max(), np.abs(B.T @ B - np.diag(b2)).max()):.1e}")
print(f"squared canonical correlations: {np.round(rho2[:4], 6)}; per-factor formula "
      f"{np.round(rho2_f, 6)}; max diff {np.abs(np.sort(rho2_f)[::-1] - rho2[:3]).max():.1e}")

# ---- sampled images and a check of their covariance ----
rng = np.random.default_rng(0)
def images(n):
    return rng.standard_normal((n, 3)) @ PAT + SD * rng.standard_normal((n, H * H))
Xtr, Xte = images(N), images(N)
Sig = PAT.T @ PAT + SD**2 * np.eye(H * H)
rel = np.linalg.norm(Xtr.T @ Xtr / N - Sig) / np.linalg.norm(Sig)
rms = np.sqrt((np.trace(Sig)**2 + np.trace(Sig @ Sig)) / N) / np.linalg.norm(Sig)
print(f"\nsample covariance of {N} training images: relative error {rel:.4f}, "
      f"expected rms {rms:.4f}, ratio {rel / rms:.3f}")
x, y, xt, yt = Xtr[:, ctx], Xtr[:, tgt], Xte[:, ctx], Xte[:, tgt]

# ---- 1. pixel prediction: the ceiling ----
R2_star = np.trace(G) / np.trace(Syy)
R2_f = (b2 * a2 / (a2 + SD**2)).sum() / np.trace(Syy)
shared = (b2.sum()) / np.trace(Syy)
W = np.linalg.lstsq(x, y, rcond=None)[0]                       # least-squares pixel predictor
dW = np.abs(LinearRegression(fit_intercept=False).fit(x, y).coef_.T - W).max()
R2_test = 1 - ((yt - xt @ W)**2).sum() / (yt**2).sum()
res = np.trace(Syx_res)
print("\n1. pixel prediction of the target block")
print(f"   best possible R^2 (closed form) {R2_star:.4f}  per-factor formula {R2_f:.4f}  "
      f"variance from the factors {shared:.4f}")
print(f"   least squares on {N} images, R^2 on {N} test images {R2_test:.4f}  "
      f"(sklearn LinearRegression coefficient diff {dW:.1e})")
print(f"   unexplained at the optimum: {res / np.trace(Syy):.4f} of the variance; of it, "
      f"brightness {b2[2] / res:.4f}, pixel detail {tgt.sum() * SD**2 / res:.4f}, "
      f"rest {1 - (b2[2] + tgt.sum() * SD**2) / res:.4f}")

# ---- 2. latent prediction with a learnable target encoder: collapse ----
# loss(m, w) = E (m.x - w.y)^2. Alternate the two exact least-squares fits: m given w, then w given m.
C = Sxy.T @ np.linalg.solve(Sxx, Sxy) @ np.linalg.inv(Syy)   # one round of both fits: w <- w C
w0 = np.random.default_rng(1).standard_normal(tgt.sum())
print("\n2. learnable target encoder, alternating exact fits (population | sampled images)")
print("   step  ||w|| exact     loss exact  | ||w|| sampled")
w, ws, hist = w0.copy(), w0.copy(), []
for k in range(STEPS + 1):
    if k in (0, 1, 2, 5, 10, 20, 50, 100, 200):
        print(f"   {k:4d}  {np.linalg.norm(w):11.4e}  {w @ Syx_res @ w:11.4e}  | {np.linalg.norm(ws):11.4e}")
    hist.append((np.linalg.norm(w), np.linalg.norm(ws)))
    if k < STEPS:
        w = w @ C
        m = np.linalg.lstsq(x, y @ ws, rcond=None)[0]           # predictor fit (context side)
        ws = np.linalg.lstsq(y, x @ m, rcond=None)[0]           # target-encoder fit
rho2_s, Us = canonical(x.T @ x / N, y.T @ y / N, x.T @ y / N)
print(f"   shrink factor per step at step {STEPS}: exact {hist[-1][0] / hist[-2][0]:.10f} "
      f"(top squared canonical correlation {rho2[0]:.10f}); sampled {hist[-1][1] / hist[-2][1]:.10f} "
      f"(sample value {rho2_s[0]:.10f})")
loss = lambda m, w: m @ Sxx @ m - 2 * m @ Sxy @ w + w @ Syy @ w     # exact E (m.x - w.y)^2
print(f"   loss at m = 0, w = 0: {loss(np.zeros(ctx.sum()), np.zeros(tgt.sum())):.1f}; frozen random target w0, best predictor: "
      f"exact {w0 @ Syx_res @ w0:.4f}, sampled {((x @ np.linalg.lstsq(x, y @ w0, rcond=None)[0] - y @ w0)**2).mean():.4f}")

# ---- 3. latent prediction under a variance constraint: which directions ----
print("\n3. unit-variance target feature, alternating fits with rescaling (exact)")
u_cca = U[:, 0] / np.sqrt(U[:, 0] @ Syy @ U[:, 0])
w = w0 / np.sqrt(w0 @ Syy @ w0)
print("   step  1-|cos| to top canonical direction   loss      1 - rho_1^2")
for k in range(STEPS + 1):
    if k in (0, 1, 5, 20, 50, 100, 200):
        print(f"   {k:4d}  {1 - abs(w @ Syy @ u_cca):.3e}  {w @ Syx_res @ w:31.12f}  {1 - rho2[0]:.12f}")
    if k < STEPS:
        w = w @ C
        w = w / np.sqrt(w @ Syy @ w)
u_rrr = np.linalg.eigh(G)[1][:, -1]                     # rank-1 pixel prediction (reduced-rank regression)
u_pca = np.linalg.eigh(Syy)[1][:, -1]                   # largest-variance target direction
u_smp = Us[:, 0]                                        # canonical direction from the sampled images
cca = CCA(n_components=1, scale=False, max_iter=2000, tol=1e-10).fit(x, y)
agree = abs(np.corrcoef(cca.transform(x, y)[1][:, 0], y @ u_smp)[0, 1])
print("   target feature                     corr^2 with: stripes   pattern   brightness | detail  | R^2 from context")
for name, u in (("latent, variance constraint (CCA)", u_cca), ("rank-1 pixel prediction", u_rrr),
                ("largest target variance (PCA)", u_pca), ("CCA on the sampled images", u_smp)):
    cz, d, p = describe(u)
    print(f"   {name:34s} {cz[0]:16.6f} {cz[1]:9.6f} {cz[2]:10.6f}   | {d:.4f}  | {p:.4f}")
print(f"   sample vs exact canonical direction: 1-|cos| {1 - abs(u_smp @ Syy @ u_cca) / np.sqrt(u_smp @ Syy @ u_smp):.2e}; "
      f"sklearn CCA vs hand: |corr| of target scores {agree:.10f}")

# ---- verdicts (criteria and hypotheses of the plan) ----
cz = {n: describe(u)[0] for n, u in (("cca", u_cca), ("rrr", u_rrr), ("pca", u_pca), ("smp", u_smp))}
picks = {n: NAMES[int(np.argmax(v))] for n, v in cz.items()}
print("\nverdicts")
print(f"   choice of factor: CCA {picks['cca']}, rank-1 pixel {picks['rrr']}, PCA {picks['pca']}, sampled CCA {picks['smp']}")
print(f"   predictability R^2: CCA {describe(u_cca)[2]:.4f} > pixel {describe(u_rrr)[2]:.4f} > PCA {describe(u_pca)[2]:.4f}: "
      f"{describe(u_cca)[2] > describe(u_rrr)[2] > describe(u_pca)[2]}")

factor loadings ||a||^2 (context), ||b||^2 (target): stripes 24.00/8.00, pattern 2.16/128.00, brightness 0.00/256.00
patterns orthogonal in context and target: max off-diagonal 4.5e-14
squared canonical correlations: [0.853333 0.678246 0.       0.      ]; per-factor formula [0.853333 0.678246 0.      ]; max diff 1.0e-15

sample covariance of 20000 training images: relative error 0.0207, expected rms 0.0179, ratio 1.159

1. pixel prediction of the target block
   best possible R^2 (closed form) 0.2087  per-factor formula 0.2087  variance from the factors 0.8596
   least squares on 20000 images, R^2 on 20000 test images 0.2062  (sklearn LinearRegression coefficient diff 0.0e+00)
   unexplained at the optimum: 0.7913 of the variance; of it, brightness 0.7095, pixel detail 0.1774, rest 0.1131

2. learnable target encoder, alternating exact fits (population | sampled images)
   step  ||w|| exact     loss exact  | ||w|| sampled
      0   6.8650e+00   1.5734e+02  |  6.8650e+00
      1   6.7754e-01   1.0627e+01  |  7.3633e-01
      2   5.1712e-01   4.9629e+00  |  5.5241e-01
      5   2.6481e-01   5.4132e-01  |  2.7832e-01
     10   1.1091e-01   2.5481e-02  |  1.1665e-01
     20   2.2495e-02   6.7190e-04  |  2.4248e-02
     50   1.9302e-04   4.9181e-08  |  2.2456e-04
    100   6.9430e-08   6.3631e-15  |  9.1702e-08
    200   8.9830e-15   1.0652e-28  |  1.5292e-14
   shrink factor per step at step 200: exact 0.8533333333 (top squared canonical correlation 0.8533333333); sampled 0.8555019738 (sample value 0.8555019738)
   loss at m = 0, w = 0: 0.0; frozen random target w0, best predictor: exact 157.3358, sampled 156.4933

3. unit-variance target feature, alternating fits with rescaling (exact)
   step  1-|cos| to top canonical direction   loss      1 - rho_1^2
      0  8.881e-01                   0.760163633290  0.146666666667
      1  7.646e-01                   0.312051607638  0.146666666667
      5  4.812e-01                   0.274622234199  0.146666666667
     20  1.380e-03                   0.147149551991  0.146666666667
     50  1.435e-09                   0.146666667169  0.146666666667
    100  2.220e-16                   0.146666666667  0.146666666667
    200  0.000e+00                   0.146666666667  0.146666666667
   target feature                     corr^2 with: stripes   pattern   brightness | detail  | R^2 from context
   latent, variance constraint (CCA)          0.888889  0.000000   0.000000   | 0.1111  | 0.8533
   rank-1 pixel prediction                    0.000000  0.992248   0.000000   | 0.0078  | 0.6782
   largest target variance (PCA)              0.000000  0.000000   0.996109   | 0.0039  | 0.0000
   CCA on the sampled images                  0.888297  0.000004   0.000001   | 0.1117  | 0.8528
   sample vs exact canonical direction: 1-|cos| 3.33e-04; sklearn CCA vs hand: |corr| of target scores 1.0000000000

verdicts
   choice of factor: CCA stripes, rank-1 pixel pattern, PCA brightness, sampled CCA stripes
   predictability R^2: CCA 0.8533 > pixel 0.6782 > PCA 0.0000: True

Every planned check passes, and the four planned expectations hold. The squared canonical correlations from the SVD match the per-factor formula to $10^{-15}$, the patterns are orthogonal to $4.5 \times 10^{-14}$, and the sampled images’ covariance error is 1.159 times its expected size. Least squares agrees with scikit-learn exactly, scikit-learn’s CCA gives the same target scores as the hand computation (|corr| printed as 1.0000000000), and the canonical direction estimated from the sampled images is within $1-|\cos| = 3.33 \times 10^{-4}$ of the exact one.

Pixel prediction hits its ceiling (D1). No predictor can explain more than $R^2_\ast = 0.2087$ of the target’s pixel variance, although the three factors carry 0.8596 of it, and least squares on the sampled images reaches 0.2062 on the test images, within finite-sample error of the ceiling. Of the 0.7913 left unexplained, the brightness makes up 0.7095 and the pixel detail 0.1774, together 0.8869, so at its optimum the pixel loss is mostly spent on what the context cannot predict. This is the situation in which the paper expects pixel targets to waste effort.

A learnable target encoder collapses (D2). The alternating fits drive $\lVert w\rVert$ from 6.865 to $8.98 \times 10^{-15}$ in 200 steps and the loss from 157.3 to $1.07 \times 10^{-28}$; on the sampled images, $\lVert w\rVert$ falls to $1.53 \times 10^{-14}$. The shrink factor at step 200 equals the top squared canonical correlation to ten decimals: 0.8533333333 for the exact iteration, and 0.8555019738 on the sampled images, which is their own sample value, 0.0022 above the exact one. The loss at $m = 0$, $w = 0$ is exactly zero. The frozen random target keeps a loss of 157.3358 (156.4933 on the sampled images), which is larger than every loss of the collapsing run after its first step, whose $w$ has shrunk. Nothing in the loss itself distinguishes a good target from a shrunken one, which is why a method that learns its targets needs a safeguard. I-JEPA’s safeguard is the EMA target encoder and the asymmetric predictor, the kind of safeguard whose analysis BYOL covers; the paper does not use a variance constraint, and its authors treat the EMA’s role as an empirical finding (§3).

A variance constraint keeps the most predictable feature (D3). With rescaling, the iteration reaches the top canonical direction to machine precision by step 100, and its loss settles at $1-\rho_1^2 = 0.146666666667$. The three target features then pick different factors. The constrained latent feature takes the stripes (squared correlation 0.888889, the pattern and brightness 0.000000), the rank-1 pixel predictor the pattern (0.992248), and the largest-variance direction the brightness (0.996109). CCA on the sampled images also picks the stripes (0.888297). Their predictability from the context orders the same way: $R^2$ 0.8533 for the constrained feature, 0.6782 for the pixel one and 0.0000 for the largest-variance one. A pixel objective weighs a factor by how much target energy it explains, so a high-energy factor that is only moderately predictable wins; the constrained latent objective weighs predictability alone and ignores energy.

The constrained feature is not the most detail-free. The table also shows, without our having predicted it, that the constrained feature carries the largest share of pixel detail, 0.1111 against 0.0078 and 0.0039. In our reading, this follows from D3: the stripes have the least target energy, $b^2 = 8$, so the pixel detail is a larger share, $\sigma^2/(b^2+\sigma^2) = 1/9$, of a feature that follows them; the constraint selects what the context predicts best, not what is cleanest. The example says nothing about which features are useful downstream, which depends on the task.

Strengths

• One view, no hand-crafted view augmentations. The method needs a single view of each image and processes the context block with the context encoder and the full image once with the target encoder, which the authors credit for its efficiency against multi-view methods such as iBOT (§4, §7).
• Strong frozen features for a masked method. Its ViT-H models’ linear-probe and low-shot results are above those of MAE, data2vec and CAE in the paper’s tables, except that ViT-H/14 only matches data2vec’s ViT-L/16 on ImageNet-1% (73.3%) (at ViT-L/16, CAE’s 78.1% is above I-JEPA’s 77.5%), and ViT-H/16 at $448^2$ matches iBOT’s ImageNet linear evaluation (81.1% against 81.0%).
• Local information. On Clevr/Dist it is well ahead of DINO and iBOT, and the supplement’s visualizations of the target encoder (Figure 7) capture objects and their pose, where samples decoded from MSN’s representations vary more in pose, scale and number of instances (Figure 8).
• Compute. Its reported pre-training cost is lower than MAE’s and iBOT’s for comparable or larger models.

Limitations

• Collapse avoidance is empirical. The paper relies on the EMA target encoder and the asymmetric predictor and offers no analysis of why they suffice; the linear example shows that without some safeguard the objective is minimized by collapse.
• Sensitive masking. The ablations move linear 1% accuracy from 54.2% to between 9.0% and 49.3% under modest changes of target scale, context scale or number of target blocks (supplement Tables 8–10), and the other masking strategies fall to 15.5–20.2% (Table 6). The ablations were run on ImageNet with ViTs.
• Semantic transfer still trails augmentation-based methods. iBOT’s linear probes are higher on CIFAR-100, Places205 and iNaturalist, and DINO’s on iNaturalist (Table 3); on Clevr/Count MAE is ahead (Table 4).
• Evaluation protocols are mixed. Table 2 takes each method’s better protocol, fine-tuning or linear probing, so its rows are not one protocol; the pixel-target ablation compares runs of different length (Table 7). Evaluating representations explains why protocols must match before comparing.
• Paper and code differ in the loss and target normalization and in the learning-rate schedule, as described under Training.

Historical Importance

I-JEPA (2023) was presented by its authors as an image instance of a joint-embedding predictive architecture, citing LeCun’s 2022 position paper for the architecture. Its main empirical point was that a masked-prediction method without view augmentations could reach the linear-probe accuracy of augmentation-based joint-embedding methods on ImageNet when the prediction is made in a learned representation space. The paper places itself closest to data2vec, which predicts representations of missing patches computed by an online target encoder, and to Context Autoencoders, which add an alignment constraint in representation space to a reconstruction loss (§4); it claims gains over both in computational efficiency and in the semantic quality of off-the-shelf features. The same group carried the objective to video in V-JEPA.

Variants

V-JEPA (Bardes et al., 2024) applies the objective to video: a ViT encodes the visible space-time patches, a predictor estimates the representations that an EMA target encoder gives the masked ones, and the authors switch the regression from I-JEPA’s $L_2$ to $\ell_1$, which they found more stable (§3.1). They also give a simple adaptation of BYOL’s argument for why an EMA target with a near-optimal predictor avoids collapse, stated as a hypothesis. Video foundation models covers V-JEPA’s objective, masking, results and its successor V-JEPA 2; it compares feature prediction with pixel prediction for video. V-JEPA’s comparison with image models also lists an I-JEPA ViT-H/16 trained on ImageNet-22K and evaluated at $512^2$, a configuration not in the CVPR paper, which scored 84.4% on ImageNet-1K with V-JEPA’s frozen attentive probe (Table 6).

Related image methods. As the I-JEPA paper describes them (§4): data2vec predicts an online target encoder’s representations of missing patches and applies to speech and text as well; Context Autoencoders combine reconstruction with an alignment term in representation space; iBOT combines a data2vec-style patch loss with DINO’s view-invariance loss; and MSN uses masking as an extra augmentation in a joint-embedding method. BYOL covers the predictor and moving-average target that several of these share, and the argument for why they keep a joint-embedding method from collapsing.

World models. The official README describes the predictor as a primitive, restricted world model that predicts high-level information about unseen regions of a static image. World models discusses the choice between predicting latents and predicting pixels in models of dynamics.

Implementations

The official PyTorch implementation, facebookresearch/ijepa, is released under the Creative Commons Attribution-NonCommercial 4.0 licence. As of November 2026 its README links four pre-trained checkpoints with their training logs and configs: ViT-H/14 at $224^2$ (300 epochs, ImageNet-1K), ViT-H/16 at $448^2$ (300 epochs, ImageNet-1K), ViT-H/14 at $224^2$ (66 epochs, ImageNet-22K) and ViT-g/16 at $224^2$ (44 epochs, ImageNet-22K). The repository was last changed in June 2023 (commit 52c1ae9). The README notes that the ViT-H/14 config is meant for 16 A100 80 GB GPUs for an effective batch size of 2,048. All experiment settings, including the mask sampler’s scales, the EMA range and the predictor’s depth and width, are set in YAML configs.