Patch Embedding

A patch embedding turns an image into the sequence of vectors a transformer consumes. It cuts the image into a grid of small patches, flattens each patch and maps it with a learned linear layer to a token of the model’s width. It is the first layer of the Vision Transformer (ViT) and its descendants, and the only point where raw pixels enter them. Its choices fix how many tokens the transformer sees, how much of each patch can survive into a token, and how the token grid responds when the image moves. A model also decides which extra tokens join the patch tokens, how the grid changes with resolution or patch size, and whether a few convolutions should replace the single linear map. This article covers the stem in general; the ViT article covers the model built on it, and positional encoding covers how tokens are told where they came from.

Definition

A patch embedding (also called a patchify layer, tokenizer or stem) maps an image to a set of token vectors. Each token comes from a fixed region of the image through the same learned function. In its basic form, the regions are non-overlapping $P \times P$ patches and the function is linear. In variants the regions overlap and the function is a small convolutional network.

Intuition

Self-attention compares every token with every other, so its cost grows with the square of the number of tokens. Attending over the 50,176 pixels of a $224 \times 224$ image is too costly, so ViT groups pixels first. A $16 \times 16$ patch is a small picture, like a word in a sentence: the embedding writes each picture as a vector, and attention relates the vectors to each other. Two consequences follow from the grouping. The vector has a fixed number of entries, so it may or may not have room for everything in the patch. And the grouping is tied to a fixed grid, so an object that moves by less than a patch lands in different patches, and its tokens change.

Formal Definition

An image $\mathbf{x} \in \mathbb{R}^{H \times W \times C}$ with $H$ and $W$ divisible by $P$ is split into $N = HW/P^2$ patches. Patch $i$, flattened, is a vector $\mathbf{x}_p^i \in \mathbb{R}^{P^2 C}$. With a learned matrix $\mathbf{E} \in \mathbb{R}^{(P^2 C) \times D}$ and bias $\mathbf{b} \in \mathbb{R}^D$, its token is

$$\mathbf{t}_i = \mathbf{x}_p^i \mathbf{E} + \mathbf{b} \in \mathbb{R}^{D}, \qquad i = 1, \dots, N.$$

ViT then prepends a learned class token and adds position embeddings (Dosovitskiy et al., 2021, Eq. 1; see the ViT paper). Because the patches tile the image without overlap, the map equals a convolution with $D$ filters of size $P \times P \times C$ and stride $P$. The Vision Transformer article checks that equality numerically, and the Transformer article counts its parameters, $P^2 C D + D$.

The general form is any function $f$ applied with stride $s$:

$$\mathbf{t}_{r,c} = f\big(\mathbf{x}[\,s r : s r + k,\; s c : s c + k\,]\big),$$

where $k$ is the region size. ViT’s stem has $k = s = P$ and linear $f$. Overlapping stems have $k > s$. Convolutional stems compose several such maps, each with its own small kernel and stride, so the overall stride is the product of the strides.

Properties

Sequence length. The number of tokens is $HW/s^2$ for total stride $s$. Halving the patch size quadruples the tokens without adding parameters to the stem. The Vision Transformer article works through the cost this adds to attention.

Width against patch size. A linear map from $P^2 C$ numbers to $D$ numbers can be inverted only if $D \geq P^2 C$ and $\mathbf{E}$ has full rank. Below that it maps a whole subspace of patch differences to the same token. The ratio $D/(P^2 C)$ varies across common models (our arithmetic from the published sizes):

Model	Patch	$P^2 C$	Width $D$	$D / (P^2 C)$
DeiT-Ti	$16 \times 16$	768	192	0.25
DeiT-S	$16 \times 16$	768	384	0.5
ViT-B/16	$16 \times 16$	768	768	1
ViT-L/16	$16 \times 16$	768	1,024	1.33
ViT-H/14	$14 \times 14$	588	1,280	2.18
ViT-B/32	$32 \times 32$	3,072	768	0.25
Swin-T (first stage)	$4 \times 4$	48	96	2

DeiT’s authors describe ViT-B’s projection as one that “conserves its overall dimension” of $3 \times 16 \times 16 = 768$ (Touvron et al., 2021, ICML). A narrower embedding need not lose much when the patches put most of their variance in a few directions that $\mathbf{E}$ keeps (our reasoning); the example below measures this on synthetic images. The LeViT authors explain a related ablation in these terms: in a low-capacity model, convolutions compress the 768-dimensional patch input effectively (Graham et al., 2021, §5.5).

Shift equivariance in whole-patch steps. A convolution is translation equivariant, but a strided one only for shifts that are multiples of its stride. Shift an image by $P$ pixels and every interior token moves one place on the grid unchanged. Shift it by one pixel and every patch contains different pixels, so every token changes. This is the subsampling problem that sampling and aliasing describes for strided convolutions and pooling in CNNs, in an extreme form: ViT subsamples by 16 in a single step (the convolutional neural network article checks the stride-2 case).

A set, not a grid. After flattening, the transformer receives the tokens as a set. Their 2D arrangement survives only through position embeddings, which is why positional encoding and the patch embedding are designed together. ViT’s authors list patch extraction and the resizing of position embeddings as the only points where 2D structure enters the model by hand (as the Vision Transformer article quotes them).

Variants

Overlapping and convolutional stems

Several 2021 models replaced the single non-overlapping projection, each with a different motivation.

• Soft split (T2T-ViT). Yuan et al. (2021) argued that ViT’s hard split of the image into tokens cannot model local structure such as edges and lines. Their tokens-to-token module splits the image into overlapping patches and concatenates the pixels or tokens in each, then alternates a transformer layer with a further soft split. Three soft splits with patch sizes 7, 3, 3 and overlaps 3, 1, 1 (strides 4, 2, 2) reduce a $224 \times 224$ image to a $14 \times 14$ grid (§3.1, §3.3). The authors note that a soft split resembles a convolution without filters, and in an ablation also tested three convolutions with kernels 7, 3, 3 and strides 4, 2, 2 in its place (§4.3).
• Convolutional token embedding (CvT). Wu et al. (2021) start each of three stages with a strided convolution over overlapping patches, followed by layer normalization; the degree of overlap is set by the stride (§3, §3.1). Each stage reduces the number of tokens and widens them, like a CNN’s stages. CvT adds no position embeddings: with them removed, CvT-13 kept its ImageNet accuracy (81.6% against 81.5% with them in every stage), while removing them from DeiT-S cost 1.8 points (Table 5). In their ablation without position embeddings, a non-overlapping patch embedding in place of the convolutional one lowered ImageNet top-1 accuracy from 81.6% to 80.7% (Table 6; the text gives the drop as 0.8%).
• A small convolutional network (LeViT). Graham et al. (2021) first grafted the early stages of a ResNet-50 below a DeiT-S and found the hybrid more accurate than either network alone on ImageNet (Table 1). LeViT then uses four $3 \times 3$ convolutions with stride 2, with channels 3, 32, 64, 128 and 256 for LeViT-256, turning a $3 \times 224 \times 224$ image into a $256 \times 14 \times 14$ map (§4.2). In the official code each convolution has no bias and is followed by batch normalization, with a Hardswish activation between convolutions. In their 100-epoch ablation of LeViT-128S, replacing the four convolutions with a single size-16 convolution hurt accuracy strongly (§5.5).

The same idea runs the other way in CNNs. ConvNeXt replaced ResNet’s stem, a $7 \times 7$ stride-2 convolution followed by max pooling, with a $4 \times 4$ stride-4 non-overlapping convolution like Swin’s patch layer. In their ResNet-50-sized model, ImageNet accuracy moved from 79.4% to 79.5%, and the authors kept the patchify stem (Liu et al., 2022, §2.2). In our reading, patchify and convolutional stems are points on one scale (kernel size against stride, and depth), and which one helps depends on the rest of the network and the training budget.

Tokens that are not patches

The patch tokens can be joined by learned tokens that come from no region of the image.

• Class token. ViT prepends one learned vector, gives it its own position embedding, and classifies from its output, following BERT. DeiT’s authors note that this departs from the pooling layers usual in computer vision and forces attention to carry information from the patches into the class token (Touvron et al., 2021, ICML, §3). The ViT authors found average pooling of the patch outputs about as accurate once its learning rate was tuned (see the Vision Transformer article). In the CaiT paper’s ablation on DeiT-S, average pooling reached 80.3% ImageNet top-1 against 79.9% with a class token (Touvron et al., 2021, ICCV, Table 2). LeViT drops the class token and average-pools its last activation map.
• Distillation token. DeiT appends a second learned token, trained to reproduce a teacher’s prediction while the class token is trained on the true label. At test time the two heads’ softmax outputs are added (§4). The learned class and distillation tokens converge to different vectors (cosine similarity 0.06), and their outputs become more similar through the network, reaching 0.93 at the last layer. Two class tokens trained on the same label instead converged to the same vector (0.999) and brought no gain (§5.2). The transformer article covers DeiT’s training recipe; knowledge distillation itself is a training method rather than part of the stem.
• Class attention. Touvron et al. (2021, ICCV) argue that ViT’s weights must serve two contradictory objectives: guiding attention between patches and summarizing them for the classifier. Inserting the class token later helps: on DeiT-S, inserting it at layer 10 of 12 gave 80.5% against 79.9% at layer 0. CaiT goes further. Its first stage is self-attention between patch tokens only. A final class-attention stage then updates only the class token, which attends to itself and to the frozen patch tokens, at a cost linear in the number of patches (§3). With 12 self-attention blocks plus 2 class-attention blocks, DeiT-S reached 80.8% (Table 2).
• Register tokens. Register tokens are the subject of Darcet et al. (2024), whose paper was not read here beyond its abstract, so this article describes them only through code. In the official DINOv2 code, the registers are learned vectors inserted after the class token, after the position embeddings have been added, so they carry no position. Their outputs are returned separately from the class and patch tokens, and outside training the model’s forward pass returns only the class token’s output. The file names of the released checkpoints with registers contain reg4, as of November 2026.

Patch merging

A hierarchical model applies patch embedding again inside the network. Swin Transformer starts from $4 \times 4$ patches, whose 48 raw values are linearly embedded to width $C$ ($C = 96$ for Swin-T). Each of its later stages begins with patch merging: the features of each $2 \times 2$ group of neighboring tokens are concatenated to $4C$ values and projected linearly to $2C$, halving the resolution (Liu et al., 2021, §3.1, §3.3). By the argument of the formal definition, this is a $2 \times 2$ convolution with stride 2 applied to the token map (our reasoning). Swin Transformer’s windowed attention and its stages belong to that model rather than to the stem.

Changing resolution or patch size

Because the stem is a convolution, it accepts any image whose sides are multiples of the stride. A larger image simply gives more tokens. What does not transfer is the position embedding, one vector per grid cell. ViT and DeiT interpolate it to the new grid before fine-tuning at higher resolution. DeiT interpolates bicubically, because bilinear interpolation shrinks the vectors’ norm (Touvron et al., 2021, ICML, §3 and §6); the positional encoding article demonstrates this.

Changing the patch size changes the stem itself. Beyer et al. (2023) found that a standard ViT evaluated at a patch size other than its training one, with the embedding weights and position embeddings resized bilinearly, degrades quickly (§3.2, Figure 3). Their PI-resize keeps tokens the same for a patch that is resized along with the weights. Write the bilinear resize of a $p \times p$ patch to $p^* \times p^*$ as a matrix $\mathbf{B} \in \mathbb{R}^{p^{*2} \times p^2}$. Then the resized weights are

$$\hat{\boldsymbol{\omega}} = (\mathbf{B}^\top)^{+} \boldsymbol{\omega}, \qquad \langle \mathbf{B}\mathbf{x}, \hat{\boldsymbol{\omega}} \rangle = \langle \mathbf{x}, \boldsymbol{\omega} \rangle \text{ for every } \mathbf{x} \text{ when } p^* \geq p.$$

For downsampling, the best weights depend in general on the distribution of patches; the authors state that the pseudo-inverse is optimal for standard Gaussian patches (§3.4, Eqs. 1–4). FlexiViT trains one ViT on $240 \times 240$ images with a patch size drawn uniformly at random each step from the sizes between 8 and 48 pixels that tile the image. The underlying parameters are a $32 \times 32$ patch kernel and a $7 \times 7$ position grid, resized on the fly (§3.3). One set of weights then serves every patch size, so accuracy can be traded against compute at deployment.

Examples

Width and shift on a synthetic image. The NumPy code below (NumPy 2.3.5; about 10 seconds on our machine, approximate since other runs shared it) measures two properties of the patch embedding. The examples in the Vision Transformer, Transformer and Positional Encoding articles cover the convolution equality, the parameter count and positions; this one measures what a token can keep, and how tokens respond to a shift.

• Part 1 embeds the $16 \times 16$ RGB patches of a $512 \times 512$ synthetic image with an affine map of width $D$: $\mathbf{t} = (\mathbf{x} - \boldsymbol{\mu})\mathbf{E}$, where $\boldsymbol{\mu}$ is the mean training patch. It then decodes each token back to pixels with the best linear decoder. The columns of $\mathbf{E}$ are either random Gaussian or the top $D$ principal directions of patches from eight other synthetic images. The code reports the fraction of the centred patch energy the round trip loses, and the reconstruction’s PSNR for pixels in $[0, 1]$.

• Part 2 slides a $512 \times 512$ window $s$ pixels to the right along a wider canvas, so the image content moves $s$ pixels left. It compares the token grids of two stems with random weights:

  • ViT-B/16’s patchify convolution;
  • LeViT-256’s four stride-2 convolutions, initialized in PyTorch’s default range, with batch normalization at its initial values in inference mode, where it only divides by $\sqrt{1 + \epsilon}$ (PyTorch’s default $\epsilon = 10^{-5}$), a factor the code omits.

  Both give $32 \times 32$ grids. The code ignores border tokens, which see zero padding in the convolutional stem. For $s \geq 16$ it compares each token with the token $\lfloor s/16 \rfloor$ columns to its right in the unshifted grid. Convolutions are cross-correlations, as in deep-learning libraries, and arrays are height × width × channels.

import numpy as np

P, C = 16, 3
DIN = P * P * C                                          # 768 numbers per 16x16 RGB patch

def make_canvas(rng, H, W, n_shapes=12, noise=0.01):
    """Synthetic RGB image in [0, 1]: shaded background, random rectangles, discs and triangles, mild noise."""
    yy, xx = np.mgrid[0:H, 0:W] / max(H, W)
    img = np.stack([rng.uniform(0.2, 0.8) + rng.uniform(-0.3, 0.3) * xx + rng.uniform(-0.3, 0.3) * yy
                    for _ in range(3)], axis=-1)
    Y, X = np.mgrid[0:H, 0:W]
    for _ in range(n_shapes):
        col, (cy, cx), r, kind = rng.uniform(0, 1, 3), rng.uniform(0, 1, 2) * (H, W), rng.uniform(8, 40), rng.integers(0, 3)
        if kind == 0:   m = (abs(Y - cy) < r) & (abs(X - cx) < 0.7 * r)                         # rectangle
        elif kind == 1: m = (Y - cy) ** 2 + (X - cx) ** 2 < r ** 2                              # disc
        else:           m = (abs(Y - cy) < r) & (abs(X - cx) < (Y - cy + r) / 2)                # triangle
        img[m] = col
    return np.clip(img + rng.normal(0, noise, img.shape), 0, 1)

def patchify(x):
    """(H, W, C) -> (N, P*P*C): non-overlapping patches in raster order (crops to a multiple of P)."""
    h, w = x.shape[0] // P * P, x.shape[1] // P * P
    return x[:h, :w].reshape(h // P, P, w // P, P, -1).transpose(0, 2, 1, 3, 4).reshape(-1, P * P * x.shape[2])

# ---- Part 1: how much of a patch survives an embedding of width D? -------------------------------
# Embedding t = (x - mu) E (an affine map, like ViT's x E + b); best linear decoder x_hat = mu + t E^+,
# i.e. the projection of x - mu onto the column space of E.
def evaluate(Xtr, Xte, widths, rng):
    mu = Xtr.mean(0)
    evals, evecs = np.linalg.eigh((Xtr - mu).T @ (Xtr - mu))
    pcs = evecs[:, ::-1]                                  # principal directions of the training patches
    Z = Xte - mu
    rows = []
    for D in widths:
        res = {}
        bases = {"random": rng.normal(0, 0.02, (DIN, D))}
        if D <= DIN:
            bases["PCA"] = pcs[:, :D]
        for name, E in bases.items():
            Zhat = (Z @ E) @ np.linalg.pinv(E)            # encode, then least-squares decode
            mse = ((Z - Zhat) ** 2).mean()
            res[name] = (((Z - Zhat) ** 2).sum() / (Z ** 2).sum(), 10 * np.log10(1 / mse))
        rows.append((D, res))
    return rows, evals.min()

widths = (48, 96, 192, 384, 576, 768, 1024)
rng = np.random.default_rng(0)
syn_tr = np.concatenate([patchify(make_canvas(np.random.default_rng(s), 256, 256)) for s in range(1, 9)])
canvas = make_canvas(np.random.default_rng(0), 512, 544)  # test image = left 512 columns; Part 2 slides along it
syn_te = patchify(canvas[:, :512])

for name, Xtr, Xte in (("synthetic shapes", syn_tr, syn_te),):
    rows, emin = evaluate(Xtr, Xte, widths, rng)
    print(f"{name}: {len(Xtr)} training patches (smallest covariance eigenvalue {emin:.1e}), {len(Xte)} test patches")
    print("    D   D/768   1-D/768 | random: energy lost  PSNR dB | PCA: energy lost  PSNR dB")
    for D, res in rows:
        line = f"{D:5d}  {D / DIN:5.2f}  {max(1 - D / DIN, 0):8.3f} | {res['random'][0]:19.3e} {res['random'][1]:8.1f} |"
        line += f" {res['PCA'][0]:16.3e} {res['PCA'][1]:8.1f}" if "PCA" in res else "        (no 1024-d PCA)"
        print(line)
    print()

# ---- Part 2: does the token grid move with the image? ---------------------------------------------
def conv2d(x, w, stride, pad):
    """Cross-correlation of x (H, W, Cin) with w (k, k, Cin, Cout), zero padding, via im2col."""
    k = w.shape[0]
    xp = np.pad(x, ((pad, pad), (pad, pad), (0, 0)))
    Ho, Wo = (xp.shape[0] - k) // stride + 1, (xp.shape[1] - k) // stride + 1
    cols = np.stack([xp[i:i + stride * Ho:stride, j:j + stride * Wo:stride] for i in range(k) for j in range(k)], 2)
    return np.einsum("hwkc,kco->hwo", cols, w.reshape(k * k, -1, w.shape[-1]))

def hardswish(x):
    return x * np.clip(x + 3, 0, 6) / 6

wr = np.random.default_rng(1)
E16 = wr.normal(0, 0.02, (P, P, C, 768))                    # ViT-B/16 patchify: one 16x16 conv, stride 16
chans = (3, 32, 64, 128, 256)                               # LeViT-256 stem: four 3x3 convs, stride 2
Ws = [wr.uniform(-1, 1, (3, 3, a, b)) / np.sqrt(9 * a) for a, b in zip(chans[:-1], chans[1:])]  # PyTorch default init range

stems = {
    "patchify 16x16/16": lambda x: conv2d(x, E16, 16, 0),
    "conv stem 4x(3x3/2)": lambda x: conv2d(hardswish(conv2d(hardswish(conv2d(hardswish(
        conv2d(x, Ws[0], 2, 1)), Ws[1], 2, 1)), Ws[2], 2, 1)), Ws[3], 2, 1),
}
# Batch norm at initialization in inference mode (mean 0, variance 1, scale 1, shift 0) divides by sqrt(1 + eps); omitted.

assert np.abs(stems["patchify 16x16/16"](canvas[:, :512]).reshape(-1, 768)
              - patchify(canvas[:, :512]) @ E16.reshape(DIN, 768)).max() < 1e-12   # conv == linear patchify
w0 = canvas[:, :512]
print("data check: window shifted by 16 px equals canvas moved 16 px:", np.array_equal(canvas[:, 16:528][:, :-16], w0[:, 16:]),
      f"| pixel range {canvas.min():.2f}..{canvas.max():.2f}")

shifts = (0, 1, 2, 4, 8, 15, 16, 17, 32)
print("\nrelative token change after shifting the image s px left (interior tokens; for s >= 16 compared with")
print("the tokens s // 16 grid columns to the right)")
print("stem                   grid  " + "".join(f"{f's={s}':>9s}" for s in shifts))
for name, f in stems.items():
    T0 = f(canvas[:, :512])
    g = T0.shape[0]
    errs = []
    for s in shifts:
        Ts, k = f(canvas[:, s:s + 512]), s // 16
        a, b = Ts[1:g - 1, 1:g - 1 - k], T0[1:g - 1, 1 + k:g - 1]     # drop border tokens that see the padding
        errs.append(np.linalg.norm(a - b) / np.linalg.norm(b))
    print(f"{name:21s} {g}x{g}  " + "".join(f"{e:9.2e}" for e in errs))

synthetic shapes: 2048 training patches (smallest covariance eigenvalue 5.2e-02), 1024 test patches
    D   D/768   1-D/768 | random: energy lost  PSNR dB | PCA: energy lost  PSNR dB
   48   0.06     0.938 |           9.490e-01     15.1 |        8.716e-03     35.5
   96   0.12     0.875 |           8.760e-01     15.4 |        6.526e-03     36.7
  192   0.25     0.750 |           7.037e-01     16.4 |        4.818e-03     38.0
  384   0.50     0.500 |           4.788e-01     18.1 |        2.858e-03     40.3
  576   0.75     0.250 |           2.670e-01     20.6 |        1.353e-03     43.6
  768   1.00     0.000 |           1.076e-24    254.6 |        1.115e-29    304.4
 1024   1.33     0.000 |           3.541e-29    299.4 |        (no 1024-d PCA)

data check: window shifted by 16 px equals canvas moved 16 px: True | pixel range 0.00..0.97

relative token change after shifting the image s px left (interior tokens; for s >= 16 compared with
the tokens s // 16 grid columns to the right)
stem                   grid        s=0      s=1      s=2      s=4      s=8     s=15     s=16     s=17     s=32
patchify 16x16/16     32x32   0.00e+00 4.12e-02 5.06e-02 6.46e-02 8.58e-02 1.10e-01 0.00e+00 3.90e-02 0.00e+00
conv stem 4x(3x3/2)   32x32   0.00e+00 4.57e-02 5.93e-02 7.80e-02 1.02e-01 1.30e-01 0.00e+00 4.33e-02 0.00e+00

From $D = 768$ on, both embeddings are exact to rounding: the energy lost is at the level of floating-point error, and PSNR is above 250 dB. Below 768, the random embedding loses close to $1 - D/768$ of the energy at every width. That is the expected share when the embedding keeps a random $D$-dimensional subspace, since such a subspace keeps on average $D/768$ of any vector’s energy (our derivation). The principal directions of other synthetic images lose under 1% even at $D = 48$, one sixteenth of the patch dimension, because shaded backgrounds and flat shapes concentrate the patches’ variation in few directions. Narrowness costs information only in proportion to how much of the patch lies outside the directions the map keeps. A trained embedding is neither random nor PCA; what it keeps is set by the task.

Part 2 shows exact shift equivariance at multiples of the stride: after a 16- or 32-pixel shift, both stems reproduce the original grid moved by one or two columns, to the last bit. At every other shift, every interior token changes. The change grows from a one-pixel shift to a 15-pixel one, and a 17-pixel shift changes the tokens about as much as a 1-pixel one. The overlapping convolutional stem is not more stable: its changes are 11–21% larger than the patchify stem’s at each shift. Its overlap blurs patch boundaries, but its four stride-2 steps still subsample by 16 overall. In our reading, the changes stay small because most patches of the synthetic image are flat or smoothly shaded, and a shift alters such content only at edges. These are untrained weights; a trained stem’s sensitivity depends on what it learned.

Common Misconceptions

• “The patch embedding is not a convolution.” For non-overlapping patches it is exactly one, with kernel and stride $P$, and implementations compute it that way.
• “A ViT’s tokens move with the image.” Only in steps of the patch size. A one-pixel shift changes every token, and an overlapping convolutional stem with the same overall stride does not remove the effect (example above).
• “Width below $P^2 C$ throws the image away.” It cannot keep every patch exactly, but when patches concentrate their variation in few directions, as the synthetic ones in the example do, a suitable projection keeps most of it.
• “The class token is necessary.” Average pooling of the patch tokens works about as well in ViT’s and CaiT’s experiments; LeViT drops the class token, and CvT adds it only in its last stage.
• “A ViT is tied to one resolution.” The stem accepts any image whose sides are multiples of the stride; only the position embeddings need resampling, and FlexiViT’s resizing extends this to the patch size.

Where It Is Used

Every vision transformer begins with some form of patch embedding. ViT and DeiT use the linear version. Masked image modeling such as the masked autoencoder embeds patches with it and then drops most of the tokens, which is possible only because each token comes from its own patch. DINO and CLIP’s ViT encoders use it, and so do SAM’s image encoder and the vision encoders of vision–language models. Hybrid models take tokens from a CNN feature map instead of pixels, one token per position. Hierarchical backbones such as Swin Transformer repeat the operation as patch merging between stages, and convolutional networks such as ConvNeXt adopted the patchify stem in place of their own.