Embeddings

An embedding is a fixed-length vector that represents an image, a region or a piece of text, computed by a map designed so that inputs that should count as similar receive nearby vectors. A photograph becomes a point $f(x)$ in a space of a few hundred or a few thousand dimensions, and questions about images (which ones show the same building, which person this is, which caption fits) become questions about distances between points. Embeddings are also called global descriptors or image representations, and in the retrieval literature neural codes when they are read from a network. They extend the idea of a local feature descriptor, which describes the neighborhood of one keypoint, to a whole image or object.

Definition

An image embedding is the output of a map $f$ from images to vectors of a fixed dimension $d$, chosen or trained so that a simple comparison of two outputs, usually the cosine similarity or the Euclidean distance, reflects a relation between the inputs that matters for a task: same object instance, same category, same identity, or a matching caption. The map can be hand-designed, as when local descriptors are pooled into one vector, or learned, as when the vector is read from a layer of a trained network. The set of all possible outputs, with the chosen comparison, is the embedding space.

Intuition

A pixel array is already a vector, but a poor one to compare. Two photographs of the same dog in different poses differ in almost every pixel, while a photograph of a dog and one of a cat on the same lawn can differ in few. An embedding is a change of coordinates in which the differences that the task ignores (position, pose, lighting, background) move a point only a little, and the differences that matter move it a lot. Nearest neighbors in the new space then share what the task cares about.

Whatever the map does not suppress stays in the vector. If a network’s outputs grow with image contrast, contrast becomes a coordinate, and a dim square can sit closer to a dim disk than to a bright square. Normalization, the subject of a section below, removes some of these unwanted coordinates after the fact.

Formal Definition

An embedding is a function

$$f : \mathcal{X} \to \mathbb{R}^d ,$$

from an input space $\mathcal{X}$ (images of a given size, image regions, or text) to $d$-dimensional real vectors. It is $\ell_2$-normalized when its outputs are rescaled to unit length,

$$u(x) = \frac{f(x)}{\lVert f(x) \rVert_2} \in \mathbb{S}^{d-1} = \{ u \in \mathbb{R}^d : \lVert u \rVert_2 = 1 \},$$

so that every image lies on the unit hypersphere. For unit vectors the cosine similarity is the dot product, and the squared Euclidean distance is a decreasing function of it:

$$\lVert u - v \rVert_2^2 = \lVert u \rVert^2 + \lVert v \rVert^2 - 2\, u^\top v = 2 - 2\cos(u, v).$$

Ranking neighbors by Euclidean distance, by cosine similarity or by dot product therefore gives the same order once the vectors are normalized. Without normalization the three differ. The comparisons themselves, and others such as the Hamming distance for binary codes, are covered under similarity and distance measures; learning $f$ so that a chosen comparison reflects a chosen relation is the subject of metric learning.

Where Embeddings Come From

Hand-designed maps

Before learned features, global image vectors were built by aggregating local descriptors such as SIFT over the whole image. The bag-of-visual-words vector quantizes each descriptor to its nearest “visual word” from a k-means vocabulary and counts the words; VLAD accumulates, for each word, the differences between the descriptors assigned to it and the word’s centroid; the Fisher vector is a probabilistic relative of VLAD (Jégou & Chum, 2012). Vectors computed more directly from the pixels, such as GIST, were also used. The only learned part of these maps is the vocabulary (or, for the Fisher vector, a Gaussian mixture), fitted without labels. In Jégou and Chum’s comparison of 128-dimensional versions on the Holidays benchmark, GIST reached 36.5% mean average precision against 45.2% for a reduced bag of words and 56.5% for an improved Fisher vector (their Table 2, which compiles most of these figures from earlier papers).

A classifier’s penultimate layer

A network trained for image classification ends in a linear layer and a softmax: each class score is a dot product between the vector entering that layer and a class weight vector. Training makes the classes linearly separable in that vector, which by itself does not require images of one class to lie close together, but in practice they often do. Krizhevsky, Sutskever and Hinton (2012, §6.1) compared images by the Euclidean distance between the 4,096-dimensional activations of AlexNet’s last hidden layer: in their Figure 4, the training images with the smallest Euclidean distance to a test image show the same kind of object, although they were not close to it in pixel space (dogs and elephants in varied poses).

Two 2014 studies measured how well such vectors work for tasks the network was not trained for:

• Off-the-shelf features. Razavian, Azizpour, Sullivan and Carlsson took the 4,096-dimensional first fully connected layer of the OverFeat network, $\ell_2$-normalized it, and used it with linear SVMs for classification and with the Euclidean distance for instance retrieval. For retrieval the plain vector reached 32.2 mAP on Oxford5k and 64.2 on Holidays; with a PCA-and-whitening chain plus a search over sub-patches (Oxford5k) or jittered copies (Holidays) it reached 68.0 and 84.3 (their Table 7). The authors reported this as better than other low-memory representations on every retrieval benchmark except Sculptures6k, leaving out methods that re-rank with spatial verification or query expansion; in the same table, however, three methods with 64k-dimensional representations score higher on Oxford5k (74.2 to 81.7 mAP).
• Neural codes. Babenko, Slesarev, Chigorin and Lempitsky compared the layers of an AlexNet-style network for retrieval, all $\ell_2$-normalized and compared with the Euclidean distance. The first fully connected layer (layer 6) was best, ahead of the last hidden layer (layer 7) and of the convolutional layer below; the class-probability output was, in the authors’ words, considerably worse (on Holidays, 0.02 mAP below layer 5). Retraining the network on 213,678 images of 672 landmarks raised Oxford mAP from 0.435 to 0.545 but lowered the score on UKB, a benchmark of small indoor objects, from 3.43 to 3.29 (out of 4); retraining on turntable views of household objects did the opposite (their Table 1).

The second result is the general lesson: an embedding encodes what its training data and labels reward, and a vector that is good for one notion of similarity can be worse for another. Transfer learning covers the use of these features for new classification tasks.

Metric and contrastive objectives

A classifier’s features cluster by class only as a side effect. Training $f$ directly on the comparison it will be used with gives more control: a loss pulls together the embeddings of pairs that should match (two photographs of one person, one landmark, or two augmented views of one image) and pushes apart those that should not. Margin losses on pairs and triplets, the triplet loss of FaceNet with its mining strategies, and proxy- and margin-based softmax losses belong to metric learning. When the matching pairs are made by augmentation instead of labels, the same idea becomes contrastive learning, one family of self-supervised learning. That article also explains why the layer before a contrastive projection head usually transfers better than the head’s output, which matters when choosing which vector to keep.

Joint image–text embeddings

Two encoders, one per modality, can be trained to map an image and a description of it to nearby points of one shared space. DeViSE (Frome et al., 2013) projected a CNN’s image features into the word-vector space of a language model, so that labels never seen with images could still be predicted from their word vectors; the multimodal learning article describes it. CLIP (Radford et al., 2021) trains both encoders from scratch on 400 million image–text pairs with a contrastive loss, and $\ell_2$-normalizes both outputs into a joint space of 512 to 1,024 dimensions depending on the model (arXiv version, Tables 19–20). Because any phrase can be embedded, the text side can define new classes at test time, as described below.

Normalization

$\ell_2$ normalization

Projecting every vector onto the unit sphere discards its length and keeps its direction. In a network of ReLU convolutions, by our reasoning, the length of a feature vector tends to grow with the input’s contrast and with how strongly the learned patterns are present, while its direction records which patterns are present; the example below shows the first effect. When length carries no information the task needs, normalization removes a nuisance coordinate; all the retrieval systems above normalized their vectors (Razavian et al., 2014; Babenko et al., 2014), and CLIP normalizes both encoders’ outputs before the dot product (Radford et al., 2021).

Wang and Isola (2020) list further reasons given in the literature for working on the hypersphere: fixed-norm vectors stabilize training wherever dot products are used, and when each class forms a well-clustered cap on the sphere, the classes are linearly separable from the rest. Normalization can also be built into training, as in contrastive losses that score cosine similarities, rather than applied afterwards.

Centering, PCA and whitening

Jégou and Chum (2012) analyzed why reducing bag-of-words and VLAD vectors with principal component analysis improved retrieval instead of only shortening the vectors, by splitting PCA into its two steps:

• Centering. A raw bag-of-words vector is non-negative, so a word absent from both images contributes nothing to their cosine similarity. Subtracting the mean vector makes a word that is rarer than average in both images count as shared evidence, which the authors called negative evidence.
• Decorrelation and whitening. Visual words that tend to occur together are counted several times when two images share the pattern behind them. Projecting onto the principal axes and dividing each coordinate by the square root of its eigenvalue equalizes the variance of all directions, which limits this co-occurrence over-counting.

Their short vector is the whitened projection, renormalized:

$$\hat{x} = \frac{\Lambda^{-1/2} P^\top (x - \mu)}{\lVert \Lambda^{-1/2} P^\top (x - \mu) \rVert_2},$$

where $\mu$ is the mean of a learning set (their Eq. 5 leaves this subtraction implicit in the PCA), the $d'$ columns of $P$ are the leading eigenvectors of its covariance, and $\Lambda = \mathrm{diag}(\lambda_1, \dots, \lambda_{d'})$ holds the corresponding eigenvalues. Comparing such vectors with the Euclidean distance resembles a Mahalanobis distance, except for the truncation and the final normalization. The renormalization mattered, by up to 10% mAP on Holidays, and whitening with a large $d'$ hurt, because dividing by small eigenvalues magnifies the noise in low-energy directions. The parameters must be learned on images separate from the test set; Jégou and Chum used an independent set of Flickr images, or Paris6k when testing on Oxford5k.

The same chain, $\ell_2$ normalization, PCA, whitening and $\ell_2$ renormalization, was then applied to CNN features: Razavian et al. (2014) reduced their 4,096-dimensional vectors to 500 dimensions this way and added a signed power transform. Babenko et al. (2014) found that PCA alone compressed layer-6 neural codes to 128 dimensions with almost no loss (Oxford mAP 0.433 against 0.435 at 4,096 dimensions) and to 16 dimensions with a graceful one (0.328), with PCA learned on 100,000 landmark images (their Table 2).

Properties

Dimension. The examples above range from 4,096 dimensions (AlexNet and OverFeat features) to 512–1,024 (CLIP’s joint space) and, after PCA, 128 or fewer. Memory and search time grow with $d$, which is one reason compression matters for retrieval over millions of images. A large $d$ does not mean the vectors use all of it: Jing, Vincent, LeCun and Tian (2022) showed that dimensional collapse, in which the embedding vectors end up spanning a lower-dimensional subspace than the space available, also happens in contrastive learning.

The hypersphere. On $\mathbb{S}^{d-1}$ all distances lie between 0 and 2, and a neighborhood is a cap around a direction. Wang and Isola (2020) describe good normalized embeddings by alignment, matching pairs close together, and uniformity, embeddings spread evenly over the sphere so that information is preserved; the contrastive learning article gives their two measures and their link to the contrastive loss.

Cones. Real embeddings are often far from uniform. Liang et al. (2022) embedded 5,000 COCO Caption images or captions with three pretrained encoders (an ImageNet ResNet-18 and CLIP’s image and text transformers), computed the cosine similarity of all pairs, and found averages of 0.56, 0.47 and 0.51 and minima of 0.23, 0.05 and 0.01: every pair at less than 90°, so each encoder’s outputs fill a narrow cone. Randomly initialized networks showed the effect even more strongly. In a two-encoder model such as CLIP the image cone and the text cone are separate, a modality gap between the two kinds of embedding. Subtracting the mean before normalizing, as in Jégou and Chum’s centering, spreads such a cone around the sphere (our observation; the example below shows it).

Neighborhoods in high dimensions. Two effects make nearest neighbors less reliable as $d$ grows (Radovanović, Nanopoulos & Ivanović, 2010). Distance concentration is the tendency of all pairwise distances to become nearly equal, so the nearest and the farthest point differ little. Hubness is the emergence of points that appear in the $k$-nearest-neighbor lists of many other points; Radovanović and colleagues showed that the distribution of these counts becomes strongly skewed as dimensionality grows, more closely tied to the data’s intrinsic dimensionality than to $d$ itself. Hubs are returned for many unrelated queries; the image–text retrieval article describes a correction for cross-modal embeddings.

Invariance is learned, not given. An embedding is invariant to exactly the changes that its training made irrelevant: retraining on landmarks helped landmark retrieval and hurt object retrieval in Babenko et al.’s experiments, and the layer that served best differed between queries. Nothing in the definition of an embedding guarantees invariance to rotation, scale or lighting.

Examples

The PyTorch script below renders $32 \times 32$ grayscale images (intensities in $[0, 1]$) of five shapes, a disk, a square, a triangle, a cross and a ring, at random positions and sizes, on random background levels, with a random contrast between 0.15 and 0.85 and pixel noise. A small CNN with three convolutional layers and global average pooling is trained for 300 Adam steps to classify the shape; its 32-dimensional pooled feature, the input to the final linear layer, is the embedding $f(x)$. On 500 held-out images the script measures how the norm of $f(x)$ follows contrast, the cosines within and between classes before and after centering, and the leave-one-out nearest-neighbor accuracy, which is Recall@1 with classes as relevance. It compares four versions of three embeddings: raw pixels, the same CNN untrained, and the trained CNN, each raw, $\ell_2$-normalized, centered and normalized, and PCA-whitened to 16 dimensions and renormalized, with the mean and PCA learned on the 2,000 training images. The last lines repeat the trained-CNN row for five seeds. It uses PyTorch 2.14 on one CPU thread and takes about 25 seconds; two runs printed identical output.

import torch
import torch.nn as nn
import torch.nn.functional as F

torch.set_num_threads(1)
H = 32
yy, xx = torch.meshgrid(torch.arange(H) - 15.5, torch.arange(H) - 15.5, indexing="ij")

def make_images(n, gen):
    """n grayscale 32x32 images in [0, 1]: one of five shapes at a random position and
    size, on a random background level, with a random contrast in [0.15, 0.85] and noise.
    Returns images (n, 1, 32, 32), labels and the contrast of each image."""
    labels = torch.randint(0, 5, (n,), generator=gen)
    contrast = 0.15 + 0.7 * torch.rand(n, generator=gen)
    r = (6 + 4 * torch.rand(n, generator=gen))[:, None, None]       # radius 6-10 px
    c = (torch.rand(n, 2, generator=gen) - 0.5)[:, :, None, None] * (30 - 2 * r[:, None])
    y, x = yy - c[:, 0], xx - c[:, 1]                                # (n, 32, 32)
    shapes = torch.stack([
        x**2 + y**2 < r**2,                                          # disk
        (x.abs() < 0.85 * r) & (y.abs() < 0.85 * r),                 # square
        (y < 0.8 * r) & (y > -r + 2 * x.abs()),                      # triangle
        ((x.abs() < 0.3 * r) | (y.abs() < 0.3 * r)) & (x.abs() < r) & (y.abs() < r),  # cross
        (x**2 + y**2 < r**2) & (x**2 + y**2 > (0.55 * r) ** 2),      # ring
    ], 1)
    m = shapes[torch.arange(n), labels].float()
    bg = 0.1 * torch.rand(n, 1, 1, generator=gen)
    imgs = (bg + contrast[:, None, None] * m)[:, None]
    imgs = imgs + 0.03 * torch.randn(imgs.shape, generator=gen)
    return imgs.clamp(0, 1), labels, contrast

class Net(nn.Module):
    """Small CNN classifier; the 32-D input to the last layer is the embedding f(x)."""
    def __init__(self):
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv2d(1, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
            nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
            nn.Conv2d(32, 32, 3, padding=1), nn.ReLU(),
            nn.AdaptiveAvgPool2d(1), nn.Flatten())
        self.classifier = nn.Linear(32, 5)
    def forward(self, x):
        return self.classifier(self.features(x))

def l2n(z):
    return z / z.norm(dim=1, keepdim=True)

def pca_whiten(ztrain, d_out):
    """Learn mean, top-d_out principal axes and eigenvalues on ztrain (Jegou & Chum, 2012)."""
    mu = ztrain.mean(0)
    evals, evecs = torch.linalg.eigh(torch.cov((ztrain - mu).T))
    idx = evals.argsort(descending=True)[:d_out]
    P, lam = evecs[:, idx], evals[idx]
    return lambda z: l2n((z - mu) @ P / lam.sqrt())

def r_at_1(z, y):
    """Leave-one-out: does each test image's nearest other image (Euclidean) share its class?"""
    d = torch.cdist(z, z)
    d.fill_diagonal_(float("inf"))
    return (y[d.argmin(1)] == y).float().mean().item()

def run(seed, verbose=False):
    torch.manual_seed(seed)
    gen = torch.Generator().manual_seed(seed)
    xtr, ytr, _ = make_images(2000, gen)
    xte, yte, cte = make_images(500, gen)
    net = Net()
    untrained = Net()
    untrained.load_state_dict(net.state_dict())          # same initialization, never trained
    opt = torch.optim.Adam(net.parameters(), lr=3e-3)
    for step in range(300):
        idx = torch.randint(0, len(xtr), (64,), generator=gen)
        loss = F.cross_entropy(net(xtr[idx]), ytr[idx])
        opt.zero_grad()
        loss.backward()
        opt.step()
    with torch.no_grad():
        acc = (net(xte).argmax(1) == yte).float().mean().item()
        maps = {"pixels (1024-D)": lambda x: x.flatten(1),
                "untrained CNN (32-D)": untrained.features,
                "trained CNN (32-D)": net.features}
        rows = {}
        for name, f in maps.items():
            ztr, zte = f(xtr), f(xte)
            white = pca_whiten(ztr, 16)
            rows[name] = [r_at_1(zte, yte), r_at_1(l2n(zte), yte),
                          r_at_1(l2n(zte - ztr.mean(0)), yte), r_at_1(white(zte), yte)]
        z = net.features(xte)
        if verbose:
            print(f"trained CNN: test classification accuracy {acc:.3f}")
            norm = z.norm(dim=1)
            print(f"correlation of ||f(x)|| with image contrast: "
                  f"{torch.corrcoef(torch.stack([norm, cte]))[0, 1]:.3f}")
            u = l2n(z)
            cos = u @ u.T
            off = ~torch.eye(len(u), dtype=torch.bool)
            same = (yte[:, None] == yte[None, :]) & off
            print(f"mean cosine, all pairs {cos[off].mean():.3f}; same class {cos[same].mean():.3f}; "
                  f"different class {cos[off & ~same].mean():.3f}")
            uc = l2n(z - net.features(xtr).mean(0))
            cc = uc @ uc.T
            print(f"after centering:  all pairs {cc[off].mean():.3f}; same class {cc[same].mean():.3f}; "
                  f"different class {cc[off & ~same].mean():.3f}")
            gap = (torch.cdist(u, u) ** 2 - (2 - 2 * cos)).abs().max().item()
            print(f"max | ||u - v||^2 - (2 - 2 cos(u, v)) | over unit vectors: {gap:.1e}")
            # what decides each test image's nearest neighbour: contrast or shape?
            low = cte < cte.quantile(0.2)                        # the 100 lowest-contrast images
            for label, emb in (("raw", z), ("l2-normalized", u)):
                d = torch.cdist(emb, emb)
                d.fill_diagonal_(float("inf"))
                nn1 = d.argmin(1)
                print(f"{label:13s} 1-NN: mean |contrast difference| {(cte - cte[nn1]).abs().mean():.3f}, "
                      f"same class {(yte[nn1] == yte).float().mean():.3f} (all), "
                      f"{(yte[nn1] == yte)[low].float().mean():.3f} (lowest-contrast 100)")
            print("\nleave-one-out R@1 on 500 test images (5 classes, chance about 0.2)")
            print(f"{'embedding':22s} {'raw':>6s} {'l2':>6s} {'center+l2':>10s} {'PCA-whiten16+l2':>16s}")
            for name, r in rows.items():
                print(f"{name:22s} {r[0]:6.3f} {r[1]:6.3f} {r[2]:10.3f} {r[3]:16.3f}")
    return rows["trained CNN (32-D)"]

if __name__ == "__main__":
    print("PyTorch", torch.__version__)
    res = torch.tensor([run(s, verbose=(s == 0)) for s in range(5)])
    print("\ntrained CNN, R@1 by seed:   raw     l2  center+l2  PCA-whiten16+l2")
    for s, r in enumerate(res.tolist()):
        print(f"  seed {s}:{'':17s} {r[0]:6.3f} {r[1]:6.3f} {r[2]:10.3f} {r[3]:16.3f}")

PyTorch 2.14.1
trained CNN: test classification accuracy 0.974
correlation of ||f(x)|| with image contrast: 0.784
mean cosine, all pairs 0.830; same class 0.978; different class 0.794
after centering:  all pairs 0.024; same class 0.495; different class -0.094
max | ||u - v||^2 - (2 - 2 cos(u, v)) | over unit vectors: 6.0e-07
raw           1-NN: mean |contrast difference| 0.060, same class 0.946 (all), 0.910 (lowest-contrast 100)
l2-normalized 1-NN: mean |contrast difference| 0.094, same class 0.988 (all), 0.980 (lowest-contrast 100)

leave-one-out R@1 on 500 test images (5 classes, chance about 0.2)
embedding                 raw     l2  center+l2  PCA-whiten16+l2
pixels (1024-D)         0.626  0.748      0.690            0.706
untrained CNN (32-D)    0.594  0.588      0.624            0.592
trained CNN (32-D)      0.946  0.988      0.962            0.970

trained CNN, R@1 by seed:   raw     l2  center+l2  PCA-whiten16+l2
  seed 0:                   0.946  0.988      0.962            0.970
  seed 1:                   0.930  0.954      0.928            0.932
  seed 2:                   0.936  0.954      0.940            0.954
  seed 3:                   0.974  0.978      0.950            0.930
  seed 4:                   0.862  0.942      0.802            0.846

The network was trained only to name shapes, yet its penultimate vector already groups them: a nearest-neighbor search over it finds a same-shape image far more often than one over raw pixels or over the same network before training. Its length, however, still tracks contrast, a nuisance the classifier could ignore because, in our reading, scaling all class scores together barely changes which one is largest. Raw nearest neighbors are therefore matched by contrast more closely than normalized ones, and the lowest-contrast images, whose vectors are the shortest, gain more from normalization than the average image. Normalizing raised Recall@1 for the trained network in all five seeds, by 0.004 to 0.080.

The cosines show a cone. Before centering, two images of different shapes still have a mean cosine near 0.8, largely, we think, because post-ReLU features are non-negative; after subtracting the training mean, unrelated images are close to orthogonal and same-class pairs keep a clearly positive cosine. Centering spreads the vectors over the sphere, but neither it nor whitening to 16 dimensions beat plain normalization for the trained network in any seed, and in the seed-0 table plain normalization also did best for raw pixels. Our reading is that on this easy five-class task the trained network’s directions were already well separated, and that whitening gave equal weight to low-variance directions carrying mostly noise, the effect Jégou and Chum reported for large $d'$. Post-processing steps are therefore choices to validate on held-out data, not rules. The printed identity check confirms $\lVert u - v \rVert^2 = 2 - 2\cos(u, v)$ to floating-point precision.

Limitations

• The similarity is the training objective’s. Two embeddings that are close are similar in whatever sense the training rewarded: object category for an ImageNet classifier, landmark identity after landmark retraining, caption match for CLIP. Babenko et al.’s results show the same vector improving on one benchmark and degrading on another.
• Nuisances leak into the vector. Contrast in the example above, and background, viewpoint or image style in real data, remain wherever training did not remove them. Normalization removes only the vector’s length.
• Embeddings from different models are not comparable. Coordinates have no fixed meaning: an independently trained model places the same images elsewhere. Shen, Xiong, Xia and Soatto (2020) compared face embeddings from an old model with those of a new model trained independently on more data and observed almost 0% accuracy in both of their IJB-C tests; updating the model meant recomputing the gallery (“backfilling”) unless the new model was trained to be compatible with the old one.
• High-dimensional neighborhoods. Distance concentration and hubs make some neighbors unreliable, and the effective dimension can be lower than $d$ suggests.
• Lost detail. A single global vector summarizes an image; small objects, spatial layout and fine-grained differences can be averaged away. Retrieval systems therefore re-rank with local features or search over sub-regions, as Razavian et al. did.

Common Misconceptions

• “The embedding is the network’s output.” For a classifier, the useful vector is usually an earlier layer: in Babenko et al.’s study the class probabilities did considerably worse for retrieval than the hidden layers, and a contrastive model’s projection head is usually discarded.
• “Cosine similarity and Euclidean distance give different neighbors.” They do for unnormalized vectors; after $\ell_2$ normalization they rank identically, since $\lVert u - v \rVert^2 = 2 - 2\cos(u, v)$.
• “More dimensions are always better.” Babenko et al.’s neural codes lost almost nothing when compressed from 4,096 to 128 dimensions by PCA, and whitening too many dimensions amplifies noise (Jégou & Chum, 2012).
• “Well-separated clusters in a t-SNE plot prove the embedding separates the classes.” The map’s cluster sizes and gaps are not faithful to the original space; nearest-neighbor accuracy or a linear probe measured in the embedding space itself is the test.

Where It Is Used

• Retrieval. Instance and category retrieval rank a gallery by distance to the query’s embedding, with exact or approximate nearest-neighbor search over millions of vectors; Recall@K and average precision score the ranking. Image–text retrieval does the same across modalities.
• Verification and identification. Face recognition and person re-identification compare a probe’s embedding with enrolled ones: verification thresholds the similarity of one pair, and identification ranks a gallery.
• Nearest-neighbor classification and linear probes. A k-nearest-neighbor vote or a linear classifier on frozen embeddings evaluates self-supervised features and adapts pretrained ones to new labels (transfer learning).
• Zero-shot classification. CLIP embeds one prompt per class, such as “a photo of a dog.”, and assigns an image to the class whose text embedding has the highest cosine similarity; its best model reached 76.2% top-1 accuracy on ImageNet zero-shot (Radford et al., 2021). The CLIP article covers the classifier, prompts and ensembling.
• Clustering and deduplication. Grouping images with k-means or a similarity threshold in embedding space can find near-duplicates and organize unlabeled collections.
• Matching inside larger systems. Learned local descriptors are embeddings of patches used for feature matching, and trackers such as DeepSORT compare unit-length appearance embeddings of detections across frames.

Visualization with t-SNE and UMAP

To look at an embedding space, its vectors are mapped to two dimensions. t-SNE (van der Maaten & Hinton, 2008) converts distances into neighbor probabilities with a Gaussian around each point, whose width is set by a user-chosen perplexity (a smooth measure of the effective number of neighbors; the authors give 5 to 50 as typical), and places points in the plane so that a heavy-tailed Student-t distribution reproduces those probabilities. The heavy tail counters the crowding problem: a ten-dimensional neighborhood has far more room at moderate distances than a two-dimensional map, so a map that keeps small distances must push moderately distant points much farther away. UMAP (McInnes, Healy & Melville, 2018) builds a neighbor graph with a different theory and optimization; its authors report visualization quality competitive with t-SNE, shorter run times, and, arguably, more preserved global structure.

Both methods preserve local neighborhoods at the expense of everything else. The t-SNE authors list among its weaknesses a cost function that is not convex, so maps can differ between runs, and sensitivity to the data’s intrinsic dimensionality; they add that intrinsically high-dimensional data cannot be fully represented in two or three dimensions. Wattenberg, Viégas and Johnson (2016) showed on synthetic Gaussian data that cluster sizes in a t-SNE map mean nothing, that distances between well-separated clusters may mean nothing, and that random noise can look structured at low perplexity. The UMAP authors warn similarly that it can find manifold structure in the noise of small, noisy datasets. A map is a picture of neighborhoods, not a measurement of the space.