InfoNCE Loss

InfoNCE is the loss that asks a model to find the true partner of a sample among $K$ candidates: the patch that really lies below a given image region, the second augmented view of the same photograph, the caption that belongs to an image. A critic scores every candidate, a softmax turns the scores into probabilities, and the loss is the cross-entropy of picking the right one. The name joins Info, for mutual information, with NCE, for noise-contrastive estimation. As Hénaff et al. (2020) describe it, the loss is the objective of Contrastive Predictive Coding (CPC; van den Oord, Li & Vinyals, 2018), and it is called InfoNCE because it was inspired by noise-contrastive estimation. In vision it is the objective of CPC on image patches, SimCLR’s NT-Xent, MoCo and the image–text loss of CLIP; instance discrimination used the older binary form of NCE. The contrastive learning article covers those methods, NT-Xent’s gradient and the temperature as a family overview. This article covers where the loss comes from, why it is a lower bound on mutual information, how tight that bound is, and what the critic and the temperature do to it.

Definition

Take a batch of $K$ pairs $(x_i, y_i)$ , each drawn from a joint distribution $p(x, y)$ . In vision, $x$ and $y$ might be two crops of one image, or a context region and the patch below it. Every $y_i$ has one positive, its own $x_i$ , and $K - 1$ negatives, the other $x_j$ in the batch. A critic $f(x, y)$ gives a real score to every pairing. The InfoNCE loss is the mean cross-entropy of identifying the positive:

\mathcal{L}_{\text{NCE}} = -\,\mathbb{E}\left[\frac{1}{K}\sum_{i=1}^{K} \log \frac{e^{f(x_i, y_i)}}{\sum_{j=1}^{K} e^{f(x_j, y_i)}}\right].

The quantity

I_{\text{NCE}} = \log K - \mathcal{L}_{\text{NCE}}

is a lower bound on the mutual information $I(X; Y)$ for any critic (Poole et al., 2019, §2.3, eqs. 8–10). Minimizing the loss raises the bound. The batch here is the full set of $K$ candidates, so $\log K$ is the loss of a critic that cannot tell the candidates apart.

Intuition

Mutual information measures how much one view says about the other. If knowing a patch’s context makes its content predictable, the two share a lot of information. Measuring that directly needs the densities $p(y \mid x)$ and $p(y)$ , which are unavailable for images. InfoNCE replaces density estimation with a recognition game. If the views share a lot of information, a good critic can pick the true partner out of many decoys. If they share none, no critic can do better than chance, and the loss stays at $\log K$ .

The game also shows the bound’s limit. With $K$ candidates, perfect recognition can only reduce the loss from $\log K$ to $0$ . So a batch of 256 can certify at most $\log 256 \approx 5.5$ nats, however much information the views share. In our reading, two views of one high-resolution photograph can share far more than that, and the bound then reports the batch size, and the critic, rather than the information.

Formal Definition

Noise-contrastive estimation

Gutmann and Hyvärinen (2010) introduced noise-contrastive estimation (NCE) to fit statistical models whose densities are known only up to a normalizing constant, such as Markov random fields. Their idea was to fit the model by logistic regression that tells observed data $x_t$ apart from artificial noise $y_t$ drawn from a known density $p_n$ , with as many noise samples as data samples. The model’s log-density enters the classifier through the log-ratio

G(u; \theta) = \ln p_m(u; \theta) - \ln p_n(u), \qquad h(u; \theta) = \frac{1}{1 + e^{-G(u;\theta)}},

and the parameters maximize

J_T(\theta) = \frac{1}{2T}\sum_{t} \Big[\ln h(x_t; \theta) + \ln\big(1 - h(y_t; \theta)\big)\Big]

(their eqs. 3–5). The normalizing constant becomes an ordinary parameter $c$ , with $\ln p_m = \ln p_m^0 + c$ . Their Theorem 1 states that, in the limit of infinite data, the objective is maximized when the model log-density equals the data log-density. This holds without any normalization constraint, and the maximizer comes out normalized automatically. There are no other extrema if the noise density is nonzero wherever the data density is. They argue that the noise should be close to the data, because otherwise the classification is too easy. They show that when the noise equals the data distribution, the estimator’s asymptotic covariance is twice the Cramér–Rao bound (§2.4). They also applied NCE to natural images: they fitted a two-layer model of $30 \times 30$ patches from van Hateren’s database and a Markov random field. In both, the learned filters (the first layer, in the two-layer model) came out Gabor-like (§3.2).

NCE is a binary classification: one sample at a time, labelled data or noise. Wu et al. (2018) used it in vision in this binary form. They approximated a softmax over every image in the training set (instance discrimination) by classifying memory-bank features against noise samples drawn uniformly. Following earlier work, they took noise samples to be $m$ times more frequent than data, and they treated the normalizing constant as a constant, estimated by Monte Carlo from the initial batches (their §3.2).

From NCE to InfoNCE

InfoNCE turns the binary decision into a $K$ -way one: which of the $K$ candidates is the positive? Hénaff et al. (2020, §2.1) give the CPC objective on images in this form. Every location’s predicted feature is scored against the true feature and against negatives drawn from other locations in the image and from other images in the batch. A softmax rates the true feature’s probability, and the usual cross-entropy is applied. The authors of SimCLR, who named their normalized, temperature-scaled version NT-Xent, note that the same loss had been used before, including in CPC and by Wu et al. (Chen et al., 2020).

Why it is a lower bound

Poole et al. (2019) gave what they call the first proof that the loss is a lower bound on mutual information; according to their footnote, the derivation in the CPC paper relied on an approximation, which they show is unnecessary. Their route, §2.3:

Mutual information does not change if independent variables are added. So $I(X_1; Y) = I(X_1, X_{2:K}; Y)$ when $X_{2:K}$ are drawn from $p(x)$ independently of $Y$ .
They apply an earlier single-sample lower bound, due to Nguyen, Wainwright and Jordan, to this multi-sample information. The critic is normalized by the batch’s own Monte Carlo estimate of the partition function, $m(y; x_{1:K}) = \frac{1}{K}\sum_i e^{f(x_i, y)}$ .
With that choice, the high-variance term of the single-sample bound becomes exactly the constant 1 (their eq. 9). What remains is $I_{\text{NCE}}$ .

Two consequences follow directly.

The bound can never exceed $\log K$ . Each term is $\log K$ plus the log of a softmax probability, and a probability is at most 1. So every batch value, not only the expectation, is at most $\log K$ . Poole et al., crediting the point to the CPC paper, conclude that the bound is loose whenever $I(X; Y) > \log K$ .
The optimal critic. Poole et al., citing Ma and Collins (2018), give the optimal critic as $f(x, y) = \log p(y \mid x) + c(y)$ , for any function $c$ of $y$ alone. So the critic needs the conditional density but not the marginal $p(y)$ . When $p(y \mid x)$ is known, it can be plugged in directly as $f$ (their eq. 12). In our reading, the bound with this critic still approaches $I(X; Y)$ only as $K$ grows, as the example below shows.

Which side is normalized. In Poole et al.’s eq. 10 the softmax for pair $i$ runs over the $y_j$ for a fixed $x_i$ . In their derivation (eqs. 8–9) and in eq. 12 it runs over the $x_j$ for a fixed $y_i$ . The stated optimal critic, $\log p(y \mid x) + c(y)$ , belongs to the second convention. We read this from the rendered pages of the PMLR copy. In our reading, both conventions give valid bounds, since mutual information is symmetric. The difference is in which free term the softmax cancels: a $c(y)$ when it runs over the $x_j$ , a $c(x)$ when it runs over the $y_j$ . This article and its example use the eq. 12 convention: the sum over the $x_j$ , as in the definition above. Symmetric losses such as CLIP’s average the two directions.

A matching upper bound

When $p(y \mid x)$ is known, Poole et al. (eq. 13) also give a leave-one-out upper bound. It is the same expression with the positive removed from the denominator:

I(X; Y) \le \mathbb{E}\left[\frac{1}{K}\sum_{i=1}^{K} \log \frac{p(y_i \mid x_i)}{\frac{1}{K-1}\sum_{j \ne i} p(y_i \mid x_j)}\right].

Together the two bounds sandwich the mutual information, with no learned variational distribution. Poole et al. found that the upper bound has large bias and variance when the batch is too small.

Properties

Saturation at $\log K$ . The bound’s bias grows with the information. Poole et al. computed the bounds with the optimal critic on a 20-dimensional correlated Gaussian (their Fig. 3). InfoNCE had low variance but high bias when the batch was small and the information large. The absolute bias grew linearly with $I$ , because the estimate stays near a constant while $I$ keeps growing. They note that the optimal critic does not depend on the batch size and can be fitted with small batches. Estimating a high mutual information accurately still needs a large batch at test time.

No estimator escapes the logarithm. McAllester and Stratos (2020) proved that the limit is not specific to InfoNCE. Any distribution-free lower bound on mutual information that holds with high confidence from $N$ samples cannot be larger than $O(\ln N)$ . Their Theorem 1.1 gives $2 \ln N + 5$ for $N \ge 50$ , with probability at least 0.96. In their Table 1, on 128-dimensional correlated Gaussians with $N = 128$ samples per step and every estimator tuned with access to the true value, CPC’s InfoNCE estimates were 2.73, 4.82 and 4.85 nats. The true values were 4.13, 18.41 and 106.29 nats, and $\ln N = 4.85$ . Their difference-of-entropies estimator, which comes with neither an upper nor a lower bound guarantee, came within 0.06 nats of the truth in the first two rows and 2.11 nats below it in the third (104.18 against 106.29).

Large $K$ . Wang and Isola (2020, Theorem 1) analyzed the loss with $M$ negatives as $M \to \infty$ . After subtracting $\log M$ , it converges to a limit whose two terms reward aligned positives and uniformly spread embeddings, with deviation $O(M^{-2/3})$ . The contrastive learning article summarizes their alignment and uniformity measures.

The bound does not explain the features. Wang and Isola (2020) note that reading the loss as a mutual-information bound is known to be inconsistent with how it behaves in practice. Citing earlier work, they say that maximizing a tighter bound often leads to worse downstream performance. In our reading, this matches the saturation result: a loss that bottoms out near $\log K$ cannot be measuring hundreds of nats of shared information. What it trains, which features and which invariances, depends on the critic’s form and the choice of views.

Temperature limits a bounded critic. In practice the critic is not a density ratio but a cosine similarity $s(x, y) \in [-1, 1]$ divided by a temperature $\tau$ , as in NT-Xent. The following derivation is ours. Write $s_{ij}$ for the similarity of $y_i$ to $x_j$ . Each term of $I_{\text{NCE}}$ is

\log K - \log\Big(1 + \sum_{j \ne i} e^{(s_{ij} - s_{ii})/\tau}\Big) \;\le\; \log K - \log\big(1 + (K - 1)\,e^{-2/\tau}\big),

because $s_{ij} - s_{ii} \ge -2$ . So whatever the encoder, a cosine critic at temperature $\tau$ cannot certify more than this cap. With $K = 256$ ( $\log K = 5.545$ ) the cap is 5.534 nats at $\tau = 0.2$ , 3.810 at $\tau = 0.5$ and 1.975 at $\tau = 1$ : small temperatures leave it at $\log K$ , large ones bring it far below. It also follows from the form (again our reasoning) that the bound is concave in $1/\tau$ . It is a linear term in $1/\tau$ minus a log-sum-exp of terms that are linear in $1/\tau$ . So for fixed embeddings any local maximum over $1/\tau$ is the global one. If some negative outscores its positive, the bound falls without limit as $\tau \to 0$ , so the maximum is not at $\tau \to 0$ . The contrastive learning article covers the other side of the temperature: how $\tau$ shares the push among hard negatives, and the limit $\tau \to 0$ , where $\tau$ times the loss tends to a zero-margin triplet loss.

A different “contrastive loss”. The name contrastive loss also denotes Hadsell, Chopra and LeCun’s margin loss on pairs (Contrastive Loss). That loss compares one pair at a time and has no softmax over candidates. The metric learning article warns about the shared name.

Examples

CPC on image patches

Hénaff et al. (2020) describe CPC on images as follows:

Each image is cut into a grid of overlapping patches. An encoder maps each patch to a vector $z_{i,j}$ .
A masked convolutional context network summarizes the rows above a position into a context vector $c_{i,j}$ .
A linear map $W_k$ predicts the feature $k$ rows below, $\hat z_{i+k,j} = W_k c_{i,j}$ .
The InfoNCE loss scores $\hat z_{i+k,j}^\top z_{i+k,j}$ against the same prediction’s products with negatives from other locations and other images.

The critic here is a bilinear product. Poole et al. call this form separable, $h(x)^\top g(y)$ . It needs only $2K$ network evaluations per batch, against $K^2$ for a critic that reads both inputs jointly. In their toy problems, the separable form gave worse estimates for the single-sample bounds than the joint form, while the multi-sample bounds, InfoNCE among them, were more robust to the choice.

Hénaff et al.’s CPC v2 made several changes:

predictions from below, left and right as well as from above;
larger patches;
layer instead of batch normalization;
stronger patch-level augmentation.

They hypothesized that batch normalization let large models find a trivial solution to CPC through the batch statistics. In their Table 1 (ImageNet, linear classifier on the frozen features, single crop), CPC v2 with a ResNet-161 reached 71.5% top-1, against 48.7% for CPC v1 as they report it. CPC v2 with a ResNet-50 reached 63.8%. In the same table, they report MoCo with a ResNet-50 at 60.6% and instance discrimination at 54.0%.

Instance discrimination, SimCLR, MoCo and CLIP

Wu et al. (2018) approximated a softmax over every training image with binary NCE. SimCLR and MoCo use the $K$ -way softmax directly, with negatives from the batch or from a queue. CLIP applies it in both directions across a batch of image–text pairs. The contrastive learning and vision–language pretraining articles describe these recipes.

Example: InfoNCE against a known mutual information

To test the bound, the true mutual information must be known exactly. The script builds such a case from image-like data:

The patches. Each $x$ is an $8 \times 8$ patch of a smooth Gaussian random field: unit variance, squared-exponential pixel covariance $C$ with a correlation length of 1.5 pixels.
The second view. $y = x + \sigma n$ , with $n$ white Gaussian noise.
The true information. For this model it is $I = \tfrac12 \sum_k \log(1 + \lambda_k/\sigma^2)$ , with $\lambda_k$ the eigenvalues of $C$ . The script picks $\sigma$ to make $I$ = 1, 3, 6 or 12 nats. Mutual information does not change when $y$ is scaled, so the form $y = \rho x + \text{noise}$ is the same family and $\rho = 1$ loses nothing.
The bounds. With the optimal critic $f = \log p(y \mid x)$ , normalized over the $x_j$ (eq. 12), the script computes InfoNCE and the leave-one-out upper bound for $K$ = 4 to 1,024. All batches are cut from one pool of 65,536 pairs per level (seeds 0–3). The tables give the mean over batches and its standard error (SE).
The checks. The script checks the data, and that adding $c(y) = -\log p(y)$ to the critic changes nothing under the eq. 12 normalization. It then replaces the critic with a cosine similarity at six temperatures, at $I = 6$ and $K = 256$ , and finds the best temperature.

Nothing is trained. It uses NumPy 2.3.5 and SciPy 1.16.3 on one thread and takes about 25–30 seconds. Two runs printed identical output.

import os
for v in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS", "VECLIB_MAXIMUM_THREADS"):
    os.environ[v] = "1"                              # one BLAS thread, set before NumPy loads
import numpy as np
from scipy.special import logsumexp
from scipy.optimize import brentq, minimize_scalar

S, ELL, M = 8, 1.5, 65536                            # 8x8 patches, correlation length 1.5 px, pairs per level
LEVELS, KS = [1.0, 3.0, 6.0, 12.0], [4, 16, 64, 256, 1024]
d = S * S
pos = np.stack(np.meshgrid(np.arange(S), np.arange(S), indexing="ij"), -1).reshape(-1, 2)
C = np.exp(-((pos[:, None] - pos[None]) ** 2).sum(-1) / (2 * ELL**2))   # pixel covariance of x
lam, U = np.linalg.eigh(C)
lam = np.clip(lam, 0, None)
L = U * np.sqrt(lam)                                 # x = L z, z ~ N(0, I), has covariance C

def true_mi(sigma):
    """I(X;Y) in nats for y = x + sigma * noise: 0.5 * sum_k log(1 + lambda_k / sigma^2)."""
    return 0.5 * np.log1p(lam / sigma**2).sum()

def mi_from_joint(sigma):
    """The same quantity from log-determinants of the covariances of x, y and (x, y)."""
    Cy = C + sigma**2 * np.eye(d)
    J = np.block([[C, C], [C, Cy]])
    ld = lambda A: np.linalg.slogdet(A + 1e-12 * np.eye(len(A)))[1]
    return 0.5 * (ld(C) + ld(Cy) - ld(J))

def log_p_y(y, sigma):
    """log N(y; 0, C + sigma^2 I), the marginal density of the second view."""
    Lc = np.linalg.cholesky(C + sigma**2 * np.eye(d))
    z = np.linalg.solve(Lc, y.T)
    return -0.5 * (z**2).sum(0) - np.log(np.diag(Lc)).sum() - 0.5 * d * np.log(2 * np.pi)

def log_cond(Y, X, sigma):
    """F[b, i, j] = log p(y_i | x_j) for every pair in each batch b (Y, X of shape (B, K, d))."""
    D = (Y**2).sum(-1)[..., :, None] + (X**2).sum(-1)[..., None, :] - 2 * Y @ X.transpose(0, 2, 1)
    return -D / (2 * sigma**2) - 0.5 * d * np.log(2 * np.pi * sigma**2)

def infonce(F, over="x"):
    """InfoNCE bound of each batch for critic values F[b, i, j] = f(x_j, y_i).
    over="x": each y_i picks its x among the K in the batch (Poole et al., eq. 12);
    over="y": each x_j picks its y instead."""
    K = F.shape[1]
    diag = np.diagonal(F, axis1=1, axis2=2)
    return (diag - logsumexp(F, axis=2 if over == "x" else 1)).mean(1) + np.log(K)

def leave_one_out(F):
    """Upper bound of Poole et al. (eq. 13): the positive is left out of the denominator."""
    K = F.shape[1]
    G = F.copy()
    G[:, np.arange(K), np.arange(K)] = -np.inf
    return (np.diagonal(F, axis1=1, axis2=2) - logsumexp(G, axis=2)).mean(1) + np.log(K - 1)

def mean_se(a):
    return a.mean(), a.std(ddof=1) / np.sqrt(len(a))

print("Data: 8x8 Gaussian-field patches x, second view y = x + sigma * noise")
print("   I   sigma   I(eigen) - I(joint)  max|cov(x) - C|  mean log p(y|x)/p(y)    z")
pools = {}
for seed, I in enumerate(LEVELS):
    sigma = np.exp(brentq(lambda t: true_mi(np.exp(t)) - I, -12, 5, xtol=1e-14))
    rng = np.random.default_rng(seed)
    x = rng.standard_normal((M, d)) @ L.T
    y = x + sigma * rng.standard_normal((M, d))
    ratio = -((y - x) ** 2).sum(1) / (2 * sigma**2) - 0.5 * d * np.log(2 * np.pi * sigma**2) - log_p_y(y, sigma)
    m, se = mean_se(ratio)
    pools[I] = (sigma, x, y)
    print(f"{I:4.0f}  {sigma:6.3f}  {true_mi(sigma) - mi_from_joint(sigma):+19.1e}  "
          f"{np.abs(np.cov(x.T, bias=True) - C).max():15.4f}  {m:20.4f}  {(m - I) / se:+5.2f}")

print("\nOptimal critic f = log p(y|x); batches of K pairs cut from each level's 65,536 pairs")
print("   I     K   log K   InfoNCE      SE   leave-one-out      SE")
nce, up, worst_cap, z_nce, z_up, crit_x, margin_y = {}, {}, -np.inf, [], [], 0.0, np.inf
for I in LEVELS:
    sigma, x, y = pools[I]
    for K in KS:
        B = M // K
        X, Y = x.reshape(B, K, d), y.reshape(B, K, d)
        F = log_cond(Y, X, sigma)
        R = F - log_p_y(y, sigma).reshape(B, K)[:, :, None]      # density-ratio critic log p(y|x)/p(y)
        b_nce, b_up = infonce(F), leave_one_out(F)
        nce[I, K], up[I, K] = mean_se(b_nce), mean_se(b_up)
        worst_cap = max(worst_cap, (b_nce - np.log(K)).max())
        z_nce.append((nce[I, K][0] - I) / nce[I, K][1])
        z_up.append((up[I, K][0] - I) / up[I, K][1])
        crit_x = max(crit_x, np.abs(infonce(R) - b_nce).max())
        margin_y = min(margin_y, infonce(R, "y").mean() - infonce(F, "y").mean())
        print(f"{I:4.0f}  {K:4d}  {np.log(K):6.3f}  {nce[I, K][0]:8.3f}  {nce[I, K][1]:6.4f}  "
              f"{up[I, K][0]:14.3f}  {up[I, K][1]:6.4f}")
gap_falls = all(np.all(np.diff([I - nce[I, K][0] for K in KS]) < 0) for I in LEVELS)
up_falls = all(np.all(np.diff([up[I, K][0] - I for K in KS]) < 0) for I in LEVELS)
print(f"largest batch value of InfoNCE - log K: {worst_cap:+.1e}")
print(f"largest (InfoNCE - I)/SE: {max(z_nce):+.2f}; smallest (leave-one-out - I)/SE: {min(z_up):+.2f}")
print(f"I - InfoNCE falls as K grows, at every I: {gap_falls}; leave-one-out - I falls too: {up_falls}")
print(f"log K - InfoNCE at I = 12: " + ", ".join(f"{np.log(K) - nce[12.0, K][0]:.4f}" for K in KS))
print(f"normalized over x: log p(y|x) and log p(y|x)/p(y) differ by at most {crit_x:.1e}")
print(f"normalized over y: log p(y|x)/p(y) beats log p(y|x) in every cell by at least {margin_y:.3f}")

print("\nCosine critic f = cos(x, y) / tau at I = 6, K = 256")
sigma, x, y = pools[6.0]
K = 256
B = M // K
X, Y = x.reshape(B, K, d), y.reshape(B, K, d)
cos = (Y / np.linalg.norm(Y, axis=-1, keepdims=True)) @ (X / np.linalg.norm(X, axis=-1, keepdims=True)).transpose(0, 2, 1)
cap = lambda tau: np.log(K) - np.log1p((K - 1) * np.exp(-2 / tau))   # largest value any cosine critic can give
print("  tau   InfoNCE      SE     cap")
worst = -np.inf
for tau in (0.02, 0.05, 0.1, 0.2, 0.5, 1.0):
    b = infonce(cos / tau)
    worst = max(worst, (b - cap(tau)).max())
    print(f"{tau:5.2f}  {b.mean():8.3f}  {mean_se(b)[1]:6.4f}  {cap(tau):6.3f}")
best = minimize_scalar(lambda beta: -infonce(beta * cos).mean(), bounds=(0.5, 500), method="bounded",
                       options={"xatol": 1e-6})
print(f"largest batch value minus cap: {worst:+.2f}")
print(f"best tau {1 / best.x:.4f}: InfoNCE {-best.fun:.3f}; optimal critic at the same K: {nce[6.0, K][0]:.3f}")

Data: 8x8 Gaussian-field patches x, second view y = x + sigma * noise
   I   sigma   I(eigen) - I(joint)  max|cov(x) - C|  mean log p(y|x)/p(y)    z
   1   5.415             +2.7e-12           0.0140                1.0043  +0.81
   3   2.882             +6.0e-12           0.0128                3.0061  +0.71
   6   1.824             +1.3e-11           0.0154                5.9922  -0.68
  12   1.059             +3.6e-11           0.0141               12.0094  +0.65

Optimal critic f = log p(y|x); batches of K pairs cut from each level's 65,536 pairs
   I     K   log K   InfoNCE      SE   leave-one-out      SE
   1     4   1.386     0.591  0.0032           1.517  0.0072
   1    16   2.773     0.863  0.0045           1.146  0.0060
   1    64   4.159     0.961  0.0052           1.044  0.0057
   1   256   5.545     0.992  0.0053           1.015  0.0055
   1  1024   6.931     1.001  0.0052           1.007  0.0053
   3     4   1.386     1.144  0.0025           5.456  0.0159
   3    16   2.773     2.007  0.0044           3.958  0.0114
   3    64   4.159     2.540  0.0060           3.397  0.0096
   3   256   5.545     2.817  0.0077           3.166  0.0097
   3  1024   6.931     2.937  0.0081           3.066  0.0088
   6     4   1.386     1.351  0.0011          13.681  0.0323
   6    16   2.773     2.632  0.0022           9.710  0.0212
   6    64   4.159     3.757  0.0039           7.982  0.0173
   6   256   5.545     4.640  0.0063           7.051  0.0154
   6  1024   6.931     5.248  0.0075           6.536  0.0120
  12     4   1.386     1.386  0.0001          40.803  0.0845
  12    16   2.773     2.771  0.0002          28.608  0.0522
  12    64   4.159     4.152  0.0005          22.706  0.0382
  12   256   5.545     5.517  0.0010          19.113  0.0305
  12  1024   6.931     6.847  0.0020          16.763  0.0252
largest batch value of InfoNCE - log K: +0.0e+00
largest (InfoNCE - I)/SE: +0.28; smallest (leave-one-out - I)/SE: +1.41
I - InfoNCE falls as K grows, at every I: True; leave-one-out - I falls too: True
log K - InfoNCE at I = 12: 0.0004, 0.0015, 0.0073, 0.0281, 0.0844
normalized over x: log p(y|x) and log p(y|x)/p(y) differ by at most 1.3e-14
normalized over y: log p(y|x)/p(y) beats log p(y|x) in every cell by at least 0.016

Cosine critic f = cos(x, y) / tau at I = 6, K = 256
  tau   InfoNCE      SE     cap
 0.02     4.386  0.0092   5.545
 0.05     4.259  0.0050   5.545
 0.10     3.188  0.0034   5.545
 0.20     1.941  0.0020   5.534
 0.50     0.868  0.0009   3.810
 1.00     0.450  0.0005   1.975
largest batch value minus cap: -0.82
best tau 0.0296: InfoNCE 4.502; optimal critic at the same K: 4.640

The data checks pass at all four levels:

The closed-form $I$ agrees with the value from the joint covariance to within $4 \times 10^{-11}$ .
The patches have the planned covariance: the largest entry error is at most 0.0154.
The mean log density ratio, whose expectation is $I$ , lies within one SE of $I$ ( $z$ from $-0.68$ to $+0.81$ ). The critic and the sampler are therefore correct.

The bound is never above $I$ or $\log K$ . No batch exceeds $\log K$ : the largest batch value of InfoNCE minus $\log K$ , over all 20 cells, is 0 to the printed precision. The mean InfoNCE stays below $I$ in 19 of the 20 cells. The exception is $I = 1$ , $K = 1024$ : the estimate, 1.001, is above $I = 1$ by less than one SE (0.0052), and the largest $(\text{InfoNCE} - I)/\text{SE}$ over the 20 cells is $+0.28$ . The bound holds for the expected value, not for each estimate. At $\log K = 6.93$ , almost seven times $I$ , the bound is tight to within the Monte Carlo noise.

The gap shrinks with $K$ at every level. Read down each level of the table:

$I = 1$ : InfoNCE rises from 0.591 to 0.863, 0.961, 0.992 and 1.001.
$I = 3$ : it rises from 1.144 to 2.007, 2.540, 2.817 and 2.937.
$I = 6$ : it rises from 1.351 to 2.632, 3.757, 4.640 and 5.248.
$I = 12$ : it rises from 1.386 to 2.771, 4.152, 5.517 and 6.847.

The printed check confirms that $I - \text{InfoNCE}$ falls at every step of $K$ at every level. Where $\log K$ is well above $I$ , the bound is nearly exact; at $I = 3$ and $I = 6$ even $K = 1{,}024$ leaves gaps of 0.063 and 0.752 nats.

Saturation. At $I = 12$ , above $\log K$ for every $K$ here, the bound sits just under $\log K$ : from 0.0004 nats below it at $K = 4$ to 0.0844 below at $K = 1{,}024$ . It is 5.15 to 10.61 nats below the truth. The loss $\log K - I_{\text{NCE}}$ is then almost zero: the critic picks the right partner nearly every time. Yet the value it reports is the logarithm of the batch size, not the 12 nats the views share. At $I = 6$ the bound is near $\log K$ for $K = 4$ and 16 (1.351 against 1.386, 2.632 against 2.773) and leaves it as $K$ grows past the information.

The sandwich. The leave-one-out bound is above $I$ in every cell. The smallest $(\text{upper} - I)/\text{SE}$ is $+1.41$ , and its excess falls with $K$ at every level, as printed. It closes in quickly when $I$ is small: 1.517 to 1.007 at $I = 1$ . When $I$ is large it is as poor from above as InfoNCE is from below: 40.803 at $K = 4$ and still 16.763 at $K = 1{,}024$ for $I = 12$ . This matches Poole et al.’s report of its bias when the batch is too small.

Which side is normalized. When the softmax runs over the $x_j$ , the critics $\log p(y \mid x)$ and $\log p(y \mid x) - \log p(y)$ give the same value in every batch, to $1.3 \times 10^{-14}$ . The $c(y)$ term cancels, as Poole et al.’s optimal critic says. When the softmax runs over the $y_j$ instead, $c(y)$ no longer cancels. The density-ratio critic then beats $\log p(y \mid x)$ in every cell, by at least 0.016 nats.

Temperature. With the cosine critic at $I = 6$ and $K = 256$ :

Over the six printed temperatures, the bound rises as $\tau$ falls: from 0.450 at $\tau = 1$ through 0.868, 1.941, 3.188 and 4.259 to 4.386 at $\tau = 0.02$ , which lies just past the peak.
The best temperature, 0.0296, gives 4.502. That is still below the optimal critic’s 4.640 at the same $K$ , because a cosine of the raw patches is not the log density ratio.
No batch comes within 0.82 nats of the cap.

The cap matters at large $\tau$ . At $\tau = 1$ , no encoder at all could push this bound above 1.975 nats with 256 candidates, whatever the views share. The best temperature here is below the 0.07–0.5 that the contrastive learning article reports from the papers. In our reading, the cause is that the noise keeps the cosine of a raw patch with its noisy partner well below 1, so the logit gaps need a small $\tau$ to become large. The best scale depends on how far apart the similarities are, not on a universal value.

Common Misconceptions

“A lower loss means more mutual information was learned.” The loss can only fall to 0, so $I_{\text{NCE}} \le \log K$ . Once the critic picks the positive reliably, the loss stops moving even if the representation keeps changing. A number near $\log K$ says the batch is too small to measure the information, not that the information is $\log K$ .
“InfoNCE estimates mutual information.” It bounds it from below, with a bias that grows with $I$ . No distribution-free high-confidence lower bound can exceed $O(\ln N)$ (McAllester & Stratos, 2020). Reported values are best read as bounds at a given $K$ and critic.
“Maximizing the bound is why contrastive features work.” Wang and Isola (2020), citing earlier work, note that maximizing a tighter bound often leads to worse downstream performance. The views, the augmentations and the critic’s form decide what the encoder keeps.
“InfoNCE is NCE.” NCE is a binary logistic regression of data against samples from a known noise density, used to fit unnormalized models (Gutmann & Hyvärinen, 2010). InfoNCE is a $K$ -way softmax, and its negatives are other samples from the data.
“The temperature is a free scale.” For a bounded similarity it caps the attainable bound: a large $\tau$ forbids confident decisions (derivation above). The contrastive learning article shows how $\tau$ also decides which negatives receive the gradient.

Where It Is Used

Self-supervised image features: CPC and CPC v2 on patches (Hénaff et al., 2020); SimCLR’s NT-Xent and MoCo, with batch or queue negatives (contrastive learning, self-supervised learning).
Image–text models: the symmetric loss of CLIP and the image–text contrastive objective of vision–language pretraining, which averages the two normalization directions.
Metric learning: the many-negative softmax beside the margin losses of metric learning and the triplet loss.
Estimating mutual information: as a low-variance, high-bias lower bound, paired with the leave-one-out upper bound when the conditional density is known (Poole et al., 2019).