Imagen

Imagen is a text-to-image system that Google Research’s Brain team presented at NeurIPS 2022 (Saharia et al., 2022). A frozen T5-XXL language encoder turns the prompt into a sequence of embeddings, and a cascade of three diffusion models turns them into an image: a $64 \times 64$ base model, then two text-conditioned super-resolution models to $256 \times 256$ and $1024 \times 1024$. All three use classifier-free guidance. The paper’s headline finding was that a language model trained on text alone is a strong text encoder for image generation, and that enlarging it helps more than enlarging the image model. It also introduced dynamic thresholding, a sampling change that allows large guidance weights, the Efficient U-Net for the super-resolution stages, and the DrawBench prompt set. The model and its code were not released.

Several published articles already describe parts of Imagen: Text-to-Image Generation places it in the history of the task and describes DrawBench and the COCO protocol, Text Conditioning covers its encoder and conditioning ablations, and Classifier-Free Guidance covers its guidance convention and thresholding. This article describes the system as a whole and links those accounts. Section, figure and appendix numbers below refer to the NeurIPS paper and its supplementary material, which carries the appendices.

Problem Addressed

By early 2022, text-to-image diffusion models conditioned on text encoders trained on image–text pairs: GLIDE trained its text transformer from scratch, and DALL·E 2 used CLIP’s text encoder (§2.1). Imagen’s authors asked whether a large language model trained on text-only corpora could serve instead. Such corpora are much larger than paired image–text data, and such models are generally much larger than the text encoders of image–text models (§2.1).

Two further problems came with the pixel-space cascade the authors chose. Guidance improves image–text alignment as its weight grows, but at large weights the images became highly saturated and unnatural (§2.3). And each super-resolution model in a cascade is trained on downsampled real images but, at sampling time, receives the imperfect output of the stage below.

Architecture

The paper calls the design simple: a text encoder and a cascade of conditional diffusion models that map the embeddings to images of increasing resolution (§2, Fig. A.4). At $1024 \times 1024$ the cascade’s stages and parameter counts are:

Stage	Input	Output	Parameters (§4.1)	Architecture
Text encoder	prompt	embedding sequence	4.6B (frozen; §4.4)	T5-XXL encoder
Base	noise, text	$64 \times 64$	2B	U-Net (§2.5, App. F.1)
Super-resolution 1	$64 \times 64$ image, text	$256 \times 256$	600M	Efficient U-Net (App. F.2)
Super-resolution 2	$256 \times 256$ image, text	$1024 \times 1024$	400M	Efficient U-Net, no self-attention (App. F.3)

Frozen text encoder

Imagen uses the encoder half of T5-XXL (Raffel et al., 2020) and keeps its weights fixed (§2.1, App. D.1). Freezing lets the embeddings be computed offline, so the encoder costs almost nothing in compute or memory during training (§2.1). The paper also trained models on frozen BERT and CLIP encoders; the Text Conditioning article reports that comparison.

Base model

The $64 \times 64$ model is a U-Net adapted from earlier diffusion work. It is conditioned on the text in two ways: an attention-pooled embedding vector is added to the diffusion timestep embedding, as class-conditional diffusion models add a class embedding, and the full sequence of token embeddings is attended to at several resolutions (§2.5). The configuration in App. F.1 lists attention and text cross-attention at resolutions 32, 16 and 8, channel multipliers $[1, 2, 3, 4]$ on an embedding width of 512, three BigGAN-style residual blocks per resolution and scale–shift normalization. Cross-attention is implemented by concatenating the text embeddings to the keys and values of each self-attention layer (App. D.3.1). Layer normalization of the text embeddings in the attention and pooling layers helped considerably (§2.5).

The paper names two sources for this U-Net: §2.5 says it adapts the U-Net of Nichol and Dhariwal’s improved DDPM, and App. F.1 says it adapts the architecture of Dhariwal and Nichol’s “Diffusion Models Beat GANs”.

Super-resolution models and the Efficient U-Net

The $64 \to 256$ model starts from the U-Net of Nichol and Dhariwal’s improved DDPM and of Palette, the authors’ image-to-image diffusion model, and is rebuilt as what they call the Efficient U-Net (§2.5, App. B.1). It makes three changes:

• Parameters move to low resolutions. More residual blocks are placed at the coarse levels, where the channel count is high, so capacity grows without the memory and compute that high-resolution blocks cost. The $64 \to 256$ configuration has 2, 4, 8 and 8 residual blocks at 128, 256, 512 and 1,024 channels, each block downsampling by 2, with self-attention and text cross-attention only in the last (App. F.2).
• Skip connections are scaled by $1/\sqrt{2}$, as in “Image Super-Resolution via Iterative Refinement” (SR3; Saharia et al., 2022) and score-based SDE models, which the authors found to speed convergence significantly with so many low-resolution blocks.
• The sampling order is reversed. A typical U-Net downsamples after the convolutions of a down block and upsamples before those of an up block; the Efficient U-Net does the opposite, which speeds the forward pass with no loss of quality they could find.

The authors report that the Efficient U-Net uses less memory, converges faster (Fig. A.14, FID-2K against TPU training days) and samples two to three times faster than the U-Net it replaces (§2.5, App. D.3.2). Figures A.27 to A.30 give its residual block, down block, up block and full layout.

The $256 \to 1024$ model uses the same configuration without self-attention layers, which lets it train on $64 \times 64 \to 256 \times 256$ crops of $1024 \times 1024$ images; at inference it receives the whole $256 \times 256$ image (§2.5). It keeps text cross-attention, which the authors found critical, as explicit cross-attention layers (App. D.3.1). Both super-resolution models attend to the text, so the prompt can still change the image after the base stage.

Noise conditioning augmentation

The super-resolution stages use noise conditioning augmentation from Cascaded Diffusion Models (Ho, Saharia et al., 2022, JMLR). During training, an augmentation level $a \in [0, 1]$ is drawn at random, the low-resolution input is corrupted with variance-preserving Gaussian noise of that level, resembling the diffusion forward process, and the model is told $a$. At sampling time the generated low-resolution image is corrupted at a chosen level, swept to find the best sample quality (§2.4; pseudocode in Fig. A.32).

The idea comes from Cascaded Diffusion Models, which traced the benefit to a train–test mismatch. A super-resolution model trained on downsampled real images sees out-of-distribution details in generated ones, and the errors compound; enough Gaussian augmentation keeps the model from upsampling those erroneous details (Ho et al., 2022, §4.3). Evidence from that paper: on ImageNet at $32 \to 64$, sample quality first improved and then degraded as the augmentation level grew, but degraded monotonically when the super-resolution model was conditioned on real low-resolution images, where there is no mismatch. In a small $16 \to 64$ pipeline without augmentation, FID against the training set was 6.02, worse than a non-cascaded $64 \times 64$ model’s 2.35; with strong augmentation it reached 2.13 (Table 2b). That paper also used blurring as the augmentation at $128$ and $256$ pixels (§4.4); Imagen uses Gaussian noise (§2.4).

Training

Each of the three models was trained with batch size 2,048 for 2.5 million steps: the base model on 256 TPU-v4 chips with Adafactor, which matched Adam’s performance with a much smaller memory footprint in initial comparisons, and each super-resolution model on 128 chips with Adam, because Adafactor hurt their quality (§4.1). The authors saw no over-fitting and believed more training might help. For guidance, the text embeddings were zeroed with 10% probability in all three models (§4.1).

The data combined internal datasets of about 460 million image–text pairs with the public LAION-400M set of about 400 million (§4.1); 400 million examples came from FIT400M, a cleaned version of the Alt-Text dataset, filtered for noise and for content such as pornographic imagery and toxic language (§6).

The objective is the standard denoising objective: with $z_t = \alpha_t x + \sigma_t \epsilon$, a network is trained so that $\hat x_\theta(z_t, \lambda_t, c) \approx x$, where $\lambda_t = \log(\alpha_t^2 / \sigma_t^2)$ is the log signal-to-noise ratio and $c$ the conditioning (App. A, Eq. 4). Imagen uses the $\epsilon$-prediction parameterization,

$$\hat x_\theta(z_t, \lambda_t, c) = \frac{z_t - \sigma_t\, \epsilon_\theta(z_t, \lambda_t, c)}{\alpha_t},$$

with a squared error on $\epsilon_\theta$ and continuous time $t \sim U(0, 1)$. The base and first super-resolution models use a cosine noise schedule; App. F.3 gives the $1024$ model “the 1000 step linear noise schedule” with endpoints $10^{-4}$ and $0.02$, while also stating continuous-time training. Learning rates are $10^{-4}$ with 10,000 warm-up steps (App. F).

Inference

Sampling runs the three stages in turn, each from pure noise, with the ancestral sampler or, for some models, DDIM (App. A). Each stage applies guidance in Imagen’s convention (Eq. 2),

$$\tilde\epsilon_\theta(z_t, c) = w\, \epsilon_\theta(z_t, c) + (1 - w)\, \epsilon_\theta(z_t),$$

in which $w = 1$ means no guidance; the Classifier-Free Guidance article relates this to the other conventions. The authors describe Imagen as depending critically on guidance and use relatively large weights in all three models (§2.2, App. D.2); the COCO results used $w = 1.35$ for the base model and $8.0$ for super-resolution (Table 1). For the super-resolution stages the augmentation level is a further sampling choice: on the $64 \to 256$ model, level 0 gave the best FID at every guidance weight, and FID improved with guidance up to weights of about 7 to 10; larger levels cost a little FID but widened the range of CLIP scores, and the authors generally used levels between 0.1 and 0.3 for their best samples (App. D.2, Fig. A.11a). With a large level and large weights, the super-resolution models can restyle a $64 \times 64$ sample when given a different prompt, such as “An oil painting of …” (Fig. A.12).

Thresholding the predicted image

Guidance extrapolates past the conditional prediction, and the paper found that large weights push the predicted clean image $\hat x_0^t$ outside $[-1, 1]$, the range of the training data. Because the sampler feeds its own output back, this train–test mismatch produced unnatural images and sometimes divergence (§2.3). Static thresholding clips $\hat x_0^t$ to $[-1, 1]$ at each step; the authors found it essential with large weights, since it prevents blank images, but still over-saturating as the weight grows. Dynamic thresholding sets $s$ to a percentile of $|\hat x_0^t|$ and pushes saturated pixels inward (§2.3). The thresholded prediction then replaces $\hat x_0^t$ in the sampler step (Fig. A.31).

The paper states dynamic thresholding in two slightly different forms. The text of §2.3 thresholds only “if $s > 1$”:

$$\hat x_0 \leftarrow \begin{cases} \operatorname{clip}(\hat x_0, -s, s) / s & s > 1, \\ \hat x_0 & s \le 1, \end{cases}$$

while the reference pseudocode (Fig. A.31b) sets $s \leftarrow \max(s, 1)$ and always clips, so that for $s \le 1$ it reduces to static thresholding; it is written s = jnp.max(s, 1.0), which we read as an element-wise maximum. The percentile used for the samples of Fig. A.10 is $p = 99.5$ (its caption), computed per image over all pixels and channels (Fig. A.31b). On the $64 \times 64$ model, dynamic thresholding gave significantly better CLIP scores and comparable or better FID-10K than static thresholding over guidance weights from 1 to 10 (App. D.2, Fig. A.8).

Example: thresholding with an exact denoiser

The Classifier-Free Guidance article’s toy example measures what guidance does to a mixture of shape images, and the Stable Diffusion article’s toy prints the largest decoded pixel value against guidance scale and step count. Neither thresholds $\hat x_0$. The script below tests §2.3’s mechanism with an exact denoiser instead of a trained network, so the outcome tests our analogue, not Imagen’s networks.

The data are $16 \times 16$ images for two captions, a ring (annuli of outer and inner radius 5 and 3, or 4 and 2) and a diamond ($|dy| + |dx| \le 5$ or $3.5$), each at three centres: twelve templates. Pixels are $-0.5$ in the background and $0.6$ inside the shape, with anti-aliased edges, so no data pixel is near $\pm 1$. Each caption is an equal-weight Gaussian mixture of its six templates with per-pixel standard deviation $\tau = 0.03$, and the unconditional model is the mixture of all twelve. The forward process is variance preserving with $\alpha_t = \cos(\pi t / 2)$ and $\sigma_t = \sin(\pi t / 2)$ (our choice of cosine schedule), and every noisy marginal is again a mixture, so

$$\hat x_0(z_t, c) = \sum_k r_k(z_t) \left[\mu_k + \frac{\alpha_t \tau^2}{\alpha_t^2 \tau^2 + \sigma_t^2} (z_t - \alpha_t \mu_k)\right]$$

is exact, with $r_k$ the posterior weights of the components $\mu_k$ of caption $c$. Guidance is applied to $\hat x_0$ as $\hat x_0(\varnothing) + w\,(\hat x_0(c) - \hat x_0(\varnothing))$, which equals the $\hat x_0$ of Eq. 2 because the map from $\epsilon$ to $\hat x_0$ is affine. DDIM (App. A, Eq. 5) runs 64 uniform steps from $t = 1$ to $0$, using the treated $\hat x_0$ in each update, and the final sample is the treated $\hat x_0$ of the last step. The four treatments are none, static, the pseudocode’s dynamic form (“dyn-code”) and the text’s (“dyn-text”), both with $p = 99.5$; the guidance weights are 1, 3, 6 and 10; and 200 samples per caption share the same initial noise across weights and treatments.

The script first checks the data and two components: $\hat x_0$ against Tweedie’s formula $(z_t + \sigma_t^2 \nabla \log p_t(z_t)) / \alpha_t$ with a finite-difference score, and the $\hat x_0$ form of guidance against Eq. 2. It then prints, for each weight and treatment, the number of out-of-range $\hat x_0$ entries at the first step beside the number derived from the templates, the number pooled over all 64 steps, the number of final pixels with $|x| \ge 0.95$ (“saturated”), the mean RMS distance from each sample to the nearest template of its caption, the number of samples whose nearest template has the right caption, and, as a property check, the number of thresholded entries still above 1.

Before the run we planned five outcomes (hypotheses H1 to H5) and fixed them with the checks below:

• H1 At $t = 1$, $\alpha_1 = \cos(\pi/2)$ is zero up to rounding, so the posterior equals the prior and the guided $\hat x_0$ is $m + w\,(m_c - m)$ for every sample, with $m$ the mean of all templates and $m_c$ that of caption $c$. The first-step count therefore follows from the templates alone: 0, 1,200, 12,800 and 25,600 of 102,400 entries for $w = 1, 3, 6, 10$. A planning script computed these counts before the plan was written, so H1 is an illustration of the derivation rather than a test.
• H2 Without thresholding, the pooled out-of-range count rises strictly from $w = 3$ to $6$ to $10$.
• H3 At $w = 6$ and $10$, static thresholding leaves more saturated pixels than either dynamic form, and some at $w = 10$.
• H4 At $w = 6$ and $10$, both dynamic forms end nearer the modes than static thresholding (a tie at the printed precision counts as a failure).
• H5 At $w = 6$ and $10$, no thresholding ends farther from the modes than static thresholding.

Each outcome is judged on seeds 0, 1 and 2, the seed being the script’s argument, and holds only if it holds on all three. Class correctness has no hypothesis.

# Imagen example: static against dynamic thresholding of x0 under strong guidance,
# with an exact Gaussian-mixture denoiser on 16x16 ring and diamond images.
import os
os.environ["OMP_NUM_THREADS"] = "1"
os.environ["OPENBLAS_NUM_THREADS"] = "1"
os.environ["MKL_NUM_THREADS"] = "1"
import sys
import numpy as np

H = W = 16
D = H * W
TAU = 0.03             # per-pixel std of each mixture component
BG, FG = -0.5, 0.6     # background and full-coverage foreground values
STEPS = 64
N_PER = 200            # samples per caption
WS = [1, 3, 6, 10]
P = 99.5
SAT = 0.95

def coverage(inside, cy, cx, ss=4):
    """Anti-aliased coverage of a shape on the 16x16 grid by ss x ss supersampling."""
    o = (np.arange(ss) + 0.5) / ss
    yy = (np.arange(H)[:, None] + o[None, :]).ravel()
    xx = (np.arange(W)[:, None] + o[None, :]).ravel()
    Y, X = np.meshgrid(yy, xx, indexing="ij")
    m = inside(Y - cy, X - cx).astype(float)
    return m.reshape(H, ss, W, ss).mean(axis=(1, 3))

def ring(r_out, r_in):
    return lambda y, x: (np.hypot(y, x) <= r_out) & (np.hypot(y, x) >= r_in)

def diamond(r):
    return lambda y, x: (np.abs(y) + np.abs(x)) <= r

centres = [(8.0, 8.0), (6.5, 9.5), (9.5, 6.5)]
temps, cap = [], []
for (cy, cx) in centres:
    for (ro, ri) in [(5.0, 3.0), (4.0, 2.0)]:
        temps.append(BG + (FG - BG) * coverage(ring(ro, ri), cy, cx)); cap.append(0)
for (cy, cx) in centres:
    for r in [5.0, 3.5]:
        temps.append(BG + (FG - BG) * coverage(diamond(r), cy, cx)); cap.append(1)
MU = np.array([t.ravel() for t in temps])      # (12, 256) component means
CAP = np.array(cap)                             # caption of each component
K = len(MU)

def sched(t):
    return np.cos(np.pi * t / 2), np.sin(np.pi * t / 2)

def x0_hat(z, t, comps):
    """Exact E[x | z_t] under the equal-weight mixture of the given components."""
    a, s = sched(t)
    mu = MU[comps]
    v = a * a * TAU ** 2 + s * s
    d2 = ((z[:, None, :] - a * mu[None]) ** 2).sum(-1)
    L = -d2 / (2 * v)
    r = np.exp(L - L.max(1, keepdims=True)); r /= r.sum(1, keepdims=True)
    cond_mean = mu[None] + (a * TAU ** 2 / v) * (z[:, None, :] - a * mu[None])
    return (r[:, :, None] * cond_mean).sum(1)

def logp(z, t, comps):
    a, s = sched(t)
    mu = MU[comps]
    v = a * a * TAU ** 2 + s * s
    d2 = ((z[:, None, :] - a * mu[None]) ** 2).sum(-1)
    L = -d2 / (2 * v) - D / 2 * np.log(2 * np.pi * v) - np.log(len(comps))
    m = L.max(1, keepdims=True)
    return (m + np.log(np.exp(L - m).sum(1, keepdims=True)))[:, 0]

ALL = np.arange(K)
CC = [np.where(CAP == c)[0] for c in (0, 1)]

def guided_x0(z, t, c, w):
    xu = x0_hat(z, t, ALL)
    xc = np.empty_like(z)
    for k in (0, 1):
        idx = c == k
        if idx.any():
            xc[idx] = x0_hat(z[idx], t, CC[k])
    return xu + w * (xc - xu), xc, xu

def treat(x, kind):
    if kind == "none":
        return x
    if kind == "static":
        return np.clip(x, -1.0, 1.0)
    s = np.percentile(np.abs(x), P, axis=1, keepdims=True)
    if kind == "dyn-code":                      # Fig. A.31: s = max(s, 1), always clip
        s = np.maximum(s, 1.0)
        return np.clip(x, -s, s) / s
    if kind == "dyn-text":                      # Sec. 2.3: only if s > 1
        big = s > 1.0
        sb = np.where(big, s, 1.0)
        return np.where(big, np.clip(x, -sb, sb) / sb, x)
    raise ValueError(kind)

def ddim(z1, c, w, kind):
    ts = np.linspace(1.0, 0.0, STEPS + 1)
    z = z1.copy()
    oor = 0
    first_oor = None
    viol = 0
    for i in range(STEPS):
        t, s_ = ts[i], ts[i + 1]
        xg, _, _ = guided_x0(z, t, c, w)
        n_out = int((np.abs(xg) > 1.0).sum())
        oor += n_out
        if i == 0:
            first_oor = n_out
        x0 = treat(xg, kind)
        if kind in ("static", "dyn-code"):
            viol += int((np.abs(x0) > 1.0 + 1e-12).sum())
        at, st = sched(t)
        as_, ss_ = sched(s_)
        z = as_ * x0 + ss_ * (z - at * x0) / st
    return z, oor, first_oor, viol

def measures(x, c):
    d = np.sqrt(((x[:, None, :] - MU[None]) ** 2).mean(-1))       # RMS to each template
    dc = np.where(CAP[None] == c[:, None], d, np.inf).min(1)
    correct = int((CAP[d.argmin(1)] == c).sum())
    sat = int((np.abs(x) >= SAT).sum())
    return dc, correct, sat

seed = int(sys.argv[1]) if len(sys.argv) > 1 else 0

print("== data checks")
print(f"templates: {K} ({(CAP == 0).sum()} rings, {(CAP == 1).sum()} diamonds), "
      f"min {MU.min():.4f}, max {MU.max():.4f}")
dmin = min(np.sqrt(((MU[i] - MU[j]) ** 2).mean()) for i in range(K) for j in range(K)
           if CAP[i] != CAP[j])
print(f"min RMS distance between templates of different captions: {dmin:.4f}")
rng = np.random.default_rng(123)
comp = rng.integers(0, K, 20000)
xd = MU[comp] + TAU * rng.standard_normal((20000, D))
dc, corr, satd = measures(xd, CAP[comp])
print(f"20000 data draws: pixels outside [-1, 1] {int((np.abs(xd) > 1).sum())}, "
      f"pixels |x| >= {SAT} {satd}, caption of nearest template correct {corr}, "
      f"mean RMS to nearest own-caption template {dc.mean():.5f}")

print("== component checks")
rng = np.random.default_rng(7)
for t in (0.3, 0.7):
    a, s = sched(t)
    z = a * MU[rng.integers(0, K, 3)] + s * rng.standard_normal((3, D))
    h = 1e-5
    g = np.zeros_like(z)
    for j in range(D):
        e = np.zeros(D); e[j] = h
        g[:, j] = (logp(z + e, t, ALL) - logp(z - e, t, ALL)) / (2 * h)
    tw = (z + s * s * g) / a
    print(f"t={t}: max |x0_hat - Tweedie(finite-difference score)| = "
          f"{np.abs(x0_hat(z, t, ALL) - tw).max():.2e}")
t = 0.5
a, s = sched(t)
z = a * MU[rng.integers(0, K, 4)] + s * rng.standard_normal((4, D))
cc = np.array([0, 1, 0, 1])
xg, xc, xu = guided_x0(z, t, cc, 6.0)
ec, eu = (z - a * xc) / s, (z - a * xu) / s
xe = (z - s * (6.0 * ec + (1 - 6.0) * eu)) / a
print(f"t=0.5, w=6: max |x-form guided x0 - Eq. 2 eps-form| = {np.abs(xg - xe).max():.2e}")

c = np.repeat([0, 1], N_PER)
m_all = MU.mean(0)
z1 = np.random.default_rng(seed).standard_normal((2 * N_PER, D))
NPIX = 2 * N_PER * D
print(f"== DDIM, {STEPS} steps, seed {seed}, {2 * N_PER} samples ({N_PER} per caption), "
      f"{NPIX} final pixels")
print(f"out-of-range: x0 entries with |x0| > 1 before treatment, pooled over {STEPS} steps "
      f"({STEPS * NPIX} entries); saturated: final pixels with |x| >= {SAT}")
print(" w  x0 treatment  step-1 out (derived)  out-of-range  saturated  RMS to mode  "
      "correct  >1 after")
for w in WS:
    derived = sum(N_PER * int((np.abs(m_all + w * (MU[CC[k]].mean(0) - m_all)) > 1.0).sum())
                  for k in (0, 1))
    for kind in ("none", "static", "dyn-code", "dyn-text"):
        x, oor, f1, viol = ddim(z1, c, float(w), kind)
        dc, corr, sat = measures(x, c)
        vtxt = f"{viol:8d}" if kind in ("static", "dyn-code") else "       -"
        print(f"{w:2d}  {kind:12s} {f1:8d} ({derived:6d})    {oor:10d}     {sat:6d}"
              f"      {dc.mean():.5f}    {corr:3d}   {vtxt}")

Output (NumPy 2.3.5; about 6 seconds per seed on a laptop CPU shared with other jobs), with seed 0:

== data checks
templates: 12 (6 rings, 6 diamonds), min -0.5000, max 0.6000
min RMS distance between templates of different captions: 0.2527
20000 data draws: pixels outside [-1, 1] 0, pixels |x| >= 0.95 0, caption of nearest template correct 20000, mean RMS to nearest own-caption template 0.02995
== component checks
t=0.3: max |x0_hat - Tweedie(finite-difference score)| = 5.94e-10
t=0.7: max |x0_hat - Tweedie(finite-difference score)| = 8.67e-09
t=0.5, w=6: max |x-form guided x0 - Eq. 2 eps-form| = 3.55e-15
== DDIM, 64 steps, seed 0, 400 samples (200 per caption), 102400 final pixels
out-of-range: x0 entries with |x0| > 1 before treatment, pooled over 64 steps (6553600 entries); saturated: final pixels with |x| >= 0.95
 w  x0 treatment  step-1 out (derived)  out-of-range  saturated  RMS to mode  correct  >1 after
 1  none                0 (     0)             0          0      0.02146    400          -
 1  static              0 (     0)             0          0      0.02146    400          0
 1  dyn-code            0 (     0)             0          0      0.02146    400          0
 1  dyn-text            0 (     0)             0          0      0.02146    400          -
 3  none             1200 (  1200)         21837          0      0.02145    400          -
 3  static           1200 (  1200)         23495          0      0.02145    400          0
 3  dyn-code         1200 (  1200)         30225          0      0.02145    400          0
 3  dyn-text         1200 (  1200)         30224          0      0.02145    400          -
 6  none            12800 ( 12800)        108630          0      0.02153    400          -
 6  static          12800 ( 12800)        131231          0      0.02149    400          0
 6  dyn-code        12800 ( 12800)        178347          0      0.02150    400          0
 6  dyn-text        12800 ( 12800)        178347          0      0.02150    400          -
10  none            25600 ( 25600)        179495          0      0.02166    400          -
10  static          25600 ( 25600)        245265          0      0.02154    400          0
10  dyn-code        25600 ( 25600)        331151          0      0.02155    400          0
10  dyn-text        25600 ( 25600)        331151          0      0.02155    400          -

With seeds 1 and 2 (the script run with the argument 1, then 2); the data and component checks print the same lines as for seed 0 and are left out:

== DDIM, 64 steps, seed 1, 400 samples (200 per caption), 102400 final pixels
out-of-range: x0 entries with |x0| > 1 before treatment, pooled over 64 steps (6553600 entries); saturated: final pixels with |x| >= 0.95
 w  x0 treatment  step-1 out (derived)  out-of-range  saturated  RMS to mode  correct  >1 after
 1  none                0 (     0)             0          0      0.02138    400          -
 1  static              0 (     0)             0          0      0.02138    400          0
 1  dyn-code            0 (     0)             0          0      0.02138    400          0
 1  dyn-text            0 (     0)             0          0      0.02138    400          -
 3  none             1200 (  1200)         20166          0      0.02138    400          -
 3  static           1200 (  1200)         21775          0      0.02138    400          0
 3  dyn-code         1200 (  1200)         27950          0      0.02138    400          0
 3  dyn-text         1200 (  1200)         27948          0      0.02138    400          -
 6  none            12800 ( 12800)        106326          0      0.02146    400          -
 6  static          12800 ( 12800)        127565          0      0.02143    400          0
 6  dyn-code        12800 ( 12800)        171728          0      0.02142    400          0
 6  dyn-text        12800 ( 12800)        171728          0      0.02142    400          -
10  none            25600 ( 25600)        178208          0      0.02159    400          -
10  static          25600 ( 25600)        242294          0      0.02147    400          0
10  dyn-code        25600 ( 25600)        325199          0      0.02147    400          0
10  dyn-text        25600 ( 25600)        325199          0      0.02147    400          -
== DDIM, 64 steps, seed 2, 400 samples (200 per caption), 102400 final pixels
out-of-range: x0 entries with |x0| > 1 before treatment, pooled over 64 steps (6553600 entries); saturated: final pixels with |x| >= 0.95
 w  x0 treatment  step-1 out (derived)  out-of-range  saturated  RMS to mode  correct  >1 after
 1  none                0 (     0)             0          0      0.02144    400          -
 1  static              0 (     0)             0          0      0.02144    400          0
 1  dyn-code            0 (     0)             0          0      0.02144    400          0
 1  dyn-text            0 (     0)             0          0      0.02144    400          -
 3  none             1200 (  1200)         20088          0      0.02143    400          -
 3  static           1200 (  1200)         21800          0      0.02143    400          0
 3  dyn-code         1200 (  1200)         27788          0      0.02143    400          0
 3  dyn-text         1200 (  1200)         27786          0      0.02143    400          -
 6  none            12800 ( 12800)        105093          0      0.02150    400          -
 6  static          12800 ( 12800)        126292          0      0.02147    400          0
 6  dyn-code        12800 ( 12800)        169996          0      0.02146    400          0
 6  dyn-text        12800 ( 12800)        169996          0      0.02146    400          -
10  none            25600 ( 25600)        175479          0      0.02164    400          -
10  static          25600 ( 25600)        238619          0      0.02151    400          0
10  dyn-code        25600 ( 25600)        322057          0      0.02151    400          0
10  dyn-text        25600 ( 25600)        322057          0      0.02151    400          -

The checks hold: the templates lie in $[-0.5, 0.6]$, no data draw has a pixel outside $[-1, 1]$ or at $|x| \ge 0.95$, the exact $\hat x_0$ matches Tweedie’s formula to about $10^{-8}$, and the two guidance forms agree to rounding. At $w = 1$ the four treatments give identical rows, with no out-of-range entry, all 400 samples nearest a template of the right caption, and a mean distance of about 0.021, below the 0.030 of the data draws. No thresholded entry exceeds 1 under static or dyn-code thresholding.

H1 held: in every row the first-step count equals the derived count. Guidance at $w = 3$ already sends 1,200 entries out of range at the first step, a quarter of all entries at $w = 10$. H2 held: without thresholding, the pooled count rises from 21,837 to 108,630 to 179,495 for seed 0, and in the same order for seeds 1 and 2. Out-of-range predictions during sampling, the condition §2.3 describes, are therefore present.

H3 failed: no final pixel reaches $|x| \ge 0.95$ under any treatment, weight or seed, so static thresholding does not over-saturate here and cannot do so more than the dynamic forms; H3 failed on all three seeds. H4 failed: at $w = 6$, the dynamic forms end slightly nearer the modes than static thresholding for seeds 1 and 2 (0.02142 against 0.02143, 0.02146 against 0.02147) but slightly farther for seed 0 (0.02150 against 0.02149); at $w = 10$ they are farther for seed 0 (0.02155 against 0.02154) and tied at the printed precision for seeds 1 and 2 (0.02147, 0.02151), so H4 failed on all three seeds. H5 held: without thresholding the samples end farther from the modes than with static thresholding at both weights and all three seeds, but by only $3 \times 10^{-5}$ to $1.3 \times 10^{-4}$, small against $\tau = 0.03$. Every sample, at every weight and under every treatment, is nearest a template of its own caption.

The output also shows something no hypothesis addressed: at $w \ge 3$, the pooled count of out-of-range raw predictions is larger with static thresholding than without, and larger again with either dynamic form (seed 0, $w = 10$: 179,495, 245,265 and 331,151). The two dynamic forms differ only at $w = 3$, by one or two entries.

In this exact model, then, the out-of-range $\hat x_0$ of the intermediate steps does not carry into the final samples, which end at the same small distance from a template whatever the treatment. The paper’s claim concerns trained networks applied to their own out-of-range outputs, and this example neither confirms nor refutes it: it shows that guidance sends the predictions out of range, as a derivation predicts, and that with an exact denoiser the treatments change the final samples only slightly. After the planned trial run showed H3 and H4 failing, the editor decided to keep the example unchanged; the run shown repeats that trial run exactly. A fourth treatment, the data and component checks and the extra printed columns (first-step counts, property check) were added to the batch plan’s description of the example in our plan, before the trial run.

Performance

COCO. Imagen’s zero-shot FID-30K at $256 \times 256$ on the COCO validation set was 7.27, against 10.39 for DALL·E 2, 12.24 for GLIDE, 26.94 for LAFITE and 17.89 for DALL·E in the zero-shot column of Table 1, and below the best model trained on COCO, Make-A-Scene at 7.55. The Fréchet Inception Distance protocol and the differences between papers’ versions of it are covered in the Text-to-Image Generation article, which also notes that Table 2’s human preference rates (39.5% for photorealism against real COCO images, 43.9% on prompts without people) differ from the 39.2% and 43.6% in the text. Alignment ratings were on par with the reference images: 91.4 against 91.9, and 92.1 against 92.2 without people (Table 2). Each human study used 200 COCO image–caption pairs and kept raters who passed at least 80% of control questions, giving 73 ratings per image for quality and 51 for alignment (§3).

DrawBench. In side-by-side human comparisons on DrawBench’s 200 prompts in 11 categories, raters preferred Imagen over DALL·E 2, GLIDE, VQ-GAN+CLIP and Latent Diffusion for both image–text alignment and fidelity (§4.3, Fig. 3; the text says “each of the three models” while comparing four). By category, Imagen was preferred to DALL·E 2 in 7 of 11 categories for alignment and all 11 for fidelity, and to GLIDE in 8 of 11 and 10 of 11 (App. E). Each category was rated by 25 raters, comparing two sets of eight samples (App. C).

Ablations (§4.4, App. D), all on CLIP-score-against-FID-10K curves swept over guidance weights:

• Encoder size against U-Net size. Larger T5 encoders improved alignment and fidelity, and scaling the encoder helped more than scaling the base U-Net from 300M to 2B parameters (Figs. A.5, A.13b; the U-Net models trained for 400,000 steps). The Text Conditioning article gives the encoder comparison in detail.
• Thresholding. Dynamic thresholding was better than static thresholding, especially at large weights (Figs. 4c, A.8, A.9).
• Noise conditioning augmentation. Super-resolution models trained without it had generally worse CLIP and FID scores, and their scores varied much less with the guidance weight; the authors hypothesize that strong augmentation weakens the low-resolution signal and makes the model rely more on the text (App. D.2, Fig. A.11b).
• Text conditioning. Cross-attention over the token sequence clearly beat mean or attention pooling (App. D.3.1).
• Efficient U-Net against the U-Net on $64 \to 256$ super-resolution, as above (App. D.3.2).

Strengths

• Language understanding from text-only models. A frozen language model gave alignment that human raters preferred over a CLIP encoder on all 11 DrawBench categories, and enlarging the frozen encoder improved both alignment and fidelity (§4.4).
• Large guidance weights. Dynamic thresholding made weights usable that otherwise gave saturated images (§2.3, §6).
• Cascaded resolution. The largest model works at $64 \times 64$ (§4.1); Cascaded Diffusion Models argues that cascades put most capacity at the low resolutions that matter most for quality and let each model be trained independently (Ho et al., 2022, §3), and augmentation makes the super-resolution stages robust to artifacts from the stage below (§2.4).
• Text at every stage. The super-resolution models attend to the prompt, so they can add detail consistent with it or restyle an image (Fig. A.12).

Limitations

• People. The photorealism preference rose from 39.5% to 43.9% (Table 2) when prompts with people were removed, which the authors read as a limited ability to generate photorealistic people (§4.2, §6).
• Bias and data. The authors’ preliminary assessment found biases toward lighter skin tones and toward Western gender stereotypes for professions, and social and cultural biases in images of activities, events and objects (§6). They cite an audit of LAION-400M that found pornographic imagery, racist slurs and harmful stereotypes, and note that Imagen, relying on text encoders trained on uncurated web-scale data, inherits the social biases and limitations of large language models (§6).
• Not released. For these reasons the authors released neither code nor a public demo (§6), so Imagen’s results cannot be reproduced outside Google.
• Sampling cost. Three diffusion models run in sequence, and guidance (Eq. 2) needs a conditional and an unconditional prediction at every step of each.

Historical Importance

Imagen appeared at NeurIPS 2022 with a zero-shot COCO FID of 7.27, which its authors report as a new state of the art, ahead of the concurrent DALL·E 2 (§4.2). Its main finding was that a frozen language model trained only on text is an effective text encoder, whose size matters more than the image model’s (§6); the Text Conditioning article follows the choice of encoder through later systems. In our reading, Imagen also showed that a pixel-space cascade, without a learned latent prior, could be preferred by human raters to DALL·E 2’s two-stage design, which its authors call simpler (§5). It carried Cascaded Diffusion Models’ noise conditioning augmentation from class-conditional ImageNet to text, and its components were reused directly by Google’s later systems, as the Variants below describe: Imagen Video extended its cascade and frozen T5-XXL encoder to video and reused noise conditioning augmentation, and Imagen Editor, Imagic, DreamBooth and Instruct-Imagen were built on the model itself.

Variants

Google built several systems on Imagen, and published papers on them:

• Imagen Video (Ho, Chan, Saharia et al., 2022) extends the design to a cascade of video diffusion models on a frozen T5-XXL encoder, described in the Video Generation article. It applies noise conditioning augmentation to all its super-resolution models (§2.5). Its authors found static clipping gave significant saturation in video and dynamic clipping alone insufficient, and alternated high and low guidance weights (“oscillating guidance”) in the first three models (§2.6.2).
• Imagen Editor (Wang et al., 2023) fine-tunes Imagen for text-guided inpainting. Every stage of the cascade is conditioned on the $1024 \times 1024$ image and mask through new convolutional downsampling encoders whose weights start at zero, so the model starts as Imagen, and training masks come from an object detector rather than at random (§3). Human raters preferred the object-masked model to a randomly masked one in 68% of alignment comparisons, and Imagen Editor to Stable Diffusion and DALL·E 2 in 78% and 77% (§1). The same paper introduced the EditBench benchmark.
• Imagic (Kawar et al., 2023) edits a single real image from a target text: it optimizes a text embedding for 100 steps to match the image, fine-tunes the $64 \times 64$ model on the image for 1,500 steps, and interpolates linearly between the optimized and target embeddings. It runs on Imagen and on Stable Diffusion (§3.2, §3.3); the Image Editing article covers text-guided editing.
• DreamBooth (Ruiz et al., 2023) fine-tunes Imagen or Stable Diffusion on a few photographs of a subject, as the Personalized Image Generation article describes.
• Instruct-Imagen (Hu et al., 2024) starts from “a version of Imagen pre-trained on internal data” whose cascade differs from the 2022 one: a $128 \times 128$ text-to-image model and a super-resolution model to $1024 \times 1024$. It adds one cross-attention layer that reads multi-modal context, growing the model from 2.51B to 2.76B parameters, and is tuned on instructions that combine text with reference images such as edges, styles and subjects (§3.1).

Google has since used the name for later models (Imagen 2, 3 and 4, as of November 2026).

Implementations

No official code was released (§6); the paper’s configurations (App. F) describe the architecture, and its JAX-style pseudocode for thresholding and conditioning augmentation is in Figs. A.31 and A.32. The example above is a NumPy illustration of the sampling step, not an implementation of the model.