DALL·E

DALL·E is OpenAI’s text-to-image model of 2021, and the name of the family that followed it. The first model, described by Ramesh et al. in “Zero-Shot Text-to-Image Generation” (ICML 2021), is a two-stage autoregressive image model. A discrete variational autoencoder (dVAE) compresses each $256 \times 256$ image into a $32 \times 32$ grid of tokens. A 12-billion-parameter sparse transformer then models the caption’s text tokens followed by the image tokens as one sequence. At sampling time, CLIP, a contrastive image–text model, picks the best of many samples. Its authors trained it on 250 million image–text pairs from the internet and evaluated it “zero-shot” on COCO captions, without training on COCO.

Three later OpenAI systems carry the line forward:

• GLIDE (Nichol et al., ICML 2022), by an overlapping team, replaced the autoregressive transformer with a text-conditioned diffusion model.
• DALL·E 2 (Ramesh et al., 2022), which the text-to-image generation article also calls unCLIP, generates a CLIP image embedding from the caption and decodes it with diffusion.
• DALL·E 3 (Betker et al., 2023) is known for training on synthetic captions.

The DALL·E 2 and DALL·E 3 reports could not be read for this article: the first is on arXiv only, and the second is on an OpenAI host that refuses automated access. The sections on them are second-hand accounts, taken from later papers that describe and evaluate those models. The DALL·E and GLIDE sections come from the papers themselves, in their ICML proceedings versions, and from OpenAI’s released dVAE code.

Problem Addressed

Before DALL·E, text-to-image models had mostly built on GANs, following Reed et al. (2016), and were evaluated on relatively small datasets such as COCO and CUB-200. Ramesh et al. (abstract, §1) describe that line as a search for better modelling assumptions on a fixed dataset: multi-scale generators, attention and auxiliary losses, and extra conditioning such as object part labels or segmentation masks. They note that samples could still suffer from object distortion, illogical object placement and unnatural blending of foreground and background. Large autoregressive transformers had meanwhile done well on text, images (Image GPT) and audio when compute, model size and data were scaled carefully. The paper’s question is whether dataset and model size had been the limit. Its answer is a deliberately simple design: one transformer, trained by maximum likelihood on text and image tokens as a single stream, at a scale far beyond earlier text-to-image models.

Modelling pixels directly was ruled out (§2) for its memory cost (a $256 \times 256$ RGB image is 196,608 values) and because likelihood objectives spend capacity on high-frequency detail, a point the autoregressive image model article covers. The authors chose the two-stage recipe they describe as similar to VQ-VAE and VQ-VAE-2: compress first, then model the compressed tokens.

Architecture

The model has three parts, trained separately:

1. the dVAE, the tokenizer and image decoder;
2. the transformer, the prior over text and image tokens;
3. a pretrained contrastive model used only to rerank samples.

The dVAE cuts the sequence the transformer must model by a factor of 192 ($3 \cdot 256^2 / 32^2$), which the authors found did not degrade visual quality much (§2, Fig. 1).

Discrete VAE

The encoder maps a $256 \times 256$ RGB image to a $32 \times 32$ grid, a spatial downsampling by 8, and outputs 8,192 logits at each position: a categorical distribution over a codebook of $K = 8{,}192$ entries (§2.1). The authors chose the large vocabulary to limit information loss (Fig. 1). The released code at commit 5be4b23 (openai/DALL-E, as of November 2026) shows the convolutional design:

• Encoder. A $7 \times 7$ input convolution is followed by four groups of two residual blocks. A $2 \times 2$ max-pooling follows each of the first three groups, and a $1 \times 1$ convolution produces the 8,192 logits. The code’s default widths are 256, 512, 1,024 and 2,048 channels.
• Decoder. A $1 \times 1$ convolution reads the one-hot (or relaxed) code at each position. Four groups of residual blocks follow, with nearest-neighbour upsampling by 2 after the first three, and a $1 \times 1$ output convolution gives six channels per pixel: two parameters for each colour channel’s likelihood.
• Residual gain. Each residual branch is multiplied by $1/L^2$, where $L = 8$ is the number of residual blocks. In our reading this is the small constant by which, according to the paper, the outgoing activations of the residual blocks are multiplied to keep training stable at initialization (§2.1).

The $1 \times 1$ convolutions at the end of the encoder and the start of the decoder are deliberate. The authors found that a small receptive field around the relaxation helped the relaxed training objective generalize to the true one (§2.1).

The reconstruction likelihood is what §2.1 calls the log-laplace distribution (the code names its constant logit_laplace_eps), derived in an appendix that is not in the ICML version (§2.1 refers to Appendix A.3). The code shows its interface. Pixels in $[0, 1]$ are mapped linearly to $[\epsilon, 1 - \epsilon]$ with $\epsilon = 0.1$ (map_pixels). The usage notebook reconstructs an image by taking the encoder’s argmax token at each position, decoding the one-hot grid, and passing the first three output channels through a sigmoid before inverting the pixel map. The model card says that the heavy compression causes a noticeable loss of detail, and that the model is not meant for high-fidelity image processing or as a general-purpose compressor. Figure 1 of the paper shows the loss directly: a cat’s fur texture, the writing on a storefront and the thin lines of an illustration are sometimes lost or distorted.

Transformer

The prior is a 12-billion-parameter decoder-only sparse transformer with 64 self-attention layers (§2.2):

• Text tokens. The lowercased caption is byte-pair encoded into at most 256 tokens from a vocabulary of 16,384.
• Image tokens. These follow as the 1,024 tokens of the $32 \times 32$ grid, from the vocabulary of 8,192.
• Attention masks. Every image token can attend to all text tokens in every layer. Text attends to text through a standard causal mask. Image-to-image attention uses a row, column or convolutional mask, depending on the layer; the full architecture is in an appendix that the ICML version does not include.
• One attention operation. Text-to-text, image-to-text and image-to-image attention share a single attention operation, which worked better than separately normalized operations (footnote 7).
• Padding. Captions shorter than 256 tokens leave positions between the last text token and the first image token. Rather than mask them out, the authors learn a separate padding token for each of the 256 positions. On Conceptual Captions this gave a higher validation loss but better performance on out-of-distribution captions.

The autoregressive image model article places this design among other token-based generators and counts its sampling steps.

Reranker

DALL·E’s generator is not used alone. Citing Razavi et al.’s VQ-VAE-2 as similar, the authors rerank its samples with a pretrained contrastive model, CLIP, which gives each candidate image a score for how well it matches the caption (§2.6). The ICML version of the DALL·E paper does not name which CLIP model it used. Its acknowledgements thank Jong Wook Kim for the PyTorch package of the contrastive model used for reranking.

Training

Objective

The paper writes the whole procedure as maximizing an evidence lower bound (ELB) on the joint likelihood of images $x$ and captions $y$, with the image tokens $z$ as latent variables. The model factorizes as $p_{\theta,\psi}(x, y, z) = p_\theta(x \mid y, z)\, p_\psi(y, z)$, which gives (§2, Eq. 1)

$$\ln p_{\theta,\psi}(x, y) \geq \mathop{\mathbb{E}}_{z \sim q_\phi(z \mid x)} \Big[ \ln p_\theta(x \mid y, z) - \beta\, D_{\mathrm{KL}}\big(q_\phi(y, z \mid x),\, p_\psi(y, z)\big) \Big],$$

where:

• $q_\phi$ is the dVAE encoder’s distribution over the $32 \times 32$ tokens;
• $p_\theta$ is the dVAE decoder;
• $p_\psi$ is the transformer’s joint distribution over text and image tokens.

The caption is assumed conditionally independent of the image given its tokens (footnote 2). The bound holds only for $\beta = 1$; in practice a larger $\beta$ helped, following β-VAE. Training proceeds in two stages, and joint training of all three parts did not improve on them in preliminary ImageNet experiments (footnote 3). The variational autoencoder article explains the bound in its usual single-latent form.

Stage one: the visual codebook

In stage one the prior $p_\psi$ is fixed to the uniform categorical distribution over the 8,192 codes, and the ELB is maximized over the encoder and decoder alone (§2.1). Because $q_\phi$ is discrete, the reparameterization gradient of an ordinary VAE does not apply. VQ-VAE handled this with nearest-neighbour assignment and the straight-through estimator. DALL·E instead uses the Gumbel-softmax relaxation, citing Jang et al. (2017) and Maddison et al. (2017), and replaces the expectation over $q_\phi$ with one over a relaxed distribution $q_\phi^\tau$ that becomes tight as the temperature $\tau \to 0$.

In the form PyTorch implements (torch.nn.functional.gumbel_softmax, v2.14.1), the relaxation adds independent standard Gumbel noise $g_k = -\log(-\log u_k)$, with $u_k$ uniform, to the logits $\theta_k$ and takes a softmax at temperature $\tau$:

$$y_k = \frac{\exp\big((\theta_k + g_k)/\tau\big)}{\sum_{j=1}^{K} \exp\big((\theta_j + g_j)/\tau\big)}.$$

The argmax of $\theta + g$ is an exact sample from $\mathrm{softmax}(\theta)$, the Gumbel-max property. The softmax replaces that argmax with a point $y$ in the probability simplex, so the decoder receives a convex mixture of code embeddings rather than one code. Because $y$ is a differentiable function of $\theta$ for fixed noise, gradients flow to the encoder. The price is that the gradient is exact for the relaxed objective, not for the discrete one. At high temperature the mixture is far from any code. As $\tau \to 0$ the mixture approaches a one-hot vector, but the Jacobian $\partial y / \partial \theta = (\mathrm{diag}(y) - y y^\top)/\tau$ is near zero for most draws and very large for the few near a tie. In our reading this is a trade between bias and variance, which the first example below measures.

The authors list three choices as especially important for stable training (§2.1):

• Annealing schedules. Specific schedules for the temperature and the step size; annealing $\tau$ to $1/16$ was enough to close the gap between the relaxed and the true validation ELB.
• $1 \times 1$ convolutions around the relaxation, as above.
• The small residual gain at initialization.

A larger KL weight, $\beta = 6.6$, gave better codebook usage and a smaller final reconstruction error, contrary to the usual trade-off between the two terms. The authors speculate that with smaller $\beta$ the noise of the relaxation makes the optimizer reduce codebook usage early in training (footnote 4). The relaxed ELB is maximized with Adam and exponentially weighted iterate averaging. The full hyperparameters are in Appendix A.2, which is absent from the ICML version.

Example: Bias and Variance of the Gumbel-Softmax Gradient

The NumPy example below (NumPy 2.3.5, 2 to 5 seconds) measures what the temperature does to the gradient, on a problem small enough that the exact gradient is known. It is a stand-in for one position of the dVAE:

• Codebook. A fixed linear “decoder” $D$ whose $K = 8$ columns are $8 \times 8$ code images: a disk, square, triangle, ring, cross, two bars and a diagonal line.
• Targets. Four $8 \times 8$ images, a disk, a square, a ring and a cross, each slightly shifted and scaled and with Gaussian noise of standard deviation 0.1.
• Encoder. A fixed soft nearest-code rule sets the logits, $\theta_k = -\lVert x - d_k \rVert^2 / 4$.
• Loss. Squared error $f(z) = \lVert x - D z \rVert^2$ stands in for DALL·E’s logit-Laplace likelihood.

The exact objective is $L(\theta) = \sum_k p_k f(e_k)$ with $p = \mathrm{softmax}(\theta)$. Its gradient is $\partial L / \partial \theta_j = p_j \big(f(e_j) - L\big)$, so no sampling is needed for the ground truth. The estimator under test is the single-sample Gumbel-softmax gradient of $f(D y)$, which is what automatic differentiation would return. For each temperature the script draws $M = 100{,}000$ estimates per target and reports four quantities, averaged over the four targets:

• Relative bias. $\lVert \bar g - g^* \rVert / \lVert g^* \rVert$, where $\bar g$ is the mean estimate and $g^*$ the exact gradient, with its Monte Carlo standard error.
• Relative variance. The trace of the estimates’ covariance divided by $\lVert g^* \rVert^2$.
• Relative MSE. Their sum, the error of one stochastic step.
• Relaxed-loss gap. $(\mathbb{E}[f(Dy)] - L)/L$, the analogue of the gap between DALL·E’s relaxed and true ELB.

The script first checks the exact gradient against finite differences of $L$, and the per-sample gradient against finite differences of $f(D\,\mathrm{softmax}((\theta + g)/\tau))$ with the noise held fixed.

import sys
import numpy as np

SEED = int(sys.argv[1]) if len(sys.argv) > 1 else 0
rng = np.random.default_rng(SEED)
yy, xx = np.mgrid[0:8, 0:8] + 0.5                     # pixel centres of an 8 x 8 image

def shape(name, cx=4.0, cy=4.0, s=1.0):
    """8 x 8 synthetic code image in [0, 1]."""
    dx, dy = (xx - cx) / s, (yy - cy) / s
    r = np.hypot(dx, dy)
    m = {"disk": r <= 2.6, "square": (abs(dx) <= 2.3) & (abs(dy) <= 2.3),
         "triangle": (dy >= -2.5) & (dy <= 2.5) & (abs(dx) <= (dy + 2.5) / 2),
         "ring": (r <= 3.2) & (r >= 1.9), "cross": (abs(dx) <= 0.8) | (abs(dy) <= 0.8),
         "hbar": abs(dy) <= 1.0, "vbar": abs(dx) <= 1.0, "diag": abs(dx - dy) <= 1.0}[name]
    return m.astype(float)

CODES = ["disk", "square", "triangle", "ring", "cross", "hbar", "vbar", "diag"]
D = np.stack([shape(c).ravel() for c in CODES], 1)    # 64 x K "decoder": code k -> image d_k
K = D.shape[1]
TARGETS = {n: np.clip(shape(n, 4 + rng.uniform(-0.5, 0.5), 4 + rng.uniform(-0.5, 0.5), rng.uniform(0.9, 1.1)).ravel()
                      + rng.normal(0, 0.1, 64), 0, 1) for n in ["disk", "square", "ring", "cross"]}

def softmax(a):
    a = a - a.max(-1, keepdims=True)
    e = np.exp(a)
    return e / e.sum(-1, keepdims=True)

def exact(theta, x):
    """Exact objective L = E_{z ~ Cat(p)} ||x - D z||^2 and its gradient p_j (f_j - L)."""
    p = softmax(theta)
    f = ((x[:, None] - D) ** 2).sum(0)                  # loss of each one-hot code
    L = p @ f
    return L, p * (f - L), p

def gumbel_softmax_grads(theta, x, tau, M, rng):
    """M single-sample Gumbel-softmax gradients of f(D y), y = softmax((theta + G) / tau)."""
    G = -np.log(rng.exponential(size=(M, K)))           # Gumbel(0, 1), as torch.nn.functional.gumbel_softmax draws it
    y = softmax((theta + G) / tau)
    r = y @ D.T - x                                      # residual of the mixed image, M x 64
    gy = 2 * r @ D                                       # d f / d y, M x K
    g = (y * gy - y * (y * gy).sum(1, keepdims=True)) / tau   # J^T gy with J = (diag(y) - y y^T) / tau
    return g, (r ** 2).sum(1)

# Checks: the exact gradient against finite differences, and the per-sample gradient with G held fixed
h, worst_exact, worst_gs, worst_abs, smallest = 1e-6, 0.0, 0.0, 0.0, np.inf
for x in TARGETS.values():
    theta = -((x[:, None] - D) ** 2).sum(0) / 4          # soft nearest-code encoder logits
    L, g, _ = exact(theta, x)
    fd = np.array([(exact(theta + h * e, x)[0] - exact(theta - h * e, x)[0]) / (2 * h) for e in np.eye(K)])
    worst_exact = max(worst_exact, np.abs(fd - g).max() / np.abs(g).max())
    for tau in [1.0, 1 / 16]:
        G = -np.log(np.random.default_rng(99).exponential(size=K))
        fy = lambda th: ((softmax((th + G) / tau) @ D.T - x) ** 2).sum()
        gs = (lambda y, gy: (y * gy - y * (y * gy).sum()) / tau)(softmax((theta + G) / tau),
                                                                   2 * (softmax((theta + G) / tau) @ D.T - x) @ D)
        fdg = np.array([(fy(theta + h * e) - fy(theta - h * e)) / (2 * h) for e in np.eye(K)])
        worst_gs = max(worst_gs, np.abs(fdg - gs).max() / max(np.abs(gs).max(), 1e-12))
        worst_abs, smallest = max(worst_abs, np.abs(fdg - gs).max()), min(smallest, np.abs(gs).max())
print(f"seed {SEED}; exact gradient vs finite differences: max relative error {worst_exact:.1e}")
print(f"per-sample gradient vs finite differences: max absolute error {worst_abs:.1e}, max relative error "
      f"{worst_gs:.1e}; smallest checked gradient (max |g|) {smallest:.1e}")
print("p_max per target:", ", ".join(f"{n} {exact(-((x[:, None] - D) ** 2).sum(0) / 4, x)[2].max():.2f}"
                                     for n, x in TARGETS.items()))

M = 100_000
print(f"{'tau':>6s} {'rel. bias':>10s} {'(MC s.e.)':>10s} {'rel. var':>10s} {'rel. MSE':>10s} {'relaxed-loss gap':>17s}")
for tau in [1 / 16, 1 / 8, 1 / 4, 1 / 2, 1, 2, 4]:
    rows = []
    for x in TARGETS.values():
        theta = -((x[:, None] - D) ** 2).sum(0) / 4
        L, g_star, _ = exact(theta, x)
        g, fy = gumbel_softmax_grads(theta, x, tau, M, rng)
        n2 = (g_star ** 2).sum()
        bias = np.sqrt(((g.mean(0) - g_star) ** 2).sum() / n2)
        var = g.var(0).sum() / n2
        rows.append((bias, np.sqrt(var / M), var, bias ** 2 + var, (fy.mean() - L) / L))
    b, se, v, mse, gap = np.mean(rows, 0)
    print(f"{tau:6.4g} {b:10.3f} {se:10.3f} {v:10.3g} {mse:10.3g} {gap:17.3f}")

Output (seed 0; two runs gave identical output):

seed 0; exact gradient vs finite differences: max relative error 7.2e-10
per-sample gradient vs finite differences: max absolute error 4.5e-09, max relative error 3.5e-02; smallest checked gradient (max |g|) 5.3e-18
p_max per target: disk 0.60, square 0.62, ring 0.68, cross 0.81
   tau  rel. bias  (MC s.e.)   rel. var   rel. MSE  relaxed-loss gap
0.0625      0.069      0.021       42.5       42.5            -0.037
 0.125      0.133      0.014       19.7       19.7            -0.071
  0.25      0.240      0.009       8.51       8.57            -0.131
   0.5      0.408      0.006       3.13        3.3            -0.225
     1      0.549      0.003      0.811       1.12            -0.326
     2      0.640      0.001        0.1       0.52            -0.298
     4      0.787      0.000     0.0058      0.631            -0.100

The checks. The exact gradient agrees with finite differences to $10^{-9}$, and the per-sample gradient agrees to an absolute error of $5 \times 10^{-9}$. With absolute errors that small, a relative error of 3.5% can arise only on a draw whose own gradient is tiny, below about $10^{-7}$, and the smallest checked gradient, $5 \times 10^{-18}$, lies below the finite differences’ round-off. In our reading these are saturated low-temperature draws, the near-zero gradients behind the variance column.

Bias against variance. The table shows the trade the paper’s wording implies. As the temperature falls from 4 to 1/16:

• the bias falls from 79% of the true gradient’s norm to 7%, still about three standard errors from zero;
• the single-sample variance rises from 0.006 to 42 times the squared norm, by a factor of two to four with each halving of $\tau$ below 1.

At high temperature the estimator is precise but points at the wrong target, the gradient of a loss on blurred mixtures of codes. At low temperature it is nearly unbiased but almost every draw is close to zero and a few are huge.

The objective. The relaxed-loss gap behaves like the gap DALL·E closed by annealing. Below $\tau = 1$, the relaxed loss approaches the true loss as $\tau$ falls, to within 4% at $\tau = 1/16$. It is lower than the true loss because a mixture of codes can fit the target better than any single code. At $\tau = 4$ the gap shrinks again; in our reading, the mixtures approach the average of all codes, whose loss need not be far from the expected loss.

The single-step error. The combined error of one step is smallest at $\tau = 2$ for this seed. The script takes the seed as its argument, and the seed also redraws the four targets. Seeds 1 and 2 give the same ordering of bias and variance at every temperature. Their MSE minimum falls at $\tau = 4$ and $\tau = 2$ respectively, so on this problem a small single-sample error calls for a temperature well above DALL·E’s final 1/16.

Two qualifications apply. DALL·E anneals the temperature, so early training sees the low-variance regime and late training the low-bias one. Its objective averages over $1{,}024$ positions and large batches, which in our reading reduces the variance that dominates at low temperature.

The check line was changed after the first run. The first run printed only the relative error of the per-sample check. Because that number read as a failure, the maximum absolute error and the smallest checked gradient were added, after a diagnostic run with the same code. The table and the data did not change.

Stage two: the prior

In stage two the dVAE is frozen, and the transformer is trained to maximize the ELB over $\psi$, which amounts to maximum likelihood of text and image tokens (§2.2):

• Image tokens. These are the argmax of the encoder’s logits, with no Gumbel noise. Footnote 6 says that sampling would be strictly correct, and that in preliminary ImageNet experiments it acted as a useful regularizer in the overparameterized regime. The authors decided against it because their 12-billion-parameter model is in the underparameterized regime.
• BPE dropout. Captions are tokenized with 10% BPE dropout (footnote 5).
• Loss weights. The cross-entropy losses for text and image tokens are each normalized by the number of tokens of that kind in the batch, then weighted 1/8 and 7/8, because the authors care mainly about modelling images.
• Optimizer. The objective is again optimized with Adam and iterate averaging.
• Overfitting. About 606,000 images were held out for validation, and the model showed no signs of overfitting at convergence.

Data

Preliminary experiments with models of up to 1.2 billion parameters used Conceptual Captions, 3.3 million image–text pairs (§2.3). For the 12-billion-parameter model the authors collected 250 million image–text pairs from the internet, a dataset of a scale similar to JFT-300M. It includes Conceptual Captions and a filtered subset of YFCC100M but not COCO. Because COCO was built from YFCC100M, the training data contain some COCO validation images, but none of their captions. The authors found about 21% of COCO validation images among their training images. The deduplication used the procedure of the CLIP paper: a contrastive model trained to find each validation image’s closest training image, a ranking by closeness, and a conservative threshold set by hand after inspection (§3.1, §3.2). The model card states that the dataset will not be released.

Mixed precision and distributed training

Most parameters, Adam moments and activations were stored in 16-bit precision. The authors call training in 16-bit precision past one billion parameters without divergence “the most challenging part of this project” (§2.4). They attribute the instability to underflow. The norms of the activation gradients fall from the earlier residual blocks to the later ones, and in a deeper, wider model the later blocks’ gradients fall below the smallest exponent of the 16-bit format and round to zero. Standard loss scaling uses one scale for the whole network. On V100 GPUs, whose 16-bit format has five exponent bits, that range was enough for language models of the same size but not for this model. The fix, per-resblock gradient scaling (Fig. 4), keeps a separate gradient scale for each residual block. It stores the identity path’s activations and gradients in 32-bit precision and sets non-finite values in each block’s activation gradient to zero, so that one overflow does not shrink the scales of all earlier blocks. Appendix D, absent from the ICML version, gives further guidelines.

The 12-billion-parameter model needs about 24 GB in 16-bit precision, more than a 16 GB V100 holds. Its parameters were therefore sharded across the eight GPUs of each machine, with the next block’s shards prefetched during computation (§2.5, Fig. 5). Bandwidth between machines was the bottleneck, so gradients were compressed with PowerSGD. A compression rate of about 85% was possible at every model size tested (Table 1: ranks 512, 640 and 896 for 2.8, 5.6 and 12.0 billion parameters).

Inference

Generation feeds the caption’s tokens to the transformer, samples 1,024 image tokens one at a time, and decodes them with the dVAE. Unless stated otherwise, the paper’s results use temperature $t = 1$, with no temperature reduction, and contrastive reranking with $N = 512$: for each caption, 512 samples are drawn and the ones CLIP scores highest are kept (§2.6). The authors liken this to a language-guided search and to the auxiliary image–text matching loss of AttnGAN. On COCO, FID and Inception Score improve clearly as $N$ grows, up to $N = 32$, after which the returns diminish (§3.1, Fig. 9c). By our arithmetic, one reranked image costs 512 × 1,024 = 524,288 transformer steps, of which only 1,024 are sequential if the 512 candidates are sampled in parallel.

Other uses come from the same model:

• Concept combinations. Prompts such as “a tapir made of accordion” (Fig. 2a), which the authors read as a rudimentary ability to compose unusual concepts (§3.3).
• Text rendering. For example, a neon sign that reads “backprop” (Fig. 2c; the text of §3.3 swaps the letters of panels b and c).
• Image-to-image translation. Given the caption “the exact same cat on the top as a sketch at the bottom” and the top 15 of the 32 token rows of a cat photo, the model completes the bottom half with a sketch of a similar cat. The same prompting works with limited reliability for colour changes, grayscale conversion, flipping and some style transfers (§3.3).

Example: How Much Reranking Buys

Reranking turns a weak generator into a better one only as far as the scorer can be trusted. The example below (NumPy 2.3.5 and scikit-learn 1.7.2, about 15 to 40 seconds) makes that concrete. The prompt is “a disk”. A stand-in generator draws $16 \times 16$ grayscale candidates, each with one filled shape:

• a disk with probability $q = 0.1$, otherwise a square or a triangle with equal chance;
• shapes of equal area, set by a size $r \in [2.5, 4.5]$, at a random centre, with random intensity and pixel noise.

A candidate is correct if it shows a disk, which is known by construction. The CLIP stand-ins are deterministic scorers: logistic regressions on the magnitude of the 2-D Fourier transform of the image, average-pooled to $R \times R$ pixels with $R$ from 16 down to 2. Each is trained on 4,000 separate balanced candidates. A coarser view makes a less reliable scorer, and reliability is measured by ROC-AUC on another 4,000 balanced candidates. An oracle that knows the label and a random scorer bracket them.

Precision of best-of-$N$ is the probability that the highest-scoring of $N$ candidates is a disk. It is computed exactly over a pool of 500,000 generated candidates for $N$ draws with replacement. If item $i$ has rank $k_i$ among the pool’s $P$ scores, from 1 for the lowest, it is the maximum of $N$ draws with probability $w_i = (k_i/P)^N - ((k_i - 1)/P)^N$, and the precision is the sum of $w_i$ over the disks. At large $N$ this sum depends on the pool’s few top-scoring items, so each precision is printed with a standard error from the weights, $\sqrt{\sum_i w_i^2 (y_i - \hat P)^2}$ for labels $y_i$ and precision $\hat P$. The pool size, the standard errors and the base check were set after a pilot. With the planned pool of 50,000, the random scorer missed its planned check, a precision equal to $q$ at every $N$, at large $N$, where the exact formula rests on few items. The pilot was reported to the editor, who decided to enlarge the pool and print the standard errors, and fixed the check before the next pilot: the random scorer must stay within two standard errors of $q$ at every $N$. The script prints that check. The script also prints the mean size of the chosen disk, to show what selection does to the distribution of accepted samples.

import numpy as np
from sklearn.linear_model import LogisticRegression

yy, xx = np.mgrid[0:16, 0:16] + 0.5
Q = 0.1                                                   # generator's chance of drawing the prompted shape

def generate(n, rng, q=Q):
    """n candidate 16 x 16 images for the prompt 'a disk': disk w.p. q, else an equal-area square or triangle."""
    kind = rng.choice(3, n, p=[q, (1 - q) / 2, (1 - q) / 2])    # 0 disk, 1 square, 2 triangle
    r = rng.uniform(2.5, 4.5, n)[:, None, None]
    cx, cy = (rng.uniform(6.0, 10.0, n)[:, None, None] for _ in range(2))
    dx, dy = xx - cx, yy - cy
    a, b = r * np.sqrt(np.pi) / 2, r * np.sqrt(2 * np.pi) / 2     # half-side; half of triangle base = half height
    masks = np.where(kind[:, None, None] == 0, np.hypot(dx, dy) <= r,
             np.where(kind[:, None, None] == 1, (abs(dx) <= a) & (abs(dy) <= a),
                      (dy >= -b) & (dy <= b) & (abs(dx) <= (dy + b) / 2)))
    img = masks * rng.uniform(0.5, 1.0, n)[:, None, None] + rng.normal(0, 0.1, (n, 16, 16))
    return img, kind, r.ravel(), masks.sum((1, 2))

def features(img, R):
    """Magnitude of the 2-D FFT of the image average-pooled to R x R (translation-invariant up to wrap-around)."""
    k = 16 // R
    pooled = img.reshape(len(img), R, k, R, k).mean((2, 4))
    return np.abs(np.fft.fft2(pooled)).reshape(len(img), -1)

def auc(s, y):
    """ROC-AUC by the rank formula (scores are continuous, so no ties)."""
    ranks = np.argsort(np.argsort(s)) + 1
    n1 = y.sum()
    return (ranks[y].sum() - n1 * (n1 + 1) / 2) / (n1 * (len(y) - n1))

def best_of_n(s, y, N):
    """Exact P(the highest-scoring of N i.i.d. draws from the pool is correct), its standard error, and the weights."""
    M = len(s)
    rank = np.argsort(np.argsort(s)) + 1                   # 1 = lowest score
    w = (rank / M) ** N - ((rank - 1) / M) ** N          # P(item is the maximum of N draws)
    P = w[y].sum()
    return P, np.sqrt((w ** 2 * (y - P) ** 2).sum()), w

# Scorers ("CLIP stand-ins"): logistic regression on FFT magnitudes, trained on balanced candidates
train, ktr, _, _ = generate(4_000, np.random.default_rng(1), q=0.5)
test, kte, _, _ = generate(4_000, np.random.default_rng(2), q=0.5)
ytr, yte = ktr == 0, kte == 0
jit = np.random.default_rng(3)
aucs = {"oracle": auc(yte + jit.uniform(0, 0.5, len(yte)), yte), "random": auc(jit.uniform(0, 1, len(yte)), yte)}
clfs = {}
for R in [16, 8, 4, 2]:
    clfs[R] = LogisticRegression(C=1.0, max_iter=5000).fit(features(train, R), ytr)
    aucs[f"FFT {R}x{R}"] = auc(clfs[R].decision_function(features(test, R)), yte)

# The generator's pool: 500,000 candidates, generated and scored in chunks of 50,000
gen, parts = np.random.default_rng(0), []
for _ in range(10):
    img, kind, radius, area = generate(50_000, gen)
    lab = kind == 0
    sc = {"oracle": lab + jit.uniform(0, 0.5, len(img)), "random": jit.uniform(0, 1, len(img))}
    sc.update({f"FFT {R}x{R}": clfs[R].decision_function(features(img, R)) for R in clfs})
    parts.append((kind, radius, area, sc))
kind, radius, area = (np.concatenate([p[j] for p in parts]) for j in range(3))
scorers = {n: np.concatenate([p[3][n] for p in parts]) for n in parts[0][3]}
label, q = kind == 0, (kind == 0).mean()
print(f"pool: {len(label)} candidates, disk fraction q = {q:.4f}; mean area in pixels: "
      + ", ".join(f"{n} {area[kind == i].mean():.1f}" for i, n in enumerate(["disk", "square", "triangle"])))

Ns = [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]
print(f"{'scorer':10s}{'AUC':>6s}  precision of the chosen image (and s.e.) at N =")
print(f"{'':16s}" + "".join(f"{n:>7d}" for n in Ns))
mean_r, z_random = {}, 0.0
for name, s in scorers.items():
    res = [best_of_n(s, label, N) for N in Ns]
    print(f"{name:10s}{aucs[name]:6.3f}" + "".join(f"{P:7.3f}" for P, _, _ in res))
    print(f"{'':16s}" + "".join(f"{'(' + format(se, '.3f')[1:] + ')':>7s}" for _, se, _ in res))
    if name == "random":
        z_random = max(abs(P - q) / se for P, se, _ in res)
    for N in [1, 512]:
        P, _, w = best_of_n(s, label, N)
        mean_r[name, N] = (w * radius)[label].sum() / P          # mean radius of the chosen disk
closed = [1 - (1 - q) ** N for N in Ns]
print(f"{'1-(1-q)^N':16s}" + "".join(f"{p:7.3f}" for p in closed))
print(f"check: random scorer within {z_random:.2f} s.e. of q at every N")
print(f"generator's disks: mean radius {radius[label].mean():.2f}; mean radius of the chosen disk at N = 1 / 512:")
print("  " + "; ".join(f"{n} {mean_r[n, 1]:.2f} / {mean_r[n, 512]:.2f}" for n in scorers))

Output (two runs gave identical output):

pool: 500000 candidates, disk fraction q = 0.1001; mean area in pixels: disk 39.6, square 39.6, triangle 39.6
scorer       AUC  precision of the chosen image (and s.e.) at N =
                      1      2      4      8     16     32     64    128    256    512
oracle     1.000  0.100  0.190  0.344  0.570  0.815  0.966  0.999  1.000  1.000  1.000
                 (.000) (.001) (.001) (.001) (.001) (.000) (.000) (.000) (.000) (.000)
random     0.494  0.100  0.100  0.100  0.101  0.100  0.099  0.097  0.096  0.095  0.093
                 (.000) (.000) (.001) (.001) (.001) (.002) (.002) (.003) (.005) (.007)
FFT 16x16  0.996  0.100  0.190  0.342  0.564  0.805  0.960  0.998  1.000  1.000  1.000
                 (.000) (.001) (.001) (.001) (.001) (.000) (.000) (.000) (.000) (.000)
FFT 8x8    0.950  0.100  0.181  0.309  0.485  0.688  0.863  0.963  0.994  0.999  1.000
                 (.000) (.001) (.001) (.001) (.002) (.001) (.001) (.000) (.000) (.000)
FFT 4x4    0.692  0.100  0.133  0.161  0.187  0.215  0.247  0.286  0.332  0.383  0.438
                 (.000) (.001) (.001) (.001) (.002) (.003) (.004) (.006) (.008) (.011)
FFT 2x2    0.544  0.100  0.108  0.106  0.103  0.102  0.105  0.110  0.114  0.114  0.106
                 (.000) (.001) (.001) (.001) (.001) (.002) (.003) (.004) (.005) (.006)
1-(1-q)^N         0.100  0.190  0.344  0.570  0.815  0.966  0.999  1.000  1.000  1.000
check: random scorer within 1.23 s.e. of q at every N
generator's disks: mean radius 3.50; mean radius of the chosen disk at N = 1 / 512:
  oracle 3.50 / 3.49; random 3.50 / 3.50; FFT 16x16 3.50 / 3.73; FFT 8x8 3.50 / 3.72; FFT 4x4 3.50 / 3.90; FFT 2x2 3.50 / 4.15

The two references. Equal areas leave no shortcut through total brightness, as the printed areas confirm. The oracle reproduces the closed form $1 - (1 - q)^N$: with a perfect scorer, reranking succeeds whenever any of the $N$ candidates is right. The random scorer stays at $q$ within its standard errors, so reranking by noise buys nothing.

The learned scorers. These fall between the references in the order of their AUC. The two good ones behave like the oracle: an AUC of 0.95 costs some precision at moderate $N$ (0.69 against 0.82 at $N = 16$), yet reaches 1.000 by $N = 512$. That happens because the highest scores this scorer gives go almost only to disks. The precision at large $N$ is set by the top of the score distribution, not by the AUC, which averages over all ranks. With an AUC of 0.69, precision climbs slowly, from 0.10 to 0.44 over the whole range. With 0.54 it barely moves. For the oracle and the good scorers, most of the gain arrives by $N = 32$ to 64. This parallels DALL·E’s diminishing FID returns past 32, though precision here is a different measure from FID.

Selection changes the distribution. The oracle picks disks without preference, so the chosen disks have the generator’s mean radius. Every learned scorer prefers larger disks, the two weak ones most: 3.73 and 3.72 for the two good scorers, 3.90 and 4.15 for the weak ones, against 3.50. In our reading, a scorer that sees a coarse view trusts the cues that survive pooling, and those are stronger for large shapes. Reranking therefore improves prompt fidelity at a cost to the variety of what is accepted, in a direction set by the scorer’s biases rather than the prompt. This connects to GLIDE’s finding, below, that CLIP guidance raised CLIP scores without convincing human raters.

Performance

Ramesh et al. compared DALL·E zero-shot with three GANs trained on COCO: AttnGAN, DM-GAN and DF-GAN, the last with the best published Inception Score and FID on COCO (§3.1). FID and IS used 30,000 captions sampled from the COCO validation set and the DM-GAN code (Fig. 9):

• Human evaluation. In a best-of-five vote against DF-GAN on COCO captions, DALL·E’s sample was chosen as more realistic 90.0% of the time and as better matching a shared caption 93.3% of the time (Fig. 7). The samples were evaluated zero-shot without temperature reduction.
• FID. DALL·E’s FID on COCO was within 2 points of the best prior approach, although it never trained on COCO captions (Fig. 9a). Removing the roughly 21% of validation images that overlap the training data made no significant difference.
• Blur. The dVAE cannot produce high-frequency detail, so the authors also computed FID and IS after Gaussian blur of both samples and validation images. With a blur of radius 1, DALL·E had the best FID by about 6 points, and the gap widened with more blur. It had the highest IS from radius 2.
• CUB. On CUB birds DALL·E was nearly 40 FID points behind the leading prior approach (Fig. 9b). The authors suspect that zero-shot models compare less favourably on specialized distributions, and suggest fine-tuning.

The paper gives these FIDs only as curves. Later papers print different single numbers for DALL·E’s zero-shot COCO FID: “∼28” (GLIDE, Table 2), 27.50 (Zhou et al., 2023, Table 1) and 17.89 (Muse, Table 2). We found no explanation of the difference; the text-to-image generation article discusses the protocol’s other problems.

GLIDE’s human evaluation compared the two models directly (Nichol et al., 2022, Table 3). GLIDE’s samples, with classifier-free guidance and no reranking, were preferred to DALL·E’s in every setting:

DALL·E setting	DALL·E temperature	GLIDE preferred, photorealism	GLIDE preferred, caption similarity
No reranking	1.0	91%	83%
No reranking	0.85	84%	80%
CLIP-reranked	1.0	89%	71%
CLIP-reranked	0.85	87%	69%
Reranked, GLIDE passed through DALL·E’s dVAE	1.0	72%	63%
Reranked, GLIDE passed through DALL·E’s dVAE	0.85	66%	61%

The last two rows blur GLIDE’s samples through DALL·E’s dVAE, a configuration the GLIDE authors describe as heavily favouring DALL·E. Reranking helped DALL·E most on caption similarity: GLIDE’s advantage there fell from 83% to 71% at temperature 1.0.

Strengths

• Simplicity and generality. One likelihood-trained transformer over a token stream, with no text-to-image-specific losses, no GAN training and no extra annotations (§1).
• Zero-shot breadth. It handles captions far from any curated dataset. Ramesh et al. report concept combinations, text rendering and rudimentary image-to-image translation as capabilities emerging from scale (§3.3, §4).
• Exact likelihood for the prior. The transformer is trained by maximum likelihood on discrete tokens, and showed no signs of overfitting at 12 billion parameters (§2.2).
• Engineering recipes. Per-resblock gradient scaling for 16-bit training and PowerSGD compression at scale are described in detail (§2.4, §2.5).

Limitations

• Detail lost to the tokenizer. The dVAE loses fine detail (Fig. 1), and the blur experiment shows that FID penalizes the loss.
• Unreliable binding. On “an illustration of a baby hedgehog in a christmas sweater walking a dog”, the model sometimes puts sweaters on both animals or draws a hedgehog walking a smaller hedgehog (§3.3).
• Specialized domains. It is far behind on CUB.
• Cost. It needs 1,024 sequential steps per image, multiplied by the reranking budget. GLIDE was preferred with roughly the same training compute and a smaller model (3.5 against 12 billion parameters), and without reranking (Nichol et al., §5.2).
• Dependence on the reranker. Reported results use $N = 512$, and the text-to-image generation article notes that this complicates comparisons with models that do not rerank. As the example shows, reranking is only as good as the scorer’s top scores.
• Closed release. As of November 2026, the official repository contains only the dVAE, whose README states that the transformer is not part of the code release. The model card says the dataset will not be released.

Historical Importance

DALL·E showed that one transformer trained by maximum likelihood on 250 million web image–text pairs could compete zero-shot with GANs trained on COCO. In our reading it also changed how the task was evaluated: GLIDE, Muse and Zhou et al. all report zero-shot COCO FIDs alongside DALL·E’s, in tables with a separate zero-shot column. Its recipe of tokenizer plus transformer was carried to video, as the video generation article describes, and to later autoregressive text-to-image models. Its pairing with CLIP used a contrastive model as a judge of generated images; in our reading, that role grew in its successors from reranking to guidance (GLIDE) to conditioning (DALL·E 2). The next two OpenAI systems, GLIDE and DALL·E 2, are diffusion models, and human raters preferred GLIDE’s samples to DALL·E’s.

Variants

GLIDE (2021–2022)

GLIDE (Nichol et al., ICML 2022) is a 3.5-billion-parameter text-conditional diffusion model at $64 \times 64$, with a 1.5-billion-parameter text-conditional upsampler to $256 \times 256$ (§4). The base model combines two parts (Appendix B.1):

• an ADM U-Net widened to 512 channels, about 2.3 billion parameters;
• a 24-block transformer text encoder of width 2,048, about 1.2 billion parameters.

The transformer’s final token embedding replaces ADM’s class embedding, and its last layer of token embeddings is projected into the attention context of every layer (§4.1). GLIDE was trained on the same dataset as DALL·E with roughly the same compute (§4.1), for 2.5 million iterations at batch size 2,048, and trained stably in 16-bit precision with ordinary loss scaling (Appendix B.1).

The paper’s main comparison is between two ways of guiding toward the caption:

• CLIP guidance uses a noise-aware CLIP model trained on noised images.
• Classifier-free guidance comes from fine-tuning the model with 20% of captions replaced by the empty sequence (§2.3, §4.2, §4.4).

Human raters preferred classifier-free guidance for both photorealism and caption similarity, although CLIP guidance raised the CLIP score more. The authors hypothesize that CLIP guidance finds adversarial examples for the evaluating CLIP model (§5.2, Table 1). GLIDE reached a zero-shot COCO FID of 12.24 at guidance scale 1.5 (Table 2). Fine-tuned with four extra input channels for a masked image and its mask, initialized to zero, it performs text-guided inpainting (§4.3).

Citing the risk of misuse, the authors released a smaller model, GLIDE (filtered), a 300-million-parameter model trained on data filtered to remove images of people, violent objects and some hate symbols (Appendix E, F.1). Red-teaming found no recognizable images of humans, but the filtered model retained and sometimes amplified dataset biases, which classifier-free guidance amplified further (Appendix E).

DALL·E 2 (unCLIP, 2022)

Second-hand account: the DALL·E 2 paper was not read, and this section reports how later readable papers describe it.

Zhou et al. (CVPR 2023) build their Corgi model on a three-part framework that, with a hierarchical diffusion decoder conditioned on both the image embedding and the text, they call similar to DALL·E 2 (§2):

• a pretrained CLIP image encoder that defines the embedding space;
• a prior that generates a CLIP image embedding from the caption; in DALL·E 2, they write, it is a diffusion model sampling from Gaussian noise $z_T \sim \mathcal{N}(0, I)$ down to the embedding $z_0$;
• a decoder, a hierarchy of diffusion models conditioned on that embedding and on the text.

They feed their own decoder the embedding as eight projected vectors, which they say follows DALL·E 2 (§3.1). They note that DALL·E 2’s image embedding enables image-to-image generation and generation under combined image and text conditions (§1). Muse’s authors describe DALL·E 2 as built on cascaded pixel-space diffusion models (Chang et al., 2023, §1).

Later papers print DALL·E 2’s zero-shot COCO FID as 10.39 (Muse, Table 2; Zhou et al., Table 1). Stable Diffusion 3’s GenEval table gives DALL·E 2 an overall score of 0.52 (Esser et al., 2024, Table 5). Its weakest categories are position (0.10) and colour attribution (0.19), the binding of colours to the right objects. The GenEval benchmark is described in the text-to-image generation article. Stable Diffusion 3’s authors also cite OpenAI’s blog post on DALL·E 2’s pre-training mitigations for their own data filtering and for the number of clusters in their deduplication (Appendix C.1, D). That post is on a host not used for this article.

DALL·E 3 (2023)

Second-hand account: the DALL·E 3 report (Betker et al., 2023) was not read for this article.

Stable Diffusion 3’s authors summarize its central finding: synthetic captions can greatly improve text-to-image models trained at scale, because human-written captions focus on the main subject and omit background, composition and any text shown (Esser et al., 2024, §5.2.2). They follow the approach with a vision–language model, CogVLM. They mix 50% original and 50% synthetic captions, so that the model does not forget concepts the captioner does not know, and they find the mix clearly better on GenEval than original captions alone (Table 4).

In their GenEval comparison, DALL·E 3 scores 0.67 overall, with the table’s highest position score (0.43) and 0.45 for colour attribution. Their largest model, at $1024^2$ after preference tuning, scores 0.74 overall (Table 5). In their human preference study on PartiPrompts, their depth-38 model compared favourably with DALL·E 3 on visual aesthetics, prompt following and typography (Fig. 6). They describe DALL·E 3 as the state of the art in prompt comprehension at the time (§5.3.1). The text conditioning article summarizes the DALL·E 3 report’s own account of its captioner and evaluations.

Implementations

• openai/DALL-E (modified MIT licence): the official PyTorch package for the dVAE, with encoder, decoder, pixel mapping and a usage notebook that loads pretrained weights from OpenAI’s servers. As of November 2026 its last commit is from 31 March 2021, and the transformer is not included.
• GLIDE (filtered): the GLIDE paper states that code and weights for the small filtered model were released in the openai/glide-text2im repository (abstract); the repository was not read for this article.
• Gumbel-softmax: PyTorch provides the relaxation as torch.nn.functional.gumbel_softmax. The function draws Gumbel noise and takes a softmax at temperature tau. With hard=True it returns one-hot samples and passes the soft sample’s gradient straight through (v2.14.1 source).