Text-to-Image Generation

Text-to-image generation is the task of producing an image from a natural-language description, the prompt or caption: “a red cube on top of a blue cube” in, a picture of that out. It is image generation with free text as the condition, and the inverse of image captioning: in the vocabulary of multimodal learning, a translation from sentence to image. Since 2021, models trained on hundreds of millions of web image–caption pairs have made the task practical, and in 2023 GenEval’s authors observed that most widely used models and checkpoints were based on Stable Diffusion (Ghosh et al., 2023, §2), a latent diffusion model conditioned on CLIP’s text encoder.

Problem Definition

Training data are pairs $(x^{(i)}, y^{(i)})$ of an image and a caption, samples from an unknown joint distribution. The task is to sample from the conditional distribution $p(x \mid y)$ for any caption $y$, including captions unlike any in the training set. A caption underspecifies its image (pose, lighting, background and style are usually left open), so there is no single correct output, and a good model returns varied images for one prompt.

Two requirements pull against each other:

• Image quality, the fidelity and diversity of image generation.
• Prompt fidelity, also called image–text alignment or prompt adherence: everything the prompt states (objects, counts, colors, positions, written text) is depicted, with attributes attached to the right objects.

Classifier-free guidance trades one for the other at sampling time, so results are often reported as curves of a text-alignment score against Fréchet Inception Distance (FID) over guidance weights (Saharia et al., 2022, §3).

Zero-shot has a narrow meaning here: following DALL·E (Ramesh et al., 2021), a model evaluated on COCO captions without training on COCO is zero-shot, and one trained on COCO’s training split is “finetuned” (Yu et al., 2022, Table 5).

Input and Output

A text encoder turns the tokenized prompt into a sequence of embeddings; the output is an image. DALL·E took up to 256 byte-pair-encoded text tokens (Ramesh et al., 2021, §2). Stable Diffusion v1 conditions its U-Net on the per-token (not pooled) embeddings of a frozen CLIP ViT-L/14 text encoder and generates $512 \times 512$ images (CompVis README, as of October 2026).

High resolutions are usually reached in stages: Imagen upsamples $64 \times 64$ samples to $256 \times 256$ and then $1024 \times 1024$ with two text-conditioned super-resolution diffusion models (Saharia et al., 2022, §2.4), GLIDE from 64 to 256 pixels (Nichol et al., 2022, §4), while latent diffusion models decode a small latent once.

Some systems keep the best of several samples by an image–text matching model, at a multiple of the sampling cost: DALL·E the best of 512 ranked by CLIP (§2.6), Parti the best of 16 ranked by a CoCa model (Yu et al., 2022, §5).

Variants

• Text-guided editing and inpainting. GLIDE was fine-tuned to fill a masked region according to a prompt (Nichol et al., 2022, §4.3).
• Image variations. DALL·E 2’s decoder generates new images from the CLIP image embedding of an existing one, keeping its content and style (Ramesh et al., 2022).
• Additional conditions, such as a layout or a semantic map (see the latent diffusion model article).

Challenges

Compositionality. Models often get the parts of a prompt right and their combination wrong. GenEval’s authors found every model they tested weak at relative position and attribute binding, with best scores of 15% and 35% of images correct, from different models (Ghosh et al., 2023, Table 2). DALL·E 2’s authors found their model worse than GLIDE at binding two colors to two objects, and its decoder’s reconstructions mixed up attributes and objects; they suspect that the CLIP image embedding does not bind attributes to objects explicitly (Ramesh et al., 2022, §7). SDXL’s authors call such leaks “concept bleeding” (a penguin asked for a blue hat and red gloves gets blue gloves and a red hat) and suggest that text encoders compressing everything into one token may contribute (Podell et al., 2024, Appendix B, arXiv v1).

Rendering text. DALL·E 2 struggled to write coherent text in images; its authors suggest that the CLIP embedding does not encode spelling precisely and that byte-pair encoding hides the letters of each word from the model (Ramesh et al., 2022, §7). SDXL still had difficulty with long, legible text (Podell et al., 2024, Appendix B). In Stable Diffusion 3’s human evaluation, removing its T5 text encoder at inference left aesthetic quality unchanged (50% win rate) and cost a little prompt adherence (46%) and more in rendering written text (38%) (Esser et al., 2024, §5.3.3, arXiv v1).

People and fine structure. Imagen’s photorealism preference rate against real COCO images rose when prompts with people were removed, which its authors read as limited ability to generate photorealistic people (Saharia et al., 2022, §4.2); SDXL’s authors list hands among its failures (Appendix B).

Bias and harmful content. Imagen’s authors cite an audit of LAION-400M that found pornographic imagery, racist slurs and harmful social stereotypes, report a preliminary finding of bias toward lighter skin tones and Western gender stereotypes for professions, and did not release the model (Saharia et al., 2022, §6). Bianchi et al. (2023) found that ordinary prompts made Stable Diffusion amplify demographic stereotypes beyond U.S. labor statistics, and that neither prompts requesting counter-stereotypes nor DALL·E’s guardrails prevented it; the Stable Diffusion article gives their figures. LAION-5B was withdrawn in December 2023 over links to suspected child sexual abuse material and re-released in 2024, as the CLIP article describes.

Memorization. Carlini et al. (2023) extracted near-copies of training images from Stable Diffusion by sampling the captions of heavily duplicated training images; image generation gives the counts and how memorization is measured.

Copyright and consent. LAION-5B’s datasheet states that the people concerned were not notified, that people shown in photographs may not have consented to their upload, and that LAION does not own the copyright of the images (Schuhmann et al., 2022, datasheet, arXiv v1). Of the training images Carlini et al. extracted, 35% carried an explicit non-permissive copyright notice (§4.2.2). Two official documents address such training (as of October 2026). The U.S. Copyright Office’s pre-publication report says it is “not possible to prejudge litigation outcomes” and expects some training uses to qualify as fair use and some not (2025, §IV.E). In Getty Images v Stability AI, Getty abandoned its training claim, acknowledging no evidence that training took place in the UK, and its outputs claim after Stability blocked the prompts concerned (¶9). The High Court dismissed the remaining secondary copyright claim, holding that a model which does not store or reproduce the works is not an “infringing copy”, and found trade mark infringement in some generated watermarks, findings it called “both historic and extremely limited in scope” ([2025] EWHC 2863 (Ch), ¶¶757–758). In December 2025 the court granted Getty permission to appeal the copyright ruling and refused Stability permission to appeal the trade mark findings ([2025] EWHC 3343 (Ch)).

Approaches

GAN-based models (2016–2020)

Reed et al. (2016) conditioned a DCGAN-style generative adversarial network on text: its generator and discriminator both receive a pretrained character-level text embedding, producing $64 \times 64$ images of birds (CUB) and flowers (Oxford-102). Their matching-aware discriminator (GAN-CLS) also sees real images with mismatched captions, so it must judge whether an image fits its text, not only whether it looks real. StackGAN (Zhang et al., 2017) added a second, text-conditioned stage that redraws a $64 \times 64$ sketch at $256 \times 256$ and raised the Inception Score on CUB from 2.88 to 3.70 over Reed et al. (Table 1). AttnGAN (Xu et al., 2018) let refinement stages attend to individual words and added a word-level image–text matching loss, raising the best reported Inception Score on COCO from 9.58 to 25.89 (Table 3).

Autoregressive models (2021–2022)

DALL·E (Ramesh et al., 2021) recast the task as sequence modeling: a discrete VAE compresses each $256 \times 256$ image into a $32 \times 32$ grid of tokens from a vocabulary of 8,192, and a 12-billion-parameter transformer models the concatenated text and image tokens autoregressively, trained on 250 million image–text pairs from the internet. Without training on COCO it reached a COCO FID within 2 points of the best GAN, and human raters preferred its samples to DF-GAN’s for realism 90.0% of the time and for matching the caption 93.3% (§3.1, Figs. 7 and 9). Parti (Yu et al., 2022) treated generation as translation with an encoder–decoder transformer over ViT-VQGAN tokens, scaled it to 20 billion parameters and reached a zero-shot COCO FID of 7.23 (Table 5, arXiv v1).

Diffusion models (2021–)

Three pixel-space diffusion model systems followed between December 2021 and May 2022. GLIDE (Nichol et al., 2022) conditioned a 3.5-billion-parameter diffusion model on text, found classifier-free guidance better than CLIP guidance in human evaluations, and was preferred to DALL·E even when DALL·E’s samples were CLIP-reranked (§5.2, Table 3). DALL·E 2, or unCLIP (Ramesh et al., 2022), splits the task in two: a prior generates a CLIP image embedding from the caption, and a diffusion decoder generates an image from that embedding; the authors report more diverse samples than GLIDE at similar photorealism. Imagen (Saharia et al., 2022) conditioned a cascade of diffusion models on a frozen text-only language model, T5-XXL (Raffel et al., 2020), and found scaling the text encoder much more effective than scaling the U-Net (§4.4).

Latent diffusion (Rombach et al., 2022) moved the diffusion process into the latent space of a pretrained autoencoder, which made training and sampling far cheaper; its 1.45-billion-parameter text-to-image model, trained on LAION-400M, was on par with GLIDE’s COCO FID with far fewer parameters (§4.3.1, Table 2). Stable Diffusion applied the design to subsets of LAION-5B and released the weights (CompVis README). SDXL (Podell et al., 2024) enlarged the U-Net and added a second text encoder, and Stable Diffusion 3 (Esser et al., 2024) replaced the U-Net with a transformer over joint text and image tokens trained as a rectified flow; the latent diffusion model article describes these architectures.

Text encoders and conditioning

Three choices of text encoder recur:

• A CLIP text encoder, trained contrastively with images, as in Stable Diffusion v1 (frozen CLIP ViT-L/14) and DALL·E 2. Its embeddings are aligned with images, but such contrastive models often behave like bags of words, insensitive to word order and to which attribute goes with which object (Yuksekgonul et al., 2023).
• A text-only language model, as in Imagen’s frozen T5-XXL. In Imagen’s comparison, T5-XXL and CLIP encoders gave similar COCO scores, but human raters preferred the T5-XXL model on all eleven DrawBench categories (Saharia et al., 2022, §4.4).
• Both. SDXL concatenates two CLIP text encoders, and Stable Diffusion 3 adds T5-XXL to two CLIP encoders, dropping each one independently during training so that any subset can be used at inference (Esser et al., 2024, §5.3.3 and Appendix B.3, arXiv v1).

The per-token embeddings usually enter the denoiser through cross-attention: queries come from image features, keys and values from the text tokens, so each image location can draw on the words relevant to it (see self-attention). Imagen found cross-attention over the token sequence clearly better than pooling it into one vector (§4.4). GLIDE appended the text tokens to the attention context of each layer (§4.1), and Stable Diffusion 3 joins text and image tokens in one attention sequence. Classifier-free guidance is used for the COCO results of GLIDE, DALL·E 2, Imagen and latent diffusion, and Parti applies it to autoregressive sampling (Yu et al., 2022, §2.4).

Datasets and Benchmarks

Training data. Web-scale image–alt-text pairs replaced curated datasets. LAION-5B (Schuhmann et al., 2022) contains 5.85 billion pairs from Common Crawl, 2.32 billion of them English, kept when OpenAI’s CLIP ViT-B/32 gave an English pair a cosine similarity of at least 0.28, or a multilingual CLIP ViT-B/32 gave a pair in another language at least 0.26; the filter removed about 90% of some 50 billion candidates (§3.1). It is distributed as URLs with metadata, and its authors note that curating with CLIP also brings in CLIP’s flaws and biases (§6). Conceptual 12M (Changpinyo et al., 2021) relaxed the filters of Conceptual Captions to collect 12.4 million pairs (Table 1). WebLI (Chen et al., 2023), built at Google for the PaLI vision–language model, holds 10 billion images and 12 billion alt-texts in 109 languages (§3.2). Imagen and Parti combined LAION-400M with internal Google data (Saharia et al., 2022, §4.1; Yu et al., 2022, §4.1).

Evaluation prompts. COCO captions, short descriptions of everyday scenes, are the standard set for FID. DrawBench (Saharia et al., 2022) has 200 prompts in 11 categories, including colors, counting, positional relations, rare words, misspellings and quoted text, for side-by-side human comparison. PartiPrompts (Yu et al., 2022) has over 1,600 English prompts labelled separately by 12 categories and 11 challenge aspects. GenEval (Ghosh et al., 2023) generates 553 prompts from templates over the 80 COCO object classes, in six tasks: single object, two objects, counting, colors, position and attribute binding.

Evaluation Metrics

Zero-shot FID on COCO. Following DALL·E and earlier GAN work, papers generate images for 30,000 captions drawn from the COCO 2014 validation set, at $256 \times 256$, and compute FID against real validation images (Saharia et al., 2022, §3; Yu et al., 2022, §5.2). The protocol has known problems:

• The reference set differs: GLIDE and Imagen compare against the full validation set, Parti against 30,000 real images, and Rombach et al. against 30,000 validation images with the torch-fidelity implementation (Appendix E.3.2, arXiv v2).
• Training data overlap: about 21% of COCO validation images had near-duplicates in DALL·E’s training data, although removing them changed its FID little (Ramesh et al., 2021, §3.1).
• DALL·E and Parti rerank several samples per caption.
• FID ignores the prompt. Retrieving the nearest training image for each caption already gave a zero-shot FID of 17.97 in Parti’s comparison (Table 5).
• It disagrees with human judgment for modern models. Over nine model and guidance-scale settings on COCO captions, win rates by FID correlated with experts’ at −0.900; the authors hypothesize that larger guidance scales give vivid images people prefer but unlike COCO’s photos (Kirstain et al., 2023, §5). SDXL’s FID was worse than Stable Diffusion 1.5’s and 2.1’s although raters clearly preferred it (Podell et al., 2024, Appendix F).

CLIP score. CLIPScore was introduced to evaluate image captions without references: $2.5 \cdot \max(\cos(\mathbf{c}, \mathbf{v}), 0)$ for the CLIP ViT-B/32 embeddings of a caption $\mathbf{c}$ and an image $\mathbf{v}$ (Hessel et al., 2021, §3). Text-to-image papers report mean prompt–image CLIP similarity with varying scalings and CLIP models; GLIDE multiplies by CLIP’s logit scale (§5.2), SDXL’s authors use OpenCLIP ViT-g/14 (Appendix F). It inherits CLIP’s weaknesses: Imagen’s authors note that CLIP is ineffective at counting (§3). Thresholded CLIPScore agreed with human annotators less often than GenEval on counting, position and attribute binding, most clearly on position (Ghosh et al., 2023, §4, Fig. 3). GLIDE’s CLIP-guided samples scored higher CLIP scores than classifier-free-guided ones while humans judged them worse at matching the prompt; its authors hypothesize that CLIP guidance finds adversarial examples for the evaluating CLIP model (Nichol et al., 2022, §5.2).

Structured checks. GenEval runs a COCO-trained Mask2Former detector on each image, checks the presence and count of the prompted objects, compares box centroids for relative position, and classifies the color of each masked object with CLIP; an image counts as correct only if every element is right. In a study of 6,000 annotations on 1,200 images, it agreed with human annotators on 83% of images, against 88% between annotators and 80% for thresholded CLIPScore (Ghosh et al., 2023, §4). Stable Diffusion 3’s largest model scored 0.74 overall at $1024^2$ after preference tuning (Esser et al., 2024, Table 5, arXiv v1).

Human evaluation remains the reference. Imagen’s raters chose between generated and real COCO images for photorealism; Table 2 prints a 39.5% preference for Imagen’s images and 43.9% on prompts without people, while the text gives 39.2% and 43.6% (Saharia et al., 2022, §4.2, in both the NeurIPS and arXiv v1 versions). PickScore, a preference model trained on over 500,000 choices by users of a generation web app, predicted held-out user preferences with 70.5% accuracy, against 68.0% for human experts (Kirstain et al., 2023, Table 1).

The zero-shot COCO FIDs below are printed in the papers named; protocols differ as described above, so in our reading differences of a point or two are not meaningful.

Model	Year	Family	Zero-shot FID-30K	Source (table, version)
DALL·E	2021	Autoregressive	∼28	Nichol et al., Table 2 (ICML); Imagen’s Table 1 prints 17.89
GLIDE	2022	Pixel diffusion	12.24	Nichol et al., Table 2 (ICML)
LDM-KL-8-G	2022	Latent diffusion	12.63	Rombach et al., Table 2 (arXiv v2)
DALL·E 2 (unCLIP)	2022	Pixel diffusion, CLIP prior	10.39	Ramesh et al., Table 2 (arXiv v1)
Imagen	2022	Pixel diffusion	7.27	Saharia et al., Table 1 (NeurIPS)
Parti (20B, CoCa-reranked)	2022	Autoregressive	7.23	Yu et al., Table 5 (arXiv v1)
Retrieval baseline	2022	Nearest training image	17.97	Yu et al., Table 5 (arXiv v1)

DALL·E’s own paper shows its FID only as curves (Ramesh et al., 2021, Fig. 9). GLIDE’s, DALL·E 2’s and Parti’s tables print “∼28” for it and Imagen’s prints 17.89; we found no explanation of the difference in these papers.

Example: A Structured Check Catches What a Bag of Attributes Misses

The data are $16 \times 16$ RGB images of two colored shapes on a noisy dark background; prompts come from an invented grammar, “⟨color⟩ ⟨shape⟩ ⟨relation⟩ ⟨color⟩ ⟨shape⟩”, with three colors, three shapes and four relations, 144 prompts in all. Five stand-in generators each answer every prompt in one fixed way: one is correct, one ignores the relation and arranges the two objects at random, one swaps the bindings by exchanging the two colors, one drops the second object, and one ignores the prompt. A rule-based detector finds colored components and classifies each shape by how much of its bounding box it fills. Two scores follow: a GenEval-like structured check that both shapes are present, carry their prompted colors and satisfy the relation; and a CLIP-score-like bag cosine between counts of color and shape words in the prompt and of colors and shapes detected in the image.

import itertools
import numpy as np
from scipy import ndimage

# Invented caption grammar: "<color> <shape> <relation> <color> <shape>"
COLORS = {"red": (0.9, 0.1, 0.1), "green": (0.1, 0.8, 0.1), "blue": (0.1, 0.2, 0.9)}
SHAPES = ["square", "circle", "triangle"]
RELS = ["left of", "right of", "above", "below"]
PROMPTS = [(c1, s1, r, c2, s2)
           for c1, c2 in itertools.permutations(COLORS, 2)
           for s1, s2 in itertools.permutations(SHAPES, 2) for r in RELS]   # 144 prompts
yy, xx = np.mgrid[-2:3, -2:3]
MASK = {"square": np.ones((5, 5), bool),
        "circle": yy**2 + xx**2 <= 6.25,
        "triangle": np.abs(xx) <= (yy + 2) / 2}       # apex at the top

def render(objs, rng):
    """objs: list of (color, shape, row, col) centers -> 16x16x3 image in [0, 1]."""
    img = rng.normal(0.05, 0.05, (16, 16, 3))
    for c, s, r, q in objs:
        img[r - 2:r + 3, q - 2:q + 3][MASK[s]] = COLORS[c]
    return np.clip(img + rng.normal(0, 0.05, img.shape), 0, 1)

def layout(rel, rng):
    """Centers (row, col) of object A and B satisfying 'A rel B'."""
    near, far, free = rng.integers(2, 5), rng.integers(11, 14), rng.integers(2, 14, 2)
    if rel == "left of":  return (free[0], near), (free[1], far)
    if rel == "right of": return (free[0], far), (free[1], near)
    if rel == "above":    return (near, free[0]), (far, free[1])
    return (far, free[0]), (near, free[1])

# Stand-in "generators": each draws an image for a prompt in one fixed way
def correct(p, rng):
    c1, s1, r, c2, s2 = p; a, b = layout(r, rng)
    return render([(c1, s1, *a), (c2, s2, *b)], rng)
def ignores_relation(p, rng):           # right objects, random arrangement
    c1, s1, _, c2, s2 = p; a, b = layout(RELS[rng.integers(4)], rng)
    return render([(c1, s1, *a), (c2, s2, *b)], rng)
def swaps_binding(p, rng):              # right layout, colors swapped between objects
    c1, s1, r, c2, s2 = p; a, b = layout(r, rng)
    return render([(c2, s1, *a), (c1, s2, *b)], rng)
def drops_object(p, rng):               # draws only the first object
    c1, s1, r, c2, s2 = p; a, _ = layout(r, rng)
    return render([(c1, s1, *a)], rng)
def ignores_prompt(p, rng):             # random colors and shapes, random arrangement
    c = list(COLORS)[rng.integers(3)], list(COLORS)[rng.integers(3)]
    s = rng.permutation(SHAPES)[:2]; a, b = layout(RELS[rng.integers(4)], rng)
    return render([(c[0], s[0], *a), (c[1], s[1], *b)], rng)

def parse(img):
    """Rule-based detector: color by dominant channel, shape by fill of the bounding box."""
    objs = []
    for c, rgb in COLORS.items():
        k = int(np.argmax(rgb))
        m = (img[..., k] > 0.5) & (img[..., k] - img.max(-1, initial=0, where=np.arange(3) != k) > 0.3)
        lab, n = ndimage.label(m)
        for sl, i in zip(ndimage.find_objects(lab), range(1, n + 1)):
            area = (lab[sl] == i).sum()
            if area < 6: continue
            fill = area / lab[sl].size
            shape = "square" if fill > 0.92 else "circle" if fill > 0.7 else "triangle"
            r, q = ndimage.center_of_mass(lab == i)
            objs.append((c, shape, r, q))
    return objs

def holds(rel, a, b, margin=2):         # a, b: (row, col) centroids
    return {"left of": a[1] < b[1] - margin, "right of": a[1] > b[1] + margin,
            "above": a[0] < b[0] - margin, "below": a[0] > b[0] + margin}[rel]

def structured(p, objs):
    """GenEval-style check: both shapes present, colors bound to them, relation holds."""
    c1, s1, r, c2, s2 = p
    shapes = {s1, s2} <= {o[1] for o in objs}
    A = [o for o in objs if o[:2] == (c1, s1)]; B = [o for o in objs if o[:2] == (c2, s2)]
    bound = bool(A and B)
    rel = bound and holds(r, A[0][2:], B[0][2:])
    return shapes, bound, rel

VOCAB = list(COLORS) + SHAPES
def bag_cosine(p, objs):
    """CLIP-score stand-in: cosine of bag-of-attribute counts, caption vs detected objects."""
    t = np.array([sum(w == v for w in (p[0], p[1], p[3], p[4])) for v in VOCAB], float)
    i = np.array([sum(v in o[:2] for o in objs) for v in VOCAB], float)
    return t @ i / (np.linalg.norm(t) * np.linalg.norm(i) + 1e-12)

GENS = [correct, ignores_relation, swaps_binding, drops_object, ignores_prompt]
print(f"{len(PROMPTS)} prompts, e.g. '{' '.join(PROMPTS[0])}'")
print(f"{'generator':18s}{'shapes':>8s}{'bound':>7s}{'all correct':>13s}{'bag cosine':>12s}{'cos=1':>7s}")
summary = {g.__name__: [] for g in GENS}
for seed in range(5):
    rng = np.random.default_rng(seed)
    for g in GENS:
        res, cos = [], []
        for p in PROMPTS:
            objs = parse(g(p, rng))
            res.append(structured(p, objs)); cos.append(bag_cosine(p, objs))
        res, cos = np.array(res).mean(0), np.array(cos)
        summary[g.__name__].append((res[2], cos.mean()))
        if seed == 0:
            print(f"{g.__name__:18s}{res[0]:8.2f}{res[1]:7.2f}{res[2]:13.2f}{cos.mean():12.3f}{(cos > 0.999).mean():7.2f}")
print(f"{'seeds 0-4':18s}{'all correct':>13s}{'bag cosine':>17s}")
for name, v in summary.items():
    v = np.array(v)
    print(f"{name:18s}{v[:, 0].min():6.2f} - {v[:, 0].max():.2f}{v[:, 1].min():9.3f} - {v[:, 1].max():.3f}")

Output (NumPy 2.3.5, SciPy 1.16.3, under a second):

144 prompts, e.g. 'red square left of green circle'
generator           shapes  bound  all correct  bag cosine  cos=1
correct               1.00   1.00         1.00       1.000   1.00
ignores_relation      1.00   1.00         0.40       1.000   1.00
swaps_binding         1.00   0.00         0.00       1.000   1.00
drops_object          0.00   0.00         0.00       0.707   0.00
ignores_prompt        0.29   0.05         0.02       0.629   0.09
seeds 0-4           all correct       bag cosine
correct             1.00 - 1.00    1.000 - 1.000
ignores_relation    0.38 - 0.55    1.000 - 1.000
swaps_binding       0.00 - 0.00    1.000 - 1.000
drops_object        0.00 - 0.00    0.707 - 0.707
ignores_prompt      0.01 - 0.03    0.618 - 0.645

The structured check separates all five generators. The bag cosine gives the correct, relation-ignoring and color-swapping generators the same 1.000 on every prompt and seed, since a red square left of a green circle and a green square left of a red circle contain the same four attributes; it notices only missing or wrong objects. The relation-ignoring generator is right on 38% to 55% of prompts over seeds 0 to 4, more than one in four, because a random placement along one axis can also satisfy a relation along the other. A real CLIP embedding is not a literal bag of words, but it can behave like one (see Text encoders above), and GenEval’s authors motivate their benchmark by noting that CLIPScore gives only a holistic measure (Ghosh et al., 2023, abstract).

Applications

Text-to-image models serve as creative and design tools: AttnGAN’s authors motivated the task by art generation and computer-aided design (Xu et al., 2018, §1), and Imagen’s expected new tools for creative practitioners (Saharia et al., 2022, §6). GenEval’s authors note research uses such as generating training data (Ghosh et al., 2023, §1). The same models are extended to editing, inpainting and image variations.

Open Problems

• Compositional prompts. Position, attribute binding and counting remained the weakest GenEval categories in 2024: Stable Diffusion 3’s best model scored 0.33 on position, against 0.99 for single objects (Esser et al., 2024, Table 5, arXiv v1).
• Evaluation. Each automatic measure above misses something: FID the prompt, CLIP score binding and counting, detector-based checks any object or attribute their detectors do not know.
• Text rendering, especially long passages (Podell et al., 2024).
• Bias, consent and memorization of training data, discussed under Challenges.