Controllable Image Generation

Controllable image generation steers a pretrained text-to-image model with information that a prompt carries badly or not at all: where objects go (boxes, layouts, per-region prompts), what shape they take (sketches, edges, segmentation maps), how the scene is arranged in depth or how a body is posed, and what an image should look like (a reference image or a style). The methods surveyed here are built almost entirely on diffusion models, most of them on Stable Diffusion. They fall into three families: models trained with the extra condition as an input, trainable add-ons to a frozen model such as ControlNet, and training-free methods that change only how a frozen model is sampled. Teaching a model a new subject from a few photographs is a related problem with its own article, personalized image generation; changing one given image is image editing.

Purpose

A caption underspecifies its image, and a text encoder that compresses a sentence into embeddings does not reliably keep where things are or which attribute belongs to which object. The text-to-image generation article documents this as the compositionality problem: models get the parts of a prompt right and their arrangement wrong. Two groups of authors put the case for other inputs in similar terms. Make-A-Scene’s authors argue that structure, form and arrangement “can only be loosely described” in text (Gafni et al., 2022, §1), and Composable Diffusion’s authors suggest that a fixed-size text encoding cannot hold arbitrarily complex descriptions (Liu et al., 2022, §1). Controllable generation adds inputs that state the missing information directly: a box says where, an edge map says what outline, a depth map says how far, a reference image says what it looks like.

The second purpose is reuse. Training a large text-to-image model for each new kind of input is expensive, and paired data for a particular condition are small. Much recent work therefore keeps a pretrained model and adds control to it, by training a module or by changing the sampler; MultiDiffusion’s authors divide existing methods into training or fine-tuning a model for the task and reusing a pretrained one (Bar-Tal et al., 2023, §1).

When to Use It

Control inputs fit when the user knows something about the image that is easier to draw or point to than to describe: a composition laid out with boxes, a product’s outline, a depth map from a 3D scene, a pose skeleton, a semantic layout for a synthetic training set. They also help when a long prompt names several objects and some go missing, since per-region prompts give each object its own condition.

The choice between families is a trade between cost and fidelity. A trained module such as ControlNet needs paired data and GPU hours for each kind of condition, but then follows that condition closely at the cost of one extra network pass per step. Training-free methods need nothing but the frozen model and work for any condition that can be expressed through it, but they add optimization or extra network passes at sampling time and, in our reading, follow the condition more loosely. MultiDiffusion’s authors, for example, present region-based generation as working from rough masks and boxes, and add a separate bootstrapping phase to follow tight masks (§4.2, §5.2). When an image with the right structure already exists, editing it is an alternative to generating from a condition.

How It Works

The conditions

The same few kinds of control recur across papers, with different representations:

Kind of control	Typical representation	Examples
Layout and boxes	Box coordinates with a class or phrase per box	LDM layout-to-image, GLIGEN, LayoutDiffusion, MIGC, BoxDiff
Regions with prompts	Masks or boxes, each with its own text	MultiDiffusion, DenseDiffusion
Sketches and edges	Binary or soft edge maps, scribbles	ControlNet (Canny, HED, scribbles), Composer, Make-A-Scene (scene sketches)
Depth and pose	Depth maps from a monocular estimator; keypoint skeletons	ControlNet, GLIGEN keypoints, Composer
Segmentation maps	Per-pixel class maps	Make-A-Scene, ControlNet (ADE20K, COCO-Stuff), FreeDoM
Reference images and style	Image embeddings, colour histograms, style images	Composer, IP-Adapter, FreeDoM

The spatial conditions are usually produced by running a detector on training images: edge detectors, monocular depth estimators, pose estimators, segmentation networks and object detectors. The ControlNet article lists the detectors and datasets used for each of its conditions. Composer’s eight representations show how far the list extends: the caption, a CLIP image embedding for semantics and style, a smoothed CIELab colour histogram, a sketch from an edge detector followed by sketch simplification, instance masks from YOLOv5, a depth map, a grayscale intensity image and a masked image for region-restricted edits (Huang et al., 2023, §2.2).

Training a conditioned model

The most direct route trains the denoiser with the condition as an input. Spatially aligned conditions are usually concatenated to the noisy input or encoded into it; sets of boxes become tokens for cross-attention. The latent diffusion model article describes both forms in Rombach et al.’s models, including a transformer over bounding boxes for layout-to-image generation.

Make-A-Scene (Gafni et al., 2022) is an autoregressive transformer over text tokens, scene tokens and image tokens. The scene is the union of panoptic, human and face segmentations, tokenized by its own VQ autoencoder (VQ-SEG), and it is optional: at inference the transformer either generates the scene tokens itself or takes them from a segmentation map the user supplies or edits (§3.1–3.3). The authors also adapted classifier-free guidance to the transformer (§3.4).

Composer (Huang et al., 2023) trains one diffusion model on all eight of its representations at once. Global representations (sentence and image embeddings, colour palette) are projected into the timestep embedding and into extra cross-attention tokens; local ones (sketch, instance masks, depth, intensity, masked image) are each encoded by convolutions to the size of the noisy input, summed and concatenated to it (§2.3). Training drops each condition independently with probability 0.5, all conditions with probability 0.1 and none with probability 0.1, and drops the intensity image with probability 0.7 because it carries most of the image’s information. The model can then decode images from any subset or combination of representations, including representations taken from different images. The 2-billion-parameter base model generates $64 \times 64$ images, upsampled by two further models to $1024 \times 1024$, and with only the caption as condition the authors report a zero-shot COCO FID of 9.2 (§2.3, §3.1, §3.5).

LayoutDiffusion (Zheng et al., 2023) is a diffusion model for layout-to-image generation trained on the layout datasets COCO-Stuff and Visual Genome. It turns the image’s patches into a “structural image patch” with region information so that image and layout can be fused in one form, and adds a layout fusion module and object-aware cross-attention so that each object’s box and class act on its own region. Controllability is measured by CAS, which the paper calls Classification Score: a ResNet-101 classifier is trained on the generated images’ crops at the layout’s boxes and tested on real images (§4). At $256 \times 256$, its Table 1 gives FID 15.61 and CAS 47.74 on COCO-Stuff, against best previous values of 29.10 (PLGAN) and 40.00 (LostGAN-v2), and FID 15.63 and CAS 48.90 on Visual Genome, against 28.06 and 34.48. The abstract states relative improvements of 46.35% and 26.70% on COCO-Stuff and 44.29% and 41.82% on Visual Genome; all but the CAS figure on COCO-Stuff match the table, against whose best previous CAS the improvement is about 19%.

Trained models give the tightest control, but each needs a dataset of images paired with the condition, and a full training run.

Trainable modules on a frozen model

The second family keeps the pretrained text-to-image model frozen and trains a small or medium module that injects the condition. ControlNet copies the frozen model’s encoder, feeds the condition to the copy and adds its outputs to the frozen U-Net through zero-initialized convolutions; the ControlNet article covers it, together with T2I-Adapter and IP-Adapter, which pursue the same goal with smaller adapters, the latter for reference images rather than spatial maps.

GLIGEN (Li et al., 2023) adds grounding to a frozen model through a new gated self-attention layer in each transformer block, described in the ControlNet article. Its input is what it calls grounding tokens: for a box with a phrase, the phrase’s text feature and a Fourier embedding of the box corners joined by an MLP (Eq. 5). Keypoints are encoded the same way, and spatially aligned maps (edges, depth, normals, semantic maps) through convolutions. Trained on detection data, it is evaluated by running a detector on its outputs (see Evaluating Control below).

MIGC (Zhou et al., 2024) treats the generation of several instances, each with its own box and description, as a set of subtasks inside Stable Diffusion’s cross-attention layers: each instance is “shaded” separately, an added enhancement attention layer strengthens the shading of each instance, and a layout attention layer and an aggregation controller combine the instances with the background (§1, Fig. 2). The authors introduce COCO-MIG, a benchmark built on COCO that requires position, attribute and count to be right at once, and report raising the average instance success rate on it from 32.39% for GLIGEN, the best baseline, to 58.43% (§1, Table 1).

Readout Guidance (Luo et al., 2024) sits between this family and the next. It trains small readout heads that predict a property such as pose, depth, edges or appearance similarity from the frozen model’s internal features at every timestep, and then guides sampling by back-propagating the difference between a readout and a user’s target through the head. The authors describe this as needing significantly fewer added parameters and training samples than earlier conditioning methods, with as few as 100 training examples (abstract, §1).

Guidance on conditions

Guidance is the sampling-time tool behind several of these methods. Classifier-free guidance and classifier guidance, as that article explains, both add to the model’s prediction a term proportional to $\nabla_{x_t} \log p_t(c \mid x_t)$ for a condition $c$, from a separate classifier or from the difference between the conditional and unconditional predictions. With several conditions there are two natural generalizations.

Products of conditions. Composable Diffusion (Liu et al., 2022, §4.2) treats each conditional model as an energy-based model and composes concepts $c_1, \dots, c_n$ by multiplying their densities, assuming the concepts are conditionally independent given the image: $p(x \mid c_1, \dots, c_n) \propto p(x) \prod_i p(x \mid c_i)/p(x)$ (Eq. 10). In noise-prediction form the composed prediction is

$$\epsilon^\ast(x_t, t) = \epsilon_\theta(x_t, t) + \sum_{i=1}^{n} w_i \big[\epsilon_\theta(x_t, t \mid c_i) - \epsilon_\theta(x_t, t)\big],$$

their Eq. 11 with one weight per concept (the paper writes a single temperature $\alpha$). With one concept this is classifier-free guidance; a concept with a negative weight is negated (“NOT”), which their Eq. 13 derives from the same factorization. They composed several short sentences with a released GLIDE model (“A pink sky in the horizon” AND “A sailboat at the sea” AND “Overwater bungalows”) and report that the composed samples captured details, such as the overwater bungalows, more accurately than GLIDE given the same text as one long sentence (§6.2, Fig. 3).

Weights per condition set. Composer writes guidance between two sets of conditions $c_1$ and $c_2$ as $\hat\epsilon = \omega\, \epsilon_\theta(x_t, c_2) + (1 - \omega)\, \epsilon_\theta(x_t, c_1)$ (§2.1, Eq. 3): conditions only in $c_2$ are emphasized with weight $\omega$, those only in $c_1$ are suppressed, and shared ones keep weight 1. InstructPix2Pix’s two-scale guidance on an image and an instruction, in the image editing article, is another instance. Which formula is right depends on how the model was trained: Composable Diffusion’s product needs a model that knows each concept separately, while Composer’s form needs one trained with conditions dropped in subsets.

Guidance from any differentiable measure. If a pretrained network can score how well a clean image matches a condition (a segmentation network, a sketch extractor, a face or style encoder), its gradient can guide sampling without a classifier trained on noisy images. FreeDoM (Yu et al., 2023) defines an energy $E(c, x_t)$, approximates it by a distance $D_\theta(c, \hat x_0(x_t))$ computed on the denoiser’s estimate of the clean image, $\hat x_0(x_t) = \mathbb{E}[x_0 \mid x_t]$, and subtracts $\rho_t \nabla_{x_t} D_\theta(c, \hat x_0(x_t))$ from each unconditional sampling step (§4.1, Eqs. 8–10). On large data domains such as ImageNet this guidance alone often failed, which the authors attribute to the unconditional direction having more freedom there, so they adopt an existing time-travel strategy: the current sample is noised back by one step and resampled, $r_t$ times at each step, during an intermediate “semantic” stage of sampling (§4.2). The conditions demonstrated include segmentation maps, sketches, landmarks, style images and CLIP-based text guidance (Fig. 1).

Training-free control through attention and features

Large text-to-image models already decide much of the layout in their attention maps, and a family of training-free methods edits those maps during sampling.

• BoxDiff (Xie et al., 2023) takes, at each denoising step, the cross-attention map of each word that has a user box, and applies three constraints to it: an inner-box constraint that pushes the strongest responses inside the box, an outer-box constraint that lowers them outside, and a corner constraint that matches the map’s projections on the two image axes to the box’s, so that the object fills the box instead of a smaller part of it. The constraints’ gradient updates the noisy latent, so the object gradually follows the box (§3). It accepts boxes or scribbles.
• DenseDiffusion (Kim et al., 2023) addresses dense captions, in which each phrase describes one region. It first shows that the layout of a generated image is closely related to both the cross-attention and the self-attention maps, then modulates both kinds of maps during the first denoising steps so that each phrase’s tokens attend inside its segment and image tokens attend mostly within their own segment, scaling the modulation by the range of the original attention scores and by the segment’s area (abstract, §3).
• FreeControl (Mo et al., 2024) works with any condition image, including ones no model was trained on. It generates a few “seed” images of the target concept with DDIM, takes self-attention features of the U-Net decoder along the way and finds by principal component analysis a small set of semantic bases; the condition image is inverted with DDIM, its projection onto these bases becomes a structure target, and an appearance guidance borrows visual detail from a sample generated without control (§4, Figs. 2–3).

Prompt-to-Prompt and plug-and-play feature injection, which edit attention or features to preserve the structure of an existing image, are described in the image editing article.

Regional prompting and MultiDiffusion

A different training-free idea runs the frozen model several times per step, once for each region or crop with its own prompt, and reconciles the results. MultiDiffusion (Bar-Tal et al., 2023) defines a new sampler $\Psi$ on top of a reference model $\Phi$, where $\Phi(I_t \mid y)$ is one denoising step of image $I_t$ with prompt $y$. Given maps $F_i$ that take a region or crop of the generated image $J_t$, prompts $y_i$ and non-negative per-pixel weights $W_i$, each step solves (§3, Eqs. 3–4)

$$J_{t-1} = \arg\min_{J} \sum_{i=1}^{n} \big\lVert W_i \otimes \big[F_i(J) - \Phi(F_i(J_t) \mid y_i)\big] \big\rVert^2 ,$$

where $\otimes$ is the pixel-wise product. When the $F_i$ are crops, this least-squares problem has a closed form: each pixel of $J_{t-1}$ is the weighted average of the steps that cover it (Eq. 5). The authors show that if every crop’s step could be followed exactly, each crop would trace an ordinary diffusion path of the reference model, so each crop would be distributed as a sample of that model (Proposition 3.1); in practice the steps of overlapping crops disagree and the average reconciles them.

Two applications follow from the choice of $F_i$ and $W_i$:

• Panoramas. $F_i$ are overlapping $64 \times 64$ crops of a wide Stable Diffusion latent, moved by 8 latent pixels, all with the same prompt and $W_i = 1$ (§4.1; Appendix C.1). Run independently from the same noise, the crops give unrelated images; fused, they give one coherent panorama (Fig. 3).
• Region-based generation. Every $F_i$ is the whole image, $W_i = M_i$ is a region mask, and $y_i$ is the region’s prompt (§4.2, Eq. 8): at each step the model is run once per prompt on the whole image, and each pixel follows the average of the steps of the regions that contain it. To follow tight masks, the authors add bootstrapping (Eq. 9): during the first 20% of the sampling process ($t > T_{\text{init}} = 800$ of 1,000), region $i$’s model sees only its own region, with the rest replaced by a noised random image of constant colour, so that each object is first formed inside its mask; afterwards each model sees the full image, which harmonizes the result (§4.2; Appendix C.2).

Because a DDIM step is an affine function of the predicted noise for a fixed input, averaging the regions’ steps by masks equals averaging their noise predictions by the same masks whenever every region sees the same $J_t$, that is, outside bootstrapping (our observation from the update formula).

How to Apply It

The papers suggest a sequence of decisions:

1. Pick the representation of control. Boxes and region prompts are cheap to draw and leave shapes free; edges, sketches, depth and segmentation maps fix shapes and must usually come from a detector or a 3D tool; reference images carry appearance rather than layout.
2. Check whether a trained module exists for that condition and the base model. ControlNets, adapters and GLIGEN-style layers are tied to the model they were trained on, as the ControlNet article explains, and give the closest adherence at the cost of one more network pass per step.
3. Otherwise, use a training-free method that matches the condition. Region prompts suit MultiDiffusion-style fusion or attention modulation; boxes suit BoxDiff-style attention constraints; an arbitrary condition that a pretrained network can measure suits energy guidance such as FreeDoM; an arbitrary condition image suits feature-based guidance such as FreeControl.
4. Set the strength. Guidance weights, control scales, bootstrapping lengths and the number of time-travel repetitions trade adherence against image quality, and the papers tune them per task.
5. Measure control, not only quality. Re-extract the condition from the generated images and compare it with the input (see Evaluating Control).

Example: composing two regions with an exact model

The script below compares global prompting with mask-based regional composition on a model small enough to be exact. Its “pretrained model” is a Gaussian mixture over $16 \times 32$ canvases: each canvas has one object (disk, square or triangle) in each $16 \times 16$ half, at one of four places, on a background level of $-0.9$ or $-0.3$ shared by the whole canvas, plus pixel noise of standard deviation 0.2, giving 288 equally likely components. Its captions name objects but never places: “disk” means any canvas containing a disk, “disk and square” any canvas containing that pair in either order. Every conditional is a sub-mixture, so the noise predictor is exact, as in the classifier-free guidance and Stable Diffusion examples. The user wants a disk on the left and a square on the right.

Five ways of sampling, all with 50 deterministic DDIM steps (see diffusion sampling) under Stable Diffusion v1’s noise schedule and from the same 300 starting noises:

• exact layout model: the sub-mixture with the disk on the left and the square on the right, a condition no caption of this model can express; it is the ceiling;
• global caption: “disk and square”;
• independent paths: the “disk” sampler and the “square” sampler run separately on the whole canvas, and the left half of the first is pasted beside the right half of the second (MultiDiffusion’s Fig. 3a contrast);
• MultiDiffusion: Eq. 8 with the two halves as masks, “disk” on the left and “square” on the right;
• + bootstrapping: the same with Eq. 9 for $t > 800$.

Each half of each sample is classified by its nearest template. The script prints the share of samples with a disk on the left, a square on the right, both (layout), and the two named objects in either order (named). It also prints the seam, the mean absolute jump between the two background columns on either side of the boundary. A row of samples drawn directly from the target components checks the classifier, and a line of statistics checks the sampler without a caption.

import sys
import numpy as np

seed = int(sys.argv[1]) if len(sys.argv) > 1 else 0

# ---- The "pretrained model": an exact Gaussian mixture over 16x32 canvases. Each canvas
# has one object (disk, square or triangle, pixels +1) in each 16x16 half, at one of four
# places, on a background level shared by the whole canvas, plus pixel noise of std 0.2.
H, W = 16, 32
yy, xx = np.mgrid[:16, :16]
def obj(k, cy, cx):
    if k == 0: return (yy - cy) ** 2 + (xx - cx) ** 2 <= 16                        # disk
    if k == 1: return (abs(yy - cy) <= 3) & (abs(xx - cx) <= 3)                    # square
    return (yy >= cy - 3) & (yy <= cy + 3) & (2 * abs(xx - cx) <= yy - (cy - 3))  # triangle
names, places, bgs, sigma = ["disk", "square", "triangle"], [(5, 5), (5, 8), (10, 5), (10, 8)], [-0.9, -0.3], 0.2
half = {(k, p, b): np.where(obj(k, *places[p]), 1.0, bgs[b])
        for k in range(3) for p in range(4) for b in range(2)}
comps = [(kl, pl, kr, pr, b) for kl in range(3) for pl in range(4)
         for kr in range(3) for pr in range(4) for b in range(2)]               # 288 components
M = np.array([np.hstack([half[kl, pl, b], half[kr, pr, b]]).ravel() for kl, pl, kr, pr, b in comps])
KL, KR = np.array([c[0] for c in comps]), np.array([c[2] for c in comps])

# Captions name classes, never places: p(x | caption) keeps the components whose two
# objects include the caption's classes. "exact" (left disk, right square) is the ceiling
# that a caption cannot express.
def only(sel): return np.where(sel, 0.0, -np.inf)                               # log weights
CAP = {"": only(KL >= 0),
       "disk": only((KL == 0) | (KR == 0)), "square": only((KL == 1) | (KR == 1)),
       "disk and square": only(((KL == 0) & (KR == 1)) | ((KL == 1) & (KR == 0))),
       "exact": only((KL == 0) & (KR == 1))}

T = 1000
ab = np.cumprod(1 - np.linspace(0.00085 ** 0.5, 0.012 ** 0.5, T) ** 2)     # Stable Diffusion v1
ts = np.linspace(T - 1, 0, 50).round().astype(int)                          # 50 DDIM steps

def eps(x, a, lw):
    """Exact noise prediction of the mixture restricted by the log weights lw."""
    v = a * sigma ** 2 + 1 - a
    d2 = (x ** 2).sum(1, keepdims=True) - 2 * np.sqrt(a) * x @ M.T + a * (M ** 2).sum(1)
    lg = lw - d2 / (2 * v)
    r = np.exp(lg - lg.max(1, keepdims=True)); r /= r.sum(1, keepdims=True)
    return np.sqrt(1 - a) * (x - np.sqrt(a) * r @ M) / v

def ddim_step(x, i, caption):                   # Phi(x_t | y): one deterministic DDIM step
    a = ab[ts[i]]
    a_next = ab[ts[i + 1]] if i + 1 < len(ts) else 1.0
    e = eps(x, a, CAP[caption])
    return np.sqrt(a_next) * (x - np.sqrt(1 - a) * e) / np.sqrt(a) + np.sqrt(1 - a_next) * e

left = np.zeros((H, W)); left[:, :16] = 1
masks = [left.ravel(), 1 - left.ravel()]         # region masks M_i, summing to 1

def sample(zT, caption):                         # ordinary sampling with one caption
    x = zT.copy()
    for i in range(len(ts)):
        x = ddim_step(x, i, caption)
    return x

def independent(zT, captions):                   # one path per region, pasted at the end
    return sum(m * sample(zT, c) for m, c in zip(masks, captions))

def multidiffusion(zT, captions, bootstrap=False, rng=None):
    """Eq. 8: each step is the mask-weighted average of the regions' DDIM steps. With
    bootstrapping (Eq. 9), while t > 800 each region's model sees its own region on a
    noised random constant background S_t instead of the rest of the canvas."""
    x = zT.copy()
    for i, t in enumerate(ts):
        new = 0
        for m, c in zip(masks, captions):
            inp = x
            if bootstrap and t > 800:
                S = np.sqrt(ab[t]) * rng.uniform(-1, 1, (len(x), 1)) \
                    + np.sqrt(1 - ab[t]) * rng.standard_normal(x.shape)
                inp = m * x + (1 - m) * S
            new = new + m * ddim_step(inp, i, c)
        x = new
    return x

# ---- Measures: each half is classified by its nearest half-template (class, place,
# background); the seam is the jump between the two background columns at the boundary.
tmpl = np.array([half[k, p, b].ravel() for k in range(3) for p in range(4) for b in range(2)])
t_cls, t_bg = np.repeat(np.arange(3), 8), np.tile([0, 1], 12)
def halves(x):
    im = x.reshape(-1, H, W)
    return [((im[:, :, s].reshape(len(x), 1, -1) - tmpl[None]) ** 2).sum(-1).argmin(1)
            for s in (slice(0, 16), slice(16, 32))]
def measures(x):
    il, ir = halves(x)
    l, r = t_cls[il] == 0, t_cls[ir] == 1
    named = (l & r) | ((t_cls[il] == 1) & (t_cls[ir] == 0))
    im = x.reshape(-1, H, W)
    seam = np.abs(im[:, :, 15].mean(1) - im[:, :, 16].mean(1)).mean()
    return l.mean(), r.mean(), (l & r).mean(), named.mean(), seam

n = 300
rng = np.random.default_rng(seed)
zT = rng.standard_normal((n, H * W))                               # shared by every method
target = np.flatnonzero((KL == 0) & (KR == 1))
x_data = M[rng.choice(target, n)] + sigma * rng.standard_normal((n, H * W))

print(f"seed {seed}; alpha_bar at t = 999: {ab[-1]:.4f}; {n} samples per row")
il, ir = halves(sample(zT, ""))
print("no caption: class shares left", np.round(np.bincount(t_cls[il], minlength=3) / n, 3),
      "right", np.round(np.bincount(t_cls[ir], minlength=3) / n, 3),
      f"| background -0.9 share {np.mean(t_bg[il] == 0):.3f} (data 0.500)")
print("method                    left disk  right square  layout  named  seam")
rows = [("data (target layout)", x_data),
        ("exact layout model", sample(zT, "exact")),
        ("global caption", sample(zT, "disk and square")),
        ("independent paths", independent(zT, ["disk", "square"])),
        ("MultiDiffusion", multidiffusion(zT, ["disk", "square"])),
        ("  + bootstrapping", multidiffusion(zT, ["disk", "square"], True, np.random.default_rng(seed + 1)))]
for name, x in rows:
    l, r, both, named, seam = measures(x)
    print(f"{name:24s}{l:9.3f}{r:14.3f}{both:8.3f}{named:7.3f}{seam:6.3f}")

Output (NumPy 2.3.5, 2 to 5 seconds; two runs gave identical output):

seed 0; alpha_bar at t = 999: 0.0047; 300 samples per row
no caption: class shares left [0.34 0.32 0.34] right [0.31  0.303 0.387] | background -0.9 share 0.230 (data 0.500)
method                    left disk  right square  layout  named  seam
data (target layout)        1.000         1.000   1.000  1.000 0.060
exact layout model          1.000         1.000   1.000  1.000 0.048
global caption              0.493         0.493   0.493  1.000 0.048
independent paths           0.607         0.597   0.337  0.413 0.062
MultiDiffusion              0.747         0.747   0.560  0.627 0.048
  + bootstrapping           0.757         0.743   0.570  0.640 0.049

The data row confirms that the measures recover the truth: every target sample is classified correctly, and its seam of 0.060 is the level that pixel noise alone produces. The exact layout model reaches the ceiling, every half correct, with a seam slightly below the data’s; in our reading, deterministic DDIM samples carry less pixel noise than the data.

The global caption does what it is asked. Every sample contains a disk and a square (named 1.000), but the caption cannot say where, so about half of them have the pair the wrong way round (layout 0.493). A sample has either the requested layout or its mirror image, which is why the left, right and layout columns are equal in that row.

Independent paths do worse than the global caption. Each path is an ordinary sample of a caption that does not fix places: the “disk” sampler’s canvases contain one of five pairs with a disk, $(d, d)$, $(d, s)$, $(d, t)$, $(s, d)$ and $(t, d)$, three of which have it on the left. That gives a left-disk share of 3/5, close to the printed 0.607, and the same for the right square. The two halves come from different samples, so the layout share is 0.337, close to the $0.6 \times 0.6 = 0.36$ expected if the two paths were independent. Fewer than half of the images contain both named objects, and the seam is the largest in the table.

MultiDiffusion raises the per-region shares to 0.747 and the layout to 0.560, with the seam of the exact model: both prompts’ models see the whole shared canvas at every step, so the two halves agree on one background. But its named column, 0.627, shows that more than a third of its images lack a disk or a square altogether. In our reading this follows from the construction: each half follows the step of a caption model that only knows its object is somewhere, and when that model’s own reading of the canvas puts its object in the other half, the half it controls renders whatever that reading has there. Bootstrapping is meant to prevent this, but here it changes every column by at most 0.013. It acts only on the steps with $t > 800$, the first 10 of the 50, which are also the noisiest ones; at $t = 999$ the signal coefficient $\bar\alpha$ is 0.0047, as printed. The two masks here are whole halves that already contain their objects, so the tightening to small masks that bootstrapping was designed for (Bar-Tal et al., Fig. 6) is not tested.

The no-caption line shows a bias that every row shares. The class shares are close to the data’s thirds, but only 23.0% of unconditional samples have the darker background, against half of the data. In our reading, this happens because $\bar\alpha$ at $t = 999$ is 0.0047 rather than 0: the model’s noisy images at that step still carry a trace of the background, while the sampler starts from noise of mean zero, which resembles the noisy version of the lighter background (nearer zero) more than the darker one.

The script takes the seed as its argument. Over seeds 0, 1 and 2, the layout share was 0.473–0.530 for the global caption, 0.303–0.357 for independent paths, 0.507–0.560 for MultiDiffusion and 0.523–0.570 with bootstrapping. The named share for MultiDiffusion was 0.580–0.627. The seam was 0.048–0.052 for the exact model and both MultiDiffusion rows against 0.062–0.074 for independent paths, and the darker-background share without a caption was 0.230–0.267.

Two changes were made to the planned example before its first run, after a pilot and by the editor’s decision. The noise schedule was changed from a linear one to Stable Diffusion v1’s, the schedule for which MultiDiffusion sets $T_{\text{init}} = 800$. The independent-paths baseline, the paper’s own contrast for fusion, was added. A second pilot then missed one planned check, that sampling without a caption gives each background level about half of the samples. By the editor’s decision the check was not redefined: the no-caption line and the printed $\bar\alpha$ were added before the first run so that the miss is reported as printed, and nothing else was changed.

Variants

• By where the condition enters: the input (concatenation, as in Composer’s local conditions and Stable Diffusion’s depth model), the attention context (boxes as tokens in GLIGEN and LDM), the skip connections of the U-Net (ControlNet), or only the sampler (guidance, attention edits, path fusion).
• By what is trained: the whole model (Make-A-Scene, Composer, LayoutDiffusion), an added module (ControlNet, GLIGEN, MIGC, readout heads), or nothing (MultiDiffusion, Composable Diffusion with an existing model, BoxDiff, DenseDiffusion, FreeDoM, FreeControl).
• By how several conditions combine: one model trained on all of them with dropout (Composer), sums of trained modules’ outputs (several ControlNets, in the ControlNet article), products of conditional densities (Composable Diffusion), or spatial fusion of per-region steps (MultiDiffusion).
• Refining a controller: ControlNet++ (Li et al., 2024) fine-tunes a ControlNet with a consistency loss: a pretrained discriminative model (for example a segmentation network) extracts the condition from the generated image, and its difference from the input condition is penalized. To avoid back-propagating through a full sampling chain, the authors noise a training image, denoise it in one step and compute the loss on that estimate (§3.2–3.3).

Best Practices

• Match the condition to what the user knows. Boxes and region prompts for composition, edges or depth for exact structure, reference images for appearance.
• Keep regional prompts consistent with each other. In MultiDiffusion’s region experiments, the background prompt is the full image caption and each foreground region gets “a {label}” (§5.2); every region’s model still sees the whole image after bootstrapping, so prompts that contradict each other compete across the whole canvas.
• Use bootstrapping-like phases for tight masks. MultiDiffusion’s bootstrapping raised its mask IoU on COCO from 0.18 to 0.26 (Table 2), making objects fit tight masks rather than appear somewhere inside them (Fig. 6).
• Evaluate with an independent extractor and report the real-image ceiling (see below).
• Report the guidance or control strength with its convention, as for classifier-free guidance.

Pitfalls

• Composition that is only local. Fusing per-region predictions makes each region follow its own prompt’s step, not a joint model of the scene; the example above shows a case where many images then miss one of the requested objects.
• Dependence on the reference model. MultiDiffusion’s authors note that their results inherit the model’s biases and failures: a vertical panorama of a waterfall prompt added a waterfall to every crop, which they try to overcome by adding a region constraint (§6, Fig. 9).
• Conflicting or redundant conditions. Composable Diffusion’s failures include concepts the base model does not understand, attribute confusion (“a red bear” for “a bear in a red forest”), and compositions that fail, mostly when objects are in the image centre; composing models trained on different datasets had limited success (§6.5, §7).
• Evaluators that the method optimizes. In our reading, a method trained or guided with a segmentation or CLIP model can score well on that same model without matching the condition any better. ControlNet++ trains with one segmentation network as its reward model and evaluates with another, Mask2Former, and compares three reward models under that fixed evaluator (§4, Table 5).
• A single seed or prompt. Training-free methods in particular are sensitive to the starting noise; MultiDiffusion’s authors name a bad seed or a biased prompt as a cause of failure (§6).

Evaluating Control

Control is evaluated by checking whether the condition can be recovered from the output, usually with a pretrained recognition model, alongside image quality (FID) and prompt agreement (CLIP score).

• Boxes and layouts. GLIGEN runs a pretrained YOLOv4 detector on generated images and scores the detected boxes against the input boxes with COCO average precision (the YOLO score). For open-vocabulary layouts on LVIS it uses a GLIP detector and reports the GLIP score, with GLIP on real images as an upper bound (Li et al., 2023, §5).
• Segmentation and regions. Generated images are segmented and compared with the input map by intersection over union: MultiDiffusion uses Mask2Former on COCO and reports the IoU of the original images, 0.43, as an upper bound (§5.2, Table 2); ControlNet++ uses mIoU on ADE20K and COCO-Stuff.
• Edges and depth. ControlNet++ re-extracts the condition from the generated images and reports the F1 score for Canny edges, SSIM for soft edges (HED and line art) and RMSE for depth (§4.1).
• Concepts. Composable Diffusion trains binary classifiers on real images for each concept and counts a generated image as correct when it shows every requested concept (§5.2).
• Users. Make-A-Scene and ControlNet rely on human ratings of quality and fidelity to the condition; the ControlNet article gives its user study.

Each automatic check measures the generator through another network’s errors. The real-image ceilings make this visible: on the same test sets real images reach only 0.43 IoU in MultiDiffusion’s protocol, 50.7 mIoU in ControlNet++’s and 25.2 GLIP AP in GLIGEN’s.

Evidence and Impact

Reported results, each on its paper’s own protocol:

• Layout-to-image. On COCO 2017 val, GLIGEN built on latent diffusion scored FID 21.04 and YOLO score AP/AP50/AP75 of 22.4/36.5/24.1, against FID 31.12 and 13.40/19.70/14.90 for LAMA, a layout-to-image model trained from scratch (Li et al., 2023, Table 2). On LVIS, its GLIP score ranged from 4.4 to 11.1 AP depending on training data, against 2.0 for LAMA and 25.2 for real images (Table 3). The authors attribute the gain to building on a large pretrained model rather than training from scratch.
• Regions without training. On 1,000 COCO validation images with two to four foreground objects (people excluded), MultiDiffusion on Stable Diffusion reached a mask IoU of 0.26 ± 0.12, against 0.16 for Stable Inpainting and 0.17 for Blended Latent Diffusion applied object by object, 0.18 without bootstrapping, and 0.43 for the real images (Bar-Tal et al., 2023, Table 2).
• Panoramas. For $512 \times 4608$ panoramas, crops of MultiDiffusion’s outputs scored FID 10.3 ± 4.8 against reference-model samples for the same prompt, against 45.5 for Stable Inpainting and 18.4 for Blended Latent Diffusion used to extrapolate outward; independent samples of the reference model scored 6.05 (Table 1, 2,000 samples for each of 8 prompts).
• Composition of concepts. On CLEVR, composing object positions, Composable Diffusion’s accuracy was 15.88 points above the best baseline (an energy-based model) with one component and 24.02 above it with three (Liu et al., 2022, §6.3, Table 1).
• Control of trained modules. ControlNet++ reports that ControlNet on SD 1.5 reached 32.55 mIoU on ADE20K segmentation-conditioned images, against 50.7 for real images under the same Mask2Former evaluation (§4.2; its ablation gives Mask2Former 56.01 mIoU on ADE20K, §4.3), and raised it to 43.64; its Table 1 shows gains of 0.1345 SSIM for line art (0.7054 to 0.8399) and 7.58 RMSE for depth (35.90 to 28.32). The abstract reports these as 11.1% mIoU, 13.4% SSIM and 7.6% RMSE, which are differences in the metrics’ own units.

In our reading, the pattern across these papers is that control learned from paired data (GLIGEN, ControlNet and its refinements) follows a condition more closely than training-free methods, while training-free methods extend control to any frozen model and any combination of conditions at sampling time. Few of these papers compare the two families on one protocol; on MIGC’s COCO-MIG, GLIGEN’s average instance success rate of 32.39% was above MultiDiffusion’s 20.12% (Zhou et al., 2024, Table 1). In the same reading, the field has moved from training conditioned models from scratch (Make-A-Scene, LayoutDiffusion) to adding control to shared base models, which lets one base model serve many kinds of control.

Risks

Control inputs make generated images more specific, including images of real scenes and people. MultiDiffusion’s authors name the risk of misuse to create fake or harmful content, and the biases that their method inherits from the underlying model, and call for research on identifying synthetic content (Appendix D). Pose- and face-related conditions raise the questions discussed in the deepfakes and personalized image generation articles.