ControlNet

ControlNet is a network architecture that adds a spatial condition, such as an edge map, a depth map, a segmentation map or a human pose skeleton, to a large pretrained text-to-image diffusion model without changing the model’s weights. Lvmin Zhang, Anyi Rao and Maneesh Agrawala of Stanford University introduced it in 2023 (ICCV 2023; the official repository’s news log begins in February 2023) and applied it to Stable Diffusion. The pretrained model is locked; a trainable copy of its encoder reads the condition, and its outputs are added to the locked model’s U-Net through zero convolutions, $1 \times 1$ convolutions whose weights and biases start at zero, so that training begins from the unchanged model. One ControlNet is trained per kind of condition. This article covers the architecture, its training and inference, ControlNet 1.0 and 1.1, and neighbouring adapters in outline; the latent diffusion article explains the model it attaches to.

Problem Addressed

Text prompts control what an image shows but are poor at saying where: exact layouts, poses, shapes and outlines are hard to express in words, and getting them right takes many rounds of prompt editing (Zhang et al., §1). Image-to-image translation models learn a mapping from a conditioning image to an output, but usually from scratch and per task. The authors’ aim was to reuse a model trained on billions of images and teach it new spatial conditions end to end.

Two obstacles stood in the way, as they describe them. Datasets for specific conditions are small, the largest usually about 100,000 pairs, which they put at 50,000 times smaller than LAION-5B; and directly fine-tuning a large model on so little data risks overfitting and catastrophic forgetting. Restricting the number or rank of trainable parameters reduces forgetting, but the authors argue that conditioning images with complex shapes and high-level semantics may need a deep network. ControlNet keeps the large model intact and uses a full-depth copy of its encoder as that deep network.

Architecture

Locked and trainable copies

Zhang et al. (§3.1) describe ControlNet on a single network block $\mathcal{F}(\cdot; \Theta)$, any unit such as a ResNet block or a transformer block, that maps a feature map $x \in \mathbb{R}^{h \times w \times c}$ to $y = \mathcal{F}(x; \Theta)$. To add a ControlNet, they lock (freeze) $\Theta$ and clone the block into a trainable copy with parameters $\Theta_c$, which receives an external conditioning vector $c$. The two are joined by zero convolutions $\mathcal{Z}(\cdot; \cdot)$ with parameters $\Theta_{z1}$ and $\Theta_{z2}$ (their Eq. 2):

$$y_c = \mathcal{F}(x; \Theta) + \mathcal{Z}\big(\mathcal{F}(x + \mathcal{Z}(c; \Theta_{z1}); \Theta_c); \Theta_{z2}\big).$$

The locked block keeps the model trained on billions of images; the trainable copy starts from the same weights, so it is a pretrained backbone for the new condition rather than a layer trained from scratch. The repository’s README puts it as: the locked copy preserves the model, the trainable copy learns the condition.

Zero convolutions

A zero convolution is a $1 \times 1$ convolution with weight and bias initialized to zero. Before training, both $\mathcal{Z}$ terms in the equation above are zero, so $y_c = y$ (Eq. 3): the network with a ControlNet attached computes exactly what the original computed. The authors’ reason is that random initial outputs would add “harmful noise” to the deep features of the locked model and, through the first zero convolution, to the trainable copy, damaging the pretrained backbone at the start of training (§1, §3.1).

A zero weight does not stop learning. For a $1 \times 1$ convolution with weight $W$ and bias $B$, the output at position $p$ and channel $i$ is $\mathcal{Z}(I)_{p,i} = B_i + \sum_j I_{p,j} W_{i,j}$, so at $W = 0$, $B = 0$ (supplementary §5)

$$\frac{\partial \mathcal{Z}_{p,i}}{\partial B_i} = 1, \qquad \frac{\partial \mathcal{Z}_{p,i}}{\partial W_{i,j}} = I_{p,j}, \qquad \frac{\partial \mathcal{Z}_{p,i}}{\partial I_{p,j}} = W_{i,j} = 0.$$

The gradient with respect to the layer’s input is zero, but the gradients of the weight and bias are not, as long as the input features $I$ and the gradient arriving from the loss are nonzero. One gradient step therefore makes $W$ nonzero, after which gradients also flow back through the layer into the trainable copy. The supplement calls this progressively growing the parameters “from zero to optimized values in a learned way”, and the repository’s FAQ answers the common objection that a zero weight blocks all gradients with the same scalar argument for $y = wx + b$. By the chain rule, the condition-side zero convolution $\mathcal{Z}(\cdot; \Theta_{z1})$ gets no gradient at the very first step either, since the gradient reaching it has to pass through the output zero convolutions; the example below prints this.

The paper places zero-initialized layers among related ideas (§2.1): Nichol and Dhariwal’s “zero module” for diffusion U-Nets, which it calls an extreme case of scaling initial weights, Stability AI’s use of zero weights in its model cards, and side-tuning, which blends a frozen network with an added one by a fixed schedule. The weight initialization article describes zero-initialized residual branches in classifiers, a similar device for starting a block as the identity.

Attaching to Stable Diffusion

Stable Diffusion’s denoiser is a U-Net with an encoder, a middle block and a skip-connected decoder; its encoder has 12 blocks at four resolutions of the $64 \times 64$ latent ($64$, $32$, $16$ and $8$), three blocks each (§3.2, Fig. 3). ControlNet makes a trainable copy of the 12 encoder blocks and the middle block, and adds the 13 outputs, each through its own zero convolution, to the U-Net’s 12 skip connections and to the middle block’s output. The cross-attention layers in the copy receive the same text embedding and timestep as the locked model. The figure uses Stable Diffusion 1.5, whose U-Net the authors say is the same as 2.1’s.

Because the locked encoder is frozen, no gradients are computed for it. On one NVIDIA A100 PCIe 40 GB, training Stable Diffusion with a ControlNet took about 23% more GPU memory and 34% more time per iteration than training Stable Diffusion alone (§3.2).

The condition encoder

Stable Diffusion runs in a $64 \times 64$ latent space, while conditions are $512 \times 512$ images. The paper (§3.2) maps a conditioning image $c_i$ to a feature map $c_f = \mathcal{E}(c_i)$ of latent size with a tiny network of four convolution layers with $4 \times 4$ kernels and stride 2, ReLU activations and 16, 32, 64 and 128 channels, initialized with Gaussian weights and trained jointly. The released code differs: its input_hint_block has eight $3 \times 3$ convolutions with SiLU activations (16, 16, 32, 32, 96, 96 and 256 channels, three of them with stride 2), ends in a zero-initialized convolution to the U-Net’s 320 channels, and adds the result after the copy’s first convolution rather than to its input (cldm/cldm.py, ControlNet 1.0 repository). The README notes that, unlike Stability AI’s depth model, which receives a $64 \times 64$ depth map, the depth ControlNet sees the full $512 \times 512$ map.

Training

The loss is the ordinary noise-prediction objective of the diffusion model, with the condition added to the network’s inputs (Eq. 5):

$$\mathcal{L} = \mathbb{E}_{z_0,\, t,\, c_t,\, c_f,\, \epsilon \sim \mathcal{N}(0, I)}\Big[\big\lVert \epsilon - \epsilon_\theta(z_t, t, c_t, c_f) \big\rVert_2^2\Big],$$

where $z_t$ is the noised latent at step $t$, $c_t$ the text prompt and $c_f$ the encoded condition. Only the ControlNet’s parameters are optimized. Half of the text prompts are replaced by empty strings during training, which the authors say increases the ControlNet’s ability to recognize semantics directly in the conditioning image, such as edges, poses or depth, in place of the prompt (§3.3).

Conditions and data

The supplement (§1.1, Table 1) lists the models trained for the paper, nearly all with batch size 32, learning rate $10^{-5}$ and no exponential moving average of the weights:

Condition	Pairs	Source	GPU hours	Base model
Canny edges	3M	internet	~600 A100	SD 1.5
Hough lines (M-LSD)	600K	Places2, BLIP captions	~150 A100	resumed from Canny
HED boundary	3M	internet	~300 A100	SD 1.5
User sketching	500K	internet	~150 A100	resumed from Canny
Human pose (OpenPifPaf)	200K	OpenPifPaf	~400 RTX 3090 Ti	SD 2.1
Human pose (OpenPose)	200K	OpenPose	~300 A100	SD 1.5
Semantic mask (COCO-Stuff)	164K	COCO, BLIP captions	~400 RTX 3090 Ti	SD 1.5
Semantic mask (ADE20K)	20K	ADE20K, BLIP captions	~200 A100	SD 1.5
Depth (MiDaS)	3M	internet	~500 A100	SD 1.5
Normal maps	25K	DIODE	~100 A100	SD 1.5
Cartoon line drawing	1M	internet	~300 A100	Waifu Diffusion

The conditions are made by running detectors on images: the Canny edge detector with random thresholds, a learned Hough-line detector, HED edge detection, MiDaS depth, OpenPose and OpenPifPaf pose estimators, and ADE20K and COCO-Stuff segmentation labels. Scribbles are synthesized from HED boundaries with random thresholds, masking, morphological transforms and non-maximum suppression. The supplement’s text and table disagree in places: its text gives the scribble model 300 GPU hours from SD 1.5 where the table has about 150 hours resumed from the Canny model, and its text gives the OpenPifPaf set 80K pairs against 200K in the table. For the two pose sets the table names the pose estimator in its source column, while the text says the images were found on the internet with it. The table’s values are given above. The RTX 3090 Ti models used a physical batch of 3 with six-fold gradient accumulation, and the ADE20K model a batch of 256.

Sudden convergence

Because the zero convolutions add nothing at first, the model produces high-quality images throughout training; it does not gradually learn to follow the condition but starts following it abruptly. The paper calls this the sudden convergence phenomenon and shows samples for an edge-map input at steps from 100 to 12,000, with step 6,133 marked as the point where the model suddenly follows the input (Fig. 4). The sources give different ranges for when it happens: “usually in less than 10K optimization steps” (§3.3), “typically somewhere between 5000 and 10000 steps” at learning rate $10^{-5}$ (supplementary §7), and “about 3k to 7k steps” in the repository’s training guide. That guide therefore advises spending a fixed compute budget on larger batches through gradient accumulation rather than on more steps after convergence, while noting that training after the sudden convergence also matters.

The repository’s tutorial trains on Fill50K, a toy set of 50,000 circle outlines paired with filled circles and colour prompts; after 4,000 steps at batch size 4, about 50 minutes on what the guide calls a PCIe 40 GB GPU, it shows sample results and says that larger batches and longer training would improve the filling. Two options there change the architecture: only_mid_control feeds only the middle-block output, and sd_locked = False also trains the locked model’s decoder blocks and output layer, which the guide calls dangerous for small datasets and pairs with a lower learning rate such as $2 \times 10^{-6}$.

Inference

The supplement’s defaults are DDIM sampling with 20 steps (see diffusion sampling) and a classifier-free guidance scale of 7.0 (the paper writes $\epsilon_{\text{prd}} = \epsilon_{\text{uc}} + \beta_{\text{cfg}}(\epsilon_{\text{c}} - \epsilon_{\text{uc}})$, §3.4, so the scale is $s$, with 1 meaning no guidance), with “ugly, low-quality” as the negative prompt for qualitative results and an empty one for comparisons (§1.2). The released demos expose a control strength that multiplies all 13 ControlNet outputs (control_scales in the code).

Guidance and the condition. With guidance, the condition can be fed to both the conditional and the unconditional prediction, or to the conditional one only. Zhang et al. (§3.4) found that feeding it to both removes the guidance when no prompt is given, while feeding it only to the conditional side makes guidance very strong. Their CFG resolution weighting adds the condition to the conditional prediction only and multiplies each of the 13 connections by $w_i = 64/h_i$, where $h_i$ is the block’s resolution (8 to 64). The released code’s guess mode (also called non-prompt mode) does something similar with different numbers: the unconditional branch gets no control, and the scales are $0.825^{12-i}$ times the control strength for $i = 0, \dots, 12$, which weights the middle block most and the highest-resolution block least (gradio_canny2image.py, where a comment calls 0.825 a magic number). Both schemes, in our reading, weight low-resolution connections more strongly than high-resolution ones. The README recommends 50 steps and a guidance scale between 3 and 5 for guess mode, in which the ControlNet tries to recognize the content of the condition map without any prompt.

Several conditions. The outputs of several ControlNets trained for the same base model can simply be added, with no extra weighting (§3.4); the paper shows depth and pose used together (Fig. 6).

Other base models. Because ControlNet does not change the base model’s topology, a trained ControlNet can be attached to community models fine-tuned from the same Stable Diffusion version without retraining; the paper shows Comic Diffusion and Protogen 3.4 (Fig. 12).

Partial use. Masking the diffusion process turns the Canny model into a tool for pen-based editing, related to image inpainting (supplementary Fig. 19), and replacing the last 50% of the denoising iterations with plain Stable Diffusion keeps only the coarse layout of the condition (supplementary Fig. 27).

Example: The Step-0 Behaviour of Zero Convolutions

The script builds the ControlNet wiring on a small convolutional U-Net: an encoder with two residual blocks at $16 \times 16$ and $8 \times 8$, a middle block at $4 \times 4$, and a decoder that receives the two encoder outputs as skip connections. The data are $16 \times 16$ images of a disk, square or triangle at every position and size that fit, and the conditions are their edge maps, a toy stand-in for Canny edges. The locked U-Net is not trained. It has random weights, because the properties checked here hold at the first training step whatever the locked weights are. The ControlNet copies the U-Net’s encoder, encodes the edge map with two convolutions, adds it after the copy’s first convolution as the released code does, and joins the copy’s three outputs to the two skip connections and the middle output. Its four $1 \times 1$ connection convolutions (one on the condition, three on the outputs) are initialized three ways: zero, as in the paper; PyTorch’s default; and Gaussian weights with variance $2/\text{fan-in}$. For 120 of the image–edge pairs (a fixed random subset, called held out in the code), each noised at $t = 100$, 500 and 900, the script reports the change in the predicted noise relative to the locked model’s prediction, and the gradient norms of one diffusion-loss step for the connection weights, the trainable copy and the condition encoder.

import copy
import math
import sys
import torch
import torch.nn as nn
import torch.nn.functional as F

seed = int(sys.argv[1]) if len(sys.argv) > 1 else 0
torch.set_num_threads(4)

# ---- Data: 16x16 images in {-1, 1} of one disk, square or triangle, radius 3-5, at every
# integer centre where it fits (600 images). The condition is the shape's edge map.
H = 16
yy, xx = torch.meshgrid(torch.arange(H), torch.arange(H), indexing="ij")
def mask(kind, cy, cx, r):
    if kind == 0: return (yy - cy) ** 2 + (xx - cx) ** 2 <= r * r              # disk
    if kind == 1: return ((yy - cy).abs() <= r - 1) & ((xx - cx).abs() <= r - 1)  # square
    top = cy - r + 1                                                           # triangle
    return (yy >= top) & (yy <= cy + r - 1) & (2 * (xx - cx).abs() <= yy - top)
masks = torch.stack([mask(k, cy, cx, r) for k in range(3) for r in (3, 4, 5)
                     for cy in range(r, H - r) for cx in range(r, H - r)]).float()   # (600, 16, 16)
def edges(m):                                   # mask minus its 4-neighbour erosion
    p = F.pad(m, (1, 1, 1, 1))
    inner = m * p[:, :-2, 1:-1] * p[:, 2:, 1:-1] * p[:, 1:-1, :-2] * p[:, 1:-1, 2:]
    return m - inner
X = (2 * masks - 1)[:, None]                    # images, (600, 1, 16, 16)
E = edges(masks)[:, None]                       # edge maps
te = torch.randperm(600, generator=torch.Generator().manual_seed(1234))[480:]   # 120 held out

T = 1000
ab = torch.cumprod(1 - torch.linspace(1e-4, 0.02, T), 0)   # alpha_bar_t, t = 1..T

# ---- A small convolutional U-Net plays the locked model. It is NOT trained here:
# its weights are random, and the step-0 properties below do not depend on them.
def temb(t, dim=32):
    f = torch.exp(-math.log(1000.0) * torch.arange(dim // 2) / (dim // 2))
    a = t.float()[:, None] * f[None]
    return torch.cat([a.sin(), a.cos()], 1)

class Res(nn.Module):
    def __init__(self, cin, cout):
        super().__init__()
        self.n1, self.c1 = nn.GroupNorm(8, cin), nn.Conv2d(cin, cout, 3, padding=1)
        self.t = nn.Linear(64, cout)
        self.n2, self.c2 = nn.GroupNorm(8, cout), nn.Conv2d(cout, cout, 3, padding=1)
        self.skip = nn.Conv2d(cin, cout, 1) if cin != cout else nn.Identity()
    def forward(self, h, e):
        z = self.c1(F.silu(self.n1(h))) + self.t(e)[:, :, None, None]
        return self.skip(h) + self.c2(F.silu(self.n2(z)))

class Encoder(nn.Module):                       # the part that ControlNet copies
    def __init__(self):
        super().__init__()
        self.time = nn.Sequential(nn.Linear(32, 64), nn.SiLU(), nn.Linear(64, 64))
        self.conv_in = nn.Conv2d(1, 32, 3, padding=1)
        self.enc1, self.down1 = Res(32, 32), nn.Conv2d(32, 64, 3, stride=2, padding=1)
        self.enc2, self.down2 = Res(64, 64), nn.Conv2d(64, 64, 3, stride=2, padding=1)
        self.mid = Res(64, 64)
    def forward(self, x, t, hint=None):
        e = self.time(temb(t))
        h = self.conv_in(x)
        if hint is not None:                    # the copy adds the encoded condition here
            h = h + hint
        s1 = self.enc1(h, e)
        s2 = self.enc2(self.down1(s1), e)
        return e, s1, s2, self.mid(self.down2(s2), e)

class UNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder = Encoder()
        self.dec2 = Res(128, 64)
        self.dec1 = Res(96, 32)
        self.out = nn.Sequential(nn.GroupNorm(8, 32), nn.SiLU(), nn.Conv2d(32, 1, 3, padding=1))
    def forward(self, x, t, control=None):     # control: residuals for skip 1, skip 2, middle
        e, s1, s2, m = self.encoder(x, t)
        if control is not None:
            s1, s2, m = s1 + control[0], s2 + control[1], m + control[2]
        h = self.dec2(torch.cat([F.interpolate(m, scale_factor=2.0), s2], 1), e)
        h = self.dec1(torch.cat([F.interpolate(h, scale_factor=2.0), s1], 1), e)
        return self.out(h)

torch.manual_seed(seed)
base = UNet().requires_grad_(False)             # the "locked" model (random weights)

# ---- ControlNet: trainable copy of the encoder, a small condition encoder, and 1x1
# connection convolutions (zero-initialized in the paper) into the locked U-Net.
class ControlNet(nn.Module):
    def __init__(self, locked, init):
        super().__init__()
        self.copy = copy.deepcopy(locked.encoder).requires_grad_(True)
        self.hint = nn.Sequential(nn.Conv2d(1, 16, 3, padding=1), nn.SiLU(),
                                  nn.Conv2d(16, 32, 3, padding=1), nn.SiLU())
        self.z_in, self.z1, self.z2, self.zm = [nn.Conv2d(c, c, 1) for c in (32, 32, 64, 64)]
        for z in (self.z_in, self.z1, self.z2, self.zm):
            if init == "zero":
                nn.init.zeros_(z.weight); nn.init.zeros_(z.bias)
            elif init == "gaussian":                # He-scaled Gaussian weights, zero bias
                nn.init.normal_(z.weight, std=math.sqrt(2 / z.in_channels)); nn.init.zeros_(z.bias)
            # "default": PyTorch's own Conv2d initialization, left as it is
    def forward(self, x, t, cond):
        _, s1, s2, m = self.copy(x, t, hint=self.z_in(self.hint(cond)))
        return [self.z1(s1), self.z2(s2), self.zm(m)]

# Step-0 checks on the 120 held-out (image, edge map) pairs at three noise levels.
g = torch.Generator().manual_seed(7)
xt = torch.cat([ab[t - 1].sqrt() * X[te] + (1 - ab[t - 1]).sqrt() * torch.randn(X[te].shape, generator=g)
                for t in (100, 500, 900)])
t = torch.tensor([100, 500, 900]).repeat_interleave(120)
c = E[te].repeat(3, 1, 1, 1)
eps = torch.randn(xt.shape, generator=g)
eps_locked = base(xt, t)

print(f"seed {seed}; locked U-Net with random weights; 360 noisy held-out images")
print("init      output change  grad connections  grad copy  grad hint encoder")
for init in ["zero", "default", "gaussian"]:
    torch.manual_seed(seed + 1)                 # same hint-encoder weights for every variant
    cn = ControlNet(base, init)
    eps_ctrl = base(xt, t, cn(xt, t, c))
    change = ((eps_ctrl - eps_locked).pow(2).mean().sqrt() / eps_locked.pow(2).mean().sqrt()).item()
    F.mse_loss(eps_ctrl, eps).backward()        # the diffusion loss of one training step
    norm = lambda ps: math.sqrt(sum(p.grad.norm().item() ** 2 for p in ps))
    print(f"{init:9s}{change:14.2e}{norm([z.weight for z in (cn.z_in, cn.z1, cn.z2, cn.zm)]):18.4f}"
          f"{norm(cn.copy.parameters()):11.4f}{norm(cn.hint.parameters()):19.4f}")
    for name, z in [("z_in", cn.z_in), ("z1", cn.z1), ("z2", cn.z2), ("zm", cn.zm)]:
        if init == "zero":
            print(f"    {name}: |dL/dW| = {z.weight.grad.norm():.4f}, |dL/db| = {z.bias.grad.norm():.4f}")

Output (PyTorch 2.14.1, CPU, 3 to 4 seconds; two runs gave identical output):

seed 0; locked U-Net with random weights; 360 noisy held-out images
init      output change  grad connections  grad copy  grad hint encoder
zero           0.00e+00            0.1139     0.0000             0.0000
    z_in: |dL/dW| = 0.0000, |dL/db| = 0.0000
    z1: |dL/dW| = 0.0779, |dL/db| = 0.0369
    z2: |dL/dW| = 0.0662, |dL/db| = 0.0326
    zm: |dL/dW| = 0.0502, |dL/db| = 0.0307
default        7.51e-01            0.2167     0.6863             0.0263
gaussian       1.09e+00            0.0945     0.5543             0.0340

With zero initialization, attaching the ControlNet leaves the locked network’s prediction exactly unchanged: Eq. 3 holds to the last bit, not approximately. With either ordinary initialization, the attached network changes the prediction by an amount comparable to the prediction itself (0.75 and 1.09 of its RMS). In a trained model this would be the “harmful noise” Zhang et al. want to avoid; the size of the change here depends on the random locked weights and says nothing about a real model.

The gradients follow the supplement’s argument. At the first step only the three output connections receive gradients, in both weight and bias, while the trainable copy, the condition encoder and the condition-side connection z_in receive exactly zero. That is the chain-rule consequence noted above: their gradients pass through the zero weights of the output connections. By the same argument, after the first update the output weights are no longer zero, so from the next step on gradients reach the copy (not run here). With ordinary initialization every part receives gradients at once, starting from a perturbed network. This toy shows only these step-0 mechanics. It says nothing about how fast control is learned or whether convergence is sudden. We planned a trained comparison of zero and ordinary initialization, with a small diffusion model trained from scratch as the locked model. The planned convolutional version was too slow for an example, so with the editor’s approval a smaller fully connected stand-in was piloted in its place; in that pilot the stand-in did not reach the base result fixed in advance, nor the planned control criterion. The editor then decided to leave the trained comparison out and keep only these step-0 checks, which we run on the planned convolutional network, untrained, since they need only one forward and backward pass.

Performance

All results below are from the paper and its supplement; ControlNet models are based on Stable Diffusion 1.5 unless stated.

• Ablation. On a scribble model tested under four prompt settings (none, an insufficient prompt that does not name the object, a conflicting prompt and a perfect prompt), replacing the zero convolutions with standard convolutions initialized with Gaussian weights dropped performance to about the level of ControlNet-lite, which replaces each trainable copy with a single convolution. The authors read this as the pretrained backbone of the trainable copy being destroyed during fine-tuning (§4.2, Fig. 8). ControlNet-lite failed with no prompt or an insufficient one. The comparison is qualitative, shown as random batches of six samples. The supplement adds lightweight layers connected to the decoder, with and without zero convolutions, and direct fine-tuning of Stable Diffusion with one added input layer, which it calls mathematically the same as Stable Diffusion 2’s depth-to-image fine-tuning (supplementary §2).
• User study. Twelve users (ten amateurs and two professionals) ranked results for 20 unseen hand-drawn sketches from five methods, giving 100 rankings each for image quality and fidelity to the sketch, on a scale of 1 (worst) to 5 (best). ControlNet averaged 4.22 for quality and 4.28 for fidelity, against 3.93 and 4.09 for ControlNet-lite, 3.21 and 2.31 for Sketch-Guided Diffusion at its default edge-guidance scale, and 1.10 and 1.02 for PITI (Table 1). The table’s caption calls the measure Average User Ranking and the text Average Human Ranking, and the tables and the user-study text spell PITI “PIPT” (§4.4 has “PTIT”).
• Segmentation fidelity. Images generated from ADE20K test-set segmentations were re-segmented with OneFormer, and the intersection over union with the input maps was measured (Table 2). ControlNet scored 0.35, ControlNet-lite 0.32, LDM 0.31, PITI 0.26 and VQGAN 0.21, against 0.58 for OneFormer on the ground-truth images.
• Image quality. On 10,000 COCO-segmentation-conditioned samples at $512 \times 512$, the Fréchet Inception Distance (FID) was 15.27 for ControlNet, 17.92 for ControlNet-lite, 19.74 for PITI and above 25 for VQGAN and LDM trained from scratch. CLIP text–image scores were 0.26 for both ControlNets and 0.17 to 0.20 for the others. Unconditioned Stable Diffusion scored FID 6.09, CLIP 0.26 and CLIP aesthetic 6.32 (Table 3). The supplement (§4.3) explains why a controlled model scores worse FID than the unconstrained one, and warns that FID alone cannot measure control, since a ControlNet that does nothing would score best.
• Against an industrial model. A depth ControlNet trained on 200,000 images on one RTX 3090 Ti for five days was compared with Stability AI’s Stable Diffusion 2 depth-to-image model, which the authors describe as trained on more than 12 million images with thousands of A100 hours. Users shown 100 labelled images from each model could tell them apart with an average precision of only 0.52 ± 0.17 on 200 new images (§4.3). The main text says one ControlNet was trained for SD 2; the supplement (§4.2) trained one each for SD 1.5 and 2.1 and used the 2.1 model in the study.
• Dataset size. Canny-edge ControlNets trained on 1,000 to 3 million images did not collapse at 1,000 images and improved with more data (Fig. 10; supplementary Fig. 22). The paper reports this qualitatively.

Strengths

• The base model is preserved. The locked weights are never changed, so text-to-image ability is kept, and with zero connections the training starts from the unchanged model (Eq. 3).
• Small data and modest hardware. The paper trained useful models from 20,000 to 3 million pairs, some on a single consumer GPU.
• Composable and transferable. Outputs of several ControlNets can be added at inference, and a ControlNet works with fine-tunes of the base model it was trained on.
• One recipe, many conditions. Edges, lines, scribbles, poses, segmentation, depth and normals all use the same architecture and loss.

Limitations

• Cost per condition and per step. Each condition needs its own trained ControlNet, a full copy of the base model’s encoder, and every denoising step runs the copy as well as the base model. Training costs about a third more time per iteration (§3.2).
• Wrong interpretations persist. When the ControlNet misreads the content of an ambiguous condition, a strong prompt cannot undo the error (supplementary Fig. 28).
• Tied to one base model. A ControlNet is trained for one U-Net layout and transfers only to models that share it.
• Training-data problems. The ControlNet 1.1 README reports that the 1.0 training sets contained a small group of greyscale human images duplicated thousands of times, low-quality and JPEG-damaged images, and some wrongly paired prompts. It also reports that HED and PiDiNet soft-edge maps can hide a corrupted greyscale copy of the original image, which the HED 1.0 model learned to restore instead of following boundaries.
• Evaluation. The quantitative results rest on a 12-person user study, segmentation IoU and FID; most comparisons in the paper are visual.

Historical Importance

ControlNet arrived as text-to-image models had become widely used but hard to steer spatially. In our reading, its main idea is a frozen pretrained model plus a full-depth trainable copy joined by connections initialized to zero. That idea made the condition-specific model an add-on that can be trained by individuals, shared and combined, rather than a fine-tune of the whole model. The repository credits a WebUI extension (Mikubill’s sd-webui-controlnet) for multi-ControlNet use and for transfer to community models, and ControlNet 1.1’s README names it the official way to combine several ControlNets. The paper and its released models were part of a wave of work on conditioning frozen text-to-image models in early 2023, alongside concurrent work such as T2I-Adapter, which the paper names.

Variants

ControlNet 1.1 (README, as of November 2026) keeps “exactly the same architecture” as 1.0, with a promise not to change it before a version 1.5. It introduces a naming scheme and 14 models for Stable Diffusion 1.5, 11 marked production-ready and 3 experimental (Shuffle, Instruct Pix2Pix and Tile). The retrained models fixed the 1.0 dataset problems described under Limitations, and several added data augmentation such as random left-right flips. Other changes include depth trained on MiDaS, LeReS and ZoeDepth maps at several resolutions; normals from a learned estimator in place of normals computed from MiDaS depth; segmentation accepting both ADE20K and COCO label colours; OpenPose with hands and face; and new line-art, inpainting and soft-edge models. The Canny 1.1 model was resumed from 1.0 and trained on eight A100 80 GB GPUs at batch size 256 for three days. Two experimental models change how the output is used: Shuffle, trained to recompose images shuffled by a random flow so that prompts or other ControlNets can change their style, requires a global average pooling of the ControlNet outputs and must be applied only on the conditional side of guidance; Tile ignores fine details and regenerates them, and for each tile of a large image reduces the influence of a global prompt that does not match the tile’s content. The 1.1 repository has no licence file (as of November 2026); the 1.0 repository is under the Apache 2.0 licence.

ControlNet-lite and other ablations replace the trainable copy with single convolution layers (see Performance).

Adapters and grounded generation pursue the same goal with smaller trainable parts; they are summarized here in outline only.

• T2I-Adapter (Mou et al., 2023), named in the ControlNet paper as concurrent work, is described in its repository’s README as an adapter of 77 million parameters, against 860 million for SD 1.4/1.5 in its table, needing “a single inference”. Its paper was not read for this article. The README also lists SDXL versions for sketch, Canny, line art, OpenPose and depth.
• IP-Adapter conditions on an image prompt instead of a spatial map. Its README describes an adapter of 22 million parameters, and its demos combine it with ControlNet. In its released code, CLIP image features are projected to a few extra tokens, each cross-attention layer gets separate key and value projections for them, and the result is added to the text cross-attention output, scaled by a user weight (ip_adapter/attention_processor.py). Its paper was not read.
• GLIGEN (Li et al., 2023) freezes the text-to-image model and inserts a new gated self-attention layer into each transformer block, between self-attention and cross-attention. The layer attends over the visual tokens together with grounding tokens, such as boxes with phrases, keypoints or spatially aligned maps, and adds its output scaled by $\beta \tanh(\gamma)$, with the learnable $\gamma$ initialized to 0 (§4.2, Eq. 8). Like a zero convolution, this starts training from the unchanged model. Setting $\beta = 0$ for the later sampling steps (scheduled sampling) trades grounding for image quality. For spatially aligned conditions, GLIGEN also concatenates a downsampled condition to the U-Net’s input, which makes the first convolution trainable.

Stable Diffusion’s own depth and inpainting checkpoints take a different route: they add zero-initialized input channels and fine-tune the whole model, as the Stable Diffusion article describes. The ControlNet supplement compares this route as its “directly finetune” ablation. Personalization methods such as DreamBooth and textual inversion, which the paper also cites, teach the model new subjects rather than new spatial conditions.

Implementations

• ControlNet 1.0 (lllyasviel/ControlNet, Apache 2.0, last commit 9 September 2023, as of November 2026). It contains the ControlLDM model (cldm/cldm.py), configurations for SD 1.5 and 2.1, tool_add_control.py, which builds an initial checkpoint by copying the U-Net encoder’s weights into the control branch, a training tutorial, and ten Gradio demos covering eight conditions, with detector code for each. The training guide calls training a ControlNet as easy as, or easier than, training a pix2pix model.
• ControlNet 1.1 (lllyasviel/ControlNet-v1-1-nightly, last commit 25 May 2023, as of November 2026). It has 14 models and demos; the README states that it is for research use and academic experiments and points users of the AUTOMATIC1111 WebUI to the separate extension.
• Model weights for both versions are distributed through the authors’ Hugging Face pages; the anime line-drawing model of 1.0 was withheld by the authors pending an assessment of its risks (1.0 README).