Image Editing

Image editing changes a given image so that it satisfies a request, such as “make it winter”, a mask with a prompt, a few coloured strokes, or a point dragged to a new place, while everything the request does not mention stays as it was. Generative image editing, the subject here, uses a learned model of images, today usually a diffusion model, so that the changed region looks real and fits its surroundings. The task is also called text-guided, prompt-based or instruction-based image editing, or semantic image manipulation. It differs from image-to-image translation, which learns one mapping between two domains, from text-to-image generation, which has no input image to preserve, and from image inpainting, which fills a hole without having to keep anything inside it.

Problem Definition

Given an image $x$ and an edit request $c$, produce an image $x'$ that (i) carries out $c$, (ii) is realistic, and (iii) agrees with $x$ wherever $c$ does not ask for change. The third requirement has no exact definition, because “what should not change” depends on the request: a style edit changes every pixel but keeps the layout, while an object replacement keeps the rest of the scene pixel for pixel. Most edits have many acceptable answers, so methods are usually stochastic, and there is rarely a ground-truth $x'$.

The requirements conflict. Meng et al. (2022, §3) state it for stroke guides as a trade-off between realism and faithfulness: the guide itself is perfectly faithful and unrealistic, while a random realistic image is not faithful. For edits made through a generator’s latent space, Tov et al. (2021) describe a distortion–editability trade-off: latent codes that reproduce a photo most exactly tend to lie where edits work less well and image quality is lower. The same tension reappears in every family below as a parameter that moves along it: the noise level of SDEdit, the guidance scale during inversion, the number of steps in which attention is injected, and the image guidance scale of InstructPix2Pix.

Input and Output

The input is an image plus one or more kinds of request:

• A guide image: a rough stroke painting, coloured strokes painted over a photo, or a collage of pasted patches (SDEdit’s high-, mid- and low-level guides, Meng et al., §3).
• A mask and a text prompt for a local edit, such as replacing an object or the background (Avrahami et al., 2022).
• A pair of captions, the source caption and an edited one, from which the change is inferred (Hertz et al., 2023; Mokady et al., 2023).
• An instruction such as “turn him into a cyborg” (Brooks et al., 2023).
• A few photos of a subject to place in new contexts (Ruiz et al., 2023; Gal et al., 2023).
• Handle and target points, optionally with a mask of the region allowed to move (Pan et al., 2023).

The output is an edited image of the same size, or several candidates to choose from.

Variants

• Global and local edits. Style, season or lighting changes touch every pixel; object replacement or attribute changes should touch a region.
• Appearance and structure. Recolouring or retexturing keeps shapes; replacing a bicycle with a car, changing pose or moving an object changes them, which is much harder to do without disturbing the rest.
• Generated and real images. A model can edit its own samples by rerunning generation with changes; a real photo must first be mapped into the model, which is the inversion problem.
• Single-turn and multi-turn. Successive edits accumulate errors (Brooks et al., 2023, §5; Zhang et al., 2023, §4.2).
• Subject-driven generation, placing a specific object or pet in new scenes, is personalization rather than editing of one image, but uses the same machinery.

Challenges

• No ground truth. Correct edits of real images are rarely available, so quantitative evaluation remains an open challenge (Mokady et al., 2023, §5.1).
• Inversion under guidance. Editing a real photo with a diffusion model needs the noise that regenerates it, and the guidance that makes text edits work amplifies the error of finding it (Mokady et al., §3.2).
• Leakage and locality. Changing one word of a prompt, even with the same seed, gave a completely different layout in Prompt-to-Prompt’s experiments (Hertz et al., arXiv v1, §4). Masks localize edits but remove the structure inside them, so details such as the replaced object’s size are lost (Mokady et al., §5.1, Fig. 7).
• Cost. Per-image optimization or guided sampling takes seconds to minutes on a GPU, which limits real-time use (Avrahami et al., §6; Mokady et al., arXiv v1, Appendix D).

Approaches

GAN inversion and latent editing

Zhu et al. (2016) proposed editing on a learned image manifold: project a photo into a GAN’s latent space, express the user’s colour and shape strokes as constraints on the latent code, and render the result, so that edits cannot “fall off” the manifold of natural images. With StyleGAN, editing became a matter of inverting a photo into the latent space and moving its code along directions found with labels or by PCA; the StyleGAN article covers those directions and the $\mathcal{W}$ and $\mathcal{W}+$ spaces. Tov et al. (2021, §4) showed that inverting into the more expressive $\mathcal{W}+$ space lowers distortion but leaves codes that are less editable and give lower perceptual quality, and designed their e4e encoder to keep codes close to $\mathcal{W}$. SDEdit’s authors cite the cost and inaccuracy of GAN inversion on many datasets, and its occasional failure to find any code that reproduces the input, as a motivation for avoiding it (Meng et al., §1, §4).

Noise and denoise: SDEdit

SDEdit (Meng et al., 2022) needs no inversion and no task-specific training: it adds Gaussian noise to the guide up to an intermediate time $t_0$ and runs the reverse process of a pretrained diffusion model from there. The noise drowns the unnatural details of strokes while the coarse layout survives, and denoising pulls the result onto the model’s image distribution. The image-to-image translation article gives the starting point in DDPM notation. The single parameter $t_0$ sets the trade-off. The authors bound the squared distance between the guide and the output, with high probability, by a quantity that grows with the noise variance at $t_0$ (Proposition 1), so a guide far from any realistic image needs a large $t_0$. With time in $[0, 1]$, they found $t_0 \in [0.3, 0.6]$ to work for reasonable guides and suggest a binary search on $t_0$ guided by the user, then fixing it for a whole task (§3). On LSUN bedroom with simulated stroke guides, SDEdit scored an $L_2$ distance to the guide of 36.76 and a kernel inception distance (KID) of 0.0030, against 52.40 and 0.0464 for the e4e GAN-inversion baseline (Table 2, arXiv v2).

For edits that must leave a region untouched, the masked variant re-noises the protected pixels of the input to the current noise level at every step, so that they return exactly to the input at $t = 0$ (Appendix C.2). With a text-to-image model the same procedure, conditioned on a new prompt, is the “image-to-image” mode of Stable Diffusion, whose strength parameter plays the role of $t_0$. Its weakness is fidelity: details such as faces drift, because everything below the noise level is regenerated (Mokady et al., §5.1).

Inversion: DDIM and null-text inversion

Deterministic DDIM sampling with $\eta = 0$ is an Euler discretization of an ODE, and running it backward maps an image to a noise vector that regenerates it (Song, Meng & Ermon, 2021; the diffusion sampling article derives the ODE). One inversion step evaluates the network at the current, less noisy point and moves to the next noise level:

$$x_{t+1} = \sqrt{\bar\alpha_{t+1}}\, \hat x_0(x_t) + \sqrt{1 - \bar\alpha_{t+1}}\; \epsilon_\theta(x_t, t, c), \qquad \hat x_0(x_t) = \frac{x_t - \sqrt{1 - \bar\alpha_t}\, \epsilon_\theta(x_t, t, c)}{\sqrt{\bar\alpha_t}},$$

in this site’s DDPM notation (Mokady et al., §3.1, write the same update in terms of ratios of $\bar\alpha$). The update reverses the ODE only in the limit of small steps (Mokady et al., §3.1), so a small error enters at each step. Mokady et al. observed that this error stays negligible for unconditional models but that classifier-free guidance (Ho & Salimans, 2022), with the scale $w = 7.5$ that Stable Diffusion uses by default (Mokady et al.’s $w$ is the $s$ convention, where 1 means no guidance), magnifies the accumulated error: inverting with guidance gives visual artifacts and a noise vector that may lie outside the Gaussian, which they found less editable (§3.2; arXiv v1, Appendix B, Fig. 9).

Null-text inversion (Mokady et al., 2023) keeps the trajectory $x^\ast_0, \dots, x^\ast_T$ of a DDIM inversion at $w = 1$ as a pivot, then generates backward with $w = 7.5$ and, at each step $t$, optimizes only the unconditional (“null text”) embedding $\varnothing_t$ used in the guidance formula, so that the guided step lands on the pivot:

$$\min_{\varnothing_t} \big\lVert x^\ast_{t-1} - x_{t-1}(\bar x_t, \varnothing_t, C) \big\rVert_2^2 ,$$

where $C$ is the embedding of the source caption and $\bar x_t$ the current point of the optimized trajectory, starting from $\bar x_T = x^\ast_T$ (their Eq. 3 and Algorithm 1). The network and the caption embedding stay fixed, so the caption can afterwards be edited as with generated images. One embedding per step reconstructed far better than one global embedding, and the DDIM pivot converged much faster than random noise (§4). With $N = 10$ iterations per step, learning rate 0.01 and early stopping, an inversion took 40 to 120 s on an A100 (arXiv v1, Appendix D). In a study of 48 images rated by 50 participants, their edits were chosen as the best of four methods in 65.1% of choices, against 16.6% for Text2LIVE, 14.5% for SDEdit and 3.8% for VQGAN+CLIP (Table 1). Prompt-to-Prompt had shown only preliminary edits of real images: its DDIM inversion often worked but was not accurate enough in many cases, which its authors attribute partly to the distortion–editability trade-off (Hertz et al., arXiv v1, §4).

Attention and feature control

In a text-conditioned diffusion model, image features attend to the prompt tokens through cross-attention, and each word gets a spatial map of where it acts. Prompt-to-Prompt (Hertz et al., 2023) observed that these maps, fixed early in sampling, largely determine the layout. It generates the source and the edited prompt from the same noise and, during the edited generation, overrides the attention maps with the source’s: for a word swap (“bicycle” to “car”) during the first part of sampling only, controlled by a step threshold $\tau$, because injecting throughout over-constrains the new object’s shape; for an added phrase only on the shared tokens; and for attention re-weighting by scaling one word’s map by a factor in $[-2, 2]$ to strengthen or weaken its effect (§3.2). The arXiv version used Imagen as the generator, a model its authors did not release for public use (Saharia et al., 2022, §6), and showed its results only qualitatively. Combined with null-text inversion, the same edits apply to real photographs on Stable Diffusion (Mokady et al.).

Plug-and-play diffusion features (Tumanyan et al., 2023) translate a guidance image to a new prompt by injecting internal features instead of cross-attention: the spatial features of one decoder layer of Stable Diffusion’s U-Net (layer 4), which they found to encode localized semantics, and the self-attention maps of the decoder layers, taken from the guidance image’s DDIM-inverted trajectory and injected up to two step thresholds (§4). They measured prompt fidelity by the CLIP image–text similarity and structure preservation by a distance between self-similarity matrices of DINO-ViT features, and their method sat at the favourable end of both, matching SDEdit’s structure distance at low noise and its CLIP similarity at high noise (§5.1, Fig. 9). It fails when the guidance image has no semantic link to the target, as with arbitrarily coloured segmentation masks (Fig. 12).

Masks and blended diffusion

When the user marks the region to change, the rest of the image can be preserved exactly; the image inpainting article describes fill-in methods, including diffusion inpainting models. Blended Diffusion (Avrahami et al., 2022) edits a masked region of a real image according to a prompt using a diffusion model trained on ImageNet and guided by CLIP. At each step it takes a CLIP-guided denoising step for the whole image, noises the input image to the same level, and combines the two with the mask, $x_{t-1} = x_{t-1,\mathrm{fg}} \odot m + x_{t-1,\mathrm{bg}} \odot (1 - m)$; the next denoising step repairs the seam, an idea the authors compare with blending each level of a Laplacian pyramid (§4.2.1). Because gradient steps on pixels can lower the CLIP loss adversarially without a visible change, they average the CLIP gradient over several projectively transformed copies of the current estimate (“extending augmentations”, §4.2.2), and rank several outputs by CLIP. In a preliminary user study on GAN-generated examples taken from earlier work (Fig. 5), their results scored 4.73 for background preservation and 4.63 for text match on a 1-to-5 scale (Table 1). An edit took about 30 s on a GPU, and the CLIP ranking could favour results with only part of the requested object (§6).

Instruction-based editing

InstructPix2Pix (Brooks et al., 2023) trains a model that edits in a single sampling pass from an instruction. Its training data are generated. The authors wrote 700 edits by hand, each an input caption from LAION-Aesthetics, an instruction and an output caption, fine-tuned GPT-3 on them, and generated 454,445 caption–instruction–caption triplets (§3.1.1). Stable Diffusion then rendered each caption pair with Prompt-to-Prompt, sampling the fraction of shared-attention steps uniformly in $[0.1, 0.9]$ for 100 pairs per caption pair and keeping those that passed CLIP-based filters (§3.1.2). The main text describes borrowing cross-attention weights; the appendix of arXiv v2 says that self-attention weights were replaced instead, for all edit types (Appendix A.2). The model starts from Stable Diffusion v1.5 with extra zero-initialized input channels that receive the input image’s latent, and uses classifier-free guidance on both conditions, with an image scale $s_I$ and a text scale $s_T$:

$$\tilde\epsilon_\theta(z_t, c_I, c_T) = \epsilon_\theta(z_t, \varnothing, \varnothing) + s_I\big(\epsilon_\theta(z_t, c_I, \varnothing) - \epsilon_\theta(z_t, \varnothing, \varnothing)\big) + s_T\big(\epsilon_\theta(z_t, c_I, c_T) - \epsilon_\theta(z_t, c_I, \varnothing)\big)$$

(Eq. 3), trained with each condition dropped in 5% of examples and both in another 5%. Raising $s_I$ keeps more of the input’s structure and raising $s_T$ strengthens the edit; values of $s_T$ from 5 to 10 and $s_I$ from 1 to 1.5 usually worked best (§3.2.1, §4.2). Training at $256 \times 256$ took 25.5 hours on eight A100 GPUs, and an edit at $512 \times 512$ about 9 s on one (arXiv v2, Appendix A.3). The model fails at viewpoint changes, swapping objects and isolating a named object, and inherits the biases of its data and models (§5, Fig. 13). Fine-tuning it on the human-made MagicBrush data improved every automatic metric in its authors’ tests (Zhang et al., 2023, Table 2).

Personalization

Some edits need a subject the model has never seen. Textual inversion (Gal et al., 2023) learns one new word embedding from typically 3 to 5 images with the generator frozen; DreamBooth (Ruiz et al., 2023) fine-tunes the whole model on a few photos bound to a rare identifier, with a prior-preservation loss; the Stable Diffusion article describes both. DreamBooth’s evaluation proposed subject fidelity as the cosine similarity of DINO ViT-S/16 embeddings between generated and real images, which they preferred to CLIP image similarity (CLIP-I) because DINO, unlike supervised networks, is not trained to ignore differences between subjects of one class, and measured prompt fidelity as image–prompt CLIP similarity (CLIP-T); MagicBrush reports all three for edits (Zhang et al., §4.1). On their 30 subjects and 25 prompts, DreamBooth with Imagen scored 0.696 DINO and 0.306 CLIP-T, with Stable Diffusion 0.668 and 0.305, and textual inversion 0.569 and 0.255, against 0.774 DINO between real images of the subject (Table 1).

Drag-based editing

DragGAN (Pan et al., 2023) lets a user click handle points and target points on a StyleGAN image and moves the content until the handles reach the targets. Each iteration takes one optimization step on the latent code with a motion supervision loss on the generator’s feature map $F$ (after StyleGAN2’s sixth block, resized to the image):

$$\mathcal{L} = \sum_{i} \sum_{q \in \Omega_1(p_i, r_1)} \big\lVert F(q) - F(q + d_i) \big\rVert_1 + \lambda \big\lVert (F - F_0) \odot (1 - M) \big\rVert_1, \qquad d_i = \frac{t_i - p_i}{\lVert t_i - p_i \rVert_2},$$

with no gradient through $F(q)$, so that the patch around each handle $p_i$ moves one small step toward its target $t_i$, while the second term holds the region outside the mask $M$ to the initial features $F_0$ (Eq. 1). A point tracking step then relocates each handle by nearest-neighbour search of its initial feature in a small window, using the same features (§3.3). Only the codes of the first six layers are optimized, in $\mathcal{W}+$ (§3.2); editing usually took 30 to 200 iterations (§3.1) and a few seconds on an RTX 3090 (§1). Real photos must first be inverted into the GAN.

DragDiffusion (Shi et al., 2024) carries the same two steps to Stable Diffusion 1.5. It first fine-tunes a rank-16 LoRA on the input image for 80 steps, about 25 s on an A100, to preserve its identity; then it DDIM-inverts the image to the 35th of 50 steps, applies motion supervision and point tracking to U-Net features of that single noisy latent, and denoises it. During denoising, keys and values of self-attention come from the original image’s latent (“reference-latent control”). It does not use classifier-free guidance in inversion or denoising, citing the error amplification above (§3–4.1).

Datasets and Benchmarks

Editing has few shared benchmarks, and most papers build their own.

Benchmark	Content	Introduced by
MagicBrush	10,388 edit turns in 5,313 sessions on real images, with instructions, masks and target images	Zhang et al. (2023)
DreamBooth dataset	30 subjects and 25 prompts; evaluation generates four images per subject and prompt	Ruiz et al. (2023)
Wild-TI2I, ImageNet-R-TI2I	148 text–image pairs (53% real images); 3 renditions from each of 10 ImageNet-R classes	Tumanyan et al. (2023)
DragBench	Images with handle–target point pairs and editable-region masks	Shi et al. (2024)

MagicBrush’s source images come from COCO, and its target images were made by crowd workers on OpenAI’s DALL·E 2 editing platform; its test split has 535 sessions (Zhang et al., §3.2, §4.2). Its ground truth is therefore one human-chosen result among many acceptable ones. Null-text inversion’s ImageNet evaluation follows a public protocol from earlier work: 1,000 test images, each with an edit replacing the main object, scored by whether a classifier recognizes the new class (Mokady et al., §5.1, Table 2).

Evaluation Metrics

Edits are scored on two axes that compete, so a single number misleads.

• Fidelity to the input: pixel $L_1$ or $L_2$ distance (to the guide, to the input outside the mask, or to a reference target), LPIPS distance, the cosine similarity of CLIP or DINO image embeddings, or a structure distance such as DINO self-similarity. DragDiffusion reports image fidelity as $1 - \mathrm{LPIPS}$ and measures dragging accuracy as the mean distance between the targets and the handles’ final positions, found by a feature-correspondence method (Shi et al., §4.2).
• Agreement with the request: CLIP similarity between the edited image and the target text (CLIP-T), or the directional CLIP similarity, the cosine between the change in image embeddings and the change in caption embeddings,

$$\cos\big(E_I(x') - E_I(x),\; E_T(c') - E_T(c)\big),$$

which Gal et al. (2022, Eq. 2) defined as a training loss for adapting a generator and InstructPix2Pix used both to filter its data and to evaluate edits. The text-to-image generation article discusses what CLIP-based scores miss.

• Human judgment: pairwise preferences or ratings of realism, preservation and text match (Meng et al.; Avrahami et al.; Mokady et al.).

Because the axes compete, methods are compared as curves. InstructPix2Pix plotted CLIP image similarity against directional similarity over 2,000 edits while sweeping each method’s control parameter, SDEdit’s strength from 0.3 to 0.9 and Prompt-to-Prompt’s injection period from 0 to 1, and reported a higher curve than both (Brooks et al., §4.1, Fig. 8). Single metrics reward doing nothing or doing too much. On MagicBrush, Text2LIVE had the best single-turn $L_1$ among the methods without masks except fine-tuned InstructPix2Pix, likely because it adds a layer that changes little, the authors suggest, and the lowest CLIP-T, while VQGAN-CLIP, which optimizes CLIP directly, had the highest CLIP-T and the worst preservation (Zhang et al., §4.2, Table 2). In null-text inversion’s ImageNet test, Text2LIVE and FlexIT stayed closer to the input in LPIPS than null-text inversion with Prompt-to-Prompt, but the new class was recognized in 47.5% and 51.3% of their edits against 61.1%; the authors argue that, when the object fills much of the image, high similarity indicates a weaker edit (Mokady et al., Table 2). Methods that optimize the evaluation network itself score well on it, as Tumanyan et al. note for VQGAN+CLIP and DiffuseIT (§5.1).

Example: SDEdit and DDIM Inversion on a Small Trained Model

The script trains a small class-conditional diffusion model from scratch on $10 \times 10$ images: a square outline, a plus or an X ($6 \times 6$) at each of 25 offsets, 75 shape images in all, with pixel noise of standard deviation 0.1 added during training. The class is dropped to a learned null embedding 10% of the time, so one residual-MLP denoiser gives unconditional, conditional and guided predictions, with guidance scale $s$ in the convention where $s = 1$ is the plain conditional model. Four experiments follow. (1) SDEdit on a faint, broken square outline with a stray blob: per $t_0$, 400 runs report the RMS distance to the guide and to the nearest training image, the share whose pixel signs match a training image exactly (“valid”, a realism check), and the share whose nearest training image is the intended square at offset (2, 2) or any square. (2) DDIM inversion in 50 steps of 75 new images (each shape and offset with new pixel noise), and regeneration; “image kept” means the nearest training image is the original’s. (3) Null-text inversion at $s = 7.5$ with Mokady et al.’s 10 iterations of learning rate 0.01 per step (here Adam, without early stopping). (4) Each square inverted to step $k$ of 50 and regenerated as a plus.

import math
import sys
import torch
import torch.nn as nn

seed = int(sys.argv[1]) if len(sys.argv) > 1 else 0
torch.manual_seed(seed)
torch.set_num_threads(4)

# Data: 10x10 images in [-1, 1] of three 6x6 shapes (square outline, plus, X)
# at every one of the 5 x 5 offsets: 75 distinct images, labelled by shape.
H, K = 10, 6
r, c = torch.meshgrid(torch.arange(K), torch.arange(K), indexing="ij")
glyphs = [(r == 0) | (r == K - 1) | (c == 0) | (c == K - 1),       # square outline
          (r == 2) | (r == 3) | (c == 2) | (c == 3),               # plus
          (r == c) | (r + c == K - 1)]                             # X
data, label = [], []
for k, g in enumerate(glyphs):
    for oy in range(H - K + 1):
        for ox in range(H - K + 1):
            img = -torch.ones(H, H)
            img[oy:oy + K, ox:ox + K][g] = 1.0
            data.append(img.flatten()); label.append(k)        # index = 25 * shape + 5 * oy + ox
data, label = torch.stack(data), torch.tensor(label)               # (75, 100), (75,)

T = 1000
ab = torch.cumprod(1 - torch.linspace(1e-4, 0.02, T), 0)          # alpha_bar_t, t = 1..T
NULL = 3                                                           # index of the null class

class Denoiser(nn.Module):
    """Residual MLP eps_theta(x_t, t, e): e is a class embedding; row 3 of the table is null."""
    def __init__(self, width=128):
        super().__init__()
        self.emb = nn.Embedding(4, 32)
        self.inp = nn.Linear(100, width)
        self.cond = nn.Sequential(nn.Linear(64, width), nn.SiLU(), nn.Linear(width, width))
        self.blocks = nn.ModuleList([nn.Sequential(nn.LayerNorm(width), nn.Linear(width, width),
                                                   nn.SiLU(), nn.Linear(width, width)) for _ in range(3)])
        self.out = nn.Sequential(nn.LayerNorm(width), nn.Linear(width, 100))
    def forward(self, x, t, e):                                    # e: class embedding vectors
        f = torch.exp(-math.log(1000.0) * torch.arange(16) / 16)
        a = t.float()[:, None] * f[None]
        h = self.inp(x) + self.cond(torch.cat([a.sin(), a.cos(), e], 1))
        for b in self.blocks:
            h = h + b(h)
        return self.out(h)

net = Denoiser()
opt = torch.optim.Adam(net.parameters(), lr=2e-3)
steps = 6000
sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, steps)
for step in range(steps):                                          # DDPM loss, 10% null class
    i = torch.randint(0, 75, (256,))
    x0, y = data[i] + 0.1 * torch.randn(256, 100), label[i].clone()  # shape plus pixel noise
    y[torch.rand(256) < 0.1] = NULL
    t = torch.randint(1, T + 1, (256,))
    eps = torch.randn_like(x0)
    a = ab[t - 1, None]
    loss = ((net(a.sqrt() * x0 + (1 - a).sqrt() * eps, t, net.emb(y)) - eps) ** 2).mean()
    opt.zero_grad(); loss.backward(); opt.step(); sched.step()
print(f"seed {seed}: final training loss {loss.item():.3f}")
for p in net.parameters():
    p.requires_grad_(False)

def eps_cfg(x, t, y, s, null_e=None):
    """Classifier-free guidance, s convention: s = 0 unconditional, s = 1 conditional."""
    n = len(x)
    tt = torch.full((n,), t)
    e_null = net.emb.weight[NULL].expand(n, -1) if null_e is None else null_e
    eu = net(x, tt, e_null)
    if s == 0:
        return eu
    return eu + s * (net(x, tt, net.emb(y)) - eu)

def ddim_step(x, e, a_from, a_to):                                 # eta = 0 DDIM, either direction
    x0_hat = (x - (1 - a_from).sqrt() * e) / a_from.sqrt()
    return a_to.sqrt() * x0_hat + (1 - a_to).sqrt() * e

def nearest(x):                                                    # nearest training image, RMS to it
    d = torch.cdist(x, data) / math.sqrt(100)
    return d.min(1)

# ---- (1) SDEdit: noise a guide to step t0, then run the reverse process (unconditional, DDPM).
g = -torch.ones(H, H)
g[2:8, 2:8][glyphs[0]] = 0.0                                       # a faint, grey square outline
g[2, 4:6] = -1.0                                                   # with a gap in its top edge
g[8:10, 0:2] = 0.6                                                 # and a stray blob
guide = g.flatten()
target = 0 * 25 + 2 * 5 + 2                                        # the square at offset (2, 2)
gen = torch.Generator().manual_seed(seed + 1)
n = 400
print(f"\nSDEdit on one guide, {n} runs per t0 (RMS in [-1, 1] units)")
print(f"guide: RMS to nearest training image {nearest(guide[None])[0].item():.3f}")
print(f"{'t0/T':>6s}{'RMS to guide':>14s}{'RMS to data':>13s}{'valid':>8s}{'= target':>10s}{'square':>8s}")
for frac in [0.02, 0.05, 0.1, 0.2, 0.3, 0.4, 0.6, 1.0]:
    t0 = int(frac * T)
    x = ab[t0 - 1].sqrt() * guide + (1 - ab[t0 - 1]).sqrt() * torch.randn(n, 100, generator=gen)
    for t in range(t0, 0, -1):                                     # ancestral steps, sigma^2 = beta_t
        a, a_prev = ab[t - 1], (ab[t - 2] if t > 1 else torch.tensor(1.0))
        beta = 1 - a / a_prev
        e = eps_cfg(x, t, None, 0)
        x = (x - beta / (1 - a).sqrt() * e) / (1 - beta).sqrt()
        if t > 1:
            x = x + beta.sqrt() * torch.randn(n, 100, generator=gen)
    d_data, j = nearest(x)
    d_guide = (x - guide).pow(2).mean(1).sqrt()
    valid = (torch.sign(x)[:, None, :] == data[None]).all(-1).any(1).float().mean()
    print(f"{frac:6.2f}{d_guide.mean():14.3f}{d_data.mean():13.3f}{valid:8.1%}"
          f"{(j == target).float().mean():10.1%}{(label[j] == 0).float().mean():8.1%}")

# ---- (2) DDIM inversion of 75 new images (each shape and offset, new pixel noise), 50 steps.
S = 50
real = data + 0.1 * torch.randn(75, 100, generator=gen)           # images to edit
tau = torch.round(torch.linspace(1, T, S)).long()                  # tau_1 < ... < tau_S
a_seq = torch.cat([torch.ones(1), ab[tau - 1]])                    # a_seq[i] = alpha_bar at tau_i

def invert(x0, y, s):
    traj, x = [x0], x0
    for i in range(1, S + 1):                                      # eps at the current point, label tau_i
        x = ddim_step(x, eps_cfg(x, int(tau[i - 1]), y, s), a_seq[i - 1], a_seq[i])
        traj.append(x)
    return traj                                                    # traj[i] = x at tau_i

def generate(xT, y, s, nulls=None):
    x = xT
    for i in range(S, 0, -1):
        e = eps_cfg(x, int(tau[i - 1]), y, s, None if nulls is None else nulls[i - 1])
        x = ddim_step(x, e, a_seq[i], a_seq[i - 1])
    return x

def rms(a, b):
    return (a - b).pow(2).mean(1).sqrt()

print(f"\nDDIM inversion and reconstruction of 75 new images, {S} steps")
print(f"{'invert s':>9s}{'generate s':>11s}{'median RMS':>12s}{'max RMS':>9s}{'image kept':>12s}{'std x_T':>9s}")
for s_inv, s_gen in [(0, 0), (1, 1), (3, 3), (7.5, 7.5), (1, 7.5)]:
    xT = invert(real, label, s_inv)[-1]
    rec = generate(xT, label, s_gen)
    err = rms(rec, real)
    kept = (nearest(rec)[1] == torch.arange(75)).float().mean()
    print(f"{s_inv:9.1f}{s_gen:11.1f}{err.median():12.4f}{err.max():9.4f}{kept:12.1%}{xT.std():9.3f}")
rec2 = generate(invert(real, label, 1)[-1], label, 1)
print(f"deterministic: two inversions and reconstructions differ by at most "
      f"{(rec2 - generate(invert(real, label, 1)[-1], label, 1)).abs().max():.1e}")

# ---- (3) Null-text inversion (Mokady et al.): keep the s = 1 trajectory as pivot, then for
# each step optimize a per-step null embedding so that the s = 7.5 step lands on the pivot.
pivot = invert(real, label, 1)
w, N = 7.5, 10
null = net.emb.weight[NULL].expand(75, -1).clone()
nulls, x = [None] * S, pivot[-1]
for i in range(S, 0, -1):
    null = null.detach().requires_grad_(True)
    opt = torch.optim.Adam([null], lr=1e-2)
    for _ in range(N):
        e = eps_cfg(x, int(tau[i - 1]), label, w, null)
        l = (ddim_step(x, e, a_seq[i], a_seq[i - 1]) - pivot[i - 1]).pow(2).mean()
        opt.zero_grad(); l.backward(); opt.step()
    null = null.detach()
    nulls[i - 1] = null
    with torch.no_grad():
        x = ddim_step(x, eps_cfg(x, int(tau[i - 1]), label, w, null), a_seq[i], a_seq[i - 1])
err = rms(x, real)
print(f"null-text inversion, s = 7.5: median RMS {err.median():.4f}, max {err.max():.4f}, "
      f"image kept {(nearest(x)[1] == torch.arange(75)).float().mean():.1%}")

# ---- (4) An edit: invert each square with s = 1 to step k of 50, then denoise from there
# as a plus with s = 3. Is the result a plus, and is it at the square's offset?
sq = label == 0
plus = torch.ones(int(sq.sum()), dtype=torch.long)
off = torch.arange(75)[sq] % 25                                    # offset index of each square
print("\nedit square -> plus: invert to step k of 50, regenerate with class plus, s = 3")
print(f"{'k':>4s}{'plus':>8s}{'same offset':>13s}{'still square':>14s}")
for k in [10, 12, 14, 16, 20, 50]:
    x = pivot[k][sq]
    for i in range(k, 0, -1):
        x = ddim_step(x, eps_cfg(x, int(tau[i - 1]), plus, 3), a_seq[i], a_seq[i - 1])
    j = nearest(x)[1]
    print(f"{k:4d}{(label[j] == 1).float().mean():8.1%}"
          f"{((label[j] == 1) & (j % 25 == off)).float().mean():13.1%}{(label[j] == 0).float().mean():14.1%}")

Output (PyTorch 2.14.1, CPU, about 17 s; two runs gave identical output):

seed 0: final training loss 0.062

SDEdit on one guide, 400 runs per t0 (RMS in [-1, 1] units)
guide: RMS to nearest training image 0.602
  t0/T  RMS to guide  RMS to data   valid  = target  square
  0.02         0.137        0.529    0.0%    100.0%  100.0%
  0.05         0.300        0.414    0.5%    100.0%  100.0%
  0.10         0.450        0.300   60.0%     99.0%   99.8%
  0.20         0.574        0.282   63.2%     66.3%   81.5%
  0.30         0.699        0.250   71.0%     27.0%   53.8%
  0.40         0.775        0.222   80.0%     10.0%   38.0%
  0.60         0.839        0.192   87.0%      3.2%   31.7%
  1.00         0.843        0.210   82.0%      2.0%   33.8%

DDIM inversion and reconstruction of 75 new images, 50 steps
 invert s generate s  median RMS  max RMS  image kept  std x_T
      0.0        0.0      0.0397   0.0492      100.0%    0.743
      1.0        1.0      0.0410   0.0669      100.0%    0.772
      3.0        3.0      0.0460   0.3528      100.0%    1.021
      7.5        7.5      0.7059   1.2430       33.3%    1.154
      1.0        7.5      0.7813   1.5083       61.3%    0.772
deterministic: two inversions and reconstructions differ by at most 0.0e+00
null-text inversion, s = 7.5: median RMS 0.0431, max 0.0754, image kept 100.0%

edit square -> plus: invert to step k of 50, regenerate with class plus, s = 3
   k    plus  same offset  still square
  10    0.0%         0.0%        100.0%
  12   16.0%         8.0%         84.0%
  14   64.0%        12.0%         36.0%
  16   96.0%        12.0%          4.0%
  20  100.0%         8.0%          0.0%
  50  100.0%         8.0%          0.0%

SDEdit. The rows trace the realism–faithfulness trade-off. At $t_0 = 0.02T$ and $0.05T$ the output stays close to the guide but almost never becomes a valid shape: too little noise to wash out the grey strokes and the blob. Around $0.1T$ most outputs become valid images and nearly all are the intended square at the intended place. Beyond that, the noise erases the guide’s information first about position and then about shape: by $0.6T$ the intended image is near its chance rate of 1 in 75 and squares are about a third of outputs, as for unconditional samples. Realism here saturates near 80–90% valid, the quality of this small model’s own samples. The window is narrow and depends on the trained model: with seeds 1 and 2 (the script’s argument), 75.2% and 1.5% of outputs were valid at $0.1T$, with the intended square in 88.7% and 79.8% of runs, and seed 2’s model had no good setting, since at $0.2T$ only 40.3% were valid and 22.2% the intended square. In our reading this is why Meng et al. suggest searching for $t_0$ per task rather than fixing one value.

Inversion. With the unconditional or the plain conditional model, the 50-step inversion regenerates each image with a median RMS error of about 0.04, well below the 0.1 pixel noise that distinguishes it from its template, so the particular image is recovered and not just its shape. At $s = 3$ the worst error is already 0.35, and at $s = 7.5$ inversion breaks down: only a third of the reconstructions are even the right image, and the inverted latents are wider than a unit Gaussian. Inverting at $s = 1$ and regenerating at $s = 7.5$, the setting of a guided edit, also fails. Null-text inversion repairs it by changing only the null embedding: every image is recovered, with errors close to unguided inversion. With seeds 1 and 2 the pattern was the same: 25.3% and 29.3% of images kept at $s = 7.5$, and 100% with null-text inversion. This is Mokady et al.’s finding reproduced at toy scale; the size of the error depends on the network and the scale.

The edit. Regenerating an inverted square as a plus changes the class once the inversion goes deep enough, from about step 12 to 16 of 50, but the plus lands at the square’s offset only 8–12% of the time, near the 4% expected by chance (seeds 1 and 2: at most 8%). Inverting less deeply keeps the square. In our reading, nothing in this network ties the position of a plus to that of a square, so swapping the condition re-samples the layout; this mirrors Prompt-to-Prompt’s observation that a changed word with the same seed gives a different layout, and it is the gap that attention injection, feature injection and trained editing models fill in large models.

Applications

Meng et al. aim SDEdit at everyday users, with or without artistic training, who want to create and edit photo-realistic images (abstract; ethics statement of arXiv v2). The papers above demonstrate object and background replacement and image extrapolation (Avrahami et al.), changes of style, season, medium and setting (Brooks et al.), translation of sketches and drawings into realistic images (Tumanyan et al.), subjects re-rendered in new contexts (Ruiz et al.), and changes of pose and shape (Pan et al.).

Risks and Provenance

Editing tools make convincing manipulation easier. The authors of the methods above say so in their own impact statements, which describe risks rather than measure them: Blended Diffusion’s authors note that their method adds nothing beyond what a professional editor could do but makes manipulation easier for people with little skill, and that it could lend credibility to fake news, erode trust in photographic evidence, and serve bullying, harassment or extortion (Avrahami et al., §7). SDEdit’s authors note that its outputs can be hard to tell from real photos and that forensic detectors mostly targeted GAN images (arXiv v2, ethics statement); DragGAN’s authors call for respecting personality rights when editing a real person’s pose or expression (§4.3); and textual inversion’s authors write that personalization could make forged images of non-public individuals more convincing, though their method did not yet preserve identity well enough for this to be a concern (Gal et al., 2023, arXiv v1, §7).

Provenance records how an image was made instead of guessing from its pixels. The C2PA technical specification, version 2.5 of October 2026, defines a manifest of signed assertions about an asset’s creation and edits, cryptographically bound to the asset’s bytes by hashes (a hard binding); a fingerprint or invisible watermark (a soft binding) lets an asset whose manifest was removed be matched to a copy kept in a provenance store (§2.3, §12.2.1, §12.2.3.1). An asset created by a generative model starts its action list with a c2pa.created action whose digital source type is trainedAlgorithmicMedia (§16.4.1.2). An optional AI-disclosure assertion, added in version 2.4, records the models used and a level of human oversight; the specification’s table pairs the source type compositeWithTrainedAlgorithmicMedia with the level human_validated for AI edits reviewed by a human editor (§16.8.2.3; Appendix F.2). The specification presents assertions as trust signals attributed to their signer, for people to weigh (§1.3), and states that a signer is not responsible for metadata it did not assert, such as an EXIF location (§12.2.2.9); it describes how to record and verify claims, not whether an editing tool records them. As of October 2026 we read no source measuring how widely editing tools write these manifests.

Open Problems

• Evaluation. InstructPix2Pix’s authors list how to evaluate instruction-based editing as an open question (Brooks et al., §5); in our reading, automatic scores trade fidelity against edit strength and human studies are not comparable across papers.
• Structural and spatial edits. Moving, swapping and counting objects, and viewpoint changes, fail in InstructPix2Pix (Brooks et al., §5, Fig. 13).
• Fast, faithful and local edits. Per-image optimization (null-text inversion, DragDiffusion’s LoRA) costs seconds to minutes, guidance still conflicts with exact reconstruction, and edits without masks leak into regions the request did not mention.
• Provenance. Signed provenance helps only where tools write and platforms keep it.