Image Inpainting

Image inpainting, also called image completion or hole filling, fills a marked region of an image with content that blends with the rest. The region may be damaged (a scratch) or unwanted (a passer-by). Outpainting, or image extension, is the same task with the unknown region outside the original borders. Large holes need invented content, making inpainting conditional image generation.

Problem Definition

Let $x \in \mathbb{R}^{H \times W \times C}$ be the image and $m \in \{0, 1\}^{H \times W}$ a mask with $m = 1$ on the missing region $\Omega$. The method sees $y = (1 - m) \odot x$ and $m$, and returns

$$\hat x = (1 - m) \odot y + m \odot g(y, m),$$

keeping the known pixels and filling $\Omega$ with the prediction $g$ (partial convolutions use the opposite convention, 1 for valid pixels). A thin scratch is almost determined by its surroundings; for a large hole many fills are plausible, and the target is a sample from $p(x_\Omega \mid y)$ rather than the one ground truth.

Input and Output

The input is the image and the mask, drawn by a user or produced by a detector, sometimes with guidance such as a sketch (DeepFill v2) or a text prompt (Stable Diffusion’s inpainting models); the output is an image of the same size. Early networks filled the hole with the mean pixel value (Pathak et al., 2016, §4; Iizuka et al., 2017, §3.5).

Variants

• Mask shape. Context Encoders filled a central square, and for feature learning random blocks or segmentation-shaped regions (§3.3), and Iizuka et al. random rectangles (§3.5). Liu et al. (2018, §4.1) released an irregular mask dataset from video occlusion estimates, with 12,000 test masks in six hole-to-image area ranges from (0.01, 0.1] to (0.5, 0.6], with and without holes near the border. DeepFill v2 generates free-form brush strokes on the fly (Yu et al., 2019, §3.4), and LaMa samples wide strokes and boxes (Suvorov et al., 2022, §2.3).
• Object removal wants background, not a new object.
• Outpainting. The context lies on one side only. Iizuka et al. name extrapolation as a failure case (§4.8), and Teterwak et al. (2019) found unmodified inpainting methods poor at it; their Boundless GAN conditions the discriminator on Inception-v3 features of the whole ground-truth image (§3.2). RePaint’s “Expand” mask keeps only the central $64 \times 64$ of a $256 \times 256$ image (Lugmayr et al., 2022, §5.3).

Challenges

Structure and texture. Diffusion fills continue edges but blur texture; texture synthesis may break long structures.

Large holes. Iizuka et al. used dilated convolutions because without them each output pixel would depend on only a $99 \times 99$ input region, too small for larger holes, against $307 \times 307$ with them (§3.2).

Invalid inputs. Ordinary convolutions mix real pixels with hole placeholders, causing colour discrepancy and blur (Liu et al., 2018, abstract).

Approaches

Diffusion PDEs. Bertalmío et al. (2000) filled a region so that isophotes, the level lines of intensity, arriving at its boundary continue inside (abstract). Their scheme approximates

$$\frac{\partial I}{\partial t} = \nabla^\perp I \cdot \nabla(\Delta I),$$

which transports the image smoothness $\Delta I$, the Laplacian, along the isophotes (as summarized by Bertalmío, Bertozzi & Sapiro, 2001, Eq. 1). Such methods suit scratches and text. The simplest relative, used in the example below, solves Laplace’s equation $\Delta u = 0$ in $\Omega$ with the known pixels as boundary values; in our reading it only blends the boundary.

Exemplar-based methods. Efros and Leung (1999) grew texture pixel by pixel, inward from a hole’s edges: they matched the known part of a pixel’s neighbourhood against every window of the sample image and drew the centre of one of the near-best matches (§2.1–2.2). Criminisi et al. (2004) copied whole patches, default $9 \times 9$, and filled first the boundary patch of highest priority

$$P(p) = C(p)\, D(p), \qquad D(p) = \frac{|\nabla I_p^\perp \cdot n_p|}{\alpha},$$

where the confidence $C(p)$ is the fraction of reliable pixels in the patch, $n_p$ is the normal to the fill front and $\alpha$ normalizes intensities (§III). The data term pulls strong edges into the hole first, so lines connect. PatchMatch (Barnes et al., 2009) finds approximate nearest-neighbour patches by random sampling and propagation of good matches to neighbours, 20 to 100 times faster than earlier search (abstract), enabling interactive completion. Exemplar methods give sharp texture but only reuse the image’s content.

Learned feed-forward models. Context Encoders trained an encoder–decoder to fill a missing region, with a masked $L_2$ loss plus a GAN loss weighted $0.999$ to $0.001$, on $128 \times 128$ images (§3.2, §5.1; arXiv v2, Appendix A). Iizuka et al. made the network fully convolutional with dilated layers and added a global discriminator on the whole image and a local one on a $128 \times 128$ patch around the hole (§3). Partial convolutions (Liu et al., 2018, Eq. 1–2) use only valid pixels, rescale the result and update the mask:

$$x' = \begin{cases} W^\top (X \odot M)\, \dfrac{\mathrm{sum}(\mathbf{1})}{\mathrm{sum}(M)} + b, & \mathrm{sum}(M) > 0 \\ 0, & \text{otherwise,} \end{cases} \qquad m' = [\mathrm{sum}(M) > 0].$$

Here $M$ marks valid pixels in the window. Liu et al. built a U-Net from them. Gated convolutions (DeepFill v2) learn a soft mask, $\phi(W_f * I) \odot \sigma(W_g * I)$, with a patch-wise spectrally normalized discriminator (Yu et al., 2019, §3.1–3.2). LaMa (Suvorov et al., 2022) uses fast Fourier convolutions, whose global branch applies a $1 \times 1$ convolution to the real 2D Fourier transform of the feature map, so every layer sees the whole image (§2.1). A perceptual loss from a dilated segmentation ResNet-50 and large training masks complete it (§2.2–2.3). Trained at $256 \times 256$, its FFC models kept much more quality than the other models at $1536 \times 1536$ (§3.3, Figs. 5–6).

Diffusion models. RePaint (Lugmayr et al., 2022) uses an unconditional DDPM trained without masks. At each reverse step, known pixels are re-noised from the input and unknown ones come from the model (shared diffusion notation):

$$x_{t-1} = (1 - m) \odot \left(\sqrt{\bar\alpha_{t-1}}\, x_0 + \sqrt{1 - \bar\alpha_{t-1}}\, \epsilon\right) + m \odot x_{t-1}^{\text{model}}, \qquad \epsilon \sim \mathcal{N}(0, I).$$

The paper’s Eq. 8 writes $\bar\alpha_t$ for this step, its Algorithm 1 scales the noise by $1 - \bar\alpha_t$ where Eq. 8a’s variance implies the square root, and its text and equations use $m$ for opposite regions. This matched texture but not semantics, so RePaint resamples: it re-noises $x_{t-1}$ back to $x_t$ and repeats the step, using $T = 250$ steps, $r = 10$ resamplings and jumps of $j = 10$ steps (§4.2, §5.1). At equal cost, resampling lowered LPIPS from 0.168 to 0.134 (32 ImageNet images, wide masks), while more sampling steps changed it little (Table 2).

Conditioned latent diffusion. Rombach et al. (2022) trained on the task itself, concatenating the masked image to the latent (arXiv v2, Table 15). On 30,000 Places crops at $512 \times 512$, their largest model reached FID 1.50 on all test masks and 9.39 on masks covering 40–50% of the image, against LaMa’s printed 2.21 and 12.0 (Table 7). Their smaller model with attention beat LaMa’s recomputed FID (2.15 against 2.23), as the text says, but not CoModGAN’s printed 1.82. Stable Diffusion’s v1.5 inpainting checkpoint follows this design: Runway’s v1.5 model card (archived June 2024) describes five extra U-Net input channels, four for the encoded masked image and one for the mask, zero-initialized and trained for 440,000 steps, with the whole image masked in 25% of examples.

Datasets and Benchmarks

Dataset	Content	Use in the papers above
Paris StreetView (Doersch et al., 2012)	Street-level views of Paris	Context Encoders’ semantic inpainting (Table 1)
Places2 (Zhou et al., 2018)	Scene photographs	Iizuka et al. trained on 8,097,967 images; Liu et al. used the original splits; LaMa and LDM tested on $512 \times 512$ crops
CelebA-HQ (Karras et al., 2018)	30,000 faces	Liu et al. split 27,000/3,000; LaMa and RePaint at $256 \times 256$
ImageNet	Object-centric photographs	Context Encoders, Liu et al., RePaint
Irregular masks (Liu et al., 2018)	55,116 training and 12,000 test masks	Hole-size-stratified evaluation

Numbers are comparable only under the same masks; RePaint reuses LaMa’s wide and narrow masks (§5.3), and Rombach et al. its mask code (arXiv v2, Appendix E.2.2).

Evaluation Metrics

Pixel metrics. $\ell_1$ error, PSNR and SSIM (see image denoising) compare the fill with the original. They suit small holes but reward averages in large ones. Pathak et al. observed that an $L_2$ loss prefers the blurry mean (§3.2): the conditional mean minimizes expected squared error, and a sharp sample from $p(x_\Omega \mid y)$ has twice that error on average, 3 dB lower PSNR (our derivation). In Liu et al.’s Table 1, PatchMatch had higher PSNR than the GAN-based methods of Iizuka et al. and Yu et al.’s earlier contextual-attention model at every hole size. The table also contradicts the text’s claim that partial convolutions beat all methods on these measures: for the smallest holes PatchMatch has lower $\ell_1$ error and, with border holes, higher SSIM, and for the largest holes without border the ordinary-convolution U-Net has higher PSNR.

Perceptual and distribution metrics. LPIPS (Zhang et al., 2018) compares deep features; FID compares feature statistics of filled and real images. LaMa argues that they suit large masks with many plausible completions (§3), but LPIPS still needs the ground truth: LaMa beat RePaint on LPIPS for wide masks on 100 CelebA-HQ images (0.045 against 0.059), but raters preferred RePaint in 58.2% of pairwise votes (Table 1). Rombach et al. attribute LaMa’s lower LPIPS to its single, more average output (§4.5).

User studies remain common: Iizuka et al.’s raters judged 77.0% of completed CelebA faces real, and 96.5% of real ones (§4.6).

Example: Smooth Fill against Patch Copying

A 10 × 10 hole is cut from 32 × 32 noisy synthetic textures: stripes and a checkerboard with period 6 pixels, and smoothed noise with no period. The fills are the known pixels’ mean, a harmonic fill (Laplace’s equation by Jacobi iteration), and a greedy exemplar fill copying the best $7 \times 7$ patch around the boundary pixel with the most known neighbours. Inside the hole the script prints PSNR, mean $3 \times 3$ local variance (texture energy) and the fill’s correlation with itself shifted by one period.

import numpy as np
from scipy.ndimage import gaussian_filter, uniform_filter

N, P = 32, 6                                   # image size, texture period in pixels
yy, xx = np.mgrid[:N, :N]
hole = np.zeros((N, N), bool); hole[11:21, 11:21] = True   # 10x10 square hole

def texture(kind, rng):
    if kind == "stripes":                      # vertical stripes, period 6
        clean = 0.5 + 0.4 * np.sign(np.sin(2 * np.pi * (xx + 0.5) / P))
    elif kind == "checker":                    # checkerboard, cells of 3 px
        clean = 0.1 + 0.8 * (((xx // 3) + (yy // 3)) % 2)
    else:                                      # stochastic: smoothed noise, no period
        g = gaussian_filter(rng.normal(size=(N, N)), 1.5, mode="wrap")
        clean = 0.5 + 0.4 * g / np.abs(g).max()
    return np.clip(clean + rng.normal(0, 0.03, (N, N)), 0, 1)

def harmonic_fill(img, hole, iters=3000):
    """Solve Laplace's equation in the hole (Jacobi), known pixels as Dirichlet data."""
    u = img.copy(); u[hole] = img[~hole].mean()
    for _ in range(iters):
        avg = 0.25 * (np.roll(u, 1, 0) + np.roll(u, -1, 0) + np.roll(u, 1, 1) + np.roll(u, -1, 1))
        u[hole] = avg[hole]
    return u

def exemplar_fill(img, hole, w=7):
    """Greedy patch copy: fill the boundary pixel with the most known neighbours first,
    copying the best SSD-matching fully known patch (Efros-Leung / Criminisi style)."""
    r = w // 2
    u, known = img.copy(), ~hole
    u[hole] = 0.0
    centres = [(i, j) for i in range(r, N - r) for j in range(r, N - r)
               if known[i - r:i + r + 1, j - r:j + r + 1].all()]
    src = np.stack([img[i - r:i + r + 1, j - r:j + r + 1] for i, j in centres])
    while not known.all():
        conf = uniform_filter(known.astype(float), w, mode="constant")
        front = np.argwhere(~known & (conf > 0))
        i, j = max(front, key=lambda p: (conf[p[0], p[1]], -p[0], -p[1]))
        i0, i1, j0, j1 = max(i - r, 0), min(i + r + 1, N), max(j - r, 0), min(j + r + 1, N)
        tgt, m = u[i0:i1, j0:j1], known[i0:i1, j0:j1]
        cand = src[:, i0 - i + r:i1 - i + r, j0 - j + r:j1 - j + r]
        ssd = (((cand - tgt) ** 2) * m).sum((1, 2))
        best = cand[np.argmin(ssd)]
        u[i0:i1, j0:j1][~m] = best[~m]
        known[i0:i1, j0:j1] = True
    return u

def psnr(a, b):  return 10 * np.log10(1.0 / ((a[hole] - b[hole]) ** 2).mean())
def local_var(a):                              # mean 3x3 local variance inside the hole
    return (uniform_filter(a**2, 3) - uniform_filter(a, 3) ** 2)[hole].mean()
def period_corr(a):                            # hole correlated with itself shifted by one period
    h = a[11:21, 11:21]
    x, y = h[:, :-P].ravel(), h[:, P:].ravel()
    if x.std() < 1e-9 or y.std() < 1e-9:       # a flat fill has no periodic structure
        return 0.0
    return np.corrcoef(x, y)[0, 1]

fills = {"mean": lambda im: np.where(hole, im[~hole].mean(), im),
         "harmonic": harmonic_fill, "exemplar": exemplar_fill}
res = {}
for seed in range(5):
    for kind in ["stripes", "checker", "stochastic"]:
        rng = np.random.default_rng(seed)
        truth = texture(kind, rng)
        damaged = np.where(hole, 0.0, truth)
        for name, f in fills.items():
            out = f(damaged) if name == "mean" else f(damaged, hole)
            res.setdefault((kind, name), []).append((psnr(out, truth), local_var(out), period_corr(out)))
        res.setdefault((kind, "truth"), []).append((np.inf, local_var(truth), period_corr(truth)))

print(f"{'texture':11s}{'fill':10s}{'PSNR dB':>9s}{'local var':>11s}{'period corr':>13s}   (seed 0)")
for (kind, name), v in res.items():
    p, lv, pc = v[0]
    ps = "--" if np.isinf(p) else f"{p:.1f}"
    pcs = "--" if kind == "stochastic" else f"{pc:.2f}"   # no period to measure
    print(f"{kind:11s}{name:10s}{ps:>9s}{lv:11.4f}{pcs:>13s}")
print("PSNR range over seeds 0-4:")
for kind in ["stripes", "checker", "stochastic"]:
    s = "  ".join(f"{n} {min(x[0] for x in res[(kind, n)]):.1f}-{max(x[0] for x in res[(kind, n)]):.1f}"
                  for n in fills)
    print(f"  {kind:11s}{s}")
wins = sum(h[0] > e[0] for h, e in zip(res[("stochastic", "harmonic")], res[("stochastic", "exemplar")]))
print(f"stochastic: harmonic PSNR above exemplar in {wins} of 5 seeds")

Output (NumPy 2.3.5, SciPy 1.16.3, under 1 second):

texture    fill        PSNR dB  local var  period corr   (seed 0)
stripes    mean            8.0     0.0164         0.00
stripes    harmonic        6.9     0.0148         0.70
stripes    exemplar       27.8     0.0977         0.99
stripes    truth            --     0.1003         0.99
checker    mean            7.9     0.0177         0.00
checker    harmonic        7.0     0.0188         0.47
checker    exemplar       28.8     0.1363         1.00
checker    truth            --     0.1394         0.99
stochastic mean           19.4     0.0011           --
stochastic harmonic       20.5     0.0010           --
stochastic exemplar       14.1     0.0082           --
stochastic truth            --     0.0044           --
PSNR range over seeds 0-4:
  stripes    mean 7.9-8.0  harmonic 6.8-7.0  exemplar 26.2-27.8
  checker    mean 7.9-8.1  harmonic 7.0-7.1  exemplar 26.9-28.8
  stochastic mean 15.5-21.8  harmonic 16.7-23.7  exemplar 14.1-19.1
stochastic: harmonic PSNR above exemplar in 3 of 5 seeds

On the periodic textures the harmonic fill is smooth, about a seventh of the true local variance, and on stripes its PSNR is even below the flat fill’s, because the dark columns flanking the hole pull the surface down. The exemplar fill copies patches in phase with the boundary and restores the texture: local variance 0.0977 against 0.1003, period correlation 0.99, PSNR 26 to 29 dB over seeds 0 to 4. On the stochastic texture no copied patch matches the hidden pixels and the ranking flips: the exemplar fill has texture energy (more than the truth in seed 0) but lower PSNR than the blurry harmonic fill in 3 of 5 seeds, the failure of pixel metrics described above.

Applications

• Object removal and restoration: wires, superimposed dates, old photographs and damaged film (Bertalmío et al., 2000, abstract), people (Criminisi et al., 2004, Fig. 1), watermarks and distracting objects (Yu et al., 2019, abstract).
• Image extension for panoramas, virtual-reality viewpoints and aspect-ratio changes (Teterwak et al., 2019, §1).
• Representation learning. Context Encoders inpainted as a pretext task for self-supervised learning, a line continued by masked autoencoders.

Open Problems

• Evaluation. For extreme masks, realistic completions can differ greatly from the ground truth, making quantitative evaluation hard (Lugmayr et al., §6).
• Generalization. RePaint attributes baseline artifacts on unusual masks to overfitting to training masks (§5.3), and LaMa shows that transfer to higher resolutions depends on the architecture.
• Speed. RePaint’s 250 steps with 10 resamplings mean thousands of network evaluations per image (our count), against one pass for LaMa.
• Manipulation. Retouching people out of photographs is older than digital inpainting; Bertalmío et al. (2001, §1) cite the “airbrushing” of political enemies. Rombach et al. (2022, §5) warn that cheaper generative models make manipulated images and misinformation easier to create.