Image Matting

Image matting estimates, for every pixel of a photograph, how much of the pixel belongs to a foreground object and what colour that object has there. The result is an alpha matte: an image-sized map of opacities between 0 (pure background) and 1 (pure foreground), usually together with the unmixed foreground colours. With the matte the object can be lifted out and composited over a new background without the jagged edges, colour fringes and lost hair strands that a binary mask produces. Chuang, Curless, Salesin and Szeliski (2001) note that the matte is also called the key, and Smith and Blinn (1996) that “key” and “keying” are the video industry’s terms for the film industry’s “matte” and “matting”; extracting a matte is called pulling it. Matting differs from image segmentation, which assigns each pixel to one region: along a soft boundary, through hair or fur, in motion blur and in semi-transparent material, a pixel’s colour is a mixture, and matting recovers the mixing proportion. Deep Image Matting and most of its successors are built on the encoder–decoder design of dense prediction.

A note on sources. Porter and Duff’s (1984) compositing paper and the alphamatting.com benchmark paper of Rhemann et al. (2009) could not be read for this article; what is said about them comes from Chuang et al. (2001), Xu et al. (2017) and Szeliski’s textbook, and is attributed to them.

Problem Definition

The forward operation is compositing. A foreground colour $F$ is laid over a background colour $B$ with opacity $\alpha$, and the observed colour is

$$C = \alpha F + (1 - \alpha) B .$$

Chuang et al. (2001) call this the compositing equation and describe it as the “over” operation; they credit Porter and Duff (1984) with introducing the alpha channel as the digital counterpart of the film matte. Smith and Blinn (1996) work in the premultiplied form, where the stored foreground colour already includes the factor $\alpha$, so that the composite is $C = C_o + (1 - \alpha_o) C_b$ with $C_o = \alpha_o F$; they note that the same arithmetic then applies to the colour and alpha channels, and that derivations without premultiplication “are not so elegant”. Matting is the inverse problem: given $C$, find $\alpha$, $F$ and, usually as a by-product, $B$.

The inverse is under-constrained. For a colour image each pixel gives three measurements, one per channel, and has seven unknowns: three for $F$, three for $B$ and one for $\alpha$ (Chuang et al., 2001; Levin, Lischinski and Weiss, 2008; Szeliski, 2022, §10.4). Even when the background is known exactly, as in front of a blue screen, there are three equations and four unknowns. Smith and Blinn (1996) use this count to show that single-backing matting, as commonly practised, has an infinity of solutions, so that no algorithm can pull a correct matte from one image without further assumptions; in practice a person tunes the controls of a matting device until the result looks right.

Every matting method therefore adds information. It comes in three forms:

• Extra measurements, by controlling the capture: a constant backing colour, several known backgrounds, or a photograph of the background without the subject.
• User input that fixes $\alpha$ in parts of the image: a trimap or scribbles (below).
• Priors on $F$, $B$ and $\alpha$: local colour models, smoothness of the foreground and background colours, or, in deep matting, the statistics of mattes learned from training data.

Szeliski (2022, §10.4.1) adds a practical condition that applies to all of these: colours should be linearized before processing, for every matting algorithm. (In our reading, the reason is that the compositing equation describes the mixing of linear light, not of gamma-encoded pixel values.)

Input and Output

Input. An image, plus one of the following:

• A trimap: a three-way labelling of the image into definite foreground ($\alpha = 1$), definite background ($\alpha = 0$) and an unknown region where $\alpha$ is to be estimated. Levin et al. (2008) describe it as a rough, typically hand-drawn segmentation. Methods that estimate $F$ and $B$ from nearby known pixels need the unknown band to be narrow, which makes trimaps laborious for hair and for objects with many holes.
• Scribbles: a few strokes with a foreground brush and a background brush, which constrain only the pixels they cover. Levin et al. (2008) designed closed-form matting so that scribbles suffice, and added brushes that set $F$ and $B$ explicitly under a stroke or declare that they are constant there.
• A background image: a photograph of the same scene without the subject (Sengupta et al., 2020; Lin et al., 2021).
• A coarse mask of any quality, from a segmentation network or a rough selection (Yu et al., 2021).
• Nothing beyond the image, in trimap-free or automatic matting.

Output. The alpha matte, at the image’s resolution, and in most methods the foreground colours $F$ in the unknown region. The foreground colour matters as much as $\alpha$: it follows from the compositing equation that pasting $\alpha C = \alpha^2 F + \alpha (1 - \alpha) B$ instead of $\alpha F$ over a new background carries the old background’s colour along the edge. Chuang et al. (2001) and Levin et al. (2008) estimate $F$ and $B$ explicitly; Levin et al. (2008, §7) solve for them after $\alpha$, by least squares on the compositing equation with smoothness priors on $F$ and $B$ that are stronger where the matte has edges.

Unlike the per-pixel labels or probabilities of semantic segmentation and salient object detection, a matte’s fractional values are mixing proportions: 0.5 means half coverage, not uncertainty.

Variants

• Constant-colour (blue screen) matting. The subject is filmed in front of a uniform backing, originally bright blue and now more often green (Szeliski, 2022, §10.4.1). Smith and Blinn (1996) give it its mathematical treatment, call it constant colour matting in general, and describe the patented solutions of Petro Vlahos, whose “color-difference” technique rests on an assumption about the subject’s blue and green channels.
• Triangulation matting. Shooting the same still object against two different known backings gives six equations for four unknowns, and the problem has a closed-form solution as long as the two backings differ at every pixel (Smith and Blinn, 1996). Ground-truth mattes for benchmarks are made this way or with more backings: Chuang et al. (2001) solved an overdetermined system from photographs against five constant colours.
• Difference matting. The background is arbitrary but known, as for a static camera. Chuang et al. (2001) describe the simple version, a thresholded difference from the background, as error-prone and producing jagged mattes.
• Natural image matting. Nothing is known about the background except what a trimap or scribbles provide. This is the main subject of research and of the benchmarks below.
• Background matting. A natural background, captured in a second photograph, replaces the blue screen (Sengupta et al., 2020).
• Trimap-free, automatic and portrait matting. The method must find the foreground itself, usually for one object category such as people.
• Video matting. Mattes for every frame, consistent over time (Lin et al., 2022).
• Smoke and shadow matting. Translucent media and cast shadows are matted with variants of the same equation: Levin et al. (2008) extract shadows and smoke by fixing the foreground colour to black or white under a scribble, and Szeliski (2022, §10.4.4) describes further shadow, smoke and flash matting methods.

Challenges

• Colour overlap. When foreground and background share colours, a colour model cannot tell which one a pixel belongs to. Levin et al. (2008, Fig. 17) show a background pixel that fits the foreground colour model better than the background’s, so that Wang and Cohen’s (2005) iterative optimization method for unified segmentation and matting, which builds global colour models, marks it as foreground and the error spreads to its neighbours.
• Textured backgrounds. Methods that assume slowly varying foreground and background colours fail when a strong background edge crosses the unknown region; global Poisson matting, which neglects these gradients, did worst in Levin et al.’s test as the background’s gradient strength grew.
• Large fractional regions. Hair, fur, smoke and motion blur produce wide areas where $\alpha$ is neither 0 nor 1, and a trimap must leave all of them unknown.
• Dependence on the trimap. Results change with the width and placement of the unknown band (Xu et al., 2017, Fig. 7), and the most important pixels, the mixed ones, are the ones the user cannot label (Levin et al., 2008).
• Ground truth. Accurate mattes of real scenes need controlled capture against several backings, which rules out moving subjects such as people and animals (Xu et al., 2017), so large training sets are built by compositing.
• Under-determination never disappears. Even with a background photograph, Sengupta et al. (2020) point out that the subject casts shadows and reflections absent from the background shot, and that two hand-held photographs can rarely be aligned exactly.

Approaches

Xu et al. (2017) sort classical natural image matting methods into sampling methods, propagation methods and combinations of the two; Sengupta et al. (2020) use the same two families. Since Deep Image Matting in 2017, learned methods have taken over: Szeliski (2022, §10.4.3) notes that most of the recent algorithms on the alphamatting.com benchmark are deep networks.

Blue screen and known backgrounds

Early blue screen techniques estimated $\alpha$ from linear combinations of the colour channels with user-tuned constants (Szeliski, 2022, §10.4.1). Smith and Blinn (1996) quote the first form from Vlahos’s patents as

$$\alpha_o = 1 - a_1 (B_f - a_2 G_f),$$

clamped to $[0, 1]$, where $B_f$ and $G_f$ are the blue and green channels of the image and $a_1$, $a_2$ are adjustment constants exposed as controls. The formula works for subjects whose blue does not exceed a multiple of their green, and Smith and Blinn analyse why this holds as often as it does. Mishima’s algorithm, as Chuang et al. (2001) describe it, fits two polyhedra around the mean backing colour in RGB space, one shrunk to contain all background samples and one grown as large as possible without containing foreground samples; a ray from the mean background colour through an observed colour meets the two surfaces at $B$ and $F$, and $\alpha$ is the fractional position of the observed colour between them. Szeliski notes that Mishima’s method can still suffer from blue spill, where translucent pixels around the object’s edges take on some of the backing colour.

Sampling-based matting

Sampling methods estimate $F$ and $B$ for each unknown pixel from known pixels nearby, then solve the compositing equation for $\alpha$. With $F$ and $B$ fixed, the least-squares $\alpha$ is the projection of $C$ onto the line through $B$ and $F$ in colour space:

$$\alpha = \frac{(C - B) \cdot (F - B)}{\lVert F - B \rVert^2} .$$

Chuang et al. (2001) summarize the two leading methods of the time:

• Knockout (a commercial tool described in patents by Berman et al.) extrapolates $F$ and $B$ into the unknown region as weighted averages of known pixels on the region’s perimeter, with weights that fall linearly to zero at twice the distance of the nearest pixel, refines $B$ by projecting $C$ onto a plane through the first background estimate, and computes $\alpha$ by projection onto several colour axes, weighted by each axis’s denominator.
• Ruzon and Tomasi treat the known pixels near a piece of the unknown region as samples, fit axis-aligned Gaussian mixtures to foreground and background, pair the components, and choose $\alpha$ as the value under which an interpolated distribution best explains $C$.

Bayesian matting (Chuang et al., 2001) makes the estimate a maximum a posteriori problem:

$$\arg\max_{F, B, \alpha} \; L(C \mid F, B, \alpha) + L(F) + L(B) + L(\alpha),$$

where $L$ is a log likelihood. The data term is Gaussian in the compositing residual, $-\lVert C - \alpha F - (1 - \alpha) B \rVert^2 / \sigma_C^2$. The foreground term comes from oriented Gaussians fitted to clusters of known and already-estimated foreground colours in a sliding window around the pixel, each sample weighted by $\alpha_i^2$ and by a spatial Gaussian with $\sigma = 8$ pixels; the background term is built the same way with weights $(1 - \alpha_i)^2$. The prior on $\alpha$ is taken as constant. The objective is not quadratic in all unknowns at once, so the method alternates: for fixed $\alpha$, $F$ and $B$ solve a $6 \times 6$ linear system; for fixed $F$ and $B$, $\alpha$ is the projection above. It starts from the mean $\alpha$ of the neighbourhood, tries every pair of foreground and background clusters, and keeps the most likely pair. The solution proceeds inward from the known regions, so estimated pixels feed the colour models of later ones. The authors formulated the method for constant-colour, difference and natural image matting and showed results on blue screen and natural images, with ground truth from five backings for the blue screen case (a thresholded difference matte appears only as a baseline); they reported fewer spill and missing-strand artifacts than Mishima, Knockout and Ruzon–Tomasi (judged visually in their figures).

Later sampling methods changed how candidate pairs are gathered and scored. In Xu et al.’s (2017) summary they sample along the nearest stretch of the trimap boundary, along rays cast from the pixel (shared sampling, Gastal and Oliveira), over the whole boundary (global sampling, He, Rhemann, Rother, Tang and Sun) or from colour clusters (comprehensive sampling, Shahrian, Rajan, Price and Cohen), and score pairs mainly by how well they reconstruct $C$, sometimes adding their distance from the pixel or the separation of the two colours. Several of them use closed-form matting as a post-process (Xu et al., 2017). In our reading, the reason is a shared weakness: each pixel is decided on its own, so when no good pair is in the sample, or a wrong pair reconstructs $C$ by accident, nothing in the method corrects it.

Propagation-based matting

Propagation methods treat the matte as a field and spread the known values into the unknown region through local affinities, solving one global problem. Poisson matting (Sun, Jia, Tang and Shum, 2004, described here through Levin et al. and Szeliski) differentiates the compositing equation, neglects the gradients of $F$ and $B$, and so approximates the matte’s gradient by $\nabla C / (F - B)$; the matte is the function whose gradient best fits this field, a Poisson equation with the trimap as boundary condition. Szeliski (2022, §10.4.3) notes that it works when $F$ and $B$ are well estimated and vary slowly. The random walk matting of Grady et al. propagates scribbles with a graph Laplacian whose affinities depend on colour distances after a learned linear transformation (Levin et al., 2008). Levin et al. (2008) regard GrabCut’s border matting as closer to feathering than to full matting, since it cannot represent wide fuzzy regions.

Closed-form matting (Levin, Lischinski and Weiss, CVPR 2006 and TPAMI 2008) derives the affinities from the compositing equation itself. The derivation below follows the TPAMI version.

Local linear model. For a grayscale image, assume that $F$ and $B$ are approximately constant in a small window $w$. Then $\alpha_i \approx a I_i + b$ for every pixel $i$ in the window, with $a = 1/(F - B)$ and $b = -B/(F - B)$. This does not require the image to be smooth: a discontinuity in $\alpha$ accounts for an edge in $I$. For colour images the model becomes $\alpha_i \approx \sum_c a^c I_i^c + b$, and Levin et al. prove that it holds whenever the foreground colours in the window lie on a line in RGB space and so do the background colours, the colour line model. A line covers, for example, the changing shading of a surface of constant albedo.

Cost. Fit the model in a window around every pixel $j$, with all windows overlapping:

$$J(\alpha, a, b) = \sum_{j} \left( \sum_{i \in w_j} \Big( \alpha_i - \sum_c a_j^c I_i^c - b_j \Big)^2 + \varepsilon \sum_c (a_j^c)^2 \right).$$

The regularizer $\varepsilon \lVert a_j \rVert^2$ makes the fit unique in flat windows and biases the matte toward smoothness, since $a_j = 0$ means $\alpha$ is constant in the window. The overlap is what lets information travel between pixels.

Eliminating $F$ and $B$. For a fixed $\alpha$, the best $(a_j, b_j)$ in each window is a small least-squares problem whose solution is linear in $\alpha$. Substituting it back leaves a quadratic form in $\alpha$ alone, $J(\alpha) = \alpha^\top L \alpha$, where $L$ is the $N \times N$ matting Laplacian with entries

$$L_{ij} = \sum_{k \mid (i, j) \in w_k} \left( \delta_{ij} - \frac{1}{|w_k|} \Big( 1 + (I_i - \mu_k)^\top \big( \Sigma_k + \tfrac{\varepsilon}{|w_k|} I_3 \big)^{-1} (I_j - \mu_k) \Big) \right),$$

with $\mu_k$ and $\Sigma_k$ the mean colour and $3 \times 3$ colour covariance of window $w_k$. Each row of $L$ sums to zero, so constant mattes cost nothing; with $\varepsilon = 0$ each colour channel of the image is also in the null space. Szeliski (2022, §10.4.3, Figure 10.43) gives a geometric reading of the same model: $\alpha$ is constant on a family of parallel planes in colour space between the foreground and background colour lines.

Constraints and solution. Scribbles or a trimap enter as a penalty with a large weight $\lambda$:

$$\alpha = \arg\min_\alpha \; \alpha^\top L \alpha + \lambda (\alpha - b_S)^\top D_S (\alpha - b_S), \qquad (L + \lambda D_S)\, \alpha = \lambda b_S ,$$

where $D_S$ is diagonal with ones at constrained pixels and $b_S$ holds their values. The matte is the global minimum of the cost, found by one sparse linear solve; this is the sense of “closed form”. Levin et al. prove that if $F$ and $B$ satisfy the colour line model in every window and the constraints agree with the true matte, the true matte is an optimal solution for $\varepsilon = 0$, and they show a synthetic smoke composite recovered exactly. The condition is necessary, not sufficient: the null space of $L$ can hold several solutions, and their exact recovery needed a constraint inside every connected component.

Parameters and cost. The authors usually used $3 \times 3$ windows and $\varepsilon$ between $10^{-7}$ and $10^{-5}$. Small $\varepsilon$ keeps the matte as sharp as the image edge but lets image noise into it; large $\varepsilon$ suppresses noise but oversmooths. Wider windows propagate further and are more stable where the colour line model holds, but they are more likely to cover three colours and break it, and they make $L$ denser; the authors instead solve on a coarser image and transfer the per-window coefficients $a$, $b$ to the fine one. Matlab’s direct solver took 20 seconds for a $200 \times 300$ image on a 2.8 GHz CPU; a coarse-to-fine scheme with clamping of near-0 and near-1 values reduced the system but could lose thin hair strands, and a multigrid solver ran in a couple of seconds with a small loss of quality.

Spectral view. $L$ can be written as $D - W$, the form of the graph Laplacian in spectral segmentation such as normalized cuts, but with a “matting affinity” $W$ that can be negative and that comes from the matting model rather than from a hand-set colour bandwidth. The smallest eigenvectors of $L$, which depend only on the image, are nearly piecewise constant over the same regions as good mattes, and Levin et al. bound how much the larger eigenvectors can contribute to a solution; they suggest using the eigenvectors to decide where scribbles are needed.

Results and limits. On 4,000 synthetic composites (2,000 background crops, each composited under a smoke matte and a feathered disk, with a uniform foreground), with trimaps for all methods and summed absolute error as the measure, Levin et al. report that all four methods tested (Wang and Cohen’s, global Poisson, random walk and closed-form) did well on smooth backgrounds, that global Poisson matting failed on strongly textured ones, and that of the other three their method was consistently the most accurate. The quadratic cost places no restriction on $\alpha$ itself, so a region with no constraints can receive any intermediate grey (their Fig. 19), and in their Fig. 18 the matte includes part of a two-coloured background between an actress’s hair and shoulder although the background scribbles covered both colours, a failure they attribute to the method’s lack of a colour model.

Example: closed-form matting against colour sampling

The script below composites a smooth foreground over three backgrounds (a smooth colour gradient, and crops of the coffee and chelsea images bundled with scikit-image) under two ground-truth mattes: a disk with a 4-pixel linear ramp, and a smaller disk with 40 thin, partly transparent strands, a crude stand-in for hair. Each composite gets two trimaps, eroding the pure foreground and pure background by 3 pixels (“narrow”) or 10 (“wide”). Three estimates are scored on the unknown pixels:

• a sampling baseline that, for each unknown pixel, takes the 10 nearest known foreground and 10 nearest known background pixels on the trimap boundary, tries all 100 pairs, and keeps the $\alpha$ of the pair that best reconstructs the pixel’s colour (no smoothing);
• closed-form matting with $3 \times 3$ windows and $\lambda = 100$, solved with SciPy’s sparse direct solver, at both ends of the $\varepsilon$ range the paper gives ($10^{-7}$ and $10^{-5}$), since the paper leaves the choice open;
• the ground-truth matte thresholded at 0.5, a perfect binary segmentation, as a reference for what a hard mask costs.

SAD here is the plain sum of absolute alpha errors over the unknown region (not divided by 1000 as in the Composition-1k code below, since the images are small); MSE is the mean squared error over the same pixels. The Laplacian sums every window, rather than only windows containing an unknown pixel as Levin et al. do for efficiency. NumPy 2.3.5, SciPy 1.16.3, scikit-image 0.25.2; the run took about 35 seconds on our machine, almost all of it in the 24 sparse solves.

import numpy as np
import scipy.sparse as sp
import scipy.sparse.linalg as spla
from scipy import ndimage as ndi
from scipy.spatial import cKDTree
from skimage import data

H = W = 160
yy, xx = np.mgrid[0:H, 0:W].astype(float)


def disk_matte():
    r = np.hypot(yy - 80, xx - 80)
    return np.clip(0.5 - (r - 45) / 4, 0, 1)


def hair_matte(n=40, seed=0):
    rng = np.random.default_rng(seed)
    r = np.hypot(yy - 80, xx - 80)
    alpha = np.clip(0.5 - (r - 40) / 2, 0, 1)
    for _ in range(n):
        th = rng.uniform(0, 2 * np.pi)
        length = rng.uniform(15, 35)
        a = rng.uniform(0.4, 0.9)
        p0 = np.array([80 + 38 * np.sin(th), 80 + 38 * np.cos(th)])
        d = np.array([np.sin(th), np.cos(th)])
        rel = np.stack([yy - p0[0], xx - p0[1]], -1)
        t = np.clip(rel @ d / length, 0, 1)
        dist = np.linalg.norm(rel - t[..., None] * length * d, axis=-1)
        strand = a * (1 - 0.7 * t) * np.clip(1 - dist / 0.8, 0, 1)
        alpha = np.maximum(alpha, strand)
    return alpha


def foreground():
    return np.stack([0.85 + 0.1 * yy / H, 0.65 + 0.1 * xx / W, 0.45 + 0.0 * xx], -1)


def background(name):
    if name == "smooth":
        return np.stack([0.2 + 0.3 * xx / W, 0.4 + 0.2 * yy / H, 0.7 + 0.0 * xx], -1)
    if name == "coffee":
        return data.coffee()[200:360, 300:460] / 255.0
    if name == "chelsea":
        return data.chelsea()[120:280, 40:200] / 255.0


def make_trimap(alpha, r):
    fg = ndi.binary_erosion(alpha >= 1.0, iterations=r) if r else alpha >= 1.0
    bg = ndi.binary_erosion(alpha <= 0.0, iterations=r) if r else alpha <= 0.0
    tri = np.full(alpha.shape, 0.5)
    tri[fg] = 1.0
    tri[bg] = 0.0
    return tri


def matting_laplacian(img, eps, rad=1):
    h, w, c = img.shape
    n = 2 * rad + 1
    m = n * n
    idx = np.arange(h * w).reshape(h, w)
    # all window centres whose window lies inside the image
    wins_img = np.lib.stride_tricks.sliding_window_view(img, (n, n), axis=(0, 1))
    wins_idx = np.lib.stride_tricks.sliding_window_view(idx, (n, n))
    I = wins_img.reshape(-1, c, m).transpose(0, 2, 1)      # (K, m, 3)
    ids = wins_idx.reshape(-1, m)                           # (K, m)
    mu = I.mean(1, keepdims=True)
    D = I - mu
    cov = np.einsum("kmi,kmj->kij", D, D) / m
    inv = np.linalg.inv(cov + (eps / m) * np.eye(c))
    vals = (1 + np.einsum("kmi,kij,knj->kmn", D, inv, D)) / m  # (K, m, m)
    vals = np.eye(m)[None] - vals
    rows = np.repeat(ids, m, axis=1).ravel()
    cols = np.tile(ids, (1, m)).ravel()
    return sp.coo_matrix((vals.ravel(), (rows, cols)), shape=(h * w, h * w)).tocsr()


def closed_form(img, tri, eps, lam=100.0):
    L = matting_laplacian(img, eps)
    known = (tri != 0.5).ravel().astype(float)
    b = np.where(tri.ravel() == 1.0, 1.0, 0.0) * known
    A = L + lam * sp.diags(known)
    alpha = spla.spsolve(A.tocsc(), lam * b)
    return np.clip(alpha.reshape(tri.shape), 0, 1)


def sampling(img, tri, k=10):
    unk = tri == 0.5
    fg_b = (tri == 1.0) & ndi.binary_dilation(unk)
    bg_b = (tri == 0.0) & ndi.binary_dilation(unk)
    fpts, bpts, upts = np.argwhere(fg_b), np.argwhere(bg_b), np.argwhere(unk)
    _, fi = cKDTree(fpts).query(upts, k=k)
    _, bi = cKDTree(bpts).query(upts, k=k)
    C = img[unk]                                            # (U, 3)
    F = img[fpts[fi, 0], fpts[fi, 1]]                       # (U, k, 3)
    B = img[bpts[bi, 0], bpts[bi, 1]]                       # (U, k, 3)
    Fp, Bp = F[:, :, None, :], B[:, None, :, :]             # pairs (U, k, k, 3)
    FB = Fp - Bp
    a = np.einsum("ujkc,ujkc->ujk", C[:, None, None, :] - Bp, FB)
    a = np.clip(a / (np.einsum("ujkc,ujkc->ujk", FB, FB) + 1e-12), 0, 1)
    rec = a[..., None] * Fp + (1 - a[..., None]) * Bp
    err = np.linalg.norm(C[:, None, None, :] - rec, axis=-1).reshape(len(C), -1)
    best = err.argmin(1)
    alpha = tri.copy()
    alpha[unk] = a.reshape(len(C), -1)[np.arange(len(C)), best]
    return alpha


def scores(pred, gt, tri):
    unk = tri == 0.5
    d = pred[unk] - gt[unk]
    return np.abs(d).sum(), (d ** 2).mean()


if __name__ == "__main__":
    F = foreground()
    mattes = {"disk": disk_matte(), "hair": hair_matte()}
    print(f"{'background':9s} {'matte':5s} {'trimap':6s} {'unknown':>7s} | "
          f"{'hard SAD':>8s} {'sampl SAD':>9s} {'CF-7 SAD':>8s} {'CF-5 SAD':>8s} | "
          f"{'sampl MSE':>9s} {'CF-7 MSE':>8s} {'CF-5 MSE':>8s}")
    rows = []
    for bname in ["smooth", "coffee", "chelsea"]:
        B = background(bname)
        for mname, gt in mattes.items():
            img = gt[..., None] * F + (1 - gt[..., None]) * B
            for tname, r in [("narrow", 3), ("wide", 10)]:
                tri = make_trimap(gt, r)
                hard = (gt >= 0.5).astype(float)
                s_h, _ = scores(hard, gt, tri)
                s_s, m_s = scores(sampling(img, tri), gt, tri)
                s_7, m_7 = scores(closed_form(img, tri, 1e-7), gt, tri)
                s_5, m_5 = scores(closed_form(img, tri, 1e-5), gt, tri)
                rows.append((s_h, s_s, s_7, s_5, m_s, m_7, m_5))
                print(f"{bname:9s} {mname:5s} {tname:6s} {int((tri == 0.5).sum()):7d} | "
                      f"{s_h:8.1f} {s_s:9.1f} {s_7:8.1f} {s_5:8.1f} | "
                      f"{m_s:9.5f} {m_7:8.5f} {m_5:8.5f}")
    mean = np.mean(rows, axis=0)
    print(f"{'mean':30s} | {mean[0]:8.1f} {mean[1]:9.1f} {mean[2]:8.1f} {mean[3]:8.1f} | "
          f"{mean[4]:9.5f} {mean[5]:8.5f} {mean[6]:8.5f}")

Output:

background matte trimap unknown | hard SAD sampl SAD CF-7 SAD CF-5 SAD | sampl MSE CF-7 MSE CF-5 MSE
smooth    disk  narrow    4536 |    282.3      12.2      2.7      2.3 |   0.00002  0.00000  0.00000
smooth    disk  wide     12204 |    282.3      74.1      5.8      3.8 |   0.00010  0.00000  0.00000
smooth    hair  narrow    9293 |    428.5      34.7      6.9     24.6 |   0.00003  0.00000  0.00004
smooth    hair  wide     19432 |    428.5     224.8     66.5     57.7 |   0.00037  0.00005  0.00004
coffee    disk  narrow    4536 |    282.3     198.3     88.8     77.4 |   0.01500  0.00287  0.00269
coffee    disk  wide     12204 |    282.3     876.6    281.3    237.4 |   0.04156  0.00450  0.00709
coffee    hair  narrow    9293 |    428.5     468.0    210.5    207.2 |   0.01771  0.00348  0.00366
coffee    hair  wide     19432 |    428.5    2286.8   1118.3   1143.0 |   0.06321  0.03289  0.03075
chelsea   disk  narrow    4536 |    282.3     204.1     55.8     53.7 |   0.00705  0.00082  0.00082
chelsea   disk  wide     12204 |    282.3     719.6    142.4    109.6 |   0.01303  0.00073  0.00065
chelsea   hair  narrow    9293 |    428.5     558.1    221.2    225.0 |   0.01048  0.00323  0.00352
chelsea   hair  wide     19432 |    428.5    1490.7    458.2    605.6 |   0.02158  0.00222  0.00457
mean                           |    355.4     595.7    221.5    228.9 |   0.01585  0.00423  0.00449

On the smooth background the foreground and background colours change so little within a $3 \times 3$ window that the colour line model holds almost exactly, and closed-form matting is nearly exact, close to what Levin et al.’s theorem predicts; its error is largest on the hair matte with the wide trimap. The sampling baseline is between about 1.4 and 20 times worse there in SAD, even though the colours are easy. A likely reason (our reading, not measured here) is that the nearest boundary samples differ slightly from the true $F$ and $B$ at the pixel, so the pair that reconstructs the colour best need not give the right $\alpha$.

The textured backgrounds raise every error. Closed-form matting stays two to almost seven times below the sampling baseline in SAD in every textured row, and its advantage is larger in MSE than in SAD in seven of the eight with $\varepsilon = 10^{-7}$ and in all eight with $10^{-5}$. Since MSE weights large errors more, this says that sampling’s errors are concentrated in fewer, larger mistakes, as expected when each pixel is decided on its own and a background pixel whose colour happens to lie between a foreground and a background sample receives a large $\alpha$ that nothing smooths away. Closed-form matting presumably degrades where the cat’s fur and the coffee scene put more than two colours into one $3 \times 3$ window, which breaks the colour line model.

The hard-mask column is the instructive one. A perfect binary segmentation is exact on every pixel whose alpha is 0 or 1, and its SAD is the same for both trimaps, since every fractional pixel already lies in the narrow band. It beats the sampling baseline in six of the eight textured rows, and beats closed-form matting with either $\varepsilon$ on the hair matte with the wide trimap on both textured backgrounds. The wide band adds only pixels whose true alpha is 0 or 1 and moves the known pixels farther from the boundary, so a matting method can only lose on the added pixels; in these rows the loss outweighs what it gains on the fractional pixels. This is why trimap width matters so much in practice, and why the benchmarks below fix the trimaps. The plan’s expectation that “the wide trimap hurts sampling more than closed-form” depends on how growth is measured: sampling’s SAD grew by more in absolute terms in all six background–matte pairs, but by a larger factor (wide over narrow SAD, against $\varepsilon = 10^{-7}$) in only four.

The choice of $\varepsilon$ changed individual rows more than the averages: $10^{-5}$ gave the lower SAD in eight of twelve rows, and $10^{-7}$ in the other four, all on the hair matte, consistent with Levin et al.’s observation that a larger $\varepsilon$ smooths the matte and costs sharp detail. SAD and MSE also disagree on the ordering of the two in places (for example coffee, disk, wide), since MSE weights a few large errors more than many small ones. One caution on scope: the foreground here is smooth by construction, which suits the colour line model, as did the uniform foreground of Levin et al.’s synthetic test; on the real foregrounds of Composition-1k, closed-form matting ranks below several sampling methods (table below).

Deep matting

Deep Image Matting (Xu, Price, Cohen and Huang, CVPR 2017) was, in its authors’ words, the first method to learn an alpha matte end-to-end from an image and a trimap; earlier deep methods had predicted trimaps for portraits and then applied closed-form matting (Shen et al., 2016), or learned to combine the outputs of closed-form and KNN matting (Cho, Tai and Kweon, 2016). The authors argued that classical methods rely mainly on colour, which fails when foreground and background colours overlap, whereas hair, fur and blurred edges have recurring structure that a network can learn.

• Network. An encoder–decoder takes a 4-channel input, the RGB patch and the trimap. The encoder has 14 convolutional layers and 5 max-pooling layers, initialized from the first 14 layers of VGG-16 (with fc6 converted to a convolution and the trimap channel’s filters set to zero); the decoder has 6 convolutional layers and 5 unpooling layers followed by an alpha prediction layer. A second stage of 4 convolutional layers without downsampling takes the image and the first-stage matte and refines it, adding its output to the input matte.
• Losses. An alpha prediction loss, $\sqrt{(\alpha_p - \alpha_g)^2 + \epsilon^2}$ with $\epsilon = 10^{-6}$, a differentiable approximation of the absolute error; and a compositional loss, the same function applied to the RGB colours of the image re-composited with the predicted alpha and the ground-truth $F$ and $B$. The two are weighted equally, and only pixels in the trimap’s unknown region contribute.
• Training. Random $320 \times 320$ crops centred on unknown pixels, crops of other sizes resized to $320 \times 320$, flipping, and trimaps randomly dilated from the ground truth, regenerated every epoch; Adam with a constant learning rate of $10^{-5}$. The refinement stage was trained after the encoder–decoder had converged, then the two were fine-tuned together.

The authors also built the Composition-1k data set described below, because the existing benchmark had too few images to train a network. On its test set their full model reached a SAD of 50.4 against 168.1 for closed-form matting (Table 1), and the compositional loss and the refinement stage each lowered the error. They reported first place by SAD on alphamatting.com at the time, a user study on 31 natural images that preferred their results, and greater robustness to trimap dilation than prior methods.

Later trimap-based networks kept this setting and the Composition-1k protocol. MatteFormer’s comparison table (Park et al., 2022, Table 1) lists, among others, IndexNet, GCA Matting, HDMatt, Mask Guided Matting and SIM, with SAD falling from 50.4 for Deep Image Matting to 31.5 for Mask Guided Matting and 28.0 for SIM. (The 31.5 is the result Yu et al. (2021, Table 1) report when only the trimap’s confident foreground is given as the guidance mask, which they call TrimapFG; with the full trimap as a soft mask their SAD is 32.1.) MatteFormer itself replaces the convolutional encoder with a Swin Transformer (Tiny, ImageNet-pretrained) whose blocks add prior-tokens, the averages of the tokens in the trimap’s foreground, background and unknown regions, to every local self-attention window, so that each window sees a global summary of the three regions; the decoder is a simple convolutional one. With six input channels (RGB and the trimap) and $512 \times 512$ training crops, it reported SAD 23.8, MSE $4.0 \times 10^{-3}$, gradient error 8.7 and connectivity error 18.9 on Composition-1k, using the official evaluation code. Mask Guided Matting (Yu et al., 2021) relaxes the input instead: one network accepts a coarse mask of any kind, from a trimap to a low-quality matte, is trained with perturbed guidance masks, and refines uncertain regions progressively through the decoder; the authors also revisit foreground colour prediction.

Trimap-free and portrait matting

Without a trimap the network must decide what the foreground is, which is a segmentation problem, before estimating fractional alpha, which is a matting problem. Most methods therefore restrict the category, usually to people.

• Implicit trimaps. Zhang et al.’s (2019) late fusion network takes a single RGB image and has two decoder branches, one classifying foreground and one background, whose outputs a fusion branch combines into alpha; the network produces a trimap implicitly. They report results on a human-image test set they assembled (40 foregrounds, 29 from the internet and 11 from Composition-1k, each composited over 25 PASCAL VOC backgrounds) and on Composition-1k.
• Background photographs. Background Matting (Sengupta et al., 2020) asks the user to photograph the scene once more without the subject, which is far less work than painting a trimap. A network takes the image, the background shot and an automatically computed soft segmentation of the person, is trained first on composites of the Adobe matting data and then, to reduce the synthetic-to-real gap, on unlabelled real images with an adversarial loss on new composites. It needs a static background and small camera motion, and it is specialized to people. Its successor (Lin et al., 2021) runs at 30 frames per second at 4K and 60 at HD on a modern GPU by computing a low-resolution result and refining it on selected high-resolution patches, and introduced the VideoMatte240K and PhotoMatte13K/85 data sets.
• No auxiliary input. Robust Video Matting (Lin, Yang, Saleemi and Sengupta, 2022) needs neither a trimap nor a background shot. It is recurrent, carrying information across frames to reduce flicker and to learn the background as it is revealed, and it is trained jointly on matting and semantic segmentation, which its authors found made it more robust than training on synthetic matting data alone. They report 4K at 76 and HD at 104 frames per second on an Nvidia GTX 1080Ti. In their review, MODNet is the most recent portrait-matting method and handles video only with a post-processing step that compares neighbouring frames.

Trimap-free methods trade generality for convenience. Yu et al. (2021) observe that fully automatic methods trained end-to-end struggle to generalize to complex real images for lack of semantic guidance, and that methods built around one kind of auxiliary input lose their appeal when that input varies.

Datasets and Benchmarks

alphamatting.com. Rhemann et al. (2009) created an online benchmark that ranks submitted methods on a fixed test set, which Szeliski (2022, §10.4.3) describes as the place to find current results; his 2021 draft counted over 60 algorithms on it. As Xu et al. (2017) describe it, it has 27 training images and 8 test images, most of them objects in front of an image shown on a monitor, and each test image comes with three trimaps, “small”, “large” and “user”. Szeliski (2022, §10.4.1) lists Rhemann et al.’s papers among those whose ground-truth mattes were made by capture against several known backings. Xu et al. argued that its size and conditions (indoor lab scenes, no people or animals) bias methods toward it, and found in their user study that some methods ranked well on it but were judged worse on natural images, which they read as a sign of overfitting to the benchmark.

Composition-1k and the Adobe matting data. Xu et al. (2017) extracted mattes and foreground colours by hand in Photoshop from images of objects on simple backgrounds, including the 27 alphamatting.com training images and frames from a video matting set, and composited each object over backgrounds from MS COCO and PASCAL VOC. Their training set has 493 objects composited onto 100 backgrounds each (49,300 images); the test set, which they call Composition-1k, has 50 objects on 20 backgrounds each (1,000 images), with trimaps randomly dilated from the ground-truth mattes. Later papers describe the released data differently: Zhang et al. (2019), Yu et al. (2021) and Park et al. (2022) give 431 training foregrounds, and Park et al. state that training backgrounds come from COCO and the test backgrounds from PASCAL VOC 2012. Composition-1k is synthetic by construction, and Xu et al. raised the concern that a network could learn cues of compositing itself (mismatched lighting or noise); they report good results on natural images nonetheless. Distinctions-646, used alongside it, has 596 training and 50 test foregrounds composited the same way, but, as Park et al. note, no official trimaps, which makes comparisons across papers hard.

Other evaluations. Levin et al.’s synthetic composites (above), Xu et al.’s 31 natural images judged by people, and the human-specific sets of the trimap-free papers (Zhang et al., 2019; Lin et al., 2021) are not shared protocols, and their numbers do not compare across papers.

Evaluation Metrics

Matting is scored against a ground-truth matte, and only over the trimap’s unknown region, since the known regions are given. Xu et al. (2017) use the four error measures that they attribute to Rhemann et al. (2009): the sum of absolute differences (SAD), the mean squared error (MSE), the gradient error and the connectivity error. Rhemann et al.’s title calls their benchmark “perceptually motivated”; their own definitions could not be read here, so the descriptions below follow the evaluation functions written by Ning Xu for Composition-1k, which MatteFormer (Park et al., 2022) calls the official evaluation code and which the GCA Matting repository distributes with its own driver script, evaluate.m. The functions read 8-bit mattes and work with alpha in $[0, 1]$; lower is better.

• SAD: $\sum_{i \in U} |\alpha_i^p - \alpha_i^g|$ over the unknown region $U$, divided by 1000; a comment in the code explains the division by the large images. It grows with the size of the unknown region, so it is not comparable between trimaps of different widths.
• MSE: the mean squared error over $U$, normalized by the number of unknown pixels. Papers print it either as a fraction (Xu et al. give 0.014 for their model) or in units of $10^{-3}$ (Park et al. give the same result as 14), so a table’s header must be checked.
• Gradient error: each matte is rescaled to $[0, 1]$ by its own minimum and maximum, filtered with first-derivative-of-Gaussian kernels with $\sigma = 1.4$, and the squared differences of the two gradient magnitudes are summed over $U$; Xu’s function returns the plain sum, and GCA Matting’s driver script divides it by 1000. By construction it compares the sharpness of the two mattes rather than their values, so an oversmoothed or noisy matte scores badly even when its values are close on average.
• Connectivity error: for thresholds $t = 0.1, 0.2, \dots, 1$, the code finds the largest 4-connected region where both mattes are at least $t$. Each pixel receives the level $l_i$ just below the threshold at which it drops out of that region (1 if it never does), and its degree of connectivity is $\varphi_i = 1 - d_i [d_i \ge 0.15]$ with $d_i = \alpha_i - l_i$. The error is $\sum_{i \in U} |\varphi_i^p - \varphi_i^g|$, again divided by 1000 in the driver script. By construction it singles out alpha that is cut off from the main connected body of the matte, such as isolated blobs. (The function also computes distance maps that do not enter the result.)

The divisions by 1000 are a convention of this code (inside the SAD function, in the driver script for gradient and connectivity); alphamatting.com and other evaluations need their own definitions checked before numbers are compared. The rows below come from the two papers’ Composition-1k tables (Xu et al.’s MSE converted to units of $10^{-3}$); the closed-form, KNN and learning-based rows in MatteFormer’s table repeat Deep Image Matting’s.

Method	SAD	MSE ($10^{-3}$)	Gradient	Connectivity	Source
Learning Based Matting	113.9	48	91.6	122.2	Xu et al. (2017), Table 1
Shared Matting	128.9	91	126.5	135.3	Xu et al. (2017), Table 1
Global Matting	133.6	68	97.6	133.3	Xu et al. (2017), Table 1
Closed-Form Matting	168.1	91	126.9	167.9	Xu et al. (2017), Table 1
KNN Matting	175.4	103	124.1	176.4	Xu et al. (2017), Table 1
Deep Image Matting	50.4	14	31.0	50.8	Xu et al. (2017), Table 1
Mask Guided Matting (TrimapFG)	31.5	6.8	13.5	27.3	Park et al. (2022), Table 1
MatteFormer	23.8	4.0	8.7	18.9	Park et al. (2022), Table 1

Two things stand out. Closed-form matting, the most accurate method in Levin et al.’s synthetic test with its uniform foreground, falls behind the sampling methods here; in our reading, real foregrounds with hair and texture break the colour line model in many windows. And in SAD the step from the best classical method in the table (learning-based matting, 113.9) to Deep Image Matting (50.4) is larger than all the improvement since (to 23.8); the earlier DCNN matting network of Cho et al., which combines classical mattes, scored 161.4 in Xu et al.’s table.

Applications

• Film and television. Compositing actors over generated or filmed scenery is matting’s original use, done in studios with blue or green screens (Smith and Blinn, 1996; Szeliski, 2022, §10.4).
• Photo editing. Cutting out an object with hair or fur intact; the deep trimap-based methods target this interactive use (Xu et al., 2017; Yu et al., 2021). Matting is one of the tools of computational photography in Szeliski’s account.
• Video conferencing and video creation. Real-time background replacement for people without a green screen, the motivation of background matting and Robust Video Matting (Lin et al., 2021, 2022); Lin et al. (2021) also note its use for privacy, hiding a user’s surroundings.

Open Problems

• Real ground truth at scale. Accurate mattes of moving people, animals and outdoor scenes cannot be captured with multiple backings, and training on composites leaves a domain gap that adversarial, semi-supervised and segmentation co-training only partly close (Sengupta et al., 2020; Lin et al., 2022).
• Benchmark validity. The main test set is synthetic, alphamatting.com is small, Distinctions-646 lacks official trimaps, and metric scales differ between papers; Xu et al.’s (2017) user study already showed that benchmark rank and human judgement can disagree.
• Generality without guidance. Most trimap-free methods target one category, usually people, and some need a static background (Sengupta et al., 2020; Yu et al., 2021).
• Foreground colour. The four standard measures score only alpha, although compositing quality depends equally on the estimated foreground colours (Yu et al., 2021, revisit their prediction).
• Temporal consistency and resolution. Video mattes must be stable across frames and computed at 4K in real time, which pushes methods toward low-resolution prediction with refinement (Lin et al., 2021, 2022).