Neural Style Transfer

Neural style transfer renders the content of one image, usually a photograph, in the visual style of another, usually a painting, by working with the activations of a convolutional network rather than with pixels. Gatys, Ecker and Bethge (2016) defined the task in this form: the output should keep the objects and layout of the content image, measured by a layer’s feature maps, and the textures, colours and strokes of the style image, measured by correlations between feature maps, the Gram matrices of their earlier texture synthesis work (Gatys et al., 2015). They found the output by optimizing the pixels of one image, which took minutes per image. Later work moved that optimization into training: feed-forward networks for one style (Johnson, Alahi and Fei-Fei, 2016; the texture networks of Ulyanov, Lebedev, Vedaldi and Lempitsky, 2016), the instance normalization that made those networks work much better (Ulyanov, Vedaldi and Lempitsky, 2017), and networks that accept any style image at test time, by adaptive instance normalization (AdaIN; Huang and Belongie, 2017) or whitening and colouring transforms (WCT; Li et al., 2017). Photorealistic and video style transfer adapt the same machinery to photographs and to frame sequences. The losses themselves, content and style reconstruction losses and the choice of layers, belong to perceptual loss; the normalization layers are defined in normalization layers. This article covers the task: its formulations, the methods and their evolution, and how results are judged.

Problem Definition

Given a content image $c$ and a style image $s$, produce an image $x$ that “looks like” the style image but shows the objects and scenery of the content image, the success criterion Gatys et al. (2016, §4) adopted while noting that it is neither mathematically precise nor universally agreed. Because there is no ground-truth stylized image, every method replaces the criterion with a surrogate computed by a fixed network $\phi$, typically VGG trained on ImageNet:

$$x^\star = \arg\min_x \; \alpha\, \mathcal{L}_{\mathrm{content}}\big(\phi(x), \phi(c)\big) + \beta\, \mathcal{L}_{\mathrm{style}}\big(\phi(x), \phi(s)\big),$$

where the content term compares feature maps position by position at one deeper layer and the style term compares statistics pooled over positions at several layers (Gatys et al., 2016, Eq. 7). The two terms pull against each other, and the ratio $\alpha/\beta$ chooses the compromise (§3.1). The exact terms, their normalizations and the weights used in the papers are given in the perceptual loss article.

The definition hides an assumption, that content and style are separable in the network’s features. Gatys et al. called this separability their key finding but also wrote that separating content from style is not necessarily a well-defined problem, since style may be the brush strokes, the colour map, the dominant forms or even the composition and choice of subject (§3, §4).

Input and Output

• Input: a content image and a style image. Per-style methods fix the style at training time and take only the content image (plus noise, in texture networks); arbitrary-style methods take both at test time. Optional inputs are the content–style weight, masks that assign styles to regions, and, for photographs, semantic segmentations of both images.
• Output: one image of the content image’s size. Fully convolutional networks trained on $256 \times 256$ images are applied at other sizes at test time (Johnson et al., 2016, §3.1).
• Related output: with the content term removed, the same machinery synthesizes a texture from an example, the problem it grew out of (Gatys et al., 2015).

Variants

• Optimization-based and feed-forward. Gatys et al. optimize each output image; feed-forward methods train a network once and stylize with one forward pass.
• Number of styles per network. One style per network (Johnson et al.; Ulyanov et al.), a fixed set of styles (conditional instance normalization of Dumoulin, Shlens and Kudlur, 2017, with 32 styles in their experiments, as Huang and Belongie describe it), or arbitrary styles never seen in training (AdaIN, WCT).
• Artistic and photorealistic. Artistic transfer accepts distorted edges and painted textures; photorealistic transfer between photographs must change colours and tones without moving or warping anything (Luan et al., 2017).
• Images and videos. Video style transfer must also keep consecutive stylized frames consistent (Huang et al., 2017).
• User controls. Colour preservation, spatial control with masks and scale control were added to the optimization framework in later work by Gatys, Ecker, Bethge, Hertzmann and Shechtman (2017), as Huang and Belongie (§2) summarize it; Huang and Belongie (§7.3) and Li et al. (§4.2) show how to obtain them in feed-forward models.
• Collection style. Transferring the style of a whole set of paintings rather than of one image is learned by unpaired image-to-image translation instead, as in the Monet and Van Gogh experiments of CycleGAN.

Challenges

• Speed against flexibility against quality. The optimization handles any pair of images but is slow; per-style networks are fast but fixed; arbitrary-style networks are fast and flexible but lose some quality. Huang and Belongie (§7.1) describe this as a three-way trade-off between speed, flexibility and quality.
• No ground truth. Any loss used to train or to rank methods is the surrogate that defines the task, so evaluation falls back on human judgment (see Evaluation Metrics).
• Sensitivity to weights and scale. The result changes with the content–style ratio, with the style layers and with image resolution. Li et al. (§4.2) attribute the scale effect to the network’s fixed receptive field, which extracts different statistics from a style image at different sizes, and changed the scale of transferred patterns by resizing the style image.
• Artefacts. Low-level noise resembling the network’s filters (Gatys et al., §4), repetitive patterns in feed-forward outputs (Johnson et al., §4.1), and painting-like distortions when both images are photographs (Luan et al., §1); the artefacts of the losses are discussed with perceptual loss.
• Semantic mismatch. A Gram matrix pools over the whole image, so the style of a large region of the style image can “spill over” onto unrelated content, such as building texture in the sky (Luan et al., §1.1).
• Temporal consistency. Stylizing video frames independently amplifies small differences between frames into flicker (Huang et al., 2017, §1).

Approaches

Optimization (Gatys et al., 2016)

The original method starts from white noise and runs L-BFGS on the pixels to minimize the combined content and style loss of a normalized VGG-19 with average pooling, matching content at conv4_2 and style at conv1_1 to conv5_1; images of about $512 \times 512$ pixels took up to an hour on an Nvidia K40 GPU (§2–§4). It grew directly out of the authors’ texture synthesis (Gatys et al., 2015), which matched Gram matrices alone; the texture-network paper (Ulyanov et al., 2016, §2) placed both in the tradition of statistical texture models, Portilla and Simoncelli’s matching of wavelet statistics in particular, with deep features in place of hand-designed filters.

The optimization is a pre-image search: many images satisfy the statistics, and which one is found depends on the starting point. Gatys et al. (§3.3) found that starting from the content or the style image instead of noise biased the result towards that image’s spatial structure but did not change it strongly, and that only noise starts produce an arbitrary number of different results. The method’s limits as its authors saw them were resolution, since both the number of pixels optimized and the network’s activations grow with image size, and low-level noise in the output, most visible when both images are photographs (§4).

Feed-forward networks for one style

In 2016 two groups replaced the per-image optimization with a network trained to minimize the same kind of loss over many content images, so that stylization becomes a single forward pass; the losses and their speed comparison are in the perceptual loss article, and the network designs differ.

• Johnson et al. (2016) used an image transformation network with two stride-2 convolutions, five residual blocks and two fractionally strided convolutions back to full resolution, with batch normalization and ReLU after every non-residual convolution and a scaled tanh at the output (§3.1). Downsampling first lets a network of the same cost be larger and gives each output pixel a larger effective receptive field, which coherent stylization of large regions needs. Each network was trained for one style on the 80,000 COCO training images resized to $256 \times 256$, for 40,000 iterations at batch size 4 (about two epochs) with Adam, taking about 4 hours on a GTX Titan X (§4.1), with a VGG-16 loss network.
• Ulyanov, Lebedev, Vedaldi and Lempitsky (2016) proposed texture networks: compact multi-scale generators that take noise tensors at several resolutions, process each with convolution blocks, upsample and concatenate them, about 65,000 parameters for texture synthesis (§3.2). For stylization the noise at each scale is concatenated with a downsampled copy of the content image, and six scales are used instead of five (§3.3). They trained with Adam for 2,000 iterations at batch size 16 and found that a very small pool of content images, for example 16, sufficed: training on more images lowered the loss on held-out images but degraded the visual quality (§3.3, §4). Generating a $256 \times 256$ texture took about 20 ms, against about 10 s for the optimization to reach the same texture loss, and 170 MB of memory against 1,100 MB (§4.2). Because the content–style weight is fixed at training time, they adjusted the trade-off at test time by scaling the input noise (Fig. 5).

Both papers found their outputs qualitatively close to the optimization’s but not always as good: Ulyanov et al. could not match it for some styles, and Johnson et al. saw more repetitive patterns.

Instance normalization

The quality gap narrowed with a one-layer change. In their CVPR 2017 paper, Ulyanov, Vedaldi and Lempitsky argued that a stylized image should not depend on the contrast of the content image, that discarding contrast with ReLUs and convolutions is unnecessarily hard, and that normalizing each image’s channels by their own mean and standard deviation, which they named instance normalization, does it directly; unlike batch normalization it is applied unchanged at test time (§4). The same authors’ 2016 arXiv report, “Instance Normalization: The Missing Ingredient for Fast Stylization”, carries the layer’s name in its title (not read here); the conference paper read here contains the argument and the experiments. Swapping batch normalization for instance normalization in Johnson et al.’s residual architecture, a network they call StyleNet, made training converge much faster to lower values of the same objective (Fig. 3a), and when the stylized outputs were used to initialize the Gatys et al. optimization, the instance-normalized network’s outputs changed little while the batch-normalized ones changed much more (Fig. 3b, e, h). For texture synthesis, by contrast, the two normalizations performed equally well, which the authors explain by the fact that only the content image, not the single learned texture, varies from input to input (§5.2). The same paper adds a diversity term, an entropy estimate from nearest-neighbour distances within a batch, so that generators sample the Julesz ensemble of images with the target statistics instead of collapsing to one output (§3); without it, a high-capacity generator learned to produce nearly the same image for every noise input (Fig. 5).

Huang and Belongie (2017, §4) offered a different explanation. Training the same networks on images first equalized in contrast (histogram equalization of the luminance) left instance normalization’s advantage significant, so contrast was not the whole story; training on images first converted to one common style with a pretrained style transfer network made it much smaller (Fig. 1). They concluded that instance normalization performs a kind of style normalization: the channel-wise mean and variance of a network’s features carry much of an image’s style, and normalizing them removes the content image’s own style, leaving the rest of the network to deal with content. How the operation differs from batch, layer and group normalization is described in the normalization layers article.

From one style to many

If the per-channel statistics carry style, a network can switch styles by changing only them. Dumoulin et al.’s conditional instance normalization learns a separate scale and shift $(\gamma_s, \beta_s)$ for each style $s$ and shares all convolution weights; the extra parameters grow linearly with the number of styles, $2FS$ for $F$ feature maps, and a new style still needs training (described by Huang and Belongie, §3.3; the same per-channel modulation from a condition reappears in conditional GANs).

Arbitrary style transfer

AdaIN. Huang and Belongie computed the scale and shift from the style image instead of learning them. Adaptive instance normalization takes content features $x$ and style features $y$ and sets each channel’s mean and standard deviation, computed over spatial positions, to those of the style (Eq. 8):

$$\mathrm{AdaIN}(x, y) = \sigma(y)\, \frac{x - \mu(x)}{\sigma(x)} + \mu(y) .$$

Their network encodes both images with VGG-19 up to relu4_1, applies AdaIN to the features, and inverts the result with a decoder trained from random initialization; the decoder mirrors the encoder, uses nearest-neighbour upsampling and reflection padding, and has no normalization layers, since instance or batch normalization would push all outputs towards one style (§6.1). The decoder is trained on COCO content images and about 80,000 paintings, mostly from WikiArt, with the AdaIN output itself as the content target and a style loss that matches channel means and standard deviations at relu1_1 to relu4_1 rather than Gram matrices; the authors found that a Gram loss gave similar results but chose the moment loss as conceptually cleaner (§6.2). On 10 unseen styles and 50 content images, the network’s average content and style losses were slightly higher than those of the per-style networks of Ulyanov, Vedaldi and Lempitsky (2017), and both reached a style loss similar to 50 to 100 iterations of the optimization (Fig. 3; for this comparison the other methods’ style losses were also changed to the moment loss). At $512 \times 512$ on a Pascal Titan X it took 0.065 s per image excluding style encoding and 0.098 s including it, against 0.038 s for a per-style network and 46.75 s for 500 iterations of the optimization (Table 1). Replacing AdaIN with concatenation of content and style features left visible contours of the style image in the output, and normalization layers in the decoder gave worse results and higher losses (§7.2). Content–style trade-off, style interpolation, colour preservation and spatial control are all done at test time with the same network (§7.3): the first two by mixing feature maps before the decoder (Eqs. 14–15), colour preservation by first matching the style image’s colours to the content image’s, and spatial control by applying AdaIN region by region with different styles. StyleGAN later borrowed AdaIN to inject learned styles into a generator.

WCT. Li et al. (2017) matched the full covariance instead of per-channel moments, without training on any style. They train decoders only to reconstruct images from VGG-19 features at relu1_1 to relu5_1. At test time, the content features $f_c$ (a $C \times HW$ matrix) are centred and whitened with the eigendecomposition of their covariance, $\hat f_c = E_c D_c^{-1/2} E_c^\top f_c$, which removes their correlations, then coloured with the style’s, $\hat f_{cs} = E_s D_s^{1/2} E_s^\top \hat f_c$, and the style mean is added back (§3.2, Eqs. 2–3). Inverting whitened features kept the global content structure but removed stroke patterns (Fig. 2), and channel-wise histogram matching, which ignores correlations between channels, transferred global colour but broke patterns apart (Fig. 3). Higher levels captured more complicated local structures and lower levels colours, so the authors applied the transform at all five levels, coarse to fine, with each level’s output fed as content to the next (§3.3). They regard AdaIN’s step as a sub-optimal approximation of WCT (§2); it matches each channel’s mean and variance but not the correlations between channels. In their comparison at $256 \times 256$ (Table 2), WCT had the lowest covariance difference to the style image, $\log L_s$ of 6.3 against 7.0 for AdaIN and 6.7 for the optimization, received 30.3% of the votes in a user study against 24.9% for AdaIN and 16.4% for the optimization, and took 0.83 s against 0.20 s for AdaIN, mostly for the eigendecomposition. The covariance difference is the statistic WCT’s transform matches in feature space, so in our reading the first row measures how directly each method targets it as much as quality.

Huang and Belongie also compared themselves with a patch-based alternative, Chen and Schmidt’s style swap, which replaces each content feature patch with its closest style patch; they report it was 1–2 orders of magnitude slower than AdaIN and failed when most content patches matched a few unrepresentative style patches (§2, §7.1).

Photorealistic style transfer

Gatys et al. (2016, §3.4) transferred a night photograph’s style onto a daytime photograph of London, and noted that photorealism was not fully preserved. Luan et al. (2017) set out to transfer the look of a reference photograph, such as time of day, weather, season or a retouching style, without such painting-like distortions. Their insight was to constrain the transformation instead of the output: every output patch should be an affine function of the input patch’s colours, which can change colours drastically but cannot move an edge, since an edge lies at the same place in all three channels (§2). They expressed the constraint as a penalty $\sum_c V_c[O]^\top M_I V_c[O]$ built on the Matting Laplacian $M_I$ of Levin et al.’s closed-form image matting, computed from the input image alone. Against spill-over, they masked the Gram matrices with semantic segmentation of both images, so that sky is matched to sky and buildings to buildings. The optimization ran in two stages, Neural Style with the masked style loss first, then the full objective with weights $\Gamma = 10^2$ and $\lambda = 10^4$, in 3–5 minutes on a Titan X (§3, §4). In a user study of photorealism their results scored below a global histogram transfer, while Neural Style and CNNMRF were rated non-photorealistic, and in a study of style faithfulness against three global colour-transfer methods they were chosen most often, more than 80% of the time (§4, Fig. 9).

Video style transfer

Running an image stylizer on each frame causes flicker. Ruder, Dosovitskiy and Brox (2016) added a temporal loss to the optimization, which, as Huang et al. (2017) describe it, penalizes the difference between the current stylized frame and the previous one warped by optical flow, and took about three minutes per frame even with precomputed flow. Huang et al. (2017), a different group from Huang and Belongie, trained a feed-forward network instead, with instance normalization and fewer channels than Johnson et al.’s, on a hybrid loss: content, style and total variation terms for each of two consecutive frames, plus the same short-term temporal loss on the network’s outputs, weighted by a per-pixel confidence that is zero at occlusions and motion boundaries (§3, Eq. 7). Optical flow is needed only for the training videos, not at test time (§3.2.2). Computing the temporal loss on the loss network’s high-level features instead of the output pixels gave strong flicker. On five MPI Sintel sequences with ground-truth flow and the style Candy, their temporal error (Eq. 8, the root of the confidence-weighted mean squared colour difference between consecutive stylized frames, the later frame warped back with the ground-truth flow) lay between Ruder et al.’s optimization, the most consistent, and Johnson et al.’s per-frame network, the least: 0.0439 against 0.0252 and 0.0770 on Alley 2 (Table 2). Adding non-consecutive frame pairs to training lowered the errors only slightly (Table 3).

Datasets and Benchmarks

Neural style transfer has no standard benchmark. Papers train on large photograph collections as content, COCO for Johnson et al., Huang and Belongie and Li et al., and ImageNet images for texture networks, and arbitrary-style methods add a collection of paintings as styles, in Huang and Belongie’s case about 80,000 mostly from WikiArt, taken from the Kaggle competition Painter by Numbers. Evaluation sets are small and chosen by each paper: 50 COCO validation images with Picasso’s The Muse for Johnson et al.’s loss comparison, 10 styles and 50 content images for AdaIN, 10 content images and 40 styles for WCT, and the MPI Sintel sequences, whose ground-truth optical flow makes temporal error measurable, for video. A set of famous paintings recurs across papers, The Starry Night and Composition VII among them, so results can be compared by eye but not by number.

Evaluation Metrics

• The objective itself. Johnson et al. and Huang and Belongie compared methods by the value of the content and style losses they reach, and Li et al. by the covariance difference to the style image. This measures how well a method minimizes the surrogate, not whether its output looks better, and it favours methods that optimize the same statistic: Huang and Belongie switched the other methods to their moment-based style loss before comparing, and Li et al.’s covariance metric is the quantity their transform matches in feature space.
• User studies. Preference votes are the main quality measure: Li et al. collected 1,200 votes from 80 participants choosing one favourite among five methods’ outputs, Luan et al. collected ratings of photorealism on a four-point scale and choices of the most faithful style, and Huang et al. compared their videos with two smartphone apps by pairwise preference. Li et al. called the evaluation of artistic style transfer an open question.
• Temporal error for video: the pixel-wise colour difference between consecutive stylized frames after warping one onto the other with ground-truth optical flow, weighted by a flow confidence that is zero at occlusions and motion boundaries (Huang et al., 2017, Eq. 8).
• Speed and memory, reported per image at fixed resolutions on a named GPU, with and without the style encoding for arbitrary-style methods (Huang and Belongie, Table 1).

Generic image metrics transfer poorly: FID and LPIPS compare against a reference image or distribution that a stylized image is not meant to match.

Example: Gram Matrices against Channel Moments

Gatys et al. match Gram matrices; Huang and Belongie match only each channel’s mean and standard deviation and report similar results, while Li et al. call that a sub-optimal approximation of matching the covariance. The example compares the two statistics on a toy scale, in three parts: texture synthesis with each statistic as the only loss, what AdaIN and WCT do to the statistics in feature space, and the content–style weight in Gatys et al.’s optimization. The perceptual loss article’s example reconstructs shapes from one layer’s features or Gram matrices to show what each layer keeps; this one compares style statistics on textures.

No pretrained weights are downloaded, so a small network trained from scratch stands in for VGG: three blocks of two $3 \times 3$ convolutions with ReLU (16, 32 and 64 channels), average pooling between blocks, global average pooling and a linear classifier. It is trained to tell apart four textures, scikit-image’s bundled brick, grass and gravel images and the brick image turned by 90°, from $48 \times 48$ patches of the top half of each image; the bottom halves are held out. Each texture is first resized to $256 \times 256$ and standardized to zero mean and unit variance, so the classes do not differ in grey level or contrast. A second network trained the same way with another seed is the judge, so the loss network does not grade itself. As in Gatys et al., each channel is divided by its mean activation, style statistics are taken at the first convolution of each block (relu1_1, relu2_1, relu3_1, after Gatys et al.’s conv1_1 to conv5_1) and content at relu2_2 (the second convolution of the penultimate block, after their conv4_2). Each loss is a squared error divided by the squared norm of its target, summed over layers, a normalization of our choosing; the mean and standard deviation loss is the moment-matching style loss of Huang and Belongie’s Eq. 13, which sums unsquared norms, in this squared and normalized form. All optimizations run L-BFGS on the pixels for 300 iterations from Gaussian noise, with no image prior, as in Gatys et al.

Besides the losses, two measures do not depend on the loss network: the judge’s accuracy, and a spectrum distance, the root-mean-square difference between the log power spectrum of a $96 \times 96$ image (Hann window, Gaussian smoothing of 2 frequency bins) and the average spectrum of the texture’s held-out half; it keeps orientation and scale and ignores where structures are. In part B, AdaIN and WCT are applied to the features of a coffee-cup photograph (coffee, cropped to the cup and standardized) for each texture; no decoder is trained, so part B measures statistics, not images. WCT drops eigenvalues below $10^{-5}$ of the largest when whitening and colouring, our choice. Part C stylizes the cup with the brick texture for five style weights $w$ in $\mathcal{L} = \mathcal{L}_{\mathrm{content}} + w\,\mathcal{L}_{\mathrm{style}}$, and measures layout as the correlation between the blurred output and the blurred content image.

The first script prepares the images with scikit-image 0.25.2 (NumPy 2.3.5) and saves them; the second runs with PyTorch 2.14.1, NumPy 2.5.4, SciPy 1.18.1 and scikit-learn 1.9.1 on four CPU threads.

# prepare.py: scikit-image 0.25.2, writes textures.npz
import numpy as np
from skimage import data, transform, color

def standardize(a):
    return ((a - a.mean()) / a.std()).astype(np.float32)

tex = {n: standardize(transform.resize(getattr(data, n)().astype(float), (256, 256), anti_aliasing=True))
       for n in ["brick", "grass", "gravel"]}
names = ["brick", "brick_rot90", "grass", "gravel"]
train = [tex["brick"][:128], np.rot90(tex["brick"][:128]), tex["grass"][:128], tex["gravel"][:128]]
test = [tex["brick"][128:], np.rot90(tex["brick"][128:]), tex["grass"][128:], tex["gravel"][128:]]
coffee = transform.resize(color.rgb2gray(data.coffee()), (100, 150), anti_aliasing=True)
content = standardize(coffee[0:96, 40:136])

np.savez("textures.npz", content=content, names=names,
         **{f"train{k}": np.ascontiguousarray(a) for k, a in enumerate(train)},
         **{f"test{k}": np.ascontiguousarray(a) for k, a in enumerate(test)})
for k, n in enumerate(names):
    print(f"{n:12s} train {train[k].shape} test {test[k].shape}")
print("content", content.shape, f"mean {content.mean():.3f} std {content.std():.3f}")

Output:

brick        train (128, 256) test (128, 256)
brick_rot90  train (256, 128) test (256, 128)
grass        train (128, 256) test (128, 256)
gravel       train (128, 256) test (128, 256)
content (96, 96) mean 0.000 std 1.000

# style_statistics.py: PyTorch 2.14.1, reads textures.npz
import numpy as np, torch, torch.nn as nn, torch.nn.functional as F
from scipy.ndimage import gaussian_filter
from sklearn.linear_model import LogisticRegression

torch.set_num_threads(4)
d = np.load("textures.npz")
names = list(d["names"]); K = len(names)
train = [d[f"train{k}"] for k in range(K)]; test = [d[f"test{k}"] for k in range(K)]
content = torch.tensor(d["content"])[None, None]

def patches(regions, n, rng, size):
    X, y = [], []
    for k, r in enumerate(regions):
        for _ in range(n):
            i, j = rng.integers(0, r.shape[0] - size + 1), rng.integers(0, r.shape[1] - size + 1)
            X.append(r[i:i + size, j:j + size]); y.append(k)
    return torch.tensor(np.array(X))[:, None], torch.tensor(y)

class Net(nn.Module):
    """VGG-like stand-in: three blocks of two 3x3 convolutions + ReLU, average pooling between blocks."""
    def __init__(s):
        super().__init__()
        c = lambda a, b: nn.Conv2d(a, b, 3, padding=1)
        s.convs = nn.ModuleList([c(1, 16), c(16, 16), c(16, 32), c(32, 32), c(32, 64), c(64, 64)])
        s.fc = nn.Linear(64, K)
        for m in s.modules():
            if isinstance(m, nn.Conv2d):
                nn.init.kaiming_normal_(m.weight, mode="fan_out", nonlinearity="relu"); nn.init.zeros_(m.bias)
    def feats(s, x):                      # relu1_1, relu1_2, relu2_1, relu2_2, relu3_1, relu3_2
        out = []
        for i, conv in enumerate(s.convs):
            if i in (2, 4): x = F.avg_pool2d(x, 2)
            x = F.relu(conv(x)); out.append(x)
        return out
    def forward(s, x): return s.fc(s.feats(x)[-1].mean((2, 3)))

def train_net(seed, X, y, epochs=8):
    torch.manual_seed(seed); net = Net(); opt = torch.optim.Adam(net.parameters(), 1e-3)
    g = torch.Generator().manual_seed(seed)
    for _ in range(epochs):
        for idx in torch.randperm(len(X), generator=g).split(64):
            loss = F.cross_entropy(net(X[idx]), y[idx]); opt.zero_grad(); loss.backward(); opt.step()
    return net.eval().requires_grad_(False)

rng = np.random.default_rng(0)
Xtr, ytr = patches(train, 1000, rng, 48); Xte, yte = patches(test, 200, rng, 48)
net, judge = train_net(0, Xtr, ytr), train_net(1, Xtr, ytr)    # loss network and an independent judge
acc = lambda m, X, y: (m(X).argmax(1) == y).float().mean().item()
ms = lambda X: np.stack([X.mean((1, 2, 3)).numpy(), X.std((1, 2, 3)).numpy()], 1)
lr = LogisticRegression(max_iter=1000).fit(ms(Xtr), ytr.numpy())
print(f"held-out 48x48 patches: loss net {acc(net, Xte, yte):.3f}, judge {acc(judge, Xte, yte):.3f}, "
      f"patch mean+std alone {lr.score(ms(Xte), yte.numpy()):.3f}")

means = [f.mean((0, 2, 3)).clamp_min(1e-8)[None, :, None, None] for f in net.feats(Xtr[:500])]
phi = lambda x: [f / m for f, m in zip(net.feats(x), means)]   # channels scaled to mean activation 1
STYLE, CONTENT = [0, 2, 4], 3                                  # relu1_1, relu2_1, relu3_1; relu2_2

def gram(f):                                                   # Johnson et al., Eq. 3
    b, c, h, w = f.shape; f = f.reshape(b, c, h * w); return f @ f.transpose(1, 2) / (c * h * w)
def musd(f):                                                   # channel means and standard deviations
    return torch.cat([f.mean((2, 3)), (f.var((2, 3), unbiased=False) + 1e-8).sqrt()], 1)
STAT = {"gram": gram, "meanstd": musd}
def style_rel(stat, fx, fs):              # per image, each layer divided by the target's squared norm
    tot = 0
    for l in STYLE:
        a, b = STAT[stat](fx[l]), STAT[stat](fs[l])
        dims = tuple(range(1, a.dim()))
        tot = tot + ((a - b) ** 2).sum(dims) / (b ** 2).sum(dims)
    return tot
def content_rel(fx, fc):
    return ((fx[CONTENT] - fc[CONTENT]) ** 2).sum((1, 2, 3)) / (fc[CONTENT] ** 2).sum((1, 2, 3))

def optimize(x0, loss_fn, iters=300):     # L-BFGS on the pixels, as Gatys et al.
    x = x0.clone().requires_grad_(True)
    opt = torch.optim.LBFGS([x], max_iter=iters, history_size=50, line_search_fn="strong_wolfe")
    def closure():
        opt.zero_grad(); l = loss_fn(x).sum(); l.backward(); return l
    opt.step(closure)
    return x.detach()

def spectrum(img):                        # smoothed log10 periodogram of a Hann-windowed image
    n = img.shape[-1]; w = np.outer(np.hanning(n), np.hanning(n))
    p = np.abs(np.fft.fft2((img - img.mean()) * w)) ** 2
    return np.log10(gaussian_filter(p, 2, mode="wrap") + 1e-12)
def ref_spectrum(region, n=96, stride=16):  # Welch average over the held-out region of a texture
    ps = [np.abs(np.fft.fft2((region[i:i+n, j:j+n] - region[i:i+n, j:j+n].mean()) * np.outer(np.hanning(n), np.hanning(n)))) ** 2
          for i in range(0, region.shape[0] - n + 1, stride) for j in range(0, region.shape[1] - n + 1, stride)]
    return np.log10(gaussian_filter(np.mean(ps, 0), 2, mode="wrap") + 1e-12)
REF = [ref_spectrum(r) for r in test]
spec_dist = lambda img, k: np.sqrt(np.mean((spectrum(img) - REF[k]) ** 2))

target = [torch.tensor(np.ascontiguousarray(test[k][16:112, 16:112]))[None, None] for k in range(K)]
real = [patches([test[k]], 8, np.random.default_rng(10 + k), 96)[0] for k in range(K)]
print("\nA. Texture synthesis from 8 noise starts per texture (spectrum distance: lower = closer)")
print(f"{'texture':12s} {'statistic':10s} {'own loss':>9s} {'other loss':>10s} {'judge':>6s} {'spectrum':>8s}")
noise = torch.randn((8, 1, 96, 96), generator=torch.Generator().manual_seed(0))
for k, n in enumerate(names):
    print(f"{n:12s} {'noise':10s} {'':>9s} {'':>10s} {(judge(noise).argmax(1) == k).float().mean():6.2f} "
          f"{np.mean([spec_dist(v[0].numpy(), k) for v in noise]):8.3f}")
    print(f"{n:12s} {'real crops':10s} {'':>9s} {'':>10s} {(judge(real[k]).argmax(1) == k).float().mean():6.2f} "
          f"{np.mean([spec_dist(r[0].numpy(), k) for r in real[k]]):8.3f}")
    print(f"{n:12s} {'other tex.':10s} {'':>9s} {'':>10s} {'':>6s} "
          f"{np.mean([spec_dist(r[0].numpy(), k) for j in range(K) if j != k for r in real[j]]):8.3f}")
    fs = phi(target[k])
    for stat in ["gram", "meanstd"]:
        other = "meanstd" if stat == "gram" else "gram"
        start = style_rel(stat, phi(noise), fs).mean().item()
        x = optimize(noise, lambda v: style_rel(stat, phi(v), fs)); fx = phi(x)
        print(f"{n:12s} {stat:10s} {style_rel(stat, fx, fs).mean() / start:9.1e} {style_rel(other, fx, fs).mean():10.3f} "
              f"{(judge(x).argmax(1) == k).float().mean():6.2f} {np.mean([spec_dist(v[0].numpy(), k) for v in x]):8.3f}")

def adain(fc, fs):                        # Huang & Belongie, Eq. 8
    mc, sc = fc.mean((2, 3), keepdim=True), fc.std((2, 3), unbiased=False, keepdim=True) + 1e-5
    msd, ssd = fs.mean((2, 3), keepdim=True), fs.std((2, 3), unbiased=False, keepdim=True)
    return ssd * (fc - mc) / sc + msd
def wct(fc, fs, rel_eps=1e-5):            # Li et al., Eqs. 2-3; tiny eigenvalues dropped
    c = fc.shape[1]; a, b = fc[0].reshape(c, -1).double(), fs[0].reshape(c, -1).double()
    ma, mb = a.mean(1, keepdim=True), b.mean(1, keepdim=True); a, b = a - ma, b - mb
    ea, Ea = torch.linalg.eigh(a @ a.T / a.shape[1]); eb, Eb = torch.linalg.eigh(b @ b.T / b.shape[1])
    ka, kb = ea > rel_eps * ea.max(), eb > rel_eps * eb.max()
    white = Ea[:, ka] @ torch.diag(ea[ka] ** -0.5) @ Ea[:, ka].T @ a
    out = Eb[:, kb] @ torch.diag(eb[kb] ** 0.5) @ Eb[:, kb].T @ white + mb
    return out.float().reshape(fc.shape)
def cov(f):
    c = f.shape[1]; a = f[0].reshape(c, -1); a = a - a.mean(1, keepdim=True); return a @ a.T / a.shape[1]
rel = lambda a, b: ((a - b).norm() / b.norm()).item()
print("\nB. Content features moved towards each texture's statistics (relative errors, mean over 4 textures)")
print(f"{'layer':8s} {'features':9s} {'mean/std':>9s} {'Gram':>9s} {'covariance':>10s}")
fc = phi(content)
for l, lname in zip(STYLE, ["relu1_1", "relu2_1", "relu3_1"]):
    rows = {"content": [], "AdaIN": [], "WCT": []}
    for k in range(K):
        fs = phi(target[k])[l]
        for key, f in [("content", fc[l]), ("AdaIN", adain(fc[l], fs)), ("WCT", wct(fc[l], fs))]:
            rows[key].append((rel(musd(f), musd(fs)), rel(gram(f), gram(fs)), rel(cov(f), cov(fs))))
    for key, v in rows.items():
        v = np.array(v).mean(0); print(f"{lname:8s} {key:9s} {v[0]:9.1e} {v[1]:9.1e} {v[2]:10.1e}")

print("\nC. Content-style trade-off, coffee cup in the style of brick")
fcon = phi(content); fs = phi(target[0]); x0 = noise[:1]
def layout(x):                            # correlation of blurred output with blurred content image
    a, b = gaussian_filter(x[0, 0].numpy(), 2), gaussian_filter(content[0, 0].numpy(), 2)
    return np.corrcoef(a.ravel(), b.ravel())[0, 1]
print(f"{'statistic':10s} {'w':>6s} {'content':>8s} {'style':>8s} {'layout':>7s} {'spectrum':>8s} {'P(brick)':>8s}")
print(f"{'(content image)':17s} {'':>8s} {'':>8s} {layout(content):7.3f} {spec_dist(content[0, 0].numpy(), 0):8.3f} "
      f"{F.softmax(judge(content), 1)[0, 0]:8.2f}")
for stat in ["gram", "meanstd"]:
    start = style_rel(stat, phi(x0), fs).item()
    for w in [0.01, 0.1, 1, 10, 100]:
        x = optimize(x0, lambda v: content_rel(phi(v), fcon) + w * style_rel(stat, phi(v), fs)); fx = phi(x)
        print(f"{stat:10s} {w:6g} {content_rel(fx, fcon).item():8.4f} {style_rel(stat, fx, fs).item() / start:8.1e} "
              f"{layout(x):7.3f} {spec_dist(x[0, 0].numpy(), 0):8.3f} {F.softmax(judge(x), 1)[0, 0]:8.2f}")

Output (five to six minutes on four CPU threads, most of it training the two networks):

held-out 48x48 patches: loss net 1.000, judge 0.998, patch mean+std alone 0.120

A. Texture synthesis from 8 noise starts per texture (spectrum distance: lower = closer)
texture      statistic   own loss other loss  judge spectrum
brick        noise                             0.00    2.198
brick        real crops                        1.00    0.190
brick        other tex.                                1.142
brick        gram         2.2e-06      0.007   1.00    0.391
brick        meanstd      9.6e-05      0.007   1.00    0.447
brick_rot90  noise                             0.00    2.197
brick_rot90  real crops                        1.00    0.179
brick_rot90  other tex.                                1.154
brick_rot90  gram         1.0e-05      0.019   1.00    0.363
brick_rot90  meanstd      3.3e-03      0.046   1.00    0.638
grass        noise                             1.00    0.716
grass        real crops                        1.00    0.138
grass        other tex.                                1.204
grass        gram         2.5e-07      0.002   1.00    0.153
grass        meanstd      9.7e-06      0.003   1.00    0.202
gravel       noise                             0.00    0.993
gravel       real crops                        1.00    0.120
gravel       other tex.                                1.010
gravel       gram         1.3e-07      0.001   1.00    0.168
gravel       meanstd      2.5e-03      0.048   1.00    0.288

B. Content features moved towards each texture's statistics (relative errors, mean over 4 textures)
layer    features   mean/std      Gram covariance
relu1_1  content     4.2e-01   6.5e-01    7.4e-01
relu1_1  AdaIN       6.1e-06   3.8e-01    5.5e-01
relu1_1  WCT         3.2e-08   8.3e-07    9.0e-06
relu2_1  content     4.0e-01   6.1e-01    8.8e-01
relu2_1  AdaIN       6.0e-06   2.2e-01    4.8e-01
relu2_1  WCT         4.8e-08   9.6e-07    6.6e-06
relu3_1  content     6.3e-01   6.7e-01    2.2e+00
relu3_1  AdaIN       4.0e-06   1.1e-01    3.2e-01
relu3_1  WCT         1.7e-04   3.4e-05    1.9e-04

C. Content-style trade-off, coffee cup in the style of brick
statistic       w  content    style  layout spectrum P(brick)
(content image)                       1.000    0.882     0.02
gram         0.01   0.0003  7.9e-03   1.000    1.109     0.02
gram          0.1   0.0137  5.5e-03   0.993    1.181     0.04
gram            1   0.1838  1.1e-03   0.896    0.947     0.64
gram           10   0.4892  8.0e-05   0.725    0.742     1.00
gram          100   0.7554  8.5e-06   0.509    0.704     1.00
meanstd      0.01   0.0001  4.1e-02   1.000    1.123     0.02
meanstd       0.1   0.0035  3.3e-02   0.998    1.219     0.03
meanstd         1   0.0651  1.5e-02   0.966    1.049     0.19
meanstd        10   0.3903  1.6e-03   0.763    0.741     1.00
meanstd       100   0.7025  1.1e-04   0.542    0.694     1.00

Both networks recognize held-out patches almost perfectly, while a classifier given only each patch’s mean and standard deviation does worse than chance (25% for four classes): after standardization the textures differ in structure, not in grey level, so the task is learnable but not trivial.

A. Synthesis. Each statistic, used alone, drives its own loss to a tiny fraction of the noise start’s, and the judge assigns every synthesis of both statistics to the right texture. That measure is at its ceiling here and cannot separate the two (it also calls all eight noise images grass, so its grass verdicts mean little). The spectrum distance does separate them: Gram syntheses are closer to the held-out texture’s spectrum than moment syntheses for all four textures, by the widest margins for the rotated brick and gravel, though still farther than real crops of the same texture and far closer than crops of other textures. For those two textures the moment-matching runs also stopped at the highest own losses, so part of the gap may be the optimizer’s shortfall rather than the statistic; the example cannot separate the two. The “other loss” column cuts the other way from what one might expect: images synthesized from channel moments alone leave Gram losses below 0.05, a squared error of a few per cent of the target Gram matrices’ squared norms summed over the three layers, and Gram syntheses leave moment losses no larger. In our reading, with 16 to 64 channels and three layers, the moments of every channel at every layer constrain the co-activations nearly as tightly as the Gram matrices do, which is in the direction of Huang and Belongie’s report that a Gram loss gave similar results; whether this holds for VGG’s hundreds of channels and for paintings the example cannot say.

B. AdaIN and WCT in feature space. AdaIN gives the content features the style’s channel means and standard deviations up to the small constant the code adds to $\sigma(x)$, yet leaves relative errors of 11–38% in the Gram matrix and 32–55% in the covariance: per-channel moments say nothing about which channels fire together, the part of the style that AdaIN does not transfer in feature space, in the step Li et al. call a sub-optimal approximation. WCT matches the means, the standard deviations, the covariance and therefore the Gram matrix at every layer; the small residual at relu3_1 comes, in our reading, from the eigenvalues it drops. Whether the decoder can turn the matched statistics into an image is the separate problem both methods solve by training a decoder, which this example does not do.

C. The trade-off. Raising the style weight moves the optimization from the content image towards the texture along the path Gatys et al. describe: the content loss grows, the layout correlation falls from 1.000 to about 0.5, the spectrum distance to the brick texture falls to about 0.70, and the judge’s probability of brick rises from near zero at the two smallest weights to 1 at $w = 10$. At small weights the output keeps the cup’s layout exactly but has a larger spectrum distance to brick than the content image itself; in our reading, from a noise start the deep content layer leaves high-frequency noise in directions it does not see, the artefact the perceptual loss article describes. The same $w$ does not mean the same thing for the two statistics: at $w = 1$ the moment loss leaves more layout and a lower brick probability than the Gram loss; in our reading, because the two losses, each normalized by its own target, differ in size and gradient. A content–style weight is therefore tied to its loss’s definition and normalization, as it was between the papers; only at the largest weights, where both statistics dominate, do their measures nearly coincide. One network, one seed and one content image make all of these figures indicative only.

Applications

• Artistic rendering of photographs and videos, the original use, in the tradition computer graphics calls non-photorealistic rendering (Gatys et al., 2016, §4); Huang et al. (2017) compared their video method with two smartphone apps that offered it.
• Photograph editing. Photorealistic transfer changes time of day, weather, season or retouching style from one reference photograph (Luan et al., 2017).
• Texture synthesis. The same networks generate textures from one example: texture networks (Ulyanov et al., 2016) and WCT applied to a noise image (Li et al., 2017, §4.3).
• Normalization of image appearance. Style transfer is one of the families of stain normalization methods for histopathology slides, which make slides from different laboratories look alike.
• Image generation. AdaIN, introduced for style transfer, became the way StyleGAN injects styles into each layer of its generator, and per-channel modulation by a condition is common in conditional GANs and image-to-image translation.

Open Problems

• What style is. The task rests on the claim that a network’s feature statistics separate style from content, which its authors called not well defined (Gatys et al., 2016, §4). Which statistic to match is still argued: Huang and Belongie found moments enough and a Gram loss similar, Li et al. found full covariances better than moments, and Luan et al. found a global Gram matrix too global for photographs.
• Evaluation. Without ground truth, papers rank methods by their own surrogate losses or by small user studies with different protocols; no shared benchmark or calibrated metric exists for stylization quality.
• Quality, speed and flexibility at once. In Huang and Belongie’s comparison the arbitrary-style network was slightly behind per-style networks and optimization on some styles, a trade-off they expected (§7.1).
• Video and resolution. Feed-forward video methods remain less temporally consistent than optimization with flow (Huang et al., 2017, Table 2), and the scale of transferred patterns is tied to the loss network’s receptive fields and the image resolution.