Autoencoder

An autoencoder is a neural network trained to output its own input. An encoder $f$ maps an image $x$ to a code $h = f(x)$, a decoder $g$ maps the code back to a reconstruction $\hat{x} = g(h)$, and training minimizes the difference between $\hat{x}$ and $x$. Copying is not the goal: the code is. Because the network must pass the image through a constrained code, either a narrow bottleneck or a penalized one, it has to learn which structure in the data is worth keeping. The autoencoder is the special case of the encoder–decoder in which the output space is the input space and no labels are needed. It is the ancestor of the variational autoencoder and the masked autoencoder, and the first stage of latent diffusion models.

Core Idea

An unconstrained network with enough capacity can learn the identity map and reconstruct every image perfectly while learning nothing. Every autoencoder is defined by how it rules out that trivial solution:

• Undercomplete autoencoders give the code fewer dimensions than the input, so the identity is impossible and the network must compress.
• Overcomplete autoencoders give the code as many dimensions as the input or more. Even a linear encoder and decoder can then copy the input, so they need another constraint: a penalty on the code (sparse, contractive) or a corrupted input (denoising) (Goodfellow, Bengio & Courville, 2016, §14.1–14.2).

Mathematical Formulation

For training images $x_1, \dots, x_N$ the basic objective is

$$\min_{\theta, \phi} \; \frac{1}{N} \sum_{i=1}^{N} L\big(x_i,\, g_\phi(f_\theta(x_i))\big) + \lambda\, \Omega,$$

where $L$ is usually the squared error, or the binary cross-entropy for intensities in $[0, 1]$ (Vincent et al., 2008), and $\Omega$ is an optional regularizer. Encoder and decoder may share weights, $W' = W^\top$ (“tied weights”).

The linear case is PCA. Take a linear encoder $h = Bx$ ($B$ is $p \times n$, $p < n$), a linear decoder $\hat{x} = Ah$, centred data and squared error. Baldi and Hornik (1989) described the whole error surface: assuming the covariance $\Sigma_{xx}$ is invertible with distinct eigenvalues, the error has, up to equivalence, a unique local and global minimum, at which $AB$ is the orthogonal projection onto the subspace spanned by the first $p$ eigenvectors $U_p$ of $\Sigma_{xx}$. Every other critical point is a saddle. At the minimum

$$A = U_p C, \qquad B = C^{-1} U_p^\top$$

for any invertible $p \times p$ matrix $C$. The network therefore finds the principal subspace, but its individual hidden units are the principal components when $C = I$; Baldi and Hornik expected gradient descent to produce a “random looking” $C$, which is why the link was not obvious in earlier simulations. Bourlard and Kamp (1988) had shown, via the singular value decomposition, that for auto-association with linear output units the optimal weights follow from the SVD of the mean-subtracted data, and that nonlinear hidden units are theoretically of no help: the output biases absorb the mean, and with sigmoid hidden units the same optimum can be approached as closely as required by keeping the units in their near-linear range. (Read in Bourlard’s 2000 reprint, IDIAP Research Report 00-16, which has minor extensions; these claims are the 1988 paper’s main result, stated in its abstract.)

The NumPy example below trains a linear autoencoder by gradient descent on 5,000 synthetic $8 \times 8$ patches (step edges at random angles, brightness ramps, pixel noise), compresses each patch to 6 numbers, and compares the result with PCA. The hand-written gradients are checked against finite differences first.

import sys
import numpy as np

rng = np.random.default_rng(0)                       # data
init = np.random.default_rng(int(sys.argv[1]) if len(sys.argv) > 1 else 0)  # weights

def make_patches(n):
    """n synthetic 8x8 patches: a step edge at a random angle and offset,
    a random brightness ramp, and pixel noise. Returned as rows of 64 values."""
    yy, xx = np.mgrid[0:8, 0:8] - 3.5
    theta = rng.uniform(0, np.pi, n)[:, None, None]
    offset = rng.uniform(-2, 2, n)[:, None, None]
    edge = np.tanh(2 * (np.cos(theta) * xx + np.sin(theta) * yy - offset))
    ramp = rng.normal(0, 0.3, (n, 2, 1, 1))
    img = (rng.uniform(0.2, 1, (n, 1, 1)) * edge
           + ramp[:, 0] * xx / 3.5 + ramp[:, 1] * yy / 3.5
           + 0.1 * rng.normal(size=(n, 8, 8)))
    return img.reshape(n, 64)

X = make_patches(5000)
X = X - X.mean(axis=0)                     # centre: the linear AE has no biases
k = 6                                       # code size: 64 pixels -> 6 numbers

def loss_and_grads(E, D, X):
    H = X @ E.T                             # codes, N x k
    R = H @ D.T - X                         # reconstruction residual, N x 64
    loss = 0.5 * np.mean(np.sum(R**2, axis=1))
    gD = R.T @ H / len(X)
    gE = (R @ D).T @ X / len(X)
    return loss, gE, gD

# check the hand-written gradients against central finite differences
E = 0.1 * init.normal(size=(k, 64)); D = 0.1 * init.normal(size=(64, k))
_, gE, gD = loss_and_grads(E, D, X[:200])
num = []
for M, g, idx in [(E, gE, (2, 10)), (D, gD, (30, 4))]:
    old = M[idx]; M[idx] = old + 1e-6; lp = loss_and_grads(E, D, X[:200])[0]
    M[idx] = old - 1e-6; lm = loss_and_grads(E, D, X[:200])[0]; M[idx] = old
    num.append(abs((lp - lm) / 2e-6 - g[idx]) / abs(g[idx]))
print(f"gradient check, max relative error: {max(num):.1e}")

# train encoder E (k x 64) and decoder D (64 x k) by plain gradient descent
for step in range(4000):
    loss, gE, gD = loss_and_grads(E, D, X)
    E -= 0.05 * gE; D -= 0.05 * gD

# PCA: eigenvectors of the covariance, largest eigenvalues first
evals, evecs = np.linalg.eigh(X.T @ X / len(X))
U = evecs[:, ::-1][:, :k]
pca_err = np.mean(np.sum((X - X @ U @ U.T) ** 2, axis=1))
ae_err = np.mean(np.sum((X - X @ E.T @ D.T) ** 2, axis=1))
print(f"mean squared error per patch: PCA {pca_err:.4f}, linear AE {ae_err:.4f}")

# principal angles between the decoder's column space and the PCA subspace
Q, _ = np.linalg.qr(D)
cosines = np.linalg.svd(U.T @ Q, compute_uv=False)
print(f"largest principal angle to the PCA subspace: {np.degrees(np.arccos(cosines.min())):.2f} deg")

# ...but the individual code units are not the principal components
C = np.corrcoef((X @ E.T).T)
print(f"largest |correlation| between two code units: {np.abs(C - np.eye(k)).max():.2f}"
      f" (PCA scores: {np.abs(np.corrcoef((X @ U).T) - np.eye(k)).max():.2f})")
print("variance captured by the top 6 of 64 components:", f"{evals[::-1][:k].sum() / evals.sum():.3f}")

Output (weight seed 0, about 5 s):

gradient check, max relative error: 8.4e-09
mean squared error per patch: PCA 2.3374, linear AE 2.3375
largest principal angle to the PCA subspace: 0.15 deg
largest |correlation| between two code units: 0.65 (PCA scores: 0.00)
variance captured by the top 6 of 64 components: 0.899

The autoencoder matches PCA’s reconstruction error, and its decoder spans the principal subspace to within a fraction of a degree. Its code units, however, are correlated with each other, unlike PCA scores, which are uncorrelated by construction: the code is the principal components mixed by some $C$. Over weight seeds 0 to 7 the largest angle ranged from 0.09° to 0.18° and the largest correlation from 0.62 to 0.84.

Why It Works

A reconstruction objective rewards a code that keeps whatever varies most across the training images and discards the rest. For a linear network “what varies most” means the leading principal directions; with nonlinear layers the code can follow curved structure that a linear subspace cannot. Vincent et al. (2008) describe the denoising variant geometrically: if images concentrate near a low-dimensional manifold, corrupted images lie farther from it, and learning to map them back teaches the network where the manifold is.

Variants

Deep autoencoders. Hinton and Salakhutdinov (2006) stacked several nonlinear layers on each side of a small linear code layer. Such networks were hard to train from random weights, so they pretrained one layer at a time as restricted Boltzmann machines, unrolled the stack into an encoder and a mirrored decoder, and fine-tuned the whole network with backpropagation. A 784-1000-500-250-30 autoencoder on MNIST reconstructed test digits with an average squared error of 3.00, against 8.01 for logistic PCA and 13.87 for PCA, all with 30 dimensions (Fig. 2B caption, for the test images shown). On synthetic curve images, their very deep autoencoder without pretraining always reconstructed the average training image.

Sparse autoencoders. A penalty $\Omega(h)$ on the code, such as an absolute-value penalty $\lambda \sum_j |h_j|$, makes most code units inactive for any one input, so even an overcomplete code must represent each image with a few active features (Goodfellow et al., 2016, §14.2.1).

Denoising autoencoders. Vincent et al. (2008) corrupted each input with a stochastic process $\tilde{x} \sim q(\tilde{x} \mid x)$, in their experiments setting a fixed number of randomly chosen inputs to 0, and trained the network to reconstruct the clean $x$ from $\tilde{x}$. The network can no longer succeed by copying, so the code need not be smaller than the input; their model selection usually chose an overcomplete first layer (typically 2,000 units for 784-pixel digits). Vincent (2011) showed that, for a tied-weight autoencoder with Gaussian corruption, the denoising objective is equivalent to matching the score of a specific energy-based model to that of a Parzen density estimate of the data. That connection reappears in diffusion models, which Rombach et al. (2022) describe as an equally weighted sequence of denoising autoencoders $\epsilon_\theta(x_t, t)$, one per time step $t$. Masking a large fraction of image patches is one corruption of this kind; see masked autoencoder.

Contractive autoencoders. Rifai et al. (2011) penalize the squared Frobenius norm of the encoder’s Jacobian,

$$\Omega(x) = \left\lVert \frac{\partial f(x)}{\partial x} \right\rVert_F^2 = \sum_{i,j} \left( \frac{\partial h_j(x)}{\partial x_i} \right)^2,$$

which makes the code insensitive to small input changes except along directions the reconstruction term needs. For a linear encoder the penalty reduces to weight decay. The authors contrast it with the denoising autoencoder, whose small-noise limit penalizes the Jacobian of the reconstruction $g(f(x))$ rather than of the code.

The PyTorch example below (PyTorch 2.14.1, CPU, one thread, about 30 s) trains a small convolutional denoising autoencoder on synthetic $32 \times 32$ images of filled disks and squares, with Gaussian noise of standard deviation 0.3 added to the input. The encoder uses two stride-2 convolutions; the decoder mirrors it with transposed convolutions. After training, it measures denoising on held-out shapes and uses the reconstruction error of clean images to separate the shapes from plus-shaped crosses, which never appear in training.

import sys
import torch
from torch import nn
import torch.nn.functional as F

torch.set_num_threads(1)
SEED = int(sys.argv[1]) if len(sys.argv) > 1 else 0

def shapes(n, g, kind="normal"):
    """n 32x32 images on a dark background. 'normal': one filled disk or square of
    random size, position and brightness. 'cross': a plus sign (never seen in training)."""
    yy, xx = torch.meshgrid(torch.arange(32.), torch.arange(32.), indexing="ij")
    c = 8 + 16 * torch.rand(n, 2, 1, 1, generator=g)
    r = 4 + 4 * torch.rand(n, 1, 1, generator=g)
    dy, dx = (yy - c[:, 0]).abs(), (xx - c[:, 1]).abs()
    if kind == "normal":
        disk = dy**2 + dx**2 <= r**2
        square = torch.maximum(dy, dx) <= r * 0.8
        is_disk = torch.rand(n, 1, 1, generator=g) < 0.5
        mask = torch.where(is_disk, disk, square)
    else:
        mask = ((dy <= 1.5) & (dx <= r)) | ((dx <= 1.5) & (dy <= r))
    level = 0.5 + 0.5 * torch.rand(n, 1, 1, generator=g)
    return (mask * level)[:, None]

class DenoisingAE(nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder = nn.Sequential(                         # 1x32x32 -> 4x8x8 code
            nn.Conv2d(1, 16, 3, stride=2, padding=1), nn.ReLU(),
            nn.Conv2d(16, 32, 3, stride=2, padding=1), nn.ReLU(),
            nn.Conv2d(32, 4, 3, padding=1))
        self.decoder = nn.Sequential(                         # 4x8x8 -> 1x32x32
            nn.ConvTranspose2d(4, 32, 2, stride=2), nn.ReLU(),
            nn.ConvTranspose2d(32, 16, 2, stride=2), nn.ReLU(),
            nn.Conv2d(16, 1, 3, padding=1))

    def forward(self, x):
        return self.decoder(self.encoder(x))

torch.manual_seed(SEED)
g = torch.Generator().manual_seed(SEED)
net = DenoisingAE()
opt = torch.optim.Adam(net.parameters(), lr=2e-3)
sigma = 0.3
for step in range(850):
    x = shapes(64, g)
    x_noisy = x + sigma * torch.randn(x.shape, generator=g)   # corrupt the input...
    loss = F.mse_loss(net(x_noisy), x)                       # ...reconstruct the clean image
    opt.zero_grad(); loss.backward(); opt.step()

gt = torch.Generator().manual_seed(1234)                     # same test set for every seed
x_test = shapes(500, gt)
x_test_noisy = x_test + sigma * torch.randn(x_test.shape, generator=gt)
x_cross = shapes(500, gt, kind="cross")
psnr = lambda a, b: (10 * torch.log10(1 / ((a - b) ** 2).mean(dim=(1, 2, 3)))).mean()
with torch.no_grad():
    den = net(x_test_noisy)
    err_normal = ((net(x_test) - x_test) ** 2).mean(dim=(1, 2, 3))
    err_cross = ((net(x_cross) - x_cross) ** 2).mean(dim=(1, 2, 3))
code = net.encoder(x_test[:1])
auroc = (err_cross[:, None] > err_normal[None, :]).float().mean()
print(f"PyTorch {torch.__version__}; code {tuple(code.shape[1:])} = {code.numel()} numbers for 1024 pixels")
print(f"test PSNR: noisy input {psnr(x_test_noisy, x_test):.2f} dB, denoised {psnr(den, x_test):.2f} dB")
print(f"reconstruction MSE: shapes {err_normal.mean():.4f}, crosses {err_cross.mean():.4f}")
print(f"AUROC of reconstruction error as a cross detector: {auroc:.3f}")

Output (seed 0; two runs gave identical output):

PyTorch 2.14.1; code (4, 8, 8) = 256 numbers for 1024 pixels
test PSNR: noisy input 10.47 dB, denoised 25.86 dB
reconstruction MSE: shapes 0.0018, crosses 0.0068
AUROC of reconstruction error as a cross detector: 0.991

Over seeds 0 to 9 the denoised PSNR ranged from 25.30 to 26.25 dB and the AUROC from 0.976 to 0.993. Crosses are reconstructed more than three times worse than the shapes the network was trained on, which is the principle behind reconstruction-based anomaly detection.

Training Considerations

• Loss and blur. Pixel-space losses such as $L^2$ or $L^1$ alone give blurry reconstructions, so latent diffusion’s autoencoder adds a perceptual loss and a patch-based adversarial loss (Rombach et al., 2022).
• Capacity versus constraint. Code size, depth and penalty strength trade reconstruction quality against how much the code has to abstract. Reconstruction error alone does not measure how useful a code is (our note): the linear autoencoder above reaches PCA’s error with any mixing $C$, and an overcomplete network can reach zero error by copying. The code is judged on the task it serves.
• Corruption level. In denoising autoencoders the noise level or masking fraction is a hyperparameter, chosen by validation in Vincent et al. (2008).

Use in Computer Vision

• Compression. Ballé, Laparra and Simoncelli (2017) built an image codec from a convolutional analysis transform, a uniform quantizer and a synthesis transform, trained end to end for rate–distortion by replacing quantization with additive uniform noise during training. On an independent test set it generally beat JPEG and JPEG 2000 in rate–distortion, and the authors reported a large improvement in visual quality, supported by MS-SSIM.
• Denoising. A denoising autoencoder is a learned denoiser; dedicated restoration networks are covered in image denoising.
• Anomaly detection. An autoencoder trained only on defect-free images reconstructs defects poorly, so the difference between input and reconstruction becomes an anomaly map. In the initial benchmark of the MVTec AD paper, convolutional autoencoders with an $L^2$ or SSIM loss outperformed the other methods on most of the ten object categories, but blurred high-frequency textures such as those of tile and zipper, causing many false positives on tile (Bergmann et al., 2019).
• Pretraining. Denoising autoencoders trained one layer at a time initialized deep networks that were then fine-tuned with labels (Vincent et al., 2008), as stacked RBMs had (Hinton & Salakhutdinov, 2006). Masked image modeling revived reconstruction as a self-supervised pretext task, with the encoder kept for transfer learning.
• Latent spaces for generation. Latent diffusion models first train an autoencoder that downsamples an RGB image by a factor $f$ into a spatial latent $z = E(x)$, regularized either by a slight KL penalty toward a standard normal or by vector quantization, and then train the diffusion model on $z$ instead of pixels (Rombach et al., 2022). In their class-conditional ImageNet comparison (arXiv v2, §4.1, Fig. 6), factors from 4 to 16 balanced training efficiency against fidelity: $f = 1$ and $2$ trained slowly, and $f = 32$ limited sample quality, which the authors attribute to information loss in the first stage.

Limitations

• A plain autoencoder is not a generative model. It defines an encoder and a decoder, but no probability distribution over codes. Decoding a random code gives no guarantee of a plausible image, because nothing in training constrains the decoder outside the codes that training images actually produce, and those may occupy an irregular, small region of code space. The variational autoencoder addresses this: its encoder outputs a distribution $q_\phi(z \mid x)$, a KL term pulls it toward the prior $p(z) = \mathcal{N}(0, I)$, and training maximizes a lower bound on $\log p_\theta(x)$, so codes can be sampled from the prior. Goodfellow et al. (2016, §14.2) contrast such models, trained to approximately maximize the probability of the data, with autoencoders trained to copy.
• Reconstruction is not understanding. Low reconstruction error can coexist with codes that are poor features for recognition, and pixel losses spend capacity on detail that may not matter.
• Anomaly detection can fail both ways. A network that generalizes well may reconstruct anomalies too, while one that blurs normal fine texture raises false alarms, as on MVTec AD’s tile images.

Important Models

Model	Year	Constraint on the code	Main use
Linear autoencoder (Bourlard & Kamp; Baldi & Hornik)	1988–89	Bottleneck, linear	Analysis: equivalent to PCA
Deep autoencoder (Hinton & Salakhutdinov)	2006	Bottleneck, RBM pretraining	Nonlinear dimensionality reduction
Denoising autoencoder (Vincent et al.)	2008	Corrupted input	Feature learning, pretraining
Contractive autoencoder (Rifai et al.)	2011	Jacobian penalty	Feature learning
End-to-end optimized codec (Ballé et al.)	2017	Quantized code, rate penalty	Image compression
Masked autoencoder (He et al.)	2022	75% of patches masked	ViT pretraining
Latent diffusion autoencoder (Rombach et al.)	2022	Downsampling, KL or VQ regularization	Latent space for diffusion