VQGAN

VQGAN is a vector-quantized autoencoder that turns an image into a short grid of discrete codes, trained with a perceptual loss and a patch-based adversarial loss so that the decoded image stays sharp even when the grid is small. Esser, Rombach and Ommer introduced it at CVPR 2021 in “Taming Transformers for High-Resolution Image Synthesis”, where it is the first of two stages: a GPT-2-style transformer then models the grid of code indices autoregressively, and the decoder turns sampled grids back into images. The paper names the autoencoder VQGAN and the whole method by its title; in later work “VQGAN” often stands for the pair, and the autoencoder alone is the archetype of what is now called an image tokenizer. It is a variant of the VQ-VAE, whose quantizer, codebook loss, commitment loss and straight-through gradient it keeps, and the first stage of latent diffusion follows its losses.

The VQ-VAE article covers the quantizer and its training, and the perceptual loss article covers feature losses and their relation to the LPIPS metric. This article covers what VQGAN changed: the losses and their adaptive balance, the context-rich codebook at high compression, the transformer prior with its sliding attention window, conditional synthesis, the results the authors reported, and the tokenizers that followed.

Problem Addressed

By 2020 transformers had become the standard architecture for language and were spreading to images, but self-attention costs grow with the square of the sequence length, and an image unrolled into pixels is a very long sequence. Esser et al. note that earlier transformers for image generation worked up to about $64 \times 64$ pixels (§3), and that methods restricting or sparsifying attention either assume independence between distant pixels or still left resolutions beyond 64 pixels too expensive (§2). Image GPT (Chen et al., 2020), as Esser et al. describe it, used a shallow VQ-VAE with a small receptive field to reach $192 \times 192$, keeping the codes as close to the pixels as possible (§2).

Two-stage models had the right shape. The VQ-VAE and VQ-VAE-2 compress an image into a grid of codes and fit an autoregressive prior to them, but their priors were convolutional (PixelCNN and PixelSNAIL), which, Esser et al. argue, makes long-range interactions hard to capture at high resolution (§2). A transformer prior needs a short sequence, and with a squared-error reconstruction loss a VQ-VAE cannot compress that far: at the downsampling factor $f = 16$, the authors’ VQ-VAE reconstructions of faces were blurry and kept little of the image (supplement, Fig. 9). Their hypothesis is that low-level structure is well described by local connectivity, the inductive bias of convolutional networks, while that bias stops paying on higher semantic levels (§1). So a convolutional stage should learn a codebook of “context-rich” visual parts that already carry local texture, and a transformer should learn only how they compose.

Architecture

Images $x \in \mathbb{R}^{H \times W \times 3}$ are encoded to $\hat z = E(x) \in \mathbb{R}^{h \times w \times n_z}$, quantized against a codebook $\mathcal{Z} = \{z_k\}_{k=1}^{K} \subset \mathbb{R}^{n_z}$ and decoded by $G$. A discriminator $D$ is used only in training. The transformer works on the indices.

Encoder, Codebook and Decoder

Each spatial vector of the encoder output is replaced by its nearest codebook entry (Eq. 2):

$$z_q = \mathbf{q}(\hat z) := \Big( \arg\min_{z_k \in \mathcal{Z}} \lVert \hat z_{ij} - z_k \rVert \Big)_{ij} \in \mathbb{R}^{h \times w \times n_z}, \qquad \hat x = G(z_q) = G\big(\mathbf{q}(E(x))\big),$$

which is equivalent to a grid $s \in \{0, \dots, |\mathcal{Z}| - 1\}^{h \times w}$ of indices with $s_{ij} = k$ when $(z_q)_{ij} = z_k$ (Eq. 8). The downsampling factor is $f = H/h = 2^m$ for $m$ downsampling blocks.

The encoder and decoder follow the architecture of DDPM (Ho et al., 2020) without its skip connections (supplement, Table 6): a convolution, $m$ stages of a residual block followed by downsampling, then a residual block, a non-local (self-attention) block and another residual block at the lowest resolution, and GroupNorm, Swish and a convolution to $n_z$ channels; the decoder mirrors it with upsampling. The single attention layer at the lowest resolution is there “to aggregate context from everywhere” (§3.1). In the released ImageNet configuration ($f = 16$) the base width is 128 channels with multipliers $(1, 1, 2, 2, 4)$, two residual blocks per stage, attention at resolution 16, and a codebook of 1,024 entries of dimension 256 (configs/imagenet_vqgan.yaml). Most experiments used $|\mathcal{Z}| = 1024$ and $n_z = 256$ (§4, supplement Table 7).

Patch Discriminator

The discriminator is the patch-based network of pix2pix (Isola et al., 2017), the PatchGAN, which classifies overlapping image patches rather than the whole image (§3.1). In the official code it is NLayerDiscriminator, with three stride-2 and two stride-1 $4 \times 4$ convolutions, batch normalization, LeakyReLU(0.2) and 64 base channels; its output is a map of logits, each of which, by our count of the layers, sees a $70 \times 70$-pixel patch, the receptive field of the default discriminator of pix2pix as the pix2pix article describes it.

Transformer

The second stage is identical to GPT-2 (supplement, §A): a decoder-only transformer with causal masking that predicts the next index. Capacity is varied mainly by depth; most models have 24 layers, 16 heads and embedding dimension 1,024, about 307 million parameters (rounded to 310M in the main text), trained on sequences of $16 \times 16 = 256$ tokens, which the authors chose as the longest feasible for that model on a GPU with 12 GB (§4, Table 7). Larger ones include a 48-layer, 1.4-billion-parameter class-conditional ImageNet model with 16,384 codes (Table 7).

Conditioning Stage

A condition $c$ enters the transformer as a prefix. A class label is a single index, so a class-conditional sequence has length 257 (Table 7). A condition with spatial extent, such as a segmentation, depth map, edge map, pose or low-resolution image, gets its own VQGAN with codebook $\mathcal{Z}_c$; its indices $r$ are prepended to $s$, and the loss is computed only on the entries of $s$ (§3.2, Eq. 10). For semantic layouts that first stage is trained with a cross-entropy reconstruction loss, because the layouts are discrete (§4.1). The authors note that this “decoder-only” strategy had been used for text summarization.

Training

The two stages are trained one after the other: first the autoencoder with its codebook and discriminator, then, with $E$, $G$ and $\mathcal{Z}$ frozen, the transformer on the index grids of the training images.

The Autoencoder Objective

The starting point is the VQ-VAE objective, which Esser et al. write as (Eq. 4)

$$\mathcal{L}_{\mathrm{VQ}}(E, G, \mathcal{Z}) = \lVert x - \hat x \rVert^2 + \lVert \mathrm{sg}[E(x)] - z_q \rVert_2^2 + \beta \lVert \mathrm{sg}[z_q] - E(x) \rVert_2^2 ,$$

with the stop-gradient $\mathrm{sg}$, the straight-through gradient from decoder to encoder, and $\beta = 0.25$ throughout (supplement, Table 7). VQGAN makes two changes “to keep good perceptual quality at increased compression rate” (§3.1). It replaces the $L^2$ term in $\mathcal{L}_{\mathrm{rec}}$ by a perceptual loss, citing LPIPS (Zhang et al., 2018), and it adds an adversarial loss from the patch discriminator (Eq. 5),

$$\mathcal{L}_{\mathrm{GAN}}(\{E, G, \mathcal{Z}\}, D) = \log D(x) + \log\big(1 - D(\hat x)\big),$$

so that the full first-stage problem is (Eq. 6)

$$\mathcal{Q}^\ast = \arg\min_{E, G, \mathcal{Z}} \; \max_{D} \; \mathbb{E}_{x \sim p(x)} \Big[ \mathcal{L}_{\mathrm{VQ}}(E, G, \mathcal{Z}) + \lambda\, \mathcal{L}_{\mathrm{GAN}}(\{E, G, \mathcal{Z}\}, D) \Big].$$

In our reading, the adversarial term judges patches, not pixels, so a reconstruction can be rewarded for realistic texture that does not match the original pixel for pixel, and the perceptual term compares feature statistics rather than pixel values; both work against the blur that the squared error gave the authors’ VQ-VAE at $f = 16$ (supplement, Fig. 9). Two worked examples on this site show the pieces separately: the pix2pix article’s example measures, in the power spectrum, what patch discriminators of different receptive fields add to a translator trained with a pixel loss, and the VQ-VAE article’s example measures how many codes a small quantizer actually uses under different codebook updates.

The Adaptive Weight

The weight $\lambda$ is not a fixed hyperparameter. It is computed at every step from the gradients of the two losses with respect to the decoder’s last layer $G_L$ (Eq. 7):

$$\lambda = \frac{\nabla_{G_L} \big[\mathcal{L}_{\mathrm{rec}}\big]}{\nabla_{G_L} \big[\mathcal{L}_{\mathrm{GAN}}\big] + \delta}, \qquad \delta = 10^{-6},$$

where $\mathcal{L}_{\mathrm{rec}}$ is the perceptual reconstruction loss. The printed equation divides the gradients themselves; the official code divides their norms, $\lambda = \lVert \nabla_{G_L} \mathcal{L}_{\mathrm{rec}} \rVert / (\lVert \nabla_{G_L} \mathcal{L}_{\mathrm{GAN}} \rVert + 10^{-4})$, clamps the result to $[0, 10^4]$, treats it as a constant in the backward pass and multiplies it by a configurable disc_weight, set to 0.8 in the released ImageNet and FacesHQ configurations (vqperceptual.py). In our reading, $\lambda$ scales the adversarial gradient arriving at the last layer to the size of the reconstruction gradient, so neither term dominates the decoder’s output whatever the raw scales of the two losses; the paper does not discuss the motivation beyond the equation.

The adversarial term is also switched off at first. The supplement states that $\lambda$ is set to zero in an initial warm-up phase, that longer warm-ups generally gave better reconstructions, and recommends at least one epoch (§A; it refers to “λ in Eq. 5”, although $\lambda$ appears in Eq. 6). The released configurations start the discriminator at step 250,001 for ImageNet and 30,001 for FacesHQ (disc_start), and before then both the adversarial term and the discriminator’s loss are multiplied by zero.

What the Official Code Does Differently

The released code (taming-transformers, as of November 2026; last commit November 2022) differs from the paper’s equations in four further ways:

• Reconstruction loss. VQLPIPSWithDiscriminator adds an $L^1$ pixel loss to the LPIPS term, $|x - \hat x| + w_p \cdot \mathrm{LPIPS}(x, \hat x)$ with $w_p = 1$, rather than replacing the pixel loss altogether.
• Adversarial loss. The default is the hinge loss: the discriminator minimizes $\tfrac12\big(\mathbb{E}[\max(0, 1 - D(x))] + \mathbb{E}[\max(0, 1 + D(\hat x))]\big)$ over its logit map, and the autoencoder minimizes $-\mathbb{E}[D(\hat x)]$. The log form of Eq. 5 is available as the “vanilla” option.
• The commitment weight. By default the quantizer applies $\beta$ to the codebook term and weight 1 to the commitment term, the reverse of Eq. 4. The code comments call this a bug kept for backward compatibility, fixed with legacy=False, and the README describes the default as corresponding to always training with $\beta = 1.0$.
• Optimization. Adam with $\beta_1 = 0.5$, $\beta_2 = 0.9$ for both the autoencoder and the discriminator, a base learning rate of $4.5 \times 10^{-6}$ multiplied by batch size, number of GPUs and gradient-accumulation steps (main.py), and codebook entries initialized uniformly in $[-1/K, 1/K]$.

Training the Transformer

With the first stage frozen, an image is encoded to its index grid, unrolled into a sequence in some order, and the transformer maximizes the log-likelihood of the sequence (Eq. 9),

$$\mathcal{L}_{\mathrm{Transformer}} = \mathbb{E}_{x \sim p(x)} \big[ -\log p(s) \big], \qquad p(s) = \prod_i p(s_i \mid s_{<i}),$$

or, with a condition, $p(s \mid c) = \prod_i p(s_i \mid s_{<i}, c)$ (Eq. 10). The cost of attention limits the sequence, so training crops images to a fixed token grid, $16 \times 16$ in most experiments; with factor $f$, crops of $16f \times 16f$ pixels (§3.2, §4.3).

The order matters. The supplement (§D) trained transformers on the codes of an ImageNet VQGAN in six orders with the same initialization and budget, and reported final validation negative log-likelihoods of 4.767 for row-major (raster) order, 4.812 for alternating rows, 4.810 for a Z-curve, 4.889 for spiral-out, 4.901 for spiral-in and 5.015 for subsampling; the two worst gave more textural samples. Row-major order is also the one the sliding-window sampler below relies on (§D).

The code adds one option the paper does not mention: with pkeep below 1, each input index of the transformer is replaced by a random code with probability $1 - p_{\mathrm{keep}}$ during training, while the targets stay the true indices (cond_transformer.py).

Inference

Reconstruction. One encoder pass, a nearest-neighbour search per position and one decoder pass. Because the tokenizer is a full autoencoder, it can encode real images, which the authors present as an advantage over GANs: their model retains “the ability to encode and reconstruct images” (§4.4).

Sampling. The transformer draws indices one at a time, and the decoder renders the finished grid in one pass. The supplement samples with temperature 1.0 and a top-$k$ cutoff of 100, raised for larger codebooks (§A); the README’s class-conditional ImageNet model is sampled with $k = 600$, nucleus sampling with $p = 0.92$ and temperature 1.0, and the authors recommend experimenting with these decoding parameters (as of November 2026). Completing an image is the same procedure started from the indices of its top half, as in the image-completion figures (Fig. 4; supplement, Fig. 15–16).

The sliding attention window. Generating beyond the training crop needs longer sequences than the transformer was trained on. Esser et al. instead apply it patch-wise (§3.2, Fig. 3): to sample index $(i, j)$ of a large grid, the transformer is run on the $16 \times 16$ window of indices around it, with the corresponding window of condition indices as its prefix when there is a spatial condition; through the causal mask, only the window’s entries before $(i, j)$ in raster order affect the prediction. In the official sampling script the window is placed so that the current position sits at local row and column 8, its centre, except near the borders, where the window is clamped to the grid and the position moves within it; the logits for that position are read from the window’s output and one index is sampled. The authors argue that the VQGAN’s context-rich codes keep this local context sufficient as long as the data’s statistics are approximately spatially invariant or a spatial condition is available. For aligned data such as faces, which violate that assumption, they condition on image coordinates instead (§3.2), as the released FacesHQ transformer does (cond_stage_key: coord). Megapixel results use this window; Fig. 5 shows landscapes of up to $1280 \times 832$ pixels generated from semantic layouts.

Speed. Sampling remains sequential, one transformer pass per token. Compression is what makes it affordable: on FacesHQ, with the sequence fixed at 256 tokens, sampling at $f = 2$, 8 and 16 was 3.86, 65.81 and 280.68 times faster than at $f = 1$, which took 7,258 seconds per sample on an NVIDIA GeForce GTX Titan X (Fig. 7). The $f = 1$ baseline replaces the VQGAN by a k-means clustering of RGB values with 512 centres, as in Image GPT (§4.3).

Performance

The paper evaluates both stages and their composition. Sample quality is measured by FID; for the semantic-synthesis comparison the authors computed it with torch-fidelity on the validation splits (Table 2).

Transformer against PixelSNAIL (Table 1). On the same VQGAN codes ($m = 4$, $f = 16$), the transformer had lower negative log-likelihood than PixelSNAIL on every task, both at equal training time (PixelSNAIL trains about twice as fast) and at equal steps: on unconditional ImageNet (310M parameters), for example, the transformer reached 4.78 at PixelSNAIL’s number of steps and 4.83 at its training time, against 4.96 for PixelSNAIL, and 4.49 and 4.57 against 4.64 on landscapes conditioned on semantic layouts (S-FLCKR). The table does not state the unit. The other tasks were Restricted ImageNet, LSUN Churches and Towers, and Restricted ImageNet conditioned on depth.

Context-rich vocabularies (§4.3). With the transformer and its 256-token sequence fixed, faces on FacesHQ, the union of CelebA-HQ and FFHQ, were incoherent at small $f$; at $f = 8$ the overall structure appeared with inconsistencies such as a half-bearded face; only $f = 16$ gave high-fidelity samples (Fig. 7). On CIFAR-10, a transformer on VQGAN codes with $f = 2$ ($16 \times 16$ tokens) improved FID by 18.63% and sampled 14.08 times faster than the same transformer on pixels quantized to 512 colours, at a fixed computational budget. The supplement (§B, Fig. 8) quantifies the trade-off on faces: the reconstruction error, measured by LPIPS, rose from 0.11 at $f = 2$ to 0.23 at $f = 8$, 0.38 at $f = 16$, 0.63 at $f = 32$ and 0.66 at $f = 64$, while the transformer’s negative log-likelihood fell from $5.66 \times 10^4$ to $6.75 \times 10^1$; the authors place the critical value for faces at $f = 16$, beyond which reconstructions, and with them samples, lose fidelity. The text of §B says the reconstruction error “decreases with f”; the figure’s values increase. On landscapes, reconstructions stayed faithful at $f = 32$ (Fig. 10), which the authors suggest may reflect a higher perceptual sensitivity to faces than to textures.

Semantic image synthesis (Table 2, $256 \times 256$). FID 22.4 on COCO-Stuff and 35.5 on ADE20K, against 22.6 and 33.9 reported for SPADE, or 23.9 and 35.7 when recalculated with the authors’ protocol on the validation splits, 111.5 and 81.8 for pix2pixHD (54.2 and 41.5 with augmentation), and 70.4 and 73.3 for CRN. The README of the released code lists 20.4 for its COCO-Stuff model, against the validation split of 5,000 images (as of November 2026).

Unconditional faces (Table 3, $256 \times 256$). FID 10.7 on CelebA-HQ, against 8.0 for PGGAN and 15.8 for DC-VAE; 11.4 on FFHQ, against 21.9 for a PixelSNAIL on the same VQGAN codes, 12.4 for BigGAN, 10.9 for U-Net GAN (7.6 with augmentation) and 3.8 for StyleGAN2 (3.6).

Class-conditional ImageNet (Table 4, $256 \times 256$, $f = 16$, 16,384 codes). With 50,000 samples the FID was 19.8, or 11.2 with VQ-VAE-2’s classifier-based rejection sampling, against 38.1 (about 10 with rejection) for VQ-VAE-2 and 7.1 for BigGAN (7.3 for BigGAN-deep). FIDs against reconstructed data, as VQ-VAE-2 reported them, were 13.5 (8.1) against 19 (5). The authors note that their transformer has about ten times fewer parameters than VQ-VAE-2, whose count they estimate at 13.5 billion from a third-party reimplementation, and that task-specialized GANs report better FIDs. A larger class-conditional model added to the repository later is listed with FIDs against the training set from 15.98 down to 5.20 for “different decoding hyperparameters” (README, as of November 2026); MaskGIT’s comparison on ImageNet $256 \times 256$ lists the 1.4-billion-parameter VQGAN at 15.78, and at 5.88 with rejection sampling at an acceptance rate of 0.05 (Chang et al., 2022, Table 1).

The tokenizer itself (Table 5). Reconstruction FID on the ImageNet validation split bounds what any model on the codes can reach. VQ-VAE-2’s two-level codes ($64 \times 64$ and $32 \times 32$, 512 entries, 5,120 positions) were reported at about 10; VQGAN with a $16 \times 16$ grid (256 positions) scored 8.0 with 1,024 entries and 4.9 with 16,384, and 1.7 with VQ-VAE-2’s two-level layout. The table’s column “Codebook Size” holds the grid sizes and “dim $\mathcal{Z}$” the number of entries. On the faces of supplement Fig. 9, with the same architecture at $f = 16$, a VQ-VAE trained with the squared error had a reconstruction FID of 254.4, against 5.7 for the VQGAN, whose codes gave the FFHQ sample FIDs of 21.93 with PixelSNAIL and 11.44 with the transformer (§4.4, Table 3). The latent diffusion paper reports a PSNR of 19.9 dB for VQGAN’s $f = 16$ autoencoder on ImageNet validation, which it took as a sign that heavy compression caps quality.

Strengths

• Short sequences at perceptual quality. A $256 \times 256$ image becomes 256 indices at $f = 16$, while reconstruction FID on ImageNet stays below VQ-VAE-2’s with twenty times fewer positions (Table 5).
• One recipe for many tasks. Unconditional, class-conditional, segmentation-, depth-, edge- and pose-conditioned synthesis and stochastic super-resolution all use the same two stages, with a second VQGAN for spatial conditions and no task-specific modules (§4.2).
• Resolution beyond the training crop. The sliding window generates images larger than the crops seen in training, given approximately stationary statistics or a spatial condition (§3.2, §4.2).
• A likelihood model that can encode. The transformer gives exact log-likelihoods of codes, and the tokenizer reconstructs real images, which the authors see as bridging adversarial and likelihood-based models (§4.4).
• A reusable first stage. The tokenizer is trained once and serves many second-stage models; MaskGIT used the same first-stage set-up unchanged (Chang et al., 2022, §3), and the latent diffusion autoencoder follows its losses.

Limitations

• Compression has a critical value. Beyond some $f$, which depends on the data ($f = 16$ for faces, at least 32 for landscapes), reconstructions and therefore samples lose fidelity (supplement, §B).
• Lossy, invented detail. An adversarially trained decoder fills in plausible texture that need not match the input; at $f = 16$, reconstruction PSNR on ImageNet was 19.9 dB in the latent diffusion paper’s comparison. Those authors took heavy compression as the price of a transformer prior and moved to a milder, two-dimensional latent for diffusion.
• Sequential sampling. One transformer pass per token, in raster order. MaskGIT, which predicts many tokens in parallel on the same kind of codes, reported 30 to 48 times faster decoding than VQGAN’s autoregressive transformer, growing with resolution (Chang et al., 2022, §4.2).
• A fixed raster order and a local window. The sliding window sees only 256 neighbouring tokens; aligned data needs coordinate conditioning, and the result depends on the scan order (supplement, §D).
• Delicate training. A warm-up without the discriminator is recommended (supplement, §A), and MAGVIT’s authors describe their tokenizer, with a LeCam regularizer and a StyleGAN discriminator, as training stably with the GAN loss from the beginning, “unlike VQGAN” (Yu et al., 2023, §3.1).
• Codebook size and use. Larger codebooks reconstruct better (Table 5), but simply enlarging the codebook does not recover what a smaller grid loses: on ImageNet, an $8 \times 8$ VQGAN had reconstruction FIDs of 17.95, 17.66 and 17.09 with 16,384, 65,536 and 131,072 entries, against 4.90 for $16 \times 16$ with 16,384 (Lee et al., 2022, Table 4). Lee et al. attribute the inefficiency of large codebooks to codebook collapse and unstable training (§3.1.1), the problem the VQ-VAE article covers.

Historical Importance

Taming Transformers gave transformers a short, perceptually rich sequence to model, at a time when pixel-space transformers stopped at a few dozen pixels per side, and its authors presented the first results of semantically guided megapixel synthesis with transformers (abstract). Its central claim was about the first stage rather than the transformer: a powerful tokenizer that captures as much context as possible is what makes high-resolution synthesis with transformers efficient (§2). Its first stage, a convolutional autoencoder with an attention layer trained with a perceptual and a patch-adversarial loss, was reused and extended by later systems. MaskGIT kept VQGAN’s first stage and replaced its prior with a masked bidirectional transformer; MAGVIT built its video tokenizer on VQGAN’s architecture; and the autoencoder of latent diffusion, and with it Stable Diffusion, follows VQGAN’s losses with a continuous or quantized latent and milder compression. Outside research papers, VQGAN was paired with CLIP in VQGAN-CLIP, which, as the image editing article describes, optimizes CLIP’s similarity directly and scores accordingly on CLIP-based metrics. The autoregressive image model article places the VQGAN transformer among autoregressive models over pixels and tokens, from PixelRNN to masked-token and next-scale prediction.

Variants

Changes in the Taming Transformers Codebase

The repository added, after the CVPR paper, a checkpoint of a VQGAN with $f = 8$ and 8,192 codes trained with Gumbel quantization, cached keys and values for faster sampling, unconditional FFHQ and CelebA-HQ models, the 1.4-billion-parameter class-conditional ImageNet model and scene-synthesis models conditioned on bounding boxes (README, as of November 2026). It lists reconstruction FIDs on ImageNet validation of 7.94 and 4.98 for the $f = 16$ VQGANs with 1,024 and 16,384 codes, close to Table 5’s 8.0 and 4.9, and 32.01 for the DALL-E discrete VAE with $f = 8$; the OpenImages $f = 8$ VQGANs released with latent diffusion are listed in the same column at 1.49 with 256 codes and 1.14 with 16,384. The README also links a third revision of the arXiv preprint with a changelog, which was not read for this article.

Later Image Tokenizers

The tokenizers that followed kept the two-stage recipe and changed the quantizer, the network or the losses. In outline, from the papers and code that could be read:

• ViT-VQGAN, from Yu et al.’s “Vector-quantized Image Modeling with Improved VQGAN” (ICLR 2022), which MaskGIT calls VIM and summarizes as using a Vision Transformer backbone to further improve the tokenization stage (Chang et al., 2022, §2.1); Parti generated from ViT-VQGAN tokens (see text-to-image generation). The paper itself could not be read for this article.
• Residual quantization (RQ-VAE) (Lee et al., 2022) quantizes each vector $D$ times with one shared codebook, each step coding the residual left by the previous ones: $k_d = \mathcal{Q}(r_{d-1}; \mathcal{C})$, $r_d = r_{d-1} - e(k_d)$, $\hat z = \sum_d e(k_d)$ (§3.1.2, Eq. 3–4). With depth $D$ it can partition the space into up to $K^D$ cells, as many as a VQ with $K^D$ codes. Trained with a patch adversarial loss and a perceptual loss as in VQGAN, an $8 \times 8 \times 4$ code map with 16,384 entries reached a reconstruction FID of 4.73 on ImageNet, against 4.90 for VQGAN’s $16 \times 16$ map as published (4.32 in Lee et al.’s reproduction), with a quarter of the positions (Table 4); with 1.4 billion parameters each, the RQ-Transformer sampled up to 7.3 times faster than a VQGAN transformer, a figure that compares its batch of 500, which the shorter sequence allowed, with the VQGAN transformer’s batch of 200 (§4.3, Fig. 4).
• 3D tokenizers for video. MAGVIT’s 3D-VQ is built on VQGAN’s encoder and decoder with all 2D convolutions expanded to 3D and with both 3D and 2D downsampling layers; it replaces zero padding with reflect padding, applies the image perceptual loss to each frame, adds LeCam regularization to the GAN loss and uses a StyleGAN discriminator inflated to 3D, and compresses video by $4 \times 8 \times 8$ (Yu et al., 2023, §3.1, Table 7); the video generation article covers token-based video models.
• Finite scalar quantization (FSQ) removes the learned codebook. The encoder projects to a few dimensions, typically fewer than ten, each bounded with a $\tanh$ and rounded to one of $L_i$ levels with a straight-through gradient; the implicit codebook is the product of the level sets, of size $\prod_i L_i$, and the index is a mixed-radix number. The official README and JAX code (Google Research, as of November 2026) state that FSQ needs no commitment or codebook loss, no EMA, reseeding or splitting, and that it does not suffer from codebook collapse; the authors used it with MaskGIT for generation and with UViM for depth estimation, colorization and panoptic segmentation.
• Lookup-free quantization (LFQ), which Infinity’s authors cite to Yu et al.’s “Language Model Beats Diffusion — Tokenizer Is Key to Visual Generation”, quantizes each dimension to its sign, $q(z) = \mathrm{sign}(z)$, as Infinity’s authors write it, so a $d$-dimensional code is one of $2^d$ binary vectors and needs no codebook lookup. Infinity adopts an entropy penalty $\mathbb{E}[H(q(z))] - H[\mathbb{E}(q(z))]$ to encourage codebook use; computing it for LFQ compares the input with the whole implicit codebook, at a cost that grows as $O(2^d)$; in their memory comparison LFQ ran out of memory from $d = 20$, while binary spherical quantization (BSQ), which normalizes to the unit sphere and approximates the penalty in $O(d)$, did not (Han et al., 2025, §3.2, Eq. 4, Table 3). Infinity’s own tokenizer uses BSQ with vocabularies up to $2^{64}$. Neither Yu et al.’s LFQ paper nor the BSQ paper could be read for this article.

Implementations

• taming-transformers (CompVis, MIT License): the VQGAN (taming/models/vqgan.py), its loss with LPIPS and the PatchGAN (taming/modules/losses/vqperceptual.py), the quantizers (taming/modules/vqvae/quantize.py), a minGPT-based transformer (taming/modules/transformer/mingpt.py), sampling scripts including the sliding-window sampler (scripts/sample_conditional.py), configurations for custom data, and links to pretrained first- and second-stage models (as of November 2026; the last commit is of November 2022). The LPIPS loss loads torchvision’s pretrained VGG-16 and a checkpoint of LPIPS’s linear layers.
• latent-diffusion (CompVis) distributes the $f = 8$ VQGANs with 256 and 16,384 codes listed in the taming README.
• FSQ has an official JAX implementation in a Colab notebook in google-research, and the README points to an external PyTorch port.