Autoregressive Image Model

An autoregressive image model is a generative model that factors the probability of an image into a chain of conditional distributions, each predicting one pixel value, or one discrete token, from those already generated. Training maximizes the exact log-likelihood of the training images, one softmax cross-entropy term per position, and sampling generates the image one position after another. PixelRNN and PixelCNN (van den Oord, Kalchbrenner & Kavukcuoglu, 2016) applied it to large-scale modelling of natural images, Image GPT (Chen et al., 2020) carried it to transformers, and later models mostly apply it not to pixels but to the short token sequences of a learned image tokenizer such as the VQ-VAE, as in DALL·E’s text-to-image generation. Two later lines change the order in which the image is produced: masked-token models such as MaskGIT generate many tokens in parallel over a few refinement steps, and VAR generates a whole token map per step, from coarse to fine.

This article covers the architecture itself: the factorization, the networks that compute the conditionals without seeing the future, the move from pixels to tokens, the decoding orders, and what each costs at sampling time.

Core Idea

Any joint distribution can be written exactly as a product of conditionals once the variables are put in an order. An autoregressive model fixes the order (row by row for images, unless stated otherwise) and learns every conditional with one shared network. Two properties follow. The likelihood of any image is a product the network computes exactly, so the model is trained by maximum likelihood with no bound or adversary, and its test likelihood is a direct measure of fit. And the conditionals can all be evaluated in parallel during training, because the whole image is known, while sampling is sequential, because each value must be drawn before the next conditional can be computed (PixelRNN paper, §2.1; gated PixelCNN paper, §2).

The architectural problem is therefore to compute $p(x_i \mid x_{<i})$ for every $i$ in one forward pass while guaranteeing that output $i$ never depends on inputs $i, i+1, \dots$. PixelRNN does it with recurrent layers that scan the image, PixelCNN with masked convolutions, and transformers with a triangular attention mask.

Mathematical Formulation

For an image $x$ of $n \times n$ pixels taken in raster order as $x_1, \dots, x_{n^2}$, PixelRNN writes (van den Oord, Kalchbrenner & Kavukcuoglu, 2016, Eq. 1)

$$p(x) = \prod_{i=1}^{n^2} p(x_i \mid x_1, \dots, x_{i-1}),$$

and splits each colour pixel into its red, green and blue values, each conditioned on the earlier channels of the same pixel (Eq. 2):

$$p(x_i \mid x_{<i}) = p(x_{i,R} \mid x_{<i})\, p(x_{i,G} \mid x_{<i}, x_{i,R})\, p(x_{i,B} \mid x_{<i}, x_{i,R}, x_{i,G}).$$

Each factor is a 256-way softmax over the possible intensity values (§2.2). Training minimizes the cross-entropy summed over all $3n^2$ values, which is the negative log-likelihood of the image. Results are reported in bits per dimension: the negative log-likelihood in bits divided by the number of values, $32 \times 32 \times 3 = 3072$ for CIFAR-10, which the authors read as the bits per colour value that a compression scheme built on the model would need (§5.1). Read as a piecewise-uniform density, a 256-way distribution has the same log-likelihood on uniformly dequantized data as on the integers, so its numbers compare directly with those of continuous densities (§5.1; see normalizing flow for dequantization).

Image GPT states the same factorization for an arbitrary permutation $\pi$ of the positions, $p(x) = \prod_i p(x_{\pi_i} \mid x_{\pi_1}, \dots, x_{\pi_{i-1}})$, and uses the identity permutation, raster order (Chen et al., 2020, §2.1). Every order gives an exact factorization of the true distribution; what changes is how hard each conditional is to learn and which conditionals a given network can represent.

Over tokens the formula is the same, with $x_i$ an index into a codebook and a softmax over the codebook entries at each position. The two later decoding schemes change the unit of the chain. MaskGIT trains on a different objective, the negative log-likelihood of randomly masked tokens given the unmasked ones (Chang et al., 2022, Eq. 1), so in our reading it is not a chain-rule model in the strict sense; VAR’s authors likewise list it among masked-prediction models, apart from the autoregressive ones (Tian et al., 2024, §2.2). VAR keeps the chain rule but over $K$ token maps of increasing resolution (Tian et al., 2024, Eq. 6):

$$p(r_1, r_2, \dots, r_K) = \prod_{k=1}^{K} p(r_k \mid r_1, \dots, r_{k-1}),$$

where all tokens of map $r_k$ are predicted in parallel given the coarser maps.

Building Blocks

Masked convolutions. A PixelCNN is a stack of convolutions that keep the spatial resolution, with no pooling, whose kernels are multiplied by a fixed binary mask that zeroes every weight looking at the current pixel’s future: the rows below and the pixels to the right in the current row (PixelRNN paper, §3.4–3.5). Mask A, used only in the first layer, also zeroes the centre, so the first layer cannot see the value it predicts; mask B, used in the later layers, keeps the centre, because by then the centre position holds features computed only from the past. For colour images the feature maps are split into three groups for R, G and B, and the masks also control which colour groups may read which (§3.4, Fig. 2). The network outputs $N \times N \times 3 \times 256$ logits in one pass (gated PixelCNN paper, §2).

Two-dimensional LSTM layers. PixelRNN’s recurrent layers compute a whole row (Row LSTM) or a whole diagonal (Diagonal BiLSTM) of LSTM states at once with convolutions. The Row LSTM sees a roughly triangular context above the pixel, so it misses part of the available context; the Diagonal BiLSTM skews the image so that diagonals become columns, runs one LSTM pass in each direction, and shifts the right-to-left output down a row so that its field covers the full context without seeing the future (§3.1–3.2). Residual connections let the authors train up to 12 LSTM layers (§3.3).

Causal attention. A decoder-only transformer achieves the same constraint by adding $-\infty$ to the attention logits above the diagonal, the “standard upper triangular mask” of Image GPT (Chen et al., 2020, §2.2). Attention, unlike a convolution stack, reaches every earlier position in one layer, at a cost quadratic in sequence length.

Image tokenizers. In two-stage models, a quantized autoencoder such as the VQ-VAE or VQGAN first maps an image to a small grid of codebook indices, the autoregressive model is fitted to those grids, and a decoder turns sampled indices back into an image.

Pixel Models

PixelRNN and PixelCNN (2016)

Van den Oord, Kalchbrenner and Kavukcuoglu (ICML 2016) introduced the Row LSTM, the Diagonal BiLSTM and, as a simplified architecture sharing their components, the PixelCNN: a first $7 \times 7$ convolution with mask A, then $3 \times 3$ convolutions with mask B (15 layers for CIFAR-10), and two $1 \times 1$ layers before the softmax (§4, Table 1). Modelling pixels as discrete values was itself a choice against the continuous densities of earlier work: on CIFAR-10 validation, a Row LSTM with the 256-way softmax reached 3.06 bits per dimension against 3.22 for the same model with a mixture of conditional Gaussian scale mixtures (§5.3), and the softmax can be multimodal or skewed without any assumed shape (Fig. 6).

Their CIFAR-10 test results (Table 5, bits per dimension, training in brackets) order the three networks by the context each sees: PixelCNN 3.14 (3.08), Row LSTM 3.07 (3.00) and Diagonal BiLSTM 3.00 (2.93). The authors read this as evidence that a large receptive field matters (§5.6). On downsampled ImageNet, for which to their knowledge no likelihoods had been published, a 12-layer Row LSTM reached 3.86 bits per dimension at $32 \times 32$ and a 4-layer one 3.63 at $64 \times 64$ on validation (Table 6). They also built a multi-scale PixelRNN that first generates a subsampled image and then the full image conditioned on it, whose $64 \times 64$ samples looked more globally coherent at a similar likelihood (§3.6, Fig. 8).

The PixelCNN’s parallelism helps only during training and when scoring test images: generation is sequential for both kinds of network, since each sampled pixel is fed back as input (§3.5).

The blind spot and the gated PixelCNN

The follow-up paper (van den Oord et al., NIPS 2016) names a flaw in stacked masked convolutions. Each mask-B kernel looks at the row above (left, centre and right), the left neighbour in the current row and its own centre. Stacking such kernels grows the field upward and to the left without limit, but up and to the right only diagonally, so a wedge of past pixels above and to the right of the current pixel is never reached. This blind spot “can cover as much as a quarter of the potential receptive field” with $3 \times 3$ filters (§2.2, Fig. 1).

The gated PixelCNN removes it with two stacks (§2.2, Fig. 2). A vertical stack, whose kernels cover the rows above and need no mask within those rows, grows a rectangular field; a horizontal stack covers the current row to the left. After each layer the vertical stack’s output is fed into the horizontal stack, never the other way, because the vertical features at a position also cover pixels to its right in the current row, which the horizontal stack must not see. The authors note that the masked $n \times n$ and $n \times 1$ convolutions can be implemented as unmasked $\lceil n/2 \rceil \times n$ and $\lceil n/2 \rceil \times 1$ convolutions followed by a shift. The horizontal stack has a residual connection; one in the vertical stack did not help in their initial experiments.

The second change is the gated activation unit, which replaces the rectified linear units between masked convolutions (Eq. 2):

$$y = \tanh(W_{k,f} * x) \odot \sigma(W_{k,g} * x),$$

with $\sigma$ the sigmoid and $k$ the layer. The authors motivate it by the multiplicative units of the LSTM, which they suggest may help PixelRNN model more complex interactions (§2.1).

The gated PixelCNN reached 3.03 bits per dimension on the CIFAR-10 test set, against 3.14 for the PixelCNN and 3.00 for PixelRNN (Table 1), and on ImageNet beat PixelRNN at both sizes: 3.83 against 3.86 at $32 \times 32$ and 3.57 against 3.63 at $64 \times 64$ (Table 2, test NLL as labelled there). The authors attribute the ImageNet gain to underfitting, since larger models did better and the simpler PixelCNN scales better, and report performance similar to the Row LSTM’s in less than half the training time, 60 hours on 32 GPUs (§3.1).

The same paper made the model conditional: a vector $h$, such as a one-hot class or a face embedding, adds a learned bias $V^\top h$ inside both halves of every gate (Eq. 4), or, through a deconvolutional network, a position-dependent bias (Eq. 5). Class conditioning on ImageNet changed the log-likelihood little but improved the samples greatly (§3.2). As a decoder in an autoencoder, the conditional PixelCNN produced varied images that share the input’s content rather than exact reconstructions (§2.4, §3.4).

PixelCNN++ (Salimans et al., ICLR 2017) is a later variant whose paper, published on OpenReview, we could not read.

Image GPT (2020)

Chen et al. (ICML 2020) trained a GPT-2-style decoder-only transformer to predict pixels in raster order, using no knowledge of the 2D structure beyond the order (abstract, §2.2). Their goal was representation learning, not generation: features from the middle layers, read by a linear classifier, gave 96.3% accuracy on CIFAR-10 for iGPT-L (Table 1), and fine-tuning the whole network gave 99.0% (abstract). The best features lay in the middle of the network, not at its end, and lower validation loss went with better linear-probe accuracy (§4.1–4.2, Fig. 2–3).

Because dense attention is quadratic in sequence length, a $224 \times 224 \times 3$ image was out of reach. The authors resized images to $32^2$, $48^2$, $96^2$ or $192^2$ pixels, and at $32^2$ and $48^2$ shortened the sequence threefold by clustering (R, G, B) values with k-means into 512 colours, so that each token is one pixel (§3.2). For $96^2$ and $192^2$ inputs, a small VQ-VAE with a 4,096-entry codebook mapped the image to a $48 \times 48$ token grid, and they note that the colour clustering can itself be read as a VQ-VAE with identity encoder and decoder (§3.2). iGPT-L has 48 layers, width 1,536 and 1.4 billion parameters (§3.3), and was trained for a million iterations at batch size 128 (§3.4). The authors list the quadratic cost of dense attention, which forced low resolutions, as a significant limitation, and note that iGPT-L needs two to three times as many parameters as similarly performing ImageNet models (§6). The ViT paper article describes how Image GPT stood relative to the Vision Transformer.

Two-Stage Token Models

A pixel model spends its capacity on high-frequency detail. Ramesh et al. (2021, §2), citing Salimans et al. (2017), state the problem directly: likelihood objectives tend to favour short-range dependencies between pixels, so most of a pixel model’s capacity would go to detail rather than the low-frequency structure that makes objects recognizable. The two-stage recipe, which DALL·E’s authors describe as similar to that of VQ-VAE and VQ-VAE-2, compresses the image first and models the compressed tokens autoregressively.

• VQ-VAE priors. The VQ-VAE (van den Oord, Vinyals & Kavukcuoglu, 2017) fits a PixelCNN to the grids of codebook indices after the autoencoder is trained, and VQ-VAE-2 (Razavi, van den Oord & Vinyals, 2019) a gated PixelCNN with self-attention to a top grid and a conditional one to a larger bottom grid; the VQ-VAE article covers both.
• VQGAN transformers. Esser, Rombach and Ommer (CVPR 2021) replaced the PixelCNN prior with a transformer predicting the next codebook index, usually over $16 \times 16 = 256$ indices with a GPT-2-medium architecture (§4); the VQGAN article covers the model, its conditioning and its sliding-window sampling. For sampling cost, their Fig. 7 reports that, at a fixed sequence of $16 \times 16$ tokens on faces, a tokenizer downsampling by 16 made sampling about 280 times faster than modelling pixels directly, where one sample took 7,258 seconds on a Titan X.
• DALL·E. Ramesh et al. (ICML 2021) trained a discrete VAE that compresses a $256 \times 256$ image into $32 \times 32 = 1{,}024$ tokens from a vocabulary of 8,192, using a Gumbel-softmax relaxation instead of VQ-VAE’s straight-through nearest-neighbour step (§2, §2.1). A 12-billion-parameter sparse transformer with 64 attention layers then models up to 256 byte-pair-encoded caption tokens followed by the 1,024 image tokens as one stream; the text–text part of the attention mask is causal, and the image–image part uses row, column or convolutional masks (§2.2). The losses on text and image tokens are weighted 1/8 and 7/8, since the authors are primarily interested in the image (§2.2). Samples are reranked: of 512 candidates per caption, a pretrained contrastive image–text model keeps the best (§2.6). The text-to-image article reports DALL·E’s results and its place among later systems.
• Parti. As the text-to-image article describes it, Parti (Yu et al., 2022) treated text-to-image generation as translation, with an encoder–decoder transformer over ViT-VQGAN tokens scaled to 20 billion parameters; VAR’s related work likewise describes it as built on ViT-VQGAN and scaled to 20 billion parameters (Tian et al., 2024, §2.2). We did not read the Parti paper.

The two stages are trained separately. DALL·E’s authors report that in preliminary ImageNet experiments they could not improve on two-stage training by training all parts jointly (§2, footnote 3).

Changing the Order

Masked-token decoding: MaskGIT (2022)

Chang et al. (CVPR 2022) kept a VQGAN tokenizer, with images compressed by a factor of 16 in each direction to a grid of tokens from a 1,024-entry codebook (§4.1), and replaced the autoregressive transformer with a bidirectional one trained like BERT on masked visual token modelling (§3.1). For each training image a ratio $r \in [0, 1)$ is drawn, $\lceil \gamma(r) N \rceil$ of the $N$ tokens are replaced by a [MASK] token, and the loss is the negative log-likelihood of the masked tokens given the rest (Eq. 1). Because the mask ratio varies, the model learns to predict tokens from any amount of context, including none.

Generation starts from an all-masked grid and repeats four steps for $T$ iterations (§3.2): predict a distribution for every masked position in parallel; sample a token at each; score each sample by its predicted probability, its “confidence”; and keep the most confident samples, re-masking the rest so that the number still masked follows the schedule $\gamma(t/T)$. Of the schedules tried, concave ones, which commit few tokens early and many late, did best, and the cosine schedule was chosen (§3.3, §4.4, Table 3). The paper adds that in practice the tokens to mask are chosen randomly with temperature annealing to encourage diversity (§3.2). The official decoding code makes this precise: it ranks positions by the log-probability of the sampled token plus Gumbel noise scaled by a temperature times $(1 - t/T)$, so the choice is noisy early and nearly greedy at the end, and the cosine schedule is $\cos(\pi r / 2)$ (google-research/maskgit, parallel_decode.py and mask_schedule.py, as of November 2026); the paper sets the temperature to 4.5 (§4.1).

With 8 iterations on ImageNet $256 \times 256$ (12 at $512 \times 512$), against 256 sequential steps for a raster-order transformer over the same $16 \times 16$ grid (Fig. 2), MaskGIT ran 30 to 48 times faster than VQGAN’s decoding, the gap growing with resolution (§4.2, Fig. 4). More iterations were not always better: each schedule had a “sweet spot”, 8 to 12 iterations for the cosine, beyond which FID worsened (§4.4, Fig. 8). Because any set of tokens can be the starting mask, the same model inpaints, outpaints and regenerates the contents of a box under a new class without retraining, a task the authors call infeasible for autoregressive models because it violates their prediction order (§4.3). In the supplementary material they show failures: semantic and colour shifts in long outpaintings, and oversmoothing or artefacts on faces, text and symmetric objects (Appendix G).

Next-scale prediction: VAR (2024)

Tian et al. (NeurIPS 2024) argue that flattening a token grid into a raster sequence causes several problems: tokens produced by a convolutional encoder depend on each other in all directions, contradicting the one-directional assumption; the model cannot, for example, predict the top of an image from its bottom; flattening breaks spatial locality; and the cost is high (§3.1). Their answer is to make the unit of the chain a whole token map. A multi-scale tokenizer, VQGAN’s architecture with a modified quantization layer and one codebook of 4,096 entries shared by all scales, encodes an image into $K$ maps $r_1, \dots, r_K$ of increasing resolution, the last at the full $16\times$-downsampled grid (§3.2, §4). It is residual: each scale quantizes, at its resolution, what the coarser scales have not yet explained (Algorithm 1). A GPT-2-style transformer with adaptive layer normalization, conditioned on the class through its start token, then predicts each map from all coarser ones, with a block-wise causal mask in training and all tokens of a map generated in parallel (§3.2, §4, Fig. 4). Ten such steps generate a $256 \times 256$ image (Table 1).

The authors bound the cost of raster generation of $n^2$ tokens by $O(n^6)$, summing the $O(i^2)$ attention cost of step $i$ over $n^2$ steps, and that of next-scale prediction with geometrically growing resolutions by $O(n^4)$ (§3.1, Appendix D). The raster bound assumes that step $i$ computes attention scores among all $i$ tokens (Lemma D.1). In our reading, with cached keys and values, which the authors note can be used at VAR’s inference (§3.2), step $i$ needs only the new token’s scores against the $i$ cached ones, $O(i)$, and the raster total is $O(n^4)$, the same order as the paper’s bound for next-scale prediction; what remains different is the number of sequential steps, $n^2$ against $K$. On ImageNet $256 \times 256$ they report VAR about 20 times faster in wall-clock time than VQGAN and ViT-VQGAN (§5.1, Table 1).

VAR’s ablation (Table 3) starts from a raster-order transformer baseline at FID 18.65 and replaces its method with next-scale prediction, other settings unchanged, which gives 5.22 at 0.013 times the inference cost; adaptive layer normalization, top-$k$ sampling, classifier-free guidance with scale 1.5 and normalized queries and keys bring a 310-million-parameter model to 3.30, and scaling to 2 billion parameters to 1.73 (Table 1 gives 1.73 for the rejection-sampled model; see below). Over 12 model sizes from 18 million to 2 billion parameters, test loss fell as a power law of model size and of training compute (§5.2, Fig. 5–6). The authors also show zero-shot inpainting, outpainting and class-conditional editing by teacher-forcing the known tokens (Appendix B). They list text conditioning and video as future work (§7).

How Data Flows

Training (teacher forcing). The whole image or token grid goes in; the masks or causal attention ensure that output position $i$ depends only on inputs before $i$; one cross-entropy per position is summed. A two-stage model first runs the frozen tokenizer to get the index grid, then trains on it exactly as on pixels.

Sampling. Starting from an empty canvas, the model is run once per position in raster order: run the network, sample position $i$ from its softmax (often at reduced temperature or with top-$k$ truncation), write the value into the input, and run again. A two-stage model then decodes the completed index grid to pixels in one pass. MaskGIT runs the network $T$ times, each time filling a scheduled number of positions anywhere in the grid; VAR runs it $K$ times, each time producing a whole finer map.

Conditioning. A class or embedding enters as a bias in every layer (gated PixelCNN), as a start token or prefix sequence (VQGAN, DALL·E, VAR), or through adaptive normalization (VAR). Inpainting and completion follow from the factorization: fixing known values and sampling the rest works directly for positions after the known ones in the order (PixelRNN’s image completions, Fig. 9), and for arbitrary regions in MaskGIT and VAR.

Sampling Cost

The number of sequential network evaluations per image is the quantity that separates these models, and the published counts make it concrete:

Model	Unit generated per step	Sequential steps per image	Source
PixelCNN on $32 \times 32$ RGB	One colour value	3,072	PixelRNN paper, §2.1 (order)
VQGAN transformer, $16 \times 16$ tokens	One token	256	MaskGIT, Table 1
DALL·E	One image token (after the caption)	1,024	Ramesh et al., §2
VQ-VAE-2	One token, two levels	5,120 (estimated)	MaskGIT, Table 1
MaskGIT, $256 \times 256$	Scheduled set of tokens	8	MaskGIT, Table 1
VAR, $256 \times 256$	Whole token map	10	VAR, Table 1

The DALL·E and PixelCNN counts are our arithmetic from the stated sizes. Each step of a token model is one pass of a large transformer, and DALL·E’s reranking multiplies the cost by the number of candidates. In our reading, the pixel models’ cost also scales with the image: the PixelRNN paper models images of at most $64 \times 64$, and Image GPT kept to $48 \times 48$ token sequences.

Example: The Blind Spot, Measured by Input Gradients

A pixel influences an output exactly when the gradient of the output with respect to that pixel is nonzero, so the receptive field of a masked network can be read off one backward pass; the receptive-field article uses the same method on unmasked stacks. The PyTorch example below (PyTorch 2.14.1, NumPy 2.5.4) builds two untrained networks of 16 channels and depth $L$ with $3 \times 3$ kernels: a single stack of masked convolutions (mask A, then mask B), and the gated PixelCNN’s vertical and horizontal stacks, with the vertical output shifted down one row before it enters the horizontal stack and a residual connection in the horizontal stack. Both use the gated activation, so that the only difference is the stack structure; the original PixelCNN used rectified linear units, whose inactive units can zero a gradient and would blur the count. The input is a $64 \times 64$ synthetic image, a disk and a square, though the set of reachable pixels does not depend on the pixel values. For the output at pixel (48, 32), the script counts the pixels with nonzero gradient, compares them with the reach predicted by adding up the kernels’ offset sets layer by layer (a Minkowski sum, computed independently of the network code), and counts the blind spot: past pixels inside the stack’s own window (the $L$ rows above, $L$ columns either side, and the $L$ pixels to the left) that get no gradient.

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)
torch.set_num_threads(1)
torch.set_default_dtype(torch.float64)

rr, cc = np.mgrid[:64, :64]    # a 64 x 64 synthetic image: a bright disk and a grey square
img = 0.2 + 0.7 * ((rr - 40) ** 2 + (cc - 24) ** 2 < 12 ** 2) + 0.4 * ((abs(rr - 20) < 8) & (abs(cc - 44) < 8))
x0 = torch.tensor(img)[None, None]
TI, TJ, C = 48, 32, 16          # target pixel (row, column) and channels

def gate(a):                    # gated activation tanh(a) * sigmoid(b), Eq. 2
    f, g = a.chunk(2, dim=1)
    return torch.tanh(f) * torch.sigmoid(g)

class MaskedConv(nn.Conv2d):    # 3 x 3 convolution, mask A (no centre) or B
    def __init__(self, cin, cout, kind):
        super().__init__(cin, cout, 3, padding=1)
        m = torch.ones(3, 3); m[2, :] = 0; m[1, 2] = 0
        if kind == "A": m[1, 1] = 0
        self.register_buffer("mask", m)
    def forward(self, x):
        return F.conv2d(x, self.weight * self.mask, self.bias, padding=1)

class PixelCNN(nn.Module):      # one stack of masked convolutions
    def __init__(self, L):
        super().__init__()
        self.convs = nn.ModuleList([MaskedConv(1 if l == 0 else C, 2 * C, "A" if l == 0 else "B")
                                    for l in range(L)])
    def forward(self, x):
        for conv in self.convs:
            x = gate(conv(x))
        return x

class GatedPixelCNN(nn.Module): # vertical stack (rows above) + horizontal stack (row so far)
    def __init__(self, L):
        super().__init__()
        cin = lambda l: 1 if l == 0 else C
        self.v = nn.ModuleList([nn.Conv2d(cin(l), 2 * C, (2, 3)) for l in range(L)])  # rows i-1..i
        self.h = nn.ModuleList([nn.Conv2d(cin(l), 2 * C, (1, 1) if l == 0 else (1, 2))
                                for l in range(L)])                            # col j-1 / cols j-1..j
        self.v2h = nn.ModuleList([nn.Conv2d(2 * C, 2 * C, 1) for _ in range(L)])
        self.res = nn.ModuleList([nn.Conv2d(C, C, 1) for _ in range(L)])
    def forward(self, x):
        v = h = x
        for l in range(len(self.v)):
            vp = self.v[l](F.pad(v, (1, 1, 1, 0)))            # sees rows <= i
            down = F.pad(vp, (0, 0, 1, 0))[:, :, :-1]          # shift down: rows < i only
            hin = F.pad(h, (1, 0, 0, 0))
            if l == 0:                                         # mask A: column j-1 only
                hin = F.pad(h, (1, 0, 0, 0))[:, :, :, :-1]
            hp = self.h[l](hin) + self.v2h[l](down)
            hn = gate(hp)
            h = hn if l == 0 else h + self.res[l](hn)          # residual in horizontal stack
            v = gate(vp)
        return h

def grad_reach(net):
    x = x0.clone().requires_grad_(True)
    net(x)[0, :, TI, TJ].sum().backward()
    return (x.grad[0, 0] != 0).numpy()

def minkowski(a, b):
    return {(p + q, r + s) for (p, r) in a for (q, s) in b}

def analytic_pixelcnn(L):       # offsets (row, col) reachable from the target
    A = {(-1, -1), (-1, 0), (-1, 1), (0, -1)}
    R = A
    for _ in range(L - 1):
        R = minkowski(R, A | {(0, 0)})
    return R

def analytic_gated(L):
    Vk = {(dr, dc) for dr in (-1, 0) for dc in (-1, 0, 1)}
    V, H = {(0, 0)}, set()
    for l in range(L):
        V = minkowski(V, Vk)                                   # reach of the vertical conv output
        down = {(dr - 1, dc) for dr, dc in V}
        Hk = {(0, -1)} if l == 0 else {(0, -1), (0, 0)}
        H = (minkowski({(0, 0)}, Hk) if l == 0 else minkowski(H, Hk) | H) | down
    return H

def to_mask(offsets):
    m = np.zeros(img.shape, bool)
    for dr, dc in offsets:
        m[TI + dr, TJ + dc] = True
    return m

rows, cols = np.indices(img.shape)
future = (rows > TI) | ((rows == TI) & (cols >= TJ))
print(f"{'L':>3} {'window':>7} | {'PixelCNN reach':>14} {'blind':>6} {'frac':>6} | "
      f"{'gated reach':>11} {'blind':>6} | {'future grads':>12} {'= analytic':>10}")
for L in [1, 2, 3, 4, 6, 8, 15]:
    window = (rows >= TI - L) & (np.abs(cols - TJ) <= L) & ~future
    p, g = grad_reach(PixelCNN(L)), grad_reach(GatedPixelCNN(L))
    ok = (p == to_mask(analytic_pixelcnn(L))).all() and (g == to_mask(analytic_gated(L))).all()
    bp, bg = (window & ~p).sum(), (window & ~g).sum()
    print(f"{L:3d} {window.sum():7d} | {p.sum():14d} {bp:6d} {bp / window.sum():6.3f} | "
          f"{g.sum():11d} {bg:6d} | {(p & future).sum() + (g & future).sum():12d} {str(ok):>10}")

Output:

  L  window | PixelCNN reach  blind   frac | gated reach  blind | future grads = analytic
  1       4 |              4      0  0.000 |           7      0 |            0       True
  2      12 |             11      1  0.083 |          17      0 |            0       True
  3      24 |             21      3  0.125 |          31      0 |            0       True
  4      40 |             34      6  0.150 |          49      0 |            0       True
  6      84 |             69     15  0.179 |          97      0 |            0       True
  8     144 |            116     28  0.194 |         161      0 |            0       True
 15     480 |            375    105  0.219 |         511      0 |            0       True

Both networks are causal, with no gradient on the target pixel or any later one, and in every row the measured reach equals the reach predicted from the kernels. The single stack misses a growing wedge: at $d$ rows above the target it reaches only $d$ columns to the right, because the only rightward step any mask-B kernel offers also moves up a row, so $L - d$ pixels of each row in the window are blind, $L(L-1)/2$ in all. Since the window holds $2L^2 + 2L$ past pixels, the blind fraction is $(L-1)/(4(L+1))$, which reaches 0.219 at depth 15 (the layer count of the PixelRNN paper’s CIFAR-10 PixelCNN, whose first layer is $7 \times 7$) and tends to one quarter, the share the gated PixelCNN paper states. The two-stack network has no blind pixels; its reach exceeds the window by one row (511 against 480) because the vertical stack’s output is shifted down before it enters the horizontal stack. The example runs in about 2 seconds.

Example: Raster Order Against Masked-Token Decoding

How much does parallel decoding lose by sampling several tokens at once? With a learned model, the answer mixes the decoding procedure with the model’s errors. The NumPy example below (NumPy 2.3.5) removes the model’s errors by using the exact conditional distributions of a known token distribution. The “images” are $8 \times 8$ token grids, as a tokenizer would produce, over a vocabulary of 4 (0 for background, 1–3 for colours); each contains one filled axis-aligned rectangle in one colour, and all 3,888 such grids are equally likely. The oracle computes $p(x_i = v \mid \text{known tokens})$ for every position by averaging over the grids consistent with the known tokens. Raster sampling with these conditionals is exact. The parallel sampler follows MaskGIT’s loop: at each of $T$ steps it samples every masked position independently from its conditional and keeps a scheduled number of the samples, with the cosine schedule $\lceil N \cos(\pi t / 2T) \rceil$ masked after step $t$ (the paper writes the ceiling, the official code the floor) and at least one new token kept per step. Which samples to keep is printed both ways: confidence keeps those with the highest probability, the noise-free limit of MaskGIT’s rule, and random keeps a uniformly random subset, the limit of infinite noise. MaskGIT’s own rule, confidence plus annealed Gumbel noise at temperature 4.5, lies between the two and was not run. A sample is valid if it is exactly one of the 3,888 grids; area TV is the total variation distance between the rectangle areas of the valid samples and the exact area distribution, and mean area is their mean area. If a parallel step leaves no consistent grid, the oracle returns uniform conditionals; such a sample can never become valid, so this choice does not affect the measures.

import itertools
import numpy as np

S, V, N = 8, 4, 64              # 8 x 8 token grid, vocabulary of 4 (0 = background), 64 tokens
n_samples = 2000

# The support: every filled rectangle in every colour, all equally likely (3888 images).
imgs, areas = [], []
for (r0, r1), (c0, c1), colour in itertools.product(
        itertools.combinations(range(S + 1), 2), itertools.combinations(range(S + 1), 2), range(1, V)):
    im = np.zeros((S, S), int); im[r0:r1, c0:c1] = colour
    imgs.append(im.ravel()); areas.append((r1 - r0) * (c1 - c0))
support, areas = np.array(imgs), np.array(areas)
onehot = np.eye(V, dtype=np.float32)[support].reshape(len(support), N * V)
index = {im.tobytes(): k for k, im in enumerate(support)}
exact_area = np.bincount(areas, minlength=N + 1) / len(areas)

def conditionals(x, known):
    """Exact p(x_i = v | known tokens) for every position: the oracle model."""
    obs = np.eye(V, dtype=np.float32)[x] * known[..., None]                  # (n, N, V)
    hits = obs.reshape(len(x), -1) @ onehot.T                                # matching known tokens
    w = (hits == known.sum(1, keepdims=True)).astype(np.float32)            # consistent images
    p = (w @ onehot).reshape(len(x), N, V)
    total = w.sum(1)[:, None, None]
    # no consistent image left (only after an invalid parallel draw): uniform fallback
    return np.where(total > 0, p / np.maximum(total, 1), 1 / V)

def sample_from(p, rng):
    u = rng.random(p.shape[:2])[..., None]
    return np.minimum((p.cumsum(-1) < u).sum(-1), V - 1)

def raster(rng):
    x, known = np.zeros((n_samples, N), int), np.zeros((n_samples, N), bool)
    for i in range(N):                       # one position per network evaluation
        x[:, i] = sample_from(conditionals(x, known), rng)[:, i]
        known[:, i] = True
    return x

def parallel(T, rule, rng):
    x, known = np.zeros((n_samples, N), int), np.zeros((n_samples, N), bool)
    for t in range(1, T + 1):
        p = conditionals(x, known)                           # one network evaluation
        draw = sample_from(p, rng)
        conf = np.take_along_axis(p, draw[..., None], -1)[..., 0]
        if rule == "random":
            conf = rng.random(conf.shape)
        conf = conf + 1e-9 * rng.random(conf.shape)          # break ties at random
        conf[known] = np.inf                                 # fixed tokens stay fixed
        n_masked = int(np.ceil(N * np.cos(np.pi / 2 * t / T)))   # cosine schedule
        n_masked = min(n_masked, N - known[0].sum() - 1) if t < T else 0
        keep = np.argsort(-conf, axis=1)[:, :N - n_masked]
        rows = np.arange(n_samples)[:, None]
        x[rows, keep] = np.where(known[rows, keep], x[rows, keep], draw[rows, keep])
        known[rows, keep] = True
    return x

def score(x):
    k = np.array([index.get(im.tobytes(), -1) for im in x])
    valid = k >= 0
    if not valid.any():
        return 0.0, 0, "-", "-"
    hist = np.bincount(areas[k[valid]], minlength=N + 1) / valid.sum()
    return (100 * valid.mean(), valid.sum(), f"{0.5 * np.abs(hist - exact_area).sum():.3f}",
            f"{areas[k[valid]].mean():.2f}")

rng = np.random.default_rng(0)
print(f"support: {len(support)} images, mean rectangle area {areas.mean():.2f} tokens")
print(f"{'sampler':28s}{'evaluations':>12s}{'valid %':>9s}{'n valid':>9s}{'area TV':>9s}{'mean area':>10s}")
rows_out = [("exact samples (reference)", 0, support[rng.integers(0, len(support), n_samples)]),
            ("raster, exact conditionals", N, raster(rng))]
for rule in ["confidence", "random"]:
    for T in [1, 2, 4, 8, 16, 64]:
        rows_out.append((f"parallel, {rule}, T={T}", T, parallel(T, rule, rng)))
for name, evals, x in rows_out:
    v, nv, tv, ma = score(x)
    print(f"{name:28s}{evals:12d}{v:9.1f}{nv:9d}{tv:>9s}{ma:>10s}")

Output:

support: 3888 images, mean rectangle area 11.11 tokens
sampler                      evaluations  valid %  n valid  area TV mean area
exact samples (reference)              0    100.0     2000    0.039     11.09
raster, exact conditionals            64    100.0     2000    0.041     11.03
parallel, confidence, T=1              1      0.0        0        -         -
parallel, confidence, T=2              2      0.5       10    0.864      1.20
parallel, confidence, T=4              4      5.2      105    0.864      1.17
parallel, confidence, T=8              8     14.4      289    0.850      1.18
parallel, confidence, T=16            16     29.2      584    0.854      1.18
parallel, confidence, T=64            64    100.0     2000    0.851      1.28
parallel, random, T=1                  1      0.1        2    0.864      1.50
parallel, random, T=2                  2      5.5      111    0.524      3.15
parallel, random, T=4                  4     23.2      465    0.340      5.04
parallel, random, T=8                  8     47.4      948    0.178      7.91
parallel, random, T=16                16     75.2     1504    0.075     10.00
parallel, random, T=64                64    100.0     2000    0.043     11.33

Raster decoding with exact conditionals reproduces the distribution, its area TV matching that of 2,000 exact samples, but it costs 64 evaluations. Parallel decoding fails for a reason the model cannot fix: tokens chosen in the same step are drawn independently from their marginals, while the true distribution ties them together (two positions may each be plausibly red or blue, but not one of each). With one step nearly every sample is invalid; with random choice of which samples to keep, validity rises steadily with $T$, and at $T = 64$, one token per step, the procedure is exact sampling in a random order, as valid and as well distributed as raster order. Even when they are valid, few-step samples are biased toward small rectangles (mean area 3 to 8 tokens at 2 to 8 steps, against 11.1), which in our reading follows from the same independence: a larger rectangle needs more jointly consistent draws.

The noise-free confidence rule does worse on both counts here. Its validity is lower than the random rule’s at every $T$ below 64, and its valid samples are almost all tiny rectangles, with a mean area near 1.2 tokens and an area TV near 0.85 even at $T = 64$, where it fixes one token per step. In our reading the reason is that background is the most probable token almost everywhere, so keeping the most confident samples commits background first and leaves room only for small rectangles; ranking by confidence is not sampling from the model’s distribution, even one token at a time. This is a property of the deterministic limit on this distribution, not a measurement of MaskGIT, which adds annealed random noise to its confidence scores, motivated in the paper by diversity, and whose authors found that a few steps with the cosine schedule gave their best FID on ImageNet. What the oracle isolates is the trade-off the method manages: fewer steps mean more tokens drawn without seeing each other, and the rule for choosing which samples to keep shapes the distribution. Two changes followed a pilot run, before the first run and with the editor’s approval: the area TV, undefined when no sample is valid, now prints “-”, and the mean-area column was added after the pilot, to show the direction of the bias. The script runs in about 30 to 40 seconds.

Training Considerations

• Overfitting and scale. The PixelRNN authors trained CIFAR-10 models with batches of 16 images, which seemed to regularize them, and found no sign of overfitting on ImageNet, where larger and deeper models kept improving (§5.2, §5.7). Gated PixelCNN’s authors read their ImageNet results as underfitting (§3.1); DALL·E’s 12-billion-parameter transformer showed no overfitting on about 606,000 held-out images (§2.2).
• Numerical stability at scale. Training DALL·E in 16-bit precision past one billion parameters without divergence was, its authors write, the most challenging part of the project; they attribute the instability to underflow of 16-bit gradients and scale the gradients of each residual block separately (§2.4, Fig. 4).
• Choosing what to model. Image GPT’s sequence length forced low resolutions and colour clustering (§3.2); the two-stage models move the same choice into the tokenizer, whose compression sets both the sequence length and the detail a sample can have. VAR’s authors keep VQGAN’s tokenizer architecture and see better tokenizers as a separate, promising route (§7).
• Sampling temperature and reranking. Reported results often depend on sampling choices: VQGAN’s face samples use top-$k$ sampling with $k = 100$ (Fig. 7), DALL·E reranks 512 samples, and MaskGIT’s ImageNet table includes rows for VQGAN and MaskGIT with classifier-based rejection sampling at an acceptance rate of 0.05 (MaskGIT, §4.2, Table 1).

Results as Published

Likelihood results compare pixel models directly, since they score the same pixels:

Model	CIFAR-10 test (bits/dim)	ImageNet $32 \times 32$	ImageNet $64 \times 64$
PixelCNN	3.14	—	—
Row LSTM	3.07	3.86 (validation)	3.63 (validation, 4 layers)
Diagonal BiLSTM	3.00	—	—
Gated PixelCNN	3.03	3.83	3.57

CIFAR-10 values are from the PixelRNN paper’s Table 5 and the gated PixelCNN paper’s Table 1; ImageNet values from the PixelRNN paper’s Table 6, labelled validation there, and the gated PixelCNN paper’s Table 2, which labels the same PixelRNN numbers as test.

Sample quality of token models is compared by FID and Inception Score on class-conditional ImageNet $256 \times 256$. The selection below is from VAR’s Table 1, which collects the other models’ numbers from their papers and from MaskGIT; “re” marks rejection sampling.

Model	Family	FID	IS	Parameters	Steps
VQ-VAE-2	Raster, two-level	31.11	~45	13.5B	5,120
VQGAN (MaskGIT’s reimplementation)	Raster	18.65	80.4	227M	256
VQGAN	Raster	15.78	74.3	1.4B	256
VQGAN-re	Raster	5.20	280.3	1.4B	256
ViT-VQGAN	Raster	4.17	175.1	1.7B	1,024
MaskGIT	Masked tokens	6.18	182.1	227M	8
VAR-d16	Next scale	3.30	274.4	310M	10
VAR-d30	Next scale	1.92	323.1	2.0B	10
VAR-d30-re	Next scale	1.73	350.2	2.0B	10
DiT-XL/2 (diffusion, for reference)	—	2.27	278.2	675M	250

Three inconsistencies in the sources: MaskGIT’s Table 1 prints VQGAN’s Inception Score as 78.3 where VAR’s prints 74.3; MaskGIT’s Table 1 gives VQGAN at an acceptance rate of 0.05 an FID of 5.88 and IS of 304.8, where VAR’s VQGAN-re row gives 5.20 and 280.3 without stating the rate; and VAR’s headline FID of 1.73 (abstract) is the rejection-sampled VAR-d30-re in its Table 1, while its ablation Table 3 lists 1.73 for the 2-billion-parameter model with top-$k$ sampling and guidance and does not mention rejection sampling. VAR’s comparison with the Diffusion Transformer uses these tables; the authors also report that DiT-XL/2 needs 45 times VAR’s wall-clock time (§5.1). The real ImageNet validation images score an FID of 1.78 in the same table, which VAR’s authors present as a reference lower bound (Fig. 3).

Use in Computer Vision

• Image generation. Class-conditional and unconditional image generation, from PixelCNN’s ImageNet samples to MaskGIT and VAR.
• Text-to-image generation. DALL·E and Parti, described in the text-to-image generation article.
• Video. Token models extend to video by tokenizing space and time; the video generation article covers VideoGPT and its successors.
• Priors and decoders. Autoregressive priors over discrete latents (the VQ-VAE family) and PixelCNN decoders inside autoencoders (gated PixelCNN paper, §2.4).
• Completion and editing. Image completion (PixelRNN, Fig. 9), inpainting, outpainting and class-conditional box editing (MaskGIT, §4.3; VAR, Appendix B); see image inpainting.
• Representation learning. Image GPT’s linear probes showed that next-pixel prediction learns features useful for classification (Chen et al., 2020); masked prediction became central to self-supervised learning, for example in the masked autoencoder.
• Compression. An exact likelihood is a code length: the PixelRNN authors read bits per dimension as the cost of compressing each colour value with the model (§5.1).

Limitations

• Sequential sampling. Raster-order models need one network evaluation per pixel or token; this motivated two-stage tokenization, masked decoding and next-scale prediction alike.
• A fixed order. The raster order puts information above and to the left of each position and nothing below it, which is why a raster model cannot inpaint an arbitrary region without help, and why MaskGIT’s authors call class-conditional box editing infeasible for autoregressive models (§4.3).
• Limited or costly context. Convolution stacks see a bounded, and with plain masks partly blind, neighbourhood; attention sees everything at quadratic cost, which limited Image GPT to low resolutions (§6).
• Likelihood is not sample quality. Pixel models with excellent likelihoods produced samples far from GANs and diffusion models in FID; the generative model article explains why the two measures diverge.
• Detail lost in tokenization. Two-stage models can generate only what their tokenizer reconstructs; DALL·E’s authors note that fine details such as fur texture, writing and thin lines are sometimes lost or distorted by its discrete VAE (Fig. 1).
• Parallel decoding trades exactness for speed. Tokens generated in the same step do not see each other, as the second example shows on an exact model; MaskGIT’s authors found that more steps can also hurt, past a “sweet spot” (§4.4).

Important Models

Model	Year	Unit and order	Contribution
PixelRNN (Row LSTM, Diagonal BiLSTM)	2016	Pixels, raster	Two-dimensional LSTM layers; discrete 256-way softmax; first ImageNet likelihoods
PixelCNN	2016	Pixels, raster	Masked convolutions; parallel training
Gated PixelCNN	2016	Pixels, raster	Vertical and horizontal stacks remove the blind spot; gated units; conditioning
VQ-VAE and VQ-VAE-2 priors	2017, 2019	Codebook indices, raster	Two-stage generation in a compressed latent
Image GPT	2020	Pixels (colour clusters) or VQ-VAE tokens, raster	Decoder-only transformer; representations from generative pre-training
VQGAN transformer	2021	Tokens, raster	Transformer prior over a perceptually trained tokenizer; sliding-window sampling
DALL·E	2021	Text then image tokens, raster	12-billion-parameter text-to-image transformer
MaskGIT	2022	Tokens, parallel masked decoding	Bidirectional transformer; 8 to 12 decoding steps
Parti	2022	Tokens, encoder–decoder	20-billion-parameter text-to-image transformer (paper not read)
VAR	2024	Token maps, coarse to fine	Next-scale prediction; 10 steps; scaling laws