Swin Transformer

The Swin Transformer is a vision transformer built to serve as a general-purpose backbone for image classification, object detection and semantic segmentation. Ze Liu, Yutong Lin, Yue Cao, Han Hu and colleagues at Microsoft Research Asia introduced it at ICCV 2021. It changes two things in ViT’s design. Instead of one grid of $16 \times 16$ patches attending globally, it starts from $4 \times 4$ patches and merges neighbouring tokens between four stages, producing feature maps at 1/4, 1/8, 1/16 and 1/32 of the input resolution, as a convolutional network does. And instead of global self-attention, each block attends inside non-overlapping windows of $7 \times 7$ tokens, with the window grid displaced by $\lfloor M/2 \rfloor$ tokens, about half a window, in every second block so that information crosses window borders. The name comes from Shifted windows. Swin Transformer V2 (CVPR 2022, a different author list led by the same first author) changed the normalization, the attention similarity and the position bias to train models of up to 3 billion parameters at high resolution.

Problem Addressed

ViT and DeiT showed that a plain transformer on image patches could classify ImageNet well, but Liu et al. (2021, §1–§2) argued that the architecture did not suit a general backbone, for two reasons they trace to differences between images and text. Visual entities vary greatly in scale, while ViT’s tokens all have one fixed scale and its output is a single low-resolution map, which detectors built on feature pyramids cannot use directly. And the cost of global self-attention is quadratic in the number of tokens, which makes dense prediction on high-resolution images intractable. Earlier attempts to bring attention into vision backbones computed it over a sliding local window around each pixel; the authors note that these had slightly better accuracy for their FLOPs than ResNet, but much higher real latency because each query has its own key set, which suits hardware poorly.

Swin’s answer is locality at a fixed window size, which makes cost linear in image size, plus a hierarchy that gives the multi-scale maps a detector or segmenter expects. Shifting the windows between blocks restores the cross-window connections that a fixed partition would lose, while every query in a window still shares one key set.

Architecture

The network has a patch embedding, four stages of Swin Transformer blocks with patch merging between them, and a head (Liu et al., 2021, §3.1, Figure 3). An $H \times W$ RGB image is split into $4 \times 4$ patches whose 48 raw values are linearly embedded to width $C$; this is the patch embedding of ViT with a smaller patch. The official code implements it as a $4 \times 4$ convolution with stride 4, followed by a LayerNorm (patch_norm=True). Swin adds no absolute position embedding by default (ape=False): position enters through a relative bias inside attention.

Stages and patch merging

Stage 1 applies Swin blocks to the $\frac{H}{4} \times \frac{W}{4}$ tokens. Each later stage begins with patch merging, which concatenates the features of each $2 \times 2$ group of neighbouring tokens into a $4C$-dimensional vector and projects it linearly to $2C$, halving the resolution and doubling the width. Stages 2, 3 and 4 therefore work at $\frac{H}{8}$, $\frac{H}{16}$ and $\frac{H}{32}$ with widths $2C$, $4C$ and $8C$ (§3.1). In the official code (PatchMerging), a LayerNorm over the $4C$ values precedes the projection, which has no bias. For classification, a final LayerNorm, global average pooling and a linear layer form the head. The patch embedding article treats merging as a second, $2 \times 2$ stride-2 patch embedding applied to the token map.

Swin Transformer block

A Swin block is a pre-norm transformer block whose multi-head self-attention (MSA) is replaced by window-based attention: LayerNorm, (shifted) window MSA, residual addition, then LayerNorm, a two-layer MLP with GELU and expansion 4, and a second residual addition (§3.1). Blocks come in pairs; the first uses regular windows (W-MSA) and the second shifted windows (SW-MSA):

$$\begin{aligned} \hat{z}^{l} &= \text{W-MSA}\big(\text{LN}(z^{l-1})\big) + z^{l-1}, & z^{l} &= \text{MLP}\big(\text{LN}(\hat{z}^{l})\big) + \hat{z}^{l},\\ \hat{z}^{l+1} &= \text{SW-MSA}\big(\text{LN}(z^{l})\big) + z^{l}, & z^{l+1} &= \text{MLP}\big(\text{LN}(\hat{z}^{l+1})\big) + \hat{z}^{l+1}. \end{aligned}$$

This is Eq. (3) of the paper. Swin V1 inherited the pre-norm placement from ViT; the layer normalization article compares it with post-norm and with Swin V2’s residual post-norm.

Window attention

The feature map is divided evenly into non-overlapping windows of $M \times M$ tokens ($M = 7$ by default), and self-attention is computed independently inside each. For an $h \times w$ token grid of width $C$, the authors give the cost of global and window MSA as

$$\Omega(\text{MSA}) = 4hwC^2 + 2(hw)^2C, \qquad \Omega(\text{W-MSA}) = 4hwC^2 + 2M^2hwC,$$

leaving out the softmax (§3.2, Eqs. 1–2). The first term covers the query, key, value and output projections and the second the products $QK^\top$ and $AV$; with $M$ fixed, window attention is linear in the number of tokens. The self-attention article works through these formulas for a $224 \times 224$ input. The official code’s flops() methods count exactly these terms, in multiply-adds.

Shifted windows and the cyclic shift

A fixed partition never lets one window exchange information with its neighbours. Swin therefore alternates two partitions: the regular one, starting at the top-left token, and one displaced by $(\lfloor M/2 \rfloor, \lfloor M/2 \rfloor)$ tokens (§3.2, Figure 2). The shifted windows straddle the borders of the previous layer’s windows and so connect them.

Shifting creates more windows, from $\lceil h/M \rceil \times \lceil w/M \rceil$ to $(\lceil h/M \rceil + 1) \times (\lceil w/M \rceil + 1)$, and the ones along the border are smaller than $M \times M$. The naive remedy pads every small window to $M \times M$ and masks the padding, which the authors note costs 2.25 times as much when the regular partition has only $2 \times 2$ windows. Their batched alternative (Figure 4) rolls the whole feature map cyclically by $\lfloor M/2 \rfloor$ tokens toward the top left, partitions it into the regular number of windows, and masks attention so that a token attends only to tokens of its own sub-window; the result is rolled back. A batched window at the bottom or right edge then contains up to four pieces that are not adjacent in the image (for example the bottom-right corner and the top-left strip), and the mask keeps them apart. In the official code the mask is an additive term of $-100$ rather than $-\infty$, which in practice removes the masked weights (the example below checks both). When a stage’s feature map is no larger than the window, the code shrinks the window to the map and turns the shift off; for Swin-T at $224 \times 224$ this applies to stage 4, whose $7 \times 7$ map is a single window.

Relative position bias

Each head adds a learned bias to the attention scores,

$$\operatorname{Attention}(Q, K, V) = \operatorname{SoftMax}\big(QK^\top/\sqrt{d} + B\big)V,$$

where $B \in \mathbb{R}^{M^2 \times M^2}$ depends only on the relative offset of query and key within the window. Because offsets along each axis lie in $[-M+1, M-1]$, the bias is read from a smaller table $\hat{B} \in \mathbb{R}^{(2M-1) \times (2M-1)}$ (§3.2, Eq. 4); the code stores one table per head and indexes it with a precomputed relative_position_index. The positional encoding article places this bias among absolute and relative encodings and gives the paper’s ablation of it. For fine-tuning with a different window size, the pre-trained table is resized by bicubic interpolation (§3.2).

The cyclic shift does not disturb the bias. Within one sub-window the roll preserves offsets between tokens, and offsets between different sub-windows are masked anyway, so indexing the bias by in-window position after the roll gives each surviving pair its true offset (our reasoning, checked numerically in the example).

Configurations

All variants use $M = 7$, a query dimension of 32 per head and MLP expansion 4; they differ in the stage-1 width $C$ and the number of blocks per stage (§3.3):

Model	$C$	Blocks per stage	Heads per stage	Parameters	FLOPs at $224^2$	Close in size to
Swin-T	96	2, 2, 6, 2	3, 6, 12, 24	29M	4.5G	ResNet-50, DeiT-S
Swin-S	96	2, 2, 18, 2	3, 6, 12, 24	50M	8.7G	ResNet-101
Swin-B	128	2, 2, 18, 2	4, 8, 16, 32	88M	15.4G	ViT-B, DeiT-B
Swin-L	192	2, 2, 18, 2	6, 12, 24, 48	197M	103.9G at $384^2$

Parameters and FLOPs are those of the paper’s Table 1; heads per stage follow from the 32-dimensional heads and come from the official configuration files. The models fine-tuned at $384 \times 384$ use windows of $12 \times 12$ in the official configurations.

Training

The paper trains classification models in two settings (§4.1). On ImageNet-1K alone, it follows DeiT’s recipe: AdamW for 300 epochs with a cosine schedule and 20 epochs of linear warm-up, batch size 1024, learning rate 0.001 and weight decay 0.05, with most of DeiT’s augmentation and regularization but without repeated augmentation and EMA, which did not help. The authors point out that DeiT found repeated augmentation crucial for stabilizing ViT. The official configurations set stochastic depth rates of 0.2, 0.3 and 0.5 for Swin-T, -S and -B. For ImageNet-22K pre-training (14.2 million images, 22K classes), AdamW runs 90 epochs with 5 of warm-up, batch size 4096, learning rate 0.001 and weight decay 0.01; fine-tuning on ImageNet-1K then takes 30 epochs at batch size 1024 with a constant learning rate of $10^{-5}$ and weight decay $10^{-8}$. The objective is the usual cross-entropy over classes.

For COCO ablations, Swin replaces the backbone in four detectors in mmdetection: Cascade Mask R-CNN, ATSS, RepPoints v2 and Sparse R-CNN, all with multi-scale training (shorter side 480 to 800, longer side at most 1333), AdamW with learning rate $10^{-4}$, weight decay 0.05, batch size 16, and a 36-epoch (3×) schedule. The system-level results use an improved Hybrid Task Cascade (HTC++) with instaboost, stronger multi-scale training, a 72-epoch schedule, soft-NMS and ImageNet-22K pre-training (§4.2). ADE20K results use UperNet in mmsegmentation (§4.3).

Inference

A Swin backbone returns four maps at strides 4, 8, 16 and 32. The official classification code fixes the input size, because each block precomputes its shift mask for one resolution; where a feature map is not divisible by the window, the paper pads it at the bottom right (§3.2, footnote 4). The authors’ speed measurements (Table 5, V100 GPU) show the cyclic implementation of shifted windows running at 755, 437 and 278 images per second for Swin-T, -S and -B, against 670, 371 and 236 with naive padding and 770, 444 and 280 with no shift at all. Variants built on sliding-window attention were 4.1, 4.0 and 3.6 times slower than Swin with a naive implementation and 1.5 times slower with a custom kernel. The authors note that their model uses built-in PyTorch functions that are not all well optimized, while ResNe(X)t benefits from cuDNN (§4.2).

Performance

Results are as printed in Liu et al. (2021), on ImageNet-1K val (single-crop top-1), COCO and ADE20K.

• ImageNet-1K, trained from scratch (Table 1a): Swin-T 81.3%, Swin-S 83.0% and Swin-B 83.5% at $224^2$, and Swin-B 84.5% at $384^2$ (the text of §4.1 gives 83.3% for Swin-B at $224^2$, while the table prints 83.5%), against 79.8% for DeiT-S and 81.8% / 83.1% for DeiT-B at $224^2$ / $384^2$. RegNetY-4G, -8G and -16G reached 80.0%, 81.7% and 82.9%.
• ImageNet-22K pre-training (Table 1b): Swin-B 85.2% at $224^2$ and 86.4% at $384^2$, and Swin-L 87.3% at $384^2$, against 84.0% for ViT-B/16 at $384^2$.
• COCO, same detector, ResNet-50 against Swin-T (Table 2a, val2017): box AP rose from 46.3 to 50.5 with Cascade Mask R-CNN, and by 3.4 to 4.2 points across the four frameworks. Swin-T also beat DeiT-S, given a deconvolution-built pyramid, by 2.5 box AP and 2.3 mask AP at 15.3 against 10.4 FPS (Table 2b). With Cascade Mask R-CNN, Swin-B reached 51.9 box AP and 45.0 mask AP, 3.6 and 3.3 above ResNeXt-101 64x4d.
• COCO system level (Table 2c): Swin-L with HTC++, ImageNet-22K pre-training and multi-scale testing reached 58.7 box AP and 51.1 mask AP on test-dev (58.0 and 50.4 on minival).
• ADE20K val (Table 3, UperNet, mIoU): Swin-T 46.1, Swin-S 49.3 against 44.0 for DeiT-S, and Swin-L (ImageNet-22K) 53.5, against 50.3 for SETR. The main paper’s table does not say whether these are single- or multi-scale; the official README lists Swin-L’s 53.5 (53.53) as its multi-scale result with flipping, against 52.05 single-scale, as of November 2026.

The ablations (Table 4, Swin-T) are the most direct evidence for the design. Without shifting, ImageNet top-1 fell from 81.3% to 80.2%, COCO box AP from 50.5 to 47.7 and ADE20K mIoU from 46.1 to 43.3. Replacing the relative bias by no position encoding gave 80.1%, 49.2 and 43.8; ViT-style absolute embeddings gave 80.5%, 49.0 and 43.2. Sliding-window attention was about as accurate as shifted windows (81.4%, 50.2 box AP, 45.8 mIoU; Table 6) while being slower, and Performer attention reached 79.0%.

Strengths

• Multi-scale maps at linear cost. The four strides match those of ResNet-style backbones, so Swin drops into FPN, Cascade Mask R-CNN, UperNet and similar heads unchanged, and cost grows with image area rather than its square.
• Hardware-friendly locality. All queries in a window share one key set, and the cyclic shift keeps the number of batched windows constant, so the shift costs little time (Table 5).
• Global reach through depth. Shifting and merging together let information cross the whole image within the first three stages (see the example).
• Transfer to dense tasks. The same backbone set the paper’s best COCO and ADE20K results, and Swin backbones were later used in detection, segmentation, restoration and depth estimation (see Variants).

Limitations

• Fixed windows and resolution coupling. The window size must divide the feature map or the map is padded, the masks are precomputed per resolution in the classification code, and transferring to larger windows needs the bias table resized. Swin V2’s first table shows ImageNet accuracy falling from 81.7% to 68.7% when a Swin-T trained at $256^2$ with $8 \times 8$ windows is tested with $24 \times 24$ windows at $768^2$ without fine-tuning.
• Slow spread of information within a stage. Inside one stage, reach grows by about one window width per block, so a large stage-1 map is far from global after its two blocks (see the example).
• Training instability at scale. The Swin V2 authors found that activations grew with depth in pre-norm Swin models and that a 658-million-parameter Swin V1 could not complete self-supervised pre-training (Liu et al., 2022, §3.1).
• Implementation complexity. Rolls, window partitions, masks and bias indexing make the block harder to implement than a convolution or global attention. The ConvNeXt authors (Liu, Mao, Wu, Feichtenhofer, Darrell and Xie, 2022, §1) make this point: cyclic shifting can make sliding-window-like attention faster, but at the cost of a more complicated design.

Historical Importance

Swin was published at ICCV 2021, the year in which ViT (ICLR) and DeiT (ICML) were published after showing that transformers could classify images, and it showed that a transformer could also be the backbone of state-of-the-art detection and segmentation systems: its COCO and ADE20K results were 2.7 box AP, 2.6 mask AP and 3.2 mIoU above the previous best entries the authors compared with. In our reading, it moved the question from whether attention could replace convolution to which of the convolutional priors (locality, hierarchy, translation-invariant relative position) a transformer should keep; the Swin V2 paper itself describes Swin as introducing hierarchy, locality and translation invariance into the transformer encoder (§3.1). ConvNeXt’s authors call it a milestone that showed for the first time that transformers could serve as a generic vision backbone, and they “modernized” a ResNet-50 step by step toward the design of Swin-T to find which differences mattered (Liu, Mao, Wu, Feichtenhofer, Darrell and Xie, 2022, §1).

Variants

Swin Transformer V2 (2022)

Swin Transformer V2 (Liu, Hu, Lin, Yao, Xie, Wei, Ning, Cao, Zhang, Dong, Wei and Guo, CVPR 2022) scales the model to 3 billion parameters and to images of up to $1536 \times 1536$, with three changes (§3.2–§3.3, Figure 1).

• Residual post-norm. Each LayerNorm moves from the start of the residual branch to its end, so that the branch output is normalized before it is added back and the main branch’s amplitude does not accumulate with depth. The layer normalization article covers this change and its ablation. The largest models also add a LayerNorm on the main branch every 6 blocks. In the official code the post-norms are initialized with zero scale and bias.
• Scaled cosine attention. The logit of a query–key pair becomes $\operatorname{Sim}(q_i, k_j) = \cos(q_i, k_j)/\tau + B_{ij}$, with a learnable $\tau$ per head and layer, kept above 0.01 (Eq. 2). The authors found that dot-product attention maps in large models, especially with res-post-norm, were often dominated by a few pixel pairs; the cosine is bounded. In the code, $1/\tau$ is a learned logit scale initialized at 10 and clamped at 100.
• Log-spaced continuous position bias (Log-CPB). A small meta-network $G$, by default a two-layer MLP with ReLU, generates the bias from relative coordinates, $B(\Delta x, \Delta y) = G(\Delta x, \Delta y)$, and the coordinates are log-spaced, $\widehat{\Delta x} = \operatorname{sign}(\Delta x)\log(1 + |\Delta x|)$ (Eqs. 3–4). Moving from $8 \times 8$ to $16 \times 16$ windows then extrapolates over a much smaller fraction of the coordinate range, 0.33 of the original instead of 1.14 times it, by the paper’s count. The official code first divides the offsets by the window size minus one (the pre-training window’s, when one is given) and multiplies them by 8, so that the pre-training window’s offsets span $[-8, 8]$, then applies $\operatorname{sign}(x)\log_2(1 + |x|)/\log_2 8$, and passes the MLP’s output through $16\,\sigma(\cdot)$.

The configurations keep V1’s stages: SwinV2-T, -S, -B and -L with $C$ = 96, 96, 128 and 192, plus SwinV2-H ($C = 352$, 658M parameters) and SwinV2-G ($C = 512$, blocks 2, 2, 42, 4, 3.0B parameters; §3.6). SwinV2-G was pre-trained first with the masked-image-modelling method SimMIM and then by classification on a privately collected 70-million-image ImageNet-22K-ext set, at $192 \times 192$ (§4.2). It reached 84.0% top-1 on ImageNet-V2, 63.1 box AP and 54.4 mask AP on COCO test-dev with HTC++, 59.9 mIoU on ADE20K val with multi-scale testing, and 86.8% on Kinetics-400 (Tables 2–5).

In ablations on ImageNet-1K (Table 6), res-post-norm and cosine attention together raised Swin-T from 81.5% to 81.7%, Swin-S from 83.2% to 83.6% and Swin-B from 83.6% to 84.1%, and ViT-B from 82.2% to 82.6%. For transfer across windows (Table 1, Swin-T trained at $256^2$ with $8 \times 8$ windows), testing at $768^2$ with $24 \times 24$ windows without fine-tuning gave 68.7% with V1’s parameterized bias and 79.1% with Log-CPB; the text of §4.3 gives 78.9% for this case, while the table prints 79.1%. After fine-tuning, the numbers were 83.2% and 84.2%.

Other variants

• Video Swin Transformer (Liu, Ning, Cao, Wei, Zhang, Lin and Hu, CVPR 2022) extends windows and shifts to space–time; the video transformers article describes it.
• Restoration and dense prediction. SwinIR builds an image restoration network from residual Swin blocks (super-resolution covers it); Mask2Former reports most of its best results with Swin-L backbones; and MiDaS v3.1 offers Swin and SwinV2 backbones.
• Released models. As of November 2026, the official repository also offers SimMIM pre-trained Swin and SwinV2 models and a mixture-of-experts variant, Swin-MoE.

Implementations

The official PyTorch code is in the microsoft/Swin-Transformer repository (MIT licence), with classification in the main repository and detection, segmentation and video in separate repositories linked from its README, as of November 2026. The model file is short: WindowAttention holds the bias table and index, SwinTransformerBlock builds the shift mask and applies torch.roll, and PatchMerging and SwinTransformer assemble the stages.

The example below re-implements the parts that matter in NumPy (float64) and checks them on a real image: scikit-image’s astronaut photograph, resized to $224 \times 224$ and cut into $56 \times 56$ tokens of $4 \times 4 \times 3$ values, which a random linear map embeds to width 96, as in Swin-T’s first stage. All weights are random (seed 0); no model is trained. The relative bias table is drawn with standard deviation 1, larger than the code’s 0.02 initialization, so that a wrong bias index would show. Grid positions are (row, column) with (0, 0) at the top left.

The first part computes shifted-window attention in two ways. sw_msa_cyclic follows the official code: roll by $-3$ along both axes, cut 64 windows, mask pairs from different regions using the code’s region labels, roll back. sw_msa_naive follows the naive solution: it forms each true shifted window, pads the small ones to $7 \times 7$ and masks the padded keys, and takes the bias from the tokens’ actual offsets. A third call drops the mask.

import numpy as np
from scipy.special import erf
from skimage import data
from skimage.transform import resize

rng = np.random.default_rng(0)
M, s, C, heads = 7, 3, 96, 3          # Swin-T stage 1: window 7, shift floor(7/2), width 96, 3 heads
d = C // heads

# ---- data: astronaut -> 224 x 224 -> 56 x 56 tokens of 4 x 4 x 3 = 48 values -> width 96
img = resize(data.astronaut(), (224, 224), anti_aliasing=True)
G = 224 // 4
X = img.reshape(G, 4, G, 4, 3).transpose(0, 2, 1, 3, 4).reshape(G, G, 48) @ (rng.standard_normal((48, C)) / np.sqrt(48))
print("tokens", X.shape, "from image", img.shape)

def new_attention_params():
    return dict(Wqkv=rng.standard_normal((C, 3 * C)) / np.sqrt(C), bqkv=rng.standard_normal(3 * C) * 0.1,
                Wo=rng.standard_normal((C, C)) / np.sqrt(C), bo=rng.standard_normal(C) * 0.1,
                table=rng.standard_normal(((2 * M - 1) ** 2, heads)))   # relative position bias table B-hat

def softmax(S):
    E = np.exp(S - S.max(-1, keepdims=True))
    return E / E.sum(-1, keepdims=True)

def attend(x, P, bias, mask=None):
    """Multi-head attention inside each window. x: (nW, N, C); bias: (heads, N, N); additive mask: (nW, N, N)."""
    nW, N, _ = x.shape
    q, k, v = (x @ P["Wqkv"] + P["bqkv"]).reshape(nW, N, 3, heads, d).transpose(2, 0, 3, 1, 4)
    S = q @ k.swapaxes(-1, -2) / np.sqrt(d) + bias
    if mask is not None:
        S = S + mask[:, None]
    return (softmax(S) @ v).transpose(0, 2, 1, 3).reshape(nW, N, C) @ P["Wo"] + P["bo"]

def rel_index(M):
    """Index into B-hat for every pair of positions in an M x M window, as in the official code."""
    yx = np.stack(np.meshgrid(np.arange(M), np.arange(M), indexing="ij")).reshape(2, -1)
    rel = yx[:, :, None] - yx[:, None, :] + (M - 1)
    return rel[0] * (2 * M - 1) + rel[1]

def partition(x, M):                   # (H, W, C) -> (windows, M*M, C)
    H, W, c = x.shape
    return x.reshape(H // M, M, W // M, M, c).transpose(0, 2, 1, 3, 4).reshape(-1, M * M, c)

def unpartition(w, M, H, W):
    return w.reshape(H // M, W // M, M, M, -1).transpose(0, 2, 1, 3, 4).reshape(H, W, -1)

def region_labels(H, W, M, s):
    """The official code's labels: after the roll, which true window each position came from."""
    lab, cnt = np.zeros((H, W)), 0
    for hs in (slice(0, -M), slice(-M, -s), slice(-s, None)):
        for ws in (slice(0, -M), slice(-M, -s), slice(-s, None)):
            lab[hs, ws] = cnt
            cnt += 1
    return lab

def sw_msa_cyclic(x, P, M, s, mask_value=-100.0, use_mask=True):
    """Efficient batch computation (Figure 4): cyclic shift, masked window attention, reverse shift."""
    H, W, _ = x.shape
    bias = P["table"][rel_index(M).ravel()].reshape(M * M, M * M, heads).transpose(2, 0, 1)
    mask = None
    if s > 0:
        x = np.roll(x, (-s, -s), axis=(0, 1))
        if use_mask:
            lab = partition(region_labels(H, W, M, s)[..., None], M)[..., 0]
            mask = np.where(lab[:, :, None] != lab[:, None, :], mask_value, 0.0)
    out = unpartition(attend(partition(x, M), P, bias, mask), M, H, W)
    return np.roll(out, (s, s), axis=(0, 1)) if s > 0 else out

def window_ids(n, M, s):
    """True window index of each row (or column) when the partition is displaced by s."""
    return (np.arange(n) + (M - s) % M) // M

def sw_msa_naive(x, P, M, s):
    """Naive solution: every true window, smaller ones padded to M x M with the padded keys masked out."""
    H, W, _ = x.shape
    ry, rx = window_ids(H, M, s), window_ids(W, M, s)
    out, n_windows, n_padded, shapes = np.zeros_like(x), 0, 0, set()
    for wy in np.unique(ry):
        for wx in np.unique(rx):
            yy, xx = np.meshgrid(np.where(ry == wy)[0], np.where(rx == wx)[0], indexing="ij")
            yy, xx, n = yy.ravel(), xx.ravel(), yy.size
            shapes.add((int(np.sum(ry == wy)), int(np.sum(rx == wx))))
            tok = np.zeros((M * M, C))
            tok[:n] = x[yy, xx]
            bias = np.zeros((heads, M * M, M * M))
            idx = (yy[:, None] - yy[None, :] + M - 1) * (2 * M - 1) + (xx[:, None] - xx[None, :] + M - 1)
            bias[:, :n, :n] = P["table"][idx].transpose(2, 0, 1)
            mask = np.zeros((1, M * M, M * M))
            mask[:, :, n:] = -np.inf
            out[yy, xx] = attend(tok[None], P, bias, mask)[0, :n]
            n_windows, n_padded = n_windows + 1, n_padded + M * M
    return out, n_windows, n_padded, shapes

# ================= Part A: the masked cyclic shift equals attention in the true shifted windows
P = new_attention_params()
naive, n_win, n_pad, shapes = sw_msa_naive(X, P, M, s)
print(f"\nPart A. true shifted windows {n_win} (padded tokens {n_pad} for {G * G} tokens), window shapes {sorted(shapes)}")
print(f"batched windows after the cyclic shift: {partition(X, M).shape[0]}")
for mv in (-100.0, -np.inf):
    cyc = sw_msa_cyclic(X, P, M, s, mask_value=mv)
    print(f"cyclic shift + mask {mv:>6}: max |difference| to naive = {np.abs(cyc - naive).max():.1e}")
nomask = sw_msa_cyclic(X, P, M, s, use_mask=False)
changed = np.abs(nomask - naive).max(-1) > 1e-9
print(f"cyclic shift, no mask: tokens that differ from naive = {changed.sum()}, max |difference| = {np.abs(nomask - naive).max():.2f}")
reg_naive = sw_msa_naive(X, P, M, 0)[0]
print(f"regular windows (s = 0): batched vs naive max |difference| = {np.abs(sw_msa_cyclic(X, P, M, 0) - reg_naive).max():.1e}")
part = [np.bincount(window_ids(G, M, sh)).tolist() for sh in (0, s)]
print("partition check, rows per window (regular, shifted):", part, "sums", [sum(p) for p in part])

tokens (56, 56, 96) from image (224, 224, 3)

Part A. true shifted windows 81 (padded tokens 3969 for 3136 tokens), window shapes [(3, 3), (3, 4), (3, 7), (4, 3), (4, 4), (4, 7), (7, 3), (7, 4), (7, 7)]
batched windows after the cyclic shift: 64
cyclic shift + mask -100.0: max |difference| to naive = 1.1e-15
cyclic shift + mask   -inf: max |difference| to naive = 1.1e-15
cyclic shift, no mask: tokens that differ from naive = 735, max |difference| = 1.28
regular windows (s = 0): batched vs naive max |difference| = 0.0e+00
partition check, rows per window (regular, shifted): [[7, 7, 7, 7, 7, 7, 7, 7], [3, 7, 7, 7, 7, 7, 7, 7, 4]] sums [56, 56]

On the $56 \times 56$ grid the shifted partition has 81 true windows of nine shapes, from $3 \times 3$ in the top-left corner to $7 \times 7$ in the interior; the naive solution pads them to 3,969 token slots for 3,136 tokens, while the cyclic shift keeps the regular 64 windows. The two computations agree to rounding error, with the code’s $-100$ as with $-\infty$: $e^{-100}$ is far below float64 resolution relative to the unmasked weights. The agreement includes the relative position bias, which confirms that indexing it by position after the roll is correct. Without the mask, exactly the 735 tokens in the last row and column of batched windows change, by up to 1.28, because their windows join pieces from opposite edges of the image; these are the tokens whose attention would wrap around.

The second part counts parameters and FLOPs of the published configurations from the official code’s module shapes and its flops() conventions, which count multiply-adds.

# ================= Part B: parameters and FLOPs of the configurations, against Table 1
def swin_size(C0, depths, nh, img=224, M=7, ncls=1000):
    """Parameter shapes of the official model; FLOPs as its flops() methods count them (multiply-adds)."""
    H = img // 4
    shapes = [(C0, 3, 4, 4), (C0,), (C0,), (C0,)]                  # patch embedding conv, bias, LayerNorm
    flops = H * H * C0 * 48 + H * H * C0
    for i, (dep, h) in enumerate(zip(depths, nh)):
        c, r = C0 * 2 ** i, H // 2 ** i
        m = min(M, r)                                               # the code shrinks the window to the map
        for _ in range(dep):
            shapes += [(c,), (c,), (c, 3 * c), (3 * c,), (c, c), (c,), ((2 * m - 1) ** 2, h),
                       (c,), (c,), (c, 4 * c), (4 * c,), (4 * c, c), (c,)]
            N = m * m
            flops += 2 * c * r * r + r * r // N * (4 * N * c * c + 2 * N * N * c) + 8 * r * r * c * c
        if i < len(depths) - 1:                                     # patch merging: LayerNorm(4c), 4c -> 2c
            shapes += [(4 * c,), (4 * c,), (4 * c, 2 * c)]
            flops += r * r * c + (r // 2) ** 2 * 8 * c * c
    f = C0 * 2 ** (len(depths) - 1)
    shapes += [(f,), (f,), (f, ncls), (ncls,)]
    flops += f * H * H // 2 ** len(depths) + f * ncls              # final norm as the code counts it, head
    return sum(int(np.prod(sh)) for sh in shapes), flops

print("\nPart B. model        params   FLOPs   Table 1")
for name, C0, dep, nh, img_size, Mw, paper in [
        ("Swin-T", 96, (2, 2, 6, 2), (3, 6, 12, 24), 224, 7, "29M 4.5G"),
        ("Swin-S", 96, (2, 2, 18, 2), (3, 6, 12, 24), 224, 7, "50M 8.7G"),
        ("Swin-B", 128, (2, 2, 18, 2), (4, 8, 16, 32), 224, 7, "88M 15.4G"),
        ("Swin-B 384", 128, (2, 2, 18, 2), (4, 8, 16, 32), 384, 12, "88M 47.0G"),
        ("Swin-L 384", 192, (2, 2, 18, 2), (6, 12, 24, 48), 384, 12, "197M 103.9G")]:
    p, f = swin_size(C0, dep, nh, img_size, Mw)
    print(f"        {name:10s} {p / 1e6:6.2f}M {f / 1e9:6.2f}G   {paper}")

Part B. model        params   FLOPs   Table 1
        Swin-T      28.29M   4.49G   29M 4.5G
        Swin-S      49.61M   8.75G   50M 8.7G
        Swin-B      87.77M  15.44G   88M 15.4G
        Swin-B 384  87.90M  47.11G   88M 47.0G
        Swin-L 384 196.74M 103.95G   197M 103.9G

The FLOPs reproduce Table 1 to the printed precision or within 0.6%, and the parameters for Swin-S, -B and -L round to the printed values. Swin-T is the exception: the code’s model has 28.29M parameters, which the paper prints as 29M; the official README lists Swin-T at 28M, and we did not find what accounts for the paper’s figure.

The last part asks how far information travels. A block lets output token $i$ depend on input token $j$ when the two share a window (attention) or $i = j$ (the residual); the MLP acts per token. Multiplying these Boolean dependency matrices block after block gives, for each output token, the set of input tokens that can reach it. Three schedules on the $56 \times 56$ grid are compared: regular windows only, Swin’s alternation of regular and shifted windows, and a $7 \times 7$ sliding window around each token, the design the paper compares with in Tables 5 and 6. The code then follows Swin-T’s four stages with patch merging, and finally pushes a random perturbation of one input token through four full random-weight Swin blocks (pre-norm LayerNorm, window attention with the cyclic shift, MLP) to see which outputs change.

# ================= Part C: how far information travels
yy, xx = np.divmod(np.arange(G * G), G)

def block_matrix(n, kind, M=7, s=3):
    """dep[i, j] = output token i of one block can depend on input token j (attention + residual)."""
    y, x = np.divmod(np.arange(n * n), n)
    if kind == "sliding":                                      # 7 x 7 neighbourhood around each query
        return (np.abs(y[:, None] - y[None]) <= M // 2) & (np.abs(x[:, None] - x[None]) <= M // 2)
    ids = window_ids(n, M, s if kind == "shifted" else 0)
    lab = ids[y] * (ids.max() + 1) + ids[x]
    return lab[:, None] == lab[None, :]

B = {k: block_matrix(G, k).astype(np.float32) for k in ("regular", "shifted", "sliding")}
schedule = {"no shift": lambda b: "regular", "Swin": lambda b: "regular" if b % 2 == 0 else "shifted",
            "sliding": lambda b: "sliding"}
centre = 28 * G + 28
counts, full = {}, {}
R = {name: np.eye(G * G, dtype=np.float32) for name in schedule}
print("\nPart C. input tokens reaching an output token (mean / min / max; centre token)")
for L in range(1, 31):
    for name, kind in schedule.items():
        R[name] = ((B[kind(L - 1)] @ R[name]) > 0).astype(np.float32)
        c = R[name].sum(1)
        counts[(name, L)] = c
        if name not in full and c.min() == G * G:
            full[name] = L
    if L <= 8:
        print(f"L={L}  " + " | ".join(f"{n}: {counts[(n, L)].mean():6.1f} {counts[(n, L)].min():5.0f} "
                                      f"{counts[(n, L)].max():5.0f}; {counts[(n, L)][centre]:5.0f}" for n in schedule))
print("blocks until every token reaches every other (within 30):", {n: full.get(n, "never") for n in schedule})
np.savez("run1_reach.npz", **{f"{n}_{L}": counts[(n, L)] for n in schedule for L in range(1, 9)})

def merge_matrix(n):
    """Patch merging: token (i, j) of the n/2 grid reads tokens (2i + a, 2j + b), a, b in {0, 1}."""
    y, x = np.divmod(np.arange(n * n), n)
    return ((y[None] // 2) * (n // 2) + x[None] // 2 == np.arange(n * n // 4)[:, None]).astype(np.float32)

print("Swin-T, fraction of the 56 x 56 stage-1 grid reaching a token at each stage end (mean, min)")
for shift in (False, True):
    D, n, out = np.eye(G * G, dtype=np.float32), G, []
    for st, depth in enumerate((2, 2, 6, 2)):
        if st > 0:
            D, n = ((merge_matrix(n) @ D) > 0).astype(np.float32), n // 2
        m = min(7, n)
        for b in range(depth):
            kind = "shifted" if (shift and b % 2 == 1 and n > 7) else "regular"
            D = ((block_matrix(n, kind, M=m, s=m // 2).astype(np.float32) @ D) > 0).astype(np.float32)
        f = D.sum(1) / (G * G)
        out.append(f"({f.mean():.4f}, {f.min():.4f})")
    print(f"  {'shifted' if shift else 'no shift'}: " + " ".join(out))

# Perturbation through four full random-weight Swin blocks (pre-norm, (S)W-MSA, MLP)
def layer_norm(x, eps=1e-5):
    return (x - x.mean(-1, keepdims=True)) / np.sqrt(x.var(-1, keepdims=True) + eps)

def gelu(x):
    return 0.5 * x * (1 + erf(x / np.sqrt(2)))

blocks = [(new_attention_params(), rng.standard_normal((C, 4 * C)) / np.sqrt(C), np.zeros(4 * C),
           rng.standard_normal((4 * C, C)) / np.sqrt(4 * C), np.zeros(C)) for _ in range(4)]

def swin_blocks(x, shift):
    for b, (P, W1, b1, W2, b2) in enumerate(blocks):
        x = x + sw_msa_cyclic(layer_norm(x), P, M, s if (shift and b % 2 == 1) else 0)
        x = x + gelu(layer_norm(x) @ W1 + b1) @ W2 + b2
    return x

Xp = X.copy()
Xp[28, 28] += rng.standard_normal(C)       # a random direction: LayerNorm removes a constant shift
for shift in (False, True):
    diff = np.abs(swin_blocks(Xp, shift) - swin_blocks(X, shift)).max(-1)
    r, c = np.nonzero(diff > 1e-12)
    print(f"perturb input token (28, 28), 4 blocks, {'shifted' if shift else 'no shift'}: "
          f"{(diff > 1e-12).sum()} output tokens change, rows {r.min()}..{r.max()}, cols {c.min()}..{c.max()}")

Part C. input tokens reaching an output token (mean / min / max; centre token)
L=1  no shift:   49.0    49    49;    49 | Swin:   49.0    49    49;    49 | sliding:   46.0    16    49;    49
L=2  no shift:   49.0    49    49;    49 | Swin:  172.3    49   196;   196 | sliding:  150.1    49   169;   169
L=3  no shift:   49.0    49    49;    49 | Swin:  370.6   196   441;   441 | sliding:  302.5   100   361;   361
L=4  no shift:   49.0    49    49;    49 | Swin:  600.2   196   784;   784 | sliding:  493.5   169   625;   625
L=5  no shift:   49.0    49    49;    49 | Swin:  885.1   441  1225;  1225 | sliding:  713.7   256   961;   961
L=6  no shift:   49.0    49    49;    49 | Swin: 1164.5   441  1764;  1764 | sliding:  954.4   361  1369;  1369
L=7  no shift:   49.0    49    49;    49 | Swin: 1482.2   784  2401;  2401 | sliding: 1207.6   484  1849;  1849
L=8  no shift:   49.0    49    49;    49 | Swin: 1764.0   784  3136;  3136 | sliding: 1465.8   625  2401;  2401
blocks until every token reaches every other (within 30): {'no shift': 'never', 'Swin': 15, 'sliding': 19}
Swin-T, fraction of the 56 x 56 stage-1 grid reaching a token at each stage end (mean, min)
  no shift: (0.0156, 0.0156) (0.0625, 0.0625) (0.2500, 0.2500) (1.0000, 1.0000)
  shifted: (0.0549, 0.0156) (0.3525, 0.1406) (1.0000, 1.0000) (1.0000, 1.0000)
perturb input token (28, 28), 4 blocks, no shift: 49 output tokens change, rows 28..34, cols 28..34
perturb input token (28, 28), 4 blocks, shifted: 784 output tokens change, rows 17..44, cols 17..44

Without the shift, reach never grows beyond a token’s own window of 49 tokens, however many blocks are stacked. With the shift, the centre token’s reach grows from $7 \times 7$ to $14 \times 14$, $21 \times 21$ and $28 \times 28$ tokens, widening by one window (7 tokens) per block along each axis, and every token reaches every other after 15 blocks; the $7 \times 7$ sliding window, which widens reach by 6 tokens per block along each axis, needs 19. From the window layout, tokens near the image border grow more slowly, because their shifted windows are the small ones, three or four tokens wide, that lie inside a single regular window; this is why the minimum lags the mean. Across Swin-T’s stages, patch merging alone would make stage 4 global, since it is a single $7 \times 7$ window over the whole image; with shifting, every token already depends on the whole image at the end of stage 3. Stage 1, however, ends with reach of only about 5% of the image on average, so detection and segmentation heads that read the stride-4 map get local features there. The perturbation through real blocks changes a $7 \times 7$ block of outputs (rows and columns 28 to 34) without the shift and a $28 \times 28$ block (17 to 44) with it: the same 49 and 784 tokens that the Boolean count gives for the centre token after four blocks, now in a network with LayerNorm, softmax and MLP. The blocks are not centred on token (28, 28) because windows are anchored to the grid, not to the token. An earlier version of this check failed in a pilot run, before the first run: it perturbed the token by the same amount in every channel, a shift that LayerNorm removes before the attention and MLP branches see it. It was replaced by a random direction before the first run, with the expected token counts, fixed beforehand, left unchanged.

The example ran with NumPy 2.3.5, SciPy 1.16.3 and scikit-image 0.25.2 and took about two to three minutes in two runs on a heavily loaded shared machine (approximate, since other jobs shared it), almost all of it in the 3,136 × 3,136 Boolean products of the last part.