Squeeze-and-Excitation

A squeeze-and-excitation (SE) block lets a convolutional network turn its own channels up or down depending on the image. It averages each feature map to a single number, feeds the vector of averages through two small fully connected layers, and multiplies every channel by the resulting weight between 0 and 1. Hu, Shen and Sun introduced it in 2017 (arXiv, published at CVPR 2018); according to the authors, an ensemble of SE networks won the ILSVRC 2017 classification task, and the block was later built into mobile and scaled architectures such as MobileNetV3 and EfficientNet. It is often called channel attention.

The original paper appeared at CVPR 2018; an extended journal version with Samuel Albanie and Enhua Wu (TPAMI, 2020; read here as arXiv v4, whose comments field calls it the journal version) adds CIFAR and mobile experiments, an ablation study and further analysis. Results below name the version they come from, because the two use different training schedules for some experiments.

Definition

An SE block is attached to a transformation $F_\text{tr}$, such as a convolution, an Inception module or the residual branch of a ResNet block, whose output $U$ has $C$ channels. It computes a vector of $C$ gates from $U$ in two steps and rescales $U$ with them:

• Squeeze: global average pooling reduces each channel to one number, giving a channel descriptor $\mathbf{z} \in \mathbb{R}^C$.
• Excitation: a bottleneck of two fully connected layers, with a ReLU between them and a sigmoid at the end, maps $\mathbf{z}$ to gates $\mathbf{s} \in (0, 1)^C$. The first layer reduces the width by a reduction ratio $r$ (16 by default).
• Scale: each channel of $U$ is multiplied by its gate.

The output has the same shape as $U$ and replaces it in the network. Hu et al. describe the block as feature recalibration: emphasizing informative channels and suppressing less useful ones, with weights that depend on the input.

Intuition

Every unit of a convolutional layer sees only its receptive field. Early in the network that field is a few pixels wide, so a unit that responds to red horizontal stripes cannot know whether the image also contains a green pattern on the other side. Hu et al. name this limitation as the motivation: each unit is unable to exploit contextual information outside its region, a problem that is more severe in lower layers.

The squeeze step summarizes the whole image per channel: the mean of the stripe detector’s map says how much red stripe there is anywhere. The excitation step looks at all these summaries together and decides, for this image, how loud each channel should be, like a mixing desk whose faders are set automatically from the overall sound. Since the gates are computed from the image, a channel can be switched on for one image and off for another, and the next layer then sees local features already modulated by global context. The second example below builds a task that a network without this global path provably cannot solve.

Formal Definition

Let $U = [\mathbf{u}_1, \dots, \mathbf{u}_C]$ with $\mathbf{u}_c \in \mathbb{R}^{H \times W}$. The squeeze is global average pooling (Hu et al., 2018, Eq. 2; see pooling):

$$z_c = F_\text{sq}(\mathbf{u}_c) = \frac{1}{H W} \sum_{i=1}^{H} \sum_{j=1}^{W} u_c(i, j).$$

The excitation is a gating function with a sigmoid $\sigma$ and a ReLU $\delta$ (Eq. 3):

$$\mathbf{s} = F_\text{ex}(\mathbf{z}, W) = \sigma\big(W_2\, \delta(W_1 \mathbf{z})\big), \qquad W_1 \in \mathbb{R}^{\frac{C}{r} \times C}, \quad W_2 \in \mathbb{R}^{C \times \frac{C}{r}},$$

and the output is channel-wise scaling (Eq. 4):

$$\tilde{\mathbf{x}}_c = F_\text{scale}(\mathbf{u}_c, s_c) = s_c\, \mathbf{u}_c.$$

The authors gave two requirements for the excitation: it must learn a nonlinear interaction between channels, and the relationship must not be mutually exclusive, so that several channels can be emphasized at once rather than one. The sigmoid satisfies the second; a softmax would not. The bottleneck limits the parameter count and, the authors state, aids generalization.

Where the block goes. For a non-residual network such as Inception, $F_\text{tr}$ is an entire Inception module, and the SE block acts on the module’s concatenated output. For a residual network, $F_\text{tr}$ is the non-identity branch: squeeze and excitation act before the summation with the identity (CVPR 2018, §3.3, Figs. 2–3). The journal version adds that in VGG-style networks the block goes after the nonlinearity that follows each convolution. In SE-ResNet-50 each bottleneck block’s SE module maps $C$ to $C/16$ to $C$, for example $1024 \to 64 \to 1024$ in the stage with $14 \times 14$ maps (Table 1).

Parameter overhead. All added parameters are in the two fully connected layers. For a network with $S$ stages, where stage $s$ repeats $N_s$ blocks with $C_s$ output channels, the added weights number (Eq. 5)

$$\frac{2}{r} \sum_{s=1}^{S} N_s\, C_s^2 ,$$

with biases, when used, adding a negligible amount (journal version). The quadratic dependence on $C_s$ puts most of the cost in the widest, last stage.

Properties

Cost. The authors reported about 3.87 GFLOPs for SE-ResNet-50 against about 3.86 for ResNet-50 at $224 \times 224$, a 0.26% relative increase, and about 2.5 million added parameters on top of about 25 million, roughly 10% (CVPR 2018, §4). The first example reproduces both from Eq. 5. Measured time grew more than the operation count suggests: a forward and backward pass over a mini-batch of 256 images took 190 ms for ResNet-50 and 209 ms for SE-ResNet-50 on a server with eight Titan X GPUs, and CPU inference on one $224 \times 224$ image 164 against 167 ms. The authors attribute part of the gap to global pooling and small inner products being less optimized in GPU libraries. Removing the SE blocks of the last stage cut the parameter increase to about 4% at a marginal cost: less than 0.1% top-1 error in the CVPR paper, less than 0.1% top-5 error in the journal version, which differ on which metric they name.

A gate, not a mixer. The scale step multiplies channel $c$ by one scalar, so output channel $c$ contains only input channel $c$; information crosses channels only through the gate. This differs from self-attention, which mixes positions with softmax weights. The CVPR paper calls the SE mechanism self-gating and contrasts it with heavier attention modules; the journal version adds that it can be regarded as a self-attention function on channels. Read as attention, an SE block attends over channels rather than positions, with independent sigmoid weights in place of a normalized distribution, and the same weights apply at every pixel. Unlike batch normalization’s learned scale $\gamma$ (see normalization layers), which is fixed after training, the gates are recomputed for each image.

Identity as a special case. If every gate equals 1, the block is the plain residual block; the journal version notes that an SE block then reduces to the identity operator. With the gates on the residual branch only, the identity path of a residual block is never rescaled.

Roles across depth. Hu et al. examined the mean excitations of SE-ResNet-50 for fifty validation images each of four ImageNet classes (goldfish, pug, plane, cliff), at fifty uniformly sampled channels of the last SE block of each stage (CVPR Fig. 5; journal Fig. 6). In early blocks such as SE_2_3 the distributions were nearly identical across classes, suggesting channel importance shared by all classes; at greater depth (SE_4_6, SE_5_1) they became class-specific. In the last stage, SE_5_2 tended toward saturation, most activations close to 1 (the CVPR text adds that the remainder were close to 0), and SE_5_3 showed a similar pattern across classes up to a change in scale. The authors concluded that the last-stage blocks matter less for recalibration, consistent with the small cost of removing them. The journal version adds that excitations also vary between images of the same class (Fig. 7).

ImageNet results. In the CVPR paper (Table 2; single 224-pixel centre crop, ImageNet validation set, 100 epochs of SGD for each model and its SE counterpart), the authors re-trained every baseline:

Model	Baseline top-1 / top-5 error (%)	With SE	GFLOPs
ResNet-50	24.80 / 7.48	23.29 / 6.62	3.86 → 3.87
ResNet-101	23.17 / 6.52	22.38 / 6.07	7.58 → 7.60
ResNet-152	22.42 / 6.34	21.57 / 5.73	11.30 → 11.32
ResNeXt-50 (32×4d)	22.11 / 5.90	21.10 / 5.49	4.24 → 4.25
VGG-16 (with BN)	27.02 / 8.81	25.22 / 7.70	15.47 → 15.48
BN-Inception	25.38 / 7.89	24.23 / 7.14	2.03 → 2.04
Inception-ResNet-v2	20.37 / 5.21	19.80 / 4.79	11.75 → 11.76

SE-ResNet-50’s top-5 error approached ResNet-101’s 6.52% at about half the computation, and SE-ResNet-101 beat ResNet-152. 1.0 MobileNet-224 and ShuffleNet 1× ($g = 3$) gained too, but the two versions print different baselines: the CVPR paper gives MobileNet 29.1% → 25.3% and ShuffleNet 33.9% → 31.7% top-1 error (Table 3); the journal version, which describes a separate training setup of about 400 epochs for the mobile models, gives 28.4% → 25.3% and 32.6% → 31.0%. On Places365, SE-ResNet-152 had 11.01% top-5 validation error against 11.61%. For COCO detection with Faster R-CNN, the CVPR paper reports AP 25.1 → 26.4 for ResNet-50 on a 40k validation set with a basic implementation, and the journal version 38.0 → 40.4 on minival with the “2x” schedule.

The ILSVRC 2017 winning entry was a small ensemble of SE networks with multi-scale, multi-crop testing, at 2.251% top-5 error on the test set, which the authors describe as about a 25% relative improvement over the 2016 winner’s 2.991%. The official ILSVRC 2017 results list their team’s (WMW) entries first by classification error, the lowest at 0.02251, in the classification-and-localization task with provided training data. One of the submitted networks, SENet-154, adds SE blocks to a modified ResNeXt; on a single $224 \times 224$ centre crop of the validation set it reached 18.68% top-1 and 4.47% top-5 error (CVPR Table 4).

Ablations. The journal version (§6–7) ablates SE-ResNet-50 under a longer schedule: about 300 epochs with plateau-based learning-rate drops, label smoothing, and fully connected layers without biases, which the authors found to help on ResNets. Its baseline and SE errors (23.30% and 22.28% top-1) are therefore not comparable with the CVPR table above.

Ablation (journal version)	Setting	Top-1 / top-5 error (%)
Reduction ratio $r$ (Table 10)	2 / 4 / 8 / 16 / 32	22.29 / 22.25 / 22.26 / 22.28 / 22.72 (top-1), at 45.7M / 35.7M / 30.7M / 28.1M / 26.9M parameters
Squeeze operator (Table 11)	max pooling / average pooling	22.57 / 6.09 vs 22.28 / 6.03
Excitation nonlinearity (Table 12)	ReLU / tanh / sigmoid	23.47 / 6.98, 23.00 / 6.38, 22.28 / 6.03
Stages with SE (Table 13)	stage 2 / stage 3 / stage 4 only, all	23.03, 23.04, 22.68, 22.28 (baseline 23.30)
Position (Table 14)	standard / SE-PRE / SE-POST / SE-Identity	22.28, 22.23, 22.78, 22.20
Inside the block (Table 15)	after the 3×3 convolution	22.48 / 6.02 at 25.8M vs 28.1M parameters
No squeeze (Table 16)	1×1 convolutions, no pooling	22.93 / 6.39 at 4.27 GFLOPs

Performance was flat from $r = 2$ to 16 while the parameter count rose steeply as $r$ fell, which is why $r = 16$ became the default; the authors add that one ratio for every layer may not be optimal. Replacing the sigmoid with ReLU made SE-ResNet-50 worse than the plain baseline. SE-PRE moves the block before the residual unit, SE-POST after the summation and its ReLU, SE-Identity onto the shortcut in parallel; only SE-POST lost accuracy, and the authors conclude that placement matters little as long as the gates act before the branches are summed. The NoSqueeze variant keeps the same number of parameters but applies the bottleneck as $1 \times 1$ convolutions at every position, so the gates see only local features: it recovered less than half of the gain while costing more computation, which the authors take as evidence that the global information matters.

Examples

Parameter and operation overhead for ResNet-50. The first example counts the weights of the SE modules that SE-ResNet-50 adds to each bottleneck block ($r = 16$; stage widths 256, 512, 1024 and 2048 with 3, 4, 6 and 3 blocks), checks them against Eq. 5, and tallies the extra arithmetic at a $224 \times 224$ input. It uses only torch (PyTorch 2.14.1).

import torch
import torch.nn as nn

class SE(nn.Module):
    """Squeeze-and-excitation on an (N, C, H, W) tensor, reduction ratio r."""
    def __init__(self, c, r=16, bias=True):
        super().__init__()
        self.fc1 = nn.Linear(c, c // r, bias=bias)     # W1: C -> C/r
        self.fc2 = nn.Linear(c // r, c, bias=bias)     # W2: C/r -> C

    def forward(self, u):
        z = u.mean(dim=(2, 3))                                  # squeeze: global average pool, (N, C)
        s = torch.sigmoid(self.fc2(torch.relu(self.fc1(z))))    # excitation: (N, C) gates in (0, 1)
        return u * s[:, :, None, None], s                       # scale: one weight per channel

# ResNet-50 stages conv2_x..conv5_x: blocks N_s, output channels C_s, middle (3x3) width, map size.
stages = [(3, 256, 64, 56), (4, 512, 128, 28), (6, 1024, 256, 14), (3, 2048, 512, 7)]
resnet50 = 25_557_032          # parameters of a 1000-class ResNet-50 (see the ResNet article)
r = 16
n_params = lambda m: sum(p.numel() for p in m.parameters())

total_w, total_b, ops_fc, ops_pool, ops_scale = 0, 0, 0, 0, 0
for i, (n, c, mid, hw) in enumerate(stages, start=2):
    w = n_params(SE(c, r, bias=False))
    b = n_params(SE(c, r)) - w
    total_w += n * w; total_b += n * b
    ops_fc += n * w                       # one multiply-add per FC weight
    ops_pool += n * c * hw * hw           # additions in the global average pool
    ops_scale += n * c * hw * hw          # multiplications in the channel-wise scaling
    print(f"stage {i}: {n} blocks x SE({c:>4}): {n * w:>9,} weights (+{n * w / 1e6:.2f}M)")

sum_nc2 = sum(n * c ** 2 for n, c, _, _ in stages)
print(f"Eq. (5), 2/r * sum N_s C_s^2:  {2 / r * sum_nc2:,.0f}")
print(f"counted FC weights:            {total_w:,}  (+ {total_b:,} biases)")
print(f"SE-ResNet-50 parameters:       {(resnet50 + total_w + total_b) / 1e6:.1f}M "
      f"(+{100 * (total_w + total_b) / resnet50:.1f}%)")
print("parameters (M) for r = 2, 4, 8, 16, 32:",
      [round((resnet50 + 2 / q * sum_nc2) / 1e6, 2) for q in (2, 4, 8, 16, 32)])
last = stages[-1][0] * 2 * stages[-1][1] ** 2 // r
print(f"share of SE weights in stage 5: {100 * last / total_w:.1f}%; without them: "
      f"+{100 * (total_w - last) / resnet50:.1f}%")
se3x3 = sum(n * n_params(SE(mid, r, bias=False)) for n, _, mid, _ in stages)
print(f"SE after each 3x3 conv instead: {se3x3:,} weights (+{se3x3 / 1e6:.2f}M)")
print(f"FC multiply-adds: {ops_fc / 1e6:.2f}M, pooling adds: {ops_pool / 1e6:.2f}M, "
      f"scaling multiplies: {ops_scale / 1e6:.2f}M")
for name, extra in [("FC only", ops_fc), ("FC + scaling", ops_fc + ops_scale),
                    ("FC + scaling + pooling", ops_fc + ops_scale + ops_pool)]:
    print(f"  3.86 G + {name:<22}= {3.86 + extra / 1e9:.4f} G")

stage 2: 3 blocks x SE( 256):    24,576 weights (+0.02M)
stage 3: 4 blocks x SE( 512):   131,072 weights (+0.13M)
stage 4: 6 blocks x SE(1024):   786,432 weights (+0.79M)
stage 5: 3 blocks x SE(2048): 1,572,864 weights (+1.57M)
Eq. (5), 2/r * sum N_s C_s^2:  2,514,944
counted FC weights:            2,514,944  (+ 16,048 biases)
SE-ResNet-50 parameters:       28.1M (+9.9%)
parameters (M) for r = 2, 4, 8, 16, 32: [45.68, 35.62, 30.59, 28.07, 26.81]
share of SE weights in stage 5: 62.5%; without them: +3.7%
SE after each 3x3 conv instead: 157,184 weights (+0.16M)
FC multiply-adds: 2.51M, pooling adds: 5.52M, scaling multiplies: 5.52M
  3.86 G + FC only               = 3.8625 G
  3.86 G + FC + scaling          = 3.8680 G
  3.86 G + FC + scaling + pooling= 3.8736 G

The 2.51 million added weights give the 28.1M of the CVPR paper’s Table 7 and the roughly 10% of its text. The per-stage increments (0.02M, 0.13M, 0.79M) added to the journal version’s rounded 25.6M give its Table 13 entries of 25.6M, 25.7M and 26.4M, and the SE-after-3×3 count (0.16M) its Table 15 entry of 25.8M; from the exact 25.56M, the last two would round to 26.3M and 25.7M. For other ratios the counts are 0.02M to 0.11M below Table 10 (45.7M, 35.7M, 30.7M, 28.1M, 26.9M). Dropping the last stage, which holds 62.5% of the SE weights, leaves an increase of 3.7%, the paper’s approximate 4%. The arithmetic adds 2.5 to 13.5 million operations depending on what is counted; added to ResNet-50’s 3.86 GFLOPs (a multiply-add count, see ResNet), it reaches the printed 3.87 only when the channel-wise scaling is counted, though since 3.86 is itself rounded, the paper’s counting convention cannot be pinned down this way.

A task only global context can solve. The second example makes $16 \times 16$ RGB images of Gaussian noise. Pattern A is red horizontal stripes in columns 0–6, pattern B green horizontal stripes in columns 9–15, each present with probability 1/2, with random stripe period (2 to 4 pixels) and phase. The label is 1 when exactly one pattern is present, an exclusive or. The network is a $3 \times 3$ convolution and a $1 \times 1$ convolution with 16 channels, global average pooling and a linear classifier, so every feature sees a $3 \times 3$ window, and no window touches both column 6 and column 9.

Our reasoning for why this network cannot solve the task without SE: the pooled vector is an average over positions, and each position’s features depend on the left region or the right region but never both, so the classifier’s score is a sum of a term that depends on A and a term that depends on B. For a fixed background, stripe period and phase, such an additive score separates at most three of the four combinations of A and B (if both mixed cases score above a threshold and both pure cases below it, the two pairs would need different sums of the same four terms). Since the noise, period and phase are drawn independently of A and B, expected accuracy is capped at 75%. A $1 \times 1$ gate at each position (the NoSqueeze variant) does not change this. An SE block after the second convolution ($r = 2$) computes its gates from the means of all channels, so the features of the left region can be scaled by a factor that depends on whether B is present.

import torch
import torch.nn as nn

torch.set_num_threads(1)
S = 16  # image size; left region columns 0-6, right region columns 9-15

def make_images(n, g):
    """RGB stripe images. A: red stripes on the left; B: green stripes on the right.
    Label 1 if exactly one of A, B is present (exclusive or), 0 otherwise."""
    x = 0.3 * torch.randn(n, 3, S, S, generator=g)
    a = torch.randint(0, 2, (n,), generator=g)
    b = torch.randint(0, 2, (n,), generator=g)
    rows = torch.arange(S)
    for i in range(n):
        period = int(torch.randint(2, 5, (1,), generator=g))
        phase = int(torch.randint(0, period, (1,), generator=g))
        stripes = ((rows + phase) % period == 0).float()[:, None]      # horizontal stripes
        if a[i]: x[i, 0, :, 0:7] += stripes
        if b[i]: x[i, 1, :, 9:16] += stripes
    return x, a, b, (a ^ b)

class SE(nn.Module):
    def __init__(self, c, r):
        super().__init__()
        self.fc1, self.fc2 = nn.Linear(c, c // r), nn.Linear(c // r, c)
    def forward(self, u):
        s = torch.sigmoid(self.fc2(torch.relu(self.fc1(u.mean(dim=(2, 3))))))
        self.gates = s.detach()
        return u * s[:, :, None, None]

class NoSqueeze(nn.Module):
    """Same gate without the pooling: 1x1 convolutions, so each position sees only its own features."""
    def __init__(self, c, r):
        super().__init__()
        self.net = nn.Sequential(nn.Conv2d(c, c // r, 1), nn.ReLU(), nn.Conv2d(c // r, c, 1), nn.Sigmoid())
    def forward(self, u):
        return u * self.net(u)

def model(gate, c=16, r=2):
    # 3x3 conv then 1x1 conv: every feature sees a 3x3 window, never both regions at once.
    layers = [nn.Conv2d(3, c, 3, padding=1), nn.ReLU(), nn.Conv2d(c, c, 1), nn.ReLU()]
    if gate: layers.append(gate(c, r))
    layers += [nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(c, 2)]
    return nn.Sequential(*layers)

def run(gate, seed, steps=600):
    torch.manual_seed(seed)
    g = torch.Generator().manual_seed(seed)
    xtr, _, _, ytr = make_images(2000, g)
    xte, ate, bte, yte = make_images(1000, g)
    net = model(gate)
    opt = torch.optim.Adam(net.parameters(), lr=3e-3)
    for step in range(steps):
        idx = torch.randint(0, len(xtr), (64,), generator=g)
        loss = nn.functional.cross_entropy(net(xtr[idx]), ytr[idx])
        opt.zero_grad(); loss.backward(); opt.step()
    with torch.no_grad():
        acc = (net(xte).argmax(1) == yte).float().mean().item()
    return acc, net, (xte, ate, bte)

names = {"plain": None, "NoSqueeze": NoSqueeze, "SE": SE}
for name, gate in names.items():
    runs = [run(gate, seed) for seed in range(5)]
    print(f"{name:>9}: test accuracy over seeds 0-4: " + " ".join(f"{r[0]:.3f}" for r in runs))

# Mean SE gates of the seed-0 model, by which patterns the image contains.
acc, net, (xte, ate, bte) = runs[0]
with torch.no_grad():
    net(xte)
gates = net[4].gates
print("mean gates of channels 0-5 (SE model, seed 0):")
for av in (0, 1):
    for bv in (0, 1):
        m = gates[(ate == av) & (bte == bv)].mean(0)
        print(f"  A={av} B={bv}:", " ".join(f"{v:.2f}" for v in m[:6].tolist()))
print("SE parameters:", sum(p.numel() for p in net[4].parameters()),
      "of", sum(p.numel() for p in net.parameters()))

    plain: test accuracy over seeds 0-4: 0.423 0.393 0.400 0.604 0.686
NoSqueeze: test accuracy over seeds 0-4: 0.371 0.366 0.422 0.489 0.707
       SE: test accuracy over seeds 0-4: 0.990 0.969 0.996 0.994 0.993
mean gates of channels 0-5 (SE model, seed 0):
  A=0 B=0: 1.00 0.03 0.00 0.11 1.00 0.98
  A=0 B=1: 0.84 0.99 0.13 0.91 0.99 0.87
  A=1 B=0: 0.86 1.00 0.11 0.91 0.99 0.87
  A=1 B=1: 0.03 1.00 0.94 0.75 0.22 0.97
SE parameters: 280 of 1034

Over seeds 0–4, the plain and NoSqueeze networks stay between 0.37 and 0.71, below the 75% bound and mostly below chance, while the SE network reaches 0.969 to 0.996 (two runs of the script gave identical output). The gates show how: channel 0 stays largely open (0.84 to 1.00) unless both patterns are present, when it falls to 0.03, and channel 2 opens only when both are present. A local feature is thus multiplied by a factor that depends on content elsewhere in the image, which no sum of local terms can express. The task is built to isolate the mechanism; in a real network, deeper layers have large receptive fields too, and the gains are the modest ones in the tables above.

Variants

• CBAM (Woo et al., ECCV 2018) applies channel attention followed by spatial attention. Its channel branch passes both the average-pooled and the max-pooled descriptor through a shared MLP and sums the results before the sigmoid; its spatial branch pools across channels by average and maximum and applies a $7 \times 7$ convolution and a sigmoid to obtain one weight per position. In the authors’ PyTorch re-implementations (90 epochs, single-crop validation error), ResNet-50 had 24.56% top-1 error, with SE 23.14% and with CBAM 22.66%, at 28.09M parameters for both (their Table 4); their Table 1 lists the SE variant at 25.92M parameters and 3.94 GFLOPs, which disagrees with Table 4.
• Gather-Excite (Hu, Shen, Albanie, Sun & Vedaldi, NeurIPS 2018) generalizes the squeeze to a gather operator over a chosen spatial extent and the excitation to an operator that resizes the gathered map back and gates the input with it. SE is the case of global average pooling with a fully connected excite. A parameter-free version with global extent, the sigmoid of each channel’s mean, reached 22.14% top-1 error on ResNet-50 against 23.30% for the baseline and 22.12% for SE under their roughly 300-epoch schedule (their Table 2). Their GE-θ+ variant, a depthwise-convolution gather combined with the SE bottleneck, reached 21.88%.
• ECA-Net (Wang et al., CVPR 2020) drops the dimensionality reduction. In their ablation on ResNet-50, gating each channel by its own learned weight (no reduction, no interaction) already slightly beat SE (77.07% against 76.71% top-1 accuracy, their Table 2), and they replaced the two fully connected layers by a 1D convolution of size $k$ across the channel axis, shared by all channels, with $k$ set from the channel count as the odd number nearest to $\log_2(C)/2 + 1/2$. They report 77.48% top-1 accuracy for ECA-Net50 against 76.71% for SENet and 75.20% for ResNet-50, both copied from Hu et al. (their Table 3), and state that the modules add 80 parameters in total.

Common Misconceptions

• “SE is attention like a transformer’s.” Its weights are independent sigmoids over channels, shared by every pixel, and nothing is averaged; the journal version’s “self-attention function on channels” is an analogy (see Properties).
• “The overhead is negligible in every sense.” It is 0.26% of ResNet-50’s FLOPs but about 10% of its parameters and of its measured training time in the authors’ setup. On mobile hardware the cost depends on the device; see MobileNet for MobileNetV4’s findings on poor support of SE on some accelerators.
• “The gain comes from the added layers.” The NoSqueeze ablation adds the same parameters without global pooling and recovers less than half of the gain.

Where It Is Used

Residual and multi-branch networks. The original paper builds SE-ResNet, SE-ResNeXt, SE-Inception (on BN-Inception), SE-Inception-ResNet-v2, SE-VGG, SE-MobileNet and SE-ShuffleNet, and SENet-154 on a modified ResNeXt. Later ResNet revisions such as ResNet-RS combine SE with other changes (see ResNet).

Mobile and searched architectures. The journal version notes that SE blocks were used as building blocks in concurrent architecture search work (MnasNet). MobileNetV3 places SE inside the inverted residual block, after the depthwise convolution, and replaces its sigmoid with a hard sigmoid (Howard et al., 2019; details in MobileNet). EfficientNet’s main block is the mobile inverted bottleneck (MBConv) with squeeze-and-excitation added (Tan & Le, 2019). In the RegNet design spaces, RegNetY is defined as RegNetX plus SE (Radosavovic et al., 2020, Fig. 12); at each compute regime the best RegNetY model they found had lower ImageNet top-1 error (mean of five training runs) than the best RegNetX, for example 20.6% against 21.4% at 4.0 GFLOPs, though the two models were selected separately rather than differing only by SE.

Beyond classification. The SE papers report gains for object detection with Faster R-CNN and for scene classification on Places365. Because the block leaves the output shape unchanged, it can be added to an existing backbone design without changing the interface to the rest of the network.