RoI Pooling

RoI pooling is the operation that lets a region-based detector look at many candidate boxes through one shared feature map. Given the map and a box, a region of interest (RoI), it returns a feature grid of fixed size, such as $7 \times 7$ per channel, whatever the box’s size or shape, so that the same fully connected or convolutional head can classify every box and refine it. The name usually means Fast R-CNN’s RoI max pooling layer (Girshick, 2015), often written RoIPool; the broader job, also called RoI feature extraction or region feature pooling, has been done by spatial pyramid pooling, RoI warping, RoIAlign, precise RoI pooling, position-sensitive RoI pooling and rotated versions of these. The operators differ mainly in how they map a real-valued box onto a discrete grid, and that choice, which Mask R-CNN’s authors say may not affect classification, decides how well masks and keypoints line up with the object.

The two-stage object detectors article places RoI pooling in the R-CNN family and shows, with a shift test, how rounding makes RoI pooling’s output jump while RoIAlign’s barely moves. This article treats the operators themselves: their definitions and conventions, their gradients, what each paper reported about them, and how much each loses on boxes of different sizes.

Problem

A two-stage detector computes a backbone feature map once per image and then needs one feature per proposal. Proposals vary from a few pixels to the whole image, and their corners fall anywhere, while a head built from fully connected layers needs a fixed-length input. R-CNN solved this by warping each image crop to $227 \times 227$ pixels and running the whole network on it, which made features expensive. SPP-net and Fast R-CNN instead crop the shared feature map and pool the crop to a fixed grid (He et al., 2014; Girshick, 2015).

Two difficulties follow. The map is coarse: at stride 16, one cell covers $16 \times 16$ pixels, so a box edge usually falls inside a cell, and a 32-pixel object spans two cells. And the pooled feature must be differentiable with respect to the map, so the backbone can learn through it, and, for some uses, with respect to the box coordinates too.

Inputs and Outputs

• Input: a feature map $F$ of $C$ channels at stride $\kappa$ (the product of the backbone’s strides), and a list of boxes in image pixels, each $(x_1, y_1, x_2, y_2)$ with an image index. Fast R-CNN writes an RoI on the feature map as $(r, c, h, w)$, its top-left corner and size (Girshick, 2015).
• Parameters: the output size $k \times k$ (Fast R-CNN writes $H \times W$), the aggregation (max or average), and for sampling-based operators the number of samples per bin and the pixel convention.
• Output: one $k \times k \times C$ tensor per box. Position-sensitive pooling outputs $k \times k$ values per class instead, and the multi-level form first chooses which pyramid level to read.

Intuition

Lay a $k \times k$ grid over the box and summarize the features under each cell of the grid, one summary per cell. The grid scales with the box, so a large box gets large bins and a small box small ones, and the output always has $k \times k$ entries. This is pooling with windows set by the box instead of by a fixed kernel and stride.

The difficulty is that the box and its bins have real-valued edges, but the map has values only at its cells. RoI pooling answers by snapping: round the box to whole cells, round each bin to whole cells, take the maximum. Snapping is cheap and exact on the grid, but every rounding can move an edge by up to half a cell, eight pixels at stride 16, and a bin of a small box may read cells that lie mostly outside it. RoIAlign answers by interpolating: keep the edges where they are and read the map at points inside each bin by bilinear interpolation, treating the map as a continuous function. Precise RoI pooling goes one step further and integrates that continuous function over each bin. Neither can add detail that the coarse map does not hold, which the example below makes visible.

Algorithm

RoI max pooling (Fast R-CNN), for one box and one channel:

1. Scale the box to feature-map coordinates by dividing by the stride, and round each coordinate to an integer cell index.
2. Take the rounded box’s height $h$ and width $w$ in cells (at least one cell each).
3. Split it into a $k \times k$ grid; bin $(i, j)$ covers rows from $\lfloor i h / k \rfloor$ to $\lceil (i+1) h / k \rceil$ and columns likewise, offset by the rounded corner. Neighbouring bins may share a row or column.
4. Output the maximum of the cells in each bin; record which cell won, for the backward pass.

RoIAlign (Mask R-CNN):

1. Scale the box by $1/\kappa$ without rounding.
2. Split it into $k \times k$ equal bins with real-valued edges.
3. In each bin, place $s \times s$ regularly spaced sample points ($s = 2$, four points, in Mask R-CNN).
4. Read the map at each point by bilinear interpolation from the four surrounding cells.
5. Average (or max) the samples of each bin.

In both, the heavy work, the backbone, runs once per image; only these steps and the head run per box.

Mathematical Formulation

Coordinates and rounding

A box edge $x$ in image pixels lies at $x / \kappa$ in feature-map units. RoI pooling replaces it by $[x / \kappa]$, where $[\cdot]$ rounds to the nearest integer, as Mask R-CNN’s authors describe it (He et al., 2017). SPP-net’s appendix in the arXiv v4 revision gives a different rule for its window mapping: with every layer padded by $\lfloor p/2 \rfloor$ for a filter of size $p$, a response at $x'$ has its receptive field centred at $\kappa x'$, and a window is mapped to cells $x' = \lfloor x/\kappa \rfloor + 1$ on the left or top and $x' = \lceil x/\kappa \rceil - 1$ on the right or bottom, so that a corner goes to the cell whose receptive-field centre is nearest (He et al., arXiv v4; the changelog says this appendix was revised for easier implementation after the journal version was accepted). The released Fast R-CNN layer computes round(x * spatial_scale) for all four coordinates and then treats the rounded right and bottom indices as inclusive, roi_width = max(roi_end_w - roi_start_w + 1, 1) (caffe-fast-rcnn, commit bcd9b4e). The rounding conventions therefore differ between papers and code, and the example measures what the inclusive end costs.

RoI pooling and its gradient

With the rounded RoI starting at cell $(\hat{y}_1, \hat{x}_1)$ and spanning $h \times w$ cells, bin $(i, j)$ for $0 \le i, j < k$ is the set of cells

$$\mathcal{R}(i, j) = \Big\{ (u, v) : \hat{y}_1 + \Big\lfloor \frac{i h}{k} \Big\rfloor \le u < \hat{y}_1 + \Big\lceil \frac{(i+1) h}{k} \Big\rceil,\; \hat{x}_1 + \Big\lfloor \frac{j w}{k} \Big\rfloor \le v < \hat{x}_1 + \Big\lceil \frac{(j+1) w}{k} \Big\rceil \Big\},$$

the bin rule of SPP-net’s implementation, which takes the floor at the left and top and the ceiling at the right and bottom, and which Fast R-CNN reuses and which R-FCN and Deformable ConvNets write out (He et al., arXiv v4; Girshick, 2015; Dai et al., 2016; Dai et al., 2017). RoI max pooling outputs $y_{ij} = \max_{p \in \mathcal{R}(i,j)} F(p)$. Girshick (2015) gives the backward pass: with $i^*(r, j)$ the input that won output $j$ of RoI $r$,

$$\frac{\partial L}{\partial x_i} = \sum_r \sum_j \big[ i = i^*(r, j) \big] \frac{\partial L}{\partial y_{rj}},$$

so a cell receives gradient from every bin, in every RoI, whose maximum it was. The output does not depend continuously on the box: moving an edge changes nothing until a rounding flips, and then the output jumps. Its derivative with respect to the box coordinates is therefore zero almost everywhere. Fast R-CNN did not need it, since its proposals were precomputed and fixed; the journal version of Faster R-CNN notes that exact joint training with network-predicted proposals would need an RoI pooling layer differentiable with respect to the box coordinates, a nontrivial problem for which it points to RoI warping (Ren et al., 2017).

RoI warping

Multi-task Network Cascades (MNC; Dai, He & Sun, 2016) needed gradients with respect to boxes predicted by an earlier stage of the network. Their RoI warping layer writes the crop-and-resize as a linear map of the feature map,

$$F^{\text{RoI}}(u', v') = \sum_{(u, v)} G(u, v; u', v' \mid B)\, F(u, v), \qquad g(u, u' \mid x, w) = \kappa_b\!\Big( x + \frac{u'}{W'} w - u \Big),$$

where $G$ factors into $g(u, u')\,g(v, v')$, $\kappa_b(t) = \max(0, 1 - |t|)$ is the bilinear kernel, $(x, w)$ are the box centre and width, and $u'$ runs over a $W' \times H'$ output grid centred on zero. The kernel is differentiable in $x$ and $w$, so the loss can pass gradients to the box. MNC warps to $28 \times 28$ and then max-pools to $7 \times 7$ for VGG-16 (Dai et al., 2016). In these equations the box enters with real-valued coordinates. Mask R-CNN’s authors, comparing with it, state that RoIWarp “was implemented in [7] as quantizing RoI just like RoIPool” (He et al., 2017), so the rounding they criticize lies in MNC’s implementation, not in the equations of the MNC paper.

RoIAlign

RoIAlign keeps $x / \kappa$ unrounded (He et al., 2017). For a box of $h \times w$ cells starting at $(y_1/\kappa, x_1/\kappa)$, bin $(i, j)$ averages $s \times s$ samples of the bilinearly interpolated map $f$:

$$r_{ij} = \frac{1}{s^2} \sum_{a=0}^{s-1} \sum_{b=0}^{s-1} f\!\left( \frac{y_1}{\kappa} + \Big(i + \frac{a + \frac12}{s}\Big) \frac{h}{k} - \delta,\; \frac{x_1}{\kappa} + \Big(j + \frac{b + \frac12}{s}\Big) \frac{w}{k} - \delta \right),$$

where $f(t_y, t_x)$ interpolates between the map values stored at integer indices. The offset $\delta$ is the pixel convention. The legacy implementation, which Detectron2 attributes to Detectron, uses $\delta = 0$, which treats cell $u$’s value as located at the cell’s corner $u$; Detectron2 uses $\delta = \frac12$ by default, which puts it at the cell’s centre $u + \frac12$, and calls this aligned=True. Its documentation explains that the legacy version “does not subtract the 0.5” and so interpolates with a slightly wrong alignment, and adds that the difference does not affect the model’s accuracy when RoIAlign is followed by convolutional layers (Detectron2 v0.6, roi_align.py). Detectron2’s pooler type ROIAlign is the legacy form and ROIAlignV2 the aligned one; torchvision’s roi_align defaults to aligned=False (torchvision v0.20.0). With $\delta = 0$, torchvision also clamps the RoI’s width and height to at least one cell, which it does not do in the aligned form.

The output is continuous in the box coordinates and differentiable almost everywhere, since the interpolation weights are piecewise linear in the sample positions; by our reasoning, the derivative jumps where a sample crosses from one cell to the next. Jiang et al. (2018) state that RoIAlign is not continuously differentiable with respect to the coordinates.

Precise RoI pooling

Precise RoI pooling (PrRoI pooling), from IoU-Net (Jiang et al., 2018), replaces the $s^2$ samples by the integral of the interpolated map over the bin. With $w_{u,v}$ the feature at cell $(u, v)$ and $f(x, y) = \sum_{u,v} \kappa_b(x - u)\,\kappa_b(y - v)\, w_{u,v}$, a bin with corners $(x_1, y_1)$ and $(x_2, y_2)$ in feature-map units pools to

$$\operatorname{PrPool}(\text{bin}, F) = \frac{\int_{y_1}^{y_2} \int_{x_1}^{x_2} f(x, y)\, dx\, dy}{(x_2 - x_1)(y_2 - y_1)}, \qquad \frac{\partial \operatorname{PrPool}}{\partial x_1} = \frac{\operatorname{PrPool}}{x_2 - x_1} - \frac{\int_{y_1}^{y_2} f(x_1, y)\, dy}{(x_2 - x_1)(y_2 - y_1)}.$$

The number of points no longer depends on a preset $s$, and the derivative with respect to each coordinate is continuous. IoU-Net uses it for its optimization-based box refinement, gradient ascent on the predicted IoU with respect to the box coordinates, and its authors prefer it to RoIAlign because of that continuity (Jiang et al., 2018). Because $f$ is piecewise bilinear, the integral has a closed form, a sum of products of one-dimensional integrals of the triangle kernel, as the example below implements.

A common form

Every operator above computes, for each bin, a weighted sum $r_{ij} = \sum_p \omega_{ij}(p; B)\, F(p)$ (or a maximum over a set of cells), with weights set by the box $B$. RoI average pooling gives equal weights to whole cells chosen by rounding; RoIAlign gives each bin $4 s^2$ bilinear weights at its sample points; PrRoI pooling gives each cell the integral of its triangle kernel over the bin. In our reading, this view explains their gradients with respect to the box: rounding makes $\omega$ piecewise constant in $B$, bilinear sampling makes it continuous but with kinks where a sample crosses a cell boundary, and integration makes it continuously differentiable.

Parameters

• Output size $k$. Set by the head that follows: Fast R-CNN replaces the network’s last max pooling layer by RoI pooling with $H$ and $W$ chosen to be compatible with the first fully connected layer, $H = W = 7$ for VGG16 (Girshick, 2015). In Mask R-CNN’s FPN head, the box branch reads $7 \times 7$ and the mask branch $14 \times 14$ RoIAlign features (He et al., 2017, Figure 3). MNC warped to $28 \times 28$ before pooling to $7 \times 7$.
• Aggregation. Fast R-CNN uses max. Mask R-CNN found RoIAlign insensitive to max or average and used average (He et al., 2017). R-FCN and Deformable ConvNets write their RoI pooling as an average (Dai et al., 2016; Dai et al., 2017).
• Samples per bin $s$. Mask R-CNN samples four points per bin and notes that a single sample at the bin centre is nearly as effective and that more than four gave diminishing returns (He et al., 2017). torchvision’s default, sampling_ratio=-1, adapts the count to the box: $\lceil h/k \rceil \times \lceil w/k \rceil$ samples, with $h$ and $w$ in feature-map cells (torchvision v0.20.0).
• Pixel convention $\delta$. Corner ($\delta = 0$) or centre ($\delta = \frac12$), as above.
• Spatial scale. $1/\kappa$, the factor that maps image coordinates to the map; in a feature pyramid each level has its own.
• Level assignment. For pyramids, the canonical size (224) and level ($k_0 = 4$) of the FPN rule, described under Variants.

Complexity

For $R$ boxes, $C$ channels and a $k \times k$ output, RoIAlign with fixed $s$ performs $4 s^2 k^2 C$ multiply-adds per box, independent of the box’s size; with adaptive sampling the cost grows with the box. RoI max pooling visits every cell of the rounded box at least once, about $hwC$ comparisons, plus the overlaps between bins. PrRoI pooling touches the cells under each bin plus a one-cell border. For typical settings, 300 to 2,000 boxes per image and $k = 7$, the pooling is cheap next to the backbone and the per-box head; the Faster R-CNN article gives timings for the whole detector. Memory is $R k^2 C$ for the output, plus, for max pooling, the argmax indices kept for the backward pass.

Implementation

The example measures how much each operator loses on boxes of different sizes. It turns the bundled camera image of scikit-image (512 × 512 grey levels) into a 32 × 32 “feature map” at stride 16 by averaging each $16 \times 16$ cell. Because the map is made from a known image, the ideal feature of any bin is known exactly: the mean of the full-resolution image over the bin’s real-valued rectangle, computed from a summed-area table. For each box size from 16 to 256 pixels (1 to 16 cells), 200 boxes with random positions and aspect ratios between 1:2 and 2:1, kept a cell away from the border, are pooled to $7 \times 7$ by six operators:

• RoI pooling with corners rounded to $[x/16]$, as Mask R-CNN describes it, and with the released Caffe layer’s inclusive end (+ 1);
• RoIWarp as Mask R-CNN describes MNC’s implementation: the rounded box, sampled bilinearly ($2 \times 2$ samples per bin, centre convention), which is our reconstruction;
• RoIAlign with $2 \times 2$ samples in the legacy ($\delta = 0$) and aligned ($\delta = \frac12$) conventions;
• PrRoI pooling, the exact integral of the interpolated map (centre convention).

All six average rather than take the maximum, so that each can be compared with an ideal that is itself an average; Fast R-CNN’s layer takes the maximum. The first table gives, for the rounding-based operators, the mean IoU between the region each bin actually reads and the bin it should read (RoIAlign’s and PrRoI’s bins coincide with the ideal ones by construction), and each operator’s mean absolute error against the ideal, in grey levels, with its standard error over the 200 boxes. The second table compares each operator with PrRoI pooling, the exact average of what the map holds. It needs NumPy 2.3.5 and scikit-image 0.25.2 and runs in a few seconds.

import numpy as np
from skimage import data

S, K = 16, 7                                   # feature stride, output grid K x K
img = data.camera().astype(float)              # 512 x 512 grey levels 0..255 (bundled)
H, W = img.shape
F = img.reshape(H // S, S, W // S, S).mean(axis=(1, 3))   # 32 x 32 "feature map": cell means

# Exact mean of the full-resolution image over real-valued rectangles (the reference).
sat = np.pad(img.cumsum(0).cumsum(1), ((1, 0), (1, 0)))
def sat_at(y, x):                              # bilinear interpolation of the SAT is exact here
    y0, x0 = np.minimum(np.floor(y).astype(int), H - 1), np.minimum(np.floor(x).astype(int), W - 1)
    dy, dx = y - y0, x - x0
    return ((1 - dy) * (1 - dx) * sat[y0, x0] + (1 - dy) * dx * sat[y0, x0 + 1]
            + dy * (1 - dx) * sat[y0 + 1, x0] + dy * dx * sat[y0 + 1, x0 + 1])
def rect_mean(y1, x1, y2, x2):
    return (sat_at(y2, x2) - sat_at(y1, x2) - sat_at(y2, x1) + sat_at(y1, x1)) / ((y2 - y1) * (x2 - x1))

def bilinear(ty, tx):                          # F at real-valued index coordinates (ty, tx)
    ty, tx = np.clip(ty, 0, F.shape[0] - 1), np.clip(tx, 0, F.shape[1] - 1)
    y0, x0 = np.minimum(ty.astype(int), F.shape[0] - 2), np.minimum(tx.astype(int), F.shape[1] - 2)
    dy, dx = ty - y0, tx - x0
    return ((1 - dy) * (1 - dx) * F[y0, x0] + (1 - dy) * dx * F[y0, x0 + 1]
            + dy * (1 - dx) * F[y0 + 1, x0] + dy * dx * F[y0 + 1, x0 + 1])

def edges(a, b):                               # K+1 bin edges of [a, b)
    return a + (b - a) * np.arange(K + 1) / K

def iou_1d(a0, a1, b0, b1):
    return np.maximum(0, np.minimum(a1, b1) - np.maximum(a0, b0))

def roi_pool(box, caffe=False):
    """Average over whole cells; returns K x K output and the per-bin read regions (cell units)."""
    q = [int(np.floor(v / S + 0.5)) for v in box]               # [x / 16]
    x1, y1, x2, y2 = q
    w = max(x2 - x1 + (1 if caffe else 0), 1)
    h = max(y2 - y1 + (1 if caffe else 0), 1)
    cx = [(x1 + int(np.floor(j * w / K)), x1 + int(np.ceil((j + 1) * w / K))) for j in range(K)]
    cy = [(y1 + int(np.floor(i * h / K)), y1 + int(np.ceil((i + 1) * h / K))) for i in range(K)]
    out = np.array([[F[a:b, c:d].mean() for (c, d) in cx] for (a, b) in cy])
    return out, cx, cy

def sampled(box, shift, s=2):
    """RoIAlign-style: s x s bilinear samples per bin of the real-valued box (cell units)."""
    x1, y1, x2, y2 = box
    ex, ey = edges(x1, x2), edges(y1, y2)
    off = (np.arange(s) + 0.5) / s
    xs = (ex[:-1, None] + off * np.diff(ex)[:, None]).ravel() - shift   # K*s sample columns
    ys = (ey[:-1, None] + off * np.diff(ey)[:, None]).ravel() - shift
    v = bilinear(ys[:, None], xs[None, :])
    return v.reshape(K, s, K, s).mean(axis=(1, 3))

def tent_int(a, b):                            # integral of the unit tent at each index over [a, b]
    G = lambda t: np.where(t <= -1, 0, np.where(t <= 0, (t + 1) ** 2 / 2,
                  np.where(t <= 1, 1 - (1 - t) ** 2 / 2, 1)))
    i = np.arange(F.shape[1])
    return G(b[:, None] - i) - G(a[:, None] - i)

def prroi(box):
    """PrRoI pooling: exact integral of the bilinear interpolant (centres at index + 0.5)."""
    x1, y1, x2, y2 = box
    ex, ey = edges(x1, x2) - 0.5, edges(y1, y2) - 0.5
    Wx, Wy = tent_int(ex[:-1], ex[1:]), tent_int(ey[:-1], ey[1:])
    return (Wy @ F @ Wx.T) / np.outer(np.diff(ey), np.diff(ex))

rng = np.random.default_rng(0)
sizes = [16, 24, 32, 48, 64, 96, 128, 192, 256]
names = ["RoIPool [x/16]", "RoIPool Caffe", "RoIWarp", "Align legacy", "Align aligned", "PrRoI"]
print("size  cells | bin IoU: Pool[x/16] PoolCaffe Warp | mean |error| in grey levels (± s.e.)")
print("            |                                   | " + "  ".join(f"{n:>13s}" for n in names))
table2 = []
for size in sizes:
    ious, errs, errs2 = [], [], []
    for _ in range(200):
        r = np.exp(rng.uniform(np.log(0.5), np.log(2)))          # aspect ratio w / h
        w, h = size * np.sqrt(r), size / np.sqrt(r)
        x1, y1 = rng.uniform(S, W - S - w), rng.uniform(S, H - S - h)
        box = (x1, y1, x1 + w, y1 + h)                           # image pixels, real-valued
        ex, ey = edges(x1, x1 + w), edges(y1, y1 + h)
        ref = rect_mean(ey[:-1, None], ex[None, :-1], ey[1:, None], ex[None, 1:])
        cb = np.array(box) / S                                    # box in cell units
        P1, cx1, cy1 = roi_pool(box)
        P2, cx2, cy2 = roi_pool(box, caffe=True)
        q = np.floor(cb + 0.5)
        qb = (q[0], q[1], max(q[2], q[0] + 1), max(q[3], q[1] + 1))    # rounded box, >= 1 cell
        outs = [P1, P2, sampled(qb, 0.5), sampled(cb, 0.0), sampled(cb, 0.5), prroi(cb)]
        errs.append([np.abs(o - ref).mean() for o in outs])
        errs2.append([np.abs(o - outs[-1]).mean() for o in outs[:-1]])   # against PrRoI
        # Geometry: IoU of each bin's read region with its ideal bin (cell units).
        bx, by = edges(cb[0], cb[2]), edges(cb[1], cb[3])
        row = []
        for cx, cy in [(cx1, cy1), (cx2, cy2)]:
            rx = np.array(cx, float); ry = np.array(cy, float)
            ix = iou_1d(rx[:, 0], rx[:, 1], bx[:-1], bx[1:])
            iy = iou_1d(ry[:, 0], ry[:, 1], by[:-1], by[1:])
            inter = np.outer(iy, ix)
            area_r = np.outer(ry[:, 1] - ry[:, 0], rx[:, 1] - rx[:, 0])
            area_b = np.outer(np.diff(by), np.diff(bx))
            row.append((inter / (area_r + area_b - inter)).mean())
        wx, wy = edges(qb[0], qb[2]), edges(qb[1], qb[3])
        ix = iou_1d(wx[:-1], wx[1:], bx[:-1], bx[1:]); iy = iou_1d(wy[:-1], wy[1:], by[:-1], by[1:])
        inter = np.outer(iy, ix)
        area_w = np.outer(np.diff(wy), np.diff(wx)); area_b = np.outer(np.diff(by), np.diff(bx))
        row.append((inter / (area_w + area_b - inter)).mean())
        ious.append(row)
    ious, errs = np.array(ious), np.array(errs)
    table2.append((size, np.mean(errs2, axis=0)))
    m, se = errs.mean(0), errs.std(0, ddof=1) / np.sqrt(len(errs))
    print(f"{size:4d} {size / S:5.1f}  |      {ious[:, 0].mean():.2f}       {ious[:, 1].mean():.2f}    "
          f"{ious[:, 2].mean():.2f} | " + "  ".join(f"{a:6.2f} ±{b:5.2f}" for a, b in zip(m, se)))

print("\nmean |difference| from PrRoI pooling (the map's own interpolant), grey levels")
print("size  " + "  ".join(f"{n:>14s}" for n in names[:-1]))
for size, m in table2:
    print(f"{size:4d}  " + "  ".join(f"{a:14.2f}" for a in m))

size  cells | bin IoU: Pool[x/16] PoolCaffe Warp | mean |error| in grey levels (± s.e.)
            |                                   | RoIPool [x/16]  RoIPool Caffe        RoIWarp   Align legacy  Align aligned          PrRoI
  16   1.0  |      0.01       0.00    0.02 |  13.45 ± 1.19   16.92 ± 1.39   13.81 ± 1.10   16.06 ± 1.35   12.37 ± 1.00   12.38 ± 1.00
  24   1.5  |      0.03       0.01    0.05 |  11.17 ± 0.95   13.91 ± 1.18   11.12 ± 0.88   13.50 ± 1.16   10.20 ± 0.80   10.21 ± 0.80
  32   2.0  |      0.05       0.02    0.09 |  10.27 ± 0.94   14.43 ± 1.10   10.16 ± 0.84   13.14 ± 1.03    9.32 ± 0.77    9.34 ± 0.77
  48   3.0  |      0.10       0.04    0.18 |   9.90 ± 0.69   13.89 ± 0.96   10.01 ± 0.66   12.93 ± 0.88    8.74 ± 0.57    8.78 ± 0.57
  64   4.0  |      0.15       0.07    0.28 |   8.83 ± 0.49   13.42 ± 0.77    8.98 ± 0.49   12.45 ± 0.71    7.68 ± 0.41    7.76 ± 0.42
  96   6.0  |      0.28       0.14    0.43 |   8.14 ± 0.39   13.90 ± 0.66    7.92 ± 0.36   12.56 ± 0.58    6.44 ± 0.30    6.62 ± 0.31
 128   8.0  |      0.35       0.20    0.51 |   7.99 ± 0.31   12.63 ± 0.52    7.12 ± 0.27   11.94 ± 0.44    5.21 ± 0.19    5.49 ± 0.20
 192  12.0  |      0.44       0.31    0.66 |   6.99 ± 0.20   11.06 ± 0.34    5.09 ± 0.15   10.38 ± 0.27    3.20 ± 0.08    3.72 ± 0.09
 256  16.0  |      0.53       0.40    0.71 |   6.38 ± 0.14    9.69 ± 0.23    4.66 ± 0.11    9.60 ± 0.16    2.23 ± 0.04    2.74 ± 0.05

mean |difference| from PrRoI pooling (the map's own interpolant), grey levels
size  RoIPool [x/16]   RoIPool Caffe         RoIWarp    Align legacy   Align aligned
  16            7.05           11.01            4.71            9.39            0.01
  24            5.93            9.38            3.63            8.33            0.02
  32            5.51           10.14            3.42            8.44            0.03
  48            5.50           10.15            4.11            9.08            0.07
  64            4.99           10.27            3.97            9.25            0.13
  96            5.41           11.49            3.93           10.05            0.30
 128            5.68           10.67            4.29           10.25            0.48
 192            5.13            9.66            3.79            9.58            0.84
 256            5.06            8.67            4.12            9.20            1.39

The geometry columns show the quantization directly. For a box one cell wide, each of RoI pooling’s bins should cover a seventh of a cell but reads at least one whole cell, so a bin and its region barely overlap (IoU 0.01, and 0.00 with the Caffe layer’s extra cell); even at 16 cells, where bins are 2.3 cells wide, the read regions overlap the ideal bins at about one half. Rounding the box but not the bins, as RoIWarp does, recovers part of the overlap but not all of it.

The feature errors tell a less obvious story. Against the full-resolution ideal, every operator is far off for small boxes, including the two exact ones: a box of one or two cells holds detail that a stride-16 map simply does not contain, and no way of reading the map can recover it. Rounding adds only about one grey level on top of that loss up to 4 cells, and its cost grows with the box, to roughly four grey levels at 16 cells, where the map does resolve structure inside the box and RoIAlign’s error keeps falling while RoI pooling’s falls more slowly. The second table separates the two sources: measured against what the map holds, RoI pooling departs by 5 to 7 grey levels at every size, slightly more for the smallest boxes, whereas aligned RoIAlign stays within a fraction of a grey level of PrRoI pooling for small boxes and drifts away only as bins grow larger than a cell and four samples under-sample them. RoIWarp sits between the two, nearer RoI pooling at every size, in line with Mask R-CNN’s finding that RoIWarp performed on par with RoI pooling. In our reading, misalignment matters most where the map holds detail at the scale of a bin; on boxes much smaller than a cell, resolution loss dominates and finer maps, which feature pyramids provide, matter more than the pooling operator.

Two results depend on choices the sources leave open. The legacy convention does worse than RoI pooling with $[x/16]$ rounding here (though slightly better than the Caffe layer’s inclusive end), because in this example’s pixel model each cell’s value belongs at the cell’s centre, so $\delta = 0$ shifts every sample by half a cell; Detectron2’s documentation notes that the shift does not matter to accuracy when convolutional layers follow; by our reasoning, learned layers can absorb a constant offset, and a learned feature has no fixed “true” location the way this block average does. And PrRoI pooling, exact with respect to the interpolated map, is slightly further from the full-resolution ideal than four-point RoIAlign on large boxes. By our reasoning, this is because the bilinear interpolant of cell averages does not preserve them: its integral over cell $u$ along one axis is $(F_{u-1} + 6F_u + F_{u+1})/8$, not $F_u$, so integrating it exactly blurs the features a little more than sampling it at four points.

The plan written before the first run expected the gap between RoI pooling and RoIAlign to shrink with box size (“gap shrinks with size”) and PrRoI pooling to do better than RoIAlign on large boxes (“PrRoI lower for large boxes”); the first run contradicted both. After it, we added the second table, the comparison with PrRoI pooling, to separate resolution loss from misalignment. Nothing else changed: the image, boxes, seed and operators are those of the first run, and the first table is as first printed.

Properties and Behavior

Training through the pooling matters. SPP-net fine-tuned only the fully connected layers after its pyramid pooling (He et al., 2014). Fast R-CNN back-propagates through RoI pooling into the convolutional layers; freezing VGG16’s thirteen convolutional layers, which emulates single-scale SPP-net training, lowered PASCAL VOC 2007 test mAP from 66.9% to 61.4% (Girshick, 2015, Table 5).

Alignment matters for pixel outputs. Mask R-CNN’s ablations, trained on COCO trainval35k and tested on minival, isolate the RoI layer (He et al., 2017, Table 2c, d):

Backbone (stride)	RoI layer	Mask AP	Mask AP$_{75}$	Box AP
ResNet-50-C4 (16)	RoIPool, max	26.9	26.4
ResNet-50-C4 (16)	RoIWarp, max / average	27.2 / 27.1	27.1 / 27.1
ResNet-50-C4 (16)	RoIAlign, max / average	30.2 / 30.3	31.8 / 31.5
ResNet-50-C5 (32)	RoIPool	23.6	21.6	28.2
ResNet-50-C5 (32)	RoIAlign	30.9	32.1	34.0

Most of the gain came at the stricter IoU of AP$_{75}$, and it grew with the stride: with stride-32 features, RoIAlign added 7.3 mask AP, so that stride-32 features with RoIAlign (30.9) beat stride-16 features with RoIAlign (30.3). With FPN, whose finest level has stride 4, the gain shrank to 1.5 mask AP and 0.5 box AP. For keypoints, where localization is more sensitive, RoIAlign raised keypoint AP from 59.8 to 64.2 even with ResNet-50-FPN (minival, Table 6). The authors conclude that alignment, not bilinear interpolation, explains the gap, since RoIWarp interpolates but rounds. The Mask R-CNN article covers the model these numbers come from. On its toy measure, the example’s second table orders the three operators the same way.

Where the RoI is pooled matters. In R-FCN’s comparisons on VOC 2007 test (ResNet-101, trained on VOC 2007+2012 trainval, no hard example mining), what the paper calls a naïve Faster R-CNN, which pooled after the last convolutional stage (conv5) and applied only a classifier per RoI, reached 68.9% mAP with a $7 \times 7$ output and 61.7% with $1 \times 1$ (Dai et al., 2016, Table 2), against the 76.4% that the paper’s text quotes from the ResNet paper for the standard ResNet detector, which inserts RoI pooling between conv4 and conv5 and runs conv5 on each RoI. The authors take this as evidence of the importance of respecting spatial information by inserting RoI pooling between layers; position-sensitive pooling is their way to keep that information without any learned layer after the RoI.

Shift behaviour. A sub-cell shift of object and box together changes RoI pooling’s output in jumps and RoIAlign’s smoothly; the two-stage object detectors article demonstrates it on a synthetic peak.

Gradients with respect to the box. Zero almost everywhere for rounding-based operators, defined almost everywhere but discontinuous at cell boundaries for bilinear sampling, and continuous for PrRoI pooling, as derived above. Faster R-CNN’s approximate joint training ignores this gradient altogether, treating the proposals as fixed inputs (Ren et al., 2017).

Limitations

• The map’s resolution caps every operator. Interpolation does not create detail finer than a cell; in the example, even exact operators stay far from the ideal on boxes of one or two cells, which suggests that for such objects a finer feature map matters more than a better pooling operator.
• Fixed grid, fixed shape. A $k \times k$ grid of axis-aligned bins treats a thin diagonal object and a square one alike, and squeezes long boxes anisotropically; deformable and rotated variants respond to this.
• Context is cut at the box. Only features inside the box are pooled, though each cell’s receptive field extends beyond it. PANet’s authors argue that context from coarser levels helps small proposals (Liu et al., 2018).
• Per-box cost. Pooling is cheap, but it fixes a per-box head after it, which R-FCN removed and one-stage detectors avoid.
• Conventions are not interchangeable. Rounding rule, inclusive or exclusive ends, the pixel convention $\delta$, the minimum RoI size and the sampling count all change outputs; so, by our reasoning, weights trained with one implementation may behave differently with another.

Variants

Spatial pyramid pooling

SPP-net (He et al., 2014) pooled each candidate window on the conv5 map (stride 16 for its ZF-5 network, in the arXiv v4 text) with a four-level pyramid of $1 \times 1$, $2 \times 2$, $3 \times 3$ and $6 \times 6$ bins, 50 bins in all, giving a 12,800-dimensional feature (256 channels × 50), with the bin rule above. RoI pooling is its one-level special case, as Girshick (2015) describes it. SPP-net also introduced the scale selection that later became level assignment: with features computed at five image scales, each window is pooled from the single scale at which its area is closest to $224 \times 224$ pixels, which the authors describe as roughly equivalent to resizing the window to $224 \times 224$ (He et al., 2014; Fast R-CNN uses the same rule for its image pyramids).

RoI warping and RoIAlign

RoI warping (MNC, 2016) introduced bilinear interpolation into region pooling to obtain gradients with respect to boxes, and RoIAlign (Mask R-CNN, 2017) removed the rounding of the box and bins. Both are described above; the image warping article covers bilinear interpolation and the backward warp it implements. The idea of a differentiable sampler appears earlier in Spatial Transformer Networks, which MNC’s authors cite as a shared motivation and distinguish from their own layer: a spatial transformer warps a whole image, while RoI warping is applied to many boxes, with a preset output size and arbitrary input sizes (Dai et al., 2016).

Precise RoI pooling

PrRoI pooling (IoU-Net, 2018) integrates the interpolated map exactly, as above. IoU-Net replaces the RoI pooling layer of its ResNet-FPN detector with it (Jiang et al., 2018).

Position-sensitive RoI pooling

R-FCN (Dai et al., 2016) moves all learned layers before the pooling. A final convolution produces $k^2 (C + 1)$ score maps, $k \times k$ for each of $C$ classes plus background, map $(i, j)$ of a class meant to respond to that part of an object (top-left, top-centre, and so on). Position-sensitive RoI pooling then pools bin $(i, j)$ of an RoI only from map $(i, j)$:

$$r_c(i, j) = \frac{1}{n} \sum_{(x, y) \in \text{bin}(i, j)} z_{i,j,c}(x + x_0, y + y_0),$$

where $(x_0, y_0)$ is the RoI’s top-left corner, $n$ the number of cells in the bin, and the bins follow the floor and ceiling rule above. The $k^2$ scores of a class vote, and a softmax over classes follows; the text says the bins vote by averaging, while the printed formula sums them, $r_c = \sum_{i,j} r_c(i, j)$ (Dai et al., 2016). A sibling bank of $4k^2$ maps, pooled the same way, gives class-agnostic box offsets. Nothing learned runs per RoI, so the per-RoI cost is negligible. Without position sensitivity ($k = 1$, a global average per RoI) R-FCN did not converge; with $3 \times 3$ it reached 75.5% and with $7 \times 7$ 76.6% mAP on VOC 2007 test, on par with the 76.4% of the standard ResNet Faster R-CNN (Table 2 and the text’s comparison, set-up as above).

RoI Transformer later used a position-sensitive RoIAlign, sampling instead of rounding, because its baseline was Light-Head R-CNN (Ding et al., 2019).

Deformable RoI pooling

Deformable ConvNets (Dai et al., 2017) add a learned offset to each bin. A first RoI pooling pass produces features from which a fully connected layer predicts normalized offsets $\Delta \hat{p}_{ij}$; these are scaled by the RoI’s size, $\Delta p_{ij} = \gamma\, \Delta \hat{p}_{ij} \circ (w, h)$ with $\gamma = 0.1$, so that the offsets are invariant to RoI size, and the bin’s cells are read at $p_0 + p + \Delta p_{ij}$ by bilinear interpolation. In the position-sensitive form, a convolution predicts offset fields over the whole image, and position-sensitive pooling of these fields gives each RoI’s offsets. With ResNet-101 on VOC 2007 test (trained on 2007+2012 trainval), deformable RoI pooling alone raised Faster R-CNN from 78.1% to 78.3% mAP at IoU 0.5 and from 62.1% to 66.6% at IoU 0.7, and the deformable position-sensitive form raised R-FCN from 80.0% to 81.2% and from 61.8% to 65.0% (Dai et al., 2017, Table 3). Note that this paper writes plain RoI pooling as an average, while Fast R-CNN’s layer takes a maximum.

Multi-level RoI assignment

A feature pyramid network offers maps at strides 4 to 32, so each RoI must first be assigned a level. Lin et al. (2017) treat the pyramid as if it came from an image pyramid and adapt the scale selection of SPP-net and Fast R-CNN: an RoI of $w \times h$ input pixels goes to level

$$k = \left\lfloor k_0 + \log_2\!\left( \sqrt{wh} / 224 \right) \right\rfloor, \qquad k_0 = 4,$$

so a $224 \times 224$ RoI is pooled from $P_4$, as in a ResNet detector that pools from C4, and halving the size moves one level finer. The feature pyramid network article works through the rule and its hard boundaries. Detectron2 implements it with a small constant inside the logarithm and clamps the level to the available range (Detectron2 v0.6, poolers.py). PANet’s adaptive feature pooling (Liu et al., 2018) instead pools each proposal from all levels with RoIAlign, passes each level’s grid through one parameter layer and fuses them by element-wise max or sum. Its authors motivate this by noting that two proposals differing by 10 pixels can go to different levels, and they report that, for small proposals that FPN would assign to the finest level, nearly 70% of the features selected by the max fusion came from higher levels.

Rotated RoI pooling

Objects in aerial images and scene text appear at every orientation, and an axis-aligned RoI around a diagonal object also holds its neighbours. Rotated variants pool from an oriented rectangle $(x_r, y_r, w_r, h_r, \theta_r)$ and output an axis-aligned $k \times k$ grid, so the features are rotation-invariant with respect to the box. RoI Transformer’s Rotated Position-Sensitive RoI Align divides the rotated RoI into $k \times k$ bins of $n \times n$ sample points and maps each point $(x, y)$, given in the RoI’s own frame, to the image by

$$\begin{pmatrix} x' \\ y' \end{pmatrix} = \begin{pmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{pmatrix} \begin{pmatrix} x - w_r/2 \\ y - h_r/2 \end{pmatrix} + \begin{pmatrix} x_r \\ y_r \end{pmatrix},$$

before bilinear interpolation (Ding et al., 2019). Oriented R-CNN’s rotated RoIAlign first turns each parallelogram proposal into an oriented rectangle by extending its shorter diagonal, then projects it to the feature map; its Eq. 5 as printed divides the width and height by the stride but floors the centre coordinates, $x_r = \lfloor x/s \rfloor$, a rounding of the kind RoIAlign removed for axis-aligned boxes (Xie et al., 2021). The angle conventions these operators assume differ between libraries, a source of silent errors that the aerial object detection article describes.