Vision Systems & Engineering

Detectron2

Facebook AI Research's PyTorch library for object detection and segmentation, covering its dataset dicts, mappers and augmentations, aspect-ratio batching, in-model normalization and padding, configs, trainers, model zoo, deployment and status as of November 2026, with a NumPy port of its resize, grouping and padding rules.

intermediate

Detectron2 is an open-source library for object detection and segmentation, written in PyTorch and released by Facebook AI Research under the Apache License 2.0. Its README calls it Facebook AI Research’s next-generation library for detection and segmentation algorithms, the successor of Detectron and maskrcnn-benchmark, and says it supports research projects and production applications at Facebook. It implements Mask R-CNN, Faster R-CNN, RetinaNet and, by its README’s list, panoptic segmentation, DensePose, Cascade R-CNN, rotated bounding boxes, PointRend, DeepLab, ViTDet and MViTv2, and it is used as a base library for research code. The README asks users to cite Wu, Kirillov, Massa, Lo and Girshick (2019).

This article describes Detectron2 from its source code and the documentation sources in its repository at commit 247691a on the main branch (27 October 2026). As of November 2026 the latest tagged release is still v0.6, published on 15 November 2021, while main continues to receive commits and the repository is not archived. The article covers the platform’s data pipeline, its model input and output formats, configs, trainers, model zoo, deployment paths and the documented differences from Detectron, and ends with an example that ports Detectron2’s training-time resizing, aspect-ratio grouping and batch padding to NumPy and measures how many pixels of a batch are padding. Detectron2 was not installed or run: the example tests our port.

Purpose

Detectron2 provides the pieces of a detection or segmentation experiment as separate, replaceable parts: a registry of datasets in a light-weight format, a mapper that loads and augments one image with its annotations, a batched data loader, meta-architectures built from registered backbones and heads, a trainer with hooks, evaluators, and export tools. Its GETTING_STARTED.md describes two command-line training scripts, tools/train_net.py and tools/plain_train_net.py, both made to train all the provided configs; plain_train_net.py has fewer default features and less abstraction, so custom logic is easier to add. The provided configs are made for 8-GPU training; for one GPU the same document shows a command that sets SOLVER.IMS_PER_BATCH 2 and SOLVER.BASE_LR 0.0025, where the base config has 16 and 0.02.

The repository’s projects/ folder holds research code built on the library: at the commit read, DeepLab, DensePose, MViTv2, Panoptic-DeepLab, PointRend, PointSup, Rethinking-BatchNorm, TensorMask, TridentNet and ViTDet. The changelog calls these projects experimental. Outside the repository, other projects are built on Detectron2: Mask2Former trains its panoptic and instance models in it, and, as the FCOS article describes, FCOS has a second implementation in AdelaiDet, which is built on Detectron2.

Core Capabilities

Datasets and metadata

A dataset is registered by name with a function that returns a list[dict], one dict per image: DatasetCatalog.register("my_dataset", fn), and DatasetCatalog.get("my_dataset") calls it. The datasets tutorial requires the function to return the same data, in the same order, each time it is called. The standard dict has file_name, height, width and image_id, plus task-specific keys:

  • annotations, for instance detection and segmentation and keypoints: a list with one dict per instance, holding bbox with its bbox_mode (BoxMode.XYXY_ABS or BoxMode.XYWH_ABS), category_id in [0,K−1][0, K-1] for KK categories (the value KK is reserved for background), segmentation as polygons or COCO RLE, keypoints and iscrowd;
  • sem_seg_file_name for semantic segmentation, and pan_seg_file_name with segments_info for panoptic segmentation.

Keypoint coordinates are real values in [0,W][0, W] and [0,H][0, H]; the tutorial notes that COCO stores keypoints as integer pixel indices in [0,W−1][0, W-1] and that Detectron2 adds 0.5 to them. Images whose annotations list is empty are removed from training by default (DATALOADER.FILTER_EMPTY_ANNOTATIONS = True, defaults.py line 128). Information shared by a whole dataset, such as thing_classes (the category names), goes in MetadataCatalog.get(name), not in each dict. The COCO article describes the annotation format these dicts resemble.

Mapper, augmentations and transforms

The data loader passes each dict to a mapper, by default DatasetMapper, which reads the image in the configured channel format, checks its shape against the dict’s width and height (setting them from the image if they are missing), applies the augmentations and converts the image to a (C, H, W) tensor (dataset_mapper.py lines 152–171; check_image_size in detection_utils.py lines 189–214). It does not change width and height, so the mapped dict still holds the original size. The data loading tutorial says that passing a different mapper, as build_detection_train_loader(cfg, mapper=...), works for most use cases of custom data loading.

The augmentation tutorial separates three objects: an Augmentation is a policy that looks at its inputs and returns a Transform; the Transform is the operation, with methods such as apply_image and apply_coords, so that the same operation can be applied to boxes, polygons, masks or a second image; and AugInput holds the inputs. The base Transform class comes from fvcore, which was not read for this article.

The default training augmentations are ResizeShortestEdge followed by a horizontal RandomFlip (detection_utils.py, build_augmentation, lines 632–656; configs/common/data/coco.py lines 20–25). ResizeShortestEdge scales the short side to a target length unless that would make the long side exceed max_size, in which case the long side is scaled to max_size; both output sizes are rounded with int(x + 0.5) (augmentation_impl.py, get_output_shape, lines 181–200). The class default for drawing the target is sample_style="range", an integer drawn uniformly from an interval (line 144, 171), but the defaults and configs use "choice" from a list: MIN_SIZE_TRAIN_SAMPLING = "choice" (defaults.py line 59), with the list (640, 672, 704, 736, 768, 800) in Base-RCNN-FPN.yaml (line 41) and MAX_SIZE_TRAIN = 1333 in defaults.py (line 61). At test time the short edge is 800 with the same cap. The resize itself, ResizeTransform.apply_image, uses PIL’s resize for uint8 images and torch.nn.functional.interpolate with align_corners=False for other dtypes (transform.py lines 117–145), so the same image resized as uint8 and as float32 goes through two different interpolation codes.

Batching and aspect-ratio grouping

build_batch_data_loader (build.py lines 294–389) divides the total batch, SOLVER.IMS_PER_BATCH (16 by default, defaults.py line 562), by the number of processes, so each GPU gets total // world_size images (line 345). The training sampler, TrainingSampler, produces an infinite stream of indices made of successive shuffles of the dataset, and each process takes every world_size-th index starting at its rank (distributed_sampler.py lines 58–70).

With DATALOADER.ASPECT_RATIO_GROUPING = True, the default (defaults.py line 119), the loader yields single mapped dicts and wraps them in AspectRatioGroupedDataset (build.py lines 359–375), which requires drop_last (line 360). The grouping has exactly two buckets: an image goes to bucket 0 if its width is greater than its height and to bucket 1 otherwise, and a batch is emitted as soon as one bucket holds a full batch (common.py lines 324–339). The code comment describes the buckets as “w > h and w < h”; by the code, a square image (w=hw = h) joins bucket 1, the portrait bucket. Because the dict’s width and height are the original sizes (above), the grouping reads the size before resizing; a shortest-edge resize keeps the orientation, so the bucket is the same. With grouping off, the loader is a plain PyTorch DataLoader with the per-GPU batch size (lines 376–389). The class docstring gives the reason for grouping: it improves training speed because the images in a batch then need less padding. The example below measures how much less.

Model input, normalization, padding and output

Every built-in model takes a list[dict], one dict per image (models tutorial). In training, a dict holds "image", a (C, H, W) tensor whose channel order is set by cfg.INPUT.FORMAT ("BGR" by default, defaults.py line 84), and "instances", an Instances object with gt_boxes, gt_classes (in [0,K)[0, K)), and gt_masks or gt_keypoints where used. For inference only "image" is required; "height" and "width" are optional and give the desired output size, which need not equal the size of the resized "image". In training the model returns a dict of losses; in inference it returns one dict per image with "instances" (pred_boxes, scores, pred_classes, and pred_masks or pred_keypoints where used), "sem_seg" or "panoptic_seg".

Normalization happens inside the model. GeneralizedRCNN.preprocess_image subtracts PIXEL_MEAN and divides by PIXEL_STD, then batches the images with ImageList.from_tensors(images, backbone.size_divisibility, ...) (rcnn.py lines 223–234). The defaults are PIXEL_MEAN = [103.530, 116.280, 123.675] and PIXEL_STD = [1, 1, 1] (defaults.py lines 42 and 46): the comments there give the mean as ImageNet’s mean pixel in BGR order and explain that for Detectron1 and MSRA pre-trained models the standard deviation has been absorbed into the first convolution’s weights, so it is set to 1; for other models they suggest ImageNet’s standard deviation. The model zoo notes that the pre-trained backbones in Detectron’s format that it lists (a ResNeXt-152 trained on ImageNet-5k and ResNets with group normalization) require slightly different settings regarding normalization and architecture. Since normalization comes first, the padding value 0 corresponds, in our reading, to a pixel equal to the mean colour.

ImageList.from_tensors takes the largest height and width in the batch, rounds each up to a multiple of size_divisibility with

Hpad=⌊Hmax⁡+s−1s⌋s,Wpad=⌊Wmax⁡+s−1s⌋s,H_{\text{pad}} = \left\lfloor \frac{H_{\max} + s - 1}{s} \right\rfloor s, \qquad W_{\text{pad}} = \left\lfloor \frac{W_{\max} + s - 1}{s} \right\rfloor s,

and copies each image into the top-left corner of a tensor filled with the pad value (image_list.py lines 88–134, the rounding at line 102). It also keeps each image’s own size (image_sizes), which GeneralizedRCNN passes on to post-processing (rcnn.py line 220). The feature pyramid network backbone sets size_divisibility to the stride of its last bottom-up input feature (fpn.py line 113) and pads to a square only if square_pad is set (default 0, line 33). That this stride is 32 for the standard R-50-FPN configs is our reading, which we did not find printed in the files read: the configs feed the FPN res2 to res5 (Base-RCNN-FPN.yaml line 8) and pool RoIs from p2 to p5 (line 23), and fpn.py names the output of a feature with stride ss “plog⁡2s\log_2 s” (line 105, docstring line 136), so res5 has stride 25=322^5 = 32. Lin et al. (2017, Section 3) give strides of 4, 8, 16 and 32 for the ResNet stages C2 to C5. Detectron2’s resnet.py was not read.

After inference, detector_postprocess rescales the predicted boxes from the resized input to the dict’s "height" and "width" (or the input size if they are absent), clips them to that size and drops empty boxes; masks are pasted at the output resolution (postprocessing.py lines 41–68; rcnn.py lines 246–248).

The detection layers themselves are owned by their articles: RoI pooling covers ROIAlign with aligned=True, the class default (roi_align.py line 8) and, as ROIAlignV2, the default pooler type (defaults.py line 308; compatibility notes), anchor boxes the anchor generator and non-maximum suppression the suppression step.

Configs: yacs and LazyConfig

Detectron2 has two config systems. The older one uses YAML files read into a yacs CfgNode (get_cfg(), merge_from_file, merge_from_list), with a _BASE_ key that loads a base file whose values the child overrides; defaults.py defines every key and its default. The configs tutorial says no new functionality or keys are added to this system since the introduction of LazyConfig. LazyConfig writes configs as Python files, loaded with LazyConfig.load into omegaconf dictionaries; objects are described by dictionaries with a _target_ key, built with LazyCall and created with instantiate, and configs can import and modify each other with ordinary Python. The configs/common/ files, such as data/coco.py and train.py, are LazyConfig building blocks; the model zoo lists configs of both kinds, trained with tools/train_net.py (YAML) or tools/lazyconfig_train_net.py (Python).

Training

The training tutorial describes two styles. A user can write the training loop directly in PyTorch, as tools/plain_train_net.py does, or use the trainer abstraction: SimpleTrainer is a minimal loop for one loss, one optimizer and one data source, with other tasks added through hooks (HookBase, with methods such as after_step), and DefaultTrainer is a SimpleTrainer built from a yacs config with default optimizer, learning-rate schedule, logging, evaluation and checkpointing. The tutorial says the trainer and hook system are deliberately kept minimal rather than powerful. Metrics go to an EventStorage and are written out by EventWriters. DetectionCheckpointer loads PyTorch .pth files and the model zoo’s .pkl files.

The base R-CNN FPN schedule has 16 images per batch, a base learning rate of 0.02 and 90,000 iterations, with the rate dropped at 60,000 and 80,000 (Base-RCNN-FPN.yaml lines 35–39); the 3x configs train for 270,000 iterations with steps at 210,000 and 250,000 (faster_rcnn_R_50_FPN_3x.yaml). The warm-up is linear over 1,000 iterations from a factor of 1/1000 (defaults.py lines 549–551). See learning rate schedules for the general form.

Model zoo

MODEL_ZOO.md documents baselines trained with Detectron2 in September and October 2019, on servers with 8 NVIDIA V100 GPUs. All COCO models were trained on train2017 and evaluated on val2017. The document warns that its default settings are not directly comparable with Detectron’s, since its default training augmentation adds scale jittering to horizontal flipping. Faster and Mask R-CNN come with three backbone combinations: FPN, C4 (a ResNet conv4 backbone with a conv5 head, the original Faster R-CNN baseline) and DC5 (dilations in conv5). The “1x” schedule is about 12 COCO epochs, which the document calls heavily under-trained, and “3x” about 37. For the R50-FPN backbone the tables list, on COCO val2017:

Model Schedule Box AP Mask AP
Faster R-CNN R50-FPN 1x 37.9 –
Faster R-CNN R50-FPN 3x 40.2 –
Mask R-CNN R50-FPN 1x 38.6 35.2
Mask R-CNN R50-FPN 3x 41.0 37.2

Box and mask AP are COCO’s average precision. A separate table of newer Mask R-CNN baselines uses large-scale jitter and 100, 200 or 400 epochs (of 118,000 COCO images each), trained from random initialization, and the document says they exceed the earlier baselines; it is a different set-up and is not compared with the table above. The zoo also lists LVIS, Cityscapes and PASCAL VOC baselines and ImageNet-pre-trained backbones, among them converted copies of MSRA’s ResNet-50 and ResNet-101. All models available through the document are licensed under Creative Commons Attribution-ShareAlike 3.0 (MODEL_ZOO.md), while the code is Apache-2.0. No weights were downloaded for this article.

Deployment

The deployment tutorial lists three export methods: tracing and scripting, which produce TorchScript for the PyTorch runtime, and caffe2_tracing, which replaces parts of the model with Caffe2 operators and can write Caffe2, TorchScript or ONNX files; the tutorial says caffe2_tracing is going to be deprecated. Traced models accept dynamic input resolution but a fixed batch size; scripted models accept a dynamic batch size. Most official GeneralizedRCNN and RetinaNet models are supported by both; Cascade R-CNN and PointRend only by tracing. Exported models often need TorchVision’s operators at run time. The PyTorch article records that TorchScript is deprecated, with deprecation warnings since PyTorch 2.10; the Detectron2 documentation read here does not discuss this, and nothing is claimed about how it affects Detectron2’s export. General serving concerns are covered in Model Deployment.

Minimal Example

Detectron2 is not installed on the machine this article was written on, and cannot be installed there. The example therefore ports four pieces of its data path at commit 247691a to NumPy 2.3.5 and Pillow 12.0.0 (Python 3.13.9): ResizeShortestEdge.get_output_shape (augmentation_impl.py lines 181–200), the uint8 branch of ResizeTransform.apply_image (transform.py lines 117–125), AspectRatioGroupedDataset (common.py lines 302–339) and the padding of ImageList.from_tensors (image_list.py lines 60–134), with the normalization of preprocess_image (rcnn.py lines 223–234). The outcomes test our port, not Detectron2. The port carries Detectron2’s copyright line, a note that it was modified and the licence’s URL at the commit.

Question. How much of a padded training batch is padding, with and without aspect-ratio grouping, and how does it change with the per-GPU batch size? The total batch of 16 gives per-GPU batches of 16, 8, 4 and 2 on 1, 2, 4 and 8 GPUs.

Data. Two sets of 480 synthetic grey uint8 images, each a horizontal gradient from 60 to 140 with one disk at 220 and one rectangle at 25, at random positions. The sizes (width × height) are landscape 640 × 480, 640 × 427, 640 × 360 and 500 × 375, portrait 480 × 640, 427 × 640 and 375 × 500, and square 512 × 512. The proportions are ours and the sources do not fix them, while the result depends on them, so two mixes are run: “mostly landscape” (96 images of each landscape size, 16 of each portrait size, 48 squares: 80 %, 10 %, 10 %) and “equal shares” (54 of each landscape size, 72 of each portrait size, 48 squares: 45 %, 45 %, 10 %). Exact counts are shuffled once with a fixed seed; that permutation is one GPU’s stream and serves every batch size, so that batch sizes differ only in batching. In Detectron2 a GPU on a run with several GPUs would see every world_size-th index of the sampler’s stream instead.

Pipeline. Each image is resized twice: for training, with a short edge drawn from (640, 672, 704, 736, 768, 800) and the 1333 cap (our seeded generator replaces Detectron2’s call to the global np.random.choice, line 173); and with the short edge fixed at 800, the test-time setting. Both use Pillow’s bilinear filter, as ResizeTransform does for uint8. Each batch is normalized with the B-channel mean 103.530 and standard deviation 1 (one grey channel stands in for three, since only height and width matter here) and padded with zeros to the largest height and width, each rounded up to a multiple of 32. The horizontal flip is left out, since it does not change an image’s size. Three arms are compared at per-GPU batch sizes 2, 4, 8 and 16:

  • (a) consecutive images from the stream, as the loader batches them with grouping off (drop_last);
  • (b) AspectRatioGroupedDataset’s two buckets, on the training resize;
  • (c) the same grouping on the fixed 800 resize, which removes the per-image short-edge draw and leaves the aspect ratios within a bucket and the rounding to 32. It is our construction for separating the two sources of padding, not a Detectron2 pipeline.

AspectRatioGroupedDataset emits a batch only when a bucket is full and never emits a partly filled bucket (common.py lines 324–339); in Detectron2 the sampler’s stream is infinite, so the remainder simply waits for more images. Our stream is finite, so whatever is left in the two buckets at its end is not batched: the left column prints these counts per bucket, and the means in such a row are over the images that were batched.

By the arithmetic of get_output_shape at short edge 800, the landscape sizes become 1067 × 800 (640 × 480 and 500 × 375), 1199 × 800 (640 × 427) and 1333 × 750 (640 × 360, capped at the long side), and the portrait sizes 800 × 1067 and 800 × 1199 (both 480 × 640 and 375 × 500 become 800 × 1067), while the square becomes 800 × 800 and shares bucket 1 with the portraits. Every short edge in the training list is a multiple of 32, so a square needs no padding on its own.

Measures and checks. M1 is the padded fraction of a batch,

M1=1−∑ihiwin HpadWpad,\text{M1} = 1 - \frac{\sum_i h_i w_i}{n\, H_{\text{pad}} W_{\text{pad}}},

printed as mean, minimum and maximum over batches. It is computed from the shapes and, independently, by counting the zeros in the padded array; since 103.530 is not an integer, no normalized image pixel can be exactly 0, so the zeros are padding. M2 is the number of batches in arm (a) that mix the two buckets. The checks:

  • C-shape: get_output_shape gives (800, 1067) for a 480 × 640 (height × width) image, (444, 1333) for a 400 × 1200 panorama, capped at the long side, and (750, 1333) for 360 × 640;
  • C-pad: one 800 × 1067 image pads to 800 × 1088;
  • C-bucket: a square goes to bucket 1, a landscape image to bucket 0, a portrait to bucket 1;
  • C-resize: every resized array has the shape get_output_shape gives (a printed count of mismatches);
  • C-data: the image counts by orientation match the planned mix, eight sizes occur, and the smallest absolute normalized pixel is above 0;
  • C-array: in every batch the padded pixels counted in the array equal nHpadWpad−∑ihiwin H_{\text{pad}} W_{\text{pad}} - \sum_i h_i w_i (a printed count of mismatching batches per row).

Hypotheses, written before any example script ran:

  • H1. Grouping (b) gives a lower mean M1 than (a) at every batch size, in both mixes.
  • H2. Mean M1 in (a) rises with batch size, from 2 to 4 to 8 to 16, in both mixes. Our reasoning: the padded size is the maximum over the batch, and the expected maximum of more images is no smaller.
  • H2b. The same reasoning applies inside a bucket, where short edges and aspect ratios still vary, so mean M1 in (b) also rises with batch size, in both mixes.
  • H3. Arm (c) gives a lower mean M1 than (b) at every batch size, in both mixes.

The comparisons are exact computations, so each margin must be above 0; margins print with six decimals and are decided unrounded.

Process. The hypotheses and checks were written before any example code existed. A planning script, a copy of the example’s functions and data generation that printed only the checks, ran on the pilot’s data before the plan was fixed; we ran it under the shell’s time command, so its elapsed time was seen before the plan was written. Every check was written before that script ran, and it met each of them. The pilot used seed 1; the first run, shown here, uses seed 0 and is otherwise the same script. Every check and hypothesis held in the pilot as well, so no outcome differs between the two. The code below is the script as first run; it runs in about 13 seconds (approximate, measured while other agents shared the machine) and printed identical output on two runs.

import os
import sys
for v in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS", "VECLIB_MAXIMUM_THREADS"):
    os.environ[v] = "1"                      # one BLAS thread, set before NumPy is imported
import numpy as np
from PIL import Image

# get_output_shape, the uint8 branch of ResizeTransform.apply_image, AspectRatioGroupedDataset
# and the padding rule of ImageList.from_tensors are ported from Detectron2, commit 247691a
# (detectron2/data/transforms/augmentation_impl.py, detectron2/data/transforms/transform.py,
# detectron2/data/common.py, detectron2/structures/image_list.py).
# Copyright (c) Facebook, Inc. and its affiliates.
# Licensed under the Apache License 2.0:
# https://github.com/facebookresearch/detectron2/blob/247691a1c292654867deb35cabb27f18d75a1363/LICENSE
# Modified (ported) by us to NumPy and Pillow, without PyTorch.

SEED = int(sys.argv[1]) if len(sys.argv) > 1 else 0
SHORT_EDGES, MAX_SIZE, TEST_EDGE, DIV = (640, 672, 704, 736, 768, 800), 1333, 800, 32
PIXEL_MEAN = np.float32(103.530)             # cfg.MODEL.PIXEL_MEAN (B channel); PIXEL_STD is 1
LAND = [(640, 480), (640, 427), (640, 360), (500, 375)]   # (width, height)
PORT = [(480, 640), (427, 640), (375, 500)]
SQUARE = (512, 512)
MIXES = {"mostly landscape": (96, 16, 48), "equal shares": (54, 72, 48)}  # per size: L, P; squares
BATCHES = (2, 4, 8, 16)


def get_output_shape(oldh, oldw, short_edge_length, max_size):   # ResizeShortestEdge
    h, w = oldh, oldw
    size = short_edge_length * 1.0
    scale = size / min(h, w)
    if h < w:
        newh, neww = size, scale * w
    else:
        newh, neww = scale * h, size
    if max(newh, neww) > max_size:
        scale = max_size * 1.0 / max(newh, neww)
        newh = newh * scale
        neww = neww * scale
    neww = int(neww + 0.5)
    newh = int(newh + 0.5)
    return (newh, neww)


def resize(img, newh, neww):                 # ResizeTransform.apply_image, uint8 branch (PIL)
    return np.asarray(Image.fromarray(img).resize((neww, newh), Image.BILINEAR))


def bucket_id(d):                            # AspectRatioGroupedDataset: 0 if w > h else 1
    return 0 if d["width"] > d["height"] else 1


def group(stream, batch_size):               # AspectRatioGroupedDataset.__iter__
    buckets, batches = [[], []], []
    for d in stream:
        bucket = buckets[bucket_id(d)]
        bucket.append(d)
        if len(bucket) == batch_size:
            batches.append(bucket[:])
            del bucket[:]
    return batches, [len(b) for b in buckets]   # incomplete buckets are never yielded


def consecutive(stream, batch_size):         # DataLoader(batch_size, drop_last=True)
    n = len(stream) // batch_size * batch_size
    return [stream[i:i + batch_size] for i in range(0, n, batch_size)], [len(stream) - n]


def pad_batch(images):                       # ImageList.from_tensors(size_divisibility=32)
    H = max(im.shape[0] for im in images)
    W = max(im.shape[1] for im in images)
    H, W = (H + DIV - 1) // DIV * DIV, (W + DIV - 1) // DIV * DIV
    batch = np.zeros((len(images), H, W), np.float32)         # pad_value 0
    for i, im in enumerate(images):
        batch[i, :im.shape[0], :im.shape[1]] = im
    return batch


def render(w, h, rng):                       # grey image: gradient, one disk, one rectangle
    yy, xx = np.mgrid[0:h, 0:w]
    img = 60 + 80 * xx / w
    cx, cy = rng.uniform(0.2, 0.8) * w, rng.uniform(0.2, 0.8) * h
    r = rng.uniform(0.1, 0.25) * min(w, h)
    img = np.where((xx - cx) ** 2 + (yy - cy) ** 2 < r * r, 220, img)
    x0, y0 = rng.integers(0, w // 2), rng.integers(0, h // 2)
    img[y0:y0 + h // 4, x0:x0 + w // 4] = 25
    return np.round(img).astype(np.uint8)


def m1(batch_dicts, key):                    # padded fraction, from shapes and from the array
    images = [(d[key].astype(np.float32) - PIXEL_MEAN) / np.float32(1.0) for d in batch_dicts]
    arr = pad_batch(images)
    n, H, W = arr.shape
    pad_shape = n * H * W - sum(im.shape[0] * im.shape[1] for im in images)
    pad_array = int(np.count_nonzero(arr == 0.0))
    return pad_shape / (n * H * W), pad_shape == pad_array


print(f"seed {SEED}")
print("C-shape:", get_output_shape(480, 640, 800, 1333), get_output_shape(400, 1200, 800, 1333),
      get_output_shape(360, 640, 800, 1333), "expected (800, 1067) (444, 1333) (750, 1333)")
print("C-pad:", pad_batch([np.ones((800, 1067), np.float32)]).shape[1:], "expected (800, 1088)")
print("C-bucket:", [bucket_id({"width": w, "height": h}) for w, h in [SQUARE, LAND[0], PORT[0]]],
      "expected [1, 0, 1]")

rng = np.random.default_rng(SEED)
for mix, (n_land, n_port, n_sq) in MIXES.items():
    sizes = [s for s in LAND for _ in range(n_land)] + [s for s in PORT for _ in range(n_port)] \
        + [SQUARE] * n_sq
    stream = [{"width": w, "height": h, "image": render(w, h, rng)} for w, h in sizes]
    stream = [stream[i] for i in rng.permutation(len(stream))]          # one GPU's stream
    bad_shape, min_abs = 0, np.inf
    for d in stream:
        h, w = d["height"], d["width"]
        for key, edge in (("train", int(rng.choice(SHORT_EDGES))), ("test", TEST_EDGE)):
            newh, neww = get_output_shape(h, w, edge, MAX_SIZE)
            d[key] = resize(d["image"], newh, neww)
            bad_shape += d[key].shape != (newh, neww)
            min_abs = min(min_abs, float(np.abs(d[key].astype(np.float32) - PIXEL_MEAN).min()))
    counts = {o: sum(f(d) for d in stream) for o, f in (
        ("landscape", lambda d: d["width"] > d["height"]),
        ("portrait", lambda d: d["width"] < d["height"]),
        ("square", lambda d: d["width"] == d["height"]))}
    planned = {"landscape": 4 * n_land, "portrait": 3 * n_port, "square": n_sq}
    per_size = sorted({(d["width"], d["height"]) for d in stream})
    print(f"\n{mix}: {counts}, as planned {counts == planned}, {len(per_size)} sizes, "
          f"C-resize shape mismatches {bad_shape}, min |normalized pixel| {min_abs:.3f}")
    print("arm batch batches images left   M1 mean    M1 min    M1 max  array!=shape  mixed")
    means = {}
    for arm, batcher, key in (("a", consecutive, "train"), ("b", group, "train"),
                              ("c", group, "test")):
        for bs in BATCHES:
            batches, left = batcher(stream, bs)
            res = [m1(b, key) for b in batches]
            fr = np.array([r[0] for r in res])
            mism = sum(not r[1] for r in res)
            mixed = sum(len({bucket_id(d) for d in b}) == 2 for b in batches)
            means[arm, bs] = fr.mean()
            mixed_txt = f"{mixed}/{len(batches)} = {mixed / len(batches):.4f}" if arm == "a" else "-"
            print(f"  {arm} {bs:5d} {len(batches):7d} {sum(map(len, batches)):6d} "
                  f"{str(left):>6} {fr.mean():9.5f} {fr.min():9.5f} {fr.max():9.5f} "
                  f"{mism:13d}  {mixed_txt}")
    for bs in BATCHES:
        d1, d3 = means["a", bs] - means["b", bs], means["b", bs] - means["c", bs]
        print(f"  H1 bs={bs:2d}: mean(a) - mean(b) = {d1:.6f} -> {'held' if d1 > 0 else 'failed'};"
              f"  H3: mean(b) - mean(c) = {d3:.6f} -> {'held' if d3 > 0 else 'failed'}")
    for arm, name in (("a", "H2 "), ("b", "H2b")):
        steps = [means[arm, BATCHES[i + 1]] - means[arm, BATCHES[i]] for i in range(3)]
        print(f"  {name} rises in ({arm}): steps " + ", ".join(f"{s:.6f}" for s in steps)
              + f" -> {'held' if all(s > 0 for s in steps) else 'failed'}")
seed 0
C-shape: (800, 1067) (444, 1333) (750, 1333) expected (800, 1067) (444, 1333) (750, 1333)
C-pad: (800, 1088) expected (800, 1088)
C-bucket: [1, 0, 1] expected [1, 0, 1]

mostly landscape: {'landscape': 384, 'portrait': 48, 'square': 48}, as planned True, 8 sizes, C-resize shape mismatches 0, min |normalized pixel| 0.470
arm batch batches images left   M1 mean    M1 min    M1 max  array!=shape  mixed
  a     2     240    480    [0]   0.16238   0.00000   0.39827             0  76/240 = 0.3167
  a     4     120    480    [0]   0.27370   0.07569   0.53332             0  70/120 = 0.5833
  a     8      60    480    [0]   0.36213   0.19977   0.55046             0  51/60 = 0.8500
  a    16      30    480    [0]   0.42350   0.27149   0.54682             0  29/30 = 0.9667
  b     2     240    480 [0, 0]   0.11588   0.00000   0.26895             0  -
  b     4     120    480 [0, 0]   0.19512   0.08990   0.29563             0  -
  b     8      60    480 [0, 0]   0.25338   0.17688   0.32364             0  -
  b    16      30    480 [0, 0]   0.28309   0.23085   0.33932             0  -
  c     2     240    480 [0, 0]   0.07396   0.00000   0.17804             0  -
  c     4     120    480 [0, 0]   0.11762   0.00000   0.26007             0  -
  c     8      60    480 [0, 0]   0.15359   0.05469   0.24620             0  -
  c    16      30    480 [0, 0]   0.16090   0.12822   0.23926             0  -
  H1 bs= 2: mean(a) - mean(b) = 0.046495 -> held;  H3: mean(b) - mean(c) = 0.041918 -> held
  H1 bs= 4: mean(a) - mean(b) = 0.078578 -> held;  H3: mean(b) - mean(c) = 0.077501 -> held
  H1 bs= 8: mean(a) - mean(b) = 0.108743 -> held;  H3: mean(b) - mean(c) = 0.099798 -> held
  H1 bs=16: mean(a) - mean(b) = 0.140408 -> held;  H3: mean(b) - mean(c) = 0.122196 -> held
  H2  rises in (a): steps 0.111321, 0.088427, 0.061375 -> held
  H2b rises in (b): steps 0.079238, 0.058262, 0.029710 -> held

equal shares: {'landscape': 216, 'portrait': 216, 'square': 48}, as planned True, 8 sizes, C-resize shape mismatches 0, min |normalized pixel| 0.470
arm batch batches images left   M1 mean    M1 min    M1 max  array!=shape  mixed
  a     2     240    480    [0]   0.20713   0.00000   0.40945             0  126/240 = 0.5250
  a     4     120    480    [0]   0.34401   0.12254   0.55417             0  102/120 = 0.8500
  a     8      60    480    [0]   0.44357   0.23805   0.55642             0  57/60 = 0.9500
  a    16      30    480    [0]   0.50718   0.41355   0.57623             0  30/30 = 1.0000
  b     2     240    480 [0, 0]   0.11040   0.00000   0.29646             0  -
  b     4     120    480 [0, 0]   0.19329   0.06446   0.38057             0  -
  b     8      60    480 [0, 0]   0.23861   0.08467   0.32285             0  -
  b    16      29    464 [8, 8]   0.26984   0.18799   0.33580             0  -
  c     2     240    480 [0, 0]   0.06962   0.00000   0.17804             0  -
  c     4     120    480 [0, 0]   0.11170   0.00000   0.20518             0  -
  c     8      60    480 [0, 0]   0.12535   0.04998   0.21875             0  -
  c    16      29    464 [8, 8]   0.13723   0.09555   0.18452             0  -
  H1 bs= 2: mean(a) - mean(b) = 0.096731 -> held;  H3: mean(b) - mean(c) = 0.040770 -> held
  H1 bs= 4: mean(a) - mean(b) = 0.150724 -> held;  H3: mean(b) - mean(c) = 0.081594 -> held
  H1 bs= 8: mean(a) - mean(b) = 0.204969 -> held;  H3: mean(b) - mean(c) = 0.113259 -> held
  H1 bs=16: mean(a) - mean(b) = 0.237343 -> held;  H3: mean(b) - mean(c) = 0.132611 -> held
  H2  rises in (a): steps 0.136889, 0.099560, 0.063607 -> held
  H2b rises in (b): steps 0.082895, 0.045315, 0.031233 -> held

Checks. All six held in both mixes. C-shape, C-pad and C-bucket held: the port’s sizes and buckets match the arithmetic of the source, including the square in bucket 1. C-resize held, with no resized array of the wrong shape, and C-data held, with the planned counts, eight sizes and a smallest normalized pixel of 0.470 (∣104−103.53∣|104 - 103.53|). C-array held in every row, so the padded fractions from the shapes and from the arrays agree exactly.

Hypotheses. H1 held at every batch size in both mixes: with grouping, a smaller share of each batch is padding, and the margin grows with the batch size in both mixes. It is larger in the equal-shares mix at each batch size, where arm (a) also has more mixed-orientation batches at each batch size; in our reading, a landscape and a portrait image in one batch force a canvas close to the larger of both dimensions, which grouping avoids. H2 held in both mixes: without grouping, the padded fraction rises at each step from 2 to 16 images per GPU, so in this set-up, running the same total batch of 16 on fewer GPUs makes each GPU’s batch carry more padding. H2b held in both mixes: the rise continues with grouping, with smaller steps than in (a) at each step. H3 held at every batch size in both mixes: fixing the short edge at 800 lowers the padded fraction further, so in this set-up the per-image draw of the short edge adds padding that grouping does not remove. Arm (c) does not reach zero, which follows from the construction stated above: within the landscape bucket, the sizes at short edge 800 differ in width (1067, 1199 and 1333) and the 1333 × 750 image is shorter, and within bucket 1 squares share batches with portraits.

The numbers depend on the mix of sizes and on the stream, which are ours; they show the mechanism of Detectron2’s batching as ported, not the padding of any real dataset.

Ecosystem and Integrations

  • PyTorch and TorchVision. INSTALL.md requires PyTorch 1.8 or later and a TorchVision that matches the installed PyTorch, and Detectron2’s layers call TorchVision operators, such as torchvision.ops.roi_align in roi_align.py. TorchVision’s own detection models, by contrast, normalize, resize and pad to a multiple of 32 inside the model, in GeneralizedRCNNTransform (torchvision/models/detection/transform.py at v0.29.2, lines 104 and 141–148), where Detectron2 resizes in the data loader and only normalizes and pads in the model.
  • Projects and downstream code. The projects/ folder (Purpose) and external projects such as Mask2Former and AdelaiDet use Detectron2 as a library; the changelog says that using it as a library is less disruptive than forking it, since API changes are much smaller in frequency and scope than code changes.
  • Export. TorchScript through tracing or scripting, and Caffe2 or ONNX through caffe2_tracing (Deployment). The compatibility notes add that most operators exist in TensorFlow and point to a conversion script in tensorpack’s Faster R-CNN.
  • Evaluation. Evaluators implement the DatasetEvaluator interface, which aggregates input and output pairs; the evaluation tutorial says the built-in ones compute metrics with standard dataset-specific APIs such as COCO’s and LVIS’s, and the LazyConfig COCO data file sets up a COCOEvaluator for the test set (configs/common/data/coco.py lines 46–48).

Alternatives

  • Detectron (Girshick et al., 2018, not read here), the Caffe2 predecessor. Its history is told in Mask R-CNN (2017 paper). Detectron2’s compatibility notes list the changes that make the two incompatible: running inference with the same weights gives different results. Box width and height are x2−x1x_2 - x_1 and y2−y1y_2 - y_1 without Detectron’s “+1”; RPN anchors are centred on feature grid points and not quantized; labels 0…K−10 \ldots K-1 are the object categories and KK is background, where Detectron used 0 for background; RoIAlign shifts RoIs by half a pixel (ROIAlignV2 is the default); and the mask pasting function differs, which the notes say can improve COCO mask AP by about 0.5 points. In training, RPN.POST_NMS_TOPK_TRAIN is per image rather than per batch, the box-regression loss is L1 instead of smooth L1, and COCO boxes and polygons are read as coordinates in [0,W][0, W], keypoints as pixel indices in [0,W−1][0, W-1].
  • maskrcnn-benchmark, which the README also names as a predecessor.
  • TorchVision’s detection models, which build Faster R-CNN, Mask R-CNN, RetinaNet and other detectors inside the PyTorch project itself (see the Faster R-CNN and Mask R-CNN articles for their models and figures).
  • MMDetection and other implementations. Detectron2’s benchmark note compares the training throughput of an end-to-end R-50-FPN Mask R-CNN with the hyperparameters of Detectron’s baseline config, which has no scale augmentation, on 8 V100 GPUs (PyTorch 1.5 for the PyTorch implementations): 62 images per second for Detectron2, 53 for mmdetection and for maskrcnn-benchmark, 50 for tensorpack, 39 for simpledet, 19 for Detectron and 14 for matterport’s Mask_RCNN, averaged over iterations 100 to 500. These are the document’s figures for that software set-up, not current ones.

Versions and Compatibility

  • Release and branch. As of November 2026 the latest release is v0.6 (15 November 2021); the repository’s main branch was last committed to on 27 October 2026 (commit 247691a, read for this article). The installation instructions build from source, python -m pip install 'git+https://github.com/facebookresearch/detectron2.git' or an editable install of a local clone, and note that Detectron2 often needs rebuilding after PyTorch is reinstalled.
  • Requirements (INSTALL.md): Linux or macOS with Python 3.7 or later; PyTorch 1.8 or later with a matching TorchVision; OpenCV only for the demo and visualization; gcc and g++ 5.4 or later to build. Most models can run inference, but not training, without GPU support, by setting MODEL.DEVICE='cpu'. Windows builds run in continuous integration, but Windows is not officially supported.
  • Stability. The changelog says that APIs listed in the API documentation are considered stable and get a deprecation warning before they break, other functions are internal, research projects are experimental, and anything documented as “default” behaviour may change as features are added. The config version has not changed since the library was open-sourced.
  • Licences. Code under the Apache License 2.0; the model zoo’s downloadable models under CC BY-SA 3.0 (MODEL_ZOO.md).

Limitations

  • No release in five years. The last tagged release is from November 2021 (as of November 2026), and the installation instructions build from main, whose state changes; in our reading, reproducing a result means recording a commit.
  • Platform scope. Training needs GPU support (INSTALL.md), Windows is not officially supported, and builds have to match the installed PyTorch and TorchVision; the installation issues section lists undefined-symbol errors and crashes when they do not.
  • Two config systems. yacs configs receive no new keys, while the newer large-scale-jitter baselines are LazyConfig Python files (configs/new_baselines/*.py, as MODEL_ZOO.md links them), so a user may meet both.
  • Conventions that change numbers. BGR input with a 0–255 mean and unit standard deviation by default; normalization and padding in the model; resizing in the data loader with rounding, and with different interpolation code for uint8 and other dtypes; boxes without the “+1”; and two aspect-ratio buckets, with squares in the portrait bucket. Pre-trained backbones in Detectron’s format need slightly different normalization and architecture settings, as MODEL_ZOO.md notes.
  • Model zoo comparisons. The baselines date from 2019, their defaults are not directly comparable with Detectron’s, and the 1x models are described in the zoo as heavily under-trained.
  • Export. Export depends on TorchScript or on caffe2_tracing, which the deployment tutorial says is going to be deprecated; batch size is fixed for traced models, and some models (Cascade R-CNN, PointRend) only support tracing.
  • Padding. Even with grouping, a batch carries padding from per-image scale jitter, from aspect ratios within a bucket and from the rounding to 32, as the example shows for its synthetic mixes.

Related

  • Faster R-CNN

    A two-stage object detector in which a region proposal network, sharing the detector's convolutional features, replaces hand-designed region proposals, making proposal generation nearly free and the whole detector one network.

  • RetinaNet

    A one-stage object detector from 2017 that predicts class scores and box offsets for dense anchors on a feature pyramid and trains its classifier with the focal loss, which down-weights the many easy background examples so that a single dense network matches two-stage accuracy.

  • RoI Pooling

    How a region-based detector turns each candidate box on a shared feature map into a fixed-size feature grid, from RoI pooling in SPP-net and Fast R-CNN through RoI warping, RoIAlign and precise RoI pooling to position-sensitive, multi-level and rotated variants.

  • Feature Pyramid Network

    A network design that turns a convolutional backbone's feature maps at strides 4 to 32 into a pyramid with strong semantic features at every scale, through a top-down pathway and lateral connections, so a detector can find small and large objects from one image scale.

  • Panoptic Segmentation

    Giving every pixel of an image a class label and, for countable objects, an instance id, so that stuff such as sky and road and things such as cars and people are segmented together without overlaps, and scoring the result with panoptic quality (PQ).

  • Deploying Vision Models

    Getting a trained vision model into use, from export and the choice of runtime to packaging the model with its preprocessing contract, serving it on a server or a device, keeping pre- and post-processing identical to training, and versioning, rolling back and monitoring it.

References

  1. Wu, Y., Kirillov, A., Massa, F., Lo, W.-Y. & Girshick, R. (2019). Detectron2. Source code.
  2. Meta AI Research. detectron2: README.md, INSTALL.md, GETTING_STARTED.md and MODEL_ZOO.md, commit 247691a (main, 27 October 2026). GitHub repository facebookresearch/detectron2 (read November 2026).
  3. Meta AI Research. detectron2: docs/tutorials (datasets.md, data_loading.md, augmentation.md, models.md, configs.md, lazyconfigs.md, training.md, deployment.md, evaluation.md) and docs/notes (compatibility.md, benchmarks.md, changelog.md), commit 247691a (main, 27 October 2026). GitHub repository facebookresearch/detectron2 (read November 2026).
  4. Meta AI Research. detectron2: detectron2/data (common.py, build.py, dataset_mapper.py, detection_utils.py, samplers/distributed_sampler.py, transforms/augmentation_impl.py, transforms/transform.py), detectron2/structures/image_list.py, detectron2/modeling (meta_arch/rcnn.py, backbone/fpn.py, postprocessing.py), detectron2/layers/roi_align.py and detectron2/config/defaults.py, commit 247691a (main, 27 October 2026). GitHub repository facebookresearch/detectron2 (read November 2026).
  5. Meta AI Research. detectron2: configs/Base-RCNN-FPN.yaml, configs/COCO-Detection/faster_rcnn_R_50_FPN_1x.yaml and _3x.yaml, configs/common/data/coco.py and configs/common/train.py, commit 247691a (main, 27 October 2026). GitHub repository facebookresearch/detectron2 (read November 2026).
  6. Meta AI Research. detectron2 releases (GitHub; latest release v0.6, 15 November 2021; read November 2026).
  7. TorchVision source code (version 0.29.2): torchvision/models/detection/transform.py.
  8. Lin, T.-Y., Dollár, P., Girshick, R., He, K., Hariharan, B. & Belongie, S. (2017). Feature Pyramid Networks for Object Detection. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 936–944.
  9. Girshick, R., Radosavovic, I., Gkioxari, G., Dollár, P. & He, K. (2018). Detectron. Source code (not read).