Vision Systems & Engineering

GPUs for Computer Vision

The graphics processing unit as a machine for vision workloads, covering SIMT execution in warps, latency hiding by many resident threads, the memory hierarchy from registers to host memory, data reuse by tiling, why convolutions suit GPUs and element-wise operations are limited by bandwidth, TF32 and reduced precision, memory capacity, host–device transfer, GPUs from other vendors and how to choose one.

intermediate

A graphics processing unit (GPU) is a processor built from many identical cores that run the same program on many data elements at once, with its own memory. Designed to accelerate computer graphics, GPUs are used to train and run vision networks because the core operations of those networks, convolutions and matrix products, are large, regular and parallel (our characterization). This article treats the GPU as a machine: how it executes threads, why it needs so many of them, how its memory is organized, which vision workloads suit it and which do not, and what to look at when choosing one. How a whole model’s speed is bounded by arithmetic and memory traffic, the roofline model, belongs to edge inference; how batching trades latency for throughput belongs to latency and throughput; 16-bit formats and Tensor Cores belong to mixed-precision training; and spreading training across several GPUs belongs to distributed training. Fixed arrays of multiply-accumulate units, another class of inference hardware, are the subject of edge AI accelerators. Writing GPU programs is the subject of CUDA and its alternatives, named here only where the hardware needs them.

Definition

A GPU is a throughput-oriented parallel processor attached to a host computer. Volkov (2016) describes GPUs as parallel processors designed for graphics, with faster memory and faster arithmetic than CPUs, that pay off only when a program offers a large degree of parallelism: serial code typically runs faster on a CPU. A GPU is therefore never used alone. The main program runs on the CPU (the host), and small functions called kernels run on the GPU (the device), each launched over a large number of threads. Numba’s documentation uses the same terms: host memory is the system’s main memory, device memory the memory on the GPU card, and a kernel a GPU function launched by the host and executed on the device.

Intuition

In our characterization, a CPU core is built to finish one sequence of instructions as soon as possible, with large caches and speculative machinery to avoid waiting for memory. A GPU is built to finish a very large number of independent pieces of work per second. It keeps thousands of threads resident at once and, whenever one group of threads waits for memory, issues instructions from another. Individual threads are not fast; the aggregate rate is.

An image suits this model well. A 1024×10241024 \times 1024 feature map with 64 channels holds about 67 million values (our arithmetic), and a convolution applies the same small computation at every position. The work divides naturally into many independent output values, which is the parallelism a GPU needs. The constraint is data: arithmetic units are fast, and they can be fed only if each value read from memory is used many times. Much of GPU programming, and much of what deep-learning libraries do on a programmer’s behalf, is arranging for that reuse.

Formal Definition

Two relations make the intuition quantitative.

Little’s law. For any concurrent process (instructions in a pipeline, memory requests in flight, warps on a multiprocessor), Volkov (2016, §3.1) uses Little’s law,

mean concurrency=mean latency×throughput,\text{mean concurrency} = \text{mean latency} \times \text{throughput},

where the three quantities are measured over the same items and the same interval. Read the other way, it gives the concurrency needed to sustain a throughput: a processor whose memory requests take LL cycles to return, and that should complete XX of them per cycle, must keep LXL X of them in flight. Since latencies on a GPU are long and peak throughputs high, the needed concurrency is large, and it comes from running many threads at once.

Traffic under tiling. Consider a matrix product C=ABC = AB with AA of size M×KM \times K and BB of size K×NK \times N, as a convolution becomes after im2col (each row of AA holds one output position’s input window). A naive kernel in which every thread computes one output and reads its operands from device memory performs MNKMNK multiply-accumulates (MACs) and 2MNK2MNK loads. A tiled kernel lets a group of threads compute a T×TT \times T block of CC: at each of ⌈K/T⌉\lceil K/T \rceil steps the group copies one T×TT \times T tile of AA and one of BB into fast on-chip memory and uses each copied value TT times. When TT divides MM, NN and KK, the loads from device memory fall to

L=2MNKT,L = \frac{2MNK}{T},

a TT-fold reduction (our derivation of the standard tiling count). When it does not, the count depends on how the ragged edge tiles are loaded. If the matrices are padded with zeros to multiples of TT, with padded sides Mp=T⌈M/T⌉M_p = T\lceil M/T \rceil and likewise NpN_p and KpK_p, every tile copy moves T2T^2 elements and

LP=2MpNpKpT.L_P = \frac{2 M_p N_p K_p}{T}.

If instead each copy loads only in-range elements and fills the rest of the tile with zeros (a bounds check), each element of AA is loaded once per column of tiles and each element of BB once per row of tiles:

LB=MK⌈NT⌉+KN⌈MT⌉.L_B = MK \left\lceil \frac{N}{T} \right\rceil + KN \left\lceil \frac{M}{T} \right\rceil .

LBL_B equals 2MNK/T2MNK/T whenever TT divides MM and NN, even if it does not divide KK; LPL_P exceeds both whenever TT fails to divide at least one of the three (both our derivation). The price of a larger tile is on-chip memory: two float32 tiles occupy 2T2×42T^2 \times 4 bytes per group of threads.

Dividing the MACs by the bytes moved to and from device memory gives what Williams, Waterman and Patterson call operational intensity (counted here in MACs): operations per byte of traffic to main memory, as distinct from traffic between the processor and its caches, as the edge inference article reports, together with the roofline built on it. Tiling raises operational intensity, and a kernel whose intensity is below the device’s ridge point is limited by memory bandwidth rather than arithmetic.

Properties

SIMT execution and warps

All instructions on NVIDIA GPUs are SIMD (single instruction, multiple data) instructions operating on vectors of 32 elements, Volkov (2016, §2.2) writes of the generations up to 2016. The programming model hides the vectors: a program is written for one scalar thread, and the hardware groups 32 threads into a warp that executes one instruction for all of them. The 32 is stable across generations as far as the sources read here go: the CUDA Samples’ histogram header defines LOG2_WARP_SIZE as 5, so WARP_SIZE is 32 (CUDA Samples v13.5), and numba 0.62.1’s source documents its cuda.warpsize as the size of a warp, adding that all architectures implemented to date have a warp size of 32. Podlozhnyuk’s 2007 whitepaper already describes warps of 32 threads on the G80.

Threads in a warp may still take different branches. Processors with this capability are said to have a SIMT (single instruction, multiple threads) architecture, a flavour of SIMD: when the threads of a warp diverge at a branch, the hardware disables some of them, runs one path, then disables the others and runs the other path, and the warp reconverges afterwards (Volkov, 2016, §2.3–2.4). The program is correct either way, but a divergent warp spends time on both paths. For vision code the consequence is that per-pixel work with data-dependent branches, such as early exits in a cascade or per-pixel loops of varying length, runs at a fraction of the rate of uniform work (our reading of the mechanism).

Warps are grouped into thread blocks. The warps of one block run on the same streaming multiprocessor (SM), access the same shared-memory unit and can be synchronized inexpensively with a barrier; different blocks access separate shared-memory partitions (Volkov, 2016, §2.10). A GPU contains many SMs, a number that varies with the price segment (§2.7), and a kernel is launched as a grid of many more blocks than can run at once.

Latency hiding and occupancy

Each warp scheduler holds a pool of resident warps and can issue from a different one every cycle; a warp whose next instruction is not ready is stalled and another is chosen, and no context switch is needed because the state of every assigned warp is equally available for issue (Volkov, 2016, §2.9). This is fine-grained multithreading; in our reading, it is how a GPU tolerates memory latencies that a CPU would cover largely with caches. The number of warps resident at the same time is the occupancy, quoted either in warps per multiprocessor or as a fraction of the maximum; how many blocks fit at once depends on the shared memory each block allocates and the registers each warp uses (§2.11). The block size chosen at launch therefore matters twice: numba’s documentation notes that it determines how many threads share an area of shared memory and that it must be large enough to fill the execution units.

Volkov argues that “latency hiding” should be understood as reaching a throughput limit, whether of the arithmetic units or of the memory system, and not as keeping the arithmetic units busy (§3.9): a memory-bound kernel whose memory system runs at its peak has hidden its latency even though its arithmetic units are mostly idle. He also finds that the concurrency needed is a number of instructions in flight, not of warps: instruction-level parallelism within a warp lets fewer warps supply it, while a mix of instruction types can require more (§3.3). In our reading, this means that more resident threads help only until some throughput limit is reached, and that a kernel can reach it with few threads if each has enough independent work.

The memory hierarchy

From fastest and smallest to slowest and largest, a GPU program sees:

  • Registers, private to each thread; Volkov describes each register as holding one 32-bit element for each of a warp’s 32 threads.
  • Shared memory, on-chip and shared by the threads of one block. Numba’s documentation calls it a manually-managed data cache, faster than device memory, allocated for the lifetime of a kernel. It is divided into banks that serve one access each at a time, so threads of a warp that hit the same bank in different words are serialized (a bank conflict); the number of banks has varied between generations (Volkov, 2016, §2.6), which is why the CUDA Samples’ histogram header keeps it as a parameter with the comment that it may change on future hardware.
  • Caches: an L1 cache in each multiprocessor and an L2 cache in each memory controller (Volkov, 2016, §2.7); in our reading, they filter some of the traffic but, unlike shared memory, are not managed by the program.
  • Device (global) memory, the large, relatively slow off-chip memory on the GPU card (numba). The memory system serves it in transactions of aligned blocks of consecutive bytes, so a warp whose 32 threads read 32 consecutive words is served by few transactions, while scattered addresses need up to one transaction each (coalescing, Volkov, 2016, §2.5). Image layouts in which neighbouring threads read neighbouring pixels coalesce; gathers through an index map, as in warping or sparse convolution, may not (our reading).
  • Host memory, the system’s main memory, reached through an interconnect and much slower to access from a kernel than device memory (our characterization).

Numba also exposes local memory, private to each thread, and constant memory, read-only, cached and off-chip.

Data reuse by tiling

Because device memory is slow compared with arithmetic, a kernel that reads every operand from it for every multiply-add cannot run near the arithmetic peak. The standard remedy is the tiling counted above: stage a block of each operand in shared memory, then let every thread of the block reuse it. The CUDA Samples’ matrixMul.cu (v13.5) is a readable example of the structure. Its header describes it as using shared memory to ensure data reuse through a tiling approach, written for clarity rather than performance; each thread block loads one BLOCK_SIZE × BLOCK_SIZE tile of each matrix into shared arrays, synchronizes, accumulates the products of the two tiles, synchronizes again and moves to the next pair of tiles. Its launch divides the matrix sides by the block size and its kernel has no bounds check, so in our reading it handles only sides that are multiples of the tile; the example below counts both ways of handling the remainder.

The same idea, keeping a working set on chip to avoid round trips to device memory, appears in other vision kernels: histograms built per block in shared memory and merged afterwards (Podlozhnyuk’s whitepaper), separable filters that stage an image tile with its border (our example), and FlashAttention, which, as the attention mechanism article describes, computes exact attention with tiling that reduces reads and writes between GPU memory levels.

Which vision workloads suit a GPU

The roofline model of the edge inference article sorts operations by operational intensity, and a GPU’s high peak arithmetic rate puts its ridge point high. Three consequences, in our reasoning:

  • Dense convolutions and matrix products reuse each input and weight many times, and they can be lowered to tiled matrix products: as the depthwise separable convolution article notes, 1×11 \times 1 convolutions map directly onto matrix multiplication and k×kk \times k ones through the im2col reordering. They can approach the arithmetic peak, and they are where Tensor Cores and reduced-precision formats pay off.
  • Element-wise operations (activations, normalization applied per element, residual additions, colour conversion, resizing) do one or a few operations per value read and written. A float32 addition of two tensors performs one operation per 12 bytes moved (our arithmetic), far below any GPU’s ridge point, so its time is set by memory bandwidth. Fusing such operations into the preceding convolution removes their round trip through memory, as the edge inference article notes for fused layers.
  • Depthwise convolutions and small layers sit in between: the depthwise separable convolution article estimates about 4 multiply-adds per element moved for a MobileNet depthwise layer against about 111 for the pointwise layer, so the depthwise stage is bandwidth-limited, and Dollár et al.’s observation, reported in latency and throughput, that GPU runtime tracks activations more closely than multiply-adds is the same effect at network scale.

Work that is irregular (per-object loops in post-processing, non-maximum suppression with data-dependent branches, small batches of small images) leaves much of a GPU idle (our reading); in Huang et al.’s detector timings, reported in latency and throughput, non-maximum suppression ran on the CPU.

Reduced precision and TF32

Recent GPUs have matrix units for reduced-precision formats, and the formats change both speed and results. FP16, BF16, the Tensor Core operation and loss scaling are treated in mixed-precision training, and integer formats in quantization. One format arises without any change to a program’s data types: TensorFloat-32 (TF32). PyTorch’s documentation (version 2.14.1, read November 2026) describes TF32 tensor cores, available on NVIDIA GPUs since Ampere, as rounding float32 inputs to 10 bits of mantissa and accumulating in FP32 precision, keeping FP32’s dynamic range; its numerical-accuracy notes say instead that only the first 10 bits of the input mantissa are read (the two notes word the input step differently), which can make even a product with the identity matrix differ from its input.

Whether TF32 is used for torch.float32 work is a setting with a history. The allow_tf32 flag introduced in PyTorch 1.7 defaulted to True from 1.7 to 1.11 and to False from 1.12 for matrix multiplications, while the cuDNN flag for convolutions defaults to True. After PyTorch 2.9, a finer fp32_precision setting per back end and operator ("ieee" or "tf32") replaces allow_tf32, which is to be deprecated, and the two styles may not be mixed. The documentation’s own A100 example measured TF32 matrix multiplication about seven times faster than FP32 with a relative error about two orders of magnitude larger against float64, and it notes that the ratio depends on the hardware generation; the numerical-accuracy notes state that most neural-network workloads converge the same with TF32 as with FP32 and recommend enabling TF32 for matrix multiplications when full float32 precision is not needed. The practical point for vision is that the same float32 PyTorch code gives different numbers on CPU and GPU, and on different PyTorch versions, and that convolutions and linear layers are governed by different flags. AMD’s MI300 also supports TF32 through hipBLASLt (PyTorch’s HIP notes); PyTorch’s numerical-accuracy notes state that its TF32 path rounds down during accumulation where NVIDIA’s rounds to nearest, giving a small negative bias.

Memory capacity

A GPU’s device memory must hold the model’s weights, the activations of the current batch (and, in training, those kept for the backward pass, as the DenseNet article measures), the optimizer state and the libraries’ workspaces. For vision the activations dominate at high resolution (our reasoning): they grow linearly with batch size and with the number of pixels (our arithmetic), so doubling an image’s side quadruples them. Capacity therefore caps the batch size and the input resolution, and when it is exceeded the options, in our summary, are smaller batches, lower resolution, reduced-precision activations, recomputation, or splitting the model across devices. The history is old: AlexNet’s network did not fit in the memory of one GPU of its time and was split across two.

Reported memory needs care. PyTorch’s caching allocator keeps freed blocks for reuse, so memory that no tensor uses still appears as used in nvidia-smi; torch.cuda.memory_allocated reports what tensors occupy and memory_reserved what the allocator holds, and empty_cache returns only unused cached memory to other applications (PyTorch documentation, CUDA semantics).

Host–device transfer and asynchronous execution

Images usually start in host memory, decoded by the CPU, and must be copied to the device. PyTorch’s documentation states that host-to-GPU copies are much faster from pinned (page-locked) memory, that a pinned tensor can be copied asynchronously with non_blocking=True so that the transfer overlaps computation, and that a DataLoader with pin_memory=True returns batches in pinned memory; it also warns that overusing pinned memory can cause serious problems when RAM runs low. The CUDA Samples’ simpleZeroCopy sample shows the other extreme, kernels reading and writing pinned system memory directly without a copy. The samples’ README describes unified virtual memory, which the CPU and GPU can both access without explicit copying, available on Linux and Windows. On systems in which CPU and GPU share one physical memory, the copy itself may be avoidable (our reading; the edge inference article covers embedded GPUs).

GPU work is asynchronous. In PyTorch, operations are queued to the device and executed later in the order queued, copies between CPU and GPU synchronize automatically, and errors from a kernel are reported only after it runs, so a stack trace may point elsewhere unless CUDA_LAUNCH_BLOCKING=1 forces synchronous execution. A timer read without synchronizing measures the launch rather than the computation; the documentation recommends torch.cuda.synchronize() or CUDA events, as latency and throughput discusses for timing in general.

GPUs from other vendors

The concepts above are not specific to NVIDIA, but the details and the software are. AMD’s GPUs are programmed through ROCm, AMD’s open-source platform, and HIP, its C++ dialect designed for porting CUDA code; PyTorch for ROCm deliberately reuses the torch.cuda interfaces, so a ROCm GPU is still addressed as device "cuda", and torch.version.hip tells the builds apart (PyTorch’s HIP notes, version 2.14.1). The official PyTorch builds also support Apple GPUs through MPS and Intel GPUs, as the PyTorch article lists. Libraries differ in which back ends they support, and features such as TF32, fused kernels or a particular runtime may exist on one vendor’s hardware only.

Examples

Counting device-memory loads under tiling

The script below models the tiled matrix product of the formal definition in NumPy and counts, from the copies its loop makes, how many elements are loaded from “device memory” into a “shared” tile buffer. It is our own model of the structure, not GPU code: no kernel runs, and nothing is copied or ported from matrixMul.cu. The outcome tests our model, not a GPU.

The workload is a 3×33 \times 3 convolution with zero padding of one pixel, written as an im2col matrix product. W1 is a synthetic 64×6464 \times 64 image with 8 channels (stripes, a disk and a rectangle per channel, integer values 0 to 7) and 16 integer filters drawn with a fixed seed from −3-3 to 33, so M=4096M = 4096 output positions, K=72K = 72 and N=16N = 16. W2 is a 30×3030 \times 30 image with 3 channels and 10 filters (M=900M = 900, K=27K = 27, N=10N = 10), whose sides are not multiples of most tile sizes. Tiles are T∈{1,2,4,8,16,32,64}T \in \{1, 2, 4, 8, 16, 32, 64\}. The operands are small integers stored as float32, so every partial sum is an integer below 2242^{24} and exact in any order; the script checks this bound, checks the im2col product against a direct correlation, and checks the tiled product against ABAB. Loads are counted under both conventions for ragged tiles: P, matrices padded with zeros in device memory, and B, bounds-checked loads. Which of the two a real kernel uses is a design choice the sources read here leave open, so both are printed. Stores count each output written once (padded outputs under P). Operational intensity is MACs (MNKMNK) per byte of loads and stores at 4 bytes per element. The shared-memory footprint 2T2×42T^2 \times 4 bytes is compared with 16 KB (16,384 bytes), the limit Podlozhnyuk’s 2007 whitepaper states for the shared memory of G8x GPUs; it is used here only as a dated, illustrative budget and says nothing about current GPUs.

The hypotheses and the closed forms above were written before any code ran, and they hold by construction: H1, that when TT divides MM, NN and KK both counts equal 2MNK/T2MNK/T; H2a, that the padded count equals 2MpNpKp/T2M_pN_pK_p/T and exceeds 2MNK/T2MNK/T in every ragged case; H2b, that the bounds-checked count equals MK⌈N/T⌉+KN⌈M/T⌉MK\lceil N/T\rceil + KN\lceil M/T\rceil and equals 2MNK/T2MNK/T exactly when TT divides MM and NN; and H3, that the footprint exceeds the budget exactly for T>45T > 45. A planning script with an independent per-tile counting loop then printed the predicted counts, the float32 bounds (398 for W1, 165 for W2) and W2’s value range of 0 to 5, before the example ran. T=64T = 64 was added at planning, before any code ran, to the batch plan’s tile sizes of 1 to 32, so that one tile exceeds the budget. The example is therefore an illustration of the derivation, not a test of it. It runs in about a second (NumPy 2.3.5, SciPy 1.16.3).

# Data reuse by shared-memory tiling, counted: a NumPy model of a tiled matrix product.
# Our own model of the tiling structure of shared-memory GEMM kernels (one T x T tile of each
# operand copied into a fast on-chip buffer per step, then a loop over the shared dimension).
# No GPU code is run, copied or ported; the counts test our model, not a GPU.
import os
os.environ["OMP_NUM_THREADS"] = "1"          # one BLAS thread, set before NumPy is imported
os.environ["OPENBLAS_NUM_THREADS"] = "1"
os.environ["MKL_NUM_THREADS"] = "1"
os.environ["VECLIB_MAXIMUM_THREADS"] = "1"
import math
import numpy as np
from scipy.signal import correlate


def make_image(h, w, c):
    """Synthetic integer image (h, w, c): stripes (0/2) + a disk (+3) + a rectangle (+2)."""
    yy, xx = np.mgrid[0:h, 0:w]
    img = np.zeros((h, w, c), dtype=np.int64)
    for ch in range(c):
        stripes = ((xx // (2 + ch)) % 2) * 2
        cy, cx, r = (h * (ch + 1)) // (c + 1), (w * (c - ch)) // (c + 1), max(3, h // 6)
        disk = ((yy - cy) ** 2 + (xx - cx) ** 2 <= r * r) * 3
        r0, c0 = h // 8 + ch, (ch * w) // (2 * c)
        rect = ((yy >= r0) & (yy < r0 + h // 4) & (xx >= c0) & (xx < c0 + w // 5)) * 2
        img[:, :, ch] = stripes + disk + rect
    return img


def im2col(img):
    """Rows: output pixels (y, x); columns: (dy, dx, c) of a 3x3 window, zero padding 1."""
    h, w, c = img.shape
    p = np.pad(img, ((1, 1), (1, 1), (0, 0)))
    cols = [p[dy:dy + h, dx:dx + w, :] for dy in range(3) for dx in range(3)]
    return np.concatenate(cols, axis=2).reshape(h * w, 9 * c)


def tiled_matmul(A, B, T):
    """C = A @ B with T x T tiles. Every output tile copies one A tile and one B tile per step
    into its 'shared' buffers; loads are counted from those copies under two conventions:
    P, matrices zero-padded in device memory (every copy moves T*T elements), and
    B, bounds-checked (only in-range elements are loaded, the rest of the buffer is zero)."""
    M, K = A.shape
    N = B.shape[1]
    nbi, nbj, nbk = math.ceil(M / T), math.ceil(N / T), math.ceil(K / T)
    Mp, Np, Kp = nbi * T, nbj * T, nbk * T
    Ap = np.zeros((Mp, Kp), np.float32); Ap[:M, :K] = A
    Bp = np.zeros((Kp, Np), np.float32); Bp[:K, :N] = B
    Va = np.zeros((Mp, Kp), bool); Va[:M, :K] = True       # in-range marks
    Vb = np.zeros((Kp, Np), bool); Vb[:K, :N] = True
    blkA = lambda X: X.reshape(nbi, T, nbk, T).transpose(0, 2, 1, 3)   # [bi, bk] -> tile
    blkB = lambda X: X.reshape(nbk, T, nbj, T).transpose(0, 2, 1, 3)   # [bk, bj] -> tile
    Ab, Bb, VAb, VBb = blkA(Ap), blkB(Bp), blkA(Va), blkB(Vb)
    acc = np.zeros((nbi, nbj, T, T), np.float32)
    loads_p = loads_b = 0
    for bk in range(nbk):                                   # steps along the shared dimension
        shA = np.empty((nbi, nbj, T, T), np.float32); shA[:] = Ab[:, bk][:, None]
        shB = np.empty((nbi, nbj, T, T), np.float32); shB[:] = Bb[bk][None, :]
        inA = np.empty((nbi, nbj, T, T), bool); inA[:] = VAb[:, bk][:, None]
        inB = np.empty((nbi, nbj, T, T), bool); inB[:] = VBb[bk][None, :]
        loads_p += shA.size + shB.size
        loads_b += int(inA.sum()) + int(inB.sum())
        acc += np.matmul(shA, shB)                          # multiply the tiles, accumulate
    Cp = acc.transpose(0, 2, 1, 3).reshape(Mp, Np)          # write each output tile once
    stores_p = Cp.size
    stores_b = int(np.outer(Va[:, 0], Vb[0, :]).sum())
    return Cp[:M, :N], loads_p, loads_b, stores_p, stores_b


def closed_forms(M, N, K, T):
    nbi, nbj, nbk = math.ceil(M / T), math.ceil(N / T), math.ceil(K / T)
    Mp, Np, Kp = nbi * T, nbj * T, nbk * T
    return 2 * Mp * Np * Kp // T, M * K * nbj + K * N * nbi, Mp * Np, M * N


rng = np.random.default_rng(0)
images = {"W1": make_image(64, 64, 8), "W2": make_image(30, 30, 3)}
weights = {"W1": rng.integers(-3, 4, size=(3, 3, 8, 16)), "W2": rng.integers(-3, 4, size=(3, 3, 3, 10))}
TILES = [1, 2, 4, 8, 16, 32, 64]
BUDGET = 16 * 1024          # bytes: the G8x shared-memory limit of the 2007 whitepaper, dated
fails = {k: 0 for k in ["D1", "D2", "C0", "C1", "C2", "H1", "H2a", "H2b", "H3"]}

for name in ["W1", "W2"]:
    img, w = images[name], weights[name]
    A = im2col(img).astype(np.float64)
    B = w.reshape(-1, w.shape[3]).astype(np.float64)
    M, K = A.shape
    N = B.shape[1]
    ok = img.dtype.kind == "i" and img.min() >= 0 and img.max() <= 7 and w.min() >= -3 and w.max() <= 3
    fails["D1"] += not ok
    bound = (np.abs(A) @ np.abs(B)).max()
    fails["D2"] += not bound < 2 ** 24
    ref = np.zeros((img.shape[0], img.shape[1], N))
    for o in range(N):
        for c in range(img.shape[2]):
            ref[:, :, o] += correlate(img[:, :, c].astype(np.float64), w[:, :, c, o].astype(np.float64),
                                      mode="same", method="direct")
    C = A @ B
    d1 = np.abs(C.reshape(ref.shape) - ref).max()
    fails["C1"] += d1 != 0
    print(f"{name}: image {img.shape} values {img.min()}..{img.max()}, weights {w.shape} values "
          f"{w.min()}..{w.max()} (D1 {'held' if ok else 'failed'}); M={M} K={K} N={N}; "
          f"max sum|a||b| = {bound:.0f} < 2^24 (D2 {'held' if bound < 2**24 else 'failed'}); "
          f"max |im2col - direct| = {d1} (C1)")
    print(f"   T  loads P    closed P   loads B    closed B   2MNK/T       stores P/B   "
          f"MAC/B P  MAC/B B  max|C-AB|  smem B")
    for T in TILES:
        Ct, lp, lb, sp, sb = tiled_matmul(A, B, T)
        LP, LB, SP, SB = closed_forms(M, N, K, T)
        d0 = np.abs(Ct.astype(np.float64) - C).max()
        fails["C0"] += d0 != 0
        fails["C2"] += (sp != SP) or (sb != SB)
        two = 2 * M * N * K                                   # compare L*T with 2MNK as integers
        divides = M % T == 0 and N % T == 0 and K % T == 0
        if divides:
            fails["H1"] += not (lp == lb and lp * T == two)
        fails["H2a"] += (lp != LP) or (not divides and not lp * T > two)
        fails["H2b"] += (lb != LB) or ((lb * T == two) != (M % T == 0 and N % T == 0)) or lb * T < two
        foot = 2 * T * T * 4
        fails["H3"] += (foot > BUDGET) != (T > 45)
        macs = M * N * K
        print(f"  {T:2d} {lp:9d} {LP:9d} {lb:9d} {LB:9d} {two / T:11.2f} {sp:7d}/{sb:<6d}"
              f" {macs / (4 * (lp + sp)):7.3f} {macs / (4 * (lb + sb)):7.3f} {d0:9.1f} {foot:6d}"
              f"{' > budget' if foot > BUDGET else ''}{'  (T divides M, N, K)' if divides else ''}")
print(f"largest T whose two float32 tiles fit in {BUDGET} bytes: "
      f"{max(t for t in range(1, 200) if 8 * t * t <= BUDGET)}")
for k, v in fails.items():
    print(f"{k}: {'held' if v == 0 else 'failed'} ({v} failing rows)")

Output (identical in two runs):

W1: image (64, 64, 8) values 0..7, weights (3, 3, 8, 16) values -3..3 (D1 held); M=4096 K=72 N=16; max sum|a||b| = 398 < 2^24 (D2 held); max |im2col - direct| = 0.0 (C1)
   T  loads P    closed P   loads B    closed B   2MNK/T       stores P/B   MAC/B P  MAC/B B  max|C-AB|  smem B
   1   9437184   9437184   9437184   9437184  9437184.00   65536/65536    0.124   0.124       0.0      8  (T divides M, N, K)
   2   4718592   4718592   4718592   4718592  4718592.00   65536/65536    0.247   0.247       0.0     32  (T divides M, N, K)
   4   2359296   2359296   2359296   2359296  2359296.00   65536/65536    0.486   0.486       0.0    128  (T divides M, N, K)
   8   1179648   1179648   1179648   1179648  1179648.00   65536/65536    0.947   0.947       0.0    512  (T divides M, N, K)
  16    655360    655360    589824    589824   589824.00   65536/65536    1.636   1.800       0.0   2048
  32    786432    786432    442368    442368   294912.00  131072/65536    1.286   2.323       0.0   8192
  64   1048576   1048576    368640    368640   147456.00  262144/65536    0.900   2.717       0.0  32768 > budget
W2: image (30, 30, 3) values 0..5, weights (3, 3, 3, 10) values -3..3 (D1 held); M=900 K=27 N=10; max sum|a||b| = 165 < 2^24 (D2 held); max |im2col - direct| = 0.0 (C1)
   T  loads P    closed P   loads B    closed B   2MNK/T       stores P/B   MAC/B P  MAC/B B  max|C-AB|  smem B
   1    486000    486000    486000    486000   486000.00    9000/9000     0.123   0.123       0.0      8  (T divides M, N, K)
   2    252000    252000    243000    243000   243000.00    9000/9000     0.233   0.241       0.0     32
   4    151200    151200    133650    133650   121500.00   10800/9000     0.375   0.426       0.0    128
   8    115712    115712     79110     79110    60750.00   14464/9000     0.467   0.689       0.0    512
  16     58368     58368     39690     39690    30375.00   14592/9000     0.833   1.248       0.0   2048
  32     59392     59392     32130     32130    15187.50   29696/9000     0.682   1.477       0.0   8192
  64    122880    122880     28350     28350     7593.75   61440/9000     0.330   1.627       0.0  32768 > budget
largest T whose two float32 tiles fit in 16384 bytes: 45
D1: held (0 failing rows)
D2: held (0 failing rows)
C0: held (0 failing rows)
C1: held (0 failing rows)
C2: held (0 failing rows)
H1: held (0 failing rows)
H2a: held (0 failing rows)
H2b: held (0 failing rows)
H3: held (0 failing rows)

Every check and hypothesis held: D1 and D2 (the data and the exactness bound), C1 and C0 (the im2col product is exact in both workloads and the tiled product in all 14 rows), C2 (the stores), H1, H2a, H2b and H3. The counts follow the closed forms row by row.

The rows where TT divides all three sides show the 1/T1/T law directly: in W1, loads halve with every doubling of TT from 1 to 8, and operational intensity roughly doubles, from 0.124 to 0.947 MACs per byte. At T=1T = 1, the naive kernel, each MAC needs two 4-byte loads, so the intensity is close to 1/8 (our arithmetic; the stores lower it slightly). Even at T=64T = 64, with bounds-checked loads, W1 reaches only 2.7 MACs per byte, because N=16N = 16 filters give each row of AA only 16 uses: once a single tile spans all of NN, AA is read once in total and further growth of TT only reduces the reloading of BB (from the closed form, MK⌈N/T⌉MK\lceil N/T \rceil stops falling at T≥NT \ge N). A layer with more output channels has more reuse to capture (our reading of the closed form).

The two conventions separate as soon as the tiles are ragged. Under bounds-checked loads the count keeps falling with TT in both workloads. Under padding it rises again past T=16T = 16, and the intensity peaks at T=16T = 16 (1.636 MACs per byte in W1, 0.833 in W2) and then falls, because the closed form 2MpNpKp/T2M_pN_pK_p/T charges the padded sides: in W1 at T=64T = 64, Np=64N_p = 64 and Kp=128K_p = 128 against N=16N = 16 and K=72K = 72, so most of each tile is zeros that are loaded and multiplied anyway (our arithmetic from the construction). W1 at T=16T = 16 also shows the case the derivation predicts, a KK that the tile does not divide: the padded count is above 2MNK/T2MNK/T, while the bounds-checked count equals it exactly, since a ragged KK only shortens the last step. The footprint grows as T2T^2: at T=64T = 64 the two tiles take 32,768 bytes, more than the 16 KB budget, and the largest square float32 tile pair that fits is T=45T = 45. Under that dated budget, tiles larger than 45 cannot be staged at all, and large tiles also leave room for fewer resident blocks (Volkov’s occupancy, above), so in our reading the best tile is a compromise between reuse, padding and on-chip capacity.

Choosing a GPU

A checklist, in our framing, for picking hardware for a vision workload; each item points to where it is treated:

  • Memory capacity. Does the model, at the needed batch size and resolution, fit with its activations and workspaces? For training, add the activations kept for the backward pass and the optimizer state.
  • Memory bandwidth against arithmetic rate. Workloads dominated by element-wise operations, depthwise convolutions or small batches are limited by bandwidth; large convolutions and matrix products by arithmetic. The ridge point of the roofline says which number matters.
  • Supported formats. Which reduced-precision formats the matrix units accelerate (TF32, FP16, BF16, INT8 and lower), and whether the model tolerates them (mixed-precision training, quantization).
  • Software stack. Whether the frameworks, inference runtimes and libraries the project needs support the vendor and generation (PyTorch builds, model deployment runtimes), and the driver and toolkit versions they require.
  • Host side. Whether the CPU can decode and preprocess images fast enough to keep the GPU fed, and how data reaches the device (real-time vision pipelines).
  • Power, heat and form factor for embedded and edge systems (edge inference), and the interconnect between GPUs when training spans several (distributed training).

In our view, peak operation counts on a datasheet answer none of these alone; measured throughput on the actual model, at the actual batch size and precision, does.

Common Misconceptions

  • “More cores means proportionally faster.” Only for work that is parallel, uniform and has enough reuse. Divergent branches run both paths, scattered memory accesses waste transactions, and bandwidth-bound operations do not speed up with more arithmetic.
  • “High occupancy is the goal.” Occupancy is a means. Volkov (2016) shows that the needed concurrency is a number of instructions in flight, which instruction-level parallelism can supply, and that latency is hidden once a throughput limit is reached; a kernel that already saturates memory bandwidth gains nothing from more warps.
  • “Float32 on a GPU is float32.” With TF32 enabled, which PyTorch 2.14.1’s notes give as the default for convolutions on Ampere and later NVIDIA GPUs, matrix units read only 10 mantissa bits of each input.
  • “The time between two Python lines is the GPU time.” GPU calls return before the work is done; without synchronization the measurement covers only the launch.
  • “nvidia-smi shows what the model needs.” A caching allocator keeps freed memory reserved, so the reported figure includes cache.
  • “Bigger tiles are always better.” Tiles reduce traffic as 1/T1/T only while they fit on chip and divide the problem; padded ragged tiles waste loads, as the example shows.

Where It Is Used

GPUs are used to train vision networks, from convolutional neural networks (AlexNet was trained on two) to vision transformers, and to run them in data centres for image classification, object detection and semantic segmentation at scale; distributed training spreads one training run across many of them. At the edge, embedded GPUs run inference on robots, vehicles and drones under the limits described in edge inference; numba’s documentation lists the Jetson Nano, Orin Nano, TX1, TX2, Xavier NX, AGX Xavier and AGX Orin among the embedded platforms its CUDA target supports. Classical vision kernels, such as filtering, warping, histograms and stereo matching, map onto the same execution model, and libraries such as OpenCV provide GPU versions of some of them. Inference runtimes compile models for specific GPUs, as model deployment describes, and PyTorch exposes the device, its streams and its allocator to Python.

Related

  • Edge Inference

    Running a vision model on the device that captures the images, such as a phone, camera, robot or microcontroller, instead of in a data centre, covering why it is done, the limits of compute, memory, power and heat, the classes of hardware and runtimes, the roofline model for finding memory-bound layers, split and offloaded computation, and how latency and energy are measured.

  • PyTorch

    An open-source Python library for building and training neural networks with tensors, automatic differentiation and GPU acceleration, run eagerly as ordinary Python code and optionally compiled or exported for deployment.

  • Model Quantization

    Storing and computing a trained network's weights and activations in low-precision formats such as INT8, INT4 or FP8, with a scale and zero point per tensor, channel or group; the integer arithmetic behind it, post-training calibration, quantization-aware training with the straight-through estimator, and what it costs in accuracy.

  • Distributed Training

    Training one neural network on many devices at once, by splitting the batch (data parallelism), the training state (sharded data parallelism, ZeRO and FSDP) or the model itself (tensor and pipeline parallelism), so that gradients match those of training on a single device while memory and communication are divided.

  • Deploying Vision Models

    Getting a trained vision model into use, from export and the choice of runtime to packaging the model with its preprocessing contract, serving it on a server or a device, keeping pre- and post-processing identical to training, and versioning, rolling back and monitoring it.

  • Convolutional Neural Network

    A neural network built from convolutional layers, small learned filters applied at every image position, interleaved with nonlinearities and downsampling, so that features grow from edges to object parts while the parameter count stays independent of image size.

References

  1. Volkov, V. (2016). Understanding Latency Hiding on GPUs. PhD dissertation, University of California, Berkeley, Technical Report No. UCB/EECS-2016-143.
  2. PyTorch. CUDA semantics, docs/source/notes/cuda.md (repository pytorch/pytorch, tag v2.14.1; BSD-style licence; read November 2026).
  3. PyTorch. Numerical accuracy, docs/source/notes/numerical_accuracy.md (repository pytorch/pytorch, tag v2.14.1; read November 2026).
  4. PyTorch. HIP (ROCm) semantics, docs/source/notes/hip.rst (repository pytorch/pytorch, tag v2.14.1; read November 2026).
  5. Numba 0.62.1. CUDA documentation: Overview, Writing CUDA Kernels and Memory management (docs/source/cuda, tag 0.62.1; BSD-2-Clause), and the warpsize docstring in numba/cuda/stubs.py (installed 0.62.1 source).
  6. NVIDIA. CUDA Samples v13.5 (4 November 2026, commit 0ec046d): cpp/0_Introduction/matrixMul/matrixMul.cu (BSD-3-Clause file header; the repository licence also refers to the CUDA EULA, not read), cpp/2_Concepts_and_Techniques/histogram/histogram_common.h, cpp/0_Introduction/simpleZeroCopy/README.md and README.md. Read for description only; no code copied.
  7. Podlozhnyuk, V. (2007). Histogram calculation in CUDA. NVIDIA whitepaper, version 1.1.1 of 9 November 2007 (cover dated September 2013), in NVIDIA/cuda-samples v13.5, cpp/2_Concepts_and_Techniques/histogram/doc/histogram.pdf.
  8. Volkov, V. & Demmel, J. W. (2008). Benchmarking GPUs to tune dense linear algebra. ACM/IEEE Conference on Supercomputing (SC '08) (not read).
  9. NVIDIA. CUDA C++ Programming Guide, CUDA C++ Best Practices Guide and cuDNN documentation (not read).