Vision Systems & Engineering

CUDA

NVIDIA's platform and programming model for general-purpose computing on its GPUs, covering kernels and their launch, the grid, block and thread hierarchy, grid-stride loops, host, device, shared and pinned memory, barriers, atomics, streams, events and CUDA graphs, the toolkit and its libraries, the Python routes and the alternatives, with a privatized image histogram run in numba's CUDA simulator.

advanced

CUDA is NVIDIA’s platform for running general-purpose programs on its GPUs. In summary, it is a programming model in which a host program on the CPU launches functions called kernels over very many GPU threads, a C++ dialect and compiler for writing them, runtime and driver interfaces for managing devices, memory and work queues, and a set of libraries built on top. In computer vision it is, in our reading, mostly invisible: PyTorch, cuDNN-backed convolutions, inference runtimes and OpenCV’s GPU modules use it underneath. It becomes visible when a pre- or post-processing step, a custom operator or a classical algorithm has no library kernel and has to be written. This article covers the programming model and the software around it. The hardware it runs on (warps and SIMT execution, latency hiding, the memory hierarchy, tiling, TF32 and host–device transfer) is the subject of GPUs for computer vision, which this article builds on.

The sources here are limited to what can be read on GitHub: numba 0.62.1’s CUDA documentation and source, the CUDA Samples at v13.5 and the whitepaper they contain, PyTorch 2.14.1’s notes, and the READMEs of numba-cuda, CuPy and Triton. NVIDIA’s own documentation sites were not used, so the CUDA C++ Programming Guide, the Best Practices Guide and the cuDNN documentation are cited “(not read)”, and this article states no compute-capability table, per-multiprocessor limit, shared-memory bank count or occupancy rule from them.

Purpose

A GPU is fast only on work that offers a great deal of parallelism, as the GPUs article explains. CUDA’s purpose is to let a programmer express such work as one scalar function run by many threads. Numba’s documentation summarizes the terms: the host is the CPU and its host memory the system’s main memory; the device is the GPU and its device memory the memory on the card; a kernel is a GPU function launched by the host and executed on the device; a device function runs on the device and can be called only from a kernel or another device function. Numba’s documentation also notes that most of the CUDA facilities it exposes map directly onto NVIDIA’s CUDA C language; in our reading, the model below therefore carries over between kernels written in C++ and in Python.

For vision work, writing CUDA directly pays off, in our reading, in three situations: an operation that no library provides (a custom loss, a non-standard resampling, a voting or accumulation step such as a histogram or Hough transform); fusing several small element-wise steps into one kernel so that intermediate images are not written to device memory, which inference optimization treats in general; and keeping a whole pipeline on the device so that frames are not copied back to the host between steps.

Core Capabilities

Kernels and their launch

A kernel is declared once and launched with an execution configuration: a number of blocks and a number of threads per block, whose product is the number of threads launched (numba’s documentation). In CUDA C++ the configuration is written between triple angle brackets, vecAdd<<<blocks, threads>>>(A, B, C, n) in the CUDA Samples’ vectorAdd.cu; numba indexes the compiled kernel instead, kernel[blocks, threads](args). A kernel cannot return a value: every result is written to an array passed to it, so a scalar result needs a one-element array (numba). A kernel is compiled once and can be launched many times with different block and grid sizes.

Launches are asynchronous: they queue the kernel’s execution on the device and return immediately, and the host waits with a synchronization call such as cuda.synchronize() in numba or cudaDeviceSynchronize() in vectorAdd.cu (numba’s documentation; the sample). Numba adds a qualification: passing an array that lives in host memory makes numba copy it back after the kernel, synchronously, so such a launch appears to run synchronously. The consequence of asynchrony that PyTorch’s notes stress applies to every CUDA program: an error in a kernel is reported only after the kernel runs, so the stack trace may point elsewhere, and setting CUDA_LAUNCH_BLOCKING=1 makes execution synchronous for debugging.

The thread hierarchy and a thread’s index

A launch is a grid of blocks, each a group of threads (numba’s documentation). Every thread runs the whole kernel and finds its work from built-in variables: threadIdx (its index within the block), blockIdx (its block’s index within the grid), blockDim (the block’s shape, the same for every block) and gridDim (the grid’s shape). Each has x, y and z components, and grids and blocks may be one-, two- or three-dimensional; numba’s documentation notes that a multidimensional configuration changes neither efficiency nor behaviour compared with a one-dimensional one of the same size, but can express an image algorithm more naturally.

The flattened global index of a thread in a one-dimensional launch is

i=blockIdx.x⋅blockDim.x+threadIdx.x,i = \texttt{blockIdx.x} \cdot \texttt{blockDim.x} + \texttt{threadIdx.x},

written threadIdx.x + blockIdx.x*blockDim.x in vectorAdd.cu and returned by cuda.grid(1) in numba, whose cuda.gridsize(1) gives the total number of threads. Because the number of blocks is rounded up, the last block can have threads beyond the end of the data, and every kernel must check its index against the array’s size; numba’s documentation says this check is required unless the block and grid sizes divide the array size, and vectorAdd.cu performs it with if(workIndex < vectorLength). For an image, a two-dimensional launch with blocks of, say, 16×1616 \times 16 threads maps one thread to each pixel, and cuda.grid(2) returns its two indices (numba’s example uses this block shape); which index is the row and which the column is the programmer’s convention, and keeping neighbouring threads on neighbouring pixels of a row is what lets their memory accesses coalesce, as the GPUs article explains.

The block is the unit of cooperation. Threads of one block share an area of shared memory and can synchronize with a barrier (numba’s documentation). The ordinary barrier does not reach across blocks; the samples’ README describes multi-block cooperative groups, available on Pascal and later GPUs, as the extension that expresses synchronization between thread blocks. In our reading, a kernel should therefore not assume any order in which its blocks run, and the numba simulator, which runs them one at a time, would not expose such an assumption. Numba’s documentation gives the two reasons the block size matters: in software it decides how many threads share one block’s shared memory, and in hardware it must be large enough to occupy the execution units, for which it defers to NVIDIA’s guide.

Grid-stride loops

Launching one thread per element ties the launch to the data size. The alternative used by both histogram kernels in the CUDA Samples is a grid-stride loop: each thread starts at its global index and advances by the total number of threads in the grid until it passes the end,

i,  i+G,  i+2G,  …with G=blockDim.x⋅gridDim.x,i,\; i + G,\; i + 2G,\; \dots \quad\text{with } G = \texttt{blockDim.x} \cdot \texttt{gridDim.x},

as in histogram256.cu’s pos += UMUL(blockDim.x, gridDim.x) and the stride = blockDim.x * gridDim.x of parallelHistogram.py (v13.5). The grid can then be sized for the device rather than for the image (histogram256.cu always launches 240 blocks, parallelHistogram.py at most 1,024), the bounds check is part of the loop condition, and a fixed number of blocks makes per-block work, such as the partial histograms below, a fixed amount. In numba the loop is for i in range(cuda.grid(1), n, cuda.gridsize(1)).

Memory spaces and host–device transfer

A CUDA program manages several memory spaces explicitly; the hardware side, and why each exists, is in the GPUs article.

  • Device (global) memory, the large off-chip memory on the GPU, visible to every thread of every kernel. In numba, cuda.to_device copies a host array to the device, cuda.device_array allocates an uninitialized device array, and a device array’s copy_to_host copies it back; numba’s documentation recommends this explicit control because its automatic transfer of NumPy arguments copies every array back after each kernel, read-only ones included. In CUDA C++, the CUDA Samples’ READMEs list cudaMalloc, cudaMemcpy and cudaFree among the runtime API calls the samples use.
  • Shared memory, on chip, allocated per block for the duration of the kernel and readable and writable by all threads of that block (below).
  • Local memory, private to each thread, for scratch arrays that do not fit in scalar variables, and constant memory, read-only, cached and off-chip, allocated by the host (numba).
  • Pinned (page-locked) host memory and mapped memory, pinned host memory mapped on to the device (numba’s pinned_array and mapped_array), which kernels read and write directly, as the simpleZeroCopy sample describes; their use for transfers, and PyTorch’s caution about overusing pinned memory, are in the GPUs article.
  • Managed (unified) memory, a single allocation that host and device can both access without explicit copies: vectorAdd.cu at v13.5 allocates its vectors with cudaMallocManaged, initializes them on the host and launches the kernel on them directly. Numba exposes it as cuda.managed_array. PyTorch’s notes do not recommend it for deep-learning workloads: for workloads that fit in GPU memory, they state, explicit placement consistently outperforms it, since there are no page faults.

Numba also defers freeing device memory: dropped arrays are queued and freed in batches (by default when ten are pending, when they reach 20% of device memory, or when an allocation fails), because a deallocation can synchronize the device. PyTorch’s caching allocator, which the GPUs article and the PyTorch article describe, instead keeps freed GPU memory for reuse.

Shared memory and barriers

Shared memory is the programmer-managed on-chip store of a block. Numba’s documentation calls it “a manually-managed data cache”; in numba it is declared inside the kernel with cuda.shared.array(shape, dtype), whose shape must be a simple constant expression, or as dynamic shared memory with shape 0 and a size in bytes given as the fourth element of the launch configuration, in which case all dynamic shared arrays of a kernel alias the same bytes and must be taken as disjoint views.

Because threads of a block run in no guaranteed order, a kernel that writes shared memory in one phase and reads it in the next needs a barrier between the phases. __syncthreads() in CUDA C++, cuda.syncthreads() in numba, waits until every thread of the block has reached it (numba describes it as the barrier of traditional multithreaded programming). The pattern is the one numba names as common: each thread fills its part of a shared array, the block synchronizes, and then threads read what others wrote. A barrier that only some threads of a block reach is an error; numba’s simulator documentation warns that if divergent threads enter different syncthreads calls “unexpected behaviour will occur” rather than a failed launch, and the same holds, in our reading, for any barrier placed inside a branch that not all threads take.

Shared memory serves two purposes in vision kernels: reuse, in which a block loads a tile of an image or a matrix once and its threads read it many times, which the GPUs article counts for a tiled matrix product; and privatization, in which each block accumulates into a private copy of a result in shared memory and merges it into device memory once, which the example below shows.

Atomic operations

When many threads update the same location, as in a histogram, a vote or a scatter, a plain read–modify–write loses updates. Atomic operations perform the whole update indivisibly. Numba 0.62.1 declares cuda.atomic.add, sub, max, min, and_, or_, xor, inc, dec, exch and compare_and_swap among others; atomic.add returns the old value, and its docstring in numba 0.62.1 lists int32, float32 and float64 operands, while the target’s type declarations also accept unsigned and 64-bit integers. Atomics work on device and shared memory alike.

Updates that collide on one address contend with each other, so, in our reading, their cost depends on how many threads collide. The CUDA Samples’ Python parallelHistogram README summarizes the trade-off: global atomics are simple but suffer “high contention on popular bins”, while privatized histograms in shared memory cost extra shared memory and synchronization. Floating-point atomic additions also make the order of a sum depend on scheduling, so results can differ from run to run in their last bits (our reasoning; mixed-precision training treats summation order).

Atomics were not always available. Podlozhnyuk’s 2007 whitepaper, written for the G80, states that G8x hardware “does not have native support for atomic shared memory operations”, and its 256-bin histogram resolved collisions within a warp in software: each thread wrote its incremented count tagged with its thread index in the five most significant bits of the counter, re-read it, and retried until its own tag stuck (Listing 3). The histogram256.cu file of CUDA Samples v13.5 no longer does this: its addByte function is a single atomicAdd on shared memory, and the tag it is passed is computed but unused.

Streams, events and graphs

A stream is a queue of work on one device: operations in the same stream run in the order they were issued, while operations in different streams may run concurrently in any relative order unless explicitly synchronized (PyTorch’s notes, which describe this as CUDA’s semantics). Streams are how a program overlaps a copy of the next frame with computation on the current one; in numba, streams are passed to copies and to kernel launches as the third element of the configuration so that they run asynchronously. The traps are those of any concurrency: work on a second stream may start before work it depends on has finished on the first, and memory freed and reused by an allocator may still be in use by a pending operation. PyTorch’s notes give its fixes, wait_stream and record_stream, and its own use of streams belongs to the PyTorch article.

Events are markers recorded into a stream, and they time GPU work: PyTorch’s notes show timing with two events and a synchronization, because a host timer around an asynchronous call measures only the launch, as latency and throughput discusses.

A CUDA graph records a sequence of kernels and their arguments once, by capturing a stream, and replays it with a single launch. PyTorch’s notes describe the gain as greatly reduced CPU overhead, since replay skips the Python, C++ and driver work of launching each kernel, at the price of fixed arguments: the same kernels on the same memory addresses, so new inputs are copied into the captured buffers before each replay. For small vision models at small batch sizes, where per-kernel launch costs can rival the arithmetic, this is the overhead that inference optimization discusses for whole-graph capture.

The toolkit and its layers

The CUDA Toolkit provides the compiler driver nvcc (at /usr/local/cuda/bin/nvcc in the samples’ build instructions), runtime libraries and the libraries below. The samples’ README, the only toolkit description read here, shows its main parts in use:

  • Compilation for architectures. GPU code is compiled for specific architectures, identified by a streaming-multiprocessor (SM) version; the samples build for all architectures supported by the release unless CMAKE_CUDA_ARCHITECTURES names one (“e.g. 90 for sm_90”), and each sample’s README lists the SM versions it supports, 5.0 to 9.0 for the histogram and reduction samples. The Tegra build instructions give 87 for Orin and 110 for Thor. Numba calls the same number the compute capability.
  • Intermediate code. Device code also exists in intermediate forms: the samples include a libNVVM category for libNVVM and NVVM IR samples and a ptxgen test invoked with -arch=compute_75, and the README names NVRTC, a runtime compilation library for CUDA C++, and nvJitLink, which links device code objects at run time. PTX code is mentioned in numba’s simulator documentation (it cannot be linked with CUDA Python functions there); its definition is in NVIDIA’s documentation, which was not read.
  • Runtime and driver APIs. The samples’ READMEs list the CUDA Runtime API functions each uses (cudaMalloc, cudaMemcpy, cudaHostAlloc, cudaDeviceSynchronize and others); numba talks to the lower-level CUDA Driver API through its own ctypes bindings or NVIDIA’s CUDA Python bindings.
  • Debugging. On-GPU debugging with cuda-gdb needs code built with nvcc -G, which disables certain compiler optimizations and is therefore off by default in the samples’ build.

The samples also mark their own scope: they are “not intended as a validation suite for CUDA” and not for performance benchmarking.

Libraries

Most vision code reaches CUDA through libraries rather than kernels. The samples’ README names cuBLAS (dense linear algebra), cuFFT (fast Fourier transforms), cuRAND (random numbers), cuSPARSE and cuSOLVER (sparse and dense solvers), NPP (NVIDIA Performance Primitives, “GPU-accelerated image, video, and signal processing functions”) and nvJPEG (JPEG decoding “for image formats commonly used in deep learning”). PyTorch’s notes show two of them under a framework: a cuBLAS workspace is allocated per handle and stream and kept until cleared, with its size set by CUBLAS_WORKSPACE_CONFIG, and cuFFT plans are kept in a per-device cache of 4,096 plans by default on CUDA 10 and newer.

cuDNN. NVIDIA’s library of deep-learning primitives, including convolutions, sits under PyTorch’s convolution and recurrent layers. Its documentation was not read, so this article states only what PyTorch’s notes say of it: cuDNN has its own TF32 setting, torch.backends.cudnn.allow_tf32, which defaults to True, unlike the matrix-multiplication flag, and its own entries in the finer fp32_precision settings, such as torch.backends.cudnn.conv.fp32_precision. The history of these flags and what TF32 does to the numbers are in the GPUs article. The practical point for vision is that, on Ampere and later NVIDIA GPUs, a float32 convolution in PyTorch may be computed with reduced-precision inputs by default.

Minimal Example

The example writes two histogram kernels for an 8-bit image and runs them in numba’s CUDA simulator, since the machine it was written on has no NVIDIA GPU. A histogram is the simplest operation in which many threads write to the same few locations; the histogram itself is the subject of image histogram. The kernels are our own, written from the algorithm in Podlozhnyuk’s whitepaper and the CUDA Samples’ descriptions; no sample code is copied.

  • Naive kernel. A grid-stride loop over the pixels, with one global atomic addition per pixel into the histogram in device memory.
  • Privatized kernel. The whitepaper’s 256-bin design: each of the block’s 6 warps (192 threads, the whitepaper’s “6 warps (192 threads) * 256 counters * 4 bytes per counter == 6KB”) owns a 256-bin sub-histogram in shared memory, with the warp index computed as threadIdx.x // 32. The block clears its shared memory, synchronizes, runs a grid-stride loop over 32-bit words that each hold four pixels, adds each pixel to its warp’s sub-histogram with a shared-memory atomic (where the G80 whitepaper used its tag scheme), synchronizes again, sums the six warp rows bin by bin and writes the block’s partial histogram to device memory with plain stores.
  • Merge kernel. The whitepaper says only that, without global atomics, “a separate final merging kernel” must combine the partial histograms. The design used here, one block per bin, each adding up that bin from every partial histogram, is the one a comment in histogram256.cu (v13.5) describes; our kernel follows that description, with 32 threads per block and a shared-memory tree reduction. The v13.5 Python sample’s “Privatized” kernel takes the other route the whitepaper names: one shared histogram per block, merged with one global atomicAdd per non-empty bin per block, and no merge kernel.

Both test images are 128×128128 \times 128 and synthetic: a textured scene (a gradient with a sinusoidal texture, a disk and a rectangle, each with its own pattern; 161 grey levels) and a flat image in which every pixel is 117, the value set by np.full in the code, the worst case for contention because every update goes to one bin. Pixels are packed four to a 32-bit word with view(np.uint32) on the C-contiguous array; on this little-endian host, byte jj of word kk is pixel 4k+j4k + j, and the script checks this by unpacking every word in NumPy. The simulator does not support bit operators applied to whole device arrays, so the kernel extracts bytes from one scalar word at a time. Both kernels launch 8 blocks of 192 threads, so the grid-stride loops run 2 or 3 iterations over the 4,096 words and 10 or 11 over the 16,384 pixels; each thread counts its own operations and adds them to a global counter once at the end, an instrumentation that is not part of the measured traffic.

What is measured: whether each kernel’s histogram equals np.bincount exactly; the number of global atomic additions (naive), shared atomic additions, partial-histogram writes and merge writes (privatized), counted inside the kernels; and a contention proxy, our own measure, computed in NumPy from the access pattern because the simulator has no warps: the largest number of updates to the same bin among the 32 threads of one 32-thread group in one step. For the naive kernel a group’s step covers 32 consecutive pixels; for the privatized kernel it covers byte jj of 32 consecutive words, that is, every fourth pixel of one 128-pixel row. The proxy counts colliding updates; it is not a timing.

The expected values were fixed before any of the example’s scripts ran, as exact integers with tolerance 0: both kernels equal np.bincount on both images; the naive kernel makes N=16,384N = 16{,}384 global atomic additions; the privatized kernel makes NN shared atomic additions, 8×256=2,0488 \times 256 = 2{,}048 partial writes and 256 merge writes; the proxy is 32 on the flat image and less than 32 on the textured one for both access patterns; and the data checks (shape, type, contiguity, distinct values, the unpacking round trip and the iteration counts) hold. A planning script then emulated the block and warp assignment in NumPy and printed every one of these values before the kernels were first run, so the example is an illustration of the mechanism with exact checks, not a test whose outcome was open. Run with numba 0.62.1’s built-in CUDA target (with NUMBA_ENABLE_CUDASIM=1) and NumPy 2.3.5, it took between about 75 and 100 seconds per run on the machine it was written on, which other jobs shared; the simulator starts one operating-system thread per CUDA thread (numba’s documentation). Every run printed identical output.

# A privatized image histogram in numba's CUDA simulator, against a naive global-atomic kernel.
# Our own kernels, written from the algorithm in Podlozhnyuk's whitepaper (per-warp sub-histograms
# in shared memory, merged afterwards); no CUDA Samples code is copied. numba 0.62.1's built-in
# CUDA target, run by its simulator: blocks run one at a time and threads are OS threads, so the
# output tests indexing, shared memory, barriers and atomics, not GPU speed or warp behaviour.
import os
os.environ["NUMBA_ENABLE_CUDASIM"] = "1"      # must be set before numba is imported
os.environ["OMP_NUM_THREADS"] = "1"; os.environ["OPENBLAS_NUM_THREADS"] = "1"
import sys
import numpy as np
import numba
from numba import cuda, int32

H = W = 128
BINS, WARP, WARPS = 256, 32, 6
THREADS = WARPS * WARP                         # 192 threads per block, as in the whitepaper
BLOCKS = 8
MERGE_THREADS = 32

@cuda.jit
def hist_naive(pixels, hist, counts, iters):
    n_atomic = 0
    n_iter = 0
    for p in range(cuda.grid(1), pixels.size, cuda.gridsize(1)):   # grid-stride loop
        cuda.atomic.add(hist, pixels[p], 1)                        # one global atomic per pixel
        n_atomic += 1
        n_iter += 1
    cuda.atomic.add(counts, 0, n_atomic)                           # instrumentation only
    iters[cuda.grid(1)] = n_iter

@cuda.jit
def hist_privatized(words, partial, counts, iters):
    s_hist = cuda.shared.array((WARPS, BINS), int32)               # one sub-histogram per warp
    t = cuda.threadIdx.x
    for i in range(t, WARPS * BINS, cuda.blockDim.x):              # clear shared memory
        s_hist[i // BINS, i % BINS] = 0
    cuda.syncthreads()
    warp = t // WARP
    n_shared = 0
    n_iter = 0
    for k in range(cuda.grid(1), words.size, cuda.gridsize(1)):    # grid-stride over uint32 words
        w = words[k]
        for j in range(4):                                         # four pixels per word
            cuda.atomic.add(s_hist, (warp, (w >> (8 * j)) & 0xFF), 1)
            n_shared += 1
        n_iter += 1
    cuda.syncthreads()                                             # all sub-histograms complete
    n_global = 0
    for b in range(t, BINS, cuda.blockDim.x):                      # merge warps, write per block
        total = 0
        for v in range(WARPS):
            total += s_hist[v, b]
        partial[cuda.blockIdx.x, b] = total
        n_global += 1
    cuda.atomic.add(counts, 0, n_shared)                           # instrumentation only
    cuda.atomic.add(counts, 1, n_global)
    iters[cuda.grid(1)] = n_iter

@cuda.jit
def merge(partial, hist, counts):
    data = cuda.shared.array(MERGE_THREADS, int32)
    b = cuda.blockIdx.x                                            # one block per bin
    t = cuda.threadIdx.x
    total = 0
    for i in range(t, partial.shape[0], cuda.blockDim.x):
        total += partial[i, b]
    data[t] = total
    cuda.syncthreads()
    stride = cuda.blockDim.x // 2
    while stride > 0:                                              # shared-memory tree reduction
        if t < stride:
            data[t] += data[t + stride]
        cuda.syncthreads()
        stride //= 2
    if t == 0:
        hist[b] = data[0]
        cuda.atomic.add(counts, 0, 1)

def textured():
    y, x = np.mgrid[0:H, 0:W].astype(np.float64)
    img = 64 + 0.25 * x + 0.2 * y + 30 * np.sin(2 * np.pi * x / 11.3) * np.cos(2 * np.pi * y / 7.9)
    disk = (y - 40) ** 2 + (x - 88) ** 2 <= 22 ** 2
    img[disk] = (190 + 25 * np.sin(2 * np.pi * (x + y) / 5.7))[disk]
    rect = (y >= 80) & (y <= 115) & (x >= 15) & (x <= 60)
    img[rect] = (30 + 3 * ((5 * x + 3 * y) % 17))[rect]
    return np.clip(np.rint(img), 0, 255).astype(np.uint8)

def contention(pix, group_words):
    # largest number of same-bin updates among the 32 threads of one group in one step
    if group_words:                    # privatized: 32 consecutive words, step = byte j of each
        g = pix.reshape(-1, WARP, 4)
        return max(np.bincount(g[m, :, j], minlength=BINS).max()
                   for m in range(g.shape[0]) for j in range(4))
    g = pix.reshape(-1, WARP)          # naive: 32 consecutive pixels per step
    return max(np.bincount(g[m], minlength=BINS).max() for m in range(g.shape[0]))

print(f"numba {numba.__version__}, NumPy {np.__version__}, NUMBA_ENABLE_CUDASIM {numba.config.ENABLE_CUDASIM}, "
      f"byteorder {sys.byteorder}")
nthreads = BLOCKS * THREADS
for name, img in [("textured", textured()), ("flat", np.full((H, W), 117, np.uint8))]:
    pix = img.ravel()
    words = pix.view(np.uint32)                        # four pixels per word, little-endian
    unpack = np.stack([(words >> np.uint32(8 * j)) & np.uint32(0xFF) for j in range(4)], 1).ravel()
    print(f"[{name}] shape {img.shape} {img.dtype} C-contiguous {img.flags['C_CONTIGUOUS']} "
          f"distinct values {np.unique(pix).size}, unpack mismatches {int((unpack != pix).sum())}")
    ref = np.bincount(pix, minlength=BINS)
    N = pix.size

    d_pix = cuda.to_device(pix)                        # host -> device copies
    d_hist = cuda.to_device(np.zeros(BINS, np.int32))
    d_cnt = cuda.to_device(np.zeros(1, np.int32))
    d_it = cuda.device_array(nthreads, np.int32)
    hist_naive[BLOCKS, THREADS](d_pix, d_hist, d_cnt, d_it)
    h_naive, c_naive, it_naive = d_hist.copy_to_host(), d_cnt.copy_to_host(), d_it.copy_to_host()

    d_words = cuda.to_device(words)
    d_part = cuda.device_array((BLOCKS, BINS), np.int32)
    d_cnt2 = cuda.to_device(np.zeros(2, np.int32))
    d_it2 = cuda.device_array(nthreads, np.int32)
    hist_privatized[BLOCKS, THREADS](d_words, d_part, d_cnt2, d_it2)
    d_hist2 = cuda.device_array(BINS, np.int32)
    d_cnt3 = cuda.to_device(np.zeros(1, np.int32))
    merge[BINS, MERGE_THREADS](d_part, d_hist2, d_cnt3)
    h_priv, c_priv, c_merge = d_hist2.copy_to_host(), d_cnt2.copy_to_host(), d_cnt3.copy_to_host()
    it_priv = d_it2.copy_to_host()

    print(f"  naive:      mismatched bins {int((h_naive != ref).sum())}, total {h_naive.sum()}, "
          f"global atomics {c_naive[0]} (N = {N}), iterations {it_naive.min()}..{it_naive.max()}")
    print(f"  privatized: mismatched bins {int((h_priv != ref).sum())}, total {h_priv.sum()}, "
          f"shared atomics {c_priv[0]} (N = {N}), partial writes {c_priv[1]} "
          f"(blocks x 256 = {BLOCKS * BINS}), merge writes {c_merge[0]} (256), "
          f"iterations {it_priv.min()}..{it_priv.max()}")
    print(f"  contention proxy (max same-bin updates per 32-thread step): "
          f"naive {contention(pix, False)}, privatized {contention(pix, True)}")
numba 0.62.1, NumPy 2.3.5, NUMBA_ENABLE_CUDASIM 1, byteorder little
[textured] shape (128, 128) uint8 C-contiguous True distinct values 161, unpack mismatches 0
  naive:      mismatched bins 0, total 16384, global atomics 16384 (N = 16384), iterations 10..11
  privatized: mismatched bins 0, total 16384, shared atomics 16384 (N = 16384), partial writes 2048 (blocks x 256 = 2048), merge writes 256 (256), iterations 2..3
  contention proxy (max same-bin updates per 32-thread step): naive 8, privatized 6
[flat] shape (128, 128) uint8 C-contiguous True distinct values 1, unpack mismatches 0
  naive:      mismatched bins 0, total 16384, global atomics 16384 (N = 16384), iterations 10..11
  privatized: mismatched bins 0, total 16384, shared atomics 16384 (N = 16384), partial writes 2048 (blocks x 256 = 2048), merge writes 256 (256), iterations 2..3
  contention proxy (max same-bin updates per 32-thread step): naive 32, privatized 32

Every check held on both images. The data checks held: the textured image has 161 distinct values and the flat one 1 (the check prints the count; the flat image’s value, 117, is set in the code), both are C-contiguous uint8 arrays, the unpacking round trip has no mismatch on this little-endian host, and every thread ran 10 or 11 iterations over pixels and 2 or 3 over words, so both full and partial sweeps of the grid occurred. The exactness check held: both kernels reproduced np.bincount exactly (0 mismatched bins, total 16,384), so the indexing, the grid-stride loops, the shared-memory clearing, both barriers and the merge tree are correct as far as the simulator can show. The counts held exactly: the naive kernel issued one global atomic per pixel, 16,384 on each image; the privatized kernel issued the same 16,384 atomics, but to shared memory, and then 2,048 plain writes of partial histograms and 256 writes in the merge. Privatization does not remove the atomic updates; it moves them into the block’s own memory, so that device memory sees blocks×256+256\text{blocks} \times 256 + 256 writes, a number set by the launch and the bin count, not by the image. The contention proxy held too: on the flat image every group of 32 threads hits one bin at every step (32 in both patterns), and on the textured image at most 8 threads of a naive group and 6 of a privatized group share a bin. The difference between the images, not between the kernels, is the point: contention depends on the data, and the privatized kernel confines it to one warp’s sub-histogram instead of a histogram shared by the whole grid.

What the example does not show is speed. Numba’s simulator documentation is explicit about its scope: it runs kernels “one block at a time”, with one operating-system thread per CUDA thread and scheduling left to the operating system; “Warps and warp-level operations are not yet implemented”; streams run sequentially and synchronously; events provide “no meaningful timing information”; and its atomics are, in numba 0.62.1’s source, Python operations under a lock. Whether privatization is faster on a GPU, and by how much, depends on hardware this example does not model; the whitepaper reports its own G80 measurements, and the Python sample’s README calls the privatized version “Significantly faster”. What the simulator does check is what the code does: indices, bounds, barriers, shared memory and atomic arithmetic. Its documentation recommends starting from code that compiles for the real CUDA target and moving to the simulator to investigate problems, which is how it is most useful.

Ecosystem and Integrations

  • Frameworks. PyTorch’s torch.cuda selects devices, allocates tensors on them, queues operators on streams and manages a caching allocator; the PyTorch article covers its use, and distributed training covers NCCL and multi-GPU training. PyTorch can also compile and load CUDA C++ at run time through torch.utils.cpp_extension.load_inline, which its notes use to build an NCCL memory allocator, and it just-in-time compiles some of its own operators on first use, caching the kernels in torch/kernels under the directory named by XDG_CACHE_HOME, or under .cache in the home directory when that variable is not set.
  • Numba and numba-cuda. Numba compiles a restricted subset of Python into CUDA kernels and device functions, whose kernels “appear to have direct access to NumPy arrays” (numba’s overview). Its built-in CUDA target is deprecated: development moved to the separate numba-cuda package, and numba’s documentation states that the built-in target “will continue to be provided by Numba through at least version 0.62”, with no new functionality; numba-cuda keeps the numba.cuda namespace, so code needs no changes. As of November 2026 the numba-cuda README (v0.30.4) says the package itself is in maintenance mode, intended to receive only security issues and critical bug fixes “through the lifetime of CUDA 13”, with new feature development targeted at Numba-CUDA-MLIR, to which it recommends upgrading as soon as practical. Numba does not implement dynamic parallelism (kernels launching kernels) or texture memory.
  • CuPy is “a NumPy/SciPy-compatible array library for GPU-accelerated computing” that runs NumPy and SciPy code on NVIDIA CUDA or AMD ROCm, and also passes arrays to CUDA C/C++ kernels through raw kernels, exposes streams and calls the CUDA runtime directly (README, v14.2.0).
  • CUDA Python. The CUDA Samples’ Python examples at v13.5 use NVIDIA’s CUDA Python with cuda.core for devices, program compilation and launches, alongside CuPy; the parallel-histogram sample compiles CUDA C source strings with cuda.core.Program and launches them with LaunchConfig.
  • OpenCV has CUDA modules (cudaarithm, cudafilters, cudaimgproc, cudaoptflow, cudastereo and others) in its contrib repository, which by that repository’s own description has less testing than the main one.
  • Inference runtimes. TensorRT and ONNX Runtime’s CUDA execution provider run exported models on NVIDIA GPUs; model deployment and inference optimization describe them.
  • Embedded devices. Numba’s documentation lists Jetson modules among the CUDA devices it supports, and the CUDA Samples build natively on Tegra boards and include cuDLA samples, whose hybrid mode programs the chips’ DLA using CUDA; edge AI accelerators and edge inference cover such hardware.

Alternatives

  • ROCm and HIP. AMD’s open-source platform and its C++ dialect, which PyTorch’s notes describe as designed to ease conversion of CUDA applications to portable C++ and used for projects “that require portability between AMD and NVIDIA”. PyTorch for ROCm reuses the torch.cuda interfaces, so an AMD GPU is still addressed as device "cuda", and torch.version.hip tells the builds apart; CuPy offers experimental ROCm 7.0 wheels (README, v14.2.0).
  • Triton, “a language and compiler for writing highly efficient custom Deep-Learning primitives”, whose README states the aim of writing fast code at higher productivity than CUDA. At v3.9.0 it supports Linux, with NVIDIA GPUs of compute capability 8.0 or later and AMD GPUs with ROCm 6.2 or later, and CPUs under development. The README cites a MAPL 2019 paper, “Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations”, for its foundations; the paper was not read.
  • OpenCL: OpenCV’s Transparent API runs ordinary functions through OpenCL when given a cv::UMat, as the OpenCV article describes.
  • Vulkan compute: Vulkan is, in the CUDA Samples README’s words, a “low-overhead, cross-platform 3D graphics and compute API”, and the samples include a simpleVulkan interoperation sample.
  • Metal on Apple GPUs, which PyTorch reaches through its MPS back end, and SYCL are further options, named here only; no source on either was read for this article.

In our reading, the choice is mostly made by the hardware: CUDA runs only on NVIDIA GPUs, and code that must also run on AMD, Intel or Apple GPUs is written against a portable layer (HIP, OpenCL, SYCL, Triton) or left to a framework that has back ends for each.

Versions and Compatibility

  • Samples and toolkit. As of November 2026 the CUDA Samples are at v13.5 (4 November 2026), and their README states that this version supports CUDA Toolkit 13.5; the Python samples target CUDA 13.x, and the Python histogram sample requires Toolkit 13.0 or newer and Python 3.10 or newer. Building with a CUDA 13.0 or later toolkit and a user-mode driver of version 580 or later on a kernel-mode driver of version 550 or earlier needs CMAKE_PREFIX_PATH pointed at the new driver library (README). Nothing here is claimed about toolkit releases beyond what the samples print.
  • Architectures. Each binary targets SM versions; numba 0.62.1 supports devices of compute capability 3.5 or greater and deprecates those below 5.0, and Triton v3.9.0 requires 8.0 or greater on NVIDIA GPUs. The SM lists in the samples’ READMEs are per sample.
  • Numba. Numba aims to support CUDA toolkits released within the last three years, with 11.2 as the minimum, and lists the Conda packages cuda-nvcc and cuda-nvrtc as required for CUDA 12; it finds a toolkit through Conda packages, CUDA_HOME or /usr/local/cuda, and needs an NVIDIA driver recent enough for the toolkit (numba 0.62.1’s overview). The example ran with numba 0.62.1’s built-in target and NumPy 2.3.5.
  • CuPy publishes separate wheels per CUDA major version (cupy-cuda12x, cupy-cuda13x) for x86_64 and aarch64 (README, v14.2.0).
  • Older code. Code written for the G80 era assumes limits that no longer hold, such as the whitepaper’s 16 KB limit on the shared-memory variables of G8x hardware; the samples’ histogram_common.h still keeps a G80-era SHARED_MEMORY_BANKS 16 as a parameter, commented “May change on future hardware”. Limits should be read from the device at run time, not from old code.

Limitations

  • One vendor. CUDA programs run on NVIDIA GPUs only; HIP and the other alternatives exist because of this.
  • Asynchronous errors and timing. Errors surface after the kernel that caused them, and host timers around kernel launches measure the launch; both need explicit synchronization or events.
  • Correctness is the programmer’s. Out-of-range threads, missing or divergent barriers, races on shared memory and forgotten synchronization between streams are silent bugs that may give plausible but wrong images. The simulator can catch indexing and logic errors on small inputs, but not warp-level or timing-dependent ones, since it implements no warps and runs blocks one at a time.
  • Python routes in transition. Numba’s built-in CUDA target is deprecated, and numba-cuda itself is in maintenance mode as of November 2026, so new Python kernel code faces a choice among numba-cuda, Numba-CUDA-MLIR, CuPy’s raw kernels, CUDA Python and Triton.
  • Default numerics. Library defaults can change results: cuDNN’s TF32 default for float32 convolutions in PyTorch, and floating-point atomics whose summation order varies from run to run.
  • Documentation and licensing outside the code. The CUDA Samples’ files read here carry BSD-3-Clause headers, but the repository’s licence also points to the CUDA EULA, which was not read for this article; the toolkit and libraries such as cuDNN have their own terms.

Related

  • PyTorch

    An open-source Python library for building and training neural networks with tensors, automatic differentiation and GPU acceleration, run eagerly as ordinary Python code and optionally compiled or exported for deployment.

  • OpenCV

    The open-source C++ library of computer vision and image processing functions, with Python bindings that work on NumPy arrays; its modules, conventions, versions from 1.0 to 5.0, contrib modules and licence.

  • Image Histogram

    The count of pixels at each intensity or color level, an empirical distribution that summarizes exposure and contrast and underlies thresholding, equalization, and histogram-based recognition and tracking.

  • Mixed-Precision Training

    Training a network with most of its arithmetic and storage in 16-bit (FP16 or BF16) or 8-bit floating point while keeping the numerically sensitive parts in FP32, using an FP32 master copy of the weights, loss scaling, FP32 accumulation and per-tensor scales for FP8.

  • Inference Optimization

    Transformations that make a trained vision network run faster or in less memory without changing the function it computes, from constant folding and operator fusion to layout changes, compilation, kernel selection and memory planning, with a worked example of what fusion stops writing to memory.

  • Distributed Training

    Training one neural network on many devices at once, by splitting the batch (data parallelism), the training state (sharded data parallelism, ZeRO and FSDP) or the model itself (tensor and pipeline parallelism), so that gradients match those of training on a single device while memory and communication are divided.

References

  1. Numba 0.62.1. CUDA documentation: Overview, Writing CUDA Kernels, Memory management, Supported Atomic Operations and Debugging CUDA Python with the CUDA Simulator (docs/source/cuda, tag 0.62.1; BSD-2-Clause), with the docstrings of numba/cuda/stubs.py and the simulator in numba/cuda/simulator (installed 0.62.1 source; read November 2026).
  2. NVIDIA. Numba CUDA Target (numba-cuda), README.md (repository NVIDIA/numba-cuda, tag v0.30.4, 3 July 2026; BSD-2-Clause; read November 2026).
  3. NVIDIA. CUDA Samples v13.5 (4 November 2026, commit 0ec046d): README.md, cpp/0_Introduction/vectorAdd/vectorAdd.cu, cpp/0_Introduction/simpleZeroCopy/README.md, cpp/2_Concepts_and_Techniques/histogram/histogram256.cu, histogram_common.h and README.md, cpp/2_Concepts_and_Techniques/reduction/README.md, cpp/8_Platform_Specific/Tegra/README.md and python/2_CoreConcepts/parallelHistogram (README.md, parallelHistogram.py). Files carry BSD-3-Clause headers; the repository licence also refers to the CUDA EULA, not read. Read for description only; no code copied.
  4. Podlozhnyuk, V. (2007). Histogram calculation in CUDA. NVIDIA whitepaper, version 1.1.1 of 9 November 2007 (cover dated September 2013), in NVIDIA/cuda-samples v13.5, cpp/2_Concepts_and_Techniques/histogram/doc/histogram.pdf.
  5. PyTorch. CUDA semantics, docs/source/notes/cuda.md (repository pytorch/pytorch, tag v2.14.1; BSD-style licence; read November 2026).
  6. PyTorch. HIP (ROCm) semantics, docs/source/notes/hip.rst (repository pytorch/pytorch, tag v2.14.1; read November 2026).
  7. CuPy developers. CuPy: NumPy & SciPy for GPU, README.md (repository cupy/cupy, tag v14.2.0, 20 August 2026; MIT; read November 2026).
  8. Triton developers. Triton, README.md (repository triton-lang/triton, tag v3.9.0, 27 October 2026; MIT; read November 2026).
  9. Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations. MAPL 2019, as cited in the Triton README (not read).
  10. NVIDIA. CUDA C++ Programming Guide, CUDA C++ Best Practices Guide, CUDA EULA and cuDNN documentation (not read).