Vision Systems & Engineering

Latency and Throughput

The two measures of a vision system's speed, how waiting, service time and batching set them, why percentiles rather than means describe latency, what parameters, multiply-adds and activations do and do not say about runtime, and how to time a component honestly.

intermediate

How fast a vision system is has two answers that are easy to confuse. Latency is how long one input waits for its result: one camera frame, one uploaded photo, one request to a server. Throughput is how many inputs the system finishes per second. A server that batches 32 images together can have high throughput and poor latency; a phone that processes one frame at a time can have good latency and modest throughput. Engineering choices about speed, from batch size and model size to where a model runs, can move one of the two at the expense of the other.

This article treats the two quantities in general: how waiting and service time make up latency, what a batch does to both, why latency is reported as a percentile, and what the common proxies for speed, parameters, multiply-adds and activations, can and cannot predict. The timing of a whole camera-to-decision chain, with its frame budget, pipelining and queue policies, is the subject of real-time vision pipelines; the roofline model and benchmark practice on devices belong to edge inference; and serving systems that batch requests are described in model deployment.

Definition

For a stream of requests, where a request is one image or one frame:

  • Latency, also called response time, is the time from a request’s arrival to the moment its result is available. It is the sum of the time the request waits (in a queue, or for a batch to fill) and its service time, the time the hardware actually spends computing on it.
  • Throughput is the number of requests completed per unit time, usually per second (images per second, frames per second, queries per second).
  • Capacity is the largest throughput the system can sustain. Requests arriving faster than the capacity pile up without limit.

Latency is a distribution over requests, not a single number, and is summarized by its percentiles: the median (p50), and the 90th and 99th percentiles (p90, p99), below which 90% and 99% of requests finish. The high percentiles are called the tail. Throughput is a rate over a period of time, and is meaningful only for a stated arrival pattern and a stated period.

Intuition

Consider an airport shuttle. A shuttle that leaves with each passenger the moment they arrive gives the shortest possible trip for that passenger but moves few passengers per hour, because every trip pays the full cost of driving. A shuttle that waits until it is full moves many more passengers per hour with the same driving, but the first passenger to board sits waiting for the rest. A sensible operator leaves when the shuttle is full or when the first passenger has waited a few minutes, whichever comes first.

Batched inference on a GPU or other accelerator behaves the same way. Running a network on one image pays fixed costs, such as launching kernels, reading the weights and per-call overheads, that a batch of many images pays only once, and the accelerator’s parallel units are better used with more images at once (our characterization; model deployment describes batching as amortizing per-call overheads and using the hardware’s parallelism). A larger batch therefore raises throughput, but each image in it waits for the others to arrive and for the whole batch to finish. At low load, waiting dominates and small batches are faster; near capacity, queues dominate and only the throughput of large batches keeps them short. Which batch size gives the lowest latency depends on the load, and the example below shows this on a simulated device.

Formal Definition

Let request ii arrive at time tit_i and its result be ready at time did_i. Its latency is

Li=di−ti=Wi+Si,L_i = d_i - t_i = W_i + S_i ,

with waiting time WiW_i and service time SiS_i. Over nn completed requests in a period of length TT, the throughput is X=n/TX = n/T, and the qq-th percentile of latency is the value L(q)L_{(q)} below which a fraction qq of the LiL_i fall. Little’s law ties the averages together: in steady state the mean number of requests in the system, waiting or in service, is Nˉ=XLˉ\bar{N} = X \bar{L}, as the real-time vision pipelines article states it for frames in flight.

An affine batch cost. A simple model of a batched accelerator (our illustrative model, not a measured one) charges a fixed overhead aa per batch and a cost cc per image:

t(b)=a+c b.t(b) = a + c\,b .

With batches of at most BB images, the capacity is the throughput with full batches,

μB=Ba+cB  →  B→∞    1c,\mu_B = \frac{B}{a + cB} \;\xrightarrow{\;B \to \infty\;}\; \frac{1}{c},

so a larger batch amortizes the overhead and throughput approaches 1/c1/c, while the service time of a full batch, a+cBa + cB, grows linearly. With requests arriving at rate λ\lambda, the system is stable only if λ<μB\lambda < \mu_B; the ratio ρ=λ/μB\rho = \lambda/\mu_B is the utilization. A request alone in an idle system has latency a+ca + c when B=1B = 1. If the server waits for a batch to fill, a request at low load waits on average about (B−1)/λ(B-1)/\lambda for B−1B - 1 more arrivals (our derivation for arrivals at rate λ\lambda), which is why practical batchers cap the wait at a timeout τ\tau: the batch is dispatched when BB requests are waiting or when the oldest has waited τ\tau.

Speeding up part of the work. If a fraction pp of a request’s time is spent in a component that is made kk times faster, the request’s time falls by the factor

S=1(1−p)+p/k≤11−p,S = \frac{1}{(1 - p) + p/k} \le \frac{1}{1 - p},

a bound known as Amdahl’s law (our derivation; the mixed precision training article gives a worked case). A network that takes half of a pipeline’s time can make the pipeline at most twice as fast, however fast the network becomes.

Properties

Batching trades latency for throughput

On a server, batching is the main lever for throughput, and serving systems expose both knobs of the model above: Model Deployment describes Clipper adapting its maximum batch size to a latency objective, and Triton’s dynamic batcher, which by default does not wait and can be told to let a request wait a bounded time for others to join. A device that processes one frame at a time has no other requests to batch with, and its latency is the service time of a batch of one, plus whatever queueing its pipeline adds. In our reading of the affine model, the timeout is pure latency at low load, where no other request arrives within τ\tau, and nearly free at high load, where batches fill before it expires.

Batching also changes where the time goes. The edge inference article’s roofline example shows batching helping a network whose memory traffic is dominated by weights, which are read once per batch, and helping much less one dominated by activations, which grow with the batch.

The tail decides deadlines

A system that must answer within a deadline misses it on its slow requests, so latency targets are written as percentiles: “p99 below 100 ms” rather than “mean below 100 ms”. The tail grows quickly as utilization approaches one, because a request that arrives behind a burst waits for the whole burst (our characterization); in the example below, two batch sizes that both keep up with the same load differ in their 99th percentile by more than a factor of two. Estimating a high percentile needs many samples: the MLPerf Inference counts that edge inference describes rise from 24,576 inferences for a 90th percentile to 270,336 for a 99th.

Throughput and the tail interact through the load. Measuring throughput by sending as many requests as the system accepts gives its capacity, at which the queue, and so the latency, has no steady value. Latency is meaningful only for a stated arrival rate below capacity, and a pair of numbers such as “400 images per second at a p99 of 50 ms” says more than either alone.

Speed proxies: parameters, multiply-adds and activations

Before a model runs on its target hardware, its cost is estimated from counts:

  • Parameters, the number of learned weights, set the storage and the weight traffic, and do not depend on the input resolution.
  • Multiply-accumulates (MACs, also called multiply-adds) count the arithmetic. Several papers call this count “FLOPs”: Dollár, Singh and Girshick (2021), Radosavovic et al. (2020) and Ma et al. (2018) all state that their FLOPs mean multiply-adds, while other papers count a multiply and an add as two operations, so “FLOPs” figures can differ by a factor of two, as the EfficientNet article shows.
  • Activations, which Dollár et al. define as the number of elements in the output tensors of the convolutional layers, measure the size of the intermediate results that have to be written and read.

For a stage of dd convolutional layers of width ww (channels) at spatial resolution r×rr \times r, Dollár et al. give the counts f=dw2r2f = dw^2r^2, p=dw2p = dw^2 and a=dwr2a = dwr^2, with the kernel size set aside. Scaling the stage so that its multiply-adds grow by a factor ss can be done by multiplying the depth by ss, or the width or the resolution by s\sqrt{s}. Their Table 1 shows that the three choices give the same multiply-adds but different parameters and activations: activations grow by s\sqrt{s} under width scaling and by ss under depth or resolution scaling, while parameters grow by ss under width or depth scaling and stay fixed under resolution scaling.

The counts matter because they predict runtime unequally. Dollár et al. scaled EfficientNet-B0 with several strategies and timed each scaled model as the time for one epoch of ImageNet training on 8 Volta GPUs, noting that inference time is highly correlated with training time. Across strategies, runtime correlated with multiply-adds at a Pearson rr of 0.81 and with parameters at 0.56, but with activations at 0.99; for scaled versions of each of three base models the correlation with activations was at least 0.99, and 0.95 across the different models. They caution that activations cannot predict runtime perfectly across different models, which may use operations with different costs, and that some large models ran slower than their activations predicted because they were limited by arithmetic. They attribute the role of activations to accelerators being memory-bandwidth bound. Radosavovic et al. (2020) make the same observation for their RegNet models: EfficientNet’s activations grow linearly with its multiply-adds, because its compound scaling enlarges resolution and depth, while those of RegNets grow with the square root, and they report RegNetX-8000 as five times faster than EfficientNet-B5 on GPUs with lower error.

Ma et al. (2018) frame the general point: multiply-adds are an indirect metric of the direct metric, speed, measured on the target platform. They name memory access cost and the degree of parallelism as factors that the count misses, and note that operations with the same count can run at different speeds on different platforms; the guidelines they derive are summarized in the MobileNet article. Their abbreviation “MAC” stands for memory access cost, not multiply-accumulate, a clash worth watching when reading efficiency papers. The edge inference article turns counts into a lower bound on time with the roofline model, which charges each layer the larger of its arithmetic time and its memory time.

Speed against accuracy

A model’s speed is only interesting next to its accuracy, and the trade-off is drawn as a curve of accuracy against latency. Huang et al. (2017) traced such curves for detectors built from three meta-architectures (Faster R-CNN, R-FCN and SSD) with several feature extractors and input sizes in one TensorFlow implementation, so that all points were measured the same way: on one machine with a GeForce GTX Titan X, at batch size 1, averaged over 500 COCO images resized and cropped to 300×300300 \times 300 or 600×600600 \times 600, including post-processing. Their non-maximum suppression ran on the CPU and took about 40 ms for the fastest models, capping their frame rate at 25 per second, and they warn that, for this reason among others, their timings are comparable among themselves but may not be directly comparable with speeds reported elsewhere, naming further differences in hardware, drivers, framework and batch size. Curves of this kind are what neural architecture search optimizes when it searches on measured latency, and what model compression papers need to report to compare methods fairly.

Measuring Speed

A measured time is a property of a configuration: the device, the runtime and its version, the number format, batch size, thread count, input size and thermal state. The edge inference article covers the practice benchmark suites encode for these, including warm-up runs, minimum durations and pinning to cores. A few points apply to any timing, in Python as elsewhere:

  • The clock. In Python, time.perf_counter() is the clock meant for measuring short durations: the documentation describes it as having the highest available resolution, as including time spent sleeping, and as meaningful only in differences between two calls; since Python 3.13 CPython uses the same clock as time.monotonic(). time.process_time() counts the CPU time of the current process and excludes sleep, so it misses time spent waiting for a device or another process. timeit uses perf_counter as its default timer.
  • The minimum against the percentile. The timeit documentation calls the mean and standard deviation of repeated timings not very useful: the lowest value is a lower bound for how fast the machine can run the code, and higher values are typically caused by other processes interfering, so the minimum is “probably the only number you should be interested in” (Python documentation, timeit). In our reading, that is the right number for comparing two implementations of a component. It answers a different question from the latency a system’s users see, which includes exactly the interference, queueing and slow cases that the minimum discards; for that question the percentiles of a realistic load are the measure (our reading).
  • What the harness turns off. timeit temporarily disables garbage collection during the timing, which makes timings more comparable but leaves out a cost the measured function may pay in use; the documentation shows how to re-enable it in the setup statement.
  • Threads and asynchronous devices. Numerical libraries may use several threads, NumPy’s BLAS library among them, so a timing depends on the thread count and on what else runs on the machine; the example below sets the BLAS thread count to one before importing NumPy. Accelerators queue work asynchronously, so a timer read before the device has finished measures the launch, not the computation (our characterization of asynchronous execution).

Examples

Dynamic batching on a simulated clock, and counting a network’s cost

The script below has two parts. Neither measures wall-clock time: the first simulates a device, and the second counts.

Part 1 simulates one accelerator serving requests that arrive as a Poisson process at rate λ\lambda, with the affine batch cost t(b)=8+2bt(b) = 8 + 2b ms, an illustrative choice of ours (capacity μB=B/(8+2B)\mu_B = B/(8 + 2B) ms−1^{-1}, from 100 requests per second at B=1B = 1 toward 1/c=5001/c = 500). The server is first-in, first-out. When it is free, it dispatches a batch at once if BB requests are waiting; otherwise it waits until BB have arrived or the oldest has waited the timeout τ\tau, and then serves up to BB of them together. Two timeouts are compared: τ=0\tau = 0, which dispatches whatever is waiting as soon as the server is free (the default behaviour of Triton’s dynamic batcher, as Model Deployment describes it), and τ=5\tau = 5 ms. The grid crosses B∈{1,2,4,8,16,32}B \in \{1, 2, 4, 8, 16, 32\} with arrival rates of 0.5, 60, 200 and 380 requests per second, with 20,000 requests per run; the same seeded exponential draws are rescaled for every rate and reused for every BB and τ\tau. Each request carries a 16×1616 \times 16 synthetic frame with a bright square in one of four column bands, and a toy batched classifier reads the band from the stacked batch, so the script checks that every result reaches the request it belongs to.

For each case the script prints the throughput X=N/(last result−first arrival)X = N/(\text{last result} - \text{first arrival}); the 50th, 90th and 99th latency percentiles over all requests (NumPy’s default linear interpolation); the mean number of requests in the system Nˉ\bar{N}, waiting or in service, averaged over the window between the arrivals of requests N/10N/10 and 9N/109N/10; and Little’s law’s relative residual ∣Nˉ−XwLˉ∣/Nˉ|\bar{N} - X_w \bar{L}|/\bar{N} in that window, with XwX_w the completions in the window per unit time and Lˉ\bar{L} their mean latency. A case with λ≥μB\lambda \ge \mu_B is marked over: it has no steady state. For each timeout and arrival seed 0 to 4, it then lists the batch sizes with the lowest 99th percentile at each load (ties at 0.01 ms), among those that keep up. Separate runs check the two limits of the model: a saturated load of 1,000 requests per second, whose throughput should approach μB\mu_B, and a vanishing load of 0.01 per second, at which a request with B=1B = 1 should take exactly a+c=10a + c = 10 ms.

Part 2 counts the parameters, multiply-accumulates and activations of a synthetic plain network: 3×33 \times 3 convolutions without bias and with zero padding of one pixel, each followed by ReLU; a stem from 3 to ww channels at r×rr \times r, and three stages of dd layers with widths ww, 2w2w and 4w4w at resolutions rr, r/2r/2 and r/4r/4, the first layer of the second and third stages having stride 2. The counts are read from the arrays of a NumPy forward pass on a synthetic r×rr \times r RGB frame, with untrained random weights, since only the shapes matter: parameters from the weight arrays, multiply-accumulates from the dimensions of each convolution’s im2col matrix product (padded positions included), and activations from the convolution outputs, Dollár et al.’s definition. Our derivation for this network, with k=3k = 3, gives the closed forms

p=k2(3w+(21d−10)w2),f=k2r2(3w+(3d−1)w2),a=wr2(4+7d)4,p = k^2\big(3w + (21d - 10)w^2\big), \qquad f = k^2 r^2 \big(3w + (3d - 1)w^2\big), \qquad a = \frac{w r^2 (4 + 7d)}{4},

against which the counts are compared exactly. The base network has w=8w = 8, d=2d = 2 and r=32r = 32; each strategy scales one dimension so that Dollár et al.’s per-stage multiply-adds would grow by s∈{4,16}s \in \{4, 16\}: width by s\sqrt{s}, depth by ss, resolution by s\sqrt{s}. For each scaled network the script prints the ratios of multiply-accumulates, parameters and activations to the base, and the exponents ep=ln⁡(p ratio)/ln⁡(f ratio)e_p = \ln(p \text{ ratio})/\ln(f \text{ ratio}) and eae_a likewise, which Dollár et al.’s Table 1 puts at (ep,ea)=(1,0.5)(e_p, e_a) = (1, 0.5) for width, (1,1)(1, 1) for depth and (0,1)(0, 1) for resolution. The im2col convolution is first checked against SciPy’s correlate.

import os
for v in ("OPENBLAS_NUM_THREADS", "OMP_NUM_THREADS", "MKL_NUM_THREADS"):
    os.environ[v] = "1"                      # one BLAS thread, set before NumPy loads
import numpy as np
from scipy.signal import correlate

# ---------- Part 1: dynamic batching on a simulated clock (times in ms) ----------
A, C = 8.0, 2.0                              # batch service time t(b) = A + C*b (illustrative)
N = 20_000                                   # requests per run
BS = [1, 2, 4, 8, 16, 32]                    # maximum batch size
TAUS = [0.0, 5.0]                            # timeout: how long the oldest request may wait for a batch to fill
LAMS = [0.5, 60, 200, 380]                   # arrival rates, requests per second

rng = np.random.default_rng(1000)            # each request carries a 16x16 frame whose label is
labels = rng.integers(0, 4, N)               # the column band (0-3) of a bright 4x4 square
imgs = rng.integers(0, 41, (N, 16, 16)).astype(np.uint8)
for L in range(4):
    imgs[np.ix_(labels == L, np.arange(6, 10), np.arange(4 * L, 4 * L + 4))] = 200

def model(batch):
    """Toy batched classifier: the brightest column band of each frame."""
    prof = batch.sum(axis=1, dtype=np.int64)              # (k, 16) column sums
    return prof.reshape(len(batch), 4, 4).sum(2).argmax(1)

def arrivals(lam, seed):
    return np.cumsum(np.random.default_rng(seed).standard_exponential(N)) * 1000 / lam

def simulate(arr, B, tau):
    """FIFO server: when free, dispatch at once if B requests wait, otherwise wait until
    B have arrived or the oldest has waited tau; then serve up to B together."""
    done = np.empty(N); h, free, correct = 0, 0.0, 0
    while h < N:
        start = max(free, arr[h])
        full = arr[h + B - 1] if h + B - 1 < N else np.inf
        t = max(start, min(arr[h] + tau, full))           # dispatch time
        k = min(B, np.searchsorted(arr, t, side="right") - h)
        free = t + A + C * k
        done[h:h + k] = free
        correct += int((model(imgs[h:h + k]) == labels[h:h + k]).sum())
        h += k
    return done - arr, done, correct

def measures(arr, lat, done):
    X = N / (done.max() - arr[0]) * 1000                  # throughput, requests/s
    t0, t1 = arr[N // 10], arr[9 * N // 10]               # steady-state window
    nbar = np.clip(np.minimum(done, t1) - np.maximum(arr, t0), 0, None).sum() / (t1 - t0)
    out = (done >= t0) & (done <= t1)
    xw, wbar = out.sum() / (t1 - t0), lat[out].mean()     # per ms, ms
    return X, np.percentile(lat, [50, 90, 99]), nbar, abs(nbar - xw * wbar) / nbar

cap = {B: B / (A + C * B) * 1000 for B in BS}             # requests/s with full batches
runs = ok = 0
best = {}
for tau in TAUS:
    print(f"tau = {tau:.0f} ms   (cap = B/(a+cB); 'over' = lambda >= cap; Nbar = mean number in system)")
    print(" lam/s   B  cap/s  thru/s    p50     p90     p99     Nbar  Little rel. resid.")
    for lam in LAMS:
        arr = arrivals(lam, 0)
        for B in BS:
            lat, done, corr = simulate(arr, B, tau); runs += 1; ok += corr
            X, (p50, p90, p99), nbar, res = measures(arr, lat, done)
            over = lam >= cap[B]
            print(f"{lam:6.1f} {B:3d} {cap[B]:6.1f} {X:7.1f} {p50:7.2f} {p90:7.2f} {p99:7.2f} {nbar:8.3f}  "
                  + ("over" if over else f"{res:.5f}"))
    # H3: which stable B gives the lowest p99 (ties at 0.01 ms), seeds 0-4
    for seed in range(5):
        sets = []
        for lam in LAMS:
            arr = arrivals(lam, seed); p = {}
            for B in BS:
                if lam < cap[B]:
                    lat, done, corr = simulate(arr, B, tau); runs += 1; ok += corr
                    p[B] = round(float(np.percentile(lat, 99)), 2)
            sets.append({B for B in p if p[B] == min(p.values())})
        common = set.intersection(*sets)
        print(f"  tau {tau:.0f}, seed {seed}: lowest-p99 B at lam {LAMS}: "
              + "  ".join("{" + ",".join(map(str, sorted(s))) + "}" for s in sets)
              + f"   B best at every load: {sorted(common) if common else 'none'}")

print("saturated, lam = 1000/s: throughput (requests/s) against B/(a+cB)")
for tau in TAUS:
    arr = arrivals(1000, 0); xs = []
    for B in BS:
        lat, done, corr = simulate(arr, B, tau); runs += 1; ok += corr
        xs.append(N / (done.max() - arr[0]) * 1000)
    err = max(abs(x / cap[B] - 1) for x, B in zip(xs, BS))
    print(f"  tau {tau:.0f}: " + " ".join(f"{x:6.1f}" for x in xs)
          + f"   theory " + " ".join(f"{cap[B]:6.1f}" for B in BS)
          + f"   max |X/cap - 1| = {err:.5f}   1/c = {1000 / C:.0f}")
for tau in TAUS:
    arr = arrivals(0.01, 0)
    lat, done, corr = simulate(arr, 1, tau); runs += 1; ok += corr
    p50, p99 = np.percentile(lat, [50, 99])
    print(f"vanishing load 0.01/s, B=1, tau {tau:.0f}: p50 {p50:.6f} p99 {p99:.6f} ms, "
          f"|p99-(a+c)| = {abs(p99 - A - C):.1e}; requests that waited: {(lat > A + C + 1e-6).sum()}")
print(f"responses with their own frame's label: {ok} of {runs * N} in {runs} runs")

# ---------- Part 2: parameters, multiply-adds and activations, counted exactly ----------
K = 3

def conv(x, W, stride):
    """3x3 convolution (cross-correlation), zero padding 1, via im2col; returns output and MACs."""
    cin, H, Wd = x.shape
    Ho, Wo = (H - 1) // stride + 1, (Wd - 1) // stride + 1
    xp = np.pad(x, ((0, 0), (1, 1), (1, 1)))
    cols = np.stack([xp[:, dy:dy + stride * Ho:stride, dx:dx + stride * Wo:stride]
                     for dy in range(K) for dx in range(K)], axis=1).reshape(cin * K * K, Ho * Wo)
    Wm = W.reshape(W.shape[0], cin * K * K)
    return (Wm @ cols).reshape(-1, Ho, Wo), Wm.shape[0] * Wm.shape[1] * cols.shape[1]

def frame(r):
    """Synthetic RGB frame: a colour gradient, a disk and a rectangle."""
    y, x = np.mgrid[0:r, 0:r] / r
    img = np.stack([x, y, 0.5 * (x + y)])
    img[:, (x - 0.35) ** 2 + (y - 0.4) ** 2 < 0.04] = np.array([[0.9], [0.2], [0.1]])
    img[:, (x > 0.6) & (x < 0.85) & (y > 0.55) & (y < 0.8)] = np.array([[0.1], [0.3], [0.9]])
    return img

def network(w, d, r):
    """Stem 3->w, then stages of d layers at widths w, 2w, 4w; stride 2 opens stages 2 and 3."""
    rng = np.random.default_rng(0)
    x, p = frame(r), 0
    macs = acts = 0
    layers = [(3, w, 1)] + [(w, w, 1)] * d
    for j in (1, 2):
        layers += [(w * 2 ** (j - 1), w * 2 ** j, 2)] + [(w * 2 ** j, w * 2 ** j, 1)] * (d - 1)
    for cin, cout, s in layers:
        W = rng.normal(0, np.sqrt(2 / (cin * K * K)), (cout, cin, K, K))
        x, m = conv(x, W, s)
        x = np.maximum(x, 0)
        p, macs, acts = p + W.size, macs + m, acts + x.size
    return p, macs, acts

def closed(w, d, r):
    return (K * K * (3 * w + (21 * d - 10) * w * w),
            K * K * r * r * (3 * w + (3 * d - 1) * w * w),
            w * r * r * (4 + 7 * d) // 4)

# library cross-check of the convolution, stride 1 and stride 2
x = frame(32); W = np.random.default_rng(1).normal(size=(4, 3, K, K))
ref = np.stack([sum(correlate(x[c], W[o, c], mode="same", method="direct") for c in range(3))
                for o in range(4)])
d1 = np.abs(conv(x, W, 1)[0] - ref).max(); d2 = np.abs(conv(x, W, 2)[0] - ref[:, ::2, ::2]).max()
print(f"im2col conv vs scipy.signal.correlate: max |diff| stride 1 {d1:.1e}, stride 2 {d2:.1e}")

w0, d0, r0 = 8, 2, 32
cases = [("base", 1, w0, d0, r0)]
for s in (4, 16):
    q = int(np.sqrt(s))
    cases += [("width", s, w0 * q, d0, r0), ("depth", s, w0, d0 * s, r0), ("resolution", s, w0, d0, r0 * q)]
print("strategy    s    w   d    r      params          MACs   activations  =closed  "
      "MACx    px     ax     e_p     e_a")
P0, M0, A0 = network(w0, d0, r0)
expo = {}
for name, s, w, d, r in sorted(cases, key=lambda c: ("bwdr".index(c[0][0]), c[1])):
    p, m, a = network(w, d, r)
    match = (p, m, a) == closed(w, d, r)
    fx, px, ax = m / M0, p / P0, a / A0
    ep, ea = (np.log(px) / np.log(fx), np.log(ax) / np.log(fx)) if s > 1 else (np.nan, np.nan)
    expo[name, s] = (ep, ea)
    print(f"{name:10s} {s:2d} {w:4d} {d:3d} {r:4d} {p:11,d} {m:13,d} {a:13,d}  {str(match):5s}  "
          f"{fx:6.3f} {px:6.3f} {ax:6.3f} {ep:7.4f} {ea:7.4f}")
dollar = {"width": (1.0, 0.5), "depth": (1.0, 1.0), "resolution": (0.0, 1.0)}
order = all(expo["width", s][1] < min(expo["depth", s][1], expo["resolution", s][1]) for s in (4, 16))
print(f"e_a(width) below e_a(depth) and e_a(resolution) at s = 4 and 16: {order}")
for name, (dp, da) in dollar.items():
    g4 = [abs(expo[name, 4][i] - t) for i, t in enumerate((dp, da))]
    g16 = [abs(expo[name, 16][i] - t) for i, t in enumerate((dp, da))]
    print(f"  {name:10s} gap to Dollar's (e_p, e_a) = ({dp:.1f}, {da:.1f}): s=4 ({g4[0]:.4f}, {g4[1]:.4f})"
          f"  s=16 ({g16[0]:.4f}, {g16[1]:.4f})  not growing: {all(b <= a + 1e-12 for a, b in zip(g4, g16))}")

Run with NumPy 2.3.5 and SciPy 1.16.3, it printed the same output in two runs, taking about 14 seconds (approximate; the machine was shared):

tau = 0 ms   (cap = B/(a+cB); 'over' = lambda >= cap; Nbar = mean number in system)
 lam/s   B  cap/s  thru/s    p50     p90     p99     Nbar  Little rel. resid.
   0.5   1  100.0     0.5   10.00   10.00   10.00    0.005  0.00000
   0.5   2  166.7     0.5   10.00   10.00   10.00    0.005  0.00000
   0.5   4  250.0     0.5   10.00   10.00   10.00    0.005  0.00000
   0.5   8  333.3     0.5   10.00   10.00   10.00    0.005  0.00000
   0.5  16  400.0     0.5   10.00   10.00   10.00    0.005  0.00000
   0.5  32  444.4     0.5   10.00   10.00   10.00    0.005  0.00000
  60.0   1  100.0    60.4   13.79   30.86   56.29    1.063  0.00000
  60.0   2  166.7    60.4   11.58   19.95   27.99    0.832  0.00001
  60.0   4  250.0    60.4   11.32   19.63   23.68    0.817  0.00001
  60.0   8  333.3    60.4   11.31   19.63   23.65    0.817  0.00001
  60.0  16  400.0    60.4   11.31   19.63   23.65    0.817  0.00001
  60.0  32  444.4    60.4   11.31   19.63   23.65    0.817  0.00001
 200.0   1  100.0   100.0 50355.77 90609.62 99631.24 5028.540  over
 200.0   2  166.7   166.6 10390.09 18650.44 20470.29 1717.228  over
 200.0   4  250.0   201.4   23.71   39.94   62.91    5.310  0.00001
 200.0   8  333.3   201.4   22.13   31.58   42.38    4.614  0.00001
 200.0  16  400.0   201.4   22.10   31.56   41.89    4.607  0.00001
 200.0  32  444.4   201.4   22.10   31.56   41.89    4.607  0.00001
 380.0   1  100.0   100.0 73873.82 132951.87 146227.02 7385.467  over
 380.0   2  166.7   166.7 33878.82 60960.73 67036.02 5640.762  over
 380.0   4  250.0   250.0 13889.80 24974.48 27446.02 3461.756  over
 380.0   8  333.3   333.1 3910.67 7004.64 7672.08 1289.751  over
 380.0  16  400.0   382.3   62.38  109.17  159.36   27.178  0.00127
 380.0  32  444.4   382.4   54.95   83.59  110.97   22.136  0.00084
  tau 0, seed 0: lowest-p99 B at lam [0.5, 60, 200, 380]: {1,2,4,8,16,32}  {8,16,32}  {16,32}  {32}   B best at every load: [32]
  tau 0, seed 1: lowest-p99 B at lam [0.5, 60, 200, 380]: {1,2,4,8,16,32}  {8,16,32}  {16,32}  {32}   B best at every load: [32]
  tau 0, seed 2: lowest-p99 B at lam [0.5, 60, 200, 380]: {1,2,4,8,16,32}  {4}  {16,32}  {32}   B best at every load: none
  tau 0, seed 3: lowest-p99 B at lam [0.5, 60, 200, 380]: {1,2,4,8,16,32}  {8,16,32}  {16,32}  {32}   B best at every load: [32]
  tau 0, seed 4: lowest-p99 B at lam [0.5, 60, 200, 380]: {1,2,4,8,16,32}  {8,16,32}  {16,32}  {32}   B best at every load: [32]
tau = 5 ms   (cap = B/(a+cB); 'over' = lambda >= cap; Nbar = mean number in system)
 lam/s   B  cap/s  thru/s    p50     p90     p99     Nbar  Little rel. resid.
   0.5   1  100.0     0.5   10.00   10.00   10.00    0.005  0.00000
   0.5   2  166.7     0.5   15.00   15.00   15.00    0.008  0.00000
   0.5   4  250.0     0.5   15.00   15.00   15.00    0.008  0.00000
   0.5   8  333.3     0.5   15.00   15.00   15.00    0.008  0.00000
   0.5  16  400.0     0.5   15.00   15.00   15.00    0.008  0.00000
   0.5  32  444.4     0.5   15.00   15.00   15.00    0.008  0.00000
  60.0   1  100.0    60.4   13.79   30.86   56.29    1.063  0.00000
  60.0   2  166.7    60.4   15.00   20.03   27.89    0.964  0.00004
  60.0   4  250.0    60.4   15.13   19.75   24.34    0.996  0.00004
  60.0   8  333.3    60.4   15.14   19.79   24.38    0.997  0.00004
  60.0  16  400.0    60.4   15.14   19.79   24.38    0.997  0.00004
  60.0  32  444.4    60.4   15.14   19.79   24.38    0.997  0.00004
 200.0   1  100.0   100.0 50355.77 90609.62 99631.24 5028.540  over
 200.0   2  166.7   166.6 10384.91 18645.27 20465.11 1716.366  over
 200.0   4  250.0   201.4   23.78   39.76   63.25    5.344  0.00002
 200.0   8  333.3   201.4   22.25   31.76   42.12    4.689  0.00003
 200.0  16  400.0   201.4   22.24   31.74   41.95    4.684  0.00003
 200.0  32  444.4   201.4   22.24   31.74   41.95    4.684  0.00003
 380.0   1  100.0   100.0 73873.82 132951.87 146227.02 7385.467  over
 380.0   2  166.7   166.7 33878.62 60957.29 67034.53 5640.543  over
 380.0   4  250.0   250.0 13888.10 24970.91 27446.18 3461.442  over
 380.0   8  333.3   333.1 3909.49 7004.27 7674.61 1289.615  over
 380.0  16  400.0   382.2   62.64  109.82  161.70   27.347  0.00145
 380.0  32  444.4   382.4   54.91   84.25  112.47   22.184  0.00200
  tau 5, seed 0: lowest-p99 B at lam [0.5, 60, 200, 380]: {1}  {4}  {16,32}  {32}   B best at every load: none
  tau 5, seed 1: lowest-p99 B at lam [0.5, 60, 200, 380]: {1}  {8,16,32}  {16,32}  {32}   B best at every load: none
  tau 5, seed 2: lowest-p99 B at lam [0.5, 60, 200, 380]: {1}  {4}  {16,32}  {32}   B best at every load: none
  tau 5, seed 3: lowest-p99 B at lam [0.5, 60, 200, 380]: {1}  {4}  {16,32}  {32}   B best at every load: none
  tau 5, seed 4: lowest-p99 B at lam [0.5, 60, 200, 380]: {1}  {4}  {16,32}  {32}   B best at every load: none
saturated, lam = 1000/s: throughput (requests/s) against B/(a+cB)
  tau 0:  100.0  166.7  250.0  333.3  399.9  444.2   theory  100.0  166.7  250.0  333.3  400.0  444.4   max |X/cap - 1| = 0.00053   1/c = 500
  tau 5:  100.0  166.7  250.0  333.3  399.9  444.2   theory  100.0  166.7  250.0  333.3  400.0  444.4   max |X/cap - 1| = 0.00047   1/c = 500
vanishing load 0.01/s, B=1, tau 0: p50 10.000000 p99 10.000000 ms, |p99-(a+c)| = 0.0e+00; requests that waited: 1
vanishing load 0.01/s, B=1, tau 5: p50 10.000000 p99 10.000000 ms, |p99-(a+c)| = 0.0e+00; requests that waited: 1
responses with their own frame's label: 4840000 of 4840000 in 242 runs
im2col conv vs scipy.signal.correlate: max |diff| stride 1 1.8e-15, stride 2 1.8e-15
strategy    s    w   d    r      params          MACs   activations  =closed  MACx    px     ax     e_p     e_a
base        1    8   2   32      18,648     3,170,304        36,864  True    1.000  1.000  1.000     nan     nan
width       4   16   2   32      74,160    12,238,848        73,728  True    3.860  3.977  2.000  1.0220  0.5131
width      16   32   2   32     295,776    48,070,656       147,456  True   15.163 15.861  4.000  1.0166  0.5099
depth       4    8   8   32      91,224    13,787,136       122,880  True    4.349  4.892  3.333  1.0801  0.8191
depth      16    8  32   32     381,528    56,254,464       466,944  True   17.744 20.459 12.667  1.0495  0.8828
resolution  4    8   2   64      18,648    12,681,216       147,456  True    4.000  1.000  4.000  0.0000  1.0000
resolution 16    8   2  128      18,648    50,724,864       589,824  True   16.000  1.000 16.000  0.0000  1.0000
e_a(width) below e_a(depth) and e_a(resolution) at s = 4 and 16: True
  width      gap to Dollar's (e_p, e_a) = (1.0, 0.5): s=4 (0.0220, 0.0131)  s=16 (0.0166, 0.0099)  not growing: True
  depth      gap to Dollar's (e_p, e_a) = (1.0, 1.0): s=4 (0.0801, 0.1809)  s=16 (0.0495, 0.1172)  not growing: True
  resolution gap to Dollar's (e_p, e_a) = (0.0, 1.0): s=4 (0.0000, 0.0000)  s=16 (0.0000, 0.0000)  not growing: True

The limits of the model. The four checks of the simulation held. At a vanishing load the median and 99th percentile with B=1B = 1 are exactly 10 ms, a+ca + c, for both timeouts (one request in 20,000 arrived while another was in service). Under saturation the throughput stays within at most 0.053% of μB\mu_B, inside the bar of 0.2%, and rises from 100.0 to 444.2 requests per second as BB grows from 1 to 32, every value below 1/c=5001/c = 500, so H1, throughput rising with BB toward 1/c1/c at high load, held for both timeouts. Little’s law held in every case that keeps up: the largest relative residual is 0.00200, against a bar of 0.01. Every one of the 4,840,000 responses carried its own frame’s label.

Low load. At 0.5 requests per second, every batch size gives 10.00 ms at every percentile when the server does not wait. With a 5 ms timeout, every B≥2B \ge 2 gives 15.00 ms, exactly τ\tau more than B=1B = 1, so H2, that a timeout raises the 99th percentile by about τ\tau at low load, held. This follows from the construction: at this rate the utilization of B=1B = 1 is 0.5×0.010=0.0050.5 \times 0.010 = 0.005, so fewer than 1% of requests find the server busy, and a lone request with B≥2B \ge 2 waits out the whole timeout.

Moderate load. At 60 requests per second B=1B = 1 still keeps up, at a utilization of 0.6, but its 99th percentile is 56.29 ms against 23.65 ms for B=8B = 8 without a timeout, with 1.063 requests in the system on average against 0.817: from the construction, requests that arrive while one is being served wait for it, and a batch serves the waiting ones together. The timeout costs most at the median here: with B=4B = 4 the median rises from 11.32 ms without it to 15.13 ms with it, while the 99th percentile moves from 23.68 to 24.34 ms.

High load. At 200 requests per second B=1B = 1 and B=2B = 2 cannot keep up: their throughput stops at their capacities, 100.0 and 166.6 per second, and their latencies, ten seconds and more, measure only how long the run lasted before the backlog was cleared. At 380 per second only B=16B = 16, at a utilization of 0.95, and B=32B = 32 keep up, and the 99th percentile of the larger batch is the lower, 110.97 ms against 159.36 ms without a timeout.

Which batch size is best. H3 was that the batch size with the lowest 99th percentile depends on the load, failing if one batch size is best or tied for best at every load. With the 5 ms timeout, H3 held on all five arrival seeds: B=1B = 1 is best at 0.5 per second, B=4B = 4 (or 8, 16 and 32 together, on seed 1) at 60, 16 and 32 at 200, and 32 at 380. Without a timeout, H3 failed on seeds 0, 1, 3 and 4, on which B=32B = 32 is best or tied for best at every load, and held on seed 2, on which B=4B = 4 alone is best at 60 per second.

The counts. The check of the im2col convolution against SciPy held, with a largest difference of 1.8×10−151.8 \times 10^{-15} at both strides against a bar of 10−910^{-9}, and H4, that the counted parameters, multiply-accumulates and activations equal the closed forms exactly, held in all seven rows. The scaled networks show Dollár et al.’s point in numbers. At s=16s = 16 the three strategies raise the multiply-accumulates by 15.2, 17.7 and 16.0 times, but the activations by 4.0 times for width, 12.7 for depth and 16.0 for resolution, and the parameters by 15.9, 20.5 and 1.0 times. Both parts of the comparison with Dollár et al.’s Table 1 held: the width exponent eae_a (0.5131 at s=4s = 4, 0.5099 at 16) is below the depth and resolution exponents at both scales, and every gap to the table’s exponents shrinks or stays at zero from s=4s = 4 to s=16s = 16.

The gaps that remain follow from the closed forms the example checks (our derivation). The table describes a stage of identical layers; this network also has a stem and two strided layers. Under width scaling the stem’s 3w3w multiply-accumulates grow only by s\sqrt{s}, so the total grows by a little less than ss (15.163 at s=16s = 16) and eae_a sits slightly above 0.5. Under depth scaling the stem’s activations wr2w r^2 do not grow with dd, while the multiply-accumulates of the layers after the stem grow as 3sd−13sd - 1 against 3d−13d - 1, a little faster than ss (17.744 at s=16s = 16, with the stem included), so eae_a falls below 1, to 0.8828 at s=16s = 16; both terms weigh less as dd grows, which is why the gap shrinks. Resolution scaling multiplies every layer’s output by ss and leaves the weights alone, so its exponents are exactly 1 and 0. In our reading, these networks would differ in speed on memory-bound hardware much more than their multiply-adds suggest, which is what Dollár et al. measured on GPUs; the example counts and does not time.

What the simulation leaves out. The device’s cost is deterministic and affine, with no variation between batches, no transfer of inputs, no preprocessing and no contention with other work, and arrivals are Poisson. A real device’s batch cost is measured, not assumed; with variable service times the tail grows further, as the real-time vision pipelines example shows for a pipeline with a randomly slow detector.

How the example was planned. The hypotheses H1 to H4 and the checks were written before any script ran. A planning script that ran this example’s set-up with extra prints then printed every outcome before the plan was frozen, so the example illustrates derivations rather than testing them. Two bars were set after that script: 0.2% for the saturated throughput and 0.01 for Little’s law’s residual. That script’s running time also served as the runtime estimate. Compared with the batch plan’s description, the example adds the zero timeout beside the 5 ms one; fixes the arrival rates, choosing 0.5 per second so that fewer than 1% of requests find B=1B = 1’s server busy; gives each request a synthetic frame with a check of its label; evaluates H3 over five arrival seeds with ties at 0.01 ms; measures the plan’s mean queue length as the mean number in the system, in a window that leaves out the first and last 10% of arrivals; checks the vanishing-load latency through its median and 99th percentile rather than every request; measures saturated throughput over the whole run; compares the counts with closed forms for a network with a stem and strided stages, and Dollár et al.’s exponents with the gaps described above; and checks the convolution against SciPy. The batch plan called the affine cost model “the shape Model Deployment describes for Clipper”; that article describes batching as amortizing per-call overheads, and the affine model here is our own illustrative choice. The pilot and the first run printed the same output.

Common Misconceptions

  • “Higher throughput means lower latency.” Batching and pipelining raise throughput while adding waiting to each request. The two improve together only when queueing dominates, as at 200 requests per second in the example, where B=1B = 1 and B=2B = 2 cannot keep up and larger batches both raise the throughput to the arrival rate and drain the queue.
  • “Batch size 1 gives the lowest latency.” Only when the server is lightly loaded. Near capacity, a batch of one leaves requests queueing behind each other; in the example B=1B = 1 cannot keep up at 200 requests per second, where B=16B = 16 keeps up with a 99th percentile of 41.89 ms.
  • “A model with fewer FLOPs is faster.” Multiply-adds are an indirect measure. Networks scaled to similar multiply-adds in the example reach 4.0 and 16.0 times the base network’s activations (width and resolution scaling at s=16s = 16), which Dollár et al. found to track GPU runtime closely; the MobileNet and edge inference articles show the same effect through memory access and the roofline.
  • “FLOPs are floating-point operations.” In several vision papers, Dollár et al.’s, Radosavovic et al.’s and Ma et al.’s among them, they are multiply-adds, half the count of operations that other papers report under the same name.
  • “Parameters measure cost.” Parameters do not depend on the input size: scaling the resolution by four in the example multiplies the arithmetic and the activations by 16 and leaves the parameters unchanged.
  • “The fastest of several timings is the system’s latency.” In our reading, the minimum is the right figure for comparing implementations of a component, because it removes interference; the latency users see includes interference and queueing, and is described by percentiles under a realistic load.

Where It Is Used

Every deployed vision model is chosen against a latency or throughput requirement, set as part of vision system design. Servers that answer many users batch requests and tune batch sizes and timeouts, as described in model deployment; devices that process one camera frame at a time care about single-image latency, energy and heat, the subject of edge inference; and systems that act on live video are judged on the end-to-end latency of the real-time pipeline around the model. Detectors such as YOLO are reported on speed–accuracy curves of the kind Huang et al. drew for object detection. The speed proxies guide the design of efficient networks such as MobileNet and EfficientNet, whose depthwise separable convolutions save multiply-adds more than memory traffic; neural architecture search can search directly on measured latency; and model compression and quantization trade accuracy for speed, a trade that is only meaningful when the speed is measured on named hardware.

Related

  • Deploying Vision Models

    Getting a trained vision model into use, from export and the choice of runtime to packaging the model with its preprocessing contract, serving it on a server or a device, keeping pre- and post-processing identical to training, and versioning, rolling back and monitoring it.

  • Model Compression

    Making a trained vision model smaller and cheaper to run by pruning weights or filters, quantizing, distilling it into a smaller network, factorizing its layers or choosing a compact architecture, and how to compare the results fairly in accuracy, size, operations and measured latency.

  • MobileNet

    Google's family of convolutional networks for phones and other small devices, from MobileNet v1 (2017), built from depthwise separable convolutions with width and resolution multipliers, through MobileNetV2's inverted residuals with linear bottlenecks and MobileNetV3's latency-driven architecture search, to MobileNetV4's universal inverted bottleneck and mobile attention (2024).

  • EfficientNet

    Google's family of convolutional networks built by scaling one searched baseline, EfficientNet-B0, in depth, width and input resolution together with a single compound coefficient (B1 to B7, 2019), followed by EfficientNetV2 (2021), which mixes fused and depthwise blocks and grows the image size during training to train faster.

  • Designing a Computer Vision System

    How a working vision system is designed from its requirements, through the choice of sensor, optics, lighting, compute and model, to the data it is trained on, its validation in the field and its maintenance once deployed, with the requirement arithmetic for a measurement station worked through in code.

  • GPUs for Computer Vision

    The graphics processing unit as a machine for vision workloads, covering SIMT execution in warps, latency hiding by many resident threads, the memory hierarchy from registers to host memory, data reuse by tiling, why convolutions suit GPUs and element-wise operations are limited by bandwidth, TF32 and reduced precision, memory capacity, host–device transfer, GPUs from other vendors and how to choose one.

References

  1. Dollár, P., Singh, M. & Girshick, R. (2021). Fast and Accurate Model Scaling. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (read in the CVF open-access copy).
  2. Radosavovic, I., Kosaraju, R. P., Girshick, R., He, K. & Dollár, P. (2020). Designing Network Design Spaces. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 10425–10433 (read in the CVF open-access copy).
  3. Huang, J., Rathod, V., Sun, C., Zhu, M., Korattikara, A., Fathi, A., Fischer, I., Wojna, Z., Song, Y., Guadarrama, S. & Murphy, K. (2017). Speed/Accuracy Trade-Offs for Modern Convolutional Object Detectors. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (read in the CVF open-access copy).
  4. Ma, N., Zhang, X., Zheng, H.-T. & Sun, J. (2018). ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design. European Conference on Computer Vision (ECCV), Lecture Notes in Computer Science, 122–138 (read in the CVF open-access copy).
  5. Little, J. D. C. (1961). A Proof for the Queuing Formula: L = λW. Operations Research, 9(3), 383–387 (not read; cited as Real-Time Vision Pipelines states it).
  6. Python Software Foundation. timeit: Measure execution time of small code snippets, Python 3.15.2 documentation (accessed November 2026).
  7. Python Software Foundation. time: Time access and conversions, Python 3.15.2 documentation (accessed November 2026).