Vision Systems & Engineering
Latency and Throughput
The two measures of a vision system's speed, how waiting, service time and batching set them, why percentiles rather than means describe latency, what parameters, multiply-adds and activations do and do not say about runtime, and how to time a component honestly.
intermediate
How fast a vision system is has two answers that are easy to confuse. Latency is how long one input waits for its result: one camera frame, one uploaded photo, one request to a server. Throughput is how many inputs the system finishes per second. A server that batches 32 images together can have high throughput and poor latency; a phone that processes one frame at a time can have good latency and modest throughput. Engineering choices about speed, from batch size and model size to where a model runs, can move one of the two at the expense of the other.
This article treats the two quantities in general: how waiting and service time make up latency, what a batch does to both, why latency is reported as a percentile, and what the common proxies for speed, parameters, multiply-adds and activations, can and cannot predict. The timing of a whole camera-to-decision chain, with its frame budget, pipelining and queue policies, is the subject of real-time vision pipelines; the roofline model and benchmark practice on devices belong to edge inference; and serving systems that batch requests are described in model deployment.
Definition
For a stream of requests, where a request is one image or one frame:
- Latency, also called response time, is the time from a request’s arrival to the moment its result is available. It is the sum of the time the request waits (in a queue, or for a batch to fill) and its service time, the time the hardware actually spends computing on it.
- Throughput is the number of requests completed per unit time, usually per second (images per second, frames per second, queries per second).
- Capacity is the largest throughput the system can sustain. Requests arriving faster than the capacity pile up without limit.
Latency is a distribution over requests, not a single number, and is summarized by its percentiles: the median (p50), and the 90th and 99th percentiles (p90, p99), below which 90% and 99% of requests finish. The high percentiles are called the tail. Throughput is a rate over a period of time, and is meaningful only for a stated arrival pattern and a stated period.
Intuition
Consider an airport shuttle. A shuttle that leaves with each passenger the moment they arrive gives the shortest possible trip for that passenger but moves few passengers per hour, because every trip pays the full cost of driving. A shuttle that waits until it is full moves many more passengers per hour with the same driving, but the first passenger to board sits waiting for the rest. A sensible operator leaves when the shuttle is full or when the first passenger has waited a few minutes, whichever comes first.
Batched inference on a GPU or other accelerator behaves the same way. Running a network on one image pays fixed costs, such as launching kernels, reading the weights and per-call overheads, that a batch of many images pays only once, and the accelerator’s parallel units are better used with more images at once (our characterization; model deployment describes batching as amortizing per-call overheads and using the hardware’s parallelism). A larger batch therefore raises throughput, but each image in it waits for the others to arrive and for the whole batch to finish. At low load, waiting dominates and small batches are faster; near capacity, queues dominate and only the throughput of large batches keeps them short. Which batch size gives the lowest latency depends on the load, and the example below shows this on a simulated device.
Formal Definition
Let request arrive at time and its result be ready at time . Its latency is
with waiting time and service time . Over completed requests in a period of length , the throughput is , and the -th percentile of latency is the value below which a fraction of the fall. Little’s law ties the averages together: in steady state the mean number of requests in the system, waiting or in service, is , as the real-time vision pipelines article states it for frames in flight.
An affine batch cost. A simple model of a batched accelerator (our illustrative model, not a measured one) charges a fixed overhead per batch and a cost per image:
With batches of at most images, the capacity is the throughput with full batches,
so a larger batch amortizes the overhead and throughput approaches , while the service time of a full batch, , grows linearly. With requests arriving at rate , the system is stable only if ; the ratio is the utilization. A request alone in an idle system has latency when . If the server waits for a batch to fill, a request at low load waits on average about for more arrivals (our derivation for arrivals at rate ), which is why practical batchers cap the wait at a timeout : the batch is dispatched when requests are waiting or when the oldest has waited .
Speeding up part of the work. If a fraction of a request’s time is spent in a component that is made times faster, the request’s time falls by the factor
a bound known as Amdahl’s law (our derivation; the mixed precision training article gives a worked case). A network that takes half of a pipeline’s time can make the pipeline at most twice as fast, however fast the network becomes.
Properties
Batching trades latency for throughput
On a server, batching is the main lever for throughput, and serving systems expose both knobs of the model above: Model Deployment describes Clipper adapting its maximum batch size to a latency objective, and Triton’s dynamic batcher, which by default does not wait and can be told to let a request wait a bounded time for others to join. A device that processes one frame at a time has no other requests to batch with, and its latency is the service time of a batch of one, plus whatever queueing its pipeline adds. In our reading of the affine model, the timeout is pure latency at low load, where no other request arrives within , and nearly free at high load, where batches fill before it expires.
Batching also changes where the time goes. The edge inference article’s roofline example shows batching helping a network whose memory traffic is dominated by weights, which are read once per batch, and helping much less one dominated by activations, which grow with the batch.
The tail decides deadlines
A system that must answer within a deadline misses it on its slow requests, so latency targets are written as percentiles: “p99 below 100 ms” rather than “mean below 100 ms”. The tail grows quickly as utilization approaches one, because a request that arrives behind a burst waits for the whole burst (our characterization); in the example below, two batch sizes that both keep up with the same load differ in their 99th percentile by more than a factor of two. Estimating a high percentile needs many samples: the MLPerf Inference counts that edge inference describes rise from 24,576 inferences for a 90th percentile to 270,336 for a 99th.
Throughput and the tail interact through the load. Measuring throughput by sending as many requests as the system accepts gives its capacity, at which the queue, and so the latency, has no steady value. Latency is meaningful only for a stated arrival rate below capacity, and a pair of numbers such as “400 images per second at a p99 of 50 ms” says more than either alone.
Speed proxies: parameters, multiply-adds and activations
Before a model runs on its target hardware, its cost is estimated from counts:
- Parameters, the number of learned weights, set the storage and the weight traffic, and do not depend on the input resolution.
- Multiply-accumulates (MACs, also called multiply-adds) count the arithmetic. Several papers call this count “FLOPs”: Dollár, Singh and Girshick (2021), Radosavovic et al. (2020) and Ma et al. (2018) all state that their FLOPs mean multiply-adds, while other papers count a multiply and an add as two operations, so “FLOPs” figures can differ by a factor of two, as the EfficientNet article shows.
- Activations, which Dollár et al. define as the number of elements in the output tensors of the convolutional layers, measure the size of the intermediate results that have to be written and read.
For a stage of convolutional layers of width (channels) at spatial resolution , Dollár et al. give the counts , and , with the kernel size set aside. Scaling the stage so that its multiply-adds grow by a factor can be done by multiplying the depth by , or the width or the resolution by . Their Table 1 shows that the three choices give the same multiply-adds but different parameters and activations: activations grow by under width scaling and by under depth or resolution scaling, while parameters grow by under width or depth scaling and stay fixed under resolution scaling.
The counts matter because they predict runtime unequally. Dollár et al. scaled EfficientNet-B0 with several strategies and timed each scaled model as the time for one epoch of ImageNet training on 8 Volta GPUs, noting that inference time is highly correlated with training time. Across strategies, runtime correlated with multiply-adds at a Pearson of 0.81 and with parameters at 0.56, but with activations at 0.99; for scaled versions of each of three base models the correlation with activations was at least 0.99, and 0.95 across the different models. They caution that activations cannot predict runtime perfectly across different models, which may use operations with different costs, and that some large models ran slower than their activations predicted because they were limited by arithmetic. They attribute the role of activations to accelerators being memory-bandwidth bound. Radosavovic et al. (2020) make the same observation for their RegNet models: EfficientNet’s activations grow linearly with its multiply-adds, because its compound scaling enlarges resolution and depth, while those of RegNets grow with the square root, and they report RegNetX-8000 as five times faster than EfficientNet-B5 on GPUs with lower error.
Ma et al. (2018) frame the general point: multiply-adds are an indirect metric of the direct metric, speed, measured on the target platform. They name memory access cost and the degree of parallelism as factors that the count misses, and note that operations with the same count can run at different speeds on different platforms; the guidelines they derive are summarized in the MobileNet article. Their abbreviation “MAC” stands for memory access cost, not multiply-accumulate, a clash worth watching when reading efficiency papers. The edge inference article turns counts into a lower bound on time with the roofline model, which charges each layer the larger of its arithmetic time and its memory time.
Speed against accuracy
A model’s speed is only interesting next to its accuracy, and the trade-off is drawn as a curve of accuracy against latency. Huang et al. (2017) traced such curves for detectors built from three meta-architectures (Faster R-CNN, R-FCN and SSD) with several feature extractors and input sizes in one TensorFlow implementation, so that all points were measured the same way: on one machine with a GeForce GTX Titan X, at batch size 1, averaged over 500 COCO images resized and cropped to or , including post-processing. Their non-maximum suppression ran on the CPU and took about 40 ms for the fastest models, capping their frame rate at 25 per second, and they warn that, for this reason among others, their timings are comparable among themselves but may not be directly comparable with speeds reported elsewhere, naming further differences in hardware, drivers, framework and batch size. Curves of this kind are what neural architecture search optimizes when it searches on measured latency, and what model compression papers need to report to compare methods fairly.
Measuring Speed
A measured time is a property of a configuration: the device, the runtime and its version, the number format, batch size, thread count, input size and thermal state. The edge inference article covers the practice benchmark suites encode for these, including warm-up runs, minimum durations and pinning to cores. A few points apply to any timing, in Python as elsewhere:
- The clock. In Python,
time.perf_counter()is the clock meant for measuring short durations: the documentation describes it as having the highest available resolution, as including time spent sleeping, and as meaningful only in differences between two calls; since Python 3.13 CPython uses the same clock astime.monotonic().time.process_time()counts the CPU time of the current process and excludes sleep, so it misses time spent waiting for a device or another process.timeitusesperf_counteras its default timer. - The minimum against the percentile. The
timeitdocumentation calls the mean and standard deviation of repeated timings not very useful: the lowest value is a lower bound for how fast the machine can run the code, and higher values are typically caused by other processes interfering, so the minimum is “probably the only number you should be interested in” (Python documentation, timeit). In our reading, that is the right number for comparing two implementations of a component. It answers a different question from the latency a system’s users see, which includes exactly the interference, queueing and slow cases that the minimum discards; for that question the percentiles of a realistic load are the measure (our reading). - What the harness turns off.
timeittemporarily disables garbage collection during the timing, which makes timings more comparable but leaves out a cost the measured function may pay in use; the documentation shows how to re-enable it in the setup statement. - Threads and asynchronous devices. Numerical libraries may use several threads, NumPy’s BLAS library among them, so a timing depends on the thread count and on what else runs on the machine; the example below sets the BLAS thread count to one before importing NumPy. Accelerators queue work asynchronously, so a timer read before the device has finished measures the launch, not the computation (our characterization of asynchronous execution).
Examples
Dynamic batching on a simulated clock, and counting a network’s cost
The script below has two parts. Neither measures wall-clock time: the first simulates a device, and the second counts.
Part 1 simulates one accelerator serving requests that arrive as a Poisson process at rate , with the affine batch cost ms, an illustrative choice of ours (capacity ms, from 100 requests per second at toward ). The server is first-in, first-out. When it is free, it dispatches a batch at once if requests are waiting; otherwise it waits until have arrived or the oldest has waited the timeout , and then serves up to of them together. Two timeouts are compared: , which dispatches whatever is waiting as soon as the server is free (the default behaviour of Triton’s dynamic batcher, as Model Deployment describes it), and ms. The grid crosses with arrival rates of 0.5, 60, 200 and 380 requests per second, with 20,000 requests per run; the same seeded exponential draws are rescaled for every rate and reused for every and . Each request carries a synthetic frame with a bright square in one of four column bands, and a toy batched classifier reads the band from the stacked batch, so the script checks that every result reaches the request it belongs to.
For each case the script prints the throughput ; the 50th, 90th and 99th latency percentiles over all requests (NumPy’s default linear interpolation); the mean number of requests in the system , waiting or in service, averaged over the window between the arrivals of requests and ; and Little’s law’s relative residual in that window, with the completions in the window per unit time and their mean latency. A case with is marked over: it has no steady state. For each timeout and arrival seed 0 to 4, it then lists the batch sizes with the lowest 99th percentile at each load (ties at 0.01 ms), among those that keep up. Separate runs check the two limits of the model: a saturated load of 1,000 requests per second, whose throughput should approach , and a vanishing load of 0.01 per second, at which a request with should take exactly ms.
Part 2 counts the parameters, multiply-accumulates and activations of a synthetic plain network: convolutions without bias and with zero padding of one pixel, each followed by ReLU; a stem from 3 to channels at , and three stages of layers with widths , and at resolutions , and , the first layer of the second and third stages having stride 2. The counts are read from the arrays of a NumPy forward pass on a synthetic RGB frame, with untrained random weights, since only the shapes matter: parameters from the weight arrays, multiply-accumulates from the dimensions of each convolution’s im2col matrix product (padded positions included), and activations from the convolution outputs, Dollár et al.’s definition. Our derivation for this network, with , gives the closed forms
against which the counts are compared exactly. The base network has , and ; each strategy scales one dimension so that Dollár et al.’s per-stage multiply-adds would grow by : width by , depth by , resolution by . For each scaled network the script prints the ratios of multiply-accumulates, parameters and activations to the base, and the exponents and likewise, which Dollár et al.’s Table 1 puts at for width, for depth and for resolution. The im2col convolution is first checked against SciPy’s correlate.
import os
for v in ("OPENBLAS_NUM_THREADS", "OMP_NUM_THREADS", "MKL_NUM_THREADS"):
os.environ[v] = "1" # one BLAS thread, set before NumPy loads
import numpy as np
from scipy.signal import correlate
# ---------- Part 1: dynamic batching on a simulated clock (times in ms) ----------
A, C = 8.0, 2.0 # batch service time t(b) = A + C*b (illustrative)
N = 20_000 # requests per run
BS = [1, 2, 4, 8, 16, 32] # maximum batch size
TAUS = [0.0, 5.0] # timeout: how long the oldest request may wait for a batch to fill
LAMS = [0.5, 60, 200, 380] # arrival rates, requests per second
rng = np.random.default_rng(1000) # each request carries a 16x16 frame whose label is
labels = rng.integers(0, 4, N) # the column band (0-3) of a bright 4x4 square
imgs = rng.integers(0, 41, (N, 16, 16)).astype(np.uint8)
for L in range(4):
imgs[np.ix_(labels == L, np.arange(6, 10), np.arange(4 * L, 4 * L + 4))] = 200
def model(batch):
"""Toy batched classifier: the brightest column band of each frame."""
prof = batch.sum(axis=1, dtype=np.int64) # (k, 16) column sums
return prof.reshape(len(batch), 4, 4).sum(2).argmax(1)
def arrivals(lam, seed):
return np.cumsum(np.random.default_rng(seed).standard_exponential(N)) * 1000 / lam
def simulate(arr, B, tau):
"""FIFO server: when free, dispatch at once if B requests wait, otherwise wait until
B have arrived or the oldest has waited tau; then serve up to B together."""
done = np.empty(N); h, free, correct = 0, 0.0, 0
while h < N:
start = max(free, arr[h])
full = arr[h + B - 1] if h + B - 1 < N else np.inf
t = max(start, min(arr[h] + tau, full)) # dispatch time
k = min(B, np.searchsorted(arr, t, side="right") - h)
free = t + A + C * k
done[h:h + k] = free
correct += int((model(imgs[h:h + k]) == labels[h:h + k]).sum())
h += k
return done - arr, done, correct
def measures(arr, lat, done):
X = N / (done.max() - arr[0]) * 1000 # throughput, requests/s
t0, t1 = arr[N // 10], arr[9 * N // 10] # steady-state window
nbar = np.clip(np.minimum(done, t1) - np.maximum(arr, t0), 0, None).sum() / (t1 - t0)
out = (done >= t0) & (done <= t1)
xw, wbar = out.sum() / (t1 - t0), lat[out].mean() # per ms, ms
return X, np.percentile(lat, [50, 90, 99]), nbar, abs(nbar - xw * wbar) / nbar
cap = {B: B / (A + C * B) * 1000 for B in BS} # requests/s with full batches
runs = ok = 0
best = {}
for tau in TAUS:
print(f"tau = {tau:.0f} ms (cap = B/(a+cB); 'over' = lambda >= cap; Nbar = mean number in system)")
print(" lam/s B cap/s thru/s p50 p90 p99 Nbar Little rel. resid.")
for lam in LAMS:
arr = arrivals(lam, 0)
for B in BS:
lat, done, corr = simulate(arr, B, tau); runs += 1; ok += corr
X, (p50, p90, p99), nbar, res = measures(arr, lat, done)
over = lam >= cap[B]
print(f"{lam:6.1f} {B:3d} {cap[B]:6.1f} {X:7.1f} {p50:7.2f} {p90:7.2f} {p99:7.2f} {nbar:8.3f} "
+ ("over" if over else f"{res:.5f}"))
# H3: which stable B gives the lowest p99 (ties at 0.01 ms), seeds 0-4
for seed in range(5):
sets = []
for lam in LAMS:
arr = arrivals(lam, seed); p = {}
for B in BS:
if lam < cap[B]:
lat, done, corr = simulate(arr, B, tau); runs += 1; ok += corr
p[B] = round(float(np.percentile(lat, 99)), 2)
sets.append({B for B in p if p[B] == min(p.values())})
common = set.intersection(*sets)
print(f" tau {tau:.0f}, seed {seed}: lowest-p99 B at lam {LAMS}: "
+ " ".join("{" + ",".join(map(str, sorted(s))) + "}" for s in sets)
+ f" B best at every load: {sorted(common) if common else 'none'}")
print("saturated, lam = 1000/s: throughput (requests/s) against B/(a+cB)")
for tau in TAUS:
arr = arrivals(1000, 0); xs = []
for B in BS:
lat, done, corr = simulate(arr, B, tau); runs += 1; ok += corr
xs.append(N / (done.max() - arr[0]) * 1000)
err = max(abs(x / cap[B] - 1) for x, B in zip(xs, BS))
print(f" tau {tau:.0f}: " + " ".join(f"{x:6.1f}" for x in xs)
+ f" theory " + " ".join(f"{cap[B]:6.1f}" for B in BS)
+ f" max |X/cap - 1| = {err:.5f} 1/c = {1000 / C:.0f}")
for tau in TAUS:
arr = arrivals(0.01, 0)
lat, done, corr = simulate(arr, 1, tau); runs += 1; ok += corr
p50, p99 = np.percentile(lat, [50, 99])
print(f"vanishing load 0.01/s, B=1, tau {tau:.0f}: p50 {p50:.6f} p99 {p99:.6f} ms, "
f"|p99-(a+c)| = {abs(p99 - A - C):.1e}; requests that waited: {(lat > A + C + 1e-6).sum()}")
print(f"responses with their own frame's label: {ok} of {runs * N} in {runs} runs")
# ---------- Part 2: parameters, multiply-adds and activations, counted exactly ----------
K = 3
def conv(x, W, stride):
"""3x3 convolution (cross-correlation), zero padding 1, via im2col; returns output and MACs."""
cin, H, Wd = x.shape
Ho, Wo = (H - 1) // stride + 1, (Wd - 1) // stride + 1
xp = np.pad(x, ((0, 0), (1, 1), (1, 1)))
cols = np.stack([xp[:, dy:dy + stride * Ho:stride, dx:dx + stride * Wo:stride]
for dy in range(K) for dx in range(K)], axis=1).reshape(cin * K * K, Ho * Wo)
Wm = W.reshape(W.shape[0], cin * K * K)
return (Wm @ cols).reshape(-1, Ho, Wo), Wm.shape[0] * Wm.shape[1] * cols.shape[1]
def frame(r):
"""Synthetic RGB frame: a colour gradient, a disk and a rectangle."""
y, x = np.mgrid[0:r, 0:r] / r
img = np.stack([x, y, 0.5 * (x + y)])
img[:, (x - 0.35) ** 2 + (y - 0.4) ** 2 < 0.04] = np.array([[0.9], [0.2], [0.1]])
img[:, (x > 0.6) & (x < 0.85) & (y > 0.55) & (y < 0.8)] = np.array([[0.1], [0.3], [0.9]])
return img
def network(w, d, r):
"""Stem 3->w, then stages of d layers at widths w, 2w, 4w; stride 2 opens stages 2 and 3."""
rng = np.random.default_rng(0)
x, p = frame(r), 0
macs = acts = 0
layers = [(3, w, 1)] + [(w, w, 1)] * d
for j in (1, 2):
layers += [(w * 2 ** (j - 1), w * 2 ** j, 2)] + [(w * 2 ** j, w * 2 ** j, 1)] * (d - 1)
for cin, cout, s in layers:
W = rng.normal(0, np.sqrt(2 / (cin * K * K)), (cout, cin, K, K))
x, m = conv(x, W, s)
x = np.maximum(x, 0)
p, macs, acts = p + W.size, macs + m, acts + x.size
return p, macs, acts
def closed(w, d, r):
return (K * K * (3 * w + (21 * d - 10) * w * w),
K * K * r * r * (3 * w + (3 * d - 1) * w * w),
w * r * r * (4 + 7 * d) // 4)
# library cross-check of the convolution, stride 1 and stride 2
x = frame(32); W = np.random.default_rng(1).normal(size=(4, 3, K, K))
ref = np.stack([sum(correlate(x[c], W[o, c], mode="same", method="direct") for c in range(3))
for o in range(4)])
d1 = np.abs(conv(x, W, 1)[0] - ref).max(); d2 = np.abs(conv(x, W, 2)[0] - ref[:, ::2, ::2]).max()
print(f"im2col conv vs scipy.signal.correlate: max |diff| stride 1 {d1:.1e}, stride 2 {d2:.1e}")
w0, d0, r0 = 8, 2, 32
cases = [("base", 1, w0, d0, r0)]
for s in (4, 16):
q = int(np.sqrt(s))
cases += [("width", s, w0 * q, d0, r0), ("depth", s, w0, d0 * s, r0), ("resolution", s, w0, d0, r0 * q)]
print("strategy s w d r params MACs activations =closed "
"MACx px ax e_p e_a")
P0, M0, A0 = network(w0, d0, r0)
expo = {}
for name, s, w, d, r in sorted(cases, key=lambda c: ("bwdr".index(c[0][0]), c[1])):
p, m, a = network(w, d, r)
match = (p, m, a) == closed(w, d, r)
fx, px, ax = m / M0, p / P0, a / A0
ep, ea = (np.log(px) / np.log(fx), np.log(ax) / np.log(fx)) if s > 1 else (np.nan, np.nan)
expo[name, s] = (ep, ea)
print(f"{name:10s} {s:2d} {w:4d} {d:3d} {r:4d} {p:11,d} {m:13,d} {a:13,d} {str(match):5s} "
f"{fx:6.3f} {px:6.3f} {ax:6.3f} {ep:7.4f} {ea:7.4f}")
dollar = {"width": (1.0, 0.5), "depth": (1.0, 1.0), "resolution": (0.0, 1.0)}
order = all(expo["width", s][1] < min(expo["depth", s][1], expo["resolution", s][1]) for s in (4, 16))
print(f"e_a(width) below e_a(depth) and e_a(resolution) at s = 4 and 16: {order}")
for name, (dp, da) in dollar.items():
g4 = [abs(expo[name, 4][i] - t) for i, t in enumerate((dp, da))]
g16 = [abs(expo[name, 16][i] - t) for i, t in enumerate((dp, da))]
print(f" {name:10s} gap to Dollar's (e_p, e_a) = ({dp:.1f}, {da:.1f}): s=4 ({g4[0]:.4f}, {g4[1]:.4f})"
f" s=16 ({g16[0]:.4f}, {g16[1]:.4f}) not growing: {all(b <= a + 1e-12 for a, b in zip(g4, g16))}")
Run with NumPy 2.3.5 and SciPy 1.16.3, it printed the same output in two runs, taking about 14 seconds (approximate; the machine was shared):
tau = 0 ms (cap = B/(a+cB); 'over' = lambda >= cap; Nbar = mean number in system)
lam/s B cap/s thru/s p50 p90 p99 Nbar Little rel. resid.
0.5 1 100.0 0.5 10.00 10.00 10.00 0.005 0.00000
0.5 2 166.7 0.5 10.00 10.00 10.00 0.005 0.00000
0.5 4 250.0 0.5 10.00 10.00 10.00 0.005 0.00000
0.5 8 333.3 0.5 10.00 10.00 10.00 0.005 0.00000
0.5 16 400.0 0.5 10.00 10.00 10.00 0.005 0.00000
0.5 32 444.4 0.5 10.00 10.00 10.00 0.005 0.00000
60.0 1 100.0 60.4 13.79 30.86 56.29 1.063 0.00000
60.0 2 166.7 60.4 11.58 19.95 27.99 0.832 0.00001
60.0 4 250.0 60.4 11.32 19.63 23.68 0.817 0.00001
60.0 8 333.3 60.4 11.31 19.63 23.65 0.817 0.00001
60.0 16 400.0 60.4 11.31 19.63 23.65 0.817 0.00001
60.0 32 444.4 60.4 11.31 19.63 23.65 0.817 0.00001
200.0 1 100.0 100.0 50355.77 90609.62 99631.24 5028.540 over
200.0 2 166.7 166.6 10390.09 18650.44 20470.29 1717.228 over
200.0 4 250.0 201.4 23.71 39.94 62.91 5.310 0.00001
200.0 8 333.3 201.4 22.13 31.58 42.38 4.614 0.00001
200.0 16 400.0 201.4 22.10 31.56 41.89 4.607 0.00001
200.0 32 444.4 201.4 22.10 31.56 41.89 4.607 0.00001
380.0 1 100.0 100.0 73873.82 132951.87 146227.02 7385.467 over
380.0 2 166.7 166.7 33878.82 60960.73 67036.02 5640.762 over
380.0 4 250.0 250.0 13889.80 24974.48 27446.02 3461.756 over
380.0 8 333.3 333.1 3910.67 7004.64 7672.08 1289.751 over
380.0 16 400.0 382.3 62.38 109.17 159.36 27.178 0.00127
380.0 32 444.4 382.4 54.95 83.59 110.97 22.136 0.00084
tau 0, seed 0: lowest-p99 B at lam [0.5, 60, 200, 380]: {1,2,4,8,16,32} {8,16,32} {16,32} {32} B best at every load: [32]
tau 0, seed 1: lowest-p99 B at lam [0.5, 60, 200, 380]: {1,2,4,8,16,32} {8,16,32} {16,32} {32} B best at every load: [32]
tau 0, seed 2: lowest-p99 B at lam [0.5, 60, 200, 380]: {1,2,4,8,16,32} {4} {16,32} {32} B best at every load: none
tau 0, seed 3: lowest-p99 B at lam [0.5, 60, 200, 380]: {1,2,4,8,16,32} {8,16,32} {16,32} {32} B best at every load: [32]
tau 0, seed 4: lowest-p99 B at lam [0.5, 60, 200, 380]: {1,2,4,8,16,32} {8,16,32} {16,32} {32} B best at every load: [32]
tau = 5 ms (cap = B/(a+cB); 'over' = lambda >= cap; Nbar = mean number in system)
lam/s B cap/s thru/s p50 p90 p99 Nbar Little rel. resid.
0.5 1 100.0 0.5 10.00 10.00 10.00 0.005 0.00000
0.5 2 166.7 0.5 15.00 15.00 15.00 0.008 0.00000
0.5 4 250.0 0.5 15.00 15.00 15.00 0.008 0.00000
0.5 8 333.3 0.5 15.00 15.00 15.00 0.008 0.00000
0.5 16 400.0 0.5 15.00 15.00 15.00 0.008 0.00000
0.5 32 444.4 0.5 15.00 15.00 15.00 0.008 0.00000
60.0 1 100.0 60.4 13.79 30.86 56.29 1.063 0.00000
60.0 2 166.7 60.4 15.00 20.03 27.89 0.964 0.00004
60.0 4 250.0 60.4 15.13 19.75 24.34 0.996 0.00004
60.0 8 333.3 60.4 15.14 19.79 24.38 0.997 0.00004
60.0 16 400.0 60.4 15.14 19.79 24.38 0.997 0.00004
60.0 32 444.4 60.4 15.14 19.79 24.38 0.997 0.00004
200.0 1 100.0 100.0 50355.77 90609.62 99631.24 5028.540 over
200.0 2 166.7 166.6 10384.91 18645.27 20465.11 1716.366 over
200.0 4 250.0 201.4 23.78 39.76 63.25 5.344 0.00002
200.0 8 333.3 201.4 22.25 31.76 42.12 4.689 0.00003
200.0 16 400.0 201.4 22.24 31.74 41.95 4.684 0.00003
200.0 32 444.4 201.4 22.24 31.74 41.95 4.684 0.00003
380.0 1 100.0 100.0 73873.82 132951.87 146227.02 7385.467 over
380.0 2 166.7 166.7 33878.62 60957.29 67034.53 5640.543 over
380.0 4 250.0 250.0 13888.10 24970.91 27446.18 3461.442 over
380.0 8 333.3 333.1 3909.49 7004.27 7674.61 1289.615 over
380.0 16 400.0 382.2 62.64 109.82 161.70 27.347 0.00145
380.0 32 444.4 382.4 54.91 84.25 112.47 22.184 0.00200
tau 5, seed 0: lowest-p99 B at lam [0.5, 60, 200, 380]: {1} {4} {16,32} {32} B best at every load: none
tau 5, seed 1: lowest-p99 B at lam [0.5, 60, 200, 380]: {1} {8,16,32} {16,32} {32} B best at every load: none
tau 5, seed 2: lowest-p99 B at lam [0.5, 60, 200, 380]: {1} {4} {16,32} {32} B best at every load: none
tau 5, seed 3: lowest-p99 B at lam [0.5, 60, 200, 380]: {1} {4} {16,32} {32} B best at every load: none
tau 5, seed 4: lowest-p99 B at lam [0.5, 60, 200, 380]: {1} {4} {16,32} {32} B best at every load: none
saturated, lam = 1000/s: throughput (requests/s) against B/(a+cB)
tau 0: 100.0 166.7 250.0 333.3 399.9 444.2 theory 100.0 166.7 250.0 333.3 400.0 444.4 max |X/cap - 1| = 0.00053 1/c = 500
tau 5: 100.0 166.7 250.0 333.3 399.9 444.2 theory 100.0 166.7 250.0 333.3 400.0 444.4 max |X/cap - 1| = 0.00047 1/c = 500
vanishing load 0.01/s, B=1, tau 0: p50 10.000000 p99 10.000000 ms, |p99-(a+c)| = 0.0e+00; requests that waited: 1
vanishing load 0.01/s, B=1, tau 5: p50 10.000000 p99 10.000000 ms, |p99-(a+c)| = 0.0e+00; requests that waited: 1
responses with their own frame's label: 4840000 of 4840000 in 242 runs
im2col conv vs scipy.signal.correlate: max |diff| stride 1 1.8e-15, stride 2 1.8e-15
strategy s w d r params MACs activations =closed MACx px ax e_p e_a
base 1 8 2 32 18,648 3,170,304 36,864 True 1.000 1.000 1.000 nan nan
width 4 16 2 32 74,160 12,238,848 73,728 True 3.860 3.977 2.000 1.0220 0.5131
width 16 32 2 32 295,776 48,070,656 147,456 True 15.163 15.861 4.000 1.0166 0.5099
depth 4 8 8 32 91,224 13,787,136 122,880 True 4.349 4.892 3.333 1.0801 0.8191
depth 16 8 32 32 381,528 56,254,464 466,944 True 17.744 20.459 12.667 1.0495 0.8828
resolution 4 8 2 64 18,648 12,681,216 147,456 True 4.000 1.000 4.000 0.0000 1.0000
resolution 16 8 2 128 18,648 50,724,864 589,824 True 16.000 1.000 16.000 0.0000 1.0000
e_a(width) below e_a(depth) and e_a(resolution) at s = 4 and 16: True
width gap to Dollar's (e_p, e_a) = (1.0, 0.5): s=4 (0.0220, 0.0131) s=16 (0.0166, 0.0099) not growing: True
depth gap to Dollar's (e_p, e_a) = (1.0, 1.0): s=4 (0.0801, 0.1809) s=16 (0.0495, 0.1172) not growing: True
resolution gap to Dollar's (e_p, e_a) = (0.0, 1.0): s=4 (0.0000, 0.0000) s=16 (0.0000, 0.0000) not growing: True
The limits of the model. The four checks of the simulation held. At a vanishing load the median and 99th percentile with are exactly 10 ms, , for both timeouts (one request in 20,000 arrived while another was in service). Under saturation the throughput stays within at most 0.053% of , inside the bar of 0.2%, and rises from 100.0 to 444.2 requests per second as grows from 1 to 32, every value below , so H1, throughput rising with toward at high load, held for both timeouts. Little’s law held in every case that keeps up: the largest relative residual is 0.00200, against a bar of 0.01. Every one of the 4,840,000 responses carried its own frame’s label.
Low load. At 0.5 requests per second, every batch size gives 10.00 ms at every percentile when the server does not wait. With a 5 ms timeout, every gives 15.00 ms, exactly more than , so H2, that a timeout raises the 99th percentile by about at low load, held. This follows from the construction: at this rate the utilization of is , so fewer than 1% of requests find the server busy, and a lone request with waits out the whole timeout.
Moderate load. At 60 requests per second still keeps up, at a utilization of 0.6, but its 99th percentile is 56.29 ms against 23.65 ms for without a timeout, with 1.063 requests in the system on average against 0.817: from the construction, requests that arrive while one is being served wait for it, and a batch serves the waiting ones together. The timeout costs most at the median here: with the median rises from 11.32 ms without it to 15.13 ms with it, while the 99th percentile moves from 23.68 to 24.34 ms.
High load. At 200 requests per second and cannot keep up: their throughput stops at their capacities, 100.0 and 166.6 per second, and their latencies, ten seconds and more, measure only how long the run lasted before the backlog was cleared. At 380 per second only , at a utilization of 0.95, and keep up, and the 99th percentile of the larger batch is the lower, 110.97 ms against 159.36 ms without a timeout.
Which batch size is best. H3 was that the batch size with the lowest 99th percentile depends on the load, failing if one batch size is best or tied for best at every load. With the 5 ms timeout, H3 held on all five arrival seeds: is best at 0.5 per second, (or 8, 16 and 32 together, on seed 1) at 60, 16 and 32 at 200, and 32 at 380. Without a timeout, H3 failed on seeds 0, 1, 3 and 4, on which is best or tied for best at every load, and held on seed 2, on which alone is best at 60 per second.
The counts. The check of the im2col convolution against SciPy held, with a largest difference of at both strides against a bar of , and H4, that the counted parameters, multiply-accumulates and activations equal the closed forms exactly, held in all seven rows. The scaled networks show Dollár et al.’s point in numbers. At the three strategies raise the multiply-accumulates by 15.2, 17.7 and 16.0 times, but the activations by 4.0 times for width, 12.7 for depth and 16.0 for resolution, and the parameters by 15.9, 20.5 and 1.0 times. Both parts of the comparison with Dollár et al.’s Table 1 held: the width exponent (0.5131 at , 0.5099 at 16) is below the depth and resolution exponents at both scales, and every gap to the table’s exponents shrinks or stays at zero from to .
The gaps that remain follow from the closed forms the example checks (our derivation). The table describes a stage of identical layers; this network also has a stem and two strided layers. Under width scaling the stem’s multiply-accumulates grow only by , so the total grows by a little less than (15.163 at ) and sits slightly above 0.5. Under depth scaling the stem’s activations do not grow with , while the multiply-accumulates of the layers after the stem grow as against , a little faster than (17.744 at , with the stem included), so falls below 1, to 0.8828 at ; both terms weigh less as grows, which is why the gap shrinks. Resolution scaling multiplies every layer’s output by and leaves the weights alone, so its exponents are exactly 1 and 0. In our reading, these networks would differ in speed on memory-bound hardware much more than their multiply-adds suggest, which is what Dollár et al. measured on GPUs; the example counts and does not time.
What the simulation leaves out. The device’s cost is deterministic and affine, with no variation between batches, no transfer of inputs, no preprocessing and no contention with other work, and arrivals are Poisson. A real device’s batch cost is measured, not assumed; with variable service times the tail grows further, as the real-time vision pipelines example shows for a pipeline with a randomly slow detector.
How the example was planned. The hypotheses H1 to H4 and the checks were written before any script ran. A planning script that ran this example’s set-up with extra prints then printed every outcome before the plan was frozen, so the example illustrates derivations rather than testing them. Two bars were set after that script: 0.2% for the saturated throughput and 0.01 for Little’s law’s residual. That script’s running time also served as the runtime estimate. Compared with the batch plan’s description, the example adds the zero timeout beside the 5 ms one; fixes the arrival rates, choosing 0.5 per second so that fewer than 1% of requests find ’s server busy; gives each request a synthetic frame with a check of its label; evaluates H3 over five arrival seeds with ties at 0.01 ms; measures the plan’s mean queue length as the mean number in the system, in a window that leaves out the first and last 10% of arrivals; checks the vanishing-load latency through its median and 99th percentile rather than every request; measures saturated throughput over the whole run; compares the counts with closed forms for a network with a stem and strided stages, and Dollár et al.’s exponents with the gaps described above; and checks the convolution against SciPy. The batch plan called the affine cost model “the shape Model Deployment describes for Clipper”; that article describes batching as amortizing per-call overheads, and the affine model here is our own illustrative choice. The pilot and the first run printed the same output.
Common Misconceptions
- “Higher throughput means lower latency.” Batching and pipelining raise throughput while adding waiting to each request. The two improve together only when queueing dominates, as at 200 requests per second in the example, where and cannot keep up and larger batches both raise the throughput to the arrival rate and drain the queue.
- “Batch size 1 gives the lowest latency.” Only when the server is lightly loaded. Near capacity, a batch of one leaves requests queueing behind each other; in the example cannot keep up at 200 requests per second, where keeps up with a 99th percentile of 41.89 ms.
- “A model with fewer FLOPs is faster.” Multiply-adds are an indirect measure. Networks scaled to similar multiply-adds in the example reach 4.0 and 16.0 times the base network’s activations (width and resolution scaling at ), which Dollár et al. found to track GPU runtime closely; the MobileNet and edge inference articles show the same effect through memory access and the roofline.
- “FLOPs are floating-point operations.” In several vision papers, Dollár et al.’s, Radosavovic et al.’s and Ma et al.’s among them, they are multiply-adds, half the count of operations that other papers report under the same name.
- “Parameters measure cost.” Parameters do not depend on the input size: scaling the resolution by four in the example multiplies the arithmetic and the activations by 16 and leaves the parameters unchanged.
- “The fastest of several timings is the system’s latency.” In our reading, the minimum is the right figure for comparing implementations of a component, because it removes interference; the latency users see includes interference and queueing, and is described by percentiles under a realistic load.
Where It Is Used
Every deployed vision model is chosen against a latency or throughput requirement, set as part of vision system design. Servers that answer many users batch requests and tune batch sizes and timeouts, as described in model deployment; devices that process one camera frame at a time care about single-image latency, energy and heat, the subject of edge inference; and systems that act on live video are judged on the end-to-end latency of the real-time pipeline around the model. Detectors such as YOLO are reported on speed–accuracy curves of the kind Huang et al. drew for object detection. The speed proxies guide the design of efficient networks such as MobileNet and EfficientNet, whose depthwise separable convolutions save multiply-adds more than memory traffic; neural architecture search can search directly on measured latency; and model compression and quantization trade accuracy for speed, a trade that is only meaningful when the speed is measured on named hardware.
Related
- Deploying Vision Models
Getting a trained vision model into use, from export and the choice of runtime to packaging the model with its preprocessing contract, serving it on a server or a device, keeping pre- and post-processing identical to training, and versioning, rolling back and monitoring it.
- Model Compression
Making a trained vision model smaller and cheaper to run by pruning weights or filters, quantizing, distilling it into a smaller network, factorizing its layers or choosing a compact architecture, and how to compare the results fairly in accuracy, size, operations and measured latency.
- MobileNet
Google's family of convolutional networks for phones and other small devices, from MobileNet v1 (2017), built from depthwise separable convolutions with width and resolution multipliers, through MobileNetV2's inverted residuals with linear bottlenecks and MobileNetV3's latency-driven architecture search, to MobileNetV4's universal inverted bottleneck and mobile attention (2024).
- EfficientNet
Google's family of convolutional networks built by scaling one searched baseline, EfficientNet-B0, in depth, width and input resolution together with a single compound coefficient (B1 to B7, 2019), followed by EfficientNetV2 (2021), which mixes fused and depthwise blocks and grows the image size during training to train faster.
- Designing a Computer Vision System
How a working vision system is designed from its requirements, through the choice of sensor, optics, lighting, compute and model, to the data it is trained on, its validation in the field and its maintenance once deployed, with the requirement arithmetic for a measurement station worked through in code.
- GPUs for Computer Vision
The graphics processing unit as a machine for vision workloads, covering SIMT execution in warps, latency hiding by many resident threads, the memory hierarchy from registers to host memory, data reuse by tiling, why convolutions suit GPUs and element-wise operations are limited by bandwidth, TF32 and reduced precision, memory capacity, host–device transfer, GPUs from other vendors and how to choose one.
References
- Dollár, P., Singh, M. & Girshick, R. (2021). Fast and Accurate Model Scaling. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (read in the CVF open-access copy).
- Radosavovic, I., Kosaraju, R. P., Girshick, R., He, K. & Dollár, P. (2020). Designing Network Design Spaces. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 10425–10433 (read in the CVF open-access copy).
- Huang, J., Rathod, V., Sun, C., Zhu, M., Korattikara, A., Fathi, A., Fischer, I., Wojna, Z., Song, Y., Guadarrama, S. & Murphy, K. (2017). Speed/Accuracy Trade-Offs for Modern Convolutional Object Detectors. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (read in the CVF open-access copy).
- Ma, N., Zhang, X., Zheng, H.-T. & Sun, J. (2018). ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design. European Conference on Computer Vision (ECCV), Lecture Notes in Computer Science, 122–138 (read in the CVF open-access copy).
- Little, J. D. C. (1961). A Proof for the Queuing Formula: L = λW. Operations Research, 9(3), 383–387 (not read; cited as Real-Time Vision Pipelines states it).
- Python Software Foundation. timeit: Measure execution time of small code snippets, Python 3.15.2 documentation (accessed November 2026).
- Python Software Foundation. time: Time access and conversions, Python 3.15.2 documentation (accessed November 2026).