Real-Time Vision Pipelines

A real-time vision pipeline is the chain that takes light arriving at a camera to a decision a system acts on: exposure and readout in the camera, transfer to a host, decoding and preprocessing, a model such as an object detector, postprocessing, and finally an output that drives a display, a robot arm or a brake. Each stage takes time, the camera delivers frames at its own rate whatever the rest of the system does, and a decision is only useful if it describes the world as it is when the decision is used. Most of the engineering of such a pipeline is about time: how often frames arrive, how long each takes to reach a decision, what happens to frames that arrive while the system is busy, and which moment each result actually describes.

This article covers the timing of the whole chain. How the camera forms and processes an image is in digital camera and image sensor; making the model itself faster belongs to quantization and model compression, and packaging it for a runtime or a device to model deployment and edge inference.

Definition

A vision pipeline is real-time when it keeps up with its input and delivers each result within a deadline set by the application. Two quantities describe it:

• Throughput: the number of frames that reach a decision per second.
• Latency: for each frame, the time from the moment the image represents (its exposure) to the moment its decision is available. This article calls it capture-to-decision latency; when the output is shown on a screen, the time until the corresponding light leaves the display is the glass-to-glass delay, from the camera lens to the display glass (Bachhuber & Steinbach, 2016).

The frame budget is the time between frames, $1/f$ for a frame rate $f$: 33.3 ms at 30 frames per second and 16.7 ms at 60. A pipeline that spends more than the budget per frame, on average, cannot process every frame. It must drop some or fall further and further behind.

The word “real-time” is used loosely. The YOLO paper called a detector real-time if it ran at 30 frames per second or better, a throughput criterion (Redmon et al., 2016). Whether a system meets its deadline depends instead on latency, and on the whole pipeline, not on the model alone.

Intuition

Picture a kitchen that receives an order every 33 ms. If one cook does everything, chopping, cooking and plating, and the three take 38 ms together, orders pile up whatever the cook does. Hiring one cook per station, a pipeline, lets three orders be in progress at once; orders now leave as fast as the slowest station works, but each order still spends at least 38 ms in the kitchen. Pipelining raises throughput and does nothing for the latency of a single order.

The pile of waiting orders is the second lesson. A queue that never discards anything keeps every order but serves each one later and later. A video pipeline usually prefers the opposite: when the detector is busy, a frame that is 100 ms old is worth less than the frame that will arrive in a moment, so the right policy is often to throw stale frames away and process the newest one.

Li, Wang and Ramanan (2020) made the third point in their streaming-perception benchmark: by the time an algorithm finishes processing a frame, the world has moved on. A detection computed from a frame captured 200 ms ago describes where a car was. A real-time pipeline is judged by how well its latest output matches the present, not the past frame it came from.

Formal Definition

Let frames be exposed at times $t_i = i/f$, and let frame $i$ produce a decision at time $d_i$. Its latency is

$$L_i = d_i - t_i .$$

Latency is a distribution over frames, not one number. It is summarized by percentiles (the median and the 90th, 99th and maximum values), because a mean hides the slow frames that cause missed deadlines.

Let stage $k$ of $K$ take service time $s_k$ for a frame, with mean $\bar{s}_k$. If one worker runs all stages in turn, it can finish at most

$$X_{\text{seq}} = \min\Big(f,\; \frac{1}{\sum_k \bar{s}_k}\Big)$$

frames per second. If every stage has its own worker and stages overlap on different frames, throughput is set by the slowest stage, while the latency of each frame is still at least the sum of its stages:

$$X_{\text{pipe}} = \min\Big(f,\; \frac{1}{\max_k \bar{s}_k}\Big), \qquad L_i \ge \sum_k s_k .$$

The utilization of the slowest stage is $\rho = f \max_k \bar{s}_k$. If $\rho > 1$ and nothing is dropped, the queue in front of that stage grows without bound, by about $f - 1/\max_k \bar{s}_k$ frames per second (a direct consequence of arrivals outpacing service). Even with $\rho < 1$, variable service times make frames wait behind slow ones, as the example below shows.

Throughput and mean latency are tied by Little’s law (Little, 1961): in a system in steady state, the average number of items inside equals the arrival rate times the average time each spends inside,

$$\bar{N} = X \, \bar{W} .$$

A pipeline that delivers 30 decisions per second with a mean latency of 100 ms holds, on average, three frames in flight. More frames in flight, whether in queues, in batches or in pipeline stages, means more latency at the same throughput.

Properties

Latency and throughput are different quantities

Throughput can be bought with parallelism: more pipeline stages, more workers, batches of frames processed together. None of these shortens a frame’s own path, and batching lengthens it, since a frame waits for the batch to fill. Latency is reduced only by doing less work per frame, doing it faster, or waiting less.

A model’s reported speed is a different quantity from a system’s latency. The YOLO paper reported 45 frames per second on a Titan X GPU without batching and concluded that it could process streaming video with less than 25 ms of latency (Redmon et al., 2016); that figure is the network’s time per image, and a deployed system adds capture, transfer, preprocessing, postprocessing and display to it. Li et al. (2020) found that many detector implementations were optimized for throughput rather than latency: image preprocessing was often done on the CPU, where it can overlap with loading the next image, and moving it to the GPU saved 21 ms of latency for a $960 \times 600$ input in their setting.

Pipelining, threads and processes

A pipeline overlaps stages by running them in separate threads, processes or devices, joined by queues. In GStreamer, an open-source multimedia framework whose documented use cases include capturing from a live camera and displaying the processed result, some elements start their own streaming threads, and the queue element is the explicit thread boundary: a producer–consumer buffer that hands data between threads safely and absorbs short hiccups (GStreamer, Clocks and synchronization; Threads; queue). The manual also warns against a thread per element, because threads have overhead.

In Python, threads have a specific limit. In CPython only one thread executes Python bytecode at a time, because of the global interpreter lock (GIL). The documentation recommends processes (multiprocessing) for CPU-bound work, and notes that some extension modules release the lock during heavy computation and that it is always released during I/O (Python documentation, threading; glossary). Threads can therefore overlap camera I/O, and computation inside native libraries that release the lock, but not pure-Python loops. As of Python 3.13 a free-threaded build can disable the GIL; as of the 3.15.2 documentation it is not the default build. Processes avoid the lock but pay for moving frames between address spaces: a $1920 \times 1080$ RGB frame is about 6.2 MB, so copying every frame at 30 frames per second moves about 187 MB/s per hop (our arithmetic); passing frames through shared memory, or keeping the pipeline in one process or on one device, avoids these copies.

The model of the example below gives each stage its own core or accelerator. On real hardware, stages compete for CPU cores, memory bandwidth and the GPU. Li et al. (2020) evaluated most of their methods under a model of a single GPU that runs one job at a time, which avoids interference between jobs and memory limits, with light CPU jobs such as box association allowed alongside.

Queues, dropped frames and back-pressure

Whenever a producer can outrun a consumer, a queue sits between them, and its policy decides what happens on overload:

• Unbounded FIFO. Every frame is kept and served in order. Nothing is dropped, but under overload latency grows without limit, and memory with it.
• Bounded FIFO with blocking (back-pressure). When the queue is full, the producer waits. Back-pressure slows a source that can be slowed, such as a file decoder. A live camera cannot be slowed, so the waiting pushes the overflow upstream until a stage that cannot wait, such as the capture driver, drops frames (our reasoning).
• Bounded FIFO dropping the newest. Frames arriving at a full queue are discarded; the queued, older frames are served. A capture driver that has no free buffer behaves this way: the Video4Linux2 (V4L2) API, the Linux camera interface, defines a dropped frame as one the device received but could not store for lack of free buffer space (Linux kernel documentation, V4L2 Buffers).
• Keep-latest (drop the oldest). A queue of length one that overwrites its content: the consumer always takes the freshest frame. Latency stays bounded and decisions are as current as the consumer’s speed allows.

The defaults of common tools differ, and they matter. GStreamer’s queue holds up to 200 buffers, 10 MB or one second of data, whichever is reached first, and blocks the upstream thread when full; its leaky property makes it drop new buffers (upstream) or old ones (downstream) instead (GStreamer, queue). The appsink element, through which applications receive frames, keeps an unlimited queue by default (max-buffers of 0), and its documentation warns that raw video can then consume a lot of memory if the application pulls too slowly; since GStreamer 1.28 its leaky-type property selects whether to drop the oldest or newest buffers, deprecating the older drop property (GStreamer, appsink). In OpenCV 5.0.0 the V4L2 back end of VideoCapture requests 4 driver buffers by default and accepts 1 to 10 through CAP_PROP_BUFFERSIZE (OpenCV, cap_v4l.cpp). Python’s queue.Queue is unbounded unless given a positive maxsize (Python documentation, queue).

A consequence follows from these definitions (our reasoning, which the example below shows): a slow loop that reads from a driver queue of several frames processes frames that waited in the queue, so its results lag by several frame periods even though no single stage is slow. Shrinking the queue, or draining it and keeping only the last frame, removes the lag at the cost of discarding frames that would have been stale anyway.

GStreamer’s latency design records the same distinction formally. When a pipeline computes how long data may wait before it is late, an element with blocking buffering adds its buffer size to the maximum latency of the elements upstream, while an element with leaky buffering, such as a leaky queue or a live source with a fixed set of buffers, takes the minimum of the two, because it discards data rather than holding it (GStreamer, Latency).

Timestamps and clock domains

Every frame should carry the time it was captured, measured as close to the sensor as possible. The time it arrived at the host, or at a given stage, includes readout, transfer, buffering and scheduling delays that vary from frame to frame; using arrival times as capture times turns that jitter into errors in anything computed from time, such as velocities, track predictions or the alignment of camera and other sensor data.

Capture timestamps come from several clocks, each its own clock domain:

• The camera’s clock. GenICam’s Standard Features Naming Convention (SFNC) defines a device timestamp counter in nanoseconds, used to tag images, chunk data and events; the timestamp attached to an image as chunk data is taken at the frame’s start (SFNC 2.8, Sections 3.66 and 24.31). A frame starts with its optional exposure period and ends when sensor readout completes (SFNC 2.8, Figure 5-3).
• The host’s monotonic clock. In V4L2, a capture buffer’s timestamp comes from clock_gettime, with a flag naming the clock (for example CLOCK_MONOTONIC), and further flags say whether it was taken at the start of exposure or at the end of the frame, when the last pixel was received (Linux kernel documentation, V4L2 Buffers). The field’s own description speaks of the first data byte, so which instant a timestamp marks must be checked through the flags for each driver. OpenCV’s V4L2 back end returns this buffer timestamp as CAP_PROP_POS_MSEC (OpenCV, cap_v4l.cpp).
• The wall clock. Python’s time.time() returns calendar time, which can be adjusted; time.monotonic() cannot go backwards and is not affected by system clock updates, so only differences between its readings are meaningful (Python documentation, time). Latencies are differences, and belong on a monotonic clock.

Multimedia frameworks add their own time base. A GStreamer pipeline selects one clock and computes a running time from it; a live source stamps each buffer with the running time at which its first byte was captured, and sinks present a buffer when the clock reaches its running time plus the pipeline’s configured latency (GStreamer, Clocks and synchronization). Without that latency compensation, the documentation explains, a sink fed by a live source would find every buffer already late and drop it (GStreamer, Latency).

The V4L2 buffer also carries a sequence number that counts frames, including dropped ones, so gaps in the sequence reveal frames lost before the application saw them (Linux kernel documentation, V4L2 Buffers). Carrying a frame index and a capture timestamp through every stage is the cheapest way to make a pipeline’s timing observable.

Exposure and the rolling shutter as timing

A frame is not an instant. Its pixels integrate light over the exposure time, so a frame stamped at the start of exposure describes, on average, a moment half an exposure later (it follows from the integration); which instant a timestamp marks (start of exposure, start of frame, end of readout) is a convention that has to be known and applied consistently.

With a rolling shutter, row $r$ starts exposing at $t_0 + r\Delta$ for a line time $\Delta$, as in the image sensor article, so different rows of one frame have different capture times. For timing this means two things (our reasoning from that model): an object’s capture time depends on where it is in the image, and the frame cannot be complete before the last row is read out, which adds up to a full readout time to the latency of a pipeline that waits for whole frames. Rolling-shutter-aware odometry and SLAM model the camera pose as a function of each row’s capture time, as the image sensor article describes; for latency figures alone, taking the time of the middle row as the frame’s time keeps the error within half a readout (again our reasoning).

Synchronizing several cameras

Stereo rigs, multi-camera robots and camera–LiDAR systems need frames that describe the same moment. In outline, there are four levels of synchronization, from strongest to weakest (our grouping):

• Hardware triggering. Every camera starts exposure on the same electrical signal. GenICam’s naming convention lets a camera take its frame-start trigger from a physical input line (SFNC 2.8, Section 5.6).
• Network triggering and synchronized clocks. SFNC’s action signals trigger actions in several devices at once, for example a frame-start trigger, usually through broadcast protocol messages (SFNC 2.8, Section 14.1). Cameras that support the Precision Time Protocol (IEEE 1588) expose controls for it (SFNC 2.8, Section 27.3); the protocol synchronizes device clocks over the network, and SFNC describes it as controlling the device’s timestamp register (Section 27.4.38), so the timestamps of synchronized cameras share one time base.
• Software-paced capture. OpenCV’s documentation of VideoCapture::grab describes calling grab() on each camera first and the slower retrieve(), which decodes the frame, afterwards, so that frames from cameras without hardware synchronization are closer in time (OpenCV, videoio.hpp).
• Matching by timestamp. Free-running cameras are paired after capture by nearest timestamp, which requires their clocks to be mapped into one domain (by estimating offset and drift) and leaves up to half a frame period of mismatch for cameras at the same rate (our reasoning).

Synchronization errors become geometry errors when the scene or the rig moves: a stereo pair exposed a few milliseconds apart sees a moving object at two different positions, which reads as a disparity error (our reasoning). The same holds for multimodal fusion of camera and LiDAR data, where each firing of a scanning LiDAR has its own time as well.

Measuring Latency End to End

Glass-to-glass measurement treats the whole system as a black box. Bachhuber and Steinbach (2016) placed a light-emitting diode in the camera’s view and a phototransistor on the display where the diode appears, and recorded with a microcontroller, sampling at 2 kHz, the time from switching the diode on to the display brightening. They reported a precision of 0.5 ms. Their model explains why the result is a distribution rather than a number: the event happens at a random point in the camera’s frame period, adding a delay uniform over about one frame period, the display refresh adds another uniform delay of up to one refresh period, and processing and transfer add a roughly fixed delay. The sum is a trapezoid-shaped distribution whose width is about $1/f_{\text{cam}} + 1/f_{\text{display}}$. For a camera at 50 Hz and a 60 Hz monitor on a desktop PC, 250 measurements ranged from 19.1 to 52.4 ms, with a standard deviation of 6.9 ms and a 95% confidence interval for the mean of 32.4 to 34.1 ms. They switched the diode on at random intervals, because with constant intervals the changing phase between the camera’s and the display’s sampling made successive measurements strongly correlated. Raising the camera’s frame rate lowered every statistic: at 25 Hz they reported a minimum, mean and maximum of 24.8, 50.4 and 78.7 ms, and at 300 Hz 8.1, 15.5 and 23 ms.

Two lessons carry over to any pipeline. The frame rate sets a floor on latency even with infinitely fast processing, since an event waits up to a frame period to be captured; and repeated measurements must be decorrelated from the frame clock, since otherwise successive samples are correlated and tell less than their number suggests.

In-pipeline measurement stamps each frame at capture and records the time it leaves each stage, using one monotonic clock or clocks mapped into one domain. The decision time minus the capture timestamp gives capture-to-decision latency per frame; the per-stage differences show where the time goes, including the time spent waiting in queues, which a profile of each stage’s runtime alone never shows. Sequence numbers count the frames lost along the way. Reports then give percentiles over many frames, together with the throughput and the number of dropped frames, since a policy that drops frames can always lower latency.

Streaming accuracy evaluates latency and accuracy together. Li et al. (2020) query a system’s latest output at every frame time and score it against the ground truth for that time, so a slow detector is judged on stale boxes. On their Argoverse-HD validation set (urban driving video at 30 frames per second, timed on a GeForce GTX 1080 Ti), an accurate detector (Hybrid Task Cascade at full resolution, 700 ms per frame) scored 38.0 average precision offline but 6.2 in streaming evaluation, and a fast one (RetinaNet at 0.2 input scale) scored only 5.5 to 6.0 because it was inaccurate. The table’s caption describes the fast detector as running within the 33 ms frame interval, although the variant with CPU preprocessing takes 36.4 ms and only the GPU-preprocessing variant (31.2 ms) does. The best single detector sat in between: Mask R-CNN with a ResNet-50 backbone at half resolution, at 57 ms per frame, reached 12.0. Adding their scheduling policy, box association and a Kalman-filter forecast raised streaming average precision to 16.7, and to 17.8 with the detector’s input scale raised to 0.75 (all from their Tables 1 and 2). Their scheduling policy sometimes waits idle for the next frame rather than start on one that will be stale by the time processing ends.

Examples

A pipeline on a simulated clock

The script below simulates a 30 frames-per-second camera feeding preprocessing, a detector and postprocessing, on a simulated clock in milliseconds, so the result does not depend on the machine it runs on. Each frame is a $48 \times 64$ synthetic image with a bright square whose position encodes the frame index. The toy detector finds the square from the pixels and decodes which frame it is looking at, so each decision’s latency is computed from the image content rather than from bookkeeping. Service times are random but drawn once per frame from a seeded generator, so every configuration processes the same frames with the same costs. Preprocessing takes 3 to 5 ms, the detector 22 ms plus an exponentially distributed extra with a mean of 9 ms, and postprocessing 2 to 4 ms; readout and transfer add a fixed 5 ms. The simulation compares one worker running all stages with one worker per stage, and three queue policies: unbounded FIFO, a 4-slot ring that drops new frames when full, and keep-latest. It stops at the end of the 10 s capture window and counts what is still waiting as backlog.

import heapq
from collections import deque
import numpy as np

T = 1000 / 30          # frame period in ms (30 fps camera)
N = 300                # frames exposed in a 10 s run
READOUT = 5.0          # exposure start to frame available on the host, ms
H, W = 48, 64

def render(i, rng):
    """Synthetic frame: noise plus a bright 6x6 square whose position encodes i."""
    img = rng.integers(0, 40, size=(H, W)).astype(np.uint8)
    r, c = 4 + i // 50, 4 + i % 50
    img[r:r + 6, c:c + 6] = 220
    return img

def detect(img):
    """Toy detector: threshold, centroid, decode the frame index from position."""
    rows, cols = np.nonzero(img > 128)
    r, c = rows.mean() - 2.5, cols.mean() - 2.5
    return int(round(r - 4)) * 50 + int(round(c - 4))

def service_times(seed):
    """Per-frame service times (ms) for preprocess, detect and postprocess."""
    rng = np.random.default_rng(seed)
    pre = 3 + rng.uniform(0, 2, N)
    det = 22 + rng.exponential(9, N)
    post = 2 + rng.uniform(0, 2, N)
    return np.stack([pre, det, post], axis=1)

def simulate(stages, policy, svc, seed):
    """stages: list of tuples of service columns run by one worker each.
    policy: 'fifo' (unbounded), 'ring4' (4 slots, drop newest), 'latest'."""
    img_rng = np.random.default_rng(seed + 1000)
    events, n = [], 0
    def push(t, kind, s, f):
        nonlocal n
        heapq.heappush(events, (t, n, kind, s, f)); n += 1
    for i in range(N):
        push(i * T + READOUT, "arrive", 0, i)
    queues = [deque() for _ in stages]
    busy = [False] * len(stages)
    dropped, max_q, lat, correct = 0, 0, [], 0
    def start(t, s, f):
        busy[s] = True
        push(t + svc[f, list(stages[s])].sum(), "done", s, f)
    while events:
        t, _, kind, s, f = heapq.heappop(events)
        if t >= N * T:                       # stop at the end of the capture window
            break
        if kind == "arrive":
            if s == 0:
                frames[f] = render(f, img_rng)
            q = queues[s]
            if not busy[s]:
                start(t, s, f)
                continue
            if policy == "latest" and q:
                q.popleft(); dropped += 1
            if policy == "ring4" and len(q) >= 4:
                dropped += 1; continue
            q.append(f)
            max_q = max(max_q, len(q))
        else:
            busy[s] = False
            if s + 1 < len(stages):
                push(t, "arrive", s + 1, f)
            else:                            # decision: run the detector on the pixels
                k = detect(frames.pop(f))
                correct += (k == f)
                lat.append(t - k * T)        # age of the frame the decision is based on
            if queues[s]:
                start(t, s, queues[s].popleft())
    backlog = sum(len(q) for q in queues) + sum(busy)
    return len(lat), dropped, backlog, max_q, np.array(lat), correct

frames = {}
SEQ = [(0, 1, 2)]               # one worker runs all three stages
PIPE = [(0,), (1,), (2,)]       # one worker per stage
configs = [("sequential", SEQ, "fifo"), ("sequential", SEQ, "ring4"),
           ("sequential", SEQ, "latest"), ("pipelined", PIPE, "fifo"),
           ("pipelined", PIPE, "latest")]

svc = service_times(seed=0)
print(f"budget {T:.1f} ms; mean service ms: pre {svc[:, 0].mean():.1f}, "
      f"detect {svc[:, 1].mean():.1f}, post {svc[:, 2].mean():.1f}, "
      f"sum {svc.sum(1).mean():.1f}; detect > budget in "
      f"{(svc[:, 1] > T).mean():.0%} of frames")
print(f"{'execution':10s} {'queue':6s} {'fps':>5s} {'drop':>4s} {'backlog':>7s} "
      f"{'maxq':>4s} {'p50':>6s} {'p90':>6s} {'p99':>6s} {'max':>6s} {'mean':>6s}  correct")
for name, stages, pol in configs:
    done, drop, back, mq, lat, ok = simulate(stages, pol, svc, seed=0)
    p50, p90, p99 = np.percentile(lat, [50, 90, 99])
    print(f"{name:10s} {pol:6s} {done / (N * T / 1000):5.1f} {drop:4d} {back:7d} "
          f"{mq:4d} {p50:6.1f} {p90:6.1f} {p99:6.1f} {lat.max():6.1f} {lat.mean():6.1f}"
          f"  {ok}/{done}")

print("p99 latency (ms) over service-time seeds 0-9: min-max")
for name, stages, pol in configs:
    p99s = [np.percentile(simulate(stages, pol, service_times(s), s)[4], 99)
            for s in range(10)]
    print(f"  {name:10s} {pol:6s} {min(p99s):7.1f} - {max(p99s):7.1f}")

Run with NumPy 2.3.5, it printed the same output in two runs:

budget 33.3 ms; mean service ms: pre 4.1, detect 31.0, post 3.0, sum 38.1; detect > budget in 28% of frames
execution  queue    fps drop backlog maxq    p50    p90    p99    max   mean  correct
sequential fifo    26.4    0      36   36  766.5 1154.7 1231.6 1233.1  698.4  264/264
sequential ring4   26.0   36       4    4  175.7  203.4  232.2  237.8  173.3  260/260
sequential latest  26.2   37       1    1   59.7   73.9   96.8  107.0   59.7  262/262
pipelined  fifo    29.9    0       1    3   49.6   85.5  112.6  126.7   55.1  299/299
pipelined  latest  29.4    5       1    1   45.7   61.4   79.7  106.5   47.8  294/294
p99 latency (ms) over service-time seeds 0-9: min-max
  sequential fifo     979.9 -  1325.4
  sequential ring4    212.3 -   237.9
  sequential latest    87.6 -   106.7
  pipelined  fifo      80.8 -   132.8
  pipelined  latest    75.7 -    88.9

The three sequential rows have the same throughput, about $1000/38.1 = 26$ decisions per second, because one worker needs 38.1 ms per frame against a 33.3 ms budget ($\rho \approx 1.14$). They differ only in which frames they process. The unbounded FIFO processes every frame in order and falls behind by the rate difference, about $30 - 26.2 \approx 3.8$ frames per second, so after 10 s 36 frames are waiting (the rate difference predicts 38) and its latest decisions are more than a second old. The ring buffer stays bounded but always full, so each processed frame has waited behind four others and decisions are about 0.18 s old: the lag of a slow loop that reads from a driver queue. Keep-latest drops about as many frames, but the frames it drops are the stale ones, and its median latency is close to the minimum possible, readout plus service time plus the wait for the next frame.

Pipelining changes throughput, not the cost of a frame. With one worker per stage, the bottleneck is the detector at $\rho = 31.0/33.3 \approx 0.93$, so the FIFO pipeline keeps up, processing 299 of 300 frames with none dropped, while its median latency of about 50 ms is still above the 43 ms of readout plus the mean stage times. The gap between its mean (55 ms) and its 99th percentile (113 ms) is the queueing described in the formal definition: in 28% of frames the detector alone exceeds the budget, and the frames behind a slow one wait. Keep-latest in front of each stage trades 5 dropped frames for a lower and steadier tail: over ten seeds its 99th percentile ranges from 76 to 89 ms against 81 to 133 ms for the FIFO pipeline. Little’s law ties the columns together: the FIFO pipeline delivers 29.9 decisions per second at a mean of 55.1 ms, so about $29.9 \times 0.0551 \approx 1.6$ frames are in flight on average. The correct column shows that every decision decoded the index of the frame it was computed from, so the latencies are those of the frames actually used.

The model is deliberately simple. It ignores the cost of handing frames between workers, contention between workers for cores or a GPU, and the global interpreter lock, all of which a real implementation adds; and it has no forecasting, which, with scheduling and box association, recovered much of the accuracy that Li et al. (2020) lost to latency.

Common Misconceptions

• “A model that runs at 30 frames per second makes a real-time system.” The model’s runtime is one term of the latency. Capture, transfer, preprocessing, queueing, postprocessing and display add to it, and a pipeline can deliver decisions at close to the frame rate while each is a second old, as the sequential FIFO above shows.
• “Pipelining reduces latency.” It raises throughput by overlapping frames. A frame’s own latency is at least the sum of its stages, plus any queueing between them.
• “Never drop a frame.” For live video, a frame that has waited behind others is often worth less than the next one. Keeping every frame trades current decisions for complete but stale ones; whether that is right depends on the application, such as recording, where completeness matters, against control, where freshness does.
• “Average latency is enough.” Deadlines are missed by the slow frames. Report percentiles and the maximum, together with throughput and dropped frames.
• “The time a frame arrives is the time it was captured.” Arrival time includes readout, transfer and buffering delays that vary per frame. Use capture timestamps from the camera or driver, on a monotonic clock, and know which instant they mark.
• “Frames with the same index from two cameras were taken at the same time.” Free-running cameras are not locked to each other’s frame timing and can drop frames independently; synchronize them in hardware or match them by timestamps in one clock domain.

Where It Is Used

Every vision system that acts on live video is a real-time pipeline. Object tracking and tracking-by-detection depend on correct timestamps, and a Kalman filter can predict a track forward to the moment a decision is used, which is how Li et al. (2020) compensated for detector latency. Visual–inertial variants of visual odometry and SLAM fuse images with inertial measurements by their timestamps, and rolling-shutter-aware variants assign times per row. Stereo rigs and camera–LiDAR systems need synchronized capture. In industrial visual inspection, frames are triggered by the production line, and a reject decision is useful only if it arrives before the part reaches the reject mechanism. Real-time detectors such as YOLO are designed and reported for speed in frames per second; once deployed, the rest of the pipeline decides the latency. Model deployment, edge inference and the design of a complete vision system build on these timing concepts, and tools such as OpenCV and GStreamer supply the capture, queueing and timestamp mechanisms described above.