Vision Systems & Engineering

Video Analytics Pipelines

How systems that analyse many live camera streams share ingest, decoding and compute among them, using per-stream knobs and resource–quality profiles, frame sampling and filtering, scheduling against quality and lag goals, and indexes for later queries; with an example that scores frame-filtering policies on sixteen synthetic streams by events found, detector calls and delay.

intermediate

A video analytics pipeline is a system that runs vision queries, such as counting cars, reading licence plates or finding a kind of event, over many live camera streams at once, on a shared cluster or a shared edge server. Each stream on its own is a real-time pipeline with a frame budget, queues and timestamps. What changes with many streams is that ingest, decoding and the detectors compete for the same machines, and the expensive models cannot be run on every frame of every stream. Such systems therefore decide, per stream and per query, how much of the video to look at and at what quality, and they schedule compute between queries whose quality and delay requirements differ. Zhang et al. (2017) describe one such system, VideoStorm, which processes thousands of queries on live streams and which this article follows for its terms: queries, knobs, configurations, resource–quality profiles and lag.

This article covers the many-stream layer. One pipeline’s timing belongs to Real-Time Vision Pipelines, batching on a server to latency and throughput and model deployment, splitting work between a device and a server to edge inference, and the chain of analytic tasks on security and traffic video to video surveillance.

Purpose

Two facts set the problem. Vision models are costly: Zhang et al. note that the best tracker of the VOT 2015 challenge processed about one frame per second on an 8-core machine, and that some of the most accurate recognition networks need 30 GFLOPs per frame (§1). And streams are many and continuous: the paper’s motivating deployments are city camera networks analysed on large clusters. Running the most accurate model on every frame of every stream is therefore rarely affordable, and a system has to trade quality for resources somewhere.

Zhang et al. identify two properties that make the trade manageable (§1). First, queries have knobs, parameters such as resolution, frame rate and internal algorithm settings, and a combination of knob values, a configuration, can lower resource demand at some loss of quality. Second, queries differ in their quality and lag goals: a car counter for traffic lights can accept approximate counts but needs them quickly, while a toll-road plate reader needs accurate reads but can wait minutes, since billing can be delayed. A video analytics pipeline uses both: it picks configurations per query and lets delay-tolerant queries fall behind during shortages so that urgent ones keep up.

When to Use It

The techniques here apply when several streams or queries share fixed compute, and when the analysis is about events or aggregates over time rather than every frame. Recording and archive search, traffic counting, retail and site monitoring, and the surveillance tasks described in Video Surveillance fit this pattern. A single camera driving a robot or a brake does not: there, every frame’s latency matters and the techniques of Real-Time Vision Pipelines apply. Approximation also has a price that must be measured per stream: Zhang et al. stress that there are no analytical models relating a video query’s resource demand to its quality, that the relation depends on the specific feed (lowering the resolution may not hurt a plate reader if the camera is zoomed in enough), and that queries must therefore be profiled on representative video (§3.2).

How It Works

Ingest and decoding

IP cameras can deliver compressed video over the network through the Real-Time Streaming Protocol (RTSP; RFC 7826, not read) with the media carried over RTP (RFC 3550, not read). FFmpeg’s documentation describes its RTSP support as a demuxer and muxer rather than a protocol handler; the demuxer handles RTSP with data over RTP, and its rtsp_transport option selects UDP, TCP interleaved in the control channel, UDP multicast, or HTTP or HTTPS tunnelling as the lower transport. Over UDP the demuxer buffers packets to reorder them, a delay that can be disabled (FFmpeg n9.0.2, protocols.texi).

Every analysed frame must be decoded first, and decoding is a per-stream cost that no later filtering removes: a filter that compares frames needs the frames. In our reading this makes decoding a fixed cost of each stream, paid whether one frame in ten or every frame reaches the detector. Decoders can skip work: FFmpeg’s skip_frame option discards frames by type, for example all non-reference frames (noref), all bidirectional frames (bidir), or all frames except keyframes (nokey, documented as “Discard all frames excepts keyframes”) (FFmpeg n9.0.2, codecs.texi). Decoding only keyframes cuts decoding work, but it also fixes the sampling to the encoder’s keyframe interval, which the analytics system may not control (our reasoning). FFmpeg covers decoding mechanics, GOPs and keyframes in detail.

Decoded frames are large. VideoStorm implemented flow control between the transforms of a query so that unprocessed video queues at the front of the query, before decoding, rather than inside it: in the authors’ benchmarks on HD video, decoded frames were up to 300 times larger than the encoded video, and buffering at the front also lets a query respond promptly to configuration changes, since no frames sit in the pipeline with old knob values (§7.1).

Queries, knobs and resource–quality profiles

In VideoStorm a query is a directed acyclic graph of transforms: a decoder transform connects to a camera and produces frames, and further transforms, such as background subtraction, object tracking, classification or a licence-plate reader, process the time-ordered stream and pass results downstream; a query may span several machines (§2). Transforms expose knobs, and those whose names start with @ can be changed while the query runs; the paper’s example decoder has an output resolution and a sampling rate, and its plate reader has minimum and maximum window sizes and a step (Figure 2). The sampling-rate convention matters when reading numbers: in the paper, a sampling rate of 0.75 drops every fourth frame (footnote 1), so the rate is the fraction of frames kept.

The resource–quality profile of a query lists its configurations with their resource demand, measured as the resources needed to keep up with the live stream, and their quality, measured as the F1 score against labelled ground truth or against an expensive “golden” configuration run on the start of the video (§3.2, §5). The space is large: the paper’s plate-reader query has 414 configurations of five knobs that span four orders of magnitude in resource demand, and profiling them all on a 10-minute video took 20 CPU days (Figure 5). Configurations with similar cost can differ widely in quality and the reverse, so only the Pareto boundary is useful: a configuration that is no better in quality and no cheaper than another is dropped (§5.2). VideoStorm’s profiler finds that boundary by greedy hill climbing on X(c)=Q(c)−βD(c)X(c) = Q(c) - \beta D(c) from the best of a few random starting configurations, caching intermediate results shared between configurations, and reports producing the same profile as a baseline search with 3.5 times less CPU, 5.4 instead of 19 CPU hours (§5.2).

Frame sampling, filtering and cascades

The simplest knob is the share of frames analysed. Fixed sampling, keeping every PP-th frame, divides the detector’s work by PP, and its effect on events follows from counting (our derivation). An event visible for dd consecutive frames contains a kept frame for min⁡(d,P)\min(d, P) of the PP possible sampling phases, so with a phase that is random relative to the event, the event is missed with probability

pmiss(d,P)=max⁡(0,  1−dP),p_{\text{miss}}(d, P) = \max\Big(0,\; 1 - \frac{d}{P}\Big),

and a detected event waits at most P−1P - 1 frames for its first kept frame. Short events are lost in proportion to how much shorter than the sampling period they are; events longer than the period are never lost, only delayed.

Content-driven filtering analyses frames only when something seems to have changed. A cheap test runs on every decoded frame, such as a difference from the last frame the detector saw, and only frames that pass reach the expensive model. Model Deployment describes the general form as a cascade: a small model filters frames on the device or at the camera, and a larger model on a server handles the frames that pass. The filter’s own errors decide what the system can find: an event the filter does not pass is invisible to the detector behind it, and a filter that responds to scene-wide changes, such as illumination, spends detector calls on frames with nothing new in them (our reasoning, which the example below measures). One guard is to force a detector call after a maximum gap, which bounds how long any long-lasting event can go unseen.

Frame differencing here is a gate, not a segmentation: Background Subtraction covers differencing, running averages and mixture models as methods for finding foreground pixels, where they are judged by pixel accuracy.

Scheduling against quality and lag goals

With profiles in hand, a scheduler allocates resources to queries and picks each query’s configuration. Zhang et al. define lag in §3.1 as the difference between the time of the last-arrived frame and the time of the last-processed frame, that is, how much video is queued unprocessed, and in §6.1 as the difference between the current time and the arrival time of the last processed frame; the two wordings differ slightly. Lag differs from per-frame latency: a query that is allocated less than its configuration’s demand keeps producing results at full quality, but later and later.

VideoStorm encodes each query’s goals in a utility function (§6.1, Eq. 1):

U(Q,L)=UB+αQ(Q−QM)+−αL(L−LM)+,U(Q, L) = U^B + \alpha^Q (Q - Q^M)_+ - \alpha^L (L - L^M)_+ ,

where QMQ^M is the minimum quality goal, LML^M the maximum lag goal, (x)+(x)_+ the positive part, and there is no bonus for lag below the goal. The cluster maximizes either the minimum utility (max-min fairness over utilities) or the sum of utilities. Because quality changes instantly with the configuration while lag accumulates, the scheduler uses model-predictive control: it predicts each query’s lag over a short horizon TT from its allocation aka_k and its configuration’s demand Dk(ck)D_k(c_k) (§6.2.1, Eq. 2),

Lk,t+T=Lk,t+T−T akDk(ck),L_{k,t+T} = L_{k,t} + T - T\,\frac{a_k}{D_k(c_k)} ,

and plugs the prediction into the utility. Allocation is greedy in small increments, for example 1% of a core: to maximize the sum, each step gives the increment to the query whose utility rises most; to maximize the minimum, it goes to the query with the lowest utility (§6.2.2). The paper’s two-query example shows the effect (§3.1, Table 1): when a machine’s capacity halves for 12 seconds, a fair scheduler drops both queries to cheap configurations, while the performance-based scheduler keeps the lag-tolerant query at its best configuration, lets it build 8 seconds of lag, within its goal, and then switches it to a cheaper configuration with spare resources so that it processes 1.5 seconds of video per second and catches up.

Placement then assigns queries to machines by the average of three scores, for packing, load balance and spreading low-lag queries so that every machine has some lag-tolerant work to delay, and migrates a query only if its score improves by more than a threshold (§6.3). Because profiles can be wrong when demand depends on the content, VideoStorm tracks the ratio of actual to profiled demand and corrects its lag prediction with it (§6.4).

Indexing for later queries

Not every question is asked while the video is live. A pipeline that keeps compact per-frame or per-segment results, such as detected classes, counts, embeddings and timestamps, lets later queries search those results instead of re-running detectors over the archive; this is our summary of the design, and Video Surveillance describes the search tasks themselves, from re-identification to text queries. Image Retrieval covers indexing feature vectors for large galleries. In our reading, the same trade as for live queries applies at ingest time: a cheaper indexing model makes the index cheaper to build but lets more frames through to expensive checks at query time.

How to Apply It

In practice the steps follow the structure above (our synthesis): measure each stream’s decode cost, which filtering cannot reduce; profile each query’s knobs on representative video from its own camera; choose a sampling or filtering policy per stream and measure what it misses on events with known times; give each query explicit quality and lag goals; and log, per stream, the detector invocations, the lag and the frames dropped.

Example: frame filtering across sixteen streams

The script below compares frame-selection policies on sixteen synthetic camera streams with exact ground truth, scored by the events they find rather than by pixels or latency. Each stream is 5 minutes of 64×6464 \times 64 greyscale video at 10 frames per second (3,000 frames): a smooth textured background between grey levels 70 and 130 and Gaussian sensor noise of standard deviation 2, rounded to 8 bits. Streams differ in activity, with 3, 10, 30 or 60 events each, four streams per level. Half the streams have a slow global illumination drift, a gain 1+0.1sin⁡(2πt/3000+φ)1 + 0.1 \sin(2\pi t / 3000 + \varphi) applied to the whole frame, squares included; applying it to the squares as well is our modelling choice. Each event is a bright square that appears inside the frame at a random time, lasts 1 to 80 frames and moves at up to half a pixel per frame along each axis. Squares have sides of 3, 5 or 8 pixels, a design choice that gives onsets of different strength, so that a single difference threshold can be high enough to pass some appearances and not others.

The “expensive” detector is a toy that reads the squares from the pixels: it thresholds at grey level 200 and returns the bounding boxes of connected components. An event counts as detected at a processed frame when a detected box overlaps its true box. Because the detector is deterministic, it is run once per frame and each policy looks up the frames it would have sent; the measure is the number of those invocations. The policies are:

  • every frame;
  • fixed sampling of every 2nd, 5th or 10th frame (VideoStorm’s sampling-rate knob at 0.5, 0.2 and 0.1), starting at frame 0;
  • a difference trigger of our own design, which sends a frame to the detector when its mean absolute difference (MAD) from the last processed frame exceeds a threshold τ\tau, at three values of τ\tau;
  • the same trigger with a forced refresh, which also sends a frame when k=10k = 10 frames have passed since the last invocation.

No read source fixes τ\tau, so a planning script set the three values from this data before the pilot, by a rule fixed before it ran: with fmf_m the largest MAD between consecutive event-free frames of the drift-free streams, and o3o_3, o5o_5 the MADs between a noisy frame with one square of side 3 or 5 and an independent noisy frame without it, τlow=fm+0.25 (min⁡o3−fm)\tau_{\text{low}} = f_m + 0.25\,(\min o_3 - f_m), τmid=fm+0.75 (min⁡o3−fm)\tau_{\text{mid}} = f_m + 0.75\,(\min o_3 - f_m) and τhigh=(max⁡o3+min⁡o5)/2\tau_{\text{high}} = (\max o_3 + \min o_5)/2. The values 2.377, 2.412 and 2.774 were written into the plan after that script printed the noise floor and onset MADs.

A false trigger is a trigger invocation on a frame with no square whose last processed frame had no square either. The table also gives, for the events lasting at least k=10k = 10 frames, how many were missed and the longest delay. Delays are counted in frames from the event’s first frame to the first processed frame in which it is detected.

import os
for v in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS", "VECLIB_MAXIMUM_THREADS"):
    os.environ[v] = "1"
import numpy as np
from scipy import ndimage

H = W = 64; FPS = 10; N = 3000; SIGMA = 2.0       # 5-minute streams of 64x64 frames at 10 fps
SIZES = (3, 5, 8); DURS = (1, 2, 3, 4, 6, 8, 12, 20, 40, 80)
LEVELS = (3, 10, 30, 60)                            # events per stream (activity level)
TAUS = (2.377, 2.412, 2.774)                        # set from a planning script (see text)
K = 10                                              # forced refresh: at most K frames between invocations

def make_stream(sid):
    level, drift = LEVELS[sid // 4], (sid // 2) % 2 == 1
    rng = np.random.default_rng(1000 + sid)
    bg = ndimage.gaussian_filter(rng.normal(size=(H, W)), 4)
    bg = 70 + 60 * (bg - bg.min()) / (bg.max() - bg.min())          # smooth texture in [70, 130]
    phi = rng.uniform(0, 2 * np.pi)
    gain = 1 + 0.1 * np.sin(2 * np.pi * np.arange(N) / N + phi) if drift else np.ones(N)
    events = []                                                      # (onset, duration, side, start, velocity)
    for _ in range(level):
        s = int(rng.choice(SIZES)); d = int(rng.choice(DURS))
        t0 = int(rng.integers(0, N - d + 1))
        v = rng.uniform(-0.5, 0.5, 2)
        lo = [max(0.0, -v[a] * (d - 1)) for a in range(2)]
        hi = [min(W - s, W - s - v[a] * (d - 1)) for a in range(2)]
        events.append((t0, d, s, [rng.uniform(lo[a], hi[a]) for a in range(2)], v))
    return dict(level=level, drift=drift, bg=bg, gain=gain, events=events)

def boxes_at(st, t):
    out = []
    for e, (t0, d, s, p0, v) in enumerate(st["events"]):
        if t0 <= t < t0 + d:
            out.append((e, int(round(p0[0] + v[0] * (t - t0))), int(round(p0[1] + v[1] * (t - t0))), s))
    return out

def render(st, t, noise_rng):
    img = st["bg"].copy()
    for _, r, c, s in boxes_at(st, t):
        img[r:r + s, c:c + s] = 240.0
    img = img * st["gain"][t] + noise_rng.normal(0, SIGMA, (H, W))
    return np.clip(np.round(img), 0, 255).astype(np.uint8)

def detect(img):
    """Toy 'expensive' detector: bright pixels, connected components, bounding boxes."""
    lab, _ = ndimage.label(img > 200)
    return [(a.start, a.stop, b.start, b.stop) for a, b in ndimage.find_objects(lab)]

def overlaps(box, r, c, s):
    r0, r1, c0, c1 = box
    return r0 < r + s and r < r1 and c0 < c + s and c < c1

# ---- generate streams, run the detector once per frame (it is deterministic), check the data ----
streams, frames, found, empty = [], [], [], []
miss = false_box = 0; inside = True; bgmax = 0; sqmin = 255
for sid in range(16):
    st = make_stream(sid); nrng = np.random.default_rng(5000 + sid)
    F = np.empty((N, H, W), np.uint8); fd = []; em = np.zeros(N, bool)
    for t in range(N):
        F[t] = render(st, t, nrng); bx = boxes_at(st, t); det = detect(F[t])
        mask = np.zeros((H, W), bool)
        for _, r, c, s in bx:
            inside &= 0 <= r and r + s <= H and 0 <= c and c + s <= W
            mask[r:r + s, c:c + s] = True
        if mask.any(): sqmin = min(sqmin, int(F[t][mask].min()))
        bgmax = max(bgmax, int(F[t][~mask].max()))
        miss += sum(not any(overlaps(b, r, c, s) for b in det) for _, r, c, s in bx)
        false_box += sum(not any(overlaps(b, r, c, s) for _, r, c, s in bx) for b in det)
        fd.append({e for e, r, c, s in bx if any(overlaps(b, r, c, s) for b in det)})
        em[t] = not bx
    streams.append(st); frames.append(F); found.append(fd); empty.append(em)

nd = [st["gain"] for st in streams if not st["drift"]]; dr = [st["gain"] for st in streams if st["drift"]]
print("data: events per stream", [len(st["events"]) for st in streams])
print(f"data: boxes inside {inside}; background max {bgmax} (<200); square min {sqmin} (>=200); "
      f"gain no-drift {min(g.min() for g in nd):.4f}-{max(g.max() for g in nd):.4f}, "
      f"drift {min(g.min() for g in dr):.4f}-{max(g.max() for g in dr):.4f}")
print(f"C1 detector on all {16 * N} frames: missed squares {miss}, false boxes {false_box}")

def run(sid, kind, P=None, tau=None, refresh=False, phase=0):
    """Return processed frame indices, and the number of false triggers."""
    F, em = frames[sid], empty[sid]
    if kind == "every": return np.arange(N), 0
    if kind == "sample": return np.arange(phase, N, P), 0
    proc, last, falsetr = [0], 0, 0
    for t in range(1, N):
        trig = np.abs(F[t].astype(np.int16) - F[last].astype(np.int16)).mean() > tau
        if trig or (refresh and t - last >= K):
            if trig and em[t] and em[last]: falsetr += 1
            proc.append(t); last = t
    return np.array(proc), falsetr

def score(sid, proc):
    """Per event: delay in frames to the first processed frame where it is detected, or None."""
    fd, out = found[sid], []
    for e, (t0, d, s, p0, v) in enumerate(streams[sid]["events"]):
        hit = [t for t in proc[(proc >= t0) & (proc < t0 + d)] if e in fd[t]]
        out.append((d, hit[0] - t0 if hit else None))
    return out

policies = [("every frame", dict(kind="every"))]
policies += [(f"sample 1/{P}", dict(kind="sample", P=P)) for P in (2, 5, 10)]
policies += [(f"trigger {tau}", dict(kind="trig", tau=tau)) for tau in TAUS]
policies += [(f"trig+refresh {tau}", dict(kind="trig", tau=tau, refresh=True)) for tau in TAUS]

print("\npolicy               inv/stream-s by level (3,10,30,60 events)   all   recall   "
      "delay med/max  false trig nodrift/drift  d>=10: missed, max delay")
res = {}
for name, kw in policies:
    inv = np.zeros(16); ev = []; ft = [0, 0]
    for sid in range(16):
        proc, f = run(sid, **kw)
        inv[sid] = proc.size / (N / FPS); ev += score(sid, proc); ft[int(streams[sid]["drift"])] += f
    lv = [inv[4 * i:4 * i + 4].mean() for i in range(4)]
    dl = [x for _, x in ev if x is not None]
    long_ = [x for d, x in ev if d >= K]
    lm = sum(x is None for x in long_); lmax = max(x for x in long_ if x is not None)
    res[name] = dict(lv=lv, det=len(dl), ft=ft, lm=lm, lmax=lmax)
    print(f"{name:20s} {lv[0]:7.4f} {lv[1]:7.4f} {lv[2]:7.4f} {lv[3]:7.4f} {inv.mean():8.4f} "
          f"{len(dl):4d}/{len(ev)} {int(np.median(dl)):5d}/{max(dl):<4d} {ft[0]:10d}/{ft[1]:<10d} "
          f"{lm:6d}, {lmax}")

ds = np.array([e[1] for st in streams for e in st["events"]])
print("\nH1 fixed sampling: misses at phase 0 against the closed form, and summed over all phases")
for P in (2, 5, 10):
    p = np.clip(1 - ds / P, 0, None); exp, sd = p.sum(), np.sqrt((p * (1 - p)).sum())
    m0 = len(ds) - res[f"sample 1/{P}"]["det"]
    msum = sum(sum(x is None for _, x in score(sid, run(sid, "sample", P=P, phase=ph)[0]))
               for ph in range(P) for sid in range(16))
    closed = int(np.clip(P - ds, 0, None).sum())
    print(f"  P={P:2d}: phase 0 missed {m0:3d}, expected {exp:7.3f} +- 3 x {sd:.3f} "
          f"[{'held' if abs(m0 - exp) <= 3 * sd else 'failed'}]; all phases missed {msum:4d}, "
          f"closed form {closed:4d} [{'held' if msum == closed else 'failed'}]")

Run with NumPy 2.3.5 and SciPy 1.16.3, it printed the same output in two runs, in about 4 seconds:

data: events per stream [3, 3, 3, 3, 10, 10, 10, 10, 30, 30, 30, 30, 60, 60, 60, 60]
data: boxes inside True; background max 151 (<200); square min 207 (>=200); gain no-drift 1.0000-1.0000, drift 0.9000-1.1000
C1 detector on all 48000 frames: missed squares 0, false boxes 0

policy               inv/stream-s by level (3,10,30,60 events)   all   recall   delay med/max  false trig nodrift/drift  d>=10: missed, max delay
every frame          10.0000 10.0000 10.0000 10.0000  10.0000  412/412     0/0             0/0               0, 0
sample 1/2            5.0000  5.0000  5.0000  5.0000   5.0000  400/412     0/1             0/0               0, 1
sample 1/5            2.0000  2.0000  2.0000  2.0000   2.0000  334/412     2/4             0/0               0, 4
sample 1/10           1.0000  1.0000  1.0000  1.0000   1.0000  263/412     3/9             0/0               0, 9
trigger 2.377         0.1850  0.4283  0.9558  1.6275   0.7992  412/412     0/2             0/282             0, 0
trigger 2.412         0.1692  0.4125  0.9417  1.6008   0.7810  412/412     0/2             0/224             0, 0
trigger 2.774         0.0858  0.2242  0.5358  0.8658   0.4279  345/412     0/74            0/88             22, 74
trig+refresh 2.377    1.0758  1.2733  1.6883  2.2492   1.5717  412/412     0/2             0/0               0, 0
trig+refresh 2.412    1.0758  1.2708  1.6833  2.2325   1.5656  412/412     0/2             0/0               0, 0
trig+refresh 2.774    1.0383  1.1392  1.3667  1.5775   1.2804  375/412     0/9             0/0               0, 9

H1 fixed sampling: misses at phase 0 against the closed form, and summed over all phases
  P= 2: phase 0 missed  12, expected  17.000 +- 3 x 2.915 [held]; all phases missed   34, closed form   34 [held]
  P= 5: phase 0 missed  78, expected  75.800 +- 3 x 5.643 [held]; all phases missed  379, closed form  379 [held]
  P=10: phase 0 missed 149, expected 139.100 +- 3 x 6.569 [held]; all phases missed 1391, closed form 1391 [held]

The data checks and the detector check held: every stream has its planned number of events, the background never reaches the detector’s threshold and the squares never fall below it, the gain is exactly 1 on the drift-free streams and stays within 0.9 to 1.1 on the drift streams, and the detector misses no square and reports no false box on any of the 48,000 frames. Every-frame processing therefore found all 412 events at delay 0, so the planned check on that row held too, which makes it the reference for the other rows.

Fixed sampling divides invocations by PP exactly, by construction. Summed over all PP phases, its misses equal the count ∑emax⁡(0,P−de)\sum_e \max(0, P - d_e) for every PP; that is an identity of the counting argument above, and its agreement checks the sampling and scoring arithmetic rather than testing anything. The hypothesis was that the phase-0 run misses as many events as the closed form expects within three standard deviations of a binomial count, and it held for all three periods. The maximum delays of 1, 4 and 9 frames are the P−1P - 1 bound of the construction.

The trigger behaves differently. Its invocations grow with activity, since each appearance, disappearance and few pixels of motion changes the frame (our reading of the construction), so it saves most on quiet streams: the hypothesis that the 3-event streams make the fewest detector calls held at all three thresholds. The hypothesis that the trigger’s recall equals every-frame recall was stated for τlow\tau_{\text{low}} and τmid\tau_{\text{mid}} only, and it held at both, with all 412 events found and a maximum delay of 2 frames. At τhigh=2.774\tau_{\text{high}} = 2.774 the trigger found 345 of 412 events, so the expectation written before the pilot, that recall there would fall below every-frame recall, held; it missed 22 of the events lasting 10 frames or more, and its longest delay was 74 frames.

Illumination drift produced every false trigger in the trigger-alone rows: 282, 224 and 88 on the drift streams against none on the drift-free streams, so the hypothesis that drift raises false triggers held at all three thresholds. In our reading, the reference frame of a trigger that has not fired for a while is old, and a slow gain change accumulates against it until the difference crosses τ\tau; on the drift-free streams, where only noise changes between empty frames, no printed trigger was false.

The forced refresh bounds the delay. With it, no event lasting at least 10 frames was missed at any threshold, and the longest delay among them was 0, 0 and 9 frames, within the k−1=9k - 1 = 9 bound, so that hypothesis held at all three thresholds. The refresh also sets a floor of about one invocation per stream-second, by construction, and raises the quiet streams’ cost from 0.1850 to 1.0758 invocations per stream-second at τlow\tau_{\text{low}}. With the refresh, the printed false-trigger counts are 0 on both kinds of stream at all three thresholds.

The pilot ran this script with the same seeds before the first run, and the two outputs are identical. Besides the three thresholds, the numerical bounds of the phase-0 check and the closed-form counts of the all-phases check were first written into the plan after the planning script printed them; the three-standard-deviation rule itself was fixed before any script ran, and the code computes the bounds it prints.

The example tests our trigger and our toy detector on synthetic video. It says nothing about any product’s filter, and the counts would change with real decoders, compression noise, camera shake and detectors that make mistakes.

Variants

  • Fixed per-stream configurations, set once from a profile, against dynamic ones that a scheduler changes as load changes, as in VideoStorm.
  • Sampling, resolution and model choice as knobs: Zhang et al. show quality against CPU demand for resolution and sampling rate of a plate reader, sampling rate of a DNN classifier and the matching metric of a tracker (Figure 3), and name a model’s compression factor as another knob (§3.2).
  • Content-driven filters of increasing cost: frame differences, small classifiers and cascades of models, as in Model Deployment’s cascades and hybrids.
  • Where filtering runs: at the camera, on an edge server or in the cluster, which decides what crosses the network (Edge Inference covers offloading and split computation).
  • Frameworks: pipelines can be assembled from multimedia-framework elements, whose GStreamer mechanisms Real-Time Vision Pipelines describes. NVIDIA DeepStream is a vendor framework named in this context; its documentation was not read for this article.
  • Live against retrospective queries, the latter served from indexes built at ingest time.

Best Practices

  • Profile on video from the camera that will run the query; the relation between knobs and quality depends on the feed (Zhang et al., §3.2).
  • Keep only Pareto-optimal configurations; the rest are never worth choosing.
  • Give each query explicit quality and lag goals, so that a scheduler can delay the tolerant ones instead of degrading all equally.
  • Queue compressed video before the decoder rather than decoded frames inside the pipeline, as VideoStorm does (§7.1).
  • Score a filtering policy on events with known start and end times: recall, delay and invocations, separately for quiet and busy streams and with and without lighting changes.
  • Bound the time between detector calls when long events must not be missed.
  • State the sampling convention: a rate of 0.75 keeps three frames in four in VideoStorm’s usage.

Pitfalls

  • Counting only the detector. Decoding, transfer and the filter itself run on every frame; a filter that sends few frames to the detector does not make a stream free.
  • Sampling below the event length. Events shorter than the sampling period are missed with probability 1−d/P1 - d/P; the loss is invisible in per-frame accuracy.
  • Thresholds set without scene changes. In the example, illumination drift caused every false trigger, and the highest threshold, which made the fewest of them, also found the fewest events; a threshold has to be checked on both.
  • Profiles that go stale. Demand depends on content, such as traffic density; VideoStorm corrects its predictions from measured usage (§6.4).
  • Comparing results across set-ups. Quality and lag improvements are specific to the queries, videos, goals and cluster of the study that measured them.
  • Fair sharing by default. Equal allocation ignores how much each query’s quality gains from more resources and how much lag it tolerates, which is the inefficiency VideoStorm’s evaluation measures.

Evidence and Impact

Zhang et al. (2017) evaluated VideoStorm on 101 Azure D3 v2 machines (4 cores and 14 GB each, one machine running the manager), with four query types (plate reader, car counter, DNN classifier, object tracker) on recorded traffic-camera video from Bellevue and Seattle, streamed at its original 14 to 30 frames per second and 240p to 1080p resolution (§8.1). The workload started with 300 queries on 300 videos, 60% with a 20-second lag goal and the rest 300 seconds, all with quality goal 0.25, and then received a burst of 200 plate-reader queries with quality goal 1.0 and a 20-second lag goal lasting NN seconds. Against a work-conservative fair scheduler (the paper’s term), maximizing the minimum utility kept quality at about 0.9 while the fair scheduler’s fell to nearly 0.5 for longer bursts, which the authors call nearly 80% better, with fewer than 5% of frames exceeding their lag goal against seven times as many under fair scheduling (§8.2.1, Figure 7). Maximizing the sum of utilities gave 25% better quality and five times better lag (§8.2.2). With VideoStorm’s allocation but round-robin placement, performance dropped by a third according to the caption of Figure 7. With profiles made wrong on purpose, half or twice the true demand for a third of the queries each, the demand correction gave quality and lag at nearly 99.6% of those with accurate profiles (§8.3.2). Migrating a query between machines took 1.3 seconds in their test (§8.3.1).

The paper states its scheduler’s speed two ways: the introduction says it allocates resources in hundreds of milliseconds even with thousands of queries (§1), while the scalability section says it makes its decisions in a few seconds with thousands of queries (§8.4, Figure 13a).

Related

  • Video Surveillance and Security

    Computer vision applied to video from fixed security and traffic cameras as its research literature describes it; the processing chain from detection to multi-camera tracking, re-identification, attributes, events and search, the PETS, DukeMTMC, CityFlow, AI City Challenge, VIRAT and MEVA benchmarks with their protocols and metrics, documented failure modes, NIST's accuracy and demographic findings, and the privacy concerns stated in official documents.

  • Deploying Vision Models

    Getting a trained vision model into use, from export and the choice of runtime to packaging the model with its preprocessing contract, serving it on a server or a device, keeping pre- and post-processing identical to training, and versioning, rolling back and monitoring it.

  • Edge Inference

    Running a vision model on the device that captures the images, such as a phone, camera, robot or microcontroller, instead of in a data centre, covering why it is done, the limits of compute, memory, power and heat, the classes of hardware and runtimes, the roofline model for finding memory-bound layers, split and offloaded computation, and how latency and energy are measured.

  • Background Subtraction

    Separating moving objects from a static camera's scene by comparing each frame with a model of the background, from frame differencing and running averages to per-pixel Gaussian mixtures, kernel density estimates, ViBe and trained networks, with the problems that break them and the CDnet benchmark that measures them.

  • Object Detection

    Finding every object of a set of categories in an image and returning, for each one, a bounding box, a class label and a confidence score.

  • Image Retrieval

    Finding the images in a collection that show the same object, place or kind of thing as a query image; instance and category retrieval, the pipeline of descriptors, indexing and ranking, spatial verification, query expansion and diffusion, deep global descriptors, and the benchmarks (UKBench, Holidays, Oxford and Paris, their revisited versions, Google Landmarks v2) with their official evaluation code.

References

  1. Zhang, H., Ananthanarayanan, G., Bodik, P., Philipose, M., Bahl, P. & Freedman, M. J. (2017). Live Video Analytics at Scale with Approximation and Delay-Tolerance. 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17), 377–392 (read in the USENIX open-access PDF).
  2. FFmpeg developers. FFmpeg Codecs Documentation: skip_frame, doc/codecs.texi (tag n9.0.2, commit 946fcce; read November 2026).
  3. FFmpeg developers. FFmpeg Protocols Documentation: rtsp, doc/protocols.texi (tag n9.0.2, commit 946fcce; read November 2026).
  4. Schulzrinne, H., Rao, A., Lanphier, R., Westerlund, M. & Stiemerling, M. (Ed.) (2016). Real-Time Streaming Protocol Version 2.0. RFC 7826 (not read).
  5. Schulzrinne, H., Casner, S., Frederick, R. & Jacobson, V. (2003). RTP: A Transport Protocol for Real-Time Applications. RFC 3550 (not read).
  6. NVIDIA. DeepStream SDK documentation (not read).