Motion Models for Tracking

A motion model is the part of a tracker that says how a target’s state evolves between observations: where an object seen at one position, moving at some speed, should be one frame later, and how uncertain that prediction is. In a Kalman filter it is the pair of the transition matrix $F$ and the process-noise covariance $Q$; in a particle filter it is the rule used to move particles. Its two halves have different jobs. The deterministic part predicts, and the process noise says how far reality may stray from the prediction, which decides how much the tracker trusts each new measurement. Lag in a turn, a track that coasts off in a straight line during an occlusion, and a gate too tight to accept the right detection are failures of the motion model, not of the filter equations.

This article covers the kinematic models used in object tracking (constant position, velocity and acceleration, the Singer model and turn models) together with their process noise; the parametric image motion models (translation, similarity, affine, homography) that trackers use for patches and for camera motion; the interacting multiple model (IMM) filter in outline; and the consistency checks (NIS and NEES) used to choose and tune a model. The filter equations themselves belong to the Kalman filter article, and are not repeated here.

Definition

A motion model, also called a dynamic model, transition model or plant model, is a stochastic description of how the state $\mathbf{x}$ of a tracked object changes over time. In discrete time it gives the distribution of $\mathbf{x}_k$ given $\mathbf{x}_{k-1}$; for the linear-Gaussian models that dominate practice, this is

$$\mathbf{x}_k = F_k \mathbf{x}_{k-1} + \mathbf{w}_k, \qquad \mathbf{w}_k \sim \mathcal{N}(\mathbf{0}, Q_k).$$

The model fixes three things: what the state contains (position only, or position with velocity, acceleration, size or turn rate), how the state is propagated ($F_k$, or a nonlinear function $f$), and the statistics of the unmodelled part ($Q_k$). The measurement model, which links state to detections, is a separate choice.

Two families appear in vision. Kinematic models describe how a point (an image position, a box centre, a 3D position from lidar) moves; they come from target tracking in radar and navigation. Parametric image motion models describe how a whole image region moves: a patch translates, rotates, scales, shears or undergoes a perspective warp. A tracker can combine them, for example by placing kinematic dynamics on the parameters of an image motion model, which is what box trackers do.

Intuition

Imagine predicting where a pedestrian’s bounding box will be in the next frame. With no knowledge of motion, the best guess is “where it was”, with an uncertainty that grows with time; that is the constant-position model. If the pedestrian has been walking steadily to the right, “where it was plus one step to the right” is better; that is the constant-velocity model, which needs velocity in the state and infers it from successive positions. If the pedestrian is speeding up or curving, a model that also carries acceleration or turn rate predicts better, but only while that acceleration persists. Each higher-order model can follow more kinds of motion, and each pays for it: more unknowns to estimate from the same noisy positions, so noisier estimates when the extra freedom is not needed.

Process noise is the dial between the model and the data. Small $Q$ says “the model is nearly right”, so the filter averages over many frames and produces smooth estimates, but it reacts slowly when the target does something the model does not describe. Large $Q$ says “the model is a rough guide”, so the filter follows each measurement closely and passes more of the detector’s jitter. A constant-velocity model with large process noise and a constant-acceleration model with small process noise can behave similarly; the structure of the model and the size of its noise have to be chosen together.

Formal Definition

From continuous to discrete time

Kinematic models are usually stated in continuous time and then discretised for a frame interval $\Delta t$. For a linear stochastic differential equation $\dot{\mathbf{x}} = A \mathbf{x} + B \tilde{w}(t)$, with $\tilde{w}$ white noise of power spectral density $\tilde{q}$, the exact discrete model has

$$F = e^{A \Delta t}, \qquad Q = \int_0^{\Delta t} e^{A \tau} B \, \tilde{q} \, B^\top e^{A^\top \tau} \, d\tau .$$

Stone Soup, an open-source tracking framework, evaluates this integral in closed form for its constant-derivative models. An alternative, which gives a different $Q$, defines the noise directly in discrete time as a random input that is constant within each interval. Both kinds are in common use, and they are not interchangeable: they disagree on the structure of $Q$ and on the units of the noise parameter.

All the models below act independently on each coordinate axis, so they are written per axis; the multi-axis model is block-diagonal, as in the example code, which builds it with a Kronecker product.

Constant position (random walk)

State: position $p$. The transition is $F = 1$, and the position diffuses with $Q = \tilde{q}\,\Delta t$. Stone Soup’s RandomWalk model describes the target as “(almost) stationary”, with velocity modelled as white noise. In vision this is the model for an object that moves little and unpredictably between frames, or whose velocity cannot be estimated reliably, such as a jittering keypoint. It cannot predict systematic motion, so a moving target is always behind its estimate unless $\tilde{q}$ is large enough to cover each frame’s displacement.

Constant velocity

State: $(p, \dot{p})$. The transition is

$$F = \begin{bmatrix} 1 & \Delta t \\ 0 & 1 \end{bmatrix}.$$

Two process-noise models are standard.

• Continuous white-noise acceleration. Acceleration is continuous-time white noise with spectral density $\tilde{q}$ (units of position$^2$/time$^3$), which gives

  $$Q = \tilde{q} \begin{bmatrix} \Delta t^3/3 & \Delta t^2/2 \\ \Delta t^2/2 & \Delta t \end{bmatrix}.$$

  This is Stone Soup’s ConstantVelocity covariance, and the constant-velocity model in Genovese’s (2001) study of IMM filters at the Johns Hopkins Applied Physics Laboratory, where $\tilde{q}$ is a “plant noise spectral density” in m$^2$/s$^3$ chosen to set the filter’s steady-state gain.

• Discrete white-noise acceleration. Acceleration is a random constant $a_k$ within each interval, independent between intervals, with variance $\sigma_a^2$. Its effect on the state is $\Gamma a_k$ with $\Gamma = (\Delta t^2/2, \Delta t)^\top$, so

  $$Q = \sigma_a^2 \, \Gamma \Gamma^\top = \sigma_a^2 \begin{bmatrix} \Delta t^4/4 & \Delta t^3/2 \\ \Delta t^3/2 & \Delta t^2 \end{bmatrix}.$$

  This is the form in the Kalman filter article’s running example, and FilterPy’s Q_discrete_white_noise, which cites Bar-Shalom, Li and Kirubarajan’s textbook (2001) for it. Here $\sigma_a$ has the units of acceleration, which makes it easier to set from physical knowledge.

It follows from the second form that $Q$ has rank one per axis: the noise can push position and velocity only along the single direction $\Gamma$. The continuous form is full rank. With the same nominal noise level the two give different gains, so a parameter tuned for one should not be copied into the other.

Constant acceleration

State: $(p, \dot{p}, \ddot{p})$, with

$$F = \begin{bmatrix} 1 & \Delta t & \Delta t^2/2 \\ 0 & 1 & \Delta t \\ 0 & 0 & 1 \end{bmatrix}.$$

The noise now drives the acceleration. With continuous white jerk of spectral density $\tilde{q}$, Stone Soup’s ConstantAcceleration model has

$$Q = \tilde{q} \begin{bmatrix} \Delta t^5/20 & \Delta t^4/8 & \Delta t^3/6 \\ \Delta t^4/8 & \Delta t^3/3 & \Delta t^2/2 \\ \Delta t^3/6 & \Delta t^2/2 & \Delta t \end{bmatrix}.$$

The discrete counterpart takes the acceleration increment over each interval as a random constant of variance $\sigma_j^2$, which gives $Q = \sigma_j^2 \Gamma \Gamma^\top$ with $\Gamma = (\Delta t^2/2, \Delta t, 1)^\top$; this is FilterPy’s three-state Q_discrete_white_noise. Because the acceleration then performs a random walk, a Wiener process in discrete time, this model is called a Wiener-process acceleration model, as against the white-noise acceleration model above; the continuous model with white jerk is its continuous-time counterpart. The textbook that FilterPy cites was not read for this article. Genovese’s (2001) constant-acceleration filter uses yet another simplification, placing all the process noise on the acceleration term, with $\tilde{q}$ in m$^2$/s$^5$.

A constant-acceleration model follows sustained acceleration without lag. Its weakness in video is that true acceleration is seldom constant: it is roughly zero most of the time, with short bursts at the start and end of manoeuvres, so the model spends most of its time estimating a noisy, unneeded acceleration.

The Singer model

Between constant velocity (acceleration is white noise) and constant acceleration (acceleration is a random walk) sits a model in which acceleration is a zero-mean, exponentially correlated process: $\dot{a} = -\alpha a + \tilde{w}$, with $1/\alpha$ the time over which a manoeuvre persists. Stone Soup implements it as Singer, with $\alpha$ given as the reciprocal of the decorrelation time. It follows from this form (our derivation, checked numerically against Stone Soup’s closed-form $Q$) that as $\alpha \to 0$, with the noise intensity of $\tilde{w}$ fixed, the acceleration becomes a random walk, as in the constant-acceleration model above. As $\alpha \to \infty$ the acceleration decorrelates within each frame and acts on the velocity as white noise, as in the constant-velocity model, with spectral density equal to that of $\tilde{w}$ divided by $\alpha^2$; if the acceleration variance were held fixed instead, that density, and with it the process noise, would vanish.

Coordinated-turn models

Many targets move at nearly constant speed along circular arcs: cars cornering, aircraft banking, players running around an opponent. The coordinated-turn model, also called the constant-turn model, rotates the velocity vector at a constant turn rate $\omega$ instead of letting it change along a fixed axis. With state $(x, \dot{x}, y, \dot{y})$ and known $\omega$, the exact transition is

$$F(\omega) = \begin{bmatrix} 1 & \dfrac{\sin \omega \Delta t}{\omega} & 0 & -\dfrac{1 - \cos \omega \Delta t}{\omega} \\ 0 & \cos \omega \Delta t & 0 & -\sin \omega \Delta t \\ 0 & \dfrac{1 - \cos \omega \Delta t}{\omega} & 1 & \dfrac{\sin \omega \Delta t}{\omega} \\ 0 & \sin \omega \Delta t & 0 & \cos \omega \Delta t \end{bmatrix},$$

which is Stone Soup’s KnownTurnRate model, with continuous white-noise acceleration on each axis as its $Q$. As $\omega \to 0$ the matrix tends to the two-axis constant-velocity matrix. Unlike the models above, the axes are coupled: the turn moves velocity from $x$ to $y$.

When the turn rate is unknown, it joins the state, $(x, \dot{x}, y, \dot{y}, \omega)$; Stone Soup models it as a random walk. The transition is then nonlinear in the state (Stone Soup’s ConstantTurn), so it needs an extended Kalman filter, an unscented filter or a particle filter. Genovese’s (2001) three-dimensional turn model instead computed the turn rate from the estimated acceleration and velocity magnitudes, $\omega = |\hat{\mathbf{a}}| / |\hat{\mathbf{v}}|$, and added a constant-speed constraint as a pseudo-measurement, a procedure he credits to Blair, Watson and Alouani; he reported that it reduces lag errors in the turn, but makes the covariance smaller than the errors from measurement noise alone. Li and Jilkov (2003) published a survey of dynamic models for manoeuvring-target tracking; it was not read for this article.

Parametric image motion models

Kinematic models move a point. A patch tracker or a camera-motion compensator needs to move a region, and uses a low-dimensional map $\mathbf{x}' = f(\mathbf{x}; \mathbf{p})$ from one frame’s pixel coordinates to the next. Szeliski (2022, Table 2.1) lists the standard hierarchy of planar transformations:

Model	Form	Degrees of freedom	Preserves
Translation	$\mathbf{x}' = \mathbf{x} + \mathbf{t}$	2	orientation
Euclidean (rigid)	$\mathbf{x}' = R \mathbf{x} + \mathbf{t}$	3	lengths
Similarity	$\mathbf{x}' = s R \mathbf{x} + \mathbf{t}$	4	angles
Affine	$\mathbf{x}' = A \bar{\mathbf{x}}$, $A$ any $2 \times 3$	6	parallelism
Projective (homography)	$\tilde{\mathbf{x}}' \sim \tilde{H} \tilde{\mathbf{x}}$	8	straight lines

Each model also preserves the properties of the rows below it. A homography describes exactly the motion of a planar surface, or of any scene seen by a camera that only rotates about its centre; the affine model drops the perspective division and, in our reading, is a fair approximation when perspective effects across the region are small; similarity covers translation, in-plane rotation and zoom.

Trackers use these models in three ways.

• Patch and template tracking. Lucas–Kanade registration generalises from translation to any of these models by linearising the warped image in $\mathbf{p}$; the Jacobian is the image gradient times the warp’s Jacobian $\partial \mathbf{x}' / \partial \mathbf{p}$ (Szeliski, 2022, §9.2). Shi and Tomasi’s feature tracker, as Szeliski describes it (§7.1.5), tracks a patch by translation between neighbouring frames and then registers it to the frame where it was first detected with an affine model, which keeps the patch’s appearance comparable as it grows or foreshortens; this is the tracker usually called KLT. Szeliski notes that Baker and Matthews’ framework for these incremental methods has been widely used as a component of object trackers.
• Camera-motion compensation. In a moving or shaking camera, every object appears to move. A tracker estimates the global image motion between frames, for example by fitting a transform to tracked background features with RANSAC, and warps its predictions before matching, so that the kinematic model describes the object’s motion in the scene rather than the camera’s (see motion estimation for global motion estimation). Szeliski (§9.2.1) notes that video stabilisation often estimates a similarity transform for the same purpose. In the BoT-SORT code, the sparse-optical-flow option tracks corner features, fits OpenCV’s estimateAffinePartial2D with RANSAC, a four-parameter transform of rotation, uniform scale and translation, and applies its $2 \times 2$ linear part to the centre, size and velocity pairs of every Kalman state and its translation to the centre. The ORB and SIFT options fit the same transform, the ECC option a Euclidean one, and the evaluation script’s default reads precomputed $2 \times 3$ matrices from files. The SORT article describes BoT-SORT’s compensation as an affine warp, which matches the $2 \times 3$ matrix the code applies; the transforms the code fits are its four- and three-parameter special cases.
• Box motion. A bounding box tracker places kinematic dynamics on the parameters of a translation-and-scale image model. The SORT and DeepSORT papers were not read for this article, so both box models are described from the reference code. SORT’s code uses the state $(x, y, s, r, \dot{x}, \dot{y}, \dot{s})$: centre, area and aspect ratio, with constant-velocity dynamics on centre and area and the aspect ratio held constant. DeepSORT’s code uses $(x, y, a, h)$ (centre, aspect ratio, height) and their velocities, also constant velocity. ByteTrack’s official code reuses DeepSORT’s filter, as the ByteTrack article reports, so these two box models serve many tracking-by-detection systems.

Properties

Model order trades lag against noise. Every model is exact for some motions and biased for others. It follows from the transition matrices that a model with a constant $n$-th derivative predicts polynomial motion of degree $n$ without bias, and that its prediction error grows with the first derivative it leaves out. The extra states must be estimated from positions alone, so each added order lets more measurement noise into the estimate. Which side of the trade matters depends on the target and the frame rate. At video frame rates the change in velocity between consecutive frames is usually small next to the detector’s noise, which plausibly explains why constant velocity is the default in multi-object tracking (the example below makes the comparison on one trajectory).

Process noise in vision trackers is often heuristic. The physically derived $Q$ matrices above couple position and velocity noise. Vision trackers frequently use diagonal matrices instead. SORT’s code keeps FilterPy’s identity default and scales the velocity entries by 0.01, and the area-velocity entry by a further 0.01, giving $Q = \mathrm{diag}(1, 1, 1, 1, 10^{-2}, 10^{-2}, 10^{-4})$, with ten times more measurement variance on area and aspect ratio than on the centre. DeepSORT’s code makes the standard deviations proportional to the current box height, $h/20$ for position and $h/160$ for velocity, with small fixed values on the aspect ratio; a comment in the code calls the scheme “a bit hacky”. Scaling the noise with box size is a reasonable fix for a real problem: in pixel units, a nearby pedestrian moves and jitters far more than a distant one.

Units and frame rate. $Q$ depends on $\Delta t$, and a value tuned at one frame rate is wrong at another unless it is recomputed from a spectral density or an acceleration variance. Models written per frame ($\Delta t = 1$) hide this dependence; dropping frames, or running the detector on every second frame, changes the effective motion between updates.

Image-plane models are not scene models. Constant velocity in the scene is not constant velocity in the image: perspective makes an object approaching the camera accelerate and grow in the image, and camera motion adds to everything. Image-plane kinematic models absorb these effects into process noise, which is one reason 3D trackers that work in world coordinates, from lidar or calibrated cameras, can use tighter models.

Interacting multiple models

Real targets switch between regimes: driving straight, then turning, then braking. No single model fits all of them, and Genovese (2001) describes the older remedy, a single filter whose gain is raised when a separate detector declares a manoeuvre, as suffering from detection delays and false alarms. The interacting multiple model (IMM) filter is the subject of Blom and Bar-Shalom’s paper of 1988 on systems with Markovian switching coefficients, which was not read for this article. As Genovese (2001) presents it, the IMM filter runs several model-matched filters in parallel and treats the active model as a Markov chain with a fixed switching matrix $\Pi$, whose entry $p_{ij}$ is the prior probability of switching from model $i$ to model $j$ in one step. Each cycle has four steps.

1. Interaction (mixing). Each filter $j$ starts not from its own previous estimate but from a mixture of all filters’ estimates, weighted by the probability that the target was in model $i$ given that it is now in model $j$: $\mu_{i \mid j} \propto p_{ij} \mu_i$. The mixed covariance includes the spread of the estimates around the mixed mean.
2. Filtering. Each filter predicts and updates from its mixed starting point with its own $F$ and $Q$.
3. Model probability update. Each filter’s innovation $\boldsymbol{\nu}_j$ and innovation covariance $S_j$ give a Gaussian likelihood $\Lambda_j = \mathcal{N}(\boldsymbol{\nu}_j; \mathbf{0}, S_j)$, and the model probabilities become $\mu_j \propto \Lambda_j \sum_i p_{ij} \mu_i$.
4. Combination. The output is the probability-weighted mean of the filters’ estimates, with a covariance that again adds the spread of the means.

With $r$ models the cost is about $r$ filters plus the mixing, and no hard decision about which model is active is ever made. Typical sets pair a low-noise constant-velocity model with a high-noise or constant-acceleration model, or add a turn model. Genovese compared single constant-velocity and constant-acceleration filters with constant-velocity pairs, constant-velocity and constant-acceleration pairs, and a three-model IMM that added his turn model, in Monte Carlo simulations (100 realizations) of five manoeuvring airborne-target models seen by two simulated radars, scored by peak RMS errors over ten selected periods. He reported that accuracy improved with model complexity for most targets, that the IMM’s performance stayed close to that of its best-matched component, that it was relatively insensitive to the switching matrix and the process-noise values, and that the choice of component models mattered most.

Consistency checks: NIS and NEES

A filter is consistent when its reported covariance matches its actual errors. Consistency matters for tracking beyond accuracy: data association gates detections with the innovation covariance, and IMM model probabilities are computed from it, so an overconfident filter rejects the right detection and an underconfident one accepts clutter. Two statistics test it.

• Normalised estimation error squared (NEES): $\epsilon_k = (\mathbf{x}_k - \hat{\mathbf{x}}_{k \mid k})^\top P_{k \mid k}^{-1} (\mathbf{x}_k - \hat{\mathbf{x}}_{k \mid k})$. It needs the true state, so it is available only in simulation or with ground truth.
• Normalised innovation squared (NIS): $\epsilon_{\nu,k} = \boldsymbol{\nu}_k^\top S_k^{-1} \boldsymbol{\nu}_k$, with $\boldsymbol{\nu}_k$ the innovation. It needs only measurements, so it can be computed on real video.

If the model is correct and the noise Gaussian, NEES is chi-square distributed with $n$ degrees of freedom (the state dimension) and NIS with $m$ (the measurement dimension); the Kalman filter article uses the same NIS for gating. A single value is too noisy to judge, so the statistic is averaged. Over $N$ independent Monte Carlo runs, $N \bar{\epsilon}_k$ is a sum of $N$ independent chi-square variables and so is chi-square with $Nn$ degrees of freedom, which gives a two-sided acceptance band

$$\left[ \frac{\chi^2_{Nn}(\alpha/2)}{N}, \; \frac{\chi^2_{Nn}(1 - \alpha/2)}{N} \right]$$

around the expected value $n$ (and likewise with $m$ for NIS). On real data, where only one run exists, the same idea applies to an average over consecutive frames, since under a correct model the innovations are also uncorrelated in time. An average above the band means the filter is overconfident: $Q$ or $R$ is too small, or the model is wrong. Below the band means it is pessimistic. These bands are standard results; the derivation given here is ours, as no source for them was read for this article.

Examples

The example compares the three kinematic models on one manoeuvring target, at three process-noise levels each. The target is the centre of an object in the image, in pixels, sampled once per frame: it moves straight at 3 px/frame for 60 frames, makes a coordinated turn at 0.05 rad/frame for 40 frames (about 115°), and moves straight again for 50 frames. A simulated detector reports its centre with Gaussian noise of 2 px per axis, and each filter is told the true $R$. Constant position uses $Q = qI$ per frame; constant velocity uses discrete white-noise acceleration; constant acceleration uses discrete Wiener-process acceleration. The process-noise level is the open choice in such an experiment, so the code prints each model at three values chosen in advance: for constant velocity, $q = 10^{-2}$ is close to the squared centripetal acceleration in the turn, $(3 \times 0.05)^2 \approx 0.02$ px$^2$/frame$^4$, and the other values are a hundred times smaller and larger. Each configuration runs 50 Monte Carlo trials on the same trajectory with fresh noise; the first 10 frames are discarded as warm-up.

import numpy as np
from scipy.stats import chi2

# Ground truth: image-plane centre of a target (pixels), one step per frame.
# Straight at 3 px/frame, a coordinated turn (0.05 rad/frame) for frames 60-99,
# then straight again.
T, w, sigma = 150, 0.05, 2.0
turn = np.zeros(T, bool); turn[60:100] = True
p, v = np.array([40.0, 400.0]), np.array([3.0, 0.0])
pos, vel, acc = [], [], []
for k in range(T):
    om = w if turn[k] else 0.0
    a = om * np.array([-v[1], v[0]])            # centripetal acceleration
    pos.append(p.copy()); vel.append(v.copy()); acc.append(a)
    if om:
        s, c = np.sin(om), np.cos(om)
        p = p + np.array([[s, c - 1], [1 - c, s]]) @ v / om
        v = np.array([[c, -s], [s, c]]) @ v
    else:
        p = p + v
pos, vel, acc = map(np.array, (pos, vel, acc))

def model(order, q):
    """Per-axis F and Q for constant position (0), velocity (1), acceleration (2)."""
    F1 = np.array([[1, 1, 0.5], [0, 1, 1], [0, 0, 1]])[:order + 1, :order + 1]
    G1 = np.array([[0.5], [1.0], [1.0]])[:order + 1]
    if order == 0:
        G1 = np.array([[1.0]])                      # random walk: Q = q per frame
    F = np.kron(np.eye(2), F1)
    Q = np.kron(np.eye(2), q * G1 @ G1.T)
    H = np.kron(np.eye(2), np.eye(1, order + 1))
    return F, Q, H

def truth_state(order, k):
    cols = [pos[k], vel[k], acc[k]][:order + 1]
    return np.concatenate([[c[0] for c in cols], [c[1] for c in cols]])

def run(order, q, n_runs=50, seed=0):
    F, Q, H = model(order, q)
    R = sigma**2 * np.eye(2)
    n = F.shape[0]
    rng = np.random.default_rng(seed)
    err2 = np.zeros((n_runs, T)); nis = np.zeros((n_runs, T)); nees = np.zeros((n_runs, T))
    for r in range(n_runs):
        z_all = pos + rng.normal(0, sigma, size=pos.shape)
        x = np.zeros(n); x[0], x[order + 1] = z_all[0]
        P = np.diag(np.tile([sigma**2, 100.0, 10.0][:order + 1], 2))
        for k in range(1, T):
            x, P = F @ x, F @ P @ F.T + Q                       # predict
            y = z_all[k] - H @ x; S = H @ P @ H.T + R           # innovation
            K = np.linalg.solve(S, H @ P).T
            x, P = x + K @ y, (np.eye(n) - K @ H) @ P           # update
            e = truth_state(order, k) - x
            err2[r, k] = (e[0]**2 + e[order + 1]**2)
            nis[r, k] = y @ np.linalg.solve(S, y)
            nees[r, k] = e @ np.linalg.solve(P, e)
    return err2, nis, nees, n

N, frames = 50, np.arange(T) >= 10
segs = {"before": frames & (np.arange(T) < 60), "turn": turn,
        "after": np.arange(T) >= 100}
lo2, hi2 = chi2.ppf([0.025, 0.975], N * 2) / N
print(f"raw detections: RMSE {np.sqrt(2) * sigma:.2f} px; "
      f"95% band for the 50-run average NIS (m=2): [{lo2:.2f}, {hi2:.2f}]")
print("              position RMSE (px)    average NIS          NIS     average NEES")
print("model  q      before  turn after   before  turn after   in band before  turn after (n)")
for name, order, qs in [("CP", 0, [1, 10, 100]), ("CV", 1, [1e-4, 1e-2, 1]),
                        ("CA", 2, [1e-6, 1e-4, 1e-2])]:
    for q in qs:
        err2, nis, nees, n = run(order, q)
        a_nis, a_nees = nis.mean(0), nees.mean(0)
        rmse = [np.sqrt(err2[:, m].mean()) for m in segs.values()]
        mnis = [a_nis[m].mean() for m in segs.values()]
        mnees = [a_nees[m].mean() for m in segs.values()]
        band = np.mean((a_nis[frames] > lo2) & (a_nis[frames] < hi2))
        print(f"{name}  {q:7.0e} " + " ".join(f"{v:5.2f}" for v in rmse) + "   "
              + " ".join(f"{v:5.2f}" for v in mnis) + f"   {band:4.0%}  "
              + " ".join(f"{v:6.1f}" for v in mnees) + f" ({n})")

Output (NumPy 2.3.5, SciPy 1.16.3; about 1.5 seconds; two runs gave identical output):

raw detections: RMSE 2.83 px; 95% band for the 50-run average NIS (m=2): [1.48, 2.59]
              position RMSE (px)    average NIS          NIS     average NEES
model  q      before  turn after   before  turn after   in band before  turn after (n)
CP    1e+00  4.91  4.88  4.92   10.52 10.42 10.48     0%    15.4   15.3   15.5 (2)
CP    1e+01  2.43  2.44  2.40    1.66  1.65  1.64    76%     1.9    1.9    1.9 (2)
CP    1e+02  2.75  2.76  2.70    0.23  0.23  0.23     0%     2.0    2.0    1.9 (2)
CV    1e-04  1.07 14.68 14.37    1.97 61.39 58.30    42%     2.9 2586.1  904.5 (4)
CV    1e-02  1.35  2.63  1.55    1.85  3.60  2.03    66%     2.2   14.8    3.1 (4)
CV    1e+00  2.09  2.12  2.05    1.52  1.54  1.48    51%     2.3    2.4    2.2 (4)
CA    1e-06  1.49  6.63  7.04    1.98 14.58 16.42    39%     4.7 1481.5  944.8 (6)
CA    1e-04  1.57  1.97  1.71    1.91  2.50  2.18    76%     3.7   20.4   12.7 (6)
CA    1e-02  1.95  1.98  1.89    1.77  1.77  1.71    83%     3.3    3.8    3.2 (6)

The rows show the lag–noise trade directly. The constant-velocity filter with the smallest process noise is the most accurate of all on the first straight segment, where its model is exactly right, and fails in the turn: it trusts its straight-line prediction so strongly that its RMSE over the turn is about 15 px, and stays as high over the 50 frames after it, while the filter slowly relearns the new heading. The same model with $q = 10^{-2}$ gives up a little accuracy on the straight and keeps the turn error below that of the raw detections. With $q = 1$ the filter is uniform across segments but only modestly better than the detections, because it follows them closely. The constant-acceleration model with $q = 10^{-4}$ has the lowest error in the turn, since it can carry the centripetal acceleration, and pays for it with noisier estimates on the straights than the matched constant-velocity filter. The constant-position model never beats about 2.4 px: whatever its noise level, it must treat a steady 3 px step as noise.

The consistency columns show what each statistic can and cannot see.

• NIS flags a mismatched model without ground truth. The low-noise constant-velocity and constant-acceleration filters have average NIS near 2 before the turn and 15 to 60 inside it, far above the band’s upper limit of 2.59. Run on real video, that jump would reveal the manoeuvre and the overconfident model.
• NIS below the band means wasted trust in the data. The constant-position filter with $q = 100$ and, more mildly, the constant-velocity filter with $q = 1$ sit at or below the band’s lower edge: their innovations are smaller than their covariance claims, so they could average more.
• NIS can look acceptable while NEES fails. The constant-acceleration filter with $q = 10^{-4}$ keeps its average NIS below the band’s upper limit in every segment, about a quarter above the expected 2 in the turn, yet its NEES in the turn is 20 against an expected 6. The errors it underestimates are in velocity and acceleration, which the innovations of a position-only measurement reveal only weakly. NEES, which needs ground truth, catches them. The band applies to one frame’s 50-run average; by our reasoning, a test on the average over all 40 turn frames would have a much narrower band and would flag the excess NIS as well, so NIS is weakened here rather than blind.
• Near-consistency does not make a model right. The constant-position filter with $q = 10$ has NEES close to its expected value of 2 in every segment and an average NIS somewhat below 2 but inside the band, because a per-frame variance of 10 px$^2$ happens to cover a 3 px step ($3^2 = 9$). Its errors are reported roughly honestly, but they are larger than those of the constant-velocity filter with $q = 10^{-2}$ and the constant-acceleration filter with $q = 10^{-4}$. Consistency says the covariance is truthful, not that a better model does not exist.

The NEES values below their expected values on the straights (for example 2.2 to 3.1 for the constant-velocity filter with $q = 10^{-2}$, against 4) have a simple cause in this simulation: the true motion on those segments is perfectly straight, with no random acceleration at all, so a constant-velocity or constant-acceleration filter with $q > 0$ overstates its uncertainty there. None of the three noise levels is right for both regimes; that is the gap an IMM filter, switching between a low-noise and a high-noise model, is designed to close.

One change was made after the first run, prompted by its output. The first version pooled frames 10–59 and 100–149 into a single “straight” segment and reported, for each configuration, the RMSE on that segment and on the turn, the mean NIS and NEES over frames 10–149, and the fraction of frames whose 50-run average NIS and NEES lay inside their 95% bands; because the post-turn frames carry the recovery from the turn, the pooled figures mixed two different behaviours. The version shown reports RMSE, average NIS and average NEES for three segments (before, during and after the turn), keeps the NIS band fraction and drops the NEES band fraction. The models, noise levels, data, seed and filter are unchanged.

Common Misconceptions

• “Constant velocity means the tracker assumes the object never changes speed.” The process noise lets velocity change; the model only says that changes are unpredictable from one frame to the next. How quickly the filter accepts a change is set by $Q$, as the three constant-velocity rows above show.
• “A higher-order model is always better.” It removes lag for one class of motion and adds noise for all others; in the example the constant-acceleration filter, at every noise level tried, is less accurate on each straight segment than the best constant-velocity filter there.
• “$q$ is just a tuning knob, so its form does not matter.” Continuous and discrete noise models have different structures and units, and $Q$ depends on the frame interval. A value copied from code written for another form or frame rate sets a different model.
• “If NIS is fine, the filter is fine.” NIS tests only the measured components. In the example a filter whose average NIS stays below the band’s upper limit has a NEES more than three times its expected value during the turn.
• “Camera motion is just more process noise.” Inflating $Q$ to cover camera motion widens every gate and passes more noise into every track. Compensating for global image motion with a parametric model, then using a tight kinematic model in the stabilised frame, keeps both prediction and association sharp.

Where It Is Used

• Multi-object tracking. Every tracking-by-detection system predicts each track with a motion model before data association; SORT, the DeepSORT paper and ByteTrack use constant-velocity box models, and in DeepSORT the prediction’s covariance also sets a Mahalanobis gate. The multi-object tracking article discusses where simple models fail, as on DanceTrack, whose dancers move in complex, non-linear patterns.
• Filtering in general. The Bayes filter, Kalman filter, extended Kalman filter and particle filter all take a motion model as their prediction step; nonlinear models such as the unknown-rate turn model are what call for the latter two.
• Patch and feature tracking. Feature tracking and Lucas–Kanade registration use parametric image motion models (translation, affine, homography) for the patch itself, often with a kinematic model predicting where to start the search.
• Camera motion and ego-motion. Global motion estimation with similarity, affine or homography models compensates camera motion in trackers and stabilisers, and filter-based visual odometry and SLAM use kinematic models of the camera itself.
• 3D tracking. Trackers on lidar and multi-camera data run the same constant-velocity, acceleration and turn models in metric world coordinates, where a vehicle’s turns can be modelled physically with coordinated-turn models and IMM filters.