Correlation Filter Tracking

Correlation filter tracking follows a target through a video by learning a filter whose correlation with an image window gives a sharp peak at the target and low values everywhere else. The filter is trained on the target’s first appearance and updated in every frame; to find the target in the next frame, the tracker correlates the filter with a window around the previous position and moves to the strongest response. Training and detection both run in the Fourier domain, where correlation becomes element-wise multiplication, so the classic trackers process hundreds of frames per second on a CPU. The family is also called discriminative correlation filters (DCF), the term Danelljan and colleagues use, and its best-known members are MOSSE, the kernelized correlation filter (KCF), DSST, SRDCF and ECO.

The Object Tracking article places correlation filters among the other tracker families in its section on discriminative correlation filters. This article covers how the filters are learned, why the circular structure of the Fourier transform makes the learning cheap and what it costs, the extensions for multi-channel features, scale and the boundary effect, failure detection, and the results reported on OTB and VOT.

Sources. The papers that define MOSSE (Bolme et al., 2010), CSK (Henriques et al., 2012) and KCF (Henriques et al., 2015) are published by IEEE and Springer, and no authors’ copy could be read while this article was drafted. Their methods are described here from the later papers of Danelljan and colleagues (DSST, SRDCF, ECO), which state MOSSE’s objective and update in full, from the OTB benchmark paper and the VOT2015 report, from OpenCV’s implementations of both trackers, and from derivations that the example below checks numerically. Nothing is attributed to those three papers beyond what the readable sources say about them.

Purpose

Model-free tracking starts from a single box in the first frame. The tracker must build an appearance model of an object it has seen once, separate it from the surrounding background, and keep the model current as the object rotates, deforms and changes lighting, all within the time budget of one frame. In single-object tracking, tracking-by-detection answers this with a classifier trained online to tell the target from nearby background patches; the bottleneck is that a classifier trained on many patches, each a shifted copy of the window, is expensive to train every frame.

Correlation filters remove the bottleneck. A filter applied by correlation is a linear classifier evaluated at every shift of a window at once, and training it by least squares on every shift of a training window has a closed form in the Fourier domain. Danelljan et al. (2015) list the two advantages that made the method suited to tracking: all shifted versions of the few available samples are used as training data implicitly, and training and detection cost a few fast Fourier transforms. With $d$ feature channels and an $M \times N$ window, scoring every location costs $O(dMN \log MN)$ instead of $O(dM^2N^2)$ for direct evaluation (SRDCF, Sec. 2.1).

When to Use It

Correlation filters suit short-term, single-object tracking where the target is given by a box, moves a limited distance between frames, and must be followed fast on modest hardware: tracking on a robot or drone CPU, a fast first stage in a larger system, or a baseline that is easy to implement and understand. ECO’s hand-crafted variant runs at 60 frames per second on a single CPU including feature extraction (Danelljan et al., 2017).

They are a poor fit when the target can move far between frames, leave the view, or stay hidden for long. The search covers one window around the last position, which is why fast motion and recovery after occlusion are the standard weaknesses (SRDCF, Sec. 1), and the example below shows trackers that do not recover from an occlusion that hides the target completely for five frames. Long-term tracking needs a re-detection component on top. They also use no knowledge learned offline about what objects look like. Siamese trackers, which learn a similarity function offline from large video datasets and apply it without online training, and later transformer trackers came to lead the benchmarks, as Object Tracking describes. Single Object Tracking, the task, compares the families on common protocols.

How It Works

MOSSE: a filter trained by least squares

The Minimum Output Sum of Squared Error (MOSSE) tracker (Bolme et al., 2010), as Danelljan et al. (2014, Sec. 2) state it, learns a filter $h$ from grayscale training windows $f_1, \dots, f_t$ of the target, each paired with a desired output $g_j$: a Gaussian peaked at the target centre. In their notation, the filter minimizes the squared error between its correlation outputs and the desired outputs,

$$\varepsilon = \sum_{j=1}^{t} \left\| h \star f_j - g_j \right\|^2 = \frac{1}{MN} \sum_{j=1}^{t} \left\| \bar{H} F_j - G_j \right\|^2 ,$$

where $\star$ is circular correlation, capital letters are discrete Fourier transforms, the bar is complex conjugation and products are element-wise. The second equality is Parseval’s theorem. Because the products are element-wise, the problem separates into one scalar least-squares problem per frequency, and the minimizer is

$$H_t = \frac{\sum_{j=1}^{t} \bar{G}_j F_j}{\sum_{j=1}^{t} \bar{F}_j F_j} .$$

To locate the target in a new frame, the tracker extracts a window $z$ at the previous position, computes the response $y = \mathcal{F}^{-1}\{\bar{H}_t Z\}$ and moves to its maximum. This is the convolution theorem at work: one product of spectra scores every shift of the window.

Online update. Instead of re-solving over all past frames, the numerator $A_t$ and denominator $B_t$ of $H_t$ are kept as running averages with a learning rate $\eta$, $A_t = (1 - \eta) A_{t-1} + \eta\, \bar{G}_t F_t$ and the same for $B_t$ with $\bar{F}_t F_t$ (Danelljan et al., 2014, Sec. 2 and eq. 5). This weights past frames with exponentially decaying weights, so the model forgets old appearances at a rate set by $\eta$. A small constant $\lambda$ added to the denominator avoids division by near-zero frequency components, a point Danelljan et al. (2014) attribute to Bolme et al.

Preprocessing. Each window is multiplied by a cosine (Hann) window before the transform, which DSST also attributes to Bolme et al. The taper makes the window’s borders fade to zero, so the periodic extension that the DFT assumes has no seams, the issue the Fourier Transform article describes under windowing, and it gives the centre more weight than the edges. OpenCV’s implementation, which cites Bolme et al., also takes the logarithm of the pixel values and normalizes each window to zero mean and unit variance, and it trains the first filter on eight copies of the first window passed through a random affine warp (in version 4.10.0 the random generator is re-seeded on every call, so the eight warps are identical).

Failure detection. Danelljan et al. (2014) note that their intensity baseline “roughly corresponds” to MOSSE but lacks MOSSE’s explicit failure detection component. OpenCV’s implementation of MOSSE detects failure with the peak-to-sidelobe ratio (PSR), which measures the sharpness of the response peak: the peak value minus the mean of the response, divided by the response’s standard deviation, computed over the whole response map. A strong, isolated peak means the filter recognizes the window; a low PSR suggests occlusion or loss. When the PSR falls below 5.7, the implementation neither moves the box nor updates the filter. How the original paper defines the sidelobe region and its threshold could not be checked.

Circulant structure: why every shift comes for free

Why is a filter trained in the Fourier domain the same as a classifier trained on shifted patches? Stack all cyclic shifts of a training window $x$ as the rows of a data matrix $X$. Such a matrix is circulant, and every circulant matrix is diagonalized by the DFT: multiplying by $X$ is circular correlation with $x$, which the DFT turns into element-wise multiplication by $\hat{x}$. Ridge regression on these rows with targets $y$, the problem $\min_w \|Xw - y\|^2 + \lambda \|w\|^2$ from least squares, therefore has the closed form above, one division per frequency, with the Gaussian $y$ assigning each shifted copy a label that decays with its distance from the target. The example below builds the $256 \times 256$ matrix of all shifts of a $16 \times 16$ patch and confirms that the explicit solution and the FFT formula agree to rounding error.

The CSK tracker (Henriques et al., 2012) is the earliest tracker in this line that the readable sources describe in these terms: Danelljan et al. (2014) describe it as learning “a kernelized least-squares classifier of the target appearance”. In the OTB benchmark of Wu, Lim and Yang (2013), CSK was the fastest of the ten best-ranked trackers, at 362 frames per second in their measurements, and the authors credit the circulant structure for that speed. The cost is that the “negative” samples are not real background patches but cyclic shifts of one window, in which the target wraps around from one border to the opposite one. This periodic assumption is the source of the boundary effect discussed below.

Kernelized correlation filters

The kernelized correlation filter (KCF; Henriques et al., 2015) extends the closed form to non-linear classifiers through the kernel trick, as in support vector machines: the VOT2015 report describes it as equivalent to kernel ridge regression trained on thousands of patches at different translations around the object, and SRDCF counts it among the trackers that learn multi-channel filters on HOG features. In the dual form the classifier is $f(z) = \sum_i \alpha_i \kappa(x_i, z)$ over the shifted training patches $x_i$, and the coefficients solve $(K + \lambda I)\alpha = y$ with the kernel matrix $K_{ij} = \kappa(x_i, x_j)$. For a kernel that treats all shifts alike, such as the Gaussian kernel or any dot-product kernel, $K$ is again circulant. It follows that the solution and the responses on a new window $z$ take the forms

$$\hat{\alpha} = \frac{\hat{y}}{\hat{k}^{xx} + \lambda} , \qquad f(z) = \mathcal{F}^{-1}\{ \hat{k}^{xz} \odot \hat{\alpha} \} ,$$

where $k^{xz}$ is the vector of kernel values between $z$ and every shift of $x$. For the Gaussian kernel, every entry of $k^{xz}$ follows from one cross-correlation, since $\|x - z_s\|^2 = \|x\|^2 + \|z\|^2 - 2\, x^\top z_s$:

$$k^{xz} = \exp\!\left( -\frac{1}{\sigma^2 n} \max\!\Big(0,\ \|x\|^2 + \|z\|^2 - 2\, \mathcal{F}^{-1}\Big\{ \textstyle\sum_c \bar{\hat{x}}_c \odot \hat{z}_c \Big\} \Big) \right) ,$$

with the sum over feature channels $c$ and $n$ the number of elements. This is the form in OpenCV’s implementation (version 4.10.0), whose defaults are $\sigma = 0.2$, $\lambda = 10^{-4}$, a model interpolation factor of 0.075 and a label width of $1/16$ of the target box’s linear size (the square root of its area, before the window is padded to twice the box); whether those match the paper could not be checked. The example verifies the dual closed form against an explicit $256 \times 256$ kernel matrix. With a linear kernel, $k^{xz}$ is the cross-correlation itself and the dual solution reduces to a MOSSE-type filter trained on a single sample; this follows from the formulas above.

In OpenCV’s implementation, KCF keeps a model of the target’s features and of the coefficients $\hat{\alpha}$ (by default as a separate numerator and denominator) and updates both by linear interpolation with the interpolation factor each frame, which plays the role of MOSSE’s running averages.

Multi-channel features

Grayscale pixels are a weak representation, and a large gain in the family came from replacing them with multi-channel features. Correlation over $d$ channels sums the per-channel correlations, $S = \sum_{l=1}^{d} h^l \star f^l$. With one training sample and a regularizer $\lambda \sum_l \|h^l\|^2$, the solution is $H^l = \bar{G} F^l / \left(\sum_{k=1}^{d} \bar{F}^k F^k + \lambda\right)$ (Danelljan et al., 2014, eq. 4). The exact solution over many samples requires a $d \times d$ linear system per frequency, which DSST avoids by updating the numerator per channel and a shared denominator as running averages, an approximation (eq. 5). Danelljan et al. (2015) argue that such approximations make the learning sub-optimal.

Histograms of oriented gradients became the standard channels: the HOG article reports KCF’s results with HOG cells against raw pixels, and DSST combined PCA-HOG computed at a $1 \times 1$ cell size with the grayscale value. In DSST’s experiment on 28 OTB sequences with scale variation, HOG improved the median distance precision of an intensity-only baseline by 11.6 points and the median overlap precision by 6.9 points, and reduced the median centre location error from 31.2 to 15.9 pixels (Danelljan et al., 2014, Sec. 4.3). Colour Names (11 channels in ECO’s set-up) were also common, and later trackers took activations of convolutional neural networks trained for classification.

How to Apply It

The example implements MOSSE, KCF with a Gaussian kernel on grayscale pixels, and KCF on HOG channels with NumPy FFTs, and runs them on a synthetic sequence in which the target’s true centre is known in every frame. It first checks the circulant identities of the previous section on a $16 \times 16$ patch.

The sequence has 160 frames of $300 \times 300$ pixels. The background is a crop of scikit-image’s grass texture. The target is a $40 \times 40$ patch of the astronaut’s face, which moves along a Lissajous path at up to 6.4 pixels per frame. Over frames 40 to 100 it blends linearly into a cat’s face (chelsea), whose pixels correlate at −0.09 with the astronaut patch. A static $60 \times 60$ patch of brick texture lies on the path, overlapping the target from frame 102 and hiding it completely in frames 113 to 117, and Gaussian noise with standard deviation 5 grey levels is added to every frame. Before the first run, a check confirmed that a fixed template matched by normalized cross-correlation finds the target exactly in frame 1 but is more than 20 pixels off in every frame from 86 on, so the sequence requires an adapting model.

All trackers search an $80 \times 80$ window, twice the target size as in DSST, and use a Gaussian label with a standard deviation of $1/16$ of the target size. MOSSE uses the log, normalization and cosine window of OpenCV’s implementation, is initialized on the first window and eight random small affine warps of it, and uses $\lambda = 0.01$ as in DSST. Both KCF versions use OpenCV’s defaults, $\sigma = 0.2$ and $\lambda = 10^{-4}$. The HOG channels come from scikit-image’s hog with 9 orientations, $4 \times 4$-pixel cells and $1 \times 1$-cell blocks, a simpler stand-in for the 31-channel HOG used, for example, in ECO, and that tracker moves in 4-pixel steps. The trackers have no scale estimation and no motion model; the target’s size is constant.

The learning rate is left open by the sources: DSST and SRDCF use 0.025, OpenCV’s KCF 0.075 and OpenCV’s MOSSE 0.2, so every tracker runs at all three. Each also runs with and without PSR gating, which holds the position and skips the update when the PSR falls below OpenCV’s threshold of 5.7. That threshold was set for OpenCV’s MOSSE, whose window equals the box; here it is applied to a padded window and to KCF responses as well. The measures are the centre error per frame, the precision at 20 pixels (the fraction of frames with centre error below 20 pixels, OTB’s representative threshold) over all frames and over the frames after the occlusion (118 to 159), and the mean PSR in four phases. The noise and the initial warps depend on the seed; seed 0 is shown in detail and five seeds are summarized. The script runs in about 45 seconds. Versions: NumPy 2.3.5, scikit-image 0.25.2.

import numpy as np
from numpy.fft import fft2, ifft2
from skimage import data, color, transform
from skimage.feature import hog

# ---------- 1. Circulant check: Fourier closed forms vs explicit regression over all shifts ----------
rng = np.random.default_rng(0)
x = rng.normal(size=(16, 16))                     # a small training patch
z = np.roll(x, (3, -5), axis=(0, 1)) + 0.1 * rng.normal(size=x.shape)  # a shifted, noisy test patch
n = x.size
d0, d1 = np.meshgrid(np.fft.fftfreq(16) * 16, np.fft.fftfreq(16) * 16, indexing="ij")
y = np.exp(-(d0**2 + d1**2) / (2 * 1.5**2))       # Gaussian label, peak at shift (0, 0)
shifts = [(i, j) for i in range(16) for j in range(16)]
Xs = np.array([np.roll(x, (-i, -j), axis=(0, 1)).ravel() for i, j in shifts])  # row s = x moved by -s
Zs = np.array([np.roll(z, (-i, -j), axis=(0, 1)).ravel() for i, j in shifts])
lam = 1e-2

# primal: ridge regression h = (X^T X + lam I)^-1 X^T y, against conj(H) = Y conj(X) / (X conj(X) + lam)
h_explicit = np.linalg.solve(Xs.T @ Xs + lam * np.eye(n), Xs.T @ y.ravel())
X, Y = fft2(x), fft2(y)
Hc = Y * np.conj(X) / (X * np.conj(X) + lam)
h_fourier = np.real(ifft2(np.conj(Hc))).ravel()
r_explicit = Zs @ h_explicit                       # responses at all shifts of z
r_fourier = np.real(ifft2(Hc * fft2(z))).ravel()

# dual: Gaussian kernel ridge regression alpha = (K + lam I)^-1 y, against alpha_hat = Y / (k_hat^xx + lam)
sig = 0.5
def gauss_corr(a, b, sig):                         # k^{ab}[s] = kappa(a, b moved by -s), all s at once
    c = np.real(ifft2(np.sum(np.conj(fft2(a, axes=(0, 1))) * fft2(b, axes=(0, 1)), axis=-1)))
    d = np.maximum(0, (a**2).sum() + (b**2).sum() - 2 * c) / a.size
    return np.exp(-d / sig**2)
kappa = lambda u, v: np.exp(-((u - v) ** 2).sum() / (n * sig**2))
K = np.array([[kappa(u, v) for v in Xs] for u in Xs])
a_explicit = np.linalg.solve(K + lam * np.eye(n), y.ravel())
Kz = np.array([[kappa(u, v) for u in Xs] for v in Zs])           # Kz[s, t] = kappa(x_t, z_s)
alpha_hat = Y / (fft2(gauss_corr(x[..., None], x[..., None], sig)) + lam)
a_fourier = np.real(ifft2(alpha_hat)).ravel()
rk_explicit = Kz @ a_explicit
rk_fourier = np.real(ifft2(alpha_hat * fft2(gauss_corr(x[..., None], z[..., None], sig)))).ravel()
print("circulant check, max abs difference (explicit 256x256 solve vs FFT):")
print(f"  linear filter {np.abs(h_explicit - h_fourier).max():.1e}, responses {np.abs(r_explicit - r_fourier).max():.1e}")
print(f"  kernel alpha  {np.abs(a_explicit - a_fourier).max():.1e}, responses {np.abs(rk_explicit - rk_fourier).max():.1e}")
pk = np.unravel_index(np.argmax(rk_fourier.reshape(16, 16)), (16, 16))
print(f"  kernel response on z peaks at shift {tuple(int(v) for v in (np.array(pk) + 8) % 16 - 8)}, true shift (3, -5)")

# ---------- 2. Synthetic sequence with known target centres ----------
T, H, W, S, P = 160, 300, 300, 40, 80   # frames, frame size, target size, search window (2x target)

def patch(img, r0, r1, c0, c1):
    g = color.rgb2gray(img[r0:r1, c0:c1]) if img.ndim == 3 else img[r0:r1, c0:c1] / 255.0
    return 255 * transform.resize(g, (S, S), anti_aliasing=True)

def make_sequence(seed):
    rng = np.random.default_rng(seed)
    bg = data.grass()[:H, :W].astype(float)
    tA = patch(data.astronaut(), 20, 160, 160, 300)     # astronaut's face
    tB = patch(data.chelsea(), 40, 240, 120, 320)       # cat's face
    occ = data.brick()[100:160, 100:160].astype(float)  # 60 x 60 occluder
    t = np.arange(T)
    cx = np.round(150 + 90 * np.sin(2 * np.pi * t / T)).astype(int)
    cy = np.round(150 + 60 * np.sin(4 * np.pi * t / T + 0.3)).astype(int)
    oy, ox = cy[115], cx[115]                            # target fully hidden in frames 113-117
    frames = []
    for k in range(T):
        a = np.clip((k - 40) / 60, 0, 1)                 # appearance blend over frames 40-100
        f = bg.copy()
        f[cy[k] - S // 2:cy[k] + S // 2, cx[k] - S // 2:cx[k] + S // 2] = (1 - a) * tA + a * tB
        f[oy - 30:oy + 30, ox - 30:ox + 30] = occ
        frames.append(f + rng.normal(0, 5, f.shape))
    return np.array(frames), np.stack([cy, cx], 1)

# ---------- 3. Trackers ----------
def window(frame, c):
    fp = np.pad(frame, P, mode="edge")
    return fp[c[0] + P - P // 2:c[0] + P + P // 2, c[1] + P - P // 2:c[1] + P + P // 2]

def gauss_label(m, sigma):
    d = np.fft.fftfreq(m) * m
    return np.exp(-(d[:, None]**2 + d[None, :]**2) / (2 * sigma**2))

def psr(r):                                          # OpenCV's MOSSE definition: whole response
    return (r.max() - r.mean()) / (r.std() + 1e-5)

def peak(r, cell):
    i, j = np.unravel_index(np.argmax(r), r.shape)
    m = r.shape[0]
    return np.array([(i + m // 2) % m - m // 2, (j + m // 2) % m - m // 2]) * cell

class Gray:                                          # log, normalize, cosine (Hann) window
    cell, m = 1, P
    hann = np.outer(np.hanning(P), np.hanning(P))
    def __call__(self, w):
        w = np.log1p(w.clip(0, None))
        return ((w - w.mean()) / (w.std() + 1e-5) * self.hann)[..., None]

class HOG:                                           # 9 orientations, 4-pixel cells, 1x1-cell blocks
    cell, m = 4, P // 4
    hann = np.outer(np.hanning(P // 4), np.hanning(P // 4))
    def __call__(self, w):
        h = hog(w.clip(0, 255) / 255, orientations=9, pixels_per_cell=(4, 4),
                cells_per_block=(1, 1), feature_vector=False)
        return h.reshape(self.m, self.m, 9) * self.hann[..., None]

class MOSSE:
    def __init__(self, frame, c, eta, seed, lam=0.01):
        self.feat, self.eta, self.lam = Gray(), eta, lam
        self.G = fft2(gauss_label(P, S / 16))
        w = window(frame, c)
        rng = np.random.default_rng(seed)
        self.A = self.B = 0
        for i in range(9):                           # the first sample plus 8 random small warps
            if i:
                M = np.eye(3)
                M[:2, :2] += rng.uniform(-0.1, 0.1, (2, 2))
                ctr = np.array([P / 2, P / 2])
                M[:2, 2] = ctr - M[:2, :2] @ ctr
                wi = transform.warp(w, transform.AffineTransform(matrix=M), mode="reflect", preserve_range=True)
            else:
                wi = w
            F = fft2(self.feat(wi)[..., 0])
            self.A = self.A + self.G * np.conj(F)
            self.B = self.B + F * np.conj(F)
    def respond(self, w):
        return np.real(ifft2(self.A / (self.B + self.lam) * fft2(self.feat(w)[..., 0])))
    def update(self, w):
        F = fft2(self.feat(w)[..., 0])
        self.A = (1 - self.eta) * self.A + self.eta * self.G * np.conj(F)
        self.B = (1 - self.eta) * self.B + self.eta * F * np.conj(F)

class KCF:
    def __init__(self, frame, c, eta, feat, lam=1e-4, sig=0.2):
        self.feat, self.eta, self.lam, self.sig = feat, eta, lam, sig
        self.Y = fft2(gauss_label(feat.m, S / 16 / feat.cell))
        self.x = self.feat(window(frame, c))
        self.alpha_hat = self.train(self.x)
    def train(self, x):
        return self.Y / (fft2(gauss_corr(x, x, self.sig)) + self.lam)
    def respond(self, w):
        return np.real(ifft2(self.alpha_hat * fft2(gauss_corr(self.x, self.feat(w), self.sig))))
    def update(self, w):
        xn = self.feat(w)
        self.alpha_hat = (1 - self.eta) * self.alpha_hat + self.eta * self.train(xn)
        self.x = (1 - self.eta) * self.x + self.eta * xn

def run(tracker, frames, gt, gate):
    c, pos, psrs = gt[0].copy(), [gt[0].copy()], [np.nan]
    for f in frames[1:]:
        r = tracker.respond(window(f, c))
        q = psr(r)
        psrs.append(q)
        if not (gate and q < 5.7):                   # gated: hold position, skip the update
            c = c + peak(r, tracker.feat.cell)
            tracker.update(window(f, c))
        pos.append(c.copy())
    err = np.linalg.norm(np.array(pos) - gt, axis=1)
    return err, np.array(psrs), np.array(pos)

# ---- experiments ----
phases = {"before": range(1, 40), "change": range(40, 101), "hidden": range(113, 118), "after": range(118, T)}
makers = {"MOSSE": lambda f, c, eta, s: MOSSE(f, c, eta, s),
          "KCF-gray": lambda f, c, eta, s: KCF(f, c, eta, Gray()),
          "KCF-HOG": lambda f, c, eta, s: KCF(f, c, eta, HOG())}
results = {}
for seed in range(5):
    frames, gt = make_sequence(seed)
    for name, mk in makers.items():
        for eta in (0.025, 0.075, 0.2):
            for gate in (False, True):
                results[name, eta, gate, seed] = run(mk(frames[0], gt[0], eta, seed), frames, gt, gate)

print("\nseed 0: precision at 20 px (all frames | after occlusion), mean PSR per phase")
print("tracker    eta    gate  P@20 all  P@20 after  lost at  final err  to occluder  PSR before/change/hidden/after")
occluder = make_sequence(0)[1][115]                  # occluder centre
for name in makers:
    for eta in (0.025, 0.075, 0.2):
        for gate in (False, True):
            err, q, pos = results[name, eta, gate, 0]
            ps = "/".join(f"{np.mean(q[list(r)]):.1f}" for r in phases.values())
            lost = np.flatnonzero(err > 20)[0] if (err > 20).any() else "-"
            print(f"{name:9s} {eta:6.3f}  {'on ' if gate else 'off'}   {np.mean(err[1:] < 20):7.2f} "
                  f"{np.mean(err[118:] < 20):10.2f} {lost!s:>8} {err[-1]:9.1f} "
                  f"{np.linalg.norm(pos[-1] - occluder):11.1f}   {ps}")

print("\nP@20 over all frames, seeds 0-4 (min-max), and number of seeds with P@20 after occlusion >= 0.9")
for name in makers:
    for eta in (0.025, 0.075, 0.2):
        for gate in (False, True):
            a = [np.mean(results[name, eta, gate, s][0][1:] < 20) for s in range(5)]
            b = [np.mean(results[name, eta, gate, s][0][118:] < 20) for s in range(5)]
            print(f"{name:9s} {eta:6.3f}  {'on ' if gate else 'off'}   {min(a):.2f}-{max(a):.2f}   "
                  f"recovered on {sum(v >= 0.9 for v in b)}/5")

circulant check, max abs difference (explicit 256x256 solve vs FFT):
  linear filter 6.1e-14, responses 5.4e-14
  kernel alpha  5.6e-16, responses 1.0e-15
  kernel response on z peaks at shift (3, -5), true shift (3, -5)

seed 0: precision at 20 px (all frames | after occlusion), mean PSR per phase
tracker    eta    gate  P@20 all  P@20 after  lost at  final err  to occluder  PSR before/change/hidden/after
MOSSE      0.025  off      0.67       0.00      102      91.8        52.2   14.0/9.9/6.1/12.2
MOSSE      0.025  on       0.57       0.00       92      37.7        74.9   14.0/9.8/3.9/3.9
MOSSE      0.075  off      0.69       0.00      110      91.4        53.2   14.5/13.5/12.1/16.9
MOSSE      0.075  on       0.69       0.00      111      91.4        53.2   14.5/13.5/11.6/16.8
MOSSE      0.200  off      0.70       0.00      112      78.1        33.5   15.1/14.2/12.9/17.6
MOSSE      0.200  on       0.70       0.00      112      78.1        33.5   15.1/14.2/12.9/17.6
KCF-gray   0.025  off      0.68       0.00      109      88.5        54.7   17.1/13.1/9.6/14.8
KCF-gray   0.025  on       0.69       0.00      111      77.5        40.2   17.1/13.1/4.8/4.9
KCF-gray   0.075  off      0.69       0.00      110      75.2        46.6   17.2/15.6/10.2/17.0
KCF-gray   0.075  on       0.69       0.00      110      89.7        53.5   17.2/15.6/5.4/14.6
KCF-gray   0.200  off      0.70       0.00      112      78.1        33.5   17.2/16.4/13.9/17.8
KCF-gray   0.200  on       0.70       0.00      112      78.1        33.5   17.2/16.4/13.9/17.8
KCF-HOG    0.025  off      0.71       0.00      114      77.9        25.3   11.7/9.2/6.6/9.8
KCF-HOG    0.025  on       0.71       0.00      114      77.9        25.3   11.7/9.2/6.4/9.7
KCF-HOG    0.075  off      0.71       0.00      114      77.9        25.3   11.8/9.7/8.5/11.9
KCF-HOG    0.075  on       0.71       0.00      114      77.9        25.3   11.8/9.7/8.5/11.9
KCF-HOG    0.200  off      0.70       0.00      113      78.8        29.1   11.7/10.6/11.0/13.1
KCF-HOG    0.200  on       0.70       0.00      113      78.8        29.1   11.7/10.6/11.0/13.1

P@20 over all frames, seeds 0-4 (min-max), and number of seeds with P@20 after occlusion >= 0.9
MOSSE      0.025  off   0.67-0.67   recovered on 0/5
MOSSE      0.025  on    0.56-0.57   recovered on 0/5
MOSSE      0.075  off   0.68-0.69   recovered on 0/5
MOSSE      0.075  on    0.69-0.69   recovered on 0/5
MOSSE      0.200  off   0.70-0.70   recovered on 0/5
MOSSE      0.200  on    0.70-0.70   recovered on 0/5
KCF-gray   0.025  off   0.68-0.68   recovered on 0/5
KCF-gray   0.025  on    0.69-0.69   recovered on 0/5
KCF-gray   0.075  off   0.69-0.69   recovered on 0/5
KCF-gray   0.075  on    0.69-0.69   recovered on 0/5
KCF-gray   0.200  off   0.70-0.70   recovered on 0/5
KCF-gray   0.200  on    0.70-0.70   recovered on 0/5
KCF-HOG    0.025  off   0.70-0.71   recovered on 0/5
KCF-HOG    0.025  on    0.70-0.71   recovered on 0/5
KCF-HOG    0.075  off   0.71-0.71   recovered on 0/5
KCF-HOG    0.075  on    0.71-0.71   recovered on 0/5
KCF-HOG    0.200  off   0.70-0.71   recovered on 0/5
KCF-HOG    0.200  on    0.70-0.71   recovered on 0/5

The circulant check shows that the FFT formulas are not an approximation to regression over shifted patches but the same solution: the explicit $256 \times 256$ solves and the closed forms agree to $10^{-13}$ or better, and the kernel classifier’s response on a shifted, noisy copy peaks at the true shift. For the trackers’ $80 \times 80$ grayscale window, the explicit kernel matrix would have 6,400 rows and about 41 million entries; the FFT version never forms it.

Before the appearance change every tracker follows the target, with mean PSRs between about 12 and 17. During the blend, the learning rate matters. MOSSE at 0.025 is lost at frame 102, earlier than any run at a higher rate, which points to a model that adapts too slowly to the blend. With gating it is lost at frame 92. Gated and ungated runs are identical until the PSR first falls below 5.7, so the gate fired by frame 92 and froze both the model and the position: gating that stops the updates when confidence is low also stops the adaptation that would restore it. Both KCF versions, and MOSSE at the higher rates, follow the blend.

The occlusion defeats every tracker. All other runs are lost between frames 109 and 114, while the target slides under the occluder. No run on any of the five seeds (90 runs in all) reaches a precision of 0.9 after the occlusion, and in all 18 seed-0 runs that precision is zero. Apart from gated MOSSE at 0.025, lost earlier, the runs end 25 to 55 pixels from the occluder’s centre (its half-width is 30 pixels) and 75 to 92 pixels from the target: they stay on or beside the occluder. By construction of the path, the target crosses the occluder vertically at 4 to 5 pixels per frame and is clear of it only from frame 127, about 50 pixels from its centre. That is beyond the 40-pixel half-width of a window centred on the occluder, and for a tracker beside the occluder it is at best near the window’s edge, which the cosine taper weights down. In our reading this is the restricted search region that Danelljan et al. (2015) attribute to standard correlation filters, combined with the absence of a motion model that could predict where the target reappears; the example does not test a larger window, a motion model or re-detection.

The PSR column shows what the ratio measures. At the lowest rate, the PSR falls during the hidden frames (MOSSE from 14.0 before the blend to 6.1; grayscale KCF from 17.1 to 9.6). At 0.2 it barely moves (MOSSE 15.1 to 12.9), presumably because during the eleven frames of partial overlap (102 to 112) a filter updated at that rate has already absorbed much of the occluder, and after the loss the PSR is higher than at the start (17.6) while the tracker sits on the wrong object. The PSR measures how well the window matches the current model, not whether the model is still the target. Gating at 5.7 left the HOG tracker’s precision, loss frame and final position unchanged (at 0.025 its PSR means shift by up to 0.2, so the gate fired there, without effect on the outcome), and for the others it changed when the target was lost by at most ten frames without preventing the loss. The five noise seeds give precisions within 0.01 of each other for every configuration.

The script was changed once after its first run. The first run printed, for seed 0, the precision over all frames and after the occlusion, the final error and the mean PSR per phase; after it showed that no run recovered, two columns were added to that table, the first frame with an error above 20 pixels and the distance of the final position from the occluder’s centre, to show when and where the trackers were lost. The sequence, the trackers, their parameters and the seed summary are unchanged. Before the first tracker run, the occluder was reduced from $80 \times 80$ at the target’s frame-128 position to $60 \times 60$ at its frame-115 position, because the first design hid the target for 17 frames rather than the few frames planned.

Variants

Scale estimation: DSST

MOSSE, CSK and KCF estimate only translation; the box keeps its first size. The Discriminative Scale Space Tracker (DSST) of Danelljan et al. (2014) learns a second, one-dimensional correlation filter over scale. After the translation filter has found the new position, DSST extracts patches of sizes $a^n P \times a^n R$ around it, for $S = 33$ scale levels $n$ with $a = 1.02$, resizes each to a fixed size and describes it by one HOG feature vector, so that the scale filter’s input is a function of the scale index. The peak of its response gives the scale change. The translation filter is $M \times N$ at twice the initial target size; the learning rate is 0.025 and $\lambda = 0.01$, the label’s standard deviation is $1/16$ of the target size for translation and 1.5 for scale, and the scale descriptor has at most 992 dimensions.

The authors compared it with an exhaustive alternative, a three-dimensional filter over position and scale searched jointly, on the 28 OTB-2013 sequences with the scale-variation attribute. Median overlap precision at IoU 0.5 rose from 37.8% for the translation-only HOG baseline to 52.2% for the exhaustive search and 75.5% for the separate scale filter, and median distance precision at 20 pixels from 74.5% to 87.6% and 93.3%, while the median speed was 44.1, 0.96 and 24.0 frames per second (Table 1). The table makes the separate filter 25 times faster than the exhaustive search; the text of Section 4.4 says 24 times. On the same sequences the best competing trackers reached a median overlap precision of 69.9% (ASLA) and a median distance precision of 76.8% (Struck) (Table 2). The authors’ project page reports that DSST won the VOT2014 challenge, among 38 participating trackers. Later trackers built on it; in VOT2015, for example, OACF extends DSST with a per-pixel likelihood map from colour histograms (Kristan et al., 2015, A.13).

Another approach, used by SAMF (Li and Zhu, 2014) and, following it, by SRDCF, applies the translation filter at several resolutions and keeps the scale with the highest score (SRDCF, Sec. 4.2).

The boundary effect and spatial regularization: SRDCF

Because the training set consists of cyclic shifts of one window, a correlation filter learns from wrapped-around images rather than real background, and its detection scores are accurate only near the centre of the window. Danelljan et al. (2015) list the consequences: inaccurate negative samples weaken the model, the effective search region is small, which hurts under fast motion and limits re-detection after occlusion, and simply enlarging the window brings so much background into the positive samples that performance drops. They call these unwanted boundary effects.

Their Spatially Regularized Discriminative Correlation Filters (SRDCF) replace the uniform regularizer $\lambda \sum_l \|h^l\|^2$ by a spatially varying penalty $\sum_l \|w \cdot h^l\|^2$, where the weight $w$ is small over the target and grows with distance from it:

$$w(m, n) = \mu + \eta \left( \frac{m}{P} \right)^2 + \eta \left( \frac{n}{Q} \right)^2 ,$$

with $P \times Q$ the target size, $\mu = 0.1$ and $\eta = 3$. Filter coefficients outside the target are suppressed, so the training window can grow to $4^2$ times the target area without the filter learning the background. Uniform weights $w = \sqrt{\lambda}$ recover the standard filter. The penalty couples frequencies, so the per-frequency closed form is lost; the authors truncate $w$’s spectrum to about 10 significant coefficients, which makes the normal equations sparse, and run 4 Gauss–Seidel iterations per frame from the previous frame’s filter, with a learning rate of 0.025, HOG with $4 \times 4$-pixel cells, and sub-grid localization by maximizing the trigonometric interpolation of the scores with Newton’s method. The MATLAB implementation ran at about 5 frames per second.

On OTB-2013 the spatial regularization and the larger window together made the difference: mean overlap precision at IoU 0.5 was 71.1% with the standard regularizer and 72.2% with the spatial one at the conventional window size, but 50.1% and 78.1% at the expanded size (SRDCF, Table 1). Galoogahi, Sim and Lucey (2015) had attacked the same problem with correlation filters with limited boundaries, a constrained single-channel formulation solved by ADMM; in SRDCF’s comparison under that method’s settings (one grayscale channel, no scale estimation), it reached 48.6% mean overlap precision on OTB-2013 against 54.3% for SRDCF.

Deep features, continuous filters and ECO

DeepSRDCF, which differs from SRDCF only in using convolutional features from a pretrained network, ranked second in VOT2015, an order of magnitude slower than SRDCF according to the challenge report. C-COT (Danelljan et al., 2016) learned the filter in the continuous spatial domain, which lets feature maps of different resolutions be combined without resampling and gives a continuous score function for sub-pixel localization; it ranked first in VOT2016 (ECO, Sec. 2).

ECO (Danelljan et al., 2017) attacked the cost and the overfitting of these large models with three changes, in outline:

• Factorized convolution. Instead of one filter per feature channel, a smaller set of $C$ basis filters is learned with a $D \times C$ projection matrix, jointly and discriminatively in the first frame by Gauss–Newton with conjugate gradient. For the VGG-m Conv-5 layer this reduced 512 channels to 64; for HOG, 31 to 10; for Colour Names, 11 to 3.
• A generative model of the training set. Instead of storing the last $M = 400$ samples, a Gaussian mixture model with $L = 50$ components summarizes them; similar samples merge into one component, so the set stays diverse rather than filled with near-copies of recent frames.
• Sparse updates. The filter is optimized only every $N_S = 6$ frames, with 5 conjugate gradient iterations, while the sample model is updated every frame. The authors report that this moderately infrequent update improved robustness, which they attribute to less overfitting to recent frames.

On VOT2016, these steps raised the expected average overlap (EAO) from C-COT’s 0.331 to 0.374, a relative gain of 13.0%, while the speed on one CPU, excluding feature extraction, rose from 0.3 to 6.0 frames per second (ECO, Table 2). ECO-HC, with HOG and Colour Names only, ran at 60 frames per second on a single CPU including feature extraction and reached an EAO of 0.322 on VOT2016 and a success AUC of 65.0% on OTB-2015, against 70.0% for ECO (Table 3, Fig. 5).

Best Practices

• Taper and pad the window. Multiply every window by a cosine window, and search a window larger than the target (twice its size in DSST); without spatial regularization, a much larger window brings background into the positive samples.
• Use multi-channel features. HOG, optionally with Colour Names, is the cheapest large gain over raw pixels in the sources above.
• Choose the learning rate for the sequence. It trades adaptation to appearance change against absorbing occluders and background. Published trackers use 0.025 (DSST, SRDCF); in the example, MOSSE at 0.025 lost the target soonest after a 60-frame appearance change, while at higher rates the PSR stayed high through the occlusion, consistent with the model absorbing the occluder.
• Estimate scale separately. A one-dimensional scale filter after translation, as in DSST, was both more accurate and much faster than a joint search in its authors’ comparison.
• Evaluate with the benchmark’s protocol. Report precision at 20 pixels and success AUC with OTB’s one-pass evaluation, or VOT’s accuracy, robustness and EAO, and compare only numbers obtained under the same protocol and dataset version.

Pitfalls

• Trusting the PSR as a correctness signal. The PSR measures how sharply the window matches the current model. Once a tracker has learned an occluder or the background, the PSR can be high while it tracks the wrong thing, as in the example; a gate on the PSR also blocks updates during genuine appearance change.
• Drift through occlusion. Every update with a wrong or occluded window moves the model toward what was there. Without a motion model or re-detection, a target that reappears outside the search window is lost for good.
• Ignoring the boundary effect. Scores far from the window centre come from wrapped-around content; enlarging the window without a spatial penalty makes results worse, as SRDCF’s 50.1% against 71.1% mean overlap precision shows.
• Mixing conventions. Correlation and convolution differ by a conjugate in the Fourier domain and by the sign of the recovered shift. Checking the closed form against an explicit solution on a small patch, as the example does, catches such errors.
• Comparing speeds across papers. Frame rates depend on hardware, language and whether feature extraction is counted. ECO’s ablation reports CPU speed without feature extraction, and VOT normalizes speed to equivalent filter operations (EFO).

Evidence and Impact

Benchmarks. OTB-2013 (Wu, Lim and Yang, 2013) has 50 annotated sequences, of which 28 carry the scale-variation attribute used by DSST, and evaluated 29 trackers with code. Its one-pass evaluation (OPE) initializes the tracker from the ground truth in the first frame and runs it to the end; temporal and spatial robustness evaluations (TRE, SRE) restart it from 20 segments of each sequence or from 12 perturbed initial boxes. The precision plot gives the fraction of frames whose centre error is within a threshold, summarized at 20 pixels, and the success plot gives the fraction of frames whose intersection over union with the ground truth exceeds a threshold, summarized by the area under the curve. OTB-2015 extends the set to 100 videos. VOT measures accuracy (overlap while tracking) and robustness (failures), restarting the tracker five frames after each failure in the 2015 protocol, and ranks trackers by expected average overlap; Object Tracking summarizes how the protocol changed later.

Results as reported. In OTB-2013, CSK was the fastest of the ten best-ranked trackers (Wu, Lim and Yang, 2013). In SRDCF’s comparison on OTB-2013, success AUCs were 51.8% for KCF, 56.0% for DSST, 57.7% for SAMF and 63.3% for SRDCF; on OTB-2015, 47.9%, 52.3%, 54.8% and 60.5% (SRDCF, Fig. 4). In ECO’s comparison on OTB-2015, SRDCF scored 60.5%, DeepSRDCF 64.3%, C-COT 69.0%, ECO 70.0% and SiameseFC, an early Siamese tracker, 57.5% (ECO, Fig. 5b). On VOT2014, SRDCF reported 0.63 mean overlap and 15.90 failures against 0.64 and 16.90 for DSST, the best of the challenge’s participants in that comparison (SRDCF, Table 3).

In VOT2015, 62 trackers took part, and the report lists many correlation filter trackers among the regression-based entries. Its winner was MDNet, a CNN trained offline on tracking sequences outside the VOT set; correlation filters took second and fourth place by EAO, DeepSRDCF with 0.32 and SRDCF with 0.29, against 0.38 for MDNet. DSST, the 2014 winner, scored 0.17 (Kristan et al., 2015, Table 1). In VOT2016, C-COT, a correlation filter on deep features, ranked first, and ECO later improved on it (ECO, Sec. 2 and Table 3).

Speed against accuracy. The arc of the family is a steady trade of speed for accuracy. DSST reports that MOSSE reached hundreds of frames per second, and ECO’s authors estimate that MOSSE is about 1,000 times faster than C-COT with about half its accuracy on VOT2016. ECO’s contribution was to cut that cost: a 20-fold speed-up over C-COT with a higher EAO, and a hand-crafted variant that runs in real time on a CPU. In the same paper the authors cite Siamese networks that track well without any model update as a reason to update less often, and in the following years the offline-trained Siamese and transformer trackers described in Object Tracking took over the benchmarks.