Single Object Tracking

Single object tracking follows one target through a video when the only thing known about it is a bounding box drawn in the first frame. The target can be anything, a person, a toy, a bird or a part of an object, and its category is not given. The tracker must report where the target is in every later frame, using only the frames seen so far. The task is also called visual object tracking, the name the VOT challenge uses, model-free tracking, and generic object tracking, the name of the GOT-10k benchmark. Because no detector for the target exists in advance, the problem is to learn what the target looks like from one example and to keep finding it while its appearance, the background and the lighting change.

Object tracking gives the overview of tracking and its families of methods. This article covers the single-object task: its definition, the evaluation protocols that decide how trackers are ranked, the main benchmarks, and transformer trackers in outline. The worked example computes the measures as the official toolkits define them and shows how the protocol’s choices move the scores.

Problem Definition

A tracker receives a video $I_1, \dots, I_T$ and a box $g_1$ around the target in the first frame, and outputs a box $b_t$ for every frame $t = 2, \dots, T$. Three assumptions, stated by the VOT challenge for the trackers it evaluates, make the task precise:

• Model-free. In the VOT2020 report’s definition, the only training information about the target is the box in the first frame. This does not forbid learning: modern trackers learn general matching functions offline from large video datasets. It forbids a model of the specific target or its class. VOT2020 did not allow trackers to identify the target’s class and apply a class-specific tracker, and banned learning on the evaluation datasets.
• Causal. The tracker may not use future frames, nor, after a re-initialization, frames before it.
• Single camera, single target. One target per sequence, given once. When several objects of known classes must be followed with identities, the task is multi-object tracking.

The first box is the whole specification, so how boxes are drawn matters. VOT2013 found that earlier annotations had been placed so that at least about 60% of the pixels in the box belonged to the target, and VOT later re-annotated its sequences with rotated boxes and, from 2020, segmentation masks. LaSOT’s annotators drew the tightest upright box around any visible part of the target in every frame, and marked frames where it was out of view or fully occluded as absent.

Short-term and long-term tracking

A short-term tracker is assumed unable to find the target again once it has lost it; a long-term tracker must re-detect it after it leaves the view or is hidden, and must report that it is absent in the meantime. VOT added a long-term challenge in 2018, adopting the categories of Lukežič and colleagues that place trackers on a spectrum (VOT2018 and VOT2020 reports):

• ST0, a short-term tracker that reports a position in every frame and neither detects occlusion nor re-detects;
• ST1, a short-term tracker that updates its model conservatively, depending on a confidence estimate;
• LT0, a pseudo long-term tracker that does not report the target when it believes it is not visible, but has no explicit re-detection;
• LT1, a re-detecting long-term tracker.

The distinction is about behaviour and output, not only about length. The VOT2020 report cites the argument that long-term tracking is defined by the number and length of the target’s disappearances and the kind of output expected, not by sequence length alone. Papers differ here: LaSOT’s own paper, STARK and OSTrack call LaSOT, whose sequences average about 2,500 frames, a long-term benchmark, while it is scored with the same one-pass protocol as short-term benchmarks.

Input and Output

The input is an RGB video, read frame by frame, and the initial box. Most benchmarks use an axis-aligned box $(x, y, w, h)$; GOT-10k’s submission format gives the top-left corner, width and height with 1-based coordinates. VOT’s short-term datasets used rotated boxes up to 2019 (VOT2015 re-annotated all its sequences with them) and masks from 2020. The output is one box (or mask) per frame. A long-term tracker also outputs a confidence score in each frame, which the long-term measures threshold to decide whether the tracker claims the target is present. OTB-style toolkits (LaSOT’s MATLAB code, pytracking, and GOT-10k’s toolkit for OTB and LaSOT) replace the tracker’s output for the first frame with the ground truth before scoring, and GOT-10k’s own evaluation leaves the first frame out.

Variants

• Short-term and long-term, as above. LaSOT, GOT-10k and TrackingNet are scored by one-pass evaluation without resets; VOT’s long-term challenges use their own measures.
• Real-time. VOT introduced a real-time challenge in 2017 (VOT2020 report). TrackingNet’s authors ran a related experiment in which frames are skipped when a tracker is slower than real time; most trackers slower than real time lost accuracy, Struck by as much as 50%, while CFNet did not degrade although it saw only every third frame.
• Masks instead of boxes. From 2020 VOT’s short-term ground truth is a segmentation mask, and 57% of the VOT-ST2020 entries reported masks. The task then meets semi-supervised video object segmentation: the VOT committee added the segmentation method STM as a baseline, and it ranked 19th, ahead of the box-based trackers ATOM and DiMP.
• Other sensors. VOT has run thermal (from 2015), RGB-thermal and RGB-depth challenges.
• Language. LaSOT gives each sequence a natural-language description of the target, so a tracker can be specified by text as well as by a box.
• Aerial video. UAV123 is a benchmark of sequences captured from drones; its paper could not be read for this article, and MixFormer’s description of it is given under Datasets and Benchmarks.

Challenges

Object tracking lists the general difficulties: occlusion, appearance change, fast motion, distractors and drift. OTB made them measurable by annotating each sequence with 11 attributes, three of them with explicit thresholds: scale variation when the box’s size ratio to the first frame leaves $[1/2, 2]$, fast motion when the target moves more than 20 pixels between frames, and low resolution when the box holds fewer than 400 pixels (Wu, Lim and Yang, 2013, Table 2). LaSOT annotates 14 attributes, including full occlusion and out-of-view.

Three difficulties are specific to the single-object task. The target is known from one example, so any appearance model starts from a single view and must be updated from the tracker’s own, possibly wrong, estimates; this is how drift starts. There is no category prior, so the tracker cannot fall back on a detector for the class. And the evaluation itself is hard: a tracker that loses the target once is scored zero for the rest of the sequence by OTB-style protocols, so scores mix how accurately a tracker localizes with how often it fails, and the choice of protocol changes the ranking, as the example below shows.

Approaches

Appearance models learned online

Until the mid-2010s, trackers built their model of the target during tracking; OTB-2013 evaluated 29 of them, run with their authors’ code and default parameters. Template trackers matched a stored patch; mean shift tracking climbs to the window whose colour histogram best matches the target’s; sparse-representation trackers such as SCM, ASLA and L1APG reconstructed candidates from templates; discriminative trackers trained a classifier online to separate target from background, as in Struck and in TLD, which tracking-by-detection describes; and correlation filters trained a linear filter over all shifts of a window in the Fourier domain. In OTB-2013, SCM ranked first by one-pass success AUC but fell below Struck under spatially perturbed initialization, and CSK, a correlation filter, was the fastest of the ten best. Correlation filters went on to win VOT2014 and VOT2016, as their article describes.

Matching learned offline

The next generation learned from large video datasets before tracking. MDNet, a CNN fine-tuned online on each sequence, was the best tracker by precision and success in the LaSOT paper’s protocol I evaluation (0.374 and 0.413 on all 1,400 sequences; VITAL led on normalized precision) and in TrackingNet’s benchmark, at a high computational cost (TrackingNet lists it at 0.625 frames per second). Siamese trackers, popularized by SiamFC, learn a similarity function between a template and a search region offline and apply it with no online training; on all 1,400 LaSOT sequences (protocol I), SiamFC reached a success score of 0.358 in real time, and the best correlation filter, ECO, 0.34 (Fan et al., 2019). Their article describes the design.

Transformer trackers

From 2021, trackers replaced correlation between template and search features with attention. Four designs show the progression, in outline:

• TransT (Chen et al., 2021) keeps a Siamese ResNet-50 backbone, with a $128 \times 128$ template and a $256 \times 256$ search region, and replaces correlation with a fusion network: ego-context augment modules based on self-attention and cross-feature augment modules based on cross-attention, with a fusion layer repeated $N = 4$ times, followed by a classification head and a box-regression head trained with GIoU and L1 losses. Its authors argue that correlation is a local linear matching that loses semantic information. It ran at about 50 frames per second on a GPU.
• STARK (Yan et al., 2021) feeds the flattened features of the template and the search region into a transformer encoder–decoder, predicts the box corners as the expectation of two corner heatmaps, and adds a dynamic template: a second template cropped from a recent frame, replaced every $T_u = 200$ frames if a score head judges the current state reliable (score above $\tau = 0.5$). The score head is trained in a second stage, after localization. No cosine window or box smoothing is used.
• OSTrack (Ye et al., 2022) is a one-stream design: template and search patches are concatenated and processed by a single vision transformer (ViT-Base pretrained with MAE), so feature extraction and template–search relation modelling happen in the same layers. Early candidate elimination drops search-region tokens that receive little attention from the template tokens, treating them as background, and keeps 70% of the candidates at each of layers 4, 7 and 10. OSTrack-256 ran at 105.4 frames per second on an RTX 2080 Ti.
• MixFormer (Cui et al., 2022) builds its backbone from mixed attention modules that extract features and integrate target information together, on a CvT backbone pretrained on ImageNet, with asymmetric attention that prunes the target-to-search cross-attention, and a score prediction module that picks which online template to keep. Its ablation shows why the score matters: adding online templates without the score lowered LaSOT AUC from 68.1 to 66.6, and with the score raised it to 69.2 (Table 6). Updating the template from the tracker’s own output is the drift problem in miniature.

Results as reported

The table collects results from the four papers. LaSOT figures are on its 280-sequence test set (protocol II); TrackingNet figures are from its evaluation server on 511 test sequences; GOT-10k figures are from its server on 180 test sequences; for the four transformer trackers they are for models trained on GOT-10k’s training set alone, as the benchmark’s one-shot protocol requires. SiamFC, ECO and MDNet are listed as in TransT’s Table 1, which, like MixFormer’s, does not state the training data behind their GOT-10k entries. All numbers are percentages.

Tracker	LaSOT AUC	LaSOT $P_{\text{norm}}$	LaSOT P	TrackingNet AUC	TrackingNet $P_{\text{norm}}$	GOT-10k AO (one-shot)
SiamFC	33.6	42.0	33.9	57.1	66.3	34.8
ECO	32.4	33.8	30.1	55.4	61.8	31.6
MDNet	39.7	46.0	37.3	60.6	70.5	29.9
TransT	64.9	73.8	69.0	81.4	86.7	67.1 (TransT-GOT)
STARK-ST101	67.1	77.0	–	82.0	86.9	68.8
OSTrack-384	71.1	81.1	77.6	83.9	88.5	73.7
MixFormer-L	70.1	79.9	76.3	83.9	88.9	71.2 (MixFormer-1k, GOT-10k only)

The GOT-10k column shows why the training data must be read with the score. TransT’s Table 1 reports an AO of 72.3 for the model trained on COCO, TrackingNet, LaSOT and GOT-10k, and 67.1 for TransT-GOT, trained on GOT-10k alone. MixFormer’s Table 3 lists 75.6 for MixFormer-L trained on all datasets, underlined as not comparable, and 71.2 and 70.7 for the 1k and 22k models trained on GOT-10k alone. STARK and OSTrack state that their GOT-10k models were retrained on its training set only. In VOT2020’s anchor-based protocol, STARK-ST50 reported an EAO of 0.308 with boxes and 0.505 with the Alpha-Refine segmentation module, and MixFormer-L 0.555; the challenge’s winner, RPT, had scored 0.530 on the public dataset (VOT2020 report, Table 8).

Datasets and Benchmarks

Benchmark	Sequences	Frames	Classes	Protocol	Main measures
OTB-2013	50 (51 targets)	over 29,000	not defined	test only, OPE, TRE, SRE	success AUC, precision at 20 px
OTB-2015 (TB-100)	100 targets	58,897	not defined	test only; TB-50 subset	as OTB-2013, plus OPER, SRER
VOT short-term	16 (2013), 60 (2015–2020)	21,356 (VOT2017)	not defined	reset-based to 2019, anchor-based from 2020; sequestered set from 2017	accuracy, robustness, EAO
VOT long-term	35 (LTB35, 2018), 50 (LTB50, 2019–2020)	14,687 (LTB35)	not defined	no resets	tracking precision, recall, F-score
LaSOT	1,400	3.52 million	70	protocol I: all; protocol II: 1,120 train, 280 test	success AUC, precision, normalized precision
GOT-10k	over 10,000 (9,335 train, 180 val, 180 test)	over 1.5 million boxes	over 560	one-shot: test classes not in training; train only on GOT-10k	AO, SR at 0.5 and 0.75
TrackingNet	30,643 (30,132 train, 511 test)	14,431,266	27	sequestered test set, server	success AUC, precision, normalized precision
UAV123	123	about 112,600	–	OPE	success AUC, precision

OTB. Wu, Lim and Yang collected 50 commonly used sequences, annotated two targets in one of them, and evaluated 29 trackers with their default parameters; the TPAMI version extends the set to 100 targets (TB-100) and selects 50 difficult ones (TB-50), with 36 body and 26 face or head sequences among the hundred. Every frame is annotated, even when the target is fully occluded, and a tracker that returns no box (TLD after it declares the target lost) is scored with its last position (Wu, Lim and Yang, 2015, Sec. 6). Later papers call the sets OTB-100 or OTB-2015, and OTB-50, which is ambiguous: TrackingNet’s comparison table uses OTB50 for the 2013 set of 51 targets, not for TB-50.

VOT. The challenge has run every year since its first edition at ICCV 2013, which compared 27 trackers on 16 sequences labelled per frame with five visual attributes (occlusion, illumination change, motion change, size change and camera motion). Its datasets are kept small and selected for diversity: VOT2015 started from a pool of 443 sequences (356 after removing duplicates, grayscale sequences and targets under 400 pixels), clustered them by visual attributes and picked 60, annotated with rotated boxes, and from VOT2017 a sequestered set of 60 more sequences, re-run by the organizers on the best submissions, limited tuning to the public data (VOT2018 and VOT2020 reports). TrackingNet’s comparison table lists VOT2017 with 21,356 annotated frames. For the VOT2019 and VOT2020 dataset updates, 1,000 sequences of GOT-10k’s training set were used and listed as prohibited for training.

LaSOT. Fan and colleagues gathered 1,400 YouTube videos under a Creative Commons licence, 20 per class for 70 classes, averaging 2,512 frames (84 seconds at 30 frames per second) by its text, 2,506 by its Table 1, and up to 11,397 frames. Protocol I evaluates on all 1,400; protocol II trains on 16 videos per class (1,120 videos, 2.83 million frames) and tests on the other 4 (280 videos, 690,000 frames), which is the split later papers report. The 2021 journal version adds an extension subset of 150 videos in 15 new classes for one-shot evaluation, according to the toolkit’s README.

GOT-10k. Its official site describes over 10,000 video segments of real-world moving objects with over 1.5 million hand-drawn boxes, backboned by WordNet, with more than 560 object classes and more than 80 motion classes, and two rules that set it apart. Under the one-shot rule, the object classes of the training and test sets do not overlap: the test set’s 180 segments cover 84 object classes and 32 motion classes. Under the unified training data rule, every submission may learn only from GOT-10k’s training set. Each frame also carries a visible-ratio label in nine levels from 0% to 100%. Test annotations are withheld; results are uploaded to a server that allows a method at most four submissions, at least 48 hours apart, and the leaderboard ranks entries by AO. The dataset is licensed under CC BY-NC-SA 4.0.

TrackingNet. Müller and colleagues built the training set from YouTube-BoundingBoxes, whose boxes are given once per second: 30,132 videos, split into 12 subsets that keep the class distribution, densified to every frame with a correlation filter tracker run forward and backward between the annotated frames; Section 3.1 names the DCF tracker, Section 4 STAPLE_CA. The 511 test videos are new, with Creative Commons licences, annotated in every frame, and scored only by an online server. The best tracker in their benchmark reached a success rate of about 60%, against about 90% on OTB, in their comparison.

UAV123. The paper could not be read for this article. Pytracking’s sequence list has 123 sequences, which sum to about 112,600 frames by our count; LaSOT’s comparison table and MixFormer give 123 sequences averaging 915 frames, and MixFormer adds that they were captured from low-altitude UAVs. TrackingNet’s comparison table lists UAV123 with 91 videos and 113,476 annotations, a different count of the same data.

Evaluation Metrics

One-pass evaluation

OTB’s one-pass evaluation (OPE) initializes the tracker with the ground truth in the first frame and lets it run to the end with no intervention (Wu, Lim and Yang, 2013). Two curves summarize the run.

The success plot uses the intersection over union (the papers call it the overlap score) $o_t = |b_t \cap g_t| / |b_t \cup g_t|$ of predicted and ground-truth boxes. For a threshold $\tau \in [0, 1]$, the success rate is the fraction of frames whose overlap exceeds it,

$$S(\tau) = \frac{1}{N} \sum_{t=1}^{N} \mathbb{1}\left[ o_t > \tau \right],$$

and trackers are ranked by the area under this curve, which OTB’s authors preferred to the success rate at a single threshold such as 0.5. In practice the area is the mean of $S$ at sampled thresholds: LaSOT’s MATLAB toolkit, pytracking and the OTB experiment of GOT-10k’s toolkit sample 21 thresholds, $0, 0.05, \dots, 1$, and GOT-10k’s own evaluation 101. The OTB authors’ MATLAB toolkit could not be read for this article; its web page now returns an error.

The precision plot uses the centre location error $e_t = \lVert c(b_t) - c(g_t) \rVert$. Its average over a sequence is a poor measure, because the error after a failure is arbitrary, so OTB plots the fraction of frames with $e_t \le \delta$ over a range of thresholds (0 to 50 pixels in the toolkits) and ranks by the value at 20 pixels. The TPAMI version notes that the centre error ignores the target’s size and scale, and moves the precision plots to its supplement.

Normalized precision, introduced with TrackingNet, divides the centre offset by the ground-truth box size, so that the threshold no longer depends on resolution and target size:

$$\tilde{e}_t = \left\lVert \left( \frac{c_x(b_t) - c_x(g_t)}{w^{g}_t}, \frac{c_y(b_t) - c_y(g_t)}{h^{g}_t} \right) \right\rVert .$$

TrackingNet’s Eq. 1 is printed as $\lVert W (C^{tr} - C^{gt}) \rVert_2$ with $W = \mathrm{diag}(BB^{gt}_x, BB^{gt}_y)$, which reads as multiplication by the box size, while its text says the precision is normalized by the box size, as the toolkits implement it. Both TrackingNet and LaSOT say trackers are ranked by the area under the normalized precision curve between 0 and 0.5. The code differs: GOT-10k’s toolkit computes that area, while pytracking and LaSOT’s own MATLAB toolkit (by default) report the value at a normalized distance of 0.2. Tracker papers’ tables label the column $P_{\text{norm}}$ without saying which.

AUC and average overlap. OTB’s TPAMI paper states that with enough uniformly sampled thresholds the success AUC of a sequence equals its average overlap, and the VOT2020 report cites the same result; GOT-10k ranks directly by the average overlap (AO), the mean overlap over frames, with success rates $SR_{0.5}$ and $SR_{0.75}$ at two thresholds. The equality is exact in the continuous limit, since $\int_0^1 \mathbb{1}[o > \tau]\, d\tau = o$. With $K$ thresholds including $\tau = 1$, where $S(1) = 0$ because no overlap exceeds 1, it follows that

$$\frac{K - 1}{K}\, \mathrm{AO} \;\le\; \mathrm{AUC}_K \;\le\; \frac{1 + (K - 1)\, \mathrm{AO}}{K},$$

(our derivation: $\mathrm{AUC}_K$ is $(K-1)/K$ times a left Riemann sum of the non-increasing $S$, which is at least the integral). A perfect tracker therefore scores $20/21 \approx 0.952$ with 21 thresholds and $100/101 \approx 0.990$ with 101, and a tracker with high overlaps has a 21-threshold AUC slightly below its AO. Comparisons are unaffected within one toolkit, but numbers from toolkits with different threshold sets are not interchangeable.

What the toolkits do with the details. Reading the code shows choices that the papers leave implicit:

• Averaging. OTB-style toolkits (LaSOT’s MATLAB code, pytracking, GOT-10k’s toolkit for OTB and LaSOT) compute curves per sequence and average them over sequences, so each sequence weighs the same. GOT-10k’s AO pools the overlaps of all frames of all sequences, so long sequences weigh more; its site defines AO as the average over all frames.
• Absent targets. OTB annotates hidden targets and scores those frames like any other. LaSOT’s MATLAB toolkit removes frames marked absent from the count of successes but keeps them in the denominator (it divides by the full sequence length), and pytracking does the same by default, so absent frames count as failures and even a perfect tracker scores below 1 on a sequence with absent frames. GOT-10k’s toolkit drops frames whose visible ratio is zero, and the first frame.
• Box conventions. LaSOT’s MATLAB code and pytracking treat boxes as inclusive pixel ranges (the right edge is $x + w - 1$ and the centre $x + (w-1)/2$); GOT-10k’s toolkit computes overlaps with continuous boxes.
• Repetitions. GOT-10k’s format expects three runs of a stochastic tracker per sequence, and its toolkit pools them; VOT ran stochastic trackers 15 times.
• Strict and non-strict thresholds. Success counts $o_t > \tau$ while precision counts $e_t \le \delta$, in every toolkit read.

Robustness to initialization: TRE and SRE

OPE initializes once, from a perfect box, so OTB added two variants (Wu, Lim and Yang, 2013, Sec. 4). Temporal robustness evaluation (TRE) splits each sequence into 20 segments and starts the tracker from the ground truth at the beginning of each, running to the end; the results of all runs are tallied. Spatial robustness evaluation (SRE) starts from 12 perturbed boxes: eight shifts by 10% of the target size, four of the centre and four of the corners, and four scalings to 0.8, 0.9, 1.1 and 1.2 times the true box. The motivation is practical: a tracker initialized by a detector starts from an imprecise box, and may be restarted at any time. In OTB-2013, average TRE scores were higher than OPE scores, which the authors attribute to the later segments being shorter, and SRE scores lower, because an imprecise first box makes trackers update with imprecise appearance and drift. The TPAMI version adds restart variants (OPER and SRER), in which a tracker is restarted when the average overlap over a moving window falls below a threshold, approximated from runs started at regular intervals, and treats SRE and SRER as the most representative measures.

VOT’s reset-based protocol

VOT took the opposite view, modelled on a user who re-initializes a tracker whenever it fails (Kristan et al., 2013). In its protocol up to 2019, a failure is declared when the overlap drops to zero, and the tracker is re-initialized five frames later, to avoid immediately failing again on the same frames. Two weakly correlated measures summarize the run:

• Accuracy, the average overlap during successful tracking, ignoring the first ten frames after each initialization because overlaps right after a perfect initialization are biased upwards (VOT2013 found the burn-in to last about ten frames);
• Robustness, the number of failures (a failure rate).

VOT2013 and VOT2014 ranked trackers on each measure per attribute, merged trackers whose differences were not statistically significant (Wilcoxon tests) or, from 2014, not larger than annotation noise, and averaged the ranks. VOT2015 replaced the averaged rank, which has no interpretation in terms of an application, with the expected average overlap (EAO). Consider sequences of length $N_s$ on which a tracker that fails stays failed: the average overlap over such a sequence, with zeros after the failure, is

$$\Phi_{N_s} = \frac{1}{N_s} \sum_{i=1}^{N_s} \Phi_i .$$

Its expectation $\hat\Phi_{N_s}$ over many sequences, plotted against $N_s$, is the expected average overlap curve, and EAO averages it over a range of typical lengths,

$$\hat\Phi = \frac{1}{N_{hi} - N_{lo}} \sum_{N_s = N_{lo}}^{N_{hi}} \hat\Phi_{N_s} .$$

The reset protocol supplies the samples: each tracking segment that ends in a failure is padded with zeros to length $N_s$, segments that end without failure are trimmed (and dropped if shorter than $N_s$), and the average over segments estimates $\hat\Phi_{N_s}$. The range $[N_{lo}, N_{hi}]$ is taken around the mode of a kernel density estimate of the dataset’s sequence lengths, so that it holds half of the probability mass: $[108, 371]$ for VOT2015’s 60 sequences. The report notes that $\hat\Phi_{N_s}$ is in principle OTB’s measure computed on $N_s$-frame sequences, but that estimating it directly would need a very large dataset because of the estimator’s variance. Speed is reported in equivalent filter operations (EFO), normalized by a filtering operation the toolkit times on each machine.

VOT2020: anchors and masks

By 2019 trackers had become robust enough that the reset points themselves mattered: where a tracker failed decided where it restarted, and a restart a few frames later might avoid a later failure, which could in principle be tuned for. A tracker could also avoid zero overlap by inflating its box; VOT disqualified one tracker for doing so (VOT2020 report, Sec. 1.1). VOT2020 replaced resets with anchors, initialization points shared by all trackers and placed every 50 frames (about 2 seconds at 25 frames per second), each manually checked to avoid an occluded target. From each anchor the tracker runs forward or backward, whichever gives the longer sub-sequence. A failure is now declared when the overlap stays below 0.1 for 10 frames, so a tracker can recover from a brief failure and an oversized box still fails; frames with a fully occluded target are ignored when detecting failures. On the sub-sequence from anchor $a$ of sequence $s$, with $N^F_{s,a}$ frames before the failure out of $N_{s,a}$,

$$A_{s,a} = \frac{1}{N^F_{s,a}} \sum_{i=1}^{N^F_{s,a}} \Omega_{s,a}(i), \qquad R_{s,a} = \frac{N^F_{s,a}}{N_{s,a}},$$

and both are averaged over anchors and sequences weighted by the frames they cover. EAO is computed as before on virtual sequences built from the sub-sequences: a sub-sequence that ended in a failure is extended with zeros, one that did not cannot be extended. The interval bounds were set from the mean plus or minus one standard deviation of the sub-sequence lengths, $[N_{lo}, N_{hi}] = [115, 755]$ on the public VOT-ST2020 data. The ground truth became segmentation masks, initialized semi-automatically and corrected frame by frame. The VOT toolkit’s code for this analysis uses the same defaults, a threshold of 0.1 and a grace period of 10 frames, and counts a frame as failing when the overlap is at most the threshold.

Long-term measures

For long-term trackers VOT adopted the tracking precision, recall and F-score of Lukežič and colleagues (VOT2018 report, Sec. 3.3). With $\theta_t$ the tracker’s confidence and $\tau_\theta$ a threshold, the prediction $A_t(\tau_\theta)$ counts only when $\theta_t \ge \tau_\theta$; with $N_p$ frames that have a prediction and $N_g$ frames in which the target is present,

$$Pr(\tau_\theta) = \frac{1}{N_p} \sum_{t:\, A_t \ne \emptyset} \Omega(A_t, G_t), \qquad Re(\tau_\theta) = \frac{1}{N_g} \sum_{t:\, G_t \ne \emptyset} \Omega(A_t, G_t),$$

and the F-measure $2\,Pr\,Re/(Pr + Re)$ is maximized over $\tau_\theta$ to give the F-score. Overlaps are averaged rather than thresholded at 0.5, so the measures need no overlap threshold, and in a short-term setting the F-measure reduces to the average overlap. VOT2018’s first long-term dataset, LTB35, holds 35 sequences and 14,687 frames with 433 disappearances, on average 12 per sequence lasting about 40 frames each.

Example: the one-pass protocol on synthetic sequences

The example implements the measures as the toolkits define them and applies them to deliberately simple trackers on synthetic sequences with known ground truth, so that what moves the scores is the protocol, not the trackers’ cleverness. Each of six grayscale sequences of $240 \times 240$ pixels shows a target cut from a scikit-image photograph (chelsea, coffee, rocket, camera, coins, astronaut), resized to its current box, moving along a smooth Lissajous-like path over a background cut from gravel, with Gaussian noise of standard deviation 4 grey levels. The sequences are 120 to 400 frames long and the targets 12 to 56 pixels wide; two targets grow and shrink by 30%. In three sequences a full-height bar of moon texture sweeps across the frame against the target’s motion and hides it completely for 8 frames. The ground truth is the box in every frame, including hidden frames, as OTB annotates, and the visible fraction of the target, as GOT-10k labels it. Before the first run, a check confirmed that a first-frame template matched by normalized cross-correlation peaks within 2 pixels of the true position in every checked visible frame at the original size, and that a box that never moves overlaps the target with an IoU above 0.5 in at most 3% of frames, so the sequences are learnable but not trivial.

Five trackers run on them:

• oracle outputs the ground truth, as a check of the measures;
• static keeps the initial box;
• TM matches the first-frame template by normalized cross-correlation in a window around its last position, with a fixed box size;
• TM+CV adds a constant-velocity prediction (a motion model): it searches around the predicted position and, when the correlation peak is below 0.5, keeps the prediction instead of the match;
• drift replaces its template with the matched patch in every frame, the naive update that lets errors accumulate.

The measures follow LaSOT’s MATLAB toolkit and pytracking: inclusive pixel boxes, success on 21 thresholds with $o_t > \tau$, precision on 0 to 50 pixels with $e_t \le \delta$, normalized precision at 0.2 and as the mean over $[0, 0.5]$, first output replaced by the ground truth, and curves averaged over sequences. The 101-threshold AUC is GOT-10k’s. Three choices that the sources leave open are printed under each option: hidden frames scored against the annotated box (OTB), counted as failures (LaSOT toolkit, pytracking) or dropped (GOT-10k); averaging over sequences or pooling frames (GOT-10k’s AO); and, for TRE, pooling the frames of all segments or averaging the segments’ scores. TRE starts at 20 evenly spaced frames, moved forward to the next frame where the target is not fully hidden; SRE uses the 12 boxes above, with corner shifts read as diagonal shifts of the box. A last block runs the VOT2013/2015 reset protocol (failure at zero overlap, restart five frames later, ten burn-in frames ignored) and reports accuracy and failures; EAO needs many more sequences and is left out. Versions: Python 3.13, NumPy 2.3.5, scikit-image 0.25.2. The script is deterministic and runs in about 100 seconds, mostly in the TRE and SRE runs.

import numpy as np
from skimage import data, transform
from skimage.feature import match_template

H = W = 240
gray = lambda im: (im.mean(axis=2) if im.ndim == 3 else im).astype(float)
SRC = {"chelsea": (data.chelsea, 40, 140), "coffee": (data.coffee, 100, 200),
       "rocket": (data.rocket, 50, 250), "camera": (data.camera, 60, 160),
       "coins": (data.coins, 50, 100), "astronaut": (data.astronaut, 0, 150)}
# target image, length, base (w, h), scale amplitude, occluded
SEQS = [("chelsea", 120, (12, 12), 0.0, False), ("coffee", 160, (20, 16), 0.0, True),
        ("rocket", 200, (28, 36), 0.3, False), ("camera", 260, (36, 36), 0.0, True),
        ("coins", 320, (44, 32), 0.3, False), ("astronaut", 400, (56, 48), 0.0, True)]

def make_sequence(i):
    """Frames, ground-truth boxes (x, y, w, h; pixel-inclusive) and visible fraction."""
    name, n, (w0, h0), amp, occluded = SEQS[i]
    rng = np.random.default_rng(i)
    fn, r0, c0 = SRC[name]
    patch = gray(fn())[r0:r0 + 200, c0:c0 + 200]
    oy, ox = rng.integers(0, 512 - H, 2)
    bg = gray(data.gravel())[oy:oy + H, ox:ox + W]
    occluder = gray(data.moon())[100:100 + H, 100:100 + W]
    t = np.arange(n)
    s = 1 + amp * np.sin(2 * np.pi * t / rng.uniform(90, 150))
    w = np.round(w0 * s).astype(int); h = np.round(h0 * s).astype(int)
    ax, ay = (W - w.max()) / 2 - 4, (H - h.max()) / 2 - 4
    px, py = rng.uniform(120, 200, 2); fx, fy = rng.uniform(0, 2 * np.pi, 2)
    cx = W / 2 + ax * np.sin(2 * np.pi * t / px + fx)
    cy = H / 2 + ay * np.sin(2 * np.pi * t / py + fy)
    gt = np.stack([np.round(cx - (w - 1) / 2), np.round(cy - (h - 1) / 2), w, h], 1).astype(int)
    bar = None
    if occluded:   # a full-height bar sweeps across the frame, centred on the target at 60% of the sequence
        k = int(0.6 * n)
        vx = cx[k + 1] - cx[k]
        u = -6.0 if vx > 0 else 6.0                       # moves against the target
        B = int(w[k] + 8 * abs(u - vx))                   # hides the target for about 8 frames
        bar = (cx[k] - B / 2 - u * k, u, B)               # left edge at frame 0, speed, width
    vis = np.ones(n)
    frames = np.empty((n, H, W), np.float32)
    for f in range(n):
        x, y, bw, bh = gt[f]
        im = bg.copy()
        im[y:y + bh, x:x + bw] = transform.resize(patch, (bh, bw), anti_aliasing=True, preserve_range=True)
        if bar is not None:
            b0 = int(round(bar[0] + bar[1] * f)); b1 = b0 + bar[2]
            lo, hi = max(b0, 0), min(b1, W)
            if lo < hi:
                im[:, lo:hi] = occluder[:, lo:hi]
                vis[f] = 1 - max(0, min(hi, x + bw) - max(lo, x)) / bw
        frames[f] = im + rng.normal(0, 4, (H, W))
    return frames, gt, vis

class Template:
    """NCC template matcher. coast: constant-velocity prediction; drift: template replaced every frame."""
    def __init__(self, coast=False, drift=False):
        self.coast, self.drift = coast, drift
    def init(self, f, box):
        x, y, w, h = (int(v) for v in box)
        x, y = min(max(x, 0), W - w), min(max(y, 0), H - h)   # keep the template inside the frame
        self.box, self.v = np.array([x, y, w, h], float), np.zeros(2)
        self.tpl = f[y:y + h, x:x + w].copy()
    def update(self, f):
        x, y, w, h = self.box
        h_t, w_t = self.tpl.shape
        px, py = (x + self.v[0], y + self.v[1]) if self.coast else (x, y)
        px, py = np.clip(px, 0, W - w_t), np.clip(py, 0, H - h_t)
        m = max(16, w_t, h_t)
        x0, y0 = int(max(round(px) - m, 0)), int(max(round(py) - m, 0))
        x1, y1 = int(min(round(px) + m + w_t, W)), int(min(round(py) + m + h_t, H))
        r = match_template(f[y0:y1, x0:x1], self.tpl)
        iy, ix = np.unravel_index(np.argmax(r), r.shape)
        nx, ny = x0 + ix, y0 + iy
        if self.coast:
            if r[iy, ix] < 0.5:            # low confidence: keep the prediction
                nx, ny = px, py
            else:
                self.v = 0.7 * self.v + 0.3 * np.array([nx - x, ny - y])
        self.box = np.array([nx, ny, w, h], float)
        if self.drift:
            self.tpl = f[int(ny):int(ny) + h_t, int(nx):int(nx) + w_t].copy()
        return self.box.copy()

class Static:
    def init(self, f, box): self.box = np.array(box, float)
    def update(self, f): return self.box.copy()

def run(tracker, frames, gt, start=0, init_box=None):
    box = gt[start] if init_box is None else init_box
    if tracker == "oracle":
        return gt[start:].astype(float)
    tracker.init(frames[start], box)
    out = [np.array(box, float)]
    for f in range(start + 1, len(frames)):
        out.append(tracker.update(frames[f]))
    out = np.array(out)
    out[0] = gt[start]                     # every toolkit replaces the first output by the GT
    return out

# ---------- measures, as in the LaSOT MATLAB toolkit and pytracking (pixel-inclusive boxes) ----------
def iou(a, b):
    x1, y1 = np.maximum(a[:, 0], b[:, 0]), np.maximum(a[:, 1], b[:, 1])
    x2 = np.minimum(a[:, 0] + a[:, 2] - 1, b[:, 0] + b[:, 2] - 1)
    y2 = np.minimum(a[:, 1] + a[:, 3] - 1, b[:, 1] + b[:, 3] - 1)
    inter = np.clip(x2 - x1 + 1, 0, None) * np.clip(y2 - y1 + 1, 0, None)
    return inter / (a[:, 2] * a[:, 3] + b[:, 2] * b[:, 3] - inter)

def centre_error(a, b, normalized=False):
    ca, cb = a[:, :2] + (a[:, 2:] - 1) / 2, b[:, :2] + (b[:, 2:] - 1) / 2
    if normalized:
        ca, cb = ca / b[:, 2:], cb / b[:, 2:]
    return np.hypot(*(ca - cb).T)

T21, T101 = np.linspace(0, 1, 21), np.linspace(0, 1, 101)
TPX, TNORM = np.arange(51), np.linspace(0, 0.5, 51)

def curves(pred, gt, hidden=None, convention="otb"):
    """Success and precision curves for one sequence.
    otb: every frame scored against the annotated box (OTB annotates hidden targets too).
    lasot: hidden frames stay in the denominator and count as failures (LaSOT MATLAB, pytracking)."""
    ov, ce, nce = iou(pred, gt), centre_error(pred, gt), centre_error(pred, gt, True)
    if convention == "lasot" and hidden is not None:
        ov, ce, nce = np.where(hidden, -1, ov), np.where(hidden, np.inf, ce), np.where(hidden, np.inf, nce)
    n = len(gt)
    return {"ao": ov[ov >= 0].mean() if convention == "otb" else np.clip(ov, 0, None).mean(),
            "s21": (ov[:, None] > T21).sum(0) / n, "s101": (ov[:, None] > T101).sum(0) / n,
            "p": (ce[:, None] <= TPX).sum(0) / n, "np": (nce[:, None] <= TNORM).sum(0) / n}

def summary(c):
    return {"AO": c["ao"], "AUC21": c["s21"].mean(), "AUC101": c["s101"].mean(), "SR0.5": c["s21"][10],
            "P@20": c["p"][20], "NP@0.2": c["np"][20], "NP-AUC": c["np"].mean()}

def mean_over_sequences(cs):
    return {k: np.mean([summary(c)[k] for c in cs]) for k in summary(cs[0])}

def sre_boxes(box):
    x, y, w, h = box
    dx, dy = round(0.1 * w), round(0.1 * h)
    shifts = [(dx, 0), (-dx, 0), (0, dy), (0, -dy), (dx, dy), (dx, -dy), (-dx, dy), (-dx, -dy)]
    out = [(x + a, y + b, w, h) for a, b in shifts]
    cx, cy = x + (w - 1) / 2, y + (h - 1) / 2
    for s in (0.8, 0.9, 1.1, 1.2):
        sw, sh = round(w * s), round(h * s)
        out.append((round(cx - (sw - 1) / 2), round(cy - (sh - 1) / 2), sw, sh))
    return out

def reset_run(tracker, frames, gt, hidden):
    """VOT2013/2015-style: failure at zero overlap, re-initialize 5 frames later,
    ignore 10 frames after every initialization when averaging the overlap."""
    n, f, fails, valid = len(gt), 0, 0, []
    while f < n:
        while f < n and hidden[f]:
            f += 1
        if f >= n:
            break
        pred = run(tracker, frames, gt, start=f)
        ov = iou(pred, gt[f:])
        zero = np.flatnonzero(ov == 0)
        end = zero[0] if len(zero) else len(ov)
        valid.extend(ov[10:end])
        if not len(zero):
            break
        fails += 1
        f = f + end + 5
    return np.mean(valid) if valid else 0.0, fails

if __name__ == "__main__":
    names = ["oracle", "static", "TM", "TM+CV", "drift"]
    make = {"oracle": lambda: "oracle", "static": Static, "TM": Template,
            "TM+CV": lambda: Template(coast=True), "drift": lambda: Template(drift=True)}
    R = {k: {"otb": [], "lasot": [], "got": [], "seqgot": [], "tre_pool": [], "tre_seg": [],
             "sre": [[] for _ in range(12)], "acc": [], "fails": 0} for k in names}
    seqinfo = []
    for i in range(len(SEQS)):
        frames, gt, vis = make_sequence(i)
        hidden = vis == 0
        n = len(gt)
        seqinfo.append((SEQS[i][0], n, tuple(gt[0, 2:]), int(hidden.sum())))
        for k in names:
            pred = run(make[k](), frames, gt)
            R[k]["otb"].append(curves(pred, gt, hidden, "otb"))
            R[k]["lasot"].append(curves(pred, gt, hidden, "lasot"))
            keep = ~hidden; keep[0] = False                 # GOT-10k: no first frame, no cover = 0
            ov = iou(pred[keep], gt[keep])
            R[k]["got"].append(ov); R[k]["seqgot"].append(ov.mean())
            # TRE: 20 start frames, moved forward to the next visible frame
            starts = [int(np.argmax(~hidden[s:])) + s for s in (np.arange(20) * n // 20)]
            runs = [curves(run(make[k](), frames, gt, start=s), gt[s:]) for s in starts]
            lens = np.array([n - s for s in starts])
            pooled = {key: np.sum([r[key] * L for r, L in zip(runs, lens)], 0) / lens.sum()
                      for key in ("s21", "s101", "p", "np")}
            R[k]["tre_pool"].append(pooled["s21"].mean())
            R[k]["tre_seg"].append(np.mean([r["s21"].mean() for r in runs]))
            for j, b in enumerate(sre_boxes(gt[0])):
                R[k]["sre"][j].append(curves(run(make[k](), frames, gt, init_box=b), gt)["s21"].mean())
            acc, fails = reset_run(make[k](), frames, gt, hidden)
            R[k]["acc"].append(acc); R[k]["fails"] += fails

    print("sequences (name, frames, first box w x h, frames with the target fully hidden):")
    print("  " + "; ".join(f"{s} {n} {w}x{h} {hd}" for s, n, (w, h), hd in seqinfo))

    print("\nOPE, mean over the 6 sequences, hidden frames scored against the annotated box (OTB)")
    cols = ["AO", "AUC21", "AUC101", "SR0.5", "P@20", "NP@0.2", "NP-AUC"]
    print(f"{'tracker':8s}" + "".join(f"{c:>8s}" for c in cols))
    for k in names:
        m = mean_over_sequences(R[k]["otb"])
        print(f"{k:8s}" + "".join(f"{m[c]:8.3f}" for c in cols))

    print("\nHidden frames and averaging: success AUC21 (OTB | LaSOT-style), AO over visible frames"
          " (mean over sequences | pooled over frames, GOT-10k)")
    for k in names:
        a = mean_over_sequences(R[k]["otb"])["AUC21"]; b = mean_over_sequences(R[k]["lasot"])["AUC21"]
        print(f"{k:8s} {a:7.3f} {b:7.3f}   {np.mean(R[k]['seqgot']):7.3f} {np.concatenate(R[k]['got']).mean():7.3f}")

    print("\nPer sequence, P@20 / NP@0.2 (OTB scoring)")
    print(f"{'sequence':16s}" + "".join(f"{k:>14s}" for k in names[1:]))
    for i, (s, n, (w, h), hd) in enumerate(seqinfo):
        row = "".join(f"{summary(R[k]['otb'][i])['P@20']:8.2f} /{summary(R[k]['otb'][i])['NP@0.2']:5.2f}"
                      for k in names[1:])
        print(f"{s + f' {w}x{h}':16s}{row}")

    print("\nRobustness to initialization (success AUC21, mean over sequences)")
    print(f"{'tracker':8s}{'OPE':>8s}{'TRE pool':>10s}{'TRE seg':>9s}{'SRE':>8s}")
    for k in names:
        ope = mean_over_sequences(R[k]["otb"])["AUC21"]
        sre = np.mean([np.mean(v) for v in R[k]["sre"]])
        print(f"{k:8s}{ope:8.3f}{np.mean(R[k]['tre_pool']):10.3f}{np.mean(R[k]['tre_seg']):9.3f}{sre:8.3f}")

    print("\nReset-based (VOT2013/2015 style): accuracy (mean over sequences), failures (total)")
    for k in names:
        print(f"{k:8s} {np.mean(R[k]['acc']):6.3f} {R[k]['fails']:4d}")

    print("\nRankings (oracle excluded)")
    def rank(score):
        return " > ".join(sorted(names[1:], key=lambda k: -score(k)))
    print("  OPE AUC21 (OTB)      ", rank(lambda k: mean_over_sequences(R[k]["otb"])["AUC21"]))
    print("  OPE P@20             ", rank(lambda k: mean_over_sequences(R[k]["otb"])["P@20"]))
    print("  OPE NP@0.2           ", rank(lambda k: mean_over_sequences(R[k]["otb"])["NP@0.2"]))
    print("  AUC21 LaSOT-style    ", rank(lambda k: mean_over_sequences(R[k]["lasot"])["AUC21"]))
    print("  AO pooled (GOT-10k)  ", rank(lambda k: np.concatenate(R[k]["got"]).mean()))
    print("  TRE pooled           ", rank(lambda k: np.mean(R[k]["tre_pool"])))
    print("  SRE                  ", rank(lambda k: np.mean([np.mean(v) for v in R[k]["sre"]])))
    print("  reset: fewest fails  ", rank(lambda k: -R[k]["fails"]))

sequences (name, frames, first box w x h, frames with the target fully hidden):
  chelsea 120 12x12 0; coffee 160 20x16 8; rocket 200 28x36 0; camera 260 36x36 8; coins 320 44x32 0; astronaut 400 56x48 8

OPE, mean over the 6 sequences, hidden frames scored against the annotated box (OTB)
tracker       AO   AUC21  AUC101   SR0.5    P@20  NP@0.2  NP-AUC
oracle     1.000   0.952   0.990   1.000   1.000   1.000   1.000
static     0.024   0.025   0.024   0.018   0.050   0.011   0.016
TM         0.679   0.650   0.673   0.678   0.741   0.678   0.687
TM+CV      0.705   0.676   0.699   0.712   0.759   0.706   0.711
drift      0.598   0.575   0.593   0.583   0.696   0.565   0.593

Hidden frames and averaging: success AUC21 (OTB | LaSOT-style), AO over visible frames (mean over sequences | pooled over frames, GOT-10k)
oracle     0.952   0.936     1.000   1.000
static     0.025   0.025     0.019   0.021
TM         0.650   0.650     0.690   0.693
TM+CV      0.676   0.669     0.710   0.721
drift      0.575   0.575     0.606   0.579

Per sequence, P@20 / NP@0.2 (OTB scoring)
sequence                static            TM         TM+CV         drift
chelsea 12x12       0.05 / 0.01    1.00 / 1.00    1.00 / 1.00    1.00 / 1.00
coffee 20x16        0.11 / 0.01    0.57 / 0.57    0.57 / 0.57    0.57 / 0.57
rocket 28x36        0.04 / 0.01    0.41 / 0.32    0.40 / 0.32    0.42 / 0.40
camera 36x36        0.02 / 0.01    0.94 / 0.94    1.00 / 0.99    0.59 / 0.58
coins 44x32         0.05 / 0.02    0.59 / 0.30    0.58 / 0.35    1.00 / 0.24
astronaut 56x48     0.03 / 0.01    0.93 / 0.93    1.00 / 1.00    0.59 / 0.59

Robustness to initialization (success AUC21, mean over sequences)
tracker      OPE  TRE pool  TRE seg     SRE
oracle     0.952     0.952    0.952   0.952
static     0.025     0.045    0.069   0.024
TM         0.650     0.645    0.684   0.540
TM+CV      0.676     0.645    0.677   0.572
drift      0.575     0.525    0.597   0.419

Reset-based (VOT2013/2015 style): accuracy (mean over sequences), failures (total)
oracle    1.000    0
static    0.131   85
TM        0.836    7
TM+CV     0.846    6
drift     0.842    7

Rankings (oracle excluded)
  OPE AUC21 (OTB)       TM+CV > TM > drift > static
  OPE P@20              TM+CV > TM > drift > static
  OPE NP@0.2            TM+CV > TM > drift > static
  AUC21 LaSOT-style     TM+CV > TM > drift > static
  AO pooled (GOT-10k)   TM+CV > TM > drift > static
  TRE pooled            TM > TM+CV > drift > static
  SRE                   TM+CV > TM > drift > static
  reset: fewest fails   TM+CV > TM > drift > static

The oracle row checks the measures and shows the offset derived above: its average overlap, precision and normalized precision are 1, but its success AUC is 0.952 with 21 thresholds and 0.990 with 101. Under the LaSOT-style rule it drops to 0.936, because the 24 hidden frames count as failures even for a perfect answer. For the three template trackers, the 101-threshold AUC lies within 0.006 of AO, and the 21-threshold AUC 0.02 to 0.03 below it, inside the bounds of the inequality: AUC with fine thresholds and average overlap are the same measure, while 21-threshold AUCs and GOT-10k’s AO are not directly comparable.

The aggregate ranking, TM+CV ahead of TM, then drift and static, is the same under success, precision and normalized precision, but the per-sequence table shows the measures disagreeing. On coins, the drift tracker is within 20 pixels of the centre in every frame (P@20 = 1.00) but within 0.2 box sizes in only 24% of frames, below TM and TM+CV on the normalized measure. In our reading, its updated template follows the growing and shrinking target loosely, staying near it without centring on it; 20 pixels is about half the width of that 44-pixel box, a lenient threshold. The static box, conversely, is within 20 pixels of the 20-pixel coffee target in 11% of frames but within 0.2 box sizes in 1%. A 20-pixel threshold equals 0.2 box sizes only for a 100-pixel box, so for the smaller targets here it is the more lenient of the two and rewards coarse localization (our reasoning); normalized precision removes the dependence on size.

How hidden frames are handled changes the scores without changing the ranking here. Counting them as failures lowers only the oracle and TM+CV, the two that overlap the annotated box while the target is hidden; the scores of TM, drift and static do not move, so, to the printed precision, their overlap in those frames was already zero. Pooling frames as GOT-10k does weights the 400-frame astronaut and 260-frame camera sequences more. On both, the drift tracker is within 20 pixels in only 59% of frames while TM+CV is in all of them, so pooled AO lowers drift (0.606 to 0.579) and raises TM+CV (0.710 to 0.721) relative to averaging over sequences.

TRE gives two different answers depending on how its runs are combined. Averaging the segments’ scores raises every tracker above its OPE score (TM+CV by only 0.001), as OTB-2013 reported, because late segments are short and short runs are easy. Pooling the segments’ frames lowers all three template trackers below OPE; in our reading, frames late in a sequence appear in many segments, and in every segment that started before a loss they are scored as failures. Under both, TM moves ahead of TM+CV: their pooled scores agree to three decimals, with TM ranked first, and the segment average is 0.684 against 0.677. In OPE, the per-sequence table puts TM+CV’s lead on the occluded camera and astronaut sequences, where coasting carried it through the occlusion; the output does not break TRE down by sequence, so it does not show which sequences reverse the order. The static box gains most from TRE in relative terms, from 0.025 to 0.069, because every restart puts it back on the target. SRE lowers every template tracker by 0.10 to 0.16 and keeps the OPE order: a template cut from a shifted or rescaled box carries its offset or wrong size through the sequence, and the drift tracker, which rebuilds its template from its own output, loses most.

The reset-based run compresses the differences. Accuracies of the three template trackers lie between 0.836 and 0.846 and failures between 6 and 7, whereas their OPE AUCs span 0.575 to 0.676: after a restart, the zeros that follow a failure in OPE are not scored, so accuracy measures localization while tracking and the failure count carries the rest. The drift tracker, last of the three in OPE, has an accuracy between TM’s and TM+CV’s here. This separation of localization from failure is what VOT designed its measures for, and EAO recombines them. The script was run as shown, with no changes after its first run.

Applications

Single-object trackers suit any setting where a target is pointed out once and must then be followed without a detector for its class: keeping a selected subject in view, following a target from a drone (the setting of UAV123), attaching effects or masks to an object in video editing, and propagating annotations between sparsely labelled frames. TrackingNet’s training set was built the last way. In larger systems a single-object tracker often runs between the detections of a slower detector, or follows an object that no detector covers. Real-time vision pipelines discuss the speed constraints.

Open Problems

• Evaluation that reflects use. One-pass AUC mixes localization with failure, reset-based measures separate them but depend on where resets fall, and anchors fix that at the cost of more runs; toolkits still differ in the details listed above.
• Training data and test classes. Large tracking datasets overlap in content and classes, and tracker papers train on several of them. GOT-10k’s one-shot rule and VOT’s prohibited-sequence lists and sequestered data are attempts to keep tests honest; results without the training data stated cannot be compared, a form of benchmark overfitting.
• Long-term re-detection. Reporting absence and finding the target again after long disappearances remains harder than short-term localization, and requires a confidence signal that most short-term designs do not produce.
• Template update. Trackers that use only the first frame cannot follow large appearance changes, and trackers that update from their own output risk drift; STARK and MixFormer gate updates with a learned score, and how to update safely is still open.
• Boxes, masks and points. VOT’s move to masks, and STM’s ranking in VOT2020, show single-object tracking converging with video object segmentation; following points rather than objects is a further variant of the same problem.