Visual Object Tracking Challenge

The Visual Object Tracking (VOT) Challenge is a series of annual benchmarks for single object tracking, held since 2013. Its results papers are written by the VOT committee with the participating teams; the reports read for this article are first-authored by Matej Kristan of the University of Ljubljana, with Jiří Matas (Czech Technical University), Aleš Leonardis (University of Birmingham) and Michael Felsberg (Linköping University) among the committee authors. Each edition published a small, carefully selected set of annotated video sequences, an evaluation toolkit that runs the tracker itself, and a joint results paper co-authored by the participants. The first challenge, VOT2013, was held at an ICCV 2013 workshop; the series ran as VOT until VOT2022 and continued from 2023 as the Visual Object Tracking and Segmentation challenge (VOTS), which merged single- and multi-target, short- and long-term tracking into one task scored on segmentation masks. In September 2026 the series received the PAMI Mark Everingham Prize at ECCV 2026 (VOT website, news of 12 September 2026).

Single Object Tracking defines the task and describes VOT’s reset-based protocol, the VOT2020 anchor-based protocol, the long-term measures and the definition of the expected average overlap (EAO). This article covers the challenge as a dataset and benchmark: its editions, how its sequences were chosen and annotated, the toolkit and its experiment stacks, the terms under which the data are distributed, and how the toolkit’s EAO estimator behaves, which the worked example measures on synthetic tracker runs. In keeping with the challenge’s own usage, “VOT” names the editions from 2013 to 2022 and “VOTS” those from 2023.

Purpose

The VOT2020 report states the initiative’s motive: before 2013 there was no consensus on performance evaluation, so tracking results could not be compared across papers, and VOT was formed to establish datasets, evaluation measures and toolkits and to provide a forum for discussing evaluation. From the start it fixed a class of trackers, which the VOT2015 report describes as single-camera, single-target, model-free, causal trackers applied to short-term tracking: the only supervised training example is the box in the first frame, and the tracker may not use future frames or, after a re-initialization, frames before it. Long-term trackers, which must re-detect a lost target, became a separate category in 2018.

Three design choices distinguish VOT from benchmarks that grew by adding sequences. The datasets were kept small and selected for diversity, which the VOT2020 report justifies by fast development-and-testing cycles. The toolkit ran each tracker itself, so the protocol, including re-initializations, was the same for every entry. And from VOT2017 the winner was decided on a sequestered dataset re-run by the committee, to reduce the influence of over-fitting without enlarging the public data (VOT2020 report; VOT website, challenges page). The VOT2015 report argues for the reset-based protocol: it notes that some papers had evaluated with the no-reset average overlap only on selected sequences, and cites the committee’s earlier analysis that this estimator has a much larger variance than the reset-based one, so that small datasets can distort it.

Dataset Size

The challenge page and the reports give the following sequence counts for the short-term (later main) datasets. Counts not stated in a source read for this article are left out.

Edition	Short-term sequences	Notes (source)
VOT2013	16	27 trackers from 51 participants (challenges page)
VOT2014	25	improved annotations, tests of statistical significance (challenges page)
VOT2015	60	plus 20 thermal sequences in VOT-TIR2015; 62 trackers (VOT2015 page and report)
VOT2016	60	the VOT2015 sequences with new annotations (challenges page)
VOT2017	refreshed	sequestered set and real-time experiment introduced (challenges page)
VOT2020	refreshed by one sequence	public and sequestered sets re-annotated with masks (VOT2020 report)
VOTS2023	144	341 targets, 1 to 8 per sequence (VOTS2023 report)

The VOT2015 report calls its dataset twice as large as VOT2014’s, while the challenge page gives 25 sequences for VOT2014 and 60 for VOT2015. VOTS2023 sequences are much longer than the short-term ones: on average about 2,000 frames, from 63 to 10,700, with a median of 1,810 (VOTS2023 report, Sec. 2.2). Single Object Tracking gives the long-term sets LTB35 and LTB50 and the 21,356 frames that TrackingNet’s comparison table lists for VOT2017.

Classes

VOT has no class labels: the target is whatever the first-frame annotation marks, and in some sequences it is a part of an object rather than the whole object (VOT2015 dataset page). Class information is excluded from the method as well as from the data. The VOT2020 rules prohibited identifying the target’s class in each sequence and applying a class-specific tracker, and banned learning on OTB, VOT, ALOV, UAV123, NUS-PRO, TempleColor and RGBT234; because part of GOT-10k’s training set was used to refresh the VOT-ST2020 data, a list of 1,000 GOT-10k sequences was prohibited for training while the rest remained allowed (VOT2020 report, Sec. 1.3). VOTS2023 added parts of objects (a foot, a hat, a hand) and transparent objects as targets, and allowed training on the training split of any dataset but not on the validation or test splits of popular tracking benchmarks (VOTS2023 report).

Annotation Types

Rotated boxes (2015–2019). VOT2013 used axis-aligned boxes (the VOT-TIR2015 page refers to its procedure as having no rotated boxes), and VOT2015 re-annotated all its targets with rotated boxes, stored as the four corner coordinates per frame (VOT2015 dataset page). The dataset page states the placement rule as at most about 30% of the pixels in the box belonging to the background, while containing most of the target; the VOT2015 report’s text states it as at least 60% of the pixels belonging to the target. For partly occluded or partly out-of-frame targets, annotators inferred the full extent, so a box includes, for example, a person’s hidden legs. Three groups of annotators each labelled a third of the sequences, the other two groups cross-checked them, a coordinator checked the result, and a rotated box was replaced by an axis-aligned one when the ratio of its shortest to its longest side exceeded 0.95, since rotation is ambiguous for round objects (VOT2015 dataset page and report). The VOT-TIR2015 thermal sequences kept axis-aligned boxes, as VOT2013 did.

Per-frame attributes. Every frame is labelled with five visual attributes, occlusion, illumination change, motion change, size change and camera motion, or as unassigned; the labels are not exclusive (VOT2015 report, Sec. 2). Per-frame rather than per-sequence labels let accuracy and robustness be computed on the frames that carry an attribute.

Segmentation masks (2020 onward). VOT had used masks since 2016 to fit its rotated boxes, but the VOT2020 report judged them too inaccurate as segmentation ground truth, so the public and sequestered short-term sets were re-annotated: initial masks from a semi-automatic method, then manual frame-by-frame correction. VOTS2023 annotated in stages: boxes drawn by hand on some frames and interpolated by a tracker, then verified; masks proposed by segmentation methods from those boxes; then editing by professional annotators and a supervisor’s check (VOTS2023 report, Sec. 2.2).

Anchors. For the VOT2020 protocol, the short-term datasets carry anchor frames, placed 50 frames apart with the first and last on the first and last frame, each checked by hand and moved by a few frames if the target is occluded there (VOT2020 report, Sec. 2.1). The toolkit reads them from the sequence data, where the sign of the value gives the direction: positive for a forward run, negative for a backward one (find_anchors in vot/experiment/multistart.py).

Practical-difference thresholds. To decide when two trackers’ differences are within annotation noise, VOT2015 had four annotators re-annotate five frames of each sequence three times and computed thresholds from the overlaps among these boxes (VOT2015 report, Sec. 4.1).

Splits

There is no training split: VOT is a test benchmark, and training on it was prohibited for the challenge. VOT2017 introduced a sequestered dataset alongside the public one: entries were ranked on the public data and the committee re-ran the top entries on the sequestered data to pick the winner. In VOT2020, the top five on the public set by EAO were re-run (VOT2020 report, Sec. 3.1). In VOTS2023 only the initialization frames were public: the ground truth of all other frames stayed on the evaluation server, participants submitted the tracker outputs that the toolkit produced, and each registered participant had 10 evaluation attempts (VOTS2023 report, Sec. 1.1).

Tasks

The sub-challenges changed from year to year. From the challenge page and the VOT2015, VOT2020 and VOTS2023 reports:

• Short-term RGB tracking (VOT-ST), every year from 2013 to 2022, with boxes up to 2019 and masks from 2020.
• Thermal infrared (VOT-TIR), introduced in 2015, which the VOT2020 report describes as evolving into the RGB+thermal challenge (VOT-RGBT) introduced in 2019.
• Real-time (VOT-RT), introduced in 2017: the same sequences, with frames processed under a time constraint.
• Long-term (VOT-LT), introduced in 2018, with the precision, recall and F-score that Single Object Tracking defines.
• RGB+depth (VOT-RGBD), introduced in 2019 as a long-term challenge.
• VOTS from 2023: one challenge in which one or more targets are tracked by segmentation over short or long sequences and may leave and re-enter the view. VOTS2024 added a sub-challenge on objects that undergo topological transformations; VOTS2025 a real-time sub-benchmark; VOTS2026 a point-tracking challenge (VOTSp2026) and a referring-tracking challenge that segments objects from descriptions (VOTSr2026), alongside the main and transformation challenges (challenges page and VOTS2026 pages). Point tracking is the subject of Point Tracking.

VOTS2026’s pages disagree on the venue as of November 2026: the challenge’s home and dataset pages name ECCV 2026 in Malmö, Sweden, while the workshop report in the site’s news says the workshop was held on 9 September 2026 at ECCV 2026 in Copenhagen, Denmark.

Evaluation

The measures themselves, accuracy, robustness and EAO under the reset and anchor protocols, are defined in Single Object Tracking. This section describes how the official toolkit computes them, where the code and the reports differ, and how the VOTS measures replaced them.

The toolkit and its stacks

VOT’s earlier toolkits were MATLAB-based (VOT2020 report); their repository, votchallenge/toolkit-legacy, is licensed GPL-3 or later and its code was not read for this article. VOT2020 introduced a re-implementation in Python, which participants were required to use, and its report announced that the MATLAB toolkits would become obsolete once the Python toolkit supported the VOT2019 challenges. The Python toolkit, votchallenge/toolkit, is licensed under GPL-3.0 and was read at commit 21a34c6 (20 October 2026). It cannot run here: it needs packages that are not installed (attributee among them) and downloads a challenge’s dataset when a workspace is set up. It was therefore read line by line, and the example below re-implements the parts it needs from the reports and that reading; no toolkit code is copied.

Each edition is an experiment stack, a YAML file that names the dataset and the analyses. At this commit:

Stack	Experiment	Analyses and parameters
vot2013 (marked deprecated)	supervised, 15 repetitions, re-initialization 5 frames after a failure	accuracy–robustness with sensitivity 30
vot2015/rgb	as above	accuracy–robustness; EAO over $[108, 371]$
vot2018/shortterm	as above, plus real-time and unsupervised experiments	EAO over $[100, 356]$
vot2019/shortterm	as above	EAO over $[46, 291]$
vot2020/shortterm, vot2022/shortterm	multistart (anchors), plus real-time and unsupervised	EAO over $[115, 755]$, accuracy–robustness
vots2023, vots2025/main, vots2026/main	unsupervised, one repetition, several objects	none listed

The VOT2022 stack’s title reads “VOT-ST2021 segmentation challenge”. The VOTS stacks only run the tracker and record its output; VOTS scores are computed by the evaluation server.

The reset protocol in the code

In the supervised experiment (vot/experiment/multirun.py), a failure is a frame whose overlap with the ground truth is at most failure_overlap, which defaults to 0, so a failure is a frame of zero overlap. The tracker is re-initialized skip_initialize frames later, 5 in the stacks. Stochastic trackers run 15 times; the experiment stops early once three runs give identical trajectories. The analyses in vot/analysis/supervised.py then compute:

• Accuracy: the mean overlap over frames that are neither an initialization, one of the 9 frames after it (a burn-in of 10 frames counted from the initialization frame), a failure frame nor a skipped frame; averaged over repetitions, then over sequences weighted by their length.
• Robustness: the number of failures, averaged over repetitions and then over sequences weighted by length. The AR plot shows $\exp(-30\,F/\bar L)$, with $F$ that weighted mean and $\bar L$ the mean sequence length, the “sensitivity” of 30 in the stack.
• The EAO curve: each run is cut into segments from an initialization frame to the frame before the next failure, and the last segment, if no failure ends it, is marked successful. All segments of all sequences and repetitions are pooled. A failed segment is padded with zeros to the length of the longest segment; a successful one is used only up to its own length. The curve at $N_s$ is the mean over the segments in use of their average overlap over frames $1, \dots, N_s$ after the initialization frame, which is left out. EAOScore averages the curve from $N_{lo}$ to $N_{hi}$.

The anchor protocol in the code

In the multistart experiment (vot/experiment/multistart.py), the tracker is initialized at every anchor and run once to the start or end of the sequence. vot/analysis/multistart.py declares a failure when the overlap is at most 0.1 for 10 consecutive frames in which the ground truth is not empty, places the failure at the first of those frames, and sets the overlaps from there on to zero. Accuracy is the sum of overlaps before the failure divided by the number of those frames, pooled over anchors and sequences; robustness is the fraction of each sub-sequence before the failure, pooled with the weights of the VOT2020 report’s equations (3) to (6). compute_eao_partial builds the EAO curve per sequence from the sub-sequences, and the aggregate pools them; a failed sub-sequence is extended with zeros, and one that did not fail is not extended beyond its length, as the report’s equation (7) states.

Code against the reports

Reading the code next to the VOT2015 and VOT2020 reports shows several differences. None changes the protocol’s intent, and each is described here as a difference, not as an error.

• The EAO normalization. Both reports write EAO as $\frac{1}{N_{hi} - N_{lo}} \sum_{N_s = N_{lo}}^{N_{hi}} \hat\Phi_{N_s}$ (VOT2015 report, Eq. 2; VOT2020 report, Eq. 8), a sum over $N_{hi} - N_{lo} + 1$ values divided by one less. The toolkit takes the mean of curve[low:high + 1]. For the reset curve this is the mean over all $N_{hi} - N_{lo} + 1$ values. The anchor curve has only high entries, so index $N_{hi}$ does not exist and the mean runs over $N_{lo}, \dots, N_{hi} - 1$.
• The length of the reset curve. The reset curve is only as long as the longest segment. If no segment reaches $N_{hi}$ frames, the mean covers fewer values than the range holds.
• The tail of a failed anchor run. Beyond the end of a failed sub-sequence, compute_eao_partial divides the sum of its overlaps by $j - 1$, while inside the run it takes the mean over $j$ frames. The value at $j$ equal to the run’s length therefore repeats the previous one, and the tail is larger than the report’s average over $j$ frames by a factor $j/(j-1)$.
• “Falls below” and “at most”. The VOT2020 report detects a tentative failure when the overlap “falls below” the threshold of 0.1; the code tests overlap <= threshold.
• Losses near the end of a run. A loss in the last 9 frames of a sub-sequence never accumulates 10 low frames, so the code counts that run as not failed, with its last overlaps at zero. The report’s definition, a failure when the tracker does not recover within the next 10 frames, does not say what happens when fewer than 10 frames remain.
• The initialization frame in the 2020 accuracy. The toolkit records the initialization frame as a special region, and its overlap function treats any region that is not a box, polygon or mask as empty, so that frame scores 0 against a non-empty ground truth. The EAO curves start after it, but the anchor accuracy sums the overlaps from the first frame of the run, including that 0, while the report’s equation (1) averages the overlaps of the frames before the failure.
• Burn-in. The supervised EAO curve declares a burn-in parameter of 10 that its computation does not use; only accuracy ignores the frames after initialization.
• Precision. The reset EAO stores overlaps as 32-bit floats; the example below uses 64-bit floats.

The range $[N_{lo}, N_{hi}]$ is not computed by the toolkit; it is a number in each stack. The VOT2015 report obtained $[108, 371]$ from a kernel density estimate of its 60 sequence lengths, with the mode at 168 and the bounds the nearest points on either side where the densities are approximately equal and the enclosed mass is 0.5 (Sec. 3.1.2, Sec. 4.2, Fig. 3). The VOT2020 report took the mean plus or minus one standard deviation of the lengths of the anchor sub-sequences, $[115, 755]$ on the public VOT-ST2020 data.

VOTS measures

VOTS2023 replaced EAO with measures for several targets that may leave the view (VOTS2023 report, Sec. 2.1). Each target is scored in each frame by the intersection over union of predicted and ground-truth masks; when the target is absent, an empty prediction scores 1 and any other prediction 0. The primary measure, the tracking quality $Q$, is the area under a success curve over thresholds, which the report shows equals the overlap averaged over the frames and targets of each sequence and then over sequences; initialization frames are excluded. Secondary measures split the result: accuracy over frames with nonzero overlap, robustness as the fraction of visible-target frames with nonzero overlap, the not-reported error (target present but reported absent), the drift-rate error (reported present but off the target), and the absence-detection quality, computed only for targets absent for at least 10 frames.

Example: how the EAO estimators behave

Single Object Tracking’s example runs the reset protocol and leaves EAO out. This example computes EAO as the toolkit does, under both protocols, for error-model trackers whose true EAO is known in closed form, and prints the code-against-report differences listed above.

Data. Sixty synthetic sequences have lengths drawn once from a log-normal distribution (median 300, clipped to 50–1,500 frames). In each, the ground truth is a $34 \times 34$ box moving on a Lissajous path. The trackers are error models built from the ground truth, not real trackers. While tracking, a tracker outputs the box shifted 6 pixels sideways, an IoU of $28/40 = 0.7$ exactly; once it loses the target it outputs a box 100 pixels away, with IoU 0. After each initialization it loses the target in each frame with probability $h$, independently, so the loss happens $k \ge 1$ frames after initialization with $k$ geometric. Four trackers differ only in $h$: T0 never fails ($h = 0$, a check), and T1, T2 and T3 have $h = 0.001$, $0.004$ and $0.016$. The initialization frame scores 0, as the toolkit’s empty initialization region does.

Ground truth. A tracker of this kind that runs from initialization without resets has overlap $a = 0.7$ in frame $i$ with probability $(1-h)^i$, so its expected average overlap over frames $1, \dots, N_s$ after initialization, the quantity EAO estimates, is

$$\hat\Phi_{N_s} = \frac{a}{N_s} \sum_{i=1}^{N_s} (1-h)^i ,$$

with the initialization frame left out as in the toolkit.

Protocols. The reset protocol follows the VOT2015 stack: 15 repetitions, failure at zero overlap, re-initialization 5 frames later, segments and EAO curve as described above. The anchor protocol follows the VOT2020 stack: one run per anchor, failure when the overlap is at most 0.1 for 10 frames, EAO curve with the code’s $j-1$ tail. The anchors are our analogue of the dataset’s annotations: every 50 frames from the first, plus the last frame, with each run going in the direction that gives the longer sub-sequence, as the VOT2020 report describes, but without the manual moves. The range for the reset protocol uses the VOT2015 rule with SciPy’s Gaussian kernel density estimate in place of the report’s batch KDE, which was not available; it is our analogue, Scott’s bandwidth is used and Silverman’s printed. The range for the anchor protocol uses the VOT2020 rule with the population standard deviation, and the sample standard deviation is printed. Outcomes that depend on the anchors or the KDE test these analogues, not the official anchor files or KDE.

Checks and measures. The script checks its curves against a direct loop over $N_s$ written from the description, its window-based failure test against a frame-by-frame counter, a hand-computed reset case, and T0, for which every curve must equal 0.7. It prints, over ten seeds of the tracker randomness with the sequence lengths fixed, each EAO against the analytic value on the same range, the code-against-report variants, the curves at three lengths, and accuracy and robustness in both protocols’ forms. The planned hypotheses were that both estimates fall below the analytic EAO in every seed for T1, T2 and T3; that the $j-1$ tail raises the anchor EAO by an amount between 0 and $1/(N_{lo} - 1)$ of the $j$-tail value, which follows from the factor $j/(j-1) \le N_{lo}/(N_{lo}-1)$ over the range; and that the toolkit’s anchor accuracy, which includes the initialization frame, lies below the accuracy without it, which in turn is at most 0.7. Versions: Python 3.13.9, NumPy 2.3.5, SciPy 1.16.3; the script runs in about 3 seconds.

# VOT example: expected average overlap (EAO) as the VOT toolkit computes it, on synthetic tracker runs.
# Re-implemented from the VOT2015 and VOT2020 reports and a reading of votchallenge/toolkit at commit
# 21a34c6 (GPL-3.0); no toolkit code is copied or run.
import os, sys
os.environ["OMP_NUM_THREADS"] = "1"; os.environ["OPENBLAS_NUM_THREADS"] = "1"
import numpy as np
from scipy.stats import gaussian_kde

A_OV = 0.7           # overlap while tracking (box shifted 6 px on a 34-px box)
TRACKERS = {"T0": 0.0, "T1": 0.001, "T2": 0.004, "T3": 0.016}
N_SEQ, REPS, SKIP, BURNIN = 60, 15, 5, 10
ANCHOR_GAP, THETA, GRACE = 50, 0.1, 10

def seq_lengths():
    rng = np.random.default_rng(2026)
    return np.clip(np.round(rng.lognormal(np.log(300), 0.6, N_SEQ)), 50, 1500).astype(int)

def iou(b1, b2):
    x1 = np.maximum(b1[:, 0], b2[:, 0]); y1 = np.maximum(b1[:, 1], b2[:, 1])
    x2 = np.minimum(b1[:, 0] + b1[:, 2], b2[:, 0] + b2[:, 2])
    y2 = np.minimum(b1[:, 1] + b1[:, 3], b2[:, 1] + b2[:, 3])
    inter = np.clip(x2 - x1, 0, None) * np.clip(y2 - y1, 0, None)
    return inter / (b1[:, 2] * b1[:, 3] + b2[:, 2] * b2[:, 3] - inter)

def gt_boxes(L, s):
    t = np.arange(L)
    cx = 200 + 120 * np.sin(2 * np.pi * t / (150 + 7 * s))
    cy = 150 + 80 * np.sin(2 * np.pi * t / (110 + 5 * s) + s)
    return np.stack([cx - 17, cy - 17, np.full(L, 34.0), np.full(L, 34.0)], 1)

def overlaps_from_state(gt, state):
    """state: 1 tracking (box 6 px off), 0 lost (box 100 px off); init frames handled by caller."""
    pred = gt.copy()
    pred[:, 0] += np.where(state == 1, 6.0, 100.0)
    return iou(pred, gt)

# ---------------- reset protocol (VOT2013-2019) ----------------
def run_reset(L, gt, h, rng):
    """Returns per-frame overlaps and labels (0 init, 1 track, 2 failure, 3 unknown)."""
    lab = np.full(L, 3); state = np.zeros(L, int)
    f = 0
    while f < L:
        lab[f] = 0; f0 = f
        k = rng.geometric(h) if h > 0 else 10**9          # failure k frames after init
        end = min(f0 + k, L)
        lab[f0 + 1:end] = 1; state[f0 + 1:end] = 1
        if f0 + k < L:
            lab[f0 + k] = 2
            f = f0 + k + SKIP
        else:
            break
    ov = overlaps_from_state(gt, state)
    ov[lab != 1] = 0.0                                     # init, failure and unknown frames score 0
    return ov, lab

def segments_2015(ov, lab):
    inits = np.flatnonzero(lab == 0); fails = np.flatnonzero(lab == 2)
    segs = [(ov[i:j], False) for i, j in zip(inits, fails)]
    if len(inits) > len(fails):
        segs.append((ov[inits[-1]:], True))
    return segs

def eao_curve_2015(segs):
    maxlen = max(len(o) for o, _ in segs)
    num = np.zeros(maxlen); den = np.zeros(maxlen); tail = np.zeros(maxlen + 1); nfail_tail = np.zeros(maxlen + 1)
    j = np.arange(maxlen, dtype=float); j[0] = 1
    for o, ok in segs:
        n = len(o); cs = np.cumsum(o); cs = cs - o[0]   # sum of frames 1..k
        num[1:n] += cs[1:n] / j[1:n]; den[1:n] += 1
        if not ok:
            tail[n] += cs[n - 1]; nfail_tail[n] += 1
    num[1:] += np.cumsum(tail[:maxlen])[1:] / j[1:]
    den[1:] += np.cumsum(nfail_tail[:maxlen])[1:]
    c = np.zeros(maxlen); c[1:] = num[1:] / np.maximum(den[1:], 1)
    return c

def eao_curve_2015_literal(segs):
    """Direct loop over N_s: a failed segment is padded with zeros and counts at every N_s,
    a successful one only while N_s is inside it."""
    maxlen = max(len(o) for o, _ in segs)
    padded = [(np.pad(o, (0, maxlen - len(o))), len(o), ok) for o, ok in segs]
    c = np.zeros(maxlen)
    for N in range(1, maxlen):
        tot, cnt = 0.0, 0
        for o, n, ok in padded:
            if N < n or not ok:
                tot += o[1:N + 1].mean(); cnt += 1
        c[N] = tot / cnt
    return c

# ---------------- anchor protocol (VOT2020-2022) ----------------
def anchors(L):
    a = list(range(0, L, ANCHOR_GAP))
    if a[-1] != L - 1: a.append(L - 1)
    return [(k, (L - k) >= (k + 1)) for k in a]           # (frame, forward?)

def detect_failure(o):
    low = (o <= THETA).astype(int)
    if len(o) < GRACE: return len(o)
    w = np.convolve(low, np.ones(GRACE, int), "valid")    # w[i] = lows in o[i:i+GRACE]
    idx = np.flatnonzero(w == GRACE)
    return int(idx[0]) if len(idx) else len(o)

def detect_failure_literal(o, strict=False):
    """Frame-by-frame counter of consecutive low frames; the failure is the first frame of the first run of GRACE."""
    run = 0
    for i, v in enumerate(o):
        low = (v < THETA) if strict else (v <= THETA)
        run = run + 1 if low else 0
        if run == GRACE:
            return i - GRACE + 1
    return len(o)

def run_anchor(n, gt, h, rng):
    k = rng.geometric(h) if h > 0 else 10**9
    state = np.zeros(n, int); state[1:min(k, n)] = 1
    o = overlaps_from_state(gt, state); o[0] = 0.0        # init frame scores 0
    p = detect_failure(o)
    ok = p >= n
    if not ok: o[p:] = 0.0
    return o, p, ok

def eao_curve_2020(runs, J, tail_div="j-1"):
    num = np.zeros(J); den = np.zeros(J)
    for o, ok in runs:
        n = len(o); cs = np.cumsum(o) - o[0]
        m = min(n, J)
        jj = np.arange(1, m); num[1:m] += cs[1:m] / jj; den[1:m] += 1
        if not ok and n < J:
            jt = np.arange(n, J)
            d = (jt - 1) if tail_div == "j-1" else jt
            num[n:J] += cs[n - 1] / d; den[n:J] += 1
    c = np.zeros(J); c[1:] = num[1:] / np.maximum(den[1:], 1)
    return c

def eao_curve_2020_literal(runs, J):
    """Direct loop over N_s: inside a run the mean of frames 1..N_s; beyond the end of a failed run,
    the sum of its overlaps divided by N_s - 1 (the code's tail)."""
    c = np.zeros(J)
    for N in range(1, J):
        tot, cnt = 0.0, 0
        for o, ok in runs:
            if N < len(o):
                tot += o[1:N + 1].mean(); cnt += 1
            elif not ok:
                tot += o[1:].sum() / (N - 1); cnt += 1
        c[N] = tot / cnt if cnt else 0.0
    return c

def analytic_curve(h, J):
    q = (1 - h) ** np.arange(1, J)
    c = np.zeros(J); c[1:] = A_OV * np.cumsum(q) / np.arange(1, J)
    return c

def kde_range(lengths, bw):
    kde = gaussian_kde(lengths.astype(float), bw_method=bw)
    x = np.arange(-1000, 4001); p = kde(x)
    m = int(np.argmax(p)); mode = int(x[m])
    lo_c, hi_c = 0.0, p[m]
    for _ in range(200):
        c = 0.5 * (lo_c + hi_c)
        i = m
        while i > 0 and p[i - 1] >= c: i -= 1
        k = m
        while k < len(p) - 1 and p[k + 1] >= c: k += 1
        mass = p[i:k + 1].sum()
        if mass > 0.5: lo_c = c
        else: hi_c = c
    return mode, int(x[i]), int(x[k]), mass, p[i], p[k], p.sum(), np.abs(np.diff(p)).max()

SEEDS = [int(x) for x in sys.argv[1].split(",")] if len(sys.argv) > 1 else list(range(10))

def run_all(h, seed, Ls, J20):
    """One seed: reset protocol (15 repetitions) and anchor protocol on all sequences."""
    rng = np.random.default_rng([seed, int(h * 1e6)])
    segs, runs = [], []
    accs, fails, ar20 = [], [], []
    undetected = 0
    for s, L in enumerate(Ls):
        gt = gt_boxes(L, s)
        a_s, f_s = [], []
        for r in range(REPS):
            ov, lab = run_reset(L, gt, h, rng)
            segs += segments_2015(ov, lab)
            mask = lab == 1
            for i in np.flatnonzero(lab == 0):
                mask[i:i + BURNIN] = False            # init frame and the next 9 frames ignored
            a_s.append(ov[mask].mean() if mask.any() else 0.0); f_s.append((lab == 2).sum())
        accs.append(np.mean(a_s)); fails.append(np.mean(f_s))
        tot_p = tot_n = 0; sum_ov = sum_ov1 = 0.0; tot_p1 = 0
        for k, fw in anchors(L):
            idx = np.arange(k, L) if fw else np.arange(k, -1, -1)
            o, p, ok = run_anchor(len(idx), gt[idx], h, rng)
            runs.append((o, ok))
            if ok and (o[1:] == 0).any(): undetected += 1   # lost in the last frames, not detected
            tot_p += p; tot_n += len(o); sum_ov += o[:p].sum()
            if p >= 1: sum_ov1 += o[1:p].sum(); tot_p1 += p - 1
        ar20.append((sum_ov, tot_p, tot_n, L, sum_ov1, tot_p1))
    return segs, runs, np.array(accs), np.array(fails), ar20, undetected

if __name__ == "__main__":
    Ls = seq_lengths()
    print("60 synthetic sequences: lengths %d to %d, median %d" % (Ls.min(), Ls.max(), np.median(Ls)))
    for bw in ("scott", "silverman"):
        mode, lo, hi, mass, plo, phi_, tot, dmax = kde_range(Ls, bw)
        print("VOT2015 rule, KDE %-9s mode %d  [N_lo, N_hi] = [%d, %d]  mass %.5f  p(lo) %.4e p(hi) %.4e  max grid step %.1e"
              % (bw, mode, lo, hi, mass, plo, phi_, dmax))
    sub = np.array([(L - k) if fw else (k + 1) for L in Ls for k, fw in anchors(L)])
    for dd in (0, 1):
        mu, sd = sub.mean(), sub.std(ddof=dd)
        print("VOT2020 rule, %d anchor sub-sequences, mean %.1f +- std (ddof=%d) %.1f  [N_lo, N_hi] = [%d, %d]"
              % (len(sub), mu, dd, sd, round(mu - sd), round(mu + sd)))
    lo15, hi15 = kde_range(Ls, "scott")[1:3]
    lo20, hi20 = int(round(sub.mean() - sub.std())), int(round(sub.mean() + sub.std()))
    J20 = max(hi20 + 1, 401)
    n15, n20 = hi15 - lo15, hi20 - lo20
    print("ranges used: 2015 [%d, %d] (Scott), 2020 [%d, %d] (ddof=0); seeds %s" % (lo15, hi15, lo20, hi20, SEEDS))

    # ---- checks on exact cases ----
    L = 45; ov = np.zeros(L); lab = np.full(L, 3)
    for f0 in (0, 11, 22):
        lab[f0] = 0; lab[f0 + 1:f0 + 6] = 1; ov[f0 + 1:f0 + 6] = A_OV; lab[f0 + 6] = 2
    lab[33] = 0; lab[34:] = 1; ov[34:] = A_OV
    c = eao_curve_2015(segments_2015(ov, lab)); jj = np.arange(1, len(c))
    hand = np.where(jj <= 5, A_OV, (3 * 5 * A_OV / jj + A_OV) / 4)
    print("check, hand-computed reset case: max |curve - hand| %.1e" % np.abs(c[1:] - hand).max())
    o = np.r_[0.0, np.full(20, A_OV), np.full(12, 0.1), np.full(20, A_OV)]
    print("check, overlap exactly 0.1 for 12 frames: failure at frame %d with <= (code), %d (= none) with < (report's 'below')"
          % (detect_failure_literal(o), detect_failure_literal(o, strict=True)))

    res = {}
    for name, h in TRACKERS.items():
        an = analytic_curve(h, 1601)
        rows = []
        for seed in SEEDS:
            segs, runs, accs, fails, ar20, und = run_all(h, seed, Ls, J20)
            c15 = eao_curve_2015(segs)
            c20 = eao_curve_2020(runs, J20); c20j = eao_curve_2020(runs, J20, "j")
            if seed == SEEDS[0]:
                lit15 = np.abs(eao_curve_2015(segs[:300]) - eao_curve_2015_literal(segs[:300])).max()
                lit20 = np.abs(eao_curve_2020(runs[:120], hi20) - eao_curve_2020_literal(runs[:120], hi20)).max()
                mism = sum(detect_failure(o) != detect_failure_literal(o) for o, _ in runs)
                print("check %s: fast vs loop curves max diff 2015 %.1e, 2020 %.1e; failure detection mismatches %d of %d"
                      % (name, lit15, lit20, mism, len(runs)))
                if h == 0:
                    print("check T0 (never fails): max |curve - a| 2015 %.1e, 2020 %.1e, 2020 with /j %.1e"
                          % (np.abs(c15[1:] - A_OV).max(), np.abs(c20[1:] - A_OV).max(), np.abs(c20j[1:] - A_OV).max()))
            w = Ls / Ls.sum()
            F = (fails * w).sum()
            so, tp, tn, LL, so1, tp1 = (np.array(v, float) for v in zip(*ar20))
            Rs = tp / tn
            rows.append(dict(
                e15=c15[lo15:hi15 + 1].mean(), e15rep=c15[lo15:hi15 + 1].sum() / n15,
                e20=c20[lo20:hi20].mean(), e20j=c20j[lo20:hi20].mean(),
                e20inc=c20[lo20:hi20 + 1].mean(), e20rep=c20[lo20:hi20 + 1].sum() / n20,
                c15=c15, c20=c20, len15=len(c15), k15=len(c15[lo15:hi15 + 1]), k20=len(c20[lo20:hi20]),
                A15=(accs * w).sum(), F=F, Fplain=fails.mean(), AR=np.exp(-30 * F / Ls.mean()),
                A20=so.sum() / tp.sum(), A20x=so1.sum() / tp1.sum(), R20=(Rs * LL).sum() / LL.sum(), und=und))
        res[name] = (h, an, rows)

    def stat(v): v = np.asarray(v); return "%.4f [%.4f, %.4f]" % (v.mean(), v.min(), v.max())
    print("\nEAO, mean [min, max] over %d seeds; gap = estimate - analytic on the same range" % len(SEEDS))
    for name, (h, an, rows) in res.items():
        a15, a20 = an[lo15:hi15 + 1].mean(), an[lo20:hi20].mean()
        g15 = np.array([r["e15"] for r in rows]) - a15; g20 = np.array([r["e20"] for r in rows]) - a20
        tag = "check" if h == 0 else "     "
        print("%s h=%.3f  VOT2015 reset: analytic %.4f  estimate %s  gap max %+.5f (<0 in %d/%d)"
              % (name, h, a15, stat([r["e15"] for r in rows]), g15.max(), (g15 < 0).sum(), len(rows)))
        print("%s          2015 curve length %d to %d frames (N_hi = %d); values averaged %d to %d of %d"
              % (tag, min(r["len15"] for r in rows), max(r["len15"] for r in rows), hi15,
                 min(r["k15"] for r in rows), max(r["k15"] for r in rows), n15 + 1))
        print("%s          VOT2020 anchor: analytic %.4f  estimate %s  gap max %+.5f (<0 in %d/%d)"
              % (tag, a20, stat([r["e20"] for r in rows]), g20.max(), (g20 < 0).sum(), len(rows)))
        print("%s          2020 values averaged %d to %d of %d" % (tag, min(r["k20"] for r in rows), max(r["k20"] for r in rows), n20))
    print("\nCode against report, mean over seeds")
    print("tracker  2015 toolkit  2015 sum/(hi-lo) | 2020 toolkit  /j tail  incl. N_hi  sum/(hi-lo) | /(j-1) gain min, max, max ratio to bound")
    for name, (h, an, rows) in res.items():
        m = lambda k: np.mean([r[k] for r in rows])
        d = np.array([r["e20"] - r["e20j"] for r in rows]); b = np.array([r["e20j"] / (lo20 - 1) for r in rows])
        idf = max(abs(r["e15rep"] - r["e15"] * (n15 + 1) / n15) for r in rows)
        print("%s%s %.5f      %.5f        |  %.5f     %.5f  %.5f     %.5f    | %.2e, %.2e, %.3f  (identity err %.0e)"
              % (name, " (check)" if h == 0 else "        ", m("e15"), m("e15rep"), m("e20"), m("e20j"), m("e20inc"), m("e20rep"), d.min(), d.max(), (d / b).max(), idf))
    print("\nEAO curve at N_s frames, mean over seeds: analytic / reset / anchor ('-': a seed's curve ends before N_s)")
    def at(rows, key, N):
        if all(len(r[key]) > N for r in rows): return "%.3f" % np.mean([r[key][N] for r in rows])
        return "- (curve length %d)" % min(len(r[key]) for r in rows)
    for name, (h, an, rows) in res.items():
        print(name, "  ".join("N=%d: %.3f / %s / %s" % (N, an[N], at(rows, "c15", N), at(rows, "c20", N)) for N in (25, 100, 400)))
    print("\nAccuracy and robustness, mean over seeds (a = %.1f)" % A_OV)
    print("tracker | 2015: A   failures (length-weighted, plain)  exp(-30F/L) | 2020: A (code)  A (init frame out)  R   undetected end losses")
    for name, (h, an, rows) in res.items():
        m = lambda k: np.mean([r[k] for r in rows])
        print("%s      | %.6f  %6.2f  %6.2f                %.3f     | %.4f          %.4f              %.4f  %d"
              % (name, m("A15"), m("F"), m("Fplain"), m("AR"), m("A20"), m("A20x"), m("R20"), sum(r["und"] for r in rows)))
    print("A2015 max |A - a| over trackers and seeds: %.1e" % max(abs(r["A15"] - A_OV) for _, _, rows in res.values() for r in rows))
    print("A2020 code < init-out <= a in all seeds: %s" % all(r["A20"] < r["A20x"] <= A_OV + 1e-12
          for n, (h, _, rows) in res.items() if h > 0 for r in rows))

60 synthetic sequences: lengths 96 to 1500, median 331
VOT2015 rule, KDE scott     mode 278  [N_lo, N_hi] = [148, 442]  mass 0.50102  p(lo) 1.3781e-03 p(hi) 1.3774e-03  max grid step 7.6e-06
VOT2015 rule, KDE silverman mode 282  [N_lo, N_hi] = [148, 448]  mass 0.49873  p(lo) 1.3539e-03 p(hi) 1.3486e-03  max grid step 7.0e-06
VOT2020 rule, 576 anchor sub-sequences, mean 441.9 +- std (ddof=0) 301.5  [N_lo, N_hi] = [140, 743]
VOT2020 rule, 576 anchor sub-sequences, mean 441.9 +- std (ddof=1) 301.8  [N_lo, N_hi] = [140, 744]
ranges used: 2015 [148, 442] (Scott), 2020 [140, 743] (ddof=0); seeds [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
check, hand-computed reset case: max |curve - hand| 1.1e-16
check, overlap exactly 0.1 for 12 frames: failure at frame 21 with <= (code), 53 (= none) with < (report's 'below')
check T0: fast vs loop curves max diff 2015 7.8e-15, 2020 8.0e-15; failure detection mismatches 0 of 576
check T0 (never fails): max |curve - a| 2015 1.9e-14, 2020 9.7e-15, 2020 with /j 9.7e-15
check T1: fast vs loop curves max diff 2015 3.4e-15, 2020 2.1e-15; failure detection mismatches 0 of 576
check T2: fast vs loop curves max diff 2015 2.1e-15, 2020 1.2e-15; failure detection mismatches 0 of 576
check T3: fast vs loop curves max diff 2015 1.7e-15, 2020 7.2e-16; failure detection mismatches 0 of 576

EAO, mean [min, max] over 10 seeds; gap = estimate - analytic on the same range
T0 h=0.000  VOT2015 reset: analytic 0.7000  estimate 0.7000 [0.7000, 0.7000]  gap max -0.00000 (<0 in 10/10)
check          2015 curve length 1500 to 1500 frames (N_hi = 442); values averaged 295 to 295 of 295
check          VOT2020 anchor: analytic 0.7000  estimate 0.7000 [0.7000, 0.7000]  gap max -0.00000 (<0 in 10/10)
check          2020 values averaged 603 to 603 of 603
T1 h=0.001  VOT2015 reset: analytic 0.6065  estimate 0.5542 [0.5438, 0.5684]  gap max -0.03816 (<0 in 10/10)
               2015 curve length 1500 to 1500 frames (N_hi = 442); values averaged 295 to 295 of 295
               VOT2020 anchor: analytic 0.5683  estimate 0.4664 [0.4497, 0.4851]  gap max -0.08314 (<0 in 10/10)
               2020 values averaged 603 to 603 of 603
T2 h=0.004  VOT2015 reset: analytic 0.4155  estimate 0.3416 [0.3355, 0.3489]  gap max -0.06661 (<0 in 10/10)
               2015 curve length 987 to 1500 frames (N_hi = 442); values averaged 295 to 295 of 295
               VOT2020 anchor: analytic 0.3444  estimate 0.2822 [0.2705, 0.3032]  gap max -0.04117 (<0 in 10/10)
               2020 values averaged 603 to 603 of 603
T3 h=0.016  VOT2015 reset: analytic 0.1561  estimate 0.1363 [0.1326, 0.1436]  gap max -0.01245 (<0 in 10/10)
               2015 curve length 397 to 521 frames (N_hi = 442); values averaged 249 to 295 of 295
               VOT2020 anchor: analytic 0.1169  estimate 0.1114 [0.1052, 0.1174]  gap max +0.00051 (<0 in 9/10)
               2020 values averaged 603 to 603 of 603

Code against report, mean over seeds
tracker  2015 toolkit  2015 sum/(hi-lo) | 2020 toolkit  /j tail  incl. N_hi  sum/(hi-lo) | /(j-1) gain min, max, max ratio to bound
T0 (check) 0.70000      0.70238        |  0.70000     0.70000  0.70000     0.70116    | 0.00e+00, 0.00e+00, 0.000  (identity err 1e-16)
T1         0.55420      0.55608        |  0.46635     0.46624  0.46606     0.46683    | 9.11e-05, 1.20e-04, 0.036  (identity err 0e+00)
T2         0.34159      0.34275        |  0.28215     0.28199  0.28193     0.28239    | 1.49e-04, 1.80e-04, 0.089  (identity err 6e-17)
T3         0.13631      0.13448        |  0.11143     0.11131  0.11133     0.11152    | 1.10e-04, 1.20e-04, 0.148  (identity err 2e-02)

EAO curve at N_s frames, mean over seeds: analytic / reset / anchor ('-': a seed's curve ends before N_s)
T0 N=25: 0.700 / 0.700 / 0.700  N=100: 0.700 / 0.700 / 0.700  N=400: 0.700 / 0.700 / 0.700
T1 N=25: 0.691 / 0.691 / 0.691  N=100: 0.666 / 0.665 / 0.665  N=400: 0.577 / 0.487 / 0.499
T2 N=25: 0.665 / 0.663 / 0.667  N=100: 0.576 / 0.561 / 0.579  N=400: 0.348 / 0.251 / 0.279
T3 N=25: 0.571 / 0.568 / 0.571  N=100: 0.345 / 0.324 / 0.347  N=400: 0.107 / - (curve length 397) / 0.101

Accuracy and robustness, mean over seeds (a = 0.7)
tracker | 2015: A   failures (length-weighted, plain)  exp(-30F/L) | 2020: A (code)  A (init frame out)  R   undetected end losses
T0      | 0.700000    0.00    0.00                1.000     | 0.6984          0.7000              1.0000  0
T1      | 0.700000    0.59    0.41                0.958     | 0.6978          0.6999              0.8006  43
T2      | 0.700000    2.39    1.63                0.839     | 0.6959          0.6997              0.5005  77
T3      | 0.700000    8.88    6.06                0.521     | 0.6885          0.6997              0.1859  25
A2015 max |A - a| over trackers and seeds: 0.0e+00
A2020 code < init-out <= a in all seeds: True

Ranges and checks. On these lengths the VOT2015 rule gives $[148, 442]$ around a mode of 278 with Scott’s bandwidth and $[148, 448]$ with Silverman’s; the enclosed mass is 0.50102 and 0.49873, within 0.0013 of 0.5 on the integer grid, and the densities at the two ends agree to within one grid step: the KDE check held. The VOT2020 rule gives a wider range, $[140, 743]$, from 576 anchor sub-sequences with a mean of 441.9 frames and a standard deviation of 301.5. The bandwidth and the choice of standard deviation move the bounds by at most six frames. The fast curves agree with the direct loops to within $10^{-14}$, the two failure tests never disagree, the hand case matches, and T0’s curves equal 0.7 under both protocols and both tails: these checks held. With an overlap of exactly 0.1 held for 12 frames, the code’s “at most” declares a failure at frame 21 and the report’s “below” declares none, as planned: this check held.

The estimators fall below the truth. For T1, T2 and T3, the reset EAO is below the analytic EAO in all 10 seeds, by 0.012 to 0.080 depending on the tracker and seed (from the printed ranges), and the anchor EAO is below it in all 10 seeds for T1 and T2: both hypotheses held for these trackers. The cause is in the estimator’s construction: a segment that ends without a failure, because the sequence ended, is used only up to its length, while a failed segment is padded with zeros to any length. At large $N_s$ the curve therefore averages over all failed segments but only over the long successful ones, and the short successful ones, which would have contributed high overlaps, are missing. The curves show this growing with $N_s$: at 25 frames the estimates are within 0.003 of the analytic value, at 400 frames T2’s reset estimate is 0.251 against 0.348. In our reading, the size of this gap depends on how long the sequences are compared with the time a tracker survives, so it is not a fixed correction. These two hypotheses were written after planning scripts had printed reset and anchor estimates below the analytic value for one seed of a tracker with T2’s hazard, so the example is an illustration rather than a test.

For T3 the reset hypothesis held in all 10 seeds, but T3’s reset curve is only 397 to 521 frames long, shorter than $N_{hi} = 442$ in some seeds, so its EAO averages 249 to 295 of the 295 values in the range while the analytic value averages all 295. In our reading, T3’s reset comparison therefore averages different sets of lengths, and its gap is not purely the censoring effect. The anchor hypothesis failed for T3: the estimate was below the analytic value in 9 of 10 seeds, and in the tenth it was above it by 0.00051. The estimate ranges from 0.1052 to 0.1174 over the seeds against an analytic value of 0.1169, and its mean is 0.0055 below; in our reading, for this fast-failing tracker the censoring effect on the anchor estimate is smaller than the spread between seeds, since its mean survival of about $1/h \approx 62$ frames is short against sub-sequences that average 442 frames, so few sub-sequences end without a failure.

Code against report. The report’s normalization, a sum over $N_{hi} - N_{lo} + 1$ values divided by $N_{hi} - N_{lo}$, raises the EAO slightly when the curve covers the whole range: for the reset protocol by the factor $295/294$, which the printed identity confirms to $10^{-16}$ or better for T0, T1 and T2. For T3 this identity check failed (error 0.02): with fewer values in the mean than the range holds, the toolkit’s mean and the report’s sum are no longer in that ratio, and the report’s form gives the lower value, 0.13448 against 0.13631. This is the toolkit’s mean behaving as described under the length of the reset curve, not an error in the code. For the anchor protocol, the $j-1$ tail raises EAO by $9 \times 10^{-5}$ to $1.8 \times 10^{-4}$, at most 15% of the bound derived above: the hypothesis held. Including index $N_{hi}$ or using the report’s sum changes the anchor EAO in the fourth decimal. All of these are far smaller than the censoring gap.

Accuracy and robustness. The reset accuracy equals 0.7 exactly for every tracker and seed (this check held), because the code averages only frames in which the tracker was tracking. Robustness distinguishes the trackers: T3 fails 8.88 times per sequence with the toolkit’s length weighting but 6.06 times on a plain average over sequences, since longer sequences have more failures, and its AR robustness is $\exp(-30 \cdot 8.88/\bar L) = 0.521$. In the anchor protocol, the code’s accuracy is below 0.7 even for T0 (0.6984), because the initialization frame’s zero enters every run; leaving that frame out gives 0.7000 for T0 and 0.6997 to 0.6999 for the others, and the hypothesis that the code’s value lies below this, which lies at most at 0.7, held in every seed. The remaining shortfall below 0.7 comes, by construction, from the 25 to 77 runs per tracker (over the ten seeds) whose loss fell in their last 9 frames: the code does not detect the loss, so the zeros after it are counted as tracked frames. The anchor robustness, 0.80, 0.50 and 0.19, is the fraction of each sub-sequence covered before the failure.

Process. The plan, with its checks and hypotheses, was fixed before a pilot on three other seeds. Before writing it, one timing run on a further seed was made with its output discarded. After the pilot, the editor approved three printing changes: the length of each reset curve and the number of values each EAO averages, and labels marking T0’s rows as checks. The first run of the script with these changes stopped with an index error while printing the curves at 400 frames, because T3’s reset curve can be shorter than that. A printing repair, which shows “–” with the curve length where a curve ends early, was made and the script run again; apart from the header of that block, every line printed by both runs is identical, and the output above is from the repaired script. During fact-checking, the three direct-loop check functions, which had followed the toolkit’s code line by line, were rewritten from the description so that no toolkit code is reproduced; the printed output is unchanged, and the rewritten script is the code shown.

License

As of November 2026, the VOT pages read for this article state no licence for the datasets. The site’s footer reads “© 2013–2026. All rights reserved.” The datasets are not offered as files to download: each dataset page says the toolkit downloads the data when an evaluation workspace is configured for a stack, so the terms are those of that route, which was not used for this article. The sequences come from many sources: for VOT2015 the dataset page lists the ALOV dataset, the OTB2 dataset, non-tracking datasets, Computer Vision Online, Professor Bob Fisher’s image database, Videezy, the Center for Research in Computer Vision at the University of Central Florida, the NYU Center for Genomics and Systems Biology, Data Wrangling, the Open Access Directory and the Learning and Recognition in Vision group at INRIA. VOTS2023 drew on LaGOT, VOT-LT2021, VOT-LT2022, UTB180, VOT-ST2022 and TOTB (VOTS2023 report). Each source may carry its own terms, so the licensing of a given sequence should be checked with the organizers. The toolkit is licensed under GPL-3.0. VOTS submissions go through an evaluation server that requires registration (VOTS2023 report).

Limitations and Bias

• Small datasets. Sixty short-term sequences per edition from 2015 keep evaluation fast but leave each result dependent on few videos; VOT’s answer was a careful selection and the sequestered set rather than size (VOT2020 report).
• Estimator bias. As the example shows, the EAO estimate depends on how sequence ends censor successful segments, so it is lower than the expected average overlap it defines, by an amount that depends on the tracker and on the sequence lengths. EAO values are comparable within one stack, not across editions with different data and ranges (our reading).
• Protocol changes. Reset-based and anchor-based EAO, boxes and masks, and VOTS’s $Q$ are different measures on different data; results from different editions are not comparable. The VOT2020 report gives the reasons for its change: reset points depended on where a tracker failed and could in principle be exploited, and one tracker had been disqualified for inflating its box to avoid zero overlap.
• Annotation conventions. The rotated-box rule placed boxes to exclude parts such as outstretched arms and inferred hidden parts, so overlaps measure agreement with that convention; the dataset page and the report state the rule in different terms.
• Training-data overlap. Trackers are trained on large tracking datasets that may share content with the test sequences; VOT’s prohibited lists address this for named datasets only, a form of benchmark overfitting.

Common Uses

VOT’s datasets and toolkit are used to benchmark short-term trackers alongside OTB. Correlation filter and Siamese tracker papers report EAO on VOT2015, VOT2016 and VOT2018, as those articles’ tables show, and transformer trackers report the VOT2020 anchor-based EAO (Single Object Tracking). With masks as ground truth from 2020, the benchmark is also used by video object segmentation methods, which VOT itself began to evaluate. The toolkit and the published results of past challenges remain available for evaluating a new tracker against the official results (VOT2015 page).

Historical Milestones

• 2013: the first challenge, with 16 sequences and 27 trackers, at ICCV 2013 (challenges page).
• 2015: a 60-sequence dataset chosen by automatic clustering and selection, rotated-box annotations, the first thermal sub-challenge, and EAO, which replaced the averaged rank as the winning measure. MDNet won with an EAO of 0.38, accuracy 0.60 and 0.69 failures on average, ahead of DeepSRDCF (0.32) and EBT (0.31), among 62 trackers (VOT2015 report, Table 1).
• 2017: a sequestered dataset and a real-time experiment (challenges page).
• 2018: a long-term sub-challenge (challenges page).
• 2020: anchor-based evaluation, segmentation masks and the Python toolkit. On the public VOT-ST2020 data, RPT led with an EAO of 0.530 (accuracy 0.700, robustness 0.869) and won on the sequestered set; the video object segmentation method STM, added by the committee as a baseline, ranked 19th with an EAO of 0.308 and the second-highest accuracy, 0.751 (VOT2020 report, Table 8 and Sec. 4.1).
• 2023: VOTS, which no longer separates single- from multi-target or short- from long-term tracking. Its 144 sequences hold 341 targets, the ground truth is sequestered, and 47 valid entries were scored; the top performer, DMAOT, built on AOT, the VOT2022 winner (VOTS2023 report, Sec. 3).
• 2026: point tracking and referring tracking joined VOTS, and the series received the PAMI Mark Everingham Prize at ECCV 2026 “for the series of Visual Object Tracking workshops and challenges” (VOT website).

Related Datasets

OTB, LaSOT, GOT-10k, TrackingNet and UAV123 are the other single-object benchmarks, described with their protocols in Single Object Tracking; they evaluate without resets. MOTChallenge is the counterpart for multiple pedestrians with identities. DAVIS and YouTube-VOS are the video object segmentation benchmarks that VOT’s mask-based editions approach. VOTS2023 reused sequences from LaGOT, UTB180 and TOTB alongside VOT’s own long-term and short-term sets.

Resources

• VOT Challenge website: editions, datasets, results, publications and news.
• VOT toolkit (GPL-3.0): the Python toolkit and experiment stacks.
• VOTS2023 leaderboard: current results for the VOTS2023 benchmark, as linked from the VOTS2023 page.