Point Tracking (Tracking Any Point)

Point tracking, or tracking any point (TAP), asks where an arbitrary point on a physical surface goes over a whole video. The input is a video and a set of query points, each a pixel position in some frame; the output is, for every query and every frame, the point’s position and whether it is visible there. The point need not be a corner or lie on a named object, and the tracker must report when it is hidden and pick it up again when it reappears. The task sits between optical flow estimation, which is dense but covers only pairs of frames, and feature tracking, which is long-range but follows only points chosen because they are easy to track. It is also called long-range or long-term point tracking; Sand and Teller (2006) called the dense, long-range version a particle video, and the PointOdyssey authors (Zheng et al., 2023) call it fine-grained long-range tracking.

The papers disagree on who introduced the task. Doersch et al. (2022) write that they formalize the problem and name it “tracking any point”, together with the TAP-Vid benchmark, which gave learned point trackers a common evaluation (abstract, §6); TAPNext’s authors also credit TAP-Vid (Zholus et al., 2025, §2.1). CoTracker’s authors write that Particle Video introduced the problem (Karaev et al., 2024, §2), and CoTracker3’s that PIPs did (Karaev et al., 2025, §2).

Problem Definition

A video $V = (I_1, \dots, I_T)$ is given with $N$ queries $q_i = (t_i, x_i, y_i)$, each a pixel in frame $t_i$ that must be visible there. For each query the tracker outputs a trajectory $\hat{\mathbf{p}}_{i,t} = (\hat{x}_{i,t}, \hat{y}_{i,t})$ and an occlusion flag $\hat{o}_{i,t} \in \{0, 1\}$ for every frame $t$. The target is the same physical point on the same surface: on frames where it is occluded, the position output is “typically assumed to be meaningless”, in TAP-Vid’s words (§3), and is not scored.

The benchmark adds two conditions. The point and its surface must be determined unambiguously from a single pixel, which excludes transparent objects, liquids and gases (§1). And the output for one query should not depend on which other queries are passed with it, because other queries can leak how pixels are grouped and move; the official repository states this rule and notes that its evaluation code does not enforce it (tapnet tapvid/README.md). Queries may lie in any frame, so a tracker must in general track forwards and backwards in time.

Compared with optical flow, the unknown is a trajectory over $T$ frames rather than one displacement field between two, and occlusion is part of the output rather than a nuisance. Compared with object tracking, and with single-object tracking in particular, where one target is given as a box in the first frame, the target is a point with no extent, so a track says how a surface rotates and deforms, which a box cannot (TAP-Vid, Figure 2).

Input and Output

• Input: a video clip, usually resized to $256 \times 256$ pixels for TAP-Vid evaluation, and query points $(t, x, y)$. TAP-Vid stores queries in the order $(t, y, x)$ and positions as $(x, y)$ (tapnet evaluation_datasets.py).
• Output: an $N \times T \times 2$ array of positions and an $N \times T$ array of occlusion flags. Some trackers also output a confidence or uncertainty per frame, which is combined with the occlusion estimate before scoring (TAPIR, §3.1; CoTracker3, §4.1).

Relation to Optical Flow and Feature Tracking

The obvious way to track a point with optical flow is to integrate it: move the point by the flow at its current position, then repeat in the next frame,

$$\hat{\mathbf{p}}_{t+1} = \hat{\mathbf{p}}_t + \mathbf{w}_t(\hat{\mathbf{p}}_t),$$

where $\mathbf{w}_t$ is the flow from frame $t$ to $t+1$, sampled with bilinear interpolation. The flow can come from any two-frame method, classical or learned, such as FlowNet, PWC-Net or RAFT. TAP-Vid’s RAFT baseline does exactly this, and marks a point occluded only when it leaves the frame (supplement, §10). Two weaknesses follow from the form of the recursion. Each step’s error is carried into every later step, so the error of a chain is a sum of per-frame errors and drifts even while the point stays visible (TAP-Vid, §5.5). And during an occlusion, the flow at the point’s position describes the occluder, so the chain follows the occluder and has no way back to the true point when it reappears (PIPs, §5.3). Sand and Teller (2006) had already noted that concatenated flow fields accumulate error and must be refined to avoid drift (§1.2). PIPs’ authors point out the other side: modern flow is accurate enough that chains often accumulate little error, and it is occlusion that breaks them (§1). Dense trajectories for action recognition end each chained track after 15 frames for the same reason.

Feature tracking with the KLT tracker avoids drift partly by comparing each feature with its first appearance, but it selects corners where the Lucas–Kanade system is well conditioned and drops tracks at occlusion rather than recovering them. Feature matching between distant frames handles large motion but finds sparse keypoints, mostly on rigid scenes, and usually ignores occlusion because structure from motion can filter errors by geometry (TAP-Vid, §2). Point tracking asks for any point, on rigid or deforming surfaces, through occlusion, which none of these provide.

Particle Video. Sand and Teller (2006) represented a video by particles, image point samples with long trajectories, and set the goal of correspondences that are both spatially dense and temporally long-range. Their algorithm sweeps forwards and then backwards through the video. In each frame it propagates particles with frame-to-frame variational flow, links neighbouring particles through a constrained Delaunay triangulation, optimizes all particle positions under an energy with three terms, prunes particles with high energy and adds new ones in gaps (§3). The data term compares each particle with its appearance in the frame where it was created, the distortion term penalizes relative motion of linked particles (weighted down across motion boundaries), and the flow term ties trajectories to the flow (§4.1). Particles are not propagated into regions the flow marks as occluded, a particle is pruned to its longest run of low-energy frames, and new particles fill the gaps (§3.1, §4.3, §5), so a track ends at an occlusion and is not re-attached afterwards. PIPs’ authors later described their model as the only work aiming at amodal trajectories that do not end at occlusions (Harley et al., 2022, §2.1). Lacking ground truth, the authors evaluated on videos whose second half was replaced by a time-reversed copy of the first, so that surviving particles should return to their start, and warned that the measure could be gamed by pruning more particles (§6). They also predicted that rendered videos with known correspondences would be needed, which is how the field later trained and evaluated.

Variants

• Query sampling. In TAP-Vid’s strided mode every visible ground-truth point is queried every 5 frames, starting at frame 0, so one trajectory yields several queries and trackers must work in both directions. In query-first mode each trajectory is queried once, at its first visible frame, and only later frames are scored, which suits online trackers and is harder because the query is on average farther from the scored frames (TAP-Vid supplement, §8).
• Online and offline. Online (causal) trackers see frames in order and can run on streams; offline trackers see the whole clip. CoTracker runs causal trackers twice, on the video and its reverse, for the strided protocol (Karaev et al., 2024, §4). CoTracker3 comes in an online, sliding-window version and an offline version that processes the clip as one window (Karaev et al., 2025, §3.2).
• Independent and joint tracking. PIPs, TAP-Net and TAPIR track each query independently; CoTracker attends across tracks and adds support points that the user did not ask for, to give context (§3.4).
• Sparse and dense. Benchmarks score a few dozen points per video, but trackers are increasingly run on grids: CoTracker reports tracking a $263 \times 263$ grid on one 80 GB GPU (Table 6).
• 3D point tracking. The tapnet repository also hosts TAPVid-3D, which asks for metric 3D trajectories and, as of November 2026, contains more than 4,000 real videos (tapnet README.md).

Challenges

• Occlusion and re-identification. A hidden point leaves no evidence in the frames where it is hidden, so a tracker must search for it when it returns, without jumping to a similar-looking point.
• Drift against jitter. Frame-by-frame integration drifts; matching every frame against the query independently does not drift but gives jittery, temporally inconsistent tracks. TAPIR’s design starts from this trade-off between TAP-Net and PIPs (§1).
• Appearance change. Rotation, scale, deformation and lighting change the point’s surroundings, so a template from the query frame matches less well far from it.
• Weak texture. Points on uniform or rotationally symmetric surfaces are ambiguous. TAP-Vid’s authors call TAP-Vid-RGB-Stacking particularly challenging for this reason (TAP-Vid, §3.1), and TAPIR improved least there (§4.1).
• Training data. Real videos with point tracks are scarce and expensive: TAP-Vid’s annotators needed about 3.3 hours per 10-second clip (§4). Trackers are therefore trained on synthetic video, and their transfer to real footage is a recurring concern (TAPIR, §1; CoTracker3, §1).
• Long videos. Window-based trackers lose points occluded for longer than their window (PIPs, §5.7), and long clips stress memory.

Approaches

Chained flow and matching baselines

TAP-Vid adapted three methods built for other tasks as baselines (§5.3). RAFT’s two-frame flow was integrated from the query point as above. COTR, a transformer for correspondence between image pairs, was applied between the query frame and every other frame, with a cycle-consistency test for occlusion: a point is marked occluded if mapping back to the query frame lands more than 48 pixels from the query (supplement, §10). Kubric-VFS-Like, released with the Kubric generator, matches contrastively learned features with a spatial soft argmax and also detects occlusion by cycle consistency. All three performed poorly: chained RAFT accumulated errors and could not detect occlusion, and COTR failed on moving objects and textureless scenes (§5.5).

PIPs: a learned particle video

Persistent Independent Particles (PIPs; Harley et al., 2022) rebuilt the particle video with components from RAFT. A 2D convolutional network computes features at one-eighth resolution for each of $T = 8$ frames. The target’s feature is sampled at the query in the first frame and copied to all frames, and its position is copied too, a zero-velocity initialization (§3.3). In each of $K = 6$ iterations, PIPs correlates the per-frame target features with each frame’s feature map, crops $7 \times 7$ local score patches at four pyramid levels around the current positions, and feeds displacements, features and scores for all 8 frames to a 12-block MLP-Mixer, which outputs updates to all positions and features at once (§3.5, §4). A linear layer on the final features predicts visibility. Supervision is an $L_1$ loss on positions after every iteration, applied even when the target is occluded, which synthetic ground truth allows, plus cross-entropy on visibility (§3.6).

Because the 8 frames are processed together, PIPs can bridge an occlusion shorter than its window by using frames before and after it. Longer videos are handled by re-starting the 8-frame model from a late frame whose visibility score is high, keeping the original target feature, so that the model does not continue from a frame where it is tracking an occluder (§3.7). The authors trained it on FlyingThings++, 8-frame trajectories mined from FlyingThings by chaining ground-truth flow with forward–backward checks, onto which objects from other videos are pasted as synthetic occluders (§5.1). On that data’s test set, the mean trajectory error of PIPs was 15.54 pixels on mostly visible and 36.67 on mostly occluded trajectories, against 24.32 and 46.73 for chained RAFT (Table 1). On BADJA keypoint propagation in animal videos, PIPs reached an average PCK-T of 62.3 against 53.5 for DINO features with a spatial window (Table 4). Each particle is tracked without looking at its neighbours, a trade of spatial for temporal context that the authors list as a limitation (§5.7).

PIPs++ (Zheng et al., 2023) replaced the MLP-Mixer with a 1D residual network convolving over time, so the model is no longer tied to 8 frames, and added templates sampled along the estimated track a few frames back, keeping the first-frame template to prevent forgetting; it drops visibility estimation (§4.2–4.3).

TAP-Net: per-frame matching through a cost volume

TAP-Net, the baseline proposed with TAP-Vid, takes the opposite route (TAP-Vid supplement, §7). A TSM-ResNet-18, which shifts features in time only in its first two layers, computes a feature grid for the whole video at one-eighth resolution. The query’s feature is bilinearly sampled and compared by dot product with every location of every frame, a cost volume per query. A small network on each frame’s slice outputs an occlusion logit and a heatmap whose soft argmax, a weighted average of positions within a radius of the heatmap’s maximum, gives the position. Training uses a Huber loss on positions of visible points and cross-entropy on occlusion, on Kubric MOVi-E. Since each frame is matched independently, TAP-Net does not drift and recovers after occlusion, but its tracks are jittery and coarse because of the 8-pixel stride (TAPIR, §1, §3.1).

TAPIR: match, then refine

TAPIR (Doersch et al., 2023) combines the two. A TAP-Net-style global match on each frame initializes the track, including an occlusion estimate and an uncertainty estimate. Refinement then repeatedly compares the query feature with features in $7 \times 7$ neighbourhoods around the current track at several resolutions, down to a stride-4 feature map, and a temporal network of depthwise convolutions, translated from PIPs’ MLP-Mixer, updates positions, occlusion, uncertainty and the query feature for all frames (§3.2). Being convolutional in time, it handles sequences of any length without PIPs’ chunking.

The uncertainty estimate is aimed at the metric. Under Average Jaccard, a visible point predicted in the wrong place counts as both a false positive and a false negative, so it is better to call an uncertain point occluded. TAPIR predicts the probability $u_t$ that its position is farther than a threshold from the truth, and outputs “visible” only if $(1 - u_t)(1 - o_t) > 0.5$ (§3.1). The authors also changed the Kubric camera to pan, because models trained on cameras that always look at one point failed on real panning video (§3.3). TAPIR runs about 150 frames per second for 50 queries at $256 \times 256$ on one GPU, against 34.5 s for PIPs on the same clip, and the authors estimate that PIPs’ chained inference takes about a month on one V100 for TAP-Vid-Kinetics (§4.1–4.2). Removing iterative refinement lowered TAPIR’s DAVIS Average Jaccard from 61.3 to 41.6, and a “simple combination” of TAP-Net initialization with PIPs-style chunked refinement reached 50.0 (Tables 4 and 5).

CoTracker: tracking points together

CoTracker (Karaev et al., 2024) tracks points jointly. A transformer operates on a grid of tokens indexed by track and time, each token holding a track’s displacement, visibility, feature and RAFT-style correlation features at four scales; attention alternates between the time axis and the track axis, and a fixed number of learned proxy tokens make track attention linear in the number of tracks (§3.1–3.2). The model works on sliding windows of 8 frames that overlap by half and is trained unrolled over several windows, like a recurrent network (§3.3). Joint tracking lets visible points inform occluded ones: on single DAVIS query points, adding support points raised Average Jaccard in query-first mode from 55.6 to 62.2, and turning off unrolled training dropped it to 44.6 (Tables 4, 5 and 7). To respect TAP-Vid’s rule that queries must not see each other, the paper evaluates one benchmark point at a time, surrounded by a local and a global grid of support points (§4.2). The repository, whose README at the pinned commit documents CoTracker3, evaluates all target points jointly by default, which its README says gives numbers similar to the paper’s; the exact numbers need a single_point=True flag (co-tracker README.md). The authors list a limitation of training only on synthetic data: CoTracker tends to track shadows together with the objects that cast them (§4.3).

Later trackers

BootsTAPIR fine-tunes TAPIR on 15 million unlabelled real videos by self-training, as CoTracker3’s authors describe it (Karaev et al., 2025, §2); the BootsTAP paper was not read for this article. LocoTrack adds 4D correlation between neighbourhoods of the query and of the track point, and TAPTR casts tracking as DETR-style decoding of point queries (CoTracker3, §2, §3.2). CoTracker3 (Karaev et al., 2025) simplifies CoTracker: it keeps iterative transformer updates, track attention and proxy tokens, takes LocoTrack’s 4D correlation with a small MLP, drops global matching as redundant, and predicts a confidence alongside visibility (§3.2–3.4). Its main change is training: models pretrained on Kubric label real videos as teachers (CoTracker3 online and offline, CoTracker and TAPIR, one drawn at random per batch), SIFT keypoints choose the query points, and a student is fine-tuned on the pseudo-labels, using 15,000 videos in the main comparison against BootsTAPIR’s 15 million (§3.1, §4.1). TAPNext (Zholus et al., 2025) treats point tracking as sequential masked token decoding: image tokens and point tokens are processed by an off-the-shelf backbone that interleaves spatial transformer layers with recurrent state-space layers over time, so the model tracks causally, frame by frame, without cost volumes, local search windows, windowed inference or iterative refinement; its authors report state-of-the-art results among online and offline trackers (abstract, §1). As of November 2026, the tapnet repository lists TAPNext++, a TAPNext checkpoint fine-tuned on 1,024-frame synthetic sequences for longer tracking and re-detection (tapnet README.md).

Results as reported

Numbers below are comparable only within a table. Table entries are Average Jaccard (AJ), position accuracy $\delta^x_{\text{avg}}$ and occlusion accuracy (OA), in percent, on TAP-Vid at $256 \times 256$ in strided mode, as TAPIR’s Table 2 reports them, with TAPIR trained on its panning version of Kubric:

Method	Kinetics AJ	Kinetics $\delta^x_{\text{avg}}$	Kinetics OA	DAVIS AJ	DAVIS $\delta^x_{\text{avg}}$	DAVIS OA	RGB-Stacking AJ
COTR	19.0	38.8	57.4	35.4	51.3	80.2	6.8
Kubric-VFS-Like	40.5	59.0	80.0	33.1	48.5	79.4	57.9
RAFT (chained)	34.5	52.5	79.7	30.0	46.3	79.6	44.0
PIPs	35.3	54.8	77.4	42.0	59.4	82.1	37.3
TAP-Net	46.6	60.9	85.0	38.4	53.1	82.3	59.9
TAPIR	57.2	70.1	87.8	61.3	73.6	88.8	62.7

TAP-Vid’s own table marks COTR’s and PIPs’ Kinetics results as computed on a random subset, because the methods were too slow for the full set, and gives PIPs 32.8 AJ there (Table 3); its supplement says PIPs was run on 167 random Kinetics videos (§10). TAPIR’s table repeats COTR’s figure without the mark and gives PIPs 35.3 without saying how that figure was obtained. TAP-Net is stronger on Kinetics and PIPs on DAVIS, which TAP-Vid’s authors attribute to PIPs’ local search and smoothness priors suiting DAVIS’s continuous shots and failing on Kinetics’s cuts and camera shake (§5.5).

In query-first mode, CoTracker3’s Table 1 reports Average Jaccard as follows (Kub: trained on Kubric only):

Method	Training data	Kinetics	RGB-Stacking	DAVIS
TAPIR	Kub	49.6	55.5	56.2
CoTracker	Kub	49.6	67.4	61.8
LocoTrack	Kub	52.9	69.7	62.9
CoTracker3 online	Kub	54.1	71.1	64.5
CoTracker3 offline	Kub	53.5	74.0	63.3
BootsTAPIR	Kub + 15M real	54.6	70.8	61.4
CoTracker3 online	Kub + 15k real	55.8	71.7	63.8
CoTracker3 offline	Kub + 15k real	54.7	74.3	64.4

The CoTracker row differs slightly from the CoTracker paper’s own result (DAVIS 62.2), and CoTracker3’s paper does not say which checkpoint it ran. CoTracker’s paper reports TAPIR at 60.3 on RGB-Stacking in query-first mode, against 55.5 in TAPIR’s and CoTracker3’s tables, because it kept only every fifth frame of RGB-Stacking for evaluation (Table 1 caption). The same method can therefore score differently under different protocols, and gaps of one or two points between papers do not establish an ordering.

Datasets and Benchmarks

TAP-Vid

TAP-Vid (Doersch et al., 2022) has four parts (§3.1, Table 2):

• TAP-Vid-Kinetics: 1,189 videos from the Kinetics-700 validation set, 10-second clips resampled to 25 frames per second (250 frames), with 26.3 annotated points per video on average. The clips are unstructured YouTube videos with camera shake, cuts and multiple moving objects. The authors committed to removing videos taken down from YouTube (§6), and later counts differ: tapnet’s tapvid/README.md describes 1,000 Kinetics videos, and CoTracker3’s authors evaluated on 1,144 (§4).
• TAP-Vid-DAVIS: the 30 videos of the DAVIS 2017 validation set used for video object segmentation, 34 to 104 frames long, annotated at 1080p with up to 5 objects and 5 points per object (21.7 points per video on average).
• TAP-Vid-Kubric: synthetic videos of the MOVi-E dataset made with the Kubric generator, about 20 objects dropped into a scene with physics simulation and ray-traced rendering, 24 frames at $256 \times 256$; 38,325 training videos and 799 for validation. Ground truth comes from the simulator, and any number of points can be sampled.
• TAP-Vid-RGB-Stacking: 50 simulated robot-stacking episodes of 250 frames with 30 points each, 20 on moving and 10 on static parts. Its objects are textureless and rotationally symmetric, and the gripper often occludes them.

The authors intend Kubric for training and the other three for testing only, recommend DAVIS for validation when many experiments are run, and discourage automated tuning on any held-out set (§4.3). Real tracks were annotated by a stable pool of 15 annotators in three stages: choose objects, click points on every other frame, and have a second annotator correct each track until no one disagrees (§4). A track assist tool computes RAFT flow for the whole video and, between two clicked points, finds the path that best follows the flow,

$$\arg\min_{\rho \in P_{s:t}} \sum_{i=s}^{t-1} \left\lVert (\rho_{i+1} - \rho_i) - F(\rho_i) \right\rVert^2 \quad \text{s.t. } \rho_s = \mathbf{p}_s,\ \rho_t = \mathbf{p}_t,$$

solved as a shortest path over pixels with a form of Dijkstra’s algorithm (§4.2; the paper writes the flow term as $F(p_i)$). Annotators clicked on average 10.2 points per visible track segment, which held about 57 annotated points, and the tool fell back to linear interpolation for clicks fewer than 5 frames apart. On 10 Kubric videos with known ground truth, assisted annotation put 99% of points within 8 pixels, 96% within 4 and 87% within 2, and on DAVIS two independent annotators agreed on occlusion 95.5% of the time and on location within 4 pixels 92.5% of the time, with a mean difference of 1.46 pixels at $256 \times 256$ (§5.2). Flow assists the annotation here but does not replace it: annotators were instructed to check every interpolated track and split those that drifted.

Other benchmarks

• JHMDB and BADJA. Before TAP-Vid, class-agnostic point trackers were evaluated by propagating human joints in JHMDB or animal keypoints in BADJA. TAP-Vid’s authors argue that joints lie inside the body rather than on its surface, are annotated even when occluded, and cover one class; fine-tuning TAP-Net on JHMDB lowered its Kinetics Average Jaccard from 46.6 to 36.4 (§5.6).
• PointOdyssey (Zheng et al., 2023): 104 synthetic videos averaging 2,000 frames, with characters animated from motion capture and camera paths mined from real videos, built for long-term tracking. It reports $\delta_{\text{avg}}$, median trajectory error and a survival rate, the average fraction of the video before the error first exceeds 50 pixels at $256 \times 256$ (§5.2).
• Dynamic Replica and RoboTAP. CoTracker and CoTracker3 also report Dynamic Replica, synthetic 300-frame sequences of articulated models where positions of occluded points are known and scored as $\delta^{\text{occ}}_{\text{avg}}$, and RoboTAP, 265 real robot-manipulation videos annotated in TAP-Vid’s format (CoTracker3, §4; tapnet README.md).

Evaluation Metrics

TAP-Vid scores position and occlusion together (§4.3). Let $E$ be the set of scored frames of a query, $v_t$ and $\hat{v}_t$ the ground-truth and predicted visibility, and $d_t = \lVert \hat{\mathbf{p}}_t - \mathbf{p}_t \rVert$ the position error after both are scaled to a $256 \times 256$ frame. For a threshold $\delta$ in pixels:

$$\mathrm{OA} = \frac{1}{|E|} \sum_{t \in E} \mathbb{1}[\hat{v}_t = v_t], \qquad \delta^x(\delta) = \frac{\sum_{t \in E} v_t \, \mathbb{1}[d_t < \delta]}{\sum_{t \in E} v_t},$$ $$\mathrm{TP} = \sum_{t \in E} v_t \hat{v}_t \mathbb{1}[d_t < \delta], \quad \mathrm{FP} = \sum_{t \in E} \hat{v}_t \left(1 - v_t \mathbb{1}[d_t < \delta]\right), \quad \mathrm{FN} = \sum_{t \in E} v_t \left(1 - \hat{v}_t \mathbb{1}[d_t < \delta]\right),$$

and $J(\delta) = \mathrm{TP} / (\mathrm{TP} + \mathrm{FP} + \mathrm{FN})$. Occlusion accuracy (OA) is plain classification accuracy of the visibility flag. Position accuracy $\delta^x$ is a PCK-style fraction computed only on frames where the ground truth is visible, ignoring the predicted flag, with the threshold in pixels of the resized frame rather than relative to an object’s size. The Jaccard measure is the intersection over union of predicted-visible-and-correct with ground-truth-visible frames. $\delta^x_{\text{avg}}$ and Average Jaccard (AJ) average over $\delta \in \{1, 2, 4, 8, 16\}$. CoTracker and CoTracker3 write $\delta^{\text{vis}}_{\text{avg}}$ for TAP-Vid’s $\delta^x_{\text{avg}}$.

The official implementation, compute_tapvid_metrics in tapnet’s evaluation_datasets.py, fixes details the paper leaves implicit:

• Scored frames. The query frame itself is never scored. In strided mode all other frames are; in query-first mode only frames after the query frame, since the tracker may assume the point is occluded before it first appears and the benchmark does not reward hard-coding that.
• Occluded ground truth. Frames where the ground truth is occluded do not enter $\delta^x$. In the Jaccard measure they count only as false positives when the tracker calls them visible. In OA they count like any other frame.
• Thresholds are strict: a point is within $\delta$ only if its squared distance is less than $\delta^2$.
• Resolution. The readers resize every video to $256 \times 256$ and multiply ground-truth coordinates, stored normalized to $[0, 1]$, by 256 on each axis. For non-square video the scaling is anisotropic, so a threshold of 1 pixel corresponds to different distances along $x$ and $y$ in the original frame. The README encourages running trackers on full-resolution video with predictions scaled to $256 \times 256$ for scoring, but treats such results as a separate category.
• Averaging. Measures are computed per video and averaged over videos, so a short video counts as much as a long one.

The code has changed. Until July 2023, query-first evaluation removed earlier frames with a loop over the batch axis instead of the query axis; since the batch held one video, frames before the query frame were still scored. A user reported this, a maintainer replied that the effect should be very minor because few points are first visible after frame 0, and commit 3c12755 of 13 July 2023 replaced the loop with the indexing above (tapnet issue #34). The bug had been copied into CoTracker’s evaluation code and was fixed there in October 2023 (co-tracker issue #20). CoTracker’s copy at the pinned commit still divides OA by the number of scored frames summed over the whole batch rather than per video, which is the same for the usual batch of one video, and its dataset reader scales the normalized coordinates by 255 rather than 256 (tap_vid_datasets.py). Query-first numbers computed with the earlier code scored those frames too.

PointOdyssey’s survival rate and median trajectory error, and the $\delta^{\text{occ}}_{\text{avg}}$ of synthetic sets with known occluded positions, extend the measures to long videos and to occluded points, which real annotations cannot score.

Example: chained flow and direct matching scored by TAP-Vid

The example builds a 40-frame synthetic video of $288 \times 384$ pixels with known trajectories and occlusion, runs two simple trackers and scores them with TAP-Vid’s measures. The background is scikit-image’s bundled gravel texture, panned by up to 24 pixels; a $96 \times 96$ patch of astronaut moves across it while rotating by 30° and growing by 20%; and a 64-pixel-wide bar of brick sweeps from right to left on top of everything, so that every track is occluded at some time. Of 28 tracks, 13 start on the visible background, 3 start hidden behind the object, and 12 lie on the object. Ground truth follows from the rendering: background points move with the pan, object points with the object’s rigid motion, and a point is occluded where a higher layer covers it. Before the first run, a separate check (not shown) sampled the frames at the ground-truth positions to confirm that visible frames show the source texture there and bar-occluded frames show the bar’s texture.

The two trackers are:

• chained flow: dense TV-L1 optical flow between consecutive frames, integrated from the query point with bilinear interpolation as TAP-Vid’s RAFT baseline integrates RAFT, forwards and, for strided queries, backwards. The sources leave occlusion handling open, so two rules are printed: never occluded (the RAFT baseline’s rule, since no point leaves the frame), and occluded when a forward–backward check fails by more than 1 pixel, the threshold used in Feature Tracking’s example;
• direct matching: the $21 \times 21$ patch around the query, taken from the query frame, is located in every other frame by normalized cross-correlation over the whole frame, refined to sub-pixel accuracy by a parabola through the peak, and called visible when the peak correlation reaches a threshold $\tau$. The threshold is a free choice, so three values are printed.

A static tracker that never moves and never reports occlusion gives a floor. The measures are tapnet’s compute_tapvid_metrics, ported line for line from the pinned commit with an option to apply the pre-July-2023 rule; queries follow sample_queries_first and sample_queries_strided, and coordinates are scaled to $256 \times 256$ as the readers do. Versions: NumPy 2.3.5, SciPy 1.16.3, scikit-image 0.25.2. The script is deterministic (two runs gave identical output) and takes 70 to 80 seconds on a laptop CPU, most of it in the 78 TV-L1 flow fields.

import numpy as np
from scipy import ndimage
from skimage import data
from skimage.feature import match_template
from skimage.registration import optical_flow_tvl1

# ---------------------------------------------------------------- synthetic video
H, W, T = 288, 384, 40
rng = np.random.default_rng(0)
gravel = data.gravel() / 255.0
patch = data.astronaut().mean(axis=2)[70:166, 190:286] / 255.0   # 96 x 96 object
brick = data.brick() / 255.0
yy, xx = np.mgrid[0:H, 0:W].astype(float)

def pan(t):        # camera pan: offset of the background texture (x, y)
    return np.array([24 * np.sin(2 * np.pi * t / 40), 12 * np.sin(2 * np.pi * t / 30)])

def obj_pose(t):   # object centre (x, y), rotation (rad), scale
    s = t / (T - 1)
    c = np.array([110 + 160 * s, 150 + 50 * np.sin(np.pi * s)])
    return c, np.deg2rad(30 * s), 1.0 + 0.2 * s

def bar_left(t):   # left edge of the 64-pixel occluding bar
    return 400.0 - 10 * t

def to_object(t, x, y):          # frame coordinates -> object coordinates
    c, th, s = obj_pose(t)
    dx, dy = x - c[0], y - c[1]
    return (np.cos(th) * dx + np.sin(th) * dy) / s, (-np.sin(th) * dx + np.cos(th) * dy) / s

def in_object(t, x, y):
    ox, oy = to_object(t, x, y)
    return (np.abs(ox) < 47.5) & (np.abs(oy) < 47.5)

def in_bar(t, x):
    return (x >= bar_left(t)) & (x < bar_left(t) + 64)

def sample(img, x, y):
    return ndimage.map_coordinates(img, [y, x], order=1, mode="nearest")

def render(t):
    b = pan(t)
    frame = sample(gravel, xx + b[0] + 64, yy + b[1] + 112)
    ox, oy = to_object(t, xx, yy)
    m = in_object(t, xx, yy)
    frame[m] = sample(patch, ox[m] + 47.5, oy[m] + 47.5)
    m = in_bar(t, xx)
    frame[m] = sample(brick, xx[m] - bar_left(t) + 100, yy[m] + 50)
    return frame + rng.normal(0, 0.01, frame.shape)

video = np.stack([render(t) for t in range(T)])

# ---------------------------------------------------------------- ground-truth tracks
def textured(img, x, y, r=5):
    return img[int(y) - r:int(y) + r + 1, int(x) - r:int(x) + r + 1].std() > 0.08

bg, under = [], []
while len(bg) < 13 or len(under) < 3:              # background points (frame-0 positions)
    x, y = rng.uniform(40, W - 40), rng.uniform(30, H - 30)
    if not textured(video[0], x, y):
        continue
    if in_object(0, x, y) and len(under) < 3:
        under.append((x, y))                       # hidden behind the object at t = 0
    elif not in_object(0, x, y) and len(bg) < 13:
        bg.append((x, y))
obj = []
while len(obj) < 12:                               # object points (object coordinates)
    ox, oy = rng.uniform(-40, 40, 2)
    if patch[int(oy + 47.5) - 4:int(oy + 47.5) + 5, int(ox + 47.5) - 4:int(ox + 47.5) + 5].std() > 0.06:
        obj.append((ox, oy))

N = 28
gt = np.zeros((N, T, 2))
gt_occ = np.zeros((N, T), bool)
for t in range(T):
    d = pan(t) - pan(0)
    c, th, s = obj_pose(t)
    for i, (x, y) in enumerate(bg + under):        # background moves with the pan
        gt[i, t] = (x - d[0], y - d[1])
        gt_occ[i, t] = in_object(t, *gt[i, t]) | in_bar(t, gt[i, t, 0])
    for j, (ox, oy) in enumerate(obj):             # object points move rigidly with scale
        px = c[0] + s * (np.cos(th) * ox - np.sin(th) * oy)
        py = c[1] + s * (np.sin(th) * ox + np.cos(th) * oy)
        gt[16 + j, t] = (px, py)
        gt_occ[16 + j, t] = in_bar(t, px)

# ---------------------------------------------------------------- TAP-Vid measures
# compute_tapvid_metrics ported from google-deepmind/tapnet, tapnet/tapvid/evaluation_datasets.py
# at commit ceedd60 (Apache-2.0); logic unchanged, comments shortened.
def compute_tapvid_metrics(query_points, gt_occluded, gt_tracks, pred_occluded, pred_tracks,
                           query_mode, get_trackwise_metrics=False, old_first_mask=False):
    summing_axis = (2,) if get_trackwise_metrics else (1, 2)
    metrics = {}
    eye = np.eye(gt_tracks.shape[2], dtype=np.int32)
    if query_mode == "first":
        query_frame_to_eval_frames = np.cumsum(eye, axis=1) - eye    # frames after the query
    elif query_mode == "strided":
        query_frame_to_eval_frames = 1 - eye                          # all frames but the query
    else:
        raise ValueError("Unknown query mode " + query_mode)
    query_frame = np.round(query_points[..., 0]).astype(np.int32)
    evaluation_points = query_frame_to_eval_frames[query_frame] > 0
    if old_first_mask:   # the rule before tapnet commit 3c12755 (13 July 2023), for comparison only
        evaluation_points = np.eye(gt_tracks.shape[2])[query_frame] == 0
        if query_mode == "first":
            for i in range(gt_occluded.shape[0]):
                index = np.where(gt_occluded[i] == 0)[0][0]
                evaluation_points[i, :index] = False
    occ_acc = np.sum(np.equal(pred_occluded, gt_occluded) & evaluation_points,
                     axis=summing_axis) / np.sum(evaluation_points, axis=summing_axis)
    metrics["occlusion_accuracy"] = occ_acc
    visible = np.logical_not(gt_occluded)
    pred_visible = np.logical_not(pred_occluded)
    all_frac_within, all_jaccard = [], []
    for thresh in [1, 2, 4, 8, 16]:
        within_dist = np.sum(np.square(pred_tracks - gt_tracks), axis=-1) < np.square(thresh)
        is_correct = np.logical_and(within_dist, visible)
        count_correct = np.sum(is_correct & evaluation_points, axis=summing_axis)
        count_visible_points = np.sum(visible & evaluation_points, axis=summing_axis)
        frac_correct = count_correct / count_visible_points
        metrics["pts_within_" + str(thresh)] = frac_correct
        all_frac_within.append(frac_correct)
        true_positives = np.sum(is_correct & pred_visible & evaluation_points, axis=summing_axis)
        gt_positives = np.sum(visible & evaluation_points, axis=summing_axis)
        false_positives = (~visible) & pred_visible
        false_positives = false_positives | ((~within_dist) & pred_visible)
        false_positives = np.sum(false_positives & evaluation_points, axis=summing_axis)
        jaccard = true_positives / (gt_positives + false_positives)
        metrics["jaccard_" + str(thresh)] = jaccard
        all_jaccard.append(jaccard)
    metrics["average_jaccard"] = np.mean(np.stack(all_jaccard, axis=1), axis=1)
    metrics["average_pts_within_thresh"] = np.mean(np.stack(all_frac_within, axis=1), axis=1)
    return metrics

def queries(mode):
    """(track, query frame) pairs as tapnet's sample_queries_first / sample_queries_strided."""
    if mode == "first":
        return [(i, int(np.argmax(~gt_occ[i]))) for i in range(N)]
    return [(i, t) for t in range(0, T, 5) for i in range(N) if not gt_occ[i, t]]

SX, SY = 256 / W, 256 / H          # TAP-Vid scores in a 256 x 256 frame
def evaluate(qs, pred_pos, pred_occ, mode, scale=True, old=False):
    sc = np.array([SX, SY]) if scale else np.ones(2)
    idx = [i for i, _ in qs]
    qp = np.array([[t, gt[i, t, 1] * sc[1], gt[i, t, 0] * sc[0]] for i, t in qs])
    return compute_tapvid_metrics(qp[None], gt_occ[idx][None], (gt[idx] * sc)[None],
                                  pred_occ[None], (pred_pos * sc)[None], mode, old_first_mask=old)

# ---------------------------------------------------------------- trackers
fwd = [optical_flow_tvl1(video[t], video[t + 1]) for t in range(T - 1)]   # (v, u): t -> t+1
bwd = [optical_flow_tvl1(video[t + 1], video[t]) for t in range(T - 1)]   # t+1 -> t

def flow_at(f, x, y):
    return np.array([sample(f[1], np.array([x]), np.array([y]))[0],
                     sample(f[0], np.array([x]), np.array([y]))[0]])

def track_flow(i, q, both_ways, fb_check):
    """Chain two-frame flow from the query; optionally flag occlusion by a forward-backward check."""
    pos, occ = np.tile(gt[i, q], (T, 1)), np.zeros(T, bool)
    steps = [(range(q, T - 1), 1, fwd, bwd)] + ([(range(q, 0, -1), -1, bwd, fwd)] if both_ways else [])
    for frames, d, there, back in steps:
        for t in frames:
            k = t if d == 1 else t - 1
            w = flow_at(there[k], *pos[t])
            nxt = np.clip(pos[t] + w, [0, 0], [W - 1, H - 1])
            pos[t + d] = nxt
            if fb_check:
                occ[t + d] = np.linalg.norm(w + flow_at(back[k], *nxt)) > 1.0
    if not both_ways:
        occ[:q] = True                     # 'first' queries track forward only
    return pos, occ

R = 10                                     # 21 x 21 template
dy, dx = np.mgrid[-R:R + 1, -R:R + 1].astype(float)
def track_match(i, q, both_ways):
    """Match the query frame's patch in every frame by NCC; return positions and peak scores."""
    x0, y0 = gt[i, q]
    tpl = sample(video[q], (x0 + dx).ravel(), (y0 + dy).ravel()).reshape(dx.shape)
    pos, score = np.tile(gt[i, q], (T, 1)), np.full(T, 1.0)
    for t in range(T):
        if t == q or (t < q and not both_ways):
            if t < q:
                score[t] = -1.0            # before a 'first' query: occluded
            continue
        r = match_template(video[t], tpl, pad_input=True)
        py, px = np.unravel_index(np.argmax(r), r.shape)
        off = []
        for a, b, c in ((r[py, px - 1] if px > 0 else r[py, px], r[py, px], r[py, px + 1] if px < W - 1 else r[py, px]),
                        (r[py - 1, px] if py > 0 else r[py, px], r[py, px], r[py + 1, px] if py < H - 1 else r[py, px])):
            den = a - 2 * b + c
            off.append(0.5 * (a - c) / den if den < 0 else 0.0)   # parabola through the peak
        pos[t], score[t] = (px + off[0], py + off[1]), r[py, px]
    return pos, score

# ---------------------------------------------------------------- checks of the port
qs = queries("strided")
idx = [i for i, _ in qs]
m = evaluate(qs, gt[idx], gt_occ[idx], "strided")
print("oracle: AJ %.3f  delta_avg %.3f  OA %.3f" % (m["average_jaccard"][0], m["average_pts_within_thresh"][0],
                                                   m["occlusion_accuracy"][0]))
shifted = gt[idx] + np.array([2.0 / SX, 0.0])      # exactly 2 px to the right in the 256 x 256 frame
m = evaluate(qs, shifted, gt_occ[idx], "strided")
print("offset of exactly 2 px: within 1/2/4 =", [round(float(m["pts_within_%d" % k][0]), 3) for k in (1, 2, 4)],
      " delta_avg %.3f" % m["average_pts_within_thresh"][0])
print("queries: first %d, strided %d; occluded fraction of ground truth %.3f"
      % (len(queries("first")), len(qs), gt_occ.mean()))

# ---------------------------------------------------------------- results
results = {}
for mode in ("first", "strided"):
    qs = queries(mode)
    both = mode == "strided"
    stat_pos = np.array([np.tile(gt[i, q], (T, 1)) for i, q in qs])
    stat_occ = np.array([np.arange(T) < q if not both else np.zeros(T, bool) for i, q in qs])
    runs = {"static": (stat_pos, stat_occ)}
    for name, fb in (("flow, no occlusion", False), ("flow, fwd-bwd check", True)):
        out = [track_flow(i, q, both, fb) for i, q in qs]
        runs[name] = (np.array([o[0] for o in out]), np.array([o[1] for o in out]))
    out = [track_match(i, q, both) for i, q in qs]
    mpos, mscore = np.array([o[0] for o in out]), np.array([o[1] for o in out])
    for tau in (0.5, 0.65, 0.8):
        runs["match, tau=%.2f" % tau] = (mpos, mscore < tau)
    results[mode] = (qs, runs)
    print(f"\nquery mode: {mode} ({len(qs)} queries)")
    print(f"{'tracker':22s} {'AJ':>6s} {'d_avg':>6s} {'OA':>6s} {'d<1':>6s} {'d<16':>6s}")
    for name, (p, o) in runs.items():
        m = evaluate(qs, p, o, mode)
        print(f"{name:22s} " + " ".join("%6.3f" % m[k][0] for k in
              ("average_jaccard", "average_pts_within_thresh", "occlusion_accuracy", "pts_within_1", "pts_within_16")))

print("\nfirst mode, scored in native pixels instead of 256 x 256:")
qs, runs = results["first"]
for name in ("flow, fwd-bwd check", "match, tau=0.65"):
    a, b = (evaluate(qs, *runs[name], "first", scale=s) for s in (True, False))
    print(f"{name:22s} AJ {a['average_jaccard'][0]:.3f} -> {b['average_jaccard'][0]:.3f}   "
          f"d_avg {a['average_pts_within_thresh'][0]:.3f} -> {b['average_pts_within_thresh'][0]:.3f}")

print("\nfirst mode, evaluation mask before / after tapnet's July 2023 fix:")
for name in ("static", "flow, fwd-bwd check", "match, tau=0.65"):
    a, b = (evaluate(qs, *runs[name], "first", old=o) for o in (True, False))
    print(f"{name:22s} OA {a['occlusion_accuracy'][0]:.3f} / {b['occlusion_accuracy'][0]:.3f}   "
          f"AJ {a['average_jaccard'][0]:.3f} / {b['average_jaccard'][0]:.3f}")

Output:

oracle: AJ 1.000  delta_avg 1.000  OA 1.000
offset of exactly 2 px: within 1/2/4 = [0.0, 0.015, 1.0]  delta_avg 0.603
queries: first 28, strided 175; occluded fraction of ground truth 0.214

query mode: first (28 queries)
tracker                    AJ  d_avg     OA    d<1   d<16
static                  0.053  0.101  0.791  0.000  0.413
flow, no occlusion      0.364  0.604  0.791  0.556  0.622
flow, fwd-bwd check     0.392  0.604  0.836  0.556  0.622
match, tau=0.50         0.444  0.695  0.791  0.626  0.743
match, tau=0.65         0.479  0.695  0.820  0.626  0.743
match, tau=0.80         0.531  0.695  0.782  0.626  0.743

query mode: strided (175 queries)
tracker                    AJ  d_avg     OA    d<1   d<16
static                  0.055  0.109  0.821  0.002  0.331
flow, no occlusion      0.361  0.589  0.821  0.568  0.598
flow, fwd-bwd check     0.393  0.589  0.852  0.568  0.598
match, tau=0.50         0.589  0.820  0.821  0.740  0.875
match, tau=0.65         0.623  0.820  0.844  0.740  0.875
match, tau=0.80         0.673  0.820  0.856  0.740  0.875

first mode, scored in native pixels instead of 256 x 256:
flow, fwd-bwd check    AJ 0.392 -> 0.382   d_avg 0.604 -> 0.592
match, tau=0.65        AJ 0.479 -> 0.467   d_avg 0.695 -> 0.683

first mode, evaluation mask before / after tapnet's July 2023 fix:
static                 OA 0.794 / 0.791   AJ 0.053 / 0.053
flow, fwd-bwd check    OA 0.838 / 0.836   AJ 0.392 / 0.392
match, tau=0.65        OA 0.822 / 0.820   AJ 0.479 / 0.479

The checks confirm the port’s conventions. The oracle scores 1 everywhere. An offset of exactly 2 pixels fails the 2-pixel test, because the comparison is strict, except for 1.5% of points. The shift is 3 native pixels along $x$, exactly 2 after scaling by $256/384$, but the shifted and the true coordinate are scaled separately and each product is rounded, so for some $x$ the difference $(x + 3) \cdot 256/384 - x \cdot 256/384$ comes out a few times $10^{-14}$ below 2 in double precision (our arithmetic, which a reader can check on any array of coordinates). Results at a threshold boundary depend on arithmetic, not on the tracker.

Direct matching beats chained flow on every measure except occlusion accuracy, and the columns show why. Both trackers place about 55–63% of visible points within 1 pixel in query-first mode, but only 62% of chained-flow points are within 16 pixels, against 74% for matching. Chained flow’s errors are rarely moderate: points are either within a pixel or more than 16 pixels off, which fits the mechanism TAP-Vid and PIPs describe, a chain that follows the occluder once the bar passes over its point and has no way back. Matching searches the whole frame and can find the point again when it reappears; its misses are, in our reading, frames where the rotated and enlarged object no longer resembles the query patch, or where gravel offers a similar patch elsewhere. Matching gains much more from strided queries (AJ 0.623 against 0.479 at $\tau = 0.65$) than chained flow (0.393 against 0.392). Our explanation: a query closer in time to each scored frame means less rotation and scaling between template and target, while every chain still has to cross the bar’s sweep in one direction or the other.

The rows for chained flow without occlusion and for the static tracker share their occlusion accuracy, 0.791 in query-first mode: a tracker that never reports occlusion scores the fraction of visible frames, so OA alone rewards nothing until occlusion is predicted. The forward–backward check raises chained flow’s OA and AJ without moving any point. For matching, raising $\tau$ from 0.5 to 0.8 raises AJ from 0.444 to 0.531 in query-first mode even though OA falls at 0.8. By the definition of the Jaccard measure, AJ can rise only if most frames newly called occluded were false positives, wrong matches or frames where the point is hidden; for a wrong match on a visible point, calling it occluded turns a false positive and a false negative into a false negative only. This is the effect TAPIR’s uncertainty estimate exploits: under Average Jaccard it pays to declare an uncertain point occluded, even at some cost in occlusion accuracy.

Scoring in native pixels instead of the $256 \times 256$ frame lowers both trackers by about 0.01: thresholds become stricter by a factor of 1.5 along $x$ and 1.125 along $y$, but the output shows that only 7% of chained-flow points and 12% of matched points have errors between 1 and 16 pixels, so few cross a threshold. On footage with many errors of a few pixels the difference would be larger, which is why results at another resolution are not comparable. The pre-July-2023 evaluation mask raises OA by 0.002 to 0.003 and leaves AJ unchanged: only the three tracks first visible after frame 0 gain extra scored frames, all of them occluded in the ground truth and in the prediction, which count as correct for OA and enter neither the numerator nor the denominator of the Jaccard measure. This agrees with the maintainer’s expectation that the effect was very minor.

The example does not test a learned tracker; it shows what the measures reward. Chained flow and per-frame matching fail in opposite ways, chaining by never recovering and matching by losing precision as appearance drifts from the query frame, which is the trade-off that TAPIR’s match-then-refine design and CoTracker’s joint tracking address.

Applications

The TAP-Vid authors expected point tracks to help structure from motion handle dynamic and deformable objects, and to let robotic manipulation methods built on semantic keypoints work with arbitrary objects (§6). TAPNext’s authors list applications that appeared within three years of the task’s formalization: robotics, action recognition, 3D and dynamic reconstruction, controllable video generation and editing, zoology and medicine (§1).

• Robotics: RoboTAP uses TAPIR tracks to carry out manipulation tasks by imitation in the real world (tapnet README.md); see robotic manipulation.
• 3D reconstruction: VGGSfM builds a differentiable structure-from-motion pipeline on a learned point tracker, as the structure from motion article describes, and VGGT predicts point tracks alongside cameras and depth (3D reconstruction).
• Video synthesis: TAPIR’s authors trained a proof-of-concept diffusion model that generates trajectories from a single image, producing plausible animations (TAPIR, abstract).
• Annotation and editing: dense long-range correspondences can propagate edits, masks and labels through a video, as Sand and Teller anticipated for matting, rotoscoping and object removal (2006, §1). CoTracker’s authors note that tracking shadows with their objects may suit video editing but not motion analysis (§4.3).

Open Problems

• Sim-to-real. Most trackers are trained on Kubric’s rigid objects falling on a floor. BootsTAPIR and CoTracker3 fine-tune on real videos without labels, by self-training and by pseudo-labels from teacher trackers; how much real data is needed, and what students learn from noisy teachers, is still being measured (CoTracker3, §1).
• Long occlusions and long videos. Window-based models can lose points hidden for longer than a window, and real benchmarks are short: TAP-Vid’s longest clips are 250 frames, while PointOdyssey’s synthetic videos average 2,000.
• Textureless and symmetric surfaces. RGB-Stacking remains hard, and TAPIR gained least there (§4.1).
• Query leakage and fair evaluation. Joint trackers can benefit from seeing several benchmark points on the same object; the benchmark forbids it, but the code does not enforce it, so evaluation protocols must be stated (tapnet tapvid/README.md; CoTracker, §4.2).
• Evaluation limits. Real annotations cover only points humans can track and cannot score positions during occlusion; scoring at $256 \times 256$ hides sub-pixel precision at high resolution; and the YouTube-based Kinetics subset is reported with different video counts (1,189, 1,000 and 1,144), so results on it may not refer to the same data.
• What a point is. TAP excludes liquids, gases and transparent surfaces (TAP-Vid, §6), and trained trackers can follow a shadow together with the object that casts it (CoTracker, §4.3).