Siamese Trackers

Siamese trackers follow a single object through a video by comparing an image of the target, the exemplar or template, with a larger search region of each new frame, using one network applied to both inputs with shared weights. The network is trained offline on many pairs of frames from annotated videos, so at test time it can locate an object it has never seen from a single box in the first frame, without updating its weights. They are the main deep-learning family of object tracking methods for single targets between the correlation filter trackers of the early 2010s and the transformer trackers that followed.

The family is named after the Siamese networks of metric learning: two identical branches that embed two inputs with the same function so that their outputs can be compared. SiamFC (Bertinetto et al., 2016) made the comparison fully convolutional: the template’s embedding is cross-correlated with the search region’s embedding, which scores every candidate position in one pass. SiamRPN (Li et al., 2018) added a region proposal head that regresses boxes, DaSiamRPN (Zhu et al., 2018) changed the training data to teach the embedding to reject similar-looking objects, SiamRPN++ (Li et al., 2019) found why deep backbones had failed in these trackers and fixed it, and SiamMask (Wang et al., 2019) added a segmentation branch. This article covers the shared design, these models, the translation-invariance argument behind SiamRPN++ and the centre bias it explains.

The SiamFC paper itself was published in the Springer workshop proceedings, which could not be read for this article. Its design is described here from the authors’ own later paper (CFNet; Valmadre et al., 2017), from SiamRPN, DaSiamRPN, SiamRPN++ and SiamMask, which restate it, and from the authors’ released code; each claim names its source.

Core Idea

A single-object tracker receives a box around an arbitrary object in the first frame and must find that object in every later frame (the task is described in Object Tracking, and its benchmarks and protocols in Single Object Tracking). Before Siamese trackers, the main approaches learned a model of the target online, from the first frame onwards: correlation filters retrained a linear template every frame in the Fourier domain, and deep trackers adapted a pretrained network to the target by stochastic gradient descent, which CFNet’s authors describe as a difficult learning problem with very little data and expensive for online adaptation (Valmadre et al., 2017, Sec. 1); GOTURN’s authors list deep trackers running between 0.8 and 15 frames per second, with the best performing at about 1 frame per second on a GPU (Held, Thrun and Savarese, 2016).

Siamese trackers move all learning offline. CFNet’s authors describe the motivation as learning an embedding in which the target can be found by similarity alone, bypassing the online learning problem (Valmadre et al., 2017, Sec. 1). A network $\varphi$ maps both the exemplar $z$ and the search image $x$ to feature maps, and a similarity function compares them. If the comparison is a cross-correlation of the two feature maps, the score of every candidate window in the search region comes out of one forward pass, as a dense response map (or score map) whose maximum marks the target. Training teaches $\varphi$ to make the target score high and everything else score low, across millions of frame pairs from videos of many object classes. At test time the exemplar is embedded once, and each frame costs one pass of the search region through $\varphi$ and one correlation.

The approach trades adaptation for prior knowledge. The original SiamFC compares every frame with the first frame’s appearance and never updates (Valmadre et al., 2017, Sec. 3.2), so it relies on the embedding having learned, offline, which changes of appearance to ignore.

Building Blocks

Exemplar and search crops. Both inputs are square crops around the target, resized to fixed sizes. In SiamFC’s tracking code the exemplar is $127 \times 127$ pixels and the search region $255 \times 255$ (its training code uses search crops of 239 pixels), and the exemplar crop includes context: for a target of width $w$ and height $h$, the crop side before resizing is $\sqrt{(w + p)(h + p)}$ with $p = (w + h)/2$ (context amount 0.5 in tracking/tracker.m). SiamRPN states the same rule for its template and a search crop of twice the template’s size (Li et al., 2018, Sec. 5.1). The search crop is centred at the target’s position in the previous frame, so the tracker searches only a neighbourhood whose size is set by the crop.

Shared embedding. SiamFC’s network is a modified AlexNet without padding (Li et al., 2019, Sec. 3.1); SiamRPN uses the same network, in which the channel groups of conv2 and conv4 are removed, citing SiamFC for the modification (Li et al., 2018, Sec. 3.1). With a total stride of 8, a $127 \times 127$ exemplar becomes a $6 \times 6 \times 256$ feature map and the search region a $22 \times 22 \times 256$ map (sizes from SiamRPN’s Fig. 2), and their correlation is a $17 \times 17$ response map (p.scoreSize = 17, p.totalStride = 8 in SiamFC’s tracker).

Cross-correlation layer. The exemplar’s feature map serves as a convolution kernel slid over the search features, summing over channels:

$$f(z, x) = \varphi(z) \star \varphi(x) + b,$$

where $b$ is a bias common to all positions (the form SiamRPN++ and DaSiamRPN give for SiamFC). SiamFC’s code follows the correlation with an “adjust” layer, a fixed gain of $10^{-3}$ and a learned bias (add_adjust_layer in training/make_siameseFC.m); CFNet, which learns both a scale and a bias, explains that they are needed to make the score range suitable for the logistic loss (Valmadre et al., 2017, Sec. 3.3). CFNet calls this operation an exhaustive search of the exemplar over the search image (Valmadre et al., 2017, Eq. 1).

Heads. SiamFC’s head is the single-channel response map itself. SiamRPN replaced it with two outputs per position, scores and box offsets for $k$ anchors, computed by an up-channel correlation; SiamRPN++ replaced that with a depthwise correlation followed by small convolutions; SiamMask added a mask per position. These are described with their models below.

How Data Flows

Training. A training example is a pair of frames from the same video, a few frames apart: CFNet sampled pairs at most 100 frames apart from the 3,862 training videos of ImageNet VID, which contain more than a million annotated frames (Valmadre et al., 2017, App. A), and SiamFC’s code uses the same range (frameRange = 100). The exemplar is cropped from one frame around the annotated box and the search region from the other. The labels form a map of the same size as the response: positions near the target are positive and the rest negative. SiamFC’s default code (training/experiment.m) labels positions within 16 pixels of the centre as positive, which at stride 8 is a radius of two cells, and weights positives and negatives so that each class contributes half of the loss (labelWeight = 'balanced'). Its training crops are centred on the object, with random translations of at most 4 pixels (maxTranslate = 4) and stretching of up to 5%. The remaining defaults are 50 epochs of 53,200 pairs, a tenth of them held out for validation (validation = 0.1), mini-batches of 8, a learning rate decreasing logarithmically from $10^{-2}$ to $10^{-5}$ and weight decay $5 \times 10^{-4}$.

Tracking. In the first frame, the exemplar is embedded once. In each later frame (tracking/tracker.m, tracking/tracker_eval.m in SiamFC’s code):

1. Crop search regions around the previous position at several scales (three, a factor of 1.0375 apart, in the default SiamFC-3s; five in SiamFC-5s).
2. Compute the $17 \times 17$ response for each scale and upsample it by a factor of 16 with bicubic interpolation; scales other than the current one are multiplied by a penalty of 0.9745, and the scale with the highest peak wins.
3. Subtract the minimum, normalize the map to sum 1, and mix it with a cosine (Hann) window: $(1 - 0.176)\,r + 0.176\,w$. The window penalizes large displacements.
4. Move the target to the maximum, converting the displacement from response cells to frame pixels, and update the size with a damping factor of 0.59.

The template is never updated. CFNet’s tracker differs in computing a new template each frame and averaging it into the old one (Valmadre et al., 2017, Sec. 3.2).

Mathematical Formulation

The loss. With labels $y[u] \in \{-1, +1\}$ on the positions $u$ of the response map $v = f(z, x)$, SiamFC minimizes an element-wise logistic loss (CFNet, Eq. 2; SiamMask, Sec. 3.1, which cites SiamFC’s Sec. 2.2 for it):

$$L(y, v) = \sum_{u} \omega[u] \, \log\!\big(1 + e^{-y[u]\, v[u]}\big),$$

with weights $\omega[u]$ that, in SiamFC’s code, give the positive and the negative positions half of the total weight each. The loss is the logistic regression loss applied independently at every position; each position is a binary classifier asking whether this window is the target.

Translation invariance. For the response map to mean the same thing everywhere, a shift of the search image must shift the response by the corresponding number of cells and change nothing else. SiamRPN writes the requirement for a fully convolutional network $h$ with total stride $k$ and translation operator $(L_\tau x)[u] = x[u - \tau]$ as

$$h(L_{k\tau} x) = L_\tau h(x),$$

and removes all padding to satisfy it (Li et al., 2018, Eq. 1). SiamRPN++ states the same property for the whole tracker, $f(z, x[\Delta\tau_j]) = f(z, x)[\Delta\tau_j]$, and calls it strict translation invariance (Li et al., 2019, Sec. 3.1); in the terms used for convolutional networks generally, it is translation equivariance of the response with respect to the search image. CFNet’s authors chose SiamFC’s architecture because it enforces the prior that the similarity function commutes with translation (Valmadre et al., 2017, Sec. 2).

A network without padding satisfies it because every output cell is computed from a window lying entirely inside the input, by the same weights everywhere; nothing in the computation depends on where the window lies. Padding breaks it near the borders: a cell whose receptive field extends past the edge of the crop sees zeros instead of image, so its features differ from those of the same content in the middle. It follows that in a deep network whose receptive field is larger than the crop, every cell sees some padding, and the features of a cell can encode its distance from the border, which is to say its absolute position.

SiamRPN’s outputs. For $k$ anchors per position, SiamRPN’s classification branch outputs $2k$ channels (target and background for each anchor) and its regression branch $4k$ channels, the normalized offsets used in Faster R-CNN’s region proposal network:

$$\delta_x = \frac{T_x - A_x}{A_w}, \quad \delta_y = \frac{T_y - A_y}{A_h}, \quad \delta_w = \ln \frac{T_w}{A_w}, \quad \delta_h = \ln \frac{T_h}{A_h},$$

where $A$ is an anchor box and $T$ the ground-truth box (Li et al., 2018, Eq. 3). The loss is a cross-entropy term for classification plus $\lambda$ times a smooth $L_1$ loss on the four offsets.

DaSiamRPN’s distractor-aware score. For proposals $p_k$, a selected target $z$ and a set of $n$ distractors $d_i$ (other proposals scoring above a threshold after non-maximum suppression), DaSiamRPN picks

$$q = \arg\max_{p_k} \; f(z, p_k) - \frac{\hat{\alpha} \sum_{i=1}^{n} \alpha_i f(d_i, p_k)}{\sum_{i=1}^{n} \alpha_i}.$$

Because cross-correlation is linear, the subtraction can be done once on the templates, $\big(\varphi(z) - \hat{\alpha} \sum_i \alpha_i \varphi(d_i) / \sum_i \alpha_i\big) \star \varphi(p_k)$, which keeps the cost of a single correlation; averaging the target and distractor templates over frames with weights $\beta_t$ turns this into incremental learning (Zhu et al., 2018, Eqs. 2–4).

Why It Works

Three properties explain the design, as its sources argue.

• Dense supervision. Every position of the response map gets a label, so one training pair supplies hundreds of positive and negative examples, all sharing the exemplar’s computation. SiamRPN credits SiamFC’s success over GOTURN to this densely supervised heatmap, compared with GOTURN’s regression of a single box, and says it makes SiamFC more robust to fast-moving objects (Li et al., 2018, Sec. 2.1).
• Weight sharing and translation equivariance. Because the same network embeds the exemplar and every candidate window, the similarity is learned once and applied everywhere. Without padding, the score of a window depends only on its content, so the network can learn only what the target looks like, not where it usually is.
• Offline data. The embedding is learned from thousands of videos rather than one sequence. SiamRPN found that adding sparsely labelled YouTube-BoundingBoxes videos to ImageNet VID raised its expected average overlap on VOT2016 from 0.317 to 0.344, with no sign of saturation (Li et al., 2018, Sec. 5.6.1).

SiamRPN also offers a learning-to-learn reading: the template branch maps the first frame’s exemplar to the weights of a detector, the correlation kernels, which the detection branch then applies to each new frame, so tracking becomes local one-shot detection (Li et al., 2018, Sec. 4.1). In our reading, this is what distinguishes the family from correlation filters, which solve a small learning problem on each sequence: a Siamese tracker’s per-sequence “learning” is a single forward pass of the exemplar.

Training Considerations

Data. SiamFC trained on ImageNet VID, which DaSiamRPN describes as about 4,000 videos annotated frame by frame in 20 categories (Zhu et al., 2018, Sec. 3.2). SiamRPN added YouTube-BoundingBoxes, sparsely annotated videos that SiamRPN puts at more than 100,000 and DaSiamRPN at more than 200,000, both annotated once every 30 frames. DaSiamRPN added still images from ImageNet DET and COCO, turned into pairs by translating, resizing and otherwise augmenting one image, to widen the range of object categories beyond VID’s 20 and YouTube-BB’s 30.

Easy and hard negatives. Most negative positions in a search crop are background with no object in it. DaSiamRPN argues that SiamFC and SiamRPN therefore learn to separate foreground from background rather than one object from another, and that the losses of semantic objects are overwhelmed by easy negatives (Zhu et al., 2018, Sec. 3.2). Its remedy is to add negative pairs whose exemplar and search region show different objects, of the same category or of different categories, a form of hard negative mining at the level of pairs. The PySOT code makes 20% of training pairs negative (DATASET.NEG = 0.2 in pysot/core/config.py), drawing the search image from a random object of any dataset.

Where the target sits in the crop. SiamFC’s training crops are centred on the target up to a few pixels, and for a network without padding this costs nothing: since the response depends only on content, a target always seen at the centre teaches the same similarity as one seen anywhere. SiamRPN++ showed that this stops being true with padding. In simulation experiments with a padded network, the authors trained with the target placed at the centre and shifted uniformly within 0, 16 or 32 pixels, then summed the response maps on test data (Li et al., 2019, Sec. 3.1, Fig. 1). With no shift, the response at the borders fell to zero whatever the test target looked like, a strong centre bias; larger shifts gradually removed it. Trained with a ResNet-50 backbone and no shift, the tracker’s expected average overlap on VOT2018 fell to 0.14; the authors concluded that a suitable shift, ±64 pixels, is vital for training a deep Siamese tracker (Fig. 2). They call the remedy spatial aware sampling; PySOT’s defaults shift the search crop by up to 64 pixels and the template by up to 4 (DATASET.SEARCH.SHIFT = 64, DATASET.TEMPLATE.SHIFT = 4). The example below reproduces the effect on a small scale.

Zhang and Peng (2019) reached a similar diagnosis in SiamDW, at the same conference: padding makes the embedding of a target near the border of the search image inconsistent with its embedding at the centre, which they call a position bias, and they removed it architecturally, with residual units that crop out the padding-affected features inside each block.

Backbone settings. SiamRPN++ fine-tuned the ImageNet-pretrained ResNet-50 with a learning rate ten times smaller than the head’s, and found fine-tuning critical (Li et al., 2019, Sec. 3.2, Table 1). Its schedule was 20 epochs on 8 GPUs with 128 pairs per mini-batch: 5 warm-up epochs at a learning rate of 0.001 that trained the proposal branches, then 15 epochs with the whole network trained end to end and the learning rate decaying exponentially from 0.005 to 0.0005 (Sec. 4.2).

Variants

SiamFC and CFNet

SiamFC is the design described above: an AlexNet-like embedding without padding, cross-correlation, the logistic loss, offline training on ImageNet VID and a tracker with a fixed template, a cosine window and a small scale pyramid. It ran at 86 frames per second in the VOT2015 comparison of the SiamRPN paper, where SiamFC-3s reached an expected average overlap (EAO) of 0.2915 and SiamFC-5s 0.275 (Li et al., 2018, Table 1).

CFNet, by the same group, inserted a correlation filter between the exemplar’s features and the cross-correlation, so that the template used for search is the ridge-regression solution learned from the exemplar’s features rather than the features themselves, and made the filter’s closed-form solution a differentiable layer so that the features could be trained through it (Valmadre et al., 2017). The filter itself is described in Correlation Filter Tracking. For shallow networks the filter helped a lot, a relative improvement of 31% and 13% in average overlap on their validation set for networks of one and two layers, but at depths three to five the difference was much less meaningful, and the authors concluded that the filter does not significantly improve a sufficiently deep Siamese network (Sec. 4.2, 5). On OTB-100 with one-pass evaluation, the OTB toolkit’s overlap score was 58.2 for SiamFC-3s, 56.8 for the two-layer CFNet and 58.8 for their own five-layer Siamese baseline (Table 1).

GOTURN

GOTURN (Held, Thrun and Savarese, 2016) is the regression counterpart, contemporary with SiamFC. It crops the previous frame around the last box and the current frame at the same place, both at twice the box size, passes each through the five convolutional layers of CaffeNet, concatenates the two pool5 outputs and regresses the box corners in the current crop with three fully connected layers of 4,096 units, trained with an $L_1$ loss. The paper does not call the architecture Siamese; SiamRPN and DaSiamRPN classify it with the Siamese trackers. It trained on 307 videos of ALOV300++ and on still images from ImageNet DET, from which pairs were made by random crops whose shifts and scale changes were drawn from Laplace distributions fitted to the motions in the training videos. It ran at 100 frames per second on a GTX 680 GPU and 165 on a Titan X, and on the 25 sequences of VOT2014 its overall rank, the average of its accuracy and robustness ranks, was better than that of every tracker in the challenge.

SiamRPN

SiamRPN (Li et al., 2018) kept SiamFC’s embedding and replaced the response map with a region proposal head. The template’s $6 \times 6 \times 256$ features pass through two convolutions that multiply their channels by $2k$ and $4k$, giving kernels of $4 \times 4 \times (2k \times 256)$ and $4 \times 4 \times (4k \times 256)$; the search features pass through two convolutions that keep 256 channels; and correlating each group of 256 template channels with the search features yields $17 \times 17 \times 2k$ scores and $17 \times 17 \times 4k$ offsets (Fig. 2). The anchors have one scale and five aspect ratios, 0.33, 0.5, 1, 2 and 3; anchors with intersection over union above 0.6 with the target are positive and below 0.3 negative, with at most 16 positives among 64 samples per pair. Because the boxes are regressed, the scale pyramid is no longer needed.

At inference the template branch runs once and its outputs become fixed convolution kernels, so tracking is a single small detection network running at 160 frames per second (Sec. 4.2, 5.2). Proposals far from the centre are discarded (only a central $7 \times 7$ region of the $17 \times 17$ map in their illustration), the rest are re-ranked with a cosine window and a penalty on changes of scale and aspect ratio, and the box size is smoothed by linear interpolation. On VOT2015 SiamRPN reached an EAO of 0.358 against SiamFC-3s’s 0.2915, on VOT2016 0.3441, and in the VOT2017 real-time experiment 0.243 against 0.182 for SiamFC (Tables 1–2, Fig. 8).

DaSiamRPN

DaSiamRPN (Zhu et al., 2018) kept SiamRPN’s network and changed what it learns from and how it is used. Training added positive pairs from detection datasets, the semantic negative pairs described above and motion blur as an augmentation; inference added the distractor-aware re-ranking and incremental template update in the formula above, with $\hat{\alpha} = 0.5$ and $\alpha_i = 1$. For long-term tracking, when the score indicates failure, the search region grows from 255 to 767 pixels, one step being enough in the authors’ experiments, with score thresholds of 0.8 to enter and 0.95 to leave the failure state. The ablation on VOT2016 attributes the gain step by step: SiamRPN’s EAO of 0.344 rose to 0.368 with detection data, 0.389 with negative pairs and 0.411 with the distractor-aware update; on the long-term UAV20L benchmark the area under the success curve rose from 45.4% to 49.8% with those steps and to 61.7% with the long-term search (Table 2). The tracker ran at 160 frames per second on short-term benchmarks and 110 on long-term ones.

SiamRPN++

SiamRPN++ (Li et al., 2019) began from the observation that Siamese trackers were all built on AlexNet-like networks and that replacing them with deeper ones such as ResNet had brought no gain. Its analysis located the cause in padding and translation invariance, as described above, and its fix was spatial aware sampling. With that in place it rebuilt SiamRPN on ResNet-50:

• Stride and size. The strides of conv4 and conv5 are set to 1, with dilated convolutions to keep the receptive field, so the effective stride is 8; a $1 \times 1$ convolution reduces each block’s output to 256 channels. Because padding is kept, the template’s feature map grows to $15 \times 15$, and only its central $7 \times 7$ is used, following CFNet (Sec. 3.2).
• Layer-wise aggregation. Separate Siamese proposal heads on conv3, conv4 and conv5 are combined by learned weighted sums of their scores and offsets.
• Depthwise cross-correlation. SiamRPN’s up-channel layer made the head far heavier than the backbone, with about 20M parameters against 4M in the feature extractor. SiamRPN++ instead passes template and search features through separate, unshared convolutions and correlates them channel by channel, producing a 256-channel response that small convolutions turn into scores and offsets. The authors report ten times fewer parameters than the up-channel layer with performance on par, and more stable training (Sec. 3.4). In PySOT the operation is a grouped convolution with one group per channel (xcorr_depthwise in pysot/core/xcorr.py), the same structure as a depthwise separable convolution’s first stage.

In the ablation (Table 1), the AlexNet version scored an EAO of 0.332 on VOT2018 and a success AUC of 0.658 on OTB2015 with up-channel correlation and 0.355 and 0.666 with depthwise correlation; ResNet-50 with all three layers, fine-tuning and depthwise correlation reached 0.414 and 0.696. The full tracker ran at 35 frames per second on a Titan Xp GPU, and variants with ResNet-18 and MobileNetV2 backbones at more than 70 (Fig. 8). The paper’s text says the regression loss is the standard smooth $L_1$ loss; the released PySOT code computes a weighted $L_1$ loss (weight_l1_loss in pysot/models/loss.py).

SiamMask

SiamMask (Wang et al., 2019) added a segmentation branch to the two designs: a two-branch variant on SiamFC and a three-branch variant on SiamRPN, both with a ResNet-50 backbone at stride 8 and depthwise correlation. Each of the $17 \times 17$ positions of the 256-channel response, called a response of a candidate window (RoW), is decoded by two $1 \times 1$ convolutions into a $63 \times 63$ mask, so each pixel classifier sees the whole window; refinement modules with skip connections sharpen the result. The mask loss is a per-pixel binary logistic loss applied only at positive positions, weighted by $\lambda_1 = 32$ against the other losses. Since tracking benchmarks need boxes, the mask is converted to an axis-aligned box, a rotated minimum-area rectangle, or a box fitted by VOT-2016’s optimization. On VOT-2018 the three-branch variant with rotated rectangles reached an EAO of 0.380 at 55 frames per second, and 0.387 at 5 frames per second with the optimization; initialized from a box and without any fine-tuning, it reached a region similarity $\mathcal{J}$ of 71.7 and a contour accuracy $\mathcal{F}$ of 67.8 on the DAVIS-2016 validation set (Tables 2, 4). This brought Siamese tracking to semi-supervised video object segmentation, whose methods are initialized with a mask and often relied on fine-tuning or other costly steps at test time; SiamMask was almost two orders of magnitude faster than accurate methods such as OnAVOS (Sec. 4.2).

After SiamRPN++

Transformer trackers replaced the plain cross-correlation with attention between template and search features, and many process both in one backbone, as Object Tracking outlines; they still compare a template with a search region.

Use in Computer Vision

Results as reported. The benchmarks and their protocols are described in Single Object Tracking; in outline, OTB’s one-pass evaluation initializes the tracker from the ground truth in the first frame, runs it to the end without correction and reports the area under the success curve (the fraction of frames whose overlap exceeds each threshold) and the precision at 20 pixels, while VOT’s protocol of these years re-initializes a tracker five frames after each failure and reports accuracy, robustness and the expected average overlap (Zhu et al., 2018, Sec. 4.2). LaSOT’s second protocol splits its 1,400 long videos into 1,120 that may be used for training and 280 for testing, with the same one-pass evaluation; LaSOT’s authors scored existing trackers on the test videos as released, without re-training them (Fan et al., 2019, Sec. 3.5, 4.4). The numbers below are the authors’ own, each from the paper named; numbers from different papers for the same tracker can differ, as noted after the table.

Benchmark and measure	SiamFC	SiamRPN	DaSiamRPN	SiamRPN++	Source
OTB2015, success AUC (one-pass)	0.582 (3s)	0.637	—	—	SiamRPN, Fig. 9
OTB2015, success AUC (one-pass)	—	0.636	0.658	0.696	SiamRPN++, Fig. 7
VOT2015, EAO	0.2915 (3s)	0.358	—	—	SiamRPN, Table 1
VOT2016, EAO	0.235	0.344	0.411	—	DaSiamRPN, Table 1
VOT2018, EAO	—	—	0.326	0.414	SiamRPN++, Table 2
LaSOT test set (280 videos, protocol II split), success AUC	0.336	—	0.415	0.496	SiamRPN++, Fig. 11

The same tracker scores differently in different papers. On OTB2015 SiamRPN reports 0.637 for itself and SiamRPN++ lists it at 0.636. DaSiamRPN’s table lists SiamRPN at 0.349 EAO on VOT2015, below SiamRPN’s own 0.358, and at 200 frames per second instead of 160 (Zhu et al., 2018, Table 1); neither paper explains the difference. On VOT2018 the two SiamRPN entries are different trackers. SiamMask lists SiamRPN at 0.244 EAO, with accuracy 0.490, robustness 0.460 and 200 frames per second (Wang et al., 2019, Table 2), the figures DaSiamRPN gives for SiamRPN on VOT2017, whose sequences VOT2018 reused unchanged (Zhu et al., 2018, Table 1; Kristan et al., 2018, Sec. 2). SiamRPN++’s entry at 0.383, with accuracy 0.586 and robustness 0.276 (Li et al., 2019, Table 2), matches the VOT2018 challenge entry named SiamRPN, which its submitters, among them DaSiamRPN’s authors, describe as SiamRPN with DaSiamRPN’s sampling strategy for semantic distractors; it won the challenge’s real-time sub-challenge (Kristan et al., 2018, Table 2, App. A.35). Correlation Filter Tracking summarizes how SiamFC compared with the correlation filters of the same period, from ECO’s comparison, which gives SiamFC a lower OTB-2015 score than SiamRPN’s and CFNet’s tables do.

Speed. The family’s practical draw was speed with deep features: tens to hundreds of frames per second on a GPU, against about one frame per second for the online-trained deep trackers it competed with, because nothing is trained at test time.

Beyond single boxes. SiamMask extended the design to masks, DaSiamRPN to long-term tracking with re-detection, and in our reading the template–search correlation is close kin to the correlation layer of optical flow networks such as FlowNet, which compares two feature maps over a range of displacements, the operation that cost volumes generalize.

Example

The example asks two questions on synthetic video where the ground truth is known exactly. Does a learned Siamese embedding track better than template matching on raw pixels? And does a padded network trained with the target always at the centre of the search crop learn the centre bias SiamRPN++ describes, while a network without padding does not?

Each frame is $128 \times 128$ pixels with a textured background and four $24 \times 24$ textured squares drawn from the same random texture generator: the target and three distractors of the same “kind”, which differ only in their particular pattern. Between frames every square moves, rotates (a random walk of 3° per frame) and changes scale slightly, and the frame’s gamma drifts, so the target’s pixels change while its identity does not. The learned trackers compare a $41 \times 41$ exemplar with a $73 \times 73$ search crop through:

• NoPad, a SiamFC-style network of five convolutions without padding (receptive field 29 pixels, stride 4), whose response covers displacements of ±16 pixels in steps of 4;
• Pad, a deeper network with zero padding and dilated convolutions whose receptive field (77 pixels) exceeds the search crop, as a ResNet’s does; following SiamRPN++, only the central $5 \times 5$ of its exemplar feature map is used as the template, and its response is restricted to the same ±16 pixels.

Each is trained with SiamFC’s logistic loss and balanced labels (positive within two cells of the target) on 3,000 frame pairs, with the target either always at the centre of the search crop (shift 0) or displaced uniformly by up to 16 pixels (shift 16). The raw-pixel baseline is normalized cross-correlation of the first frame’s $25 \times 25$ target patch, which is already invariant to brightness and contrast changes. All trackers keep their first-frame template, as SiamFC does, and are printed with and without SiamFC’s cosine window. Tracking is scored on 30 test sequences of 60 frames by the mean centre error, the precision at 10 pixels and the area under the success curve for fixed $24 \times 24$ boxes. A probe places the target at known displacements in 1,000 held-out search crops and fits the slope of predicted against true displacement: 1 means unbiased, 0 means the network always answers “centre”.

Before training, a check without any learning confirmed that the task is learnable but not trivial: on 500 pairs, normalized cross-correlation found the target ahead of every location at least 8 pixels away in all pairs rotated by less than 10°, in 71% of pairs rotated by 10–20° and in 6% of pairs rotated by 20–30°. The script runs with PyTorch 2.14.1 and SciPy, without torchvision, and takes the training seed as an argument; it took between four and five and a half minutes per seed on a laptop CPU with four threads, with two runs sharing the processor. After the first run, the two lines that printed elapsed times (for building the data and for the whole run) were removed from the output, because timing depends on the machine; no other line of the script changed. The output below is for seed 0.

import sys
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from scipy.ndimage import gaussian_filter, map_coordinates

torch.set_num_threads(4)
H = 128                     # frame size
OBJ, EX, SX, R = 24, 41, 73, 16   # object, exemplar crop, search crop, search range (px)

def texture(rng, n, sigma):
    t = gaussian_filter(rng.standard_normal((n, n)), sigma)
    return (t - t.mean()) / t.std()

def render(bg, objs, gamma, rng):
    """Paste rotated, scaled textured squares (tex, cx, cy, angle, scale) on bg."""
    img = bg.copy()
    for tex, cx, cy, ang, sc in objs:          # the target is the last object
        x0, x1 = max(int(cx) - 20, 0), min(int(cx) + 21, H)
        y0, y1 = max(int(cy) - 20, 0), min(int(cy) + 21, H)
        if x0 >= x1 or y0 >= y1:
            continue
        yy, xx = np.mgrid[y0:y1, x0:x1].astype(float)
        c, s = np.cos(ang), np.sin(ang)
        u = (c * (xx - cx) + s * (yy - cy)) / sc
        v = (-s * (xx - cx) + c * (yy - cy)) / sc
        inside = (np.abs(u) < OBJ / 2) & (np.abs(v) < OBJ / 2)
        vals = map_coordinates(tex, [v[inside] + OBJ / 2 - 0.5, u[inside] + OBJ / 2 - 0.5],
                               order=1, mode="nearest")
        img[y0:y1, x0:x1][inside] = 0.5 + 0.2 * vals
    img = np.clip(img, 0, 1) ** gamma
    return img + 0.03 * rng.standard_normal(img.shape)

def crop(img, cx, cy, size):
    pad = np.pad(img, size, constant_values=img.mean())
    x0 = int(round(cx)) - size // 2 + size
    y0 = int(round(cy)) - size // 2 + size
    return pad[y0:y0 + size, x0:x0 + size]

def make_pair(rng):
    """Two frames of one scene: exemplar crop at A, a wide crop around the target at B."""
    bg = 0.5 + 0.08 * texture(rng, H, 3)
    texs = [texture(rng, OBJ, 1.5) for _ in range(4)]
    objA, objB = [], []
    for t in texs[1:]:                          # three distractors of the same kind
        q, a = rng.uniform(0, H, 2), rng.uniform(0, 2 * np.pi)
        objA.append((t, *q, a, rng.uniform(0.9, 1.1)))
        objB.append((t, *(q + rng.uniform(-12, 12, 2)), a + np.radians(rng.uniform(-30, 30)),
                     rng.uniform(0.9, 1.1)))
    pa = rng.uniform(40, 88, 2)
    pb = pa + rng.uniform(-12, 12, 2)
    angA = rng.uniform(0, 2 * np.pi)
    dang = rng.uniform(-30, 30)
    objA.append((texs[0], *pa, angA, rng.uniform(0.9, 1.1)))
    objB.append((texs[0], *pb, angA + np.radians(dang), rng.uniform(0.9, 1.1)))
    fa = render(bg, objA, rng.uniform(0.7, 1.4), rng)
    fb = render(bg, objB, rng.uniform(0.7, 1.4), rng)
    z = crop(fa, *pa, EX)
    wide = crop(fb, *pb, SX + 2 * R)            # search crops are cut from this
    sub = pb - np.round(pb)                     # sub-pixel offset of the target in `wide`
    return z, wide, sub, dang, fa, fb, pa, pb

class SiamNet(nn.Module):
    def __init__(self, pad):
        super().__init__()
        if pad:   # zero padding, receptive field 77 px > search crop
            spec = [(5, 2, 2, 1), (3, 1, 1, 1), (3, 2, 1, 1), (3, 1, 1, 1),
                    (3, 1, 2, 2), (3, 1, 4, 4), (3, 1, 1, 1)]
        else:     # no padding (SiamFC-style), receptive field 29 px
            spec = [(5, 2, 0, 1), (3, 1, 0, 1), (3, 2, 0, 1), (3, 1, 0, 1), (3, 1, 0, 1)]
        layers, cin = [], 1
        for i, (k, s, p, d) in enumerate(spec):
            layers.append(nn.Conv2d(cin, 32, k, s, p, dilation=d))
            cin = 32
            if i < len(spec) - 1:
                layers.append(nn.ReLU())
        self.f, self.pad = nn.Sequential(*layers), pad
        self.off = 28 if pad else 16            # response cell i <-> displacement 4*i - off
        self.scale = nn.Parameter(torch.tensor(0.01))
        self.bias = nn.Parameter(torch.tensor(0.0))

    def template(self, z):
        t = self.f(z)
        return t[:, :, 3:8, 3:8] if self.pad else t   # padded: keep the centre 5x5 of 11x11

    def forward(self, z, x, t=None):
        t = self.template(z) if t is None else t
        fx = self.f(x)
        b = x.shape[0]
        out = F.conv2d(fx.reshape(1, -1, *fx.shape[2:]), t, groups=b)  # cross-correlation
        return out.reshape(b, *out.shape[2:]) * self.scale + self.bias

def prep(a):
    return torch.tensor((np.asarray(a) - 0.5) / 0.25, dtype=torch.float32).unsqueeze(-3)

def labels(net, d):
    """Logistic-loss labels: +1 within 8 px (2 cells) of the true displacement, balanced weights."""
    n = 2 * net.off // 4 + 1
    g = 4 * torch.arange(n, dtype=torch.float32) - net.off
    dist = torch.sqrt((g[None, None, :] - d[:, 0, None, None]) ** 2 +
                      (g[None, :, None] - d[:, 1, None, None]) ** 2)
    y = torch.where(dist <= 8, 1.0, -1.0)
    pos = (y > 0).float()
    w = 0.5 * pos / pos.sum((1, 2), keepdim=True) + 0.5 * (1 - pos) / (1 - pos).sum((1, 2), keepdim=True)
    return y, w

def train(pad, shift, pairs, epochs, seed):
    torch.manual_seed(seed)
    rng = np.random.default_rng(seed)
    net = SiamNet(pad)
    opt = torch.optim.Adam(net.parameters(), lr=1e-3)
    zs = prep(np.stack([p[0] for p in pairs]))
    for ep in range(epochs):
        for idx in np.array_split(rng.permutation(len(pairs)), len(pairs) // 32):
            d = rng.integers(-shift, shift + 1, (len(idx), 2)) if shift else np.zeros((len(idx), 2), int)
            xs = [pairs[i][1][R - d[k, 1]:R - d[k, 1] + SX, R - d[k, 0]:R - d[k, 0] + SX]
                  for k, i in enumerate(idx)]   # target sits at +d from the crop centre
            dt = torch.tensor(d + np.stack([pairs[i][2] for i in idx]), dtype=torch.float32)
            y, w = labels(net, dt)
            s = net(zs[idx], prep(np.stack(xs)))
            loss = (w * F.softplus(-y * s)).sum((1, 2)).mean()
            opt.zero_grad()
            loss.backward()
            opt.step()
    return net

def centre_map(net, s):
    """Response restricted to displacements -16..16, upsampled to 1 px (bicubic, as SiamFC)."""
    i0 = (net.off - R) // 4
    s = s[..., i0:i0 + 9, i0:i0 + 9]
    return F.interpolate(s.unsqueeze(1), size=(33, 33), mode="bicubic", align_corners=True)[:, 0]

def ncc_map(tpl, x):
    """Normalized cross-correlation of a 25x25 template at displacements -16..16."""
    win = np.lib.stride_tricks.sliding_window_view(x[SX // 2 - 28:SX // 2 + 29, SX // 2 - 28:SX // 2 + 29], (25, 25))
    t = (tpl - tpl.mean()) / tpl.std()
    w = win - win.mean((2, 3), keepdims=True)
    w = w / (w.std((2, 3), keepdims=True) + 1e-8)
    return (w * t).mean((2, 3))

HANN = np.outer(np.hanning(33), np.hanning(33))
HANN = HANN / HANN.sum()

def peak(r, w_inf):
    """SiamFC's rule: normalize the map to sum 1, mix with a cosine window, take the argmax."""
    r = r - r.min()
    r = (1 - w_inf) * r / r.sum() + w_inf * HANN
    iy, ix = np.unravel_index(np.argmax(r), r.shape)
    return np.array([ix - R, iy - R], float)

def make_sequence(rng, n=60):
    bg = 0.5 + 0.08 * texture(rng, H, 3)
    objs = []
    for _ in range(4):                           # 3 distractors, then the target
        objs.append(dict(t=texture(rng, OBJ, 1.5), p=rng.uniform(30, 98, 2), v=rng.uniform(-2, 2, 2),
                         a=rng.uniform(0, 2 * np.pi), s=1.0))
    gamma, frames, gt = 1.0, [], []
    for _ in range(n):
        frames.append(render(bg, [(o["t"], *o["p"], o["a"], o["s"]) for o in objs], gamma, rng))
        gt.append(objs[-1]["p"].copy())
        for o in objs:
            o["v"] = np.clip(o["v"] + rng.normal(0, 0.5, 2), -4, 4)
            o["p"] = o["p"] + o["v"]
            for k in range(2):
                if not 20 <= o["p"][k] <= H - 20:
                    o["v"][k] = -o["v"][k]
                    o["p"][k] = np.clip(o["p"][k], 20, H - 20)
            o["a"] += np.radians(rng.normal(0, 3))
            o["s"] = np.clip(o["s"] + rng.normal(0, 0.01), 0.9, 1.1)
        gamma = float(np.clip(gamma + rng.normal(0, 0.03), 0.7, 1.4))
    return frames, np.array(gt)

def track(frames, gt, w_inf, net=None):
    c = gt[0].copy()
    out = [c.copy()]
    if net is None:
        tpl = crop(frames[0], *c, 25)
    else:
        with torch.no_grad():
            t = net.template(prep(crop(frames[0], *c, EX))[None])
    for f in frames[1:]:
        x = crop(f, *c, SX)
        if net is None:
            r = ncc_map(tpl, x)
        else:
            with torch.no_grad():
                r = centre_map(net, net(None, prep(x)[None], t))[0].numpy()
        c = np.round(c) + peak(r, w_inf)
        out.append(c.copy())
    return np.array(out)

def scores(pred, gt):
    e = np.linalg.norm(pred - gt, axis=1)[1:]
    ov = np.clip(OBJ - np.abs(pred - gt), 0, None).prod(1)[1:]
    iou = ov / (2 * OBJ * OBJ - ov)
    auc = np.mean([(iou > th).mean() for th in np.linspace(0, 1, 21)])
    return e.mean(), (e <= 10).mean(), auc

def probe(net, pairs, rng):
    """Centre-bias probe: target placed at a known displacement d in the search crop."""
    d = rng.integers(-R, R + 1, (len(pairs), 2))
    xs = np.stack([p[1][R - d[k, 1]:R - d[k, 1] + SX, R - d[k, 0]:R - d[k, 0] + SX]
                   for k, p in enumerate(pairs)])
    dt = d + np.stack([p[2] for p in pairs])
    with torch.no_grad():
        r = centre_map(net, net(prep(np.stack([p[0] for p in pairs])), prep(xs))).numpy()
    pred = np.stack([peak(m, 0.0) for m in r])
    slope = (pred * dt).sum() / (dt * dt).sum()
    err = np.linalg.norm(pred - dt, axis=1)
    big = np.abs(d).max(1)
    return slope, err[big <= 4].mean(), err[big >= 12].mean()

if __name__ == "__main__":
    seed = int(sys.argv[1]) if len(sys.argv) > 1 else 0
    rng = np.random.default_rng(1000)
    train_pairs = [make_pair(rng)[:4] for _ in range(3000)]
    probe_pairs = [make_pair(np.random.default_rng(5000 + i))[:4] for i in range(1000)]
    seqs = [make_sequence(np.random.default_rng(9000 + i)) for i in range(30)]
    print(f"torch {torch.__version__}, training seed {seed}")
    print(f"{'tracker':<16}{'window':>7}{'centre err':>11}{'prec@10':>9}{'success':>9}")
    for w_inf in (0.0, 0.176):
        res = np.mean([scores(track(f, g, w_inf), g) for f, g in seqs], 0)
        print(f"{'NCC on pixels':<16}{w_inf:>7}{res[0]:>11.2f}{res[1]:>9.3f}{res[2]:>9.3f}")
    nets = {}
    for pad in (False, True):
        for shift in (0, R):
            nets[(pad, shift)] = net = train(pad, shift, train_pairs, epochs=6, seed=seed)
            name = f"{'Pad' if pad else 'NoPad'} shift {shift}"
            for w_inf in (0.0, 0.176):
                res = np.mean([scores(track(f, g, w_inf, net), g) for f, g in seqs], 0)
                print(f"{name:<16}{w_inf:>7}{res[0]:>11.2f}{res[1]:>9.3f}{res[2]:>9.3f}")
    print(f"{'probe':<16}{'slope':>7}{'err |d|<=4':>11}{'err |d|>=12':>12}")
    for (pad, shift), net in nets.items():
        sl, e_small, e_big = probe(net, probe_pairs, np.random.default_rng(7))
        print(f"{('Pad' if pad else 'NoPad') + f' shift {shift}':<16}{sl:>7.3f}{e_small:>11.2f}{e_big:>12.2f}")

torch 2.14.1, training seed 0
tracker          window centre err  prec@10  success
NCC on pixels       0.0      12.51    0.753    0.679
NCC on pixels     0.176       3.06    0.939    0.802
NoPad shift 0       0.0       9.98    0.788    0.669
NoPad shift 0     0.176      17.23    0.604    0.426
NoPad shift 16      0.0      16.72    0.642    0.520
NoPad shift 16    0.176      18.69    0.513    0.366
Pad shift 0         0.0      31.43    0.200    0.162
Pad shift 0       0.176      31.25    0.186    0.155
Pad shift 16        0.0      14.57    0.684    0.567
Pad shift 16      0.176      14.58    0.664    0.494
probe             slope err |d|<=4 err |d|>=12
NoPad shift 0     0.961       2.80        2.24
NoPad shift 16    0.940       3.65        3.10
Pad shift 0       0.007       3.52       16.01
Pad shift 16      0.938       3.40        2.85

The centre bias. The probe reproduces SiamRPN++’s finding in its most extreme form. The padded network trained with centred targets has a slope of 0.007: whatever the displacement of the target, it answers “no displacement”, with an error of 16 pixels for targets 12 or more pixels from the centre. In our reading, its response no longer encodes where the target is, only that the target is usually at the centre, which the padding lets it detect. In tracking it therefore leaves the box at, or very near, the target’s first position and loses the target as soon as it moves away. Trained with shifted targets, the same network localizes again (slope 0.938) and tracks. The network without padding shows no such dependence: its slopes are 0.96 and 0.94 for the two training regimes, because a network that cannot see the crop border cannot learn where the centre is. Seeds 1 and 2 gave the same picture: slopes of 0.000 for the padded network with centred targets, 0.89 and 0.93 with shifted targets, and 0.93 to 0.98 for the network without padding.

Learned against raw pixels. The answer depends on the cosine window, a choice the example prints both ways. Without the window, the learned network without padding, trained with centred targets, had a lower mean centre error than normalized cross-correlation in all three seeds, 9.98, 3.88 and 4.31 pixels against 12.51, though its success AUC beat the baseline’s 0.679 only in seeds 1 and 2 (0.801 and 0.772; 0.669 in seed 0). With SiamFC’s window weight of 0.176, normalized cross-correlation was the best tracker of all, at 3.06 pixels, while the window made the network without padding clearly worse in every seed (from 9.98 to 17.23 pixels for centred training in seed 0) and moved the padded networks’ errors only a little, in either direction. The window suppresses jumps far from the previous position, which in our reading is how raw-pixel matching fails here (to a distractor), but it also pulls the estimate towards the previous position; how much it pulls depends on how peaked the normalized response is, and in our reading SiamFC’s weight, chosen for its own response maps, does not transfer to these. The example’s networks are small and trained for 6 epochs on 3,000 pairs, so the example cannot show the advantage that offline training on large video datasets gives the published trackers.

An unexpected difference. Without padding, training with shifted targets should make no difference to what the network can learn, and the probe agrees, yet these networks tracked worse than those trained with centred targets in all three seeds (16.72, 17.07 and 8.17 pixels against 9.98, 3.88 and 4.31 without the window). The probe’s errors are also slightly larger for shifted training, so a likely cause is that labels at arbitrary sub-cell positions make precise localization harder to learn in a few epochs, and tracking compounds small errors into losses; the example does not test this.

Limitations

• Local search. The tracker looks only in a window around the last position. A target that moves farther between frames, or reappears elsewhere after an occlusion, is lost unless a re-detection mechanism such as DaSiamRPN’s growing search region is added (Zhu et al., 2018, Sec. 3.4).
• No adaptation by default. A fixed template cannot follow large appearance changes; SiamRPN++’s authors attribute their remaining robustness gap with correlation filters to the lack of template adaptation (Li et al., 2019, Sec. 4.4). Updating the template, as CFNet and DaSiamRPN do, reintroduces the risk, familiar from correlation filters, of absorbing an occluder into the template.
• Distractors. An embedding trained mainly against background scores similar objects highly, and the cosine window is not a dependable defence when the objects’ motion is erratic (Zhu et al., 2018, Sec. 3.1, 3.3).
• Position bias. Padded backbones learn where targets usually are unless training displaces them; in SiamRPN++’s experiment, VOT EAO rose steeply as the displacement range grew to 48–64 pixels and changed little beyond (Li et al., 2019, Fig. 2).
• Coarse boxes from scores alone. SiamFC’s response gives a position, and scale comes from a pyramid; regression heads (SiamRPN) and masks (SiamMask) were added to recover the extent and shape of the target.
• Benchmark numbers vary between papers. As the table shows, the same tracker is reported with different scores by different authors, and benchmarks such as OTB and VOT do not enforce a validation/test split; CFNet’s authors chose their hyperparameters on separate validation videos for this reason (Valmadre et al., 2017, Sec. 4.1).