Siamese Network

A Siamese network compares two inputs by passing each through the same network, with the same weights, and then comparing the two outputs. The shared network, called a branch, tower or sub-network, turns an image or patch into a vector. A comparison module then turns the two vectors into a distance or a similarity score. Chopra, Hadsell and LeCun (2005) state the consequence that gives the design its value: because the same function with the same parameters processes both inputs, the resulting similarity is symmetric. The name appears in the title of Bromley et al.’s signature-verification paper (1993), which Chopra et al., Zagoruyko and Komodakis (2015) and Žbontar and LeCun (2016) all cite for the architecture. The design reaches well beyond pairs of signatures:

• face verification (DeepFace);
• one-shot recognition (few-shot recognition);
• patch matching for wide-baseline stereo and descriptors;
• the matching cost of stereo networks;
• tracking and change detection;
• recent self-supervised methods, which Chen and He (2021) say generally involve some form of Siamese network.

The losses that train such a network are covered in metric learning, contrastive loss and triplet loss. This article covers the architecture: what sharing weights guarantees, what the alternatives that do not share them give up or gain, and what each costs when many items must be compared.

Core Idea

Comparing two images directly in pixel space fails for the reasons that motivate metric learning: the pixels that change most between two photographs of the same thing, such as lighting, position and pose, are not the ones that decide whether it is the same thing. A Siamese network learns a map $f_\theta$ into a space where a simple comparison works, and it learns that map from pairs (or triplets) labelled as matching or not. Chopra et al. (2005) wanted a system for problems in which the number of categories is very large, each category has few examples and only some categories are known during training. A function that maps any input to a point, trained so that distances between points carry the meaning of “same” or “different”, can compare faces of people never seen in training.

Two facts make the shared map more than a convenience:

1. The comparison is consistent. Both inputs are described in the same coordinates, so the comparison of $a$ with $b$ is the comparison of $b$ with $a$, and an input compared with itself gives the most similar possible result, provided the comparison module itself is symmetric (see Mathematical Formulation).
2. The description can be computed once per item. The branch’s output depends on one input only, so a collection of $N$ images needs $N$ forward passes, after which any pair is compared by a cheap operation on stored vectors. This is the property that Žbontar and LeCun (2016) built their fast stereo network around, and that Zagoruyko and Komodakis (2015) give up in their more accurate 2-channel network.

Building Blocks

• Branch (encoder). Any network that maps one input to a vector or feature map: the convolutional networks of Chopra et al. (2005), Zagoruyko and Komodakis (2015) and MatchNet (Han et al., 2015); DeepFace’s face-recognition network without its top layer (Taigman et al., 2014); a ResNet or ViT in self-supervised learning. The branches are copies of one set of weights, not separate networks.
• Comparison module. Three forms recur:
  • a fixed distance or similarity, such as the Euclidean distance (Simo-Serra et al., 2015), the $\ell_1$ distance (Chopra et al., 2005) or the cosine similarity (Žbontar and LeCun’s fast architecture);
  • a learned head on a symmetric combination of the two vectors, such as DeepFace’s weighted absolute difference $\sum_i \alpha_i \lvert f_1[i] - f_2[i] \rvert$ followed by a logistic unit;
  • a learned head on the concatenated vectors, as in Zagoruyko and Komodakis’s “siam” network, MatchNet’s metric network and Žbontar and LeCun’s accurate architecture.
• Loss. A pair loss on the distance (Chopra et al.’s loss, the contrastive loss), a cross-entropy loss on a same/different output (DeepFace, MatchNet, Žbontar and LeCun’s accurate architecture), a hinge on scores (Zagoruyko and Komodakis; Žbontar and LeCun’s fast architecture, which compares a positive and a negative pair), or a ranking loss over triplets.

How Data Flows

In training, a pair $(a, b)$ with a label (matching or not) enters the two branches. Each branch produces a vector, $u = f_\theta(a)$ and $v = f_\theta(b)$, the comparison module produces a scalar, and the loss compares that scalar with the label. Backpropagation sends gradients down both branches; because the branches are one set of weights, the two contributions add up. Chopra et al. (2005) update the shared parameter vector with the sum of the gradients contributed by the two sub-networks.

At test time the network can be taken apart. MatchNet’s authors describe this as a two-stage pipeline (Han et al., 2015, Sec. 4.2). To score $N_1 \times N_2$ pairs of patches from two images, they first run the feature network once on every patch, then pair the stored features and run only the metric network on each pair. On one NVIDIA K40 GPU they report 3.56K patches per second for the feature network (without bottleneck) and 416.6K pairs per second for the metric network ($B = 128$, $F = 512$). Žbontar and LeCun (2016, Sec. 3.3) go further for stereo:

• they run the branch once over the whole image rather than once per patch;
• with a cosine comparison, the only step repeated for every disparity is a dot product;
• with a learned head, its fully connected layers, rewritten as $1 \times 1$ convolutions, still run once per disparity, and they call this the bottleneck of the accurate architecture.

For retrieval or verification against a gallery, the gallery’s vectors are computed once and stored, and a query costs one forward pass plus comparisons. With a fixed distance those comparisons can use a nearest-neighbor index, which Simo-Serra et al. (2015) give as a reason for comparing their descriptors by the Euclidean distance rather than a learned metric network.

Mathematical Formulation

Let $f_\theta$ be the branch and $h$ the comparison module. A Siamese network scores a pair as

$$s(a, b) = h\big(f_\theta(a), f_\theta(b)\big).$$

Symmetry. If $h$ is symmetric, $h(u, v) = h(v, u)$, then $s(a, b) = s(b, a)$ for every pair, for any $\theta$, trained or not. This holds for a distance, for the cosine similarity and for any head applied to $\lvert u - v \rvert$ or $u \odot v$. It fails for a head on the concatenation $[u; v]$, which sees the two vectors in different input positions. It also fails for a pseudo-Siamese network, $s(a, b) = h(f_\theta(a), g_\phi(b))$ with separate weights, and for a 2-channel network $s(a, b) = F_\psi([a; b])$, which stacks the two inputs as channels of one image.

Self-comparison. If $h$ is a distance, $s(a, a) = d(f_\theta(a), f_\theta(a)) = 0$ for every input, so an input is always at least as close to itself as to anything else. If $h$ acts on $\lvert u - v \rvert$, then $s(a, a) = h(\mathbf{0})$ is the same constant for every input. Neither holds for a pseudo-Siamese network: $d(f_\theta(a), g_\phi(a))$ is zero only if training has made the two branches agree on $a$.

Gradient. With a loss $L(s(a, b))$ and shared parameters $\theta$, the chain rule gives

$$\frac{\partial L}{\partial \theta} = \frac{\partial L}{\partial u}\,\frac{\partial f_\theta(a)}{\partial \theta} + \frac{\partial L}{\partial v}\,\frac{\partial f_\theta(b)}{\partial \theta},$$

the sum of the gradients that each branch would receive on its own, which is the update Chopra et al. (2005) describe. A pseudo-Siamese network updates $\theta$ with the first term and $\phi$ with the second.

Cost of comparing $N$ items. To score all pairs of $N$ distinct items, a Siamese network needs $N$ branch passes and $N(N-1)/2$ evaluations of a symmetric head, or $N(N-1)$ of an asymmetric one. A pseudo-Siamese network needs $2N$ passes, since each item may appear on either side. A 2-channel network needs a full pass for every ordered pair, $N(N-1)$ passes, because nothing is computed from one input alone. Zagoruyko and Komodakis (2015) make the same point in words: the 2-channel network needs all combinations of patches to be tested against each other in a brute-force manner.

Triplet networks. A triplet network runs three copies of the branch on an anchor $x$, a positive $x^+$ of the same class and a negative $x^-$ of another. Wang et al. (2014) fed a query, a positive and a negative image into three identical networks with shared architecture and parameters and placed a parameter-free ranking layer on top that evaluates a hinge loss on the triplet. Hoffer and Ailon’s triplet network, read here only through Hoffer’s Torch code, whose README says it replicates the paper’s results (TripletNet, commit 98d80f5; the paper itself was not read), builds the copies by cloning one embedding network with shared weights and biases, computes the Euclidean distances $d^- = \lVert f(x) - f(x^-) \rVert$ and $d^+ = \lVert f(x) - f(x^+) \rVert$, and in DistanceRatioCriterion applies a softmax to the pair $(d^-, d^+)$ and a mean-squared error against the target $(1, 0)$. Writing $\sigma$ for the logistic function, the softmax gives $(\sigma(d^- - d^+), \sigma(d^+ - d^-))$, and the squared error averaged over the two entries simplifies to

$$\ell(x, x^+, x^-) = \tfrac{1}{2}\big[(\sigma(d^- - d^+) - 1)^2 + \sigma(d^+ - d^-)^2\big] = \sigma\big(d^+ - d^-\big)^2 .$$

It follows that the loss depends only on the difference $d^+ - d^-$. It is unchanged when both distances grow by the same amount, but not when the embedding is rescaled: multiplying $f$ by $c > 1$ multiplies the difference by $c$, which lowers the loss of every triplet already in the right order and raises the loss of every triplet in the wrong order, while the order itself does not change. The example below checks this numerically. FaceNet’s margin-based triplet loss is covered, with the mining it needs, in its own article.

Why It Works

Sharing weights builds an assumption into the architecture: both inputs are the same kind of thing and should be described the same way. Several consequences follow.

• Fewer parameters. A Siamese network has the parameters of one branch. DeepFace’s Siamese network has roughly the same number of parameters as the original network, since much of it is shared between the two replicas, but requires twice the computation (Taigman et al., 2014).
• A comparison that transfers. The learned map is applied identically to items the network has never seen, which is what Chopra et al. (2005) needed for faces of people absent from training.
• Invariance from data. Chopra et al. contrast learned invariances with hand-designed ones such as tangent distance or elastic matching. Two images that the labels call “same” are pushed to one point of the output space, so the branch must discard whatever differs between them. Chen and He (2021) make the same argument for self-supervised learning: just as convolution’s weight sharing models translation invariance, weight-sharing Siamese networks can model invariance to more complicated transformations, such as data augmentations.

Weight sharing does not stop the network from collapsing. Without a term that keeps different inputs apart, a constant branch gives every pair the same output and minimizes any loss that only pulls matching pairs together. Chopra et al. (2005) call this a catastrophic collapse and introduce a contrastive term to prevent it. Chen and He (2021) list contrastive negatives, clustering and BYOL’s momentum encoder as strategies that self-supervised Siamese networks use against it, and report that their SimSiam learns useful representations with none of negative pairs, large batches or a momentum encoder, with a stop-gradient operation playing an essential role in preventing collapse. The losses are covered in contrastive loss, and the collapse argument in self-supervised learning.

Training Considerations

• Pairs and their balance. Training sets of pairs are lopsided: for their AR/FERET experiments Chopra et al. (2005) report that taking all combinations of two images gave 71,628 genuine and 11,096,376 impostor pairs, and they trained on 140,000 pairs split evenly between the two. Most negative pairs soon contribute nothing. Simo-Serra et al. (2015) observed that once their network performed reasonably, most non-corresponding pairs already lay beyond the loss’s margin, so the gradient estimate became small and noisy, and they searched batches for hard negatives (and positives) instead. Metric learning covers sampling and mining in general.
• What to train. DeepFace trains only the two topmost layers of its Siamese network, to prevent overfitting on the verification task (Taigman et al., 2014); the DeepFace article explains why the pairs available from LFW could not constrain more.
• Augmentation. Zagoruyko and Komodakis (2015) flipped both patches of a pair horizontally and vertically and rotated them by 90°, 180° and 270°. Žbontar and LeCun (2016) took positive examples up to one pixel away from the true match, because their cross-based cost aggregation worked better when near matches also scored low.
• Loss per architecture. Žbontar and LeCun used a hinge loss on their fast architecture, whose final cosine similarity does not suit a cross-entropy loss, and binary cross-entropy on the accurate one, where experiments showed it beat the hinge.
• Symmetric training data for asymmetric models. A concatenation head or a 2-channel network scores $(a, b)$ and $(b, a)$ differently unless training shows it both orders. In our reading, presenting each pair in both orders, or symmetrizing the score at test time, is the simple remedy; none of the papers read here prescribes it.

Variants

Design	Weights of the two branches	Comparison	Passes to score all pairs of $N$ items	Symmetric by construction
Siamese, fixed distance	shared	distance or cosine	$N$	yes
Siamese, symmetric learned head	shared	head on $\lvert u-v \rvert$	$N$	yes
Siamese, concatenation head	shared	head on $[u; v]$	$N$, plus an $N(N-1)$-fold head	no
Pseudo-Siamese	separate	any	$2N$	no
2-channel	none: one network on the stacked pair	final layer	$N(N-1)$	no
Triplet network	shared, three copies	two distances, ranking loss	$N$	—

Siamese, pseudo-Siamese and 2-channel (Zagoruyko & Komodakis, 2015). Zagoruyko and Komodakis compared the three basic designs for $64 \times 64$ grayscale patches, all trained from scratch with a hinge loss:

• Siamese (“siam”): two branches with the same architecture and the same weights, whose outputs are concatenated and passed to two fully connected layers.
• Pseudo-Siamese: the same structure with uncoupled weights, which in their words gives more flexibility than a Siamese network but keeps its efficiency at test time.
• 2-channel: stacks the two patches as a 2-channel image from the first layer on. It has no direct notion of a descriptor and must test every combination of patches.

They added a central-surround two-stream version, in which one stream sees the central $32 \times 32$ crop at full resolution and the other the whole patch downsampled by two. They also tried spatial pyramid pooling for patches of any size.

On the local image patches benchmark (Yosemite, Notre Dame and Liberty, with patches around Difference-of-Gaussians points), the error measure is the false positive rate at 95% recall (FPR95, lower is better). Averaged over the six train–test combinations (their Table 1), the results were:

• 2ch-2stream 4.19, 2ch-deep 4.27 and 2ch 5.63;
• siam 10.07 and pseudo-siam 9.62;
• siam-2stream 7.63;
• 9.67 for siam-2stream’s branches compared by the $\ell_2$ distance;
• 10.38 for the earlier descriptor of Simonyan, Vedaldi and Zisserman.

Their conclusion was that processing the two patches jointly from the first layer matters. The text says that the pseudo-Siamese network was better than the corresponding Siamese one. That holds for the mean over six combinations (9.62 against 10.07). Over the four combinations of earlier work, mean(1,4), the table gives 10.33 for pseudo-siam against 10.31 for siam, so there pseudo-siam is marginally worse, not better.

Learned metric networks. MatchNet (Han et al., 2015) pairs a convolutional feature network with a metric network of three fully connected layers on the concatenated features and a two-way softmax, trained with cross-entropy. Its authors describe their two-tower structure with tied parameters as related to the Siamese network, which in their description uses carefully designed loss functions instead of a learned metric network. They note that patches from two quite different domains could call for towers that share fewer layers or differ in structure.

Fast against accurate (Žbontar & LeCun, 2016). Žbontar and LeCun’s MC-CNN offers two architectures for stereo matching costs:

• Fast: a Siamese network whose branches end in normalized vectors compared by a dot product.
• Accurate: concatenates the two vectors and passes them through fully connected layers with a sigmoid output.

On KITTI 2012, measured as the percentage of pixels whose disparity is off by more than three pixels on the test set, the accurate architecture had an error of 2.43% at 67 seconds per image pair and the fast one 2.82% at 0.8 seconds (their Table 1, leaderboard as of October 2015). On KITTI 2015 their Table 2 gives 3.89% for the accurate and 4.62% for the fast architecture. The text gives 4.46% for the fast one. Cost volume and stereo matching describe the rest of the pipeline.

Triplet networks. These are described under Mathematical Formulation (Wang et al., 2014; Hoffer’s code for Hoffer and Ailon’s network); triplet loss covers FaceNet’s margin-based loss and its mining.

Siamese networks in self-supervised learning. Chen and He (2021) define Siamese networks as weight-sharing neural networks applied on two or more inputs, and note that recent self-supervised methods such as SimCLR, SwAV, BYOL and MoCo generally involve some form of one: the two inputs are two augmentations of one image, and the methods differ in how they avoid collapse. They treat MoCo and BYOL, whose second branch is a momentum encoder, as Siamese networks with “indirect” weight-sharing. Their SimSiam uses directly shared weights with a predictor on one branch and a stop-gradient on the other. Contrastive learning and self-supervised learning cover these methods.

Example: What Sharing Guarantees, and What It Costs

The script below builds five pair networks with fixed random weights and trains none of them. With random weights, any property that holds is guaranteed by the architecture, not learned. The five networks are:

1. siamese, L2: a shared branch compared by the Euclidean distance;
2. siamese, |u-v| head: a shared branch with a logistic head on $\lvert u - v \rvert$ in the style of DeepFace;
3. siamese, concat head: a shared branch with a ReLU layer of 32 units on $[u; v]$ in the style of Zagoruyko and Komodakis’s “siam”;
4. pseudo-siamese, L2: two independently drawn branches;
5. 2-channel: the same trunk with a 2-channel first layer.

Each branch is a $7 \times 7$ convolution with 8 filters, ReLU, $2 \times 2$ max-pooling, a $5 \times 5$ convolution with 16 filters, ReLU and global average pooling, giving a 16-dimensional vector. Weights are He-normal, without biases, for weight seeds 0–4.

The data are twelve $64 \times 64$ grayscale patches, four each from scikit-image’s bundled camera, astronaut and coffee images on a fixed grid, and a copy of each. In one condition the copies are exact duplicates. In the other each copy is cropped 3 rows down and 4 columns right of its base and carries Gaussian noise of standard deviation 0.05 (intensities in $[0, 1]$; arrays are indexed row, column), which gives 24 items. The script prints:

• the largest asymmetry $\lvert s(a,b) - s(b,a) \rvert$ over all ordered pairs, also divided by the standard deviation of the scores;
• a self-comparison measure;
• the share of items whose nearest other item is their own copy (or base), for the two distance networks;
• the trunk passes, head evaluations and multiply–accumulate operations (MACs) needed to score every pair of distinct items, at $N = 24$ and by formula at $N = 10{,}000$;
• the loss of Hoffer and Ailon’s triplet network, as Hoffer’s code computes it and in the closed form above, on 96 triplets: each base, its shifted copy, and each of the eight bases from the two other images. The embeddings are scaled by $c$.

Distances are smaller for more similar pairs; the scores of the untrained heads have no meaningful direction and are used only for the symmetry and self-comparison checks.

The hypotheses and tolerances were fixed before the script was first run:

• H1: networks 1 and 2 are exactly symmetric; the others have a relative asymmetry of at least 0.05 in every seed.
• H2: $d(a, a) = 0$ for network 1; $s(a, a)$ is constant for network 2; the pseudo-Siamese $d(a,a)$ is at least 0.1 of the median $d(a, b)$; and the self-scores of networks 3 and 5 spread by at least 0.05 of the score standard deviation.
• H3: network 1 finds every exact duplicate, and network 4 at most half of them.
• H4: network 1 finds fewer shifted copies than raw pixels do (0.958) in every seed, and network 4’s mean over the seeds is no higher than network 1’s.
• H5: the pass and evaluation counts equal the formulas.
• H6: the explicit and closed-form triplet losses agree within $10^{-12}$; the share of correctly ordered triplets does not change with $c$; and the mean loss falls with $c$ for correctly ordered triplets and rises for wrongly ordered ones.

The script uses NumPy 2.3.5 and scikit-image 0.25.2 on one BLAS thread and runs in about 3 seconds; two runs printed identical output. After the plan’s pilot, and before the first run, two columns were added to the cost table on the editor’s approval: the head MACs and the total MACs at $N = 10{,}000$. They print arithmetic the text below relies on and decide no check.

import os
for v in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS"):
    os.environ[v] = "1"                      # one BLAS thread: deterministic sums
import numpy as np
from numpy.lib.stride_tricks import sliding_window_view
from skimage import data, color

# ---------------------------------------------------------------- data
P, SHIFT, SIGMA = 64, (3, 4), 0.05           # patch size, copy offset (rows, cols), noise std
IMAGES = [data.camera() / 255.0, color.rgb2gray(data.astronaut()), color.rgb2gray(data.coffee())]

def make_items(shift, sigma, rng):
    """12 base patches (4 per image, on a fixed grid) and, for each, a copy cropped
    `shift` pixels away plus Gaussian noise. Items 0-11 are bases, 12-23 their copies."""
    bases, copies = [], []
    for img in IMAGES:
        h, w = img.shape
        for r in (h // 4 - P // 2, 3 * h // 4 - P // 2):
            for c in (w // 4 - P // 2, 3 * w // 4 - P // 2):
                bases.append(img[r:r + P, c:c + P])
                copies.append(img[r + shift[0]:r + shift[0] + P, c + shift[1]:c + shift[1] + P]
                              + sigma * rng.standard_normal((P, P)))
    return np.array(bases + copies)

# ---------------------------------------------------------------- fixed random networks
PASSES = [0]                                 # forward passes through a convolutional trunk

def conv(x, w):
    """Valid cross-correlation: x (C, H, W), w (K, C, k, k) -> (K, H-k+1, W-k+1)."""
    K, C, k, _ = w.shape
    win = sliding_window_view(x, (k, k), axis=(1, 2))           # (C, H', W', k, k)
    Hh, Ww = win.shape[1:3]
    cols = win.transpose(1, 2, 0, 3, 4).reshape(Hh * Ww, C * k * k)
    return (cols @ w.reshape(K, -1).T).T.reshape(K, Hh, Ww)

def trunk(x, W1, W2):
    """conv 7x7 (8) -> ReLU -> 2x2 max-pool -> conv 5x5 (16) -> ReLU -> global average pool."""
    PASSES[0] += 1
    h = np.maximum(conv(x, W1), 0)                                # (8, 58, 58)
    h = h.reshape(8, 29, 2, 29, 2).max(axis=(2, 4))               # (8, 29, 29)
    return np.maximum(conv(h, W2), 0).mean(axis=(1, 2))           # (16,)

def trunk_macs(c_in):                                             # multiply-adds per pass
    return 58 * 58 * 8 * c_in * 7 * 7 + 25 * 25 * 16 * 8 * 5 * 5

def he(rng, *shape):
    return rng.standard_normal(shape) * np.sqrt(2.0 / np.prod(shape[1:]))

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def make_models(seed):
    """Five ways to compare two patches, all with fixed random weights (no training)."""
    rng = np.random.default_rng(seed)
    W = (he(rng, 8, 1, 7, 7), he(rng, 16, 8, 5, 5))              # the shared branch
    W_b = (he(rng, 8, 1, 7, 7), he(rng, 16, 8, 5, 5))            # a second, unshared branch
    W_2ch = (he(rng, 8, 2, 7, 7), he(rng, 16, 8, 5, 5))          # 2-channel trunk
    alpha = rng.standard_normal(16) / 4                           # weights of the |u - v| head
    H1, h2 = he(rng, 32, 32), rng.standard_normal(32) / np.sqrt(32)  # concatenation head
    w_out = rng.standard_normal(16) / 4                           # 2-channel decision layer
    f = lambda x: trunk(x[None], *W)
    g = lambda x: trunk(x[None], *W_b)
    l2 = lambda u, v: np.linalg.norm(u - v)
    # name: (branch for first input, branch for second input, comparison, head cost in MACs)
    return {
        "siamese, L2":          (f, f, l2, 16),
        "siamese, |u-v| head":  (f, f, lambda u, v: sigmoid(alpha @ np.abs(u - v)), 16),
        "siamese, concat head": (f, f, lambda u, v: h2 @ np.maximum(H1 @ np.concatenate([u, v]), 0), 32 * 32 + 32),
        "pseudo-siamese, L2":   (f, g, l2, 16),
        "2-channel":            (None, None, lambda a, b: w_out @ trunk(np.stack([a, b]), *W_2ch), 16),
    }

def score_matrix(model, X):
    """S[i, j] = s(X_i, X_j) for every ordered pair, self-pairs on the diagonal."""
    e1, e2, cmp, _ = model
    if e1 is None:                                                # 2-channel: a full pass per pair
        return np.array([[cmp(a, b) for b in X] for a in X])
    U = [e1(x) for x in X]
    V = U if e2 is e1 else [e2(x) for x in X]
    return np.array([[cmp(u, v) for v in V] for u in U])

def all_pairs_cost(model, X, symmetric):
    """Score every pair of distinct items (both orders unless the head is symmetric);
    count trunk passes and head evaluations."""
    e1, e2, cmp, _ = model
    n, PASSES[0], evals = len(X), 0, 0
    pairs = [(i, j) for i in range(n) for j in range(n) if (i < j if symmetric else i != j)]
    if e1 is None:
        for i, j in pairs:
            cmp(X[i], X[j])
        return PASSES[0], len(pairs)
    U = [e1(x) for x in X]
    V = U if e2 is e1 else [e2(x) for x in X]
    for i, j in pairs:
        cmp(U[i], V[j])
    return PASSES[0], len(pairs)

# ---------------------------------------------------------------- data checks
X_dup = make_items((0, 0), 0.0, np.random.default_rng(100))     # copies are exact duplicates
X = make_items(SHIFT, SIGMA, np.random.default_rng(100))         # shifted, noisy copies
X_clean = make_items(SHIFT, 0.0, np.random.default_rng(100))     # the same copies without noise
n, half = len(X), len(X) // 2
partner = (np.arange(n) + half) % n
off = ~np.eye(n, dtype=bool)
r, c = SHIFT
print(f"items {X.shape}; duplicates identical: {bool((X_dup[:half] == X_dup[half:]).all())}; "
      f"noise-free copies are bases moved by {SHIFT}: "
      f"{bool((X_clean[half:, :-r, :-c] == X_clean[:half, r:, c:]).all())}; "
      f"noise std {np.std(X[half:] - X_clean[half:]):.4f}")

def retrieval(S):
    """Share of items whose nearest other item (smallest score) is their own copy/base."""
    D = np.where(off, S, np.inf)
    return np.mean(D.argmin(1) == partner)

pix = lambda Y: np.linalg.norm(Y.reshape(n, 1, -1) - Y.reshape(1, n, -1), axis=2)
print(f"raw pixels, L2: copy found as nearest item: duplicates {retrieval(pix(X_dup)):.3f}, "
      f"shifted+noisy {retrieval(pix(X)):.3f} (chance {1 / (n - 1):.3f})")

# ---------------------------------------------------------------- symmetry and self-comparison
SEEDS = range(5)
rows = {}
for seed in SEEDS:
    for name, model in make_models(seed).items():
        S = score_matrix(model, X)
        asym = np.abs(S - S.T)[off].max()
        rec = rows.setdefault(name, {"asym": [], "rel": [], "self": [], "dup": [], "shift": []})
        rec["asym"].append(asym)
        rec["rel"].append(asym / S[off].std())
        d = np.diag(S)
        if name.endswith("L2"):                                   # distances: d(a, a) vs d(a, b)
            rec["self"].append(d.min() / np.median(S[off]) if name.startswith("pseudo") else d.max())
            rec["dup"].append(retrieval(score_matrix(model, X_dup)))
            rec["shift"].append(retrieval(S))
        else:                                                     # scores: spread of s(a, a)
            rec["self"].append(np.ptp(d) / S[off].std() if "concat" in name or "2-ch" in name else np.ptp(d))

rng_txt = lambda v, fmt: f"{min(v):{fmt}} to {max(v):{fmt}}"
print("\nweight seeds 0-4, 24 items, all ordered pairs")
print("model                  max|s(a,b)-s(b,a)|      relative to std      self-comparison")
for name, rec in rows.items():
    self_txt = {"siamese, L2": "max d(a,a)", "pseudo-siamese, L2": "min d(a,a)/median d(a,b)",
                "siamese, |u-v| head": "spread of s(a,a)"}.get(name, "spread of s(a,a)/std")
    print(f"{name:22s} {rng_txt(rec['asym'], '.1e'):22s} {rng_txt(rec['rel'], '.3f'):20s} "
          f"{self_txt} {rng_txt(rec['self'], '.3g')}")

print("\ncopy found as nearest item (query through branch 1, others through branch 2)")
for name in ("siamese, L2", "pseudo-siamese, L2"):
    print(f"{name:22s} duplicates {rng_txt(rows[name]['dup'], '.3f')}   "
          f"shifted+noisy {rng_txt(rows[name]['shift'], '.3f')}   per seed "
          + " ".join(f"{v:.3f}" for v in rows[name]["shift"]))

# ---------------------------------------------------------------- cost of comparing everything
print("\ncost of scoring all pairs of distinct items (seed 0)")
print("model                  trunk passes  head evals   trunk MACs, N=24   N=10,000: trunk MACs   head MACs   total MACs")
for name, model in make_models(0).items():
    sym = name in ("siamese, L2", "siamese, |u-v| head")
    passes, evals = all_pairs_cost(model, X, sym)
    m = trunk_macs(2 if model[0] is None else 1)
    N = 10_000
    big = {"2-channel": N * (N - 1), "pseudo-siamese, L2": 2 * N}.get(name, N) * m
    head = (N * (N - 1) // 2 if sym else N * (N - 1)) * model[3]
    print(f"{name:22s} {passes:12d} {evals:11d} {passes * m:18.3e} {big:22.3e} {head:11.3e} {big + head:12.3e}")

# ---------------------------------------------------------------- triplet network loss
U = np.array([make_models(0)["siamese, L2"][0](x) for x in X])
img_of = np.arange(n) % half // 4                                 # source image of each item
trip = [(a, a + half, b) for a in range(half) for b in range(half) if img_of[b] != img_of[a]]
A, Pz, Ng = (U[[t[k] for t in trip]] for k in range(3))
print(f"\ntriplet network on {len(trip)} triplets (anchor, its shifted copy, a patch of another image)")
print("scale c   correctly ordered   mean loss (correct)   mean loss (wrong)   max |softmax+MSE - closed form|")
for scale in (0.5, 1, 2, 4):
    dp = np.linalg.norm(scale * (A - Pz), axis=1)                 # d+
    dn = np.linalg.norm(scale * (A - Ng), axis=1)                 # d-
    s = np.exp(np.stack([dn, dp], 1))
    s /= s.sum(1, keepdims=True)                                  # softmax over (d-, d+)
    mse = ((s[:, 0] - 1) ** 2 + s[:, 1] ** 2) / 2                 # target (1, 0), averaged
    closed = sigmoid(dp - dn) ** 2
    ok = dp < dn
    wrong = f"{mse[~ok].mean():.4f}" if (~ok).any() else "none"
    print(f"{scale:7g}   {ok.mean():17.3f}   {mse[ok].mean():19.4f}   {wrong:>17s}   {np.abs(mse - closed).max():.1e}")

items (24, 64, 64); duplicates identical: True; noise-free copies are bases moved by (3, 4): True; noise std 0.0500
raw pixels, L2: copy found as nearest item: duplicates 1.000, shifted+noisy 0.958 (chance 0.043)

weight seeds 0-4, 24 items, all ordered pairs
model                  max|s(a,b)-s(b,a)|      relative to std      self-comparison
siamese, L2            0.0e+00 to 0.0e+00     0.000 to 0.000       max d(a,a) 0 to 0
siamese, |u-v| head    0.0e+00 to 0.0e+00     0.000 to 0.000       spread of s(a,a) 0 to 0
siamese, concat head   8.7e-02 to 1.1e+00     0.410 to 3.956       spread of s(a,a)/std 1.29 to 4.52
pseudo-siamese, L2     3.7e-01 to 2.2e+00     0.260 to 2.836       min d(a,a)/median d(a,b) 0.318 to 0.383
2-channel              8.8e-02 to 6.9e-01     1.252 to 3.138       spread of s(a,a)/std 1.8 to 3.9

copy found as nearest item (query through branch 1, others through branch 2)
siamese, L2            duplicates 1.000 to 1.000   shifted+noisy 0.042 to 0.708   per seed 0.500 0.375 0.042 0.625 0.708
pseudo-siamese, L2     duplicates 0.083 to 0.333   shifted+noisy 0.000 to 0.250   per seed 0.083 0.000 0.042 0.250 0.083

cost of scoring all pairs of distinct items (seed 0)
model                  trunk passes  head evals   trunk MACs, N=24   N=10,000: trunk MACs   head MACs   total MACs
siamese, L2                      24         276          7.965e+07              3.319e+10   7.999e+08    3.399e+10
siamese, |u-v| head              24         276          7.965e+07              3.319e+10   7.999e+08    3.399e+10
siamese, concat head             24         552          7.965e+07              3.319e+10   1.056e+11    1.388e+11
pseudo-siamese, L2               48         552          1.593e+08              6.637e+10   1.600e+09    6.797e+10
2-channel                       552         552          2.560e+09              4.637e+14   1.600e+09    4.637e+14

triplet network on 96 triplets (anchor, its shifted copy, a patch of another image)
scale c   correctly ordered   mean loss (correct)   mean loss (wrong)   max |softmax+MSE - closed form|
    0.5               0.958                0.2004              0.2575   1.4e-16
      1               0.958                0.1615              0.2652   1.1e-16
      2               0.958                0.1116              0.2809   1.1e-16
      4               0.958                0.0688              0.3136   1.1e-16

The two data checks hold: the duplicates are identical, the noise-free copies are their bases moved by (3, 4), and the noise has the planned standard deviation. Raw pixels find the partner for every duplicate and for 23 of 24 shifted copies (0.958), so the labels exist and the task is easy in pixel space.

Symmetry and self-comparison. All parts of H1 and H2 held, in every seed and for every network:

• Networks 1 and 2 have an asymmetry of exactly zero in all five seeds. Network 1 has $d(a, a) = 0$ for every item, and network 2 gives every item the same self-score.
• Weight sharing alone is not enough. The concatenation head shares its branch and is still asymmetric, by 0.410 to 3.956 times the score standard deviation, because it sees its two inputs in different positions. Its self-scores also vary across items.
• The pseudo-Siamese pair is asymmetric (0.260 to 2.836). Its distance from an item to itself is 0.318 to 0.383 of the median distance between different items, not zero, because the two branches describe the same patch in unrelated coordinates.
• The 2-channel network is asymmetric (1.252 to 3.138), with self-scores that spread by 1.8 to 3.9 standard deviations.

Retrieval. H3 held. The shared network finds every exact duplicate in every seed, a direct consequence of $d(a, a) = 0$. The pseudo-Siamese network finds 2 to 8 of the 24 duplicates (0.083 to 0.333).

H4 held, with a caveat. The shared random network found the shifted, noisy copy for 0.042 to 0.708 of the items depending on the seed, below the 0.958 of raw pixels in every seed. Weight sharing guarantees identity, not tolerance to shifts: the invariance Chopra et al. describe comes from training. The pseudo-Siamese network did worse on average: 0.092 against 0.450, means computed from the printed per-seed values. It was not worse in every seed: in seed 2 both found the copy for one item in 24 (0.042).

A related published example is the random-feature stand-in in LPIPS, which scores a one-pixel shift against other distortions on a two-alternative test. The measure here is different, nearest-item retrieval with shared against unshared branches.

Cost. H5 held exactly. The counted passes are $N = 24$ for the three Siamese networks, $2N = 48$ for the pseudo-Siamese pair and $N(N-1) = 552$ for the 2-channel network. The symmetric heads needed 276 evaluations and the asymmetric ones 552.

At $N = 10{,}000$ the 2-channel network needs about $4.6 \times 10^{14}$ MACs, about 13,600 times the $3.4 \times 10^{10}$ of the Siamese network with a distance. The concatenation head, small as it is (1,056 MACs per pair), evaluated on all $N(N-1)$ ordered pairs, costs about three times as much as running the trunk on every item. This is the same kind of bottleneck that Žbontar and LeCun describe for the fully connected layers of their accurate architecture. The pseudo-Siamese pair doubles the trunk cost.

Triplet loss. H6 held. The explicit softmax-and-squared-error loss and the closed form $\sigma(d^+ - d^-)^2$ agree to about $10^{-16}$ at every scale. The share of correctly ordered triplets is 0.958 (92 of 96) at every scale, since rescaling cannot change which distance is smaller. Scaling the embedding from 0.5 to 4 lowers the mean loss of the correctly ordered triplets from 0.2004 to 0.0688 and raises that of the four wrongly ordered ones from 0.2575 to 0.3136. A network trained with this loss can lower it by growing its embedding as well as by reordering triplets. This follows from the closed form; the example does not train a network to show it.

Use in Computer Vision

• Verification. Signatures (Bromley et al.’s network, as later papers cite it), faces (Chopra et al., 2005; DeepFace’s Siamese verification head) and face recognition systems that compare learned embeddings.
• One-shot recognition. As few-shot recognition describes it, Koch, Zemel and Salakhutdinov’s convolutional Siamese network on Omniglot verifies pairs of characters and classifies a new character by its most similar support image.
• Patch matching and descriptors. Zagoruyko and Komodakis’s networks, MatchNet and Simo-Serra et al.’s Euclidean-distance descriptors. Feature descriptors and learned local features cover learned descriptors as a family.
• Stereo and flow. MC-CNN’s matching cost (cost volume). As their articles describe, FlowNet’s FlowNetC and PWC-Net run the two frames through identical feature streams before comparing them, and DUSt3R encodes both views with one ViT encoder.
• Tracking. Siamese trackers pass a target template and a search region through one shared network and cross-correlate the results.
• Change detection. Siamese encoders for bitemporal images, including FC-Siam-diff’s absolute-difference skip connections (change detection), where early-fusion designs (the 2-channel idea) are the main alternative.
• Retrieval and re-identification. Image retrieval and person re-identification compare a query embedding with stored gallery embeddings, the setting in which computing each description once pays off most. Wang et al. (2014) trained their triplet network for fine-grained image similarity.
• Self-supervised pretraining. Chen and He (2021) describe SimCLR, MoCo, BYOL, SwAV and SimSiam as Siamese networks applied to two augmented views of one image.

Limitations

• Late interaction. A branch must describe each input without seeing the other, so the comparison can only use what survived in the vector. Zagoruyko and Komodakis’s 2-channel networks, which compare the patches from the first layer, had the lowest FPR95 on their benchmark, and Žbontar and LeCun’s learned head was more accurate than the dot product. The price is computation that grows with the number of pairs instead of the number of items.
• No invariance without training. Weight sharing guarantees that identical inputs match, not that a shifted, relit or re-posed view does; the example’s random network missed most shifted copies in some seeds. The invariances come from the pairs used in training.
• Collapse. A loss without a term that separates non-matching inputs is minimized by a constant branch (Chopra et al., 2005); Siamese methods guard against it with negatives, a margin, clustering, a momentum encoder or a stop-gradient.
• Asymmetric inputs. When the two inputs come from different sources, such as different sensors, sketches and photographs, or a template and a search region, identical treatment can be the wrong assumption. MatchNet’s authors suggest towers that share fewer layers in such cases, and pseudo-Siamese networks keep separate weights.
• Pair sampling. The number of possible pairs grows quadratically and most are uninformative, so training depends on sampling and mining (metric learning).

Important Models

• Bromley et al. (1993): the signature-verification paper whose title names the “Siamese” network and which later papers cite for the architecture (not read here).
• Chopra, Hadsell & LeCun (2005): convolutional Siamese network for face verification with a contrastive energy-based loss; Hadsell, Chopra and LeCun (2006) used the architecture to learn invariant low-dimensional mappings.
• DeepFace (Taigman et al., 2014): Siamese verification on top of a face-classification network.
• Deep ranking (Wang et al., 2014): triplet network with three shared-weight multi-scale networks for fine-grained image similarity.
• Zagoruyko & Komodakis (2015): Siamese, pseudo-Siamese, 2-channel and central-surround networks for comparing patches.
• MatchNet (Han et al., 2015): a feature tower and a learned metric network, run as a two-stage pipeline.
• Simo-Serra et al. (2015): Siamese training of 128-dimensional descriptors compared by the Euclidean distance, with hard-pair mining.
• Koch et al. (2015): Siamese network for one-shot character recognition.
• MC-CNN (Žbontar & LeCun, 2016): fast (cosine) and accurate (learned head) Siamese networks for stereo matching costs.
• SiamFC and later Siamese trackers: covered in Siamese trackers.
• SimSiam (Chen & He, 2021): a Siamese network for self-supervised learning without negatives or a momentum encoder.