Video Object Segmentation

Video object segmentation (VOS) produces a pixel mask for one or more chosen objects in every frame of a video, keeping each object’s identity from frame to frame. In its most studied form the objects are specified by their masks in the first frame and the method must carry them through the rest of the video, past motion, deformation, occlusion and look-alike objects; other forms take user scribbles or clicks, or no input at all. The task sits between semantic segmentation, which labels pixels without following objects over time, and object tracking, which follows objects but usually reports boxes. Its first-frame form is often called one-shot or semi-supervised VOS. The image segmentation hub places it among the other segmentation tasks; this article covers its settings, the main families of methods, the DAVIS and YouTube-VOS benchmarks and the J and F measures they share.

Problem Definition

A video is a sequence of frames $I_1, \dots, I_T$. For each target object $o$, a method outputs a binary mask $M_t^o$ in every frame, or equivalently one label map per frame whose values are object identities, with 0 for background. Perazzi et al. (2016), who introduced the DAVIS benchmark, define the task as a binary labelling problem that separates foreground objects from the background of a video, a pixel-accurate spatio-temporal bipartition. Two features set it apart from per-frame segmentation:

• The targets are instances, not classes. The object to follow is whatever the first-frame mask or the user’s input designates. It need not belong to a category seen in training; YouTube-VOS deliberately holds out categories for testing.
• Identity must persist. An object that leaves the view or hides behind another and comes back must receive the same label. In the evaluation code a correct empty prediction for an absent object scores as perfect (see Evaluation Metrics).

The DAVIS site separates the settings by the human input available at test time, not during training: the semi-supervised setting, which the site says is better called human-guided, gives the mask of each object of interest in the first frame; the unsupervised, or human non-guided, setting gives nothing.

Input and Output

• Input: the frames, plus the setting’s guidance: first-frame masks (semi-supervised), a bounding box (a variant studied by MaskTrack’s authors), scribbles or clicks added over several rounds (interactive), or nothing (unsupervised).
• Output: a mask per object per frame. Benchmarks store them as indexed PNG images in which each pixel holds the identity of the object it belongs to, so the masks of different objects do not overlap (DAVIS 2019 unsupervised track).
• Online or offline: many methods process frames in order, using only the past, which suits streaming and long videos; others see the whole clip. OSVOS segments every frame independently, so frame order does not matter for it.

Variants

• Semi-supervised VOS. The first-frame masks are given and nothing else; this is the protocol of the DAVIS semi-supervised track and of YouTube-VOS. The DAVIS 2019 rules forbid using the released object categories as extra information.
• Unsupervised VOS. No input at all. In DAVIS 2016, whose sequences each contain one object or two connected ones, it meant separating that foreground from the background, which the authors describe as extending salient object detection to video. The DAVIS 2019 unsupervised track redefined it for several objects: methods return a set of non-overlapping object candidates spanning the video, which should include at least the objects that capture human attention, those most likely to be followed by gaze.
• Interactive VOS. A user marks errors with scribbles, the method updates every frame, and the loop repeats: the video counterpart of interactive segmentation of single images. Quality is measured against the time and the number of interactions spent (DAVIS 2018 and 2019 interactive tracks).
• Promptable visual segmentation. SAM 2 (Ravi et al., 2025) generalizes the task so that clicks, boxes or masks may be given on any frame, and refined at any time; semi-supervised VOS is the special case with one mask prompt on the first frame. SAM is the image model it extends.
• Video instance segmentation. Detect, segment and track every object of a fixed category set, with no first-frame guidance and with a class label for each track (Yang, Fan & Xu, 2019); see below.
• Related formulations. Motion segmentation groups pixels by their motion and is the older unsupervised form; the Freiburg–Berkeley Motion Segmentation (FBMS) set, which YouTube-VOS’s authors count as 59 videos with 139 objects, was built for it. In referring VOS the target is given by a language expression instead of a mask; Seo, Lee and Han (2020) built the Refer-Youtube-VOS dataset for it on top of YouTube-VOS. Long-term VOS concerns videos of minutes rather than seconds.

Challenges

• Occlusion and disocclusion. A hidden object must be reported as absent and found again when it returns; pixels uncovered by a moving object have no counterpart in the previous frame. Propagation methods suffer from both, as the example below shows.
• Drift. Each frame’s errors feed the next frame’s input. DAVIS 2016 found that semi-supervised methods that propagate frame to frame lose accuracy over time, which its decay statistic measures.
• Distractors. Objects that look like the target, such as other people in a crowd or fish in a shoal, attract matching-based methods. FEELVOS’s authors show a shoal of similar fish causing their method to lose some of them.
• Appearance change. Rotation, deformation, lighting and scale change make later frames look unlike the first. DAVIS 2016 defines 15 attributes, among them occlusion, fast motion, motion blur, appearance change, scale variation and background clutter, tags each sequence with those that apply, and reports results per attribute.
• Long videos. A memory of past frames grows with the video; XMem’s authors note that attention-based methods of their time usually struggled with videos longer than a minute on consumer-grade hardware.
• Annotation cost. Dense masks are expensive. DAVIS annotated every frame of short clips; YouTube-VOS annotated every fifth frame to scale up; and SAM 2’s authors note that drawing a high-quality first-frame mask is itself costly at inference time.

Approaches

The methods differ in where they keep what they know about the target: in the weights of a network fine-tuned on the first frame, in the previous frame’s mask, in features of the first frame to match against, or in a memory of many past frames. SAM 2’s authors summarize the deep-learning history in roughly that order: online fine-tuning first, then offline-trained models conditioned on the first frame or on the previous one, then models that attend to all past frames with recurrent networks and transformers.

Before deep learning

DAVIS 2016 grouped the earlier literature by supervision. Unsupervised methods assumed that the object moves differently from its surroundings: Papazoglou and Ferrari (2013), for example, compute optical flow, detect motion boundaries, turn them into inside–outside maps of likely object pixels and refine those with appearance models; they evaluated on more than 1,400 shots and report speeds orders of magnitude above proposal-based methods. Semi-supervised methods propagated the given annotation, often by minimizing an energy defined on a graph, some with fully connected graphs or higher-order potentials for long-range links. One evaluated method, NLC, describes each frame with about 2,000 superpixels; another, KEY, which DAVIS ran with ground-truth hypothesis selection, refines its segments with spatio-temporal graph cuts. On DAVIS 2016 the best of twelve tested methods reached a mean J of 0.641 (NLC, unsupervised), and the authors noted that propagating methods decayed over time while methods with cues at many frames decayed less.

Online fine-tuning

OSVOS (Caelles et al., 2017) adapted a network to the target instead of following it. A fully convolutional network built on VGG and pretrained on ImageNet is first trained on the DAVIS training masks to separate foreground objects from background (the “parent network”), and is then fine-tuned at test time on the one annotated frame. Every other frame is segmented independently, with no temporal model, which the authors argue lets it segment through occlusions and avoids propagating errors. It reached a mean J of 79.8 on the DAVIS 2016 validation set against 68.0 for the previous best; without the first-frame fine-tuning it fell to 52.5, and without the parent network to 64.6 (both without its boundary snapping step). Fine-tuning costs time: the authors report 71.5% at 181 ms per frame and 79.7% at 7.83 s per frame, depending on how long it runs.

Propagation

MaskTrack (Perazzi et al., 2017) feeds the previous frame’s mask, dilated to a rough blob, as a fourth input channel next to the RGB frame, so the network becomes a mask refiner that knows which object to segment. Because the input is only a mask, training pairs can be synthesized from still images by deforming annotated masks with affine and thin-plate-spline warps; the paper trains on about $10^4$ images, and found that removing the deformations collapsed mean IoU on DAVIS from 74.8 to 17.1. The authors tried warping the previous mask with optical flow and found that flow errors offset the gains, so they copy the mask unwarped; a second stream that reads the flow magnitude as an image raised DAVIS mIoU from 74.8 to 78.4, and a CRF to 80.3, but they report that flow-based gains of 1 to 4 points did not hold consistently across datasets. YouTube-VOS’s authors replaced explicit propagation with a convolutional LSTM that carries hidden state across frames, trained end to end on their larger dataset.

Matching

Matching methods compare each pixel of the current frame with the pixels of reference frames in a learned embedding space. PML (Chen et al., 2018) trains an embedding with a modified triplet loss and labels each pixel by its nearest neighbours among the annotated pixels, at 275 ms per frame; it reached J&F 77.4 on the DAVIS 2016 validation set in the paper’s table (its introduction gives 77.5). The same formulation accepts clicked points instead of a mask. FEELVOS (Voigtlaender et al., 2019) combines a global match to the first frame, the distance from each pixel to its nearest first-frame neighbour of each object, with a local match to the previous frame restricted to a window of $k = 15$ positions on its stride-4 embedding map, and feeds both distance maps to a segmentation head trained end to end. Its ablation on the DAVIS 2017 validation set (training on DAVIS 2017 only) shows the two references are complementary: the full model scored J&F 69.1, 56.1 without the first-frame match, 54.9 without any previous-frame match, and 64.2 when the previous-frame match was global rather than local; the local window, the authors explain, uses the small motion between frames to avoid false matches. The first-frame distance maps alone are noisy, with false matches on background regions of similar appearance; the metric learning article covers the embedding losses.

Memory networks

STM (Oh et al., 2019) generalized matching from one or two reference frames to a memory of many. Each memory frame is encoded together with its mask into a key map, used for addressing, and a value map, which carries the mask information; the query frame is encoded into a key and a value as well. Every query location $i$ reads from all memory locations $j$ across space and time:

$$\mathbf{y}_i = \left[\, \mathbf{v}^Q_i,\ \frac{1}{Z} \sum_{j} \exp\!\left(\mathbf{k}^Q_i \cdot \mathbf{k}^M_j\right) \mathbf{v}^M_j \right], \qquad Z = \sum_j \exp\!\left(\mathbf{k}^Q_i \cdot \mathbf{k}^M_j\right),$$

where $[\cdot, \cdot]$ is concatenation and a decoder turns $\mathbf{y}$ into the mask. This is attention with the memory as keys and values, which the authors relate to non-local self-attention while noting that it attends to other frames, not to the frame itself. At inference the memory holds the first frame, the previous frame and one frame in every five. STM pretrains on synthetic clips made by deforming still images and then trains on video. Its memory ablation separates the references that earlier families used: on the YouTube-VOS 2018 validation set (overall score) and DAVIS 2017 validation set (mean J), the first frame alone gave 68.9 and 67.0, the previous frame alone 69.7 and 69.6, both together 78.4 and 77.7, and adding a frame every five steps 79.4 and 79.2, at 0.16 s per frame on DAVIS 2016 against 0.06–0.07 s for the smaller memories. The DAVIS values equal those of its model trained with YouTube-VOS data. Online fine-tuning becomes unnecessary: adding a frame to the memory adapts the model without changing its weights.

STCN (Cheng, Tai & Tang, 2021) computes keys from the image alone, so that the affinity between two frames does not depend on the mask and is computed once per frame and shared by all objects, and replaces the dot product with the negative squared Euclidean distance. The authors found that with the dot product a small fixed subset of memory locations dominates the weights regardless of the query, while the L2 similarity gives every memory location a chance to contribute. With a memory of every fifth frame it reached J&F 85.4 on the DAVIS 2017 validation set at 20.2 frames per second.

XMem (Cheng & Schwing, 2022) targets long videos with three stores modelled on the Atkinson–Shiffrin model of human memory: a sensory memory (the hidden state of a GRU, updated every frame), a high-resolution working memory of recent frames, and a compact long-term memory. When the working memory reaches $T_{\max} = 10$ frames, the first frame and the most recent $T_{\min} - 1 = 4$ frames stay as a high-resolution buffer, and the other $T_{\max} - T_{\min}$ frames are consolidated into the long-term memory: the most-used keys are kept as prototypes ($P = 128$ per consolidation), and their values are aggregated from all candidates (“memory potentiation”) rather than subsampled. With at most 10,000 long-term elements, the authors state that XMem never uses more than 1.4 GB of GPU memory. On the Long-time Video dataset played three times over, STCN fell from J&F 87.3 to 84.6 while XMem went from 89.8 to 90.0.

Cutie (Cheng et al., 2024) argues that pixel-level memory reading is noisy in the presence of distractors and adds object-level reading: a small set of object queries summarizes the target and interacts with the pixel features through a query-based object transformer, with masked attention that separates foreground and background. On MOSE, a dataset with heavy occlusion and crowded scenes, recent methods scored more than 20 points of J&F lower than on DAVIS 2017, by the authors’ count; on the MOSE validation set Cutie-base reached 64.0 against 56.3 for XMem when neither was trained on MOSE.

SAM 2 (Ravi et al., 2025) carries the memory design into a promptable foundation model. A streaming image encoder processes one frame at a time; memory attention conditions each frame on a memory bank that holds a first-in-first-out queue of up to $N$ recent frames (six by default) and up to $M$ prompted frames, plus a list of object pointer vectors; and an extra head predicts whether the object is present in the current frame, since it may be occluded. Trained on its SA-V dataset together with existing video and image data, SAM 2 (Hiera-L) reached J&F 90.7 on DAVIS 2017 validation and 78.4 on SA-V test, where the best earlier method in the authors’ comparison reached 62.8.

Reported results

Each figure is as reported by the method’s authors on the named validation set. Training data and backbones differ, so rows are comparable only roughly: STM’s DAVIS figures use YouTube-VOS training data, Cutie’s row is the variant trained without MOSE, and SAM 2 trains on SA-V and other data.

Method	Year	DAVIS 2016 val	DAVIS 2017 val, J&F	YouTube-VOS val, G
OSVOS	2017	J 79.8	–	–
PML	2018	J&F 77.4	–	–
FEELVOS	2019	J&F 81.7	71.5	–
STM	2019	J 88.7, F 89.9	J 79.2, F 84.3	79.4 (2018)
STCN	2021	J&F 91.6	85.4	83.0 (2018)
XMem	2022	J&F 91.5	86.2	85.7 (2018)
Cutie-base	2024	–	88.8	86.1 (2019)
SAM 2 (Hiera-L)	2025	–	90.7	89.3 (2019)

MaskTrack reports mIoU 74.8 on DAVIS (80.3 with flow and CRF) without naming a split; it describes DAVIS as its 50 videos. The SAM 2 paper’s comparison table gives Cutie-base 87.9 and XMem 86.0 on DAVIS 2017, against 88.8 and 86.2 in their own papers; the XMem value matches the version Cutie’s authors retrained with MOSE data. The table above uses each method’s own paper.

Unsupervised and interactive methods

Unsupervised methods replace the first-frame mask with a prior on what an object is. Before deep learning the prior was mostly motion, as in Papazoglou and Ferrari’s motion boundaries; DAVIS 2016 also evaluated a salient object detector on single frames and on inter-frame motion as reference points, and found motion saliency far stronger than colour saliency. In the multi-object DAVIS 2019 track a method must also decide how many objects there are, and is scored only on the annotated ones it matches (see below).

Interactive methods turn a user’s corrections into masks for the whole video. IPN (Oh et al., 2019) splits the work between an interaction network, which converts scribbles on one frame into a mask, and a propagation network, which carries the mask through the other frames, and trains them over several simulated rounds in which positive scribbles are sampled from missed object regions and negative ones from false positives. The SAM family brings the same loop to clicks and boxes.

Video instance segmentation in outline

Yang, Fan and Xu (2019) defined video instance segmentation: for a fixed set of $K$ categories, find every object, give it a class, a confidence and a mask in each frame where it appears, and keep its identity through occlusion and re-entry, which they distinguish from multi-object tracking by counting still objects as instances. Their YouTube-VIS dataset has 2,883 videos, 40 categories, 4,883 objects and about 131,000 masks, split into 2,238 training, 302 validation and 343 test videos. Evaluation borrows average precision from COCO at ten IoU thresholds from 0.5 to 0.95, with a spatio-temporal IoU between a true track $m^i$ and a predicted track $\tilde{m}^j$, both padded with empty masks to all $T$ frames:

$$\mathrm{IoU}(i, j) = \frac{\sum_{t=1}^{T} \lvert m_t^i \cap \tilde{m}_t^j \rvert}{\sum_{t=1}^{T} \lvert m_t^i \cup \tilde{m}_t^j \rvert}.$$

Their baseline, MaskTrack R-CNN, adds a tracking branch to Mask R-CNN and reached 30.3 AP on validation and 32.3 on test; FEELVOS, used as a mask-propagation baseline, reached 26.9 AP on validation. Video instance segmentation is to semi-supervised VOS what instance segmentation is to segmenting a given object: the method must find the objects itself.

Example: propagation against matching under occlusion

The script builds a 48-frame, $96 \times 128$ video with known masks and known motion. A textured disk cut from scikit-image’s coffee image moves right at 2 pixels per frame over a background cut from chelsea that pans left at 1 pixel per frame. A static bar textured from brick stands in front of the disk’s path, so the disk enters it, is fully hidden for frames 22–27 and re-emerges; the ground truth is the visible part of the disk, and is empty while it is hidden. A second disk cut from the same image, which the first frame never shows, starts entering from the right on a lower row at frame 30 and is visible from frame 31, as a distractor. Five methods start from the true first-frame mask:

• Flow propagation warps the previous predicted mask into the current frame, $M_t(\mathbf{x}) = M_{t-1}(\mathbf{x} + \mathbf{w}(\mathbf{x}))$, with backward flow $\mathbf{w}$ from frame $t$ to frame $t-1$, bilinear interpolation and a threshold of 0.5. The flow is either the true flow, known by construction, or estimated with skimage.registration.optical_flow_tvl1. We chose TV-L1 over scikit-image’s optical_flow_ilk because its total-variation term allows flow discontinuities at object boundaries, where window-based Lucas–Kanade flow blurs them (optical flow estimation describes both).
• The true flow with an occlusion mask additionally sets to background every pixel whose source location in frame $t-1$ is covered by a different layer, as flow methods with an occlusion output or a forward–backward check would. What propagation should do with such pixels is a choice the sources leave open, so both versions are shown.
• First-frame matching labels each pixel by its nearest neighbour, in Gaussian-smoothed RGB, among all first-frame pixels, segmenting each frame independently, as the global matching of PML and FEELVOS does with learned features.
• First + previous keeps a first-frame match only within 8 pixels of the previous predicted mask, a crude version of FEELVOS’s local window and of STM’s first-and-previous memory; if the previous mask is empty, it falls back to global matching.

J and F follow the DAVIS 2017 evaluation code: the same empty-mask conventions, the boundary map of seg2bmap, a tolerance of $\lceil 0.008 \times \text{diagonal} \rceil$ pixels, the exclusion of the first and last frames, and the code’s recall and decay statistics. The code dilates with OpenCV; the script uses SciPy’s binary dilation with the same disk-shaped structuring element, which computes the same operation on binary maps. It runs in a few seconds with NumPy 2.3.5, SciPy 1.16.3 and scikit-image 0.25.2.

import numpy as np
from scipy import ndimage
from scipy.spatial import cKDTree
from skimage import data, color
from skimage.morphology import disk
from skimage.registration import optical_flow_tvl1

# ---------- J and F as in the DAVIS 2017 evaluation code (davis2017/metrics.py) ----------
def db_eval_iou(gt, pred):
    union = np.sum(gt | pred)
    return 1.0 if union == 0 else np.sum(gt & pred) / union      # empty vs empty counts as 1

def seg2bmap(seg):
    """1-pixel boundary map, offset half a pixel towards the origin (Martin's seg2bmap)."""
    e = np.zeros_like(seg); s = np.zeros_like(seg); se = np.zeros_like(seg)
    e[:, :-1] = seg[:, 1:]; s[:-1, :] = seg[1:, :]; se[:-1, :-1] = seg[1:, 1:]
    b = (seg ^ e) | (seg ^ s) | (seg ^ se)
    b[-1, :] = seg[-1, :] ^ e[-1, :]; b[:, -1] = seg[:, -1] ^ s[:, -1]; b[-1, -1] = 0
    return b

def f_measure(pred, gt, bound_th=0.008):
    bound_pix = bound_th if bound_th >= 1 else np.ceil(bound_th * np.linalg.norm(pred.shape))
    fg_b, gt_b = seg2bmap(pred), seg2bmap(gt)
    fg_dil = ndimage.binary_dilation(fg_b, structure=disk(bound_pix))
    gt_dil = ndimage.binary_dilation(gt_b, structure=disk(bound_pix))
    n_fg, n_gt = fg_b.sum(), gt_b.sum()
    if n_fg == 0 and n_gt > 0:   precision, recall = 1, 0
    elif n_fg > 0 and n_gt == 0: precision, recall = 0, 1
    elif n_fg == 0 and n_gt == 0: precision, recall = 1, 1
    else: precision, recall = (fg_b & gt_dil).sum() / n_fg, (gt_b & fg_dil).sum() / n_gt
    return 0.0 if precision + recall == 0 else 2 * precision * recall / (precision + recall)

def db_statistics(v):
    """Mean, recall (fraction of frames > 0.5) and decay (first minus last quarter), as in davis2017/utils.py."""
    ids = (np.round(np.linspace(1, len(v), 5) + 1e-10) - 1).astype(np.uint8)
    bins = [v[ids[i]:ids[i + 1] + 1] for i in range(4)]
    return np.mean(v), np.mean(v > 0.5), np.mean(bins[0]) - np.mean(bins[3])

# ---------- protocol check: the boundary tolerance at 96 x 128 ----------
H, W, T = 96, 128, 48
sq = np.zeros((H, W), bool); sq[36:60, 50:74] = True
print("boundary tolerance (px):", int(np.ceil(0.008 * np.hypot(H, W))))
print("F of a 24x24 square shifted by 0..4 px:",
      [round(float(f_measure(np.roll(sq, d, axis=1), sq)), 3) for d in range(5)])

# ---------- synthetic video with known masks and known flow ----------
bgtex = data.chelsea().astype(float) / 255                 # pans 1 px/frame to the left
cof = data.coffee().astype(float) / 255
obj_tex, dis_tex = cof[180:205, 250:275], cof[180:205, 300:325]
occ = np.repeat((0.3 + 0.4 * data.brick()[:H, :36] / 255.0)[..., None], 3, axis=2)
OCC_X0, OCC_X1, R = 46, 82, 12
yy, xx = np.mgrid[:H, :W]
occluder = (xx >= OCC_X0) & (xx < OCC_X1)

def disk_at(cy, cx):
    return (yy - cy) ** 2 + (xx - cx) ** 2 <= R * R

def paste(frame, tex, cy, cx):
    m = disk_at(cy, cx)
    frame[m] = tex[yy[m] - cy + R, xx[m] - cx + R]
    return m

frames, gts, flows, layers = [], [], [], []
for t in range(T):
    f = bgtex[30:30 + H, 150 + t:150 + t + W].copy()
    flow_x = np.ones((H, W))                               # background: came from x + 1
    obj = paste(f, obj_tex, 44, 14 + 2 * t); flow_x[obj] = -2  # object moves right 2 px/frame
    dis = np.zeros((H, W), bool)
    if t >= 30:                                            # distractor enters from the right at frame 30
        dis = paste(f, dis_tex, 76, 140 - 2 * (t - 30)); flow_x[dis] = 2
    f[occluder] = occ[:, :OCC_X1 - OCC_X0][np.nonzero(occluder)[0], xx[occluder] - OCC_X0]
    flow_x[occluder] = 0                                   # occluder is static and in front
    layer = np.zeros((H, W), int); layer[obj] = 1; layer[dis] = 2; layer[occluder] = 3
    layers.append(layer)
    frames.append(f); gts.append(obj & ~occluder); flows.append(np.stack([np.zeros((H, W)), flow_x]))
gray = [color.rgb2gray(f) for f in frames]

# ---------- methods ----------
def warp(mask, flow):
    """Backward warp: mask_t(x) = mask_{t-1}(x + w(x)), bilinear, then threshold 0.5."""
    coords = np.stack([yy + flow[0], xx + flow[1]])
    return ndimage.map_coordinates(mask.astype(float), coords, order=1, cval=0) > 0.5

def propagate(get_flow, occlusion_mask=False):
    preds = [gts[0]]
    for t in range(1, T):
        m = warp(preds[-1], get_flow(t))
        if occlusion_mask:   # pixels whose correspondence in frame t-1 is covered by another layer -> background
            src_x = np.clip(xx + flows[t][1].astype(int), 0, W - 1)
            m &= layers[t - 1][yy, src_x] == layers[t]
        preds.append(m)
    return preds

tvl1 = {t: optical_flow_tvl1(gray[t], gray[t - 1]) for t in range(1, T)}   # reference t, moving t-1
feats = [np.stack([ndimage.gaussian_filter(f[..., c], 1.0) for c in range(3)], -1).reshape(-1, 3) for f in frames]
tree, ref_lab = cKDTree(feats[0]), gts[0].ravel()
nn_fg = [ref_lab[tree.query(feats[t])[1]].reshape(H, W) for t in range(T)]

def combined():
    preds = [gts[0]]
    for t in range(1, T):
        prev = preds[-1]
        if prev.any():
            preds.append(nn_fg[t] & ndimage.binary_dilation(prev, structure=np.ones((17, 17), bool)))
        else:
            preds.append(nn_fg[t])                         # nothing to follow: fall back to global matching
    return preds

methods = {
    "propagation, oracle flow": propagate(lambda t: flows[t]),
    "oracle flow + occl. mask": propagate(lambda t: flows[t], occlusion_mask=True),
    "propagation, TV-L1 flow": propagate(lambda t: tvl1[t]),
    "first-frame matching": [gts[0]] + nn_fg[1:],
    "first + previous": combined(),
}

epe = np.array([np.hypot(*(tvl1[t] - flows[t])) for t in range(1, T)])
objm = np.array(gts[1:])
print(f"TV-L1 endpoint error vs true flow: all pixels {epe.mean():.2f} px, visible object {epe[objm].mean():.2f} px")

# ---------- evaluation on frames 1..T-2 (first and last excluded, as in the DAVIS code) ----------
ev = range(1, T - 1)
phases = {"before (1-21)": range(1, 22), "hidden (22-27)": range(22, 28), "after (28-46)": range(28, 47)}
print(f"\n{'method':26s} {'J':>6s} {'J-rec':>6s} {'J-dec':>6s} {'F':>6s} {'F-rec':>6s} {'F-dec':>6s} {'J&F':>6s}")
per_phase = {}
for name, preds in methods.items():
    J = np.array([db_eval_iou(gts[t], preds[t]) for t in ev])
    F = np.array([f_measure(preds[t], gts[t]) for t in ev])
    (jm, jr, jd), (fm, fr, fd) = db_statistics(J), db_statistics(F)
    print(f"{name:26s} {jm:6.3f} {jr:6.3f} {jd:6.3f} {fm:6.3f} {fr:6.3f} {fd:6.3f} {(jm + fm) / 2:6.3f}")
    per_phase[name] = (J, preds)

print(f"\n{'mean J by phase':26s}" + "".join(f"{p:>16s}" for p in phases) + f"{'px on distractor':>18s}{'other FP px':>13s}")
for name, (J, preds) in per_phase.items():
    row = "".join(f"{J[[t - 1 for t in r]].mean():16.3f}" for r in phases.values())
    on_dis = np.mean([(preds[t] & disk_at(76, 140 - 2 * (t - 30))).sum() for t in range(31, T - 1)])
    other = np.mean([(preds[t] & ~gts[t] & ~disk_at(76, 140 - 2 * (t - 30))).sum() if t >= 30 else
                     (preds[t] & ~gts[t]).sum() for t in ev])
    print(f"{name:26s}{row}{on_dis:18.1f}{other:13.1f}")

boundary tolerance (px): 2
F of a 24x24 square shifted by 0..4 px: [1.0, 1.0, 1.0, 0.542, 0.521]
TV-L1 endpoint error vs true flow: all pixels 0.09 px, visible object 1.11 px

method                          J  J-rec  J-dec      F  F-rec  F-dec    J&F
propagation, oracle flow    0.174  0.130  0.551  0.228  0.261  0.629  0.201
oracle flow + occl. mask    0.587  0.587  1.000  0.587  0.587  1.000  0.587
propagation, TV-L1 flow     0.198  0.174  0.636  0.207  0.130  0.591  0.202
first-frame matching        0.606  0.674  0.446  0.622  0.826  0.363  0.614
first + previous            0.631  0.674  0.099  0.641  0.696  0.217  0.636

mean J by phase              before (1-21)  hidden (22-27)   after (28-46)  px on distractor  other FP px
propagation, oracle flow             0.381           0.000           0.000               0.0        690.6
oracle flow + occl. mask             1.000           1.000           0.000               0.0          0.0
propagation, TV-L1 flow              0.433           0.000           0.000               0.0        413.2
first-frame matching                 0.862           0.000           0.514             275.7         14.5
first + previous                     0.890           0.000           0.545               8.2          8.7

The first line shows how lenient F is at this size: the tolerance is $\lceil 0.008 \times 160 \rceil = 2$ pixels, so a mask shifted by up to 2 pixels keeps a perfect F; at 3 pixels F roughly halves, in our reading because the left and right edges leave the tolerance band while the top and bottom edges stay on their rows. At DAVIS’s 480p resolution the same rule gives $\lceil 0.008 \times 979.7 \rceil = 8$ pixels. TV-L1’s flow is accurate on the background and off by about a pixel on the moving disk, which we attribute to its regularization smoothing the flow across the object boundary.

Propagation with the true flow fails before any occlusion: in frames 1–21 its mean J is 0.381, with about 690 false-positive pixels per frame. In our reading these are disoccluded background pixels: background uncovered behind the moving disk truly came from a place that the disk covered in the previous frame, so the warp copies the disk’s label onto it, and the error then travels with the background. The occlusion-masked case confirms it: with those pixels set to background, the false positives vanish and J is perfect until the occlusion. That case also reports the hidden frames correctly as empty, and then never recovers, because a re-emerging pixel’s source lies behind the bar, where the mask is 0. It is the extreme of the decay that DAVIS measures: J decay 1.000. TV-L1 propagation behaves like the uncorrected oracle, with fewer false-positive pixels (413 against 691 per frame), and is lost after the occlusion too.

Matching to the first frame does not depend on the previous frame, so it finds the disk again when it emerges (J 0.514 after the occlusion), but it labels about 276 pixels of the distractor per frame as the object, and its few scattered false matches score J = 0 in every hidden frame, since any foreground pixel in a frame whose ground truth is empty gives J = 0 and F = 0. Restricting matches to a window around the previous mask cuts the distractor pixels to about 8 per frame and gives the best J&F, 0.636. It still scores 0 on the hidden frames: with nothing to follow, it falls back to global matching and picks up the same stray matches. The recall and decay columns summarize the same behaviour: the propagation methods have large decays, and matching has a sizeable one too (0.446), which in our reading comes from the distractor and the partly hidden object in the last quarter rather than from accumulated error. The ranking mirrors STM’s memory ablation on real data, where the first and previous frames together beat either alone, but on one synthetic clip it shows mechanisms, not the size of their effects.

Three changes were made after the first run. The first run had a single true-flow propagation case; after its trail of false positives, the occlusion-masked case was added as the second answer to the open choice above, leaving the other cases unchanged. After the second run, the last table’s phase boundaries were found to be set one frame too late (a count of ground-truth pixels showed that frame 28 still has one visible pixel) and were corrected to the geometry; this changed only the phase averages. The first run also printed the tolerance check with NumPy’s type names, which was changed to plain numbers.

Datasets and Benchmarks

Before DAVIS, the most widely adopted set was SegTrack (Tsai et al., 2010), six low-resolution videos that DAVIS 2016’s authors considered no longer representative; FBMS, with masks on a sparse subset of frames, and YouTube-Objects were used as well.

• DAVIS 2016 (Perazzi et al., 2016): 50 sequences, 3,455 frames, all captured at 24 fps in Full HD and annotated in every frame with a single object or two spatially connected objects per sequence, spanning humans, animals, vehicles and objects. Each sequence is tagged with challenge attributes. The authors evaluated at 480p, because several methods could not run at full resolution; the DAVIS 2017 download page likewise states that the official metrics are computed on the 480p images and annotations. The first frame, given as input, and the last frame, which some methods did not process, are excluded from evaluation. Later papers use its 20-video validation set.
• DAVIS 2017 (DAVIS challenge pages): multiple objects per sequence. Train and validation together have 90 sequences, the 50 of DAVIS 2016 re-annotated with multiple objects plus 40 new ones, split 60 to 30; Test-Dev 2017 and Test-Challenge 2017 add 30 new sequences each with ground truth withheld. STM’s authors count 59 objects in the 30 validation videos. The 2018 challenge added an interactive track with simulated scribbles, and the 2019 challenge an unsupervised track, which re-annotated the 90 train and validation sequences for the unsupervised definition and added 30 new Test-Dev and 30 new Test-Challenge sequences. As of November 2026 the DAVIS site describes itself as in maintenance mode, with no further challenges or leaderboard updates, and says its evaluation servers continue to run on Codalab.
• YouTube-VOS (Xu et al., 2018): the first large-scale VOS dataset. The ECCV paper describes 3,252 clips of 3 to 6 seconds over 78 categories with 133,886 object annotations, split into 2,796 training, 134 validation and 322 test clips, with 10 categories held out as unseen. Annotators labelled up to five objects per clip every fifth frame of 30 fps video, an annotation rate of 6 fps, on the reasoning that five consecutive frames are strongly correlated. The released version differs from the paper: the dataset page lists, for the 2018 challenge, 3,471 training videos (65 categories, 5,945 objects), 474 validation videos (65 seen and 26 unseen categories) and 508 test videos (65 seen and 29 unseen); the 2019 version keeps the 3,471 training videos with more objects (6,459) and has 507 validation and 541 test videos; the 2022 test set has 746 videos, including 116 long videos of 20 to 120 seconds annotated at 1 fps. Some objects first appear after the first frame. As of November 2026 the site states that, after CodaLab services were discontinued, it released the test-set ground truth of the 2019 VOS benchmark and of the 2019, 2021 and 2022 VIS benchmarks.
• Newer sets target the failure cases of the older ones: MOSE (heavy occlusion and crowded scenes), BURST and LVOS (over a thousand frames per video, as Cutie’s authors describe them; BURST is scored with HOTA), the Long-time Video dataset (three videos with more than 7,000 frames in total, as XMem’s authors describe it) and SAM 2’s SA-V, whose validation and test sets cover open-world objects.

DAVIS remains small: STM’s authors read the jump in mean J from 69.2 to 79.2 on DAVIS 2017 validation when adding YouTube-VOS training data as a sign that DAVIS alone is too small to train a network that generalizes, and in a cross-dataset test their DAVIS-trained model scored 56.3 overall on YouTube-VOS, while the YouTube-VOS-trained model reached a mean J of 78.6 on DAVIS 2017.

Evaluation Metrics

DAVIS 2016 introduced three per-frame measures, and later benchmarks kept the first two.

Region similarity $\mathcal{J}$ is the Jaccard index, the intersection over union of the predicted mask $M$ and the ground-truth mask $G$:

$$\mathcal{J} = \frac{\lvert M \cap G \rvert}{\lvert M \cup G \rvert}.$$

The evaluation code sets $\mathcal{J} = 1$ when both masks are empty, so a frame in which the object is absent scores 1 for an empty prediction and 0 for any foreground pixel.

Contour accuracy $\mathcal{F}$ treats the mask as a set of contours and computes boundary precision and recall $P_c$ and $R_c$ between the contour points of $M$ and $G$, combined as an F-measure, $\mathcal{F} = 2 P_c R_c / (P_c + R_c)$. The paper describes a bipartite matching of contour points, approximated with morphological operators for speed. In the code, $B(M)$ and $B(G)$ are 1-pixel boundary maps; a boundary pixel counts as matched if it lies within a disk of radius $\lceil 0.008 \sqrt{H^2 + W^2} \rceil$ pixels of the other boundary:

$$P_c = \frac{\lvert B(M) \cap \mathrm{dil}(B(G)) \rvert}{\lvert B(M) \rvert}, \qquad R_c = \frac{\lvert B(G) \cap \mathrm{dil}(B(M)) \rvert}{\lvert B(G) \rvert}.$$

When neither mask has a boundary, $\mathcal{F} = 1$; when only the prediction has one, precision is 0 and $\mathcal{F} = 0$; when only the ground truth has one, recall is 0. The paper shows cases where the two measures disagree, a missed head and foot that costs many pixels but little boundary, and a small area with a very inaccurate outline, and found the two only partly correlated over the dataset.

Temporal stability $\mathcal{T}$ matched shape-context descriptors along the contours of consecutive masks by dynamic time warping and took the mean matching cost, so that jitter is penalized while smooth motion is not. Because occlusions and strong deformations look like instability to it, the authors computed it on a subset of sequences. The DAVIS 2017 code does not implement it and rejects a request for it.

Statistics. For a measure $C$ the 2016 paper reports the mean over sequences; the recall, the fraction of sequences scoring above $\tau = 0.5$; and the decay, the difference between the scores of the first and last quarters of each sequence, averaged over sequences. The DAVIS 2017 code computes these per object, with recall as the fraction of frames above 0.5, and averages over objects. Its headline score is

$$\mathcal{J}\&\mathcal{F} = \tfrac{1}{2}\left(\overline{\mathcal{J}} + \overline{\mathcal{F}}\right),$$

with each mean taken over all objects of all sequences, so a sequence with three objects weighs three times as much as a sequence with one. The semi-supervised code does not use the void pixels marked in the annotations.

Protocol by setting.

• YouTube-VOS computes J and F separately on seen and unseen categories, averaged over the corresponding objects, and ranks by the mean of the four scores, $\mathcal{G} = (\mathcal{J}_s + \mathcal{F}_s + \mathcal{J}_u + \mathcal{F}_u)/4$, called the overall score.
• DAVIS unsupervised (2019): each annotated object is matched to one of up to 20 predicted object tracks by bipartite matching on the sequence-level mean of J and F, with the Hungarian algorithm in the code; extra predicted objects are not penalized, and the score is the mean over matched objects. DAVIS 2016 had already used a bipartite matching on J for unsupervised methods that output several segments.
• DAVIS interactive (2018, 2019): a server simulates a user who sends scribbles, receives the masks and replies with corrective scribbles, for up to 8 interactions with at most 30 seconds per object per interaction. Methods are ranked by the area under the curve of quality against time, and by the quality interpolated at 60 seconds; quality was J in 2018 and J&F from 2019.
• Video instance segmentation: AP and AR with the spatio-temporal IoU given above.

Applications

DAVIS 2016’s authors name rotoscoping for video editing and post-production, action recognition, object tracking and video summarization as uses of pixel-accurate object masks, and chose the boundary and temporal measures with editing in mind, where contour accuracy and stability take the most manual work. Cutie’s authors add robotics and reducing the cost of data annotation, and combining VOS with SAM-style models for segmenting anything in video. Object masks over time are also an input to other video understanding tasks, such as action recognition.

Open Problems

• Occlusion, distractors and crowds. Cutie’s authors report that recent methods such as XMem score more than 20 points of J&F lower on MOSE than on DAVIS 2017, and on SA-V test SAM 2 leads the best earlier method in its comparison by more than 15 points.
• Long videos. Memory must stay bounded without forgetting the target; XMem and the long-term datasets address this, but most benchmarks still use clips of a few seconds.
• Recovery and absence. Reporting an object as absent and finding it again after a long gap remains hard; the empty-mask convention makes every false positive in such frames cost a full point of J and F.
• Evaluation. DAVIS is small and its leaderboards are frozen; YouTube-VOS annotates only every fifth frame; and the temporal stability measure that DAVIS proposed for editing applications is no longer computed.
• Guidance. A first-frame mask is expensive to draw. Promptable models accept clicks and boxes on any frame, but how to compare methods that use different amounts of user input is not settled.