DAVIS

DAVIS (Densely Annotated VIdeo Segmentation) is a benchmark for video object segmentation, first released in 2016. Federico Perazzi, Jordi Pont-Tuset and colleagues at ETH Zurich and Disney Research released its first version with their CVPR 2016 paper: 50 short Full HD clips in which every frame carries a pixel-accurate mask of the foreground object. From 2017 to 2020 organizers including Pont-Tuset and Perazzi ran a yearly DAVIS Challenge at CVPR on a multi-object extension, DAVIS 2017, adding an interactive track in 2018 and an unsupervised track, with its own annotations and test sets, in 2019. The region and boundary measures $\mathcal{J}$ and $\mathcal{F}$ it introduced are also used by later video segmentation benchmarks. As of November 2026 the site describes the standalone DAVIS initiative as being in maintenance mode: no more challenges are planned and the leaderboards are no longer updated, but the site says it will continue running the evaluation servers on Codalab.

This article covers the dataset itself: its releases, files, annotation format, licence, challenge rules and the official evaluation code, with the 2019 unsupervised protocol reimplemented in detail. The measures $\mathcal{J}$ and $\mathcal{F}$, their statistics and the methods evaluated on DAVIS belong to the video object segmentation article, which this one links rather than repeats.

Purpose

The 2016 paper argues that video object segmentation lacked a dataset of its own. The most widely adopted set, SegTrack, had six low-resolution videos; FBMS, built for motion segmentation, annotated only a sparse subset of frames; and the Berkeley video segmentation set had fragmented ground truth, with most objects covered by several disjoint segments. The authors wanted enough high-resolution data to delay saturation and overfitting, dense per-frame ground truth, and a standard evaluation with several complementary measures. They also chose clips with one object, or two spatially connected objects, so that methods segmenting one object could be compared fairly with methods segmenting several, and so that the target is unambiguous for fully automatic methods (§3).

The 2017 challenge moved to several distinct objects per sequence, the setting for which every method from FEELVOS (2019) on in the results table of the video object segmentation article reports a score, and the 2019 challenge redefined the unsupervised task for that multi-object setting.

Dataset Size

• DAVIS 2016: 50 sequences with 3,455 annotated frames, all captured at 24 fps in Full HD (1080p), each about 2–4 seconds long (paper, §3).
• DAVIS 2017 (semi-supervised): 90 Train and Val sequences, the 60 training and 30 validation sequences, plus Test-Dev 2017 and Test-Challenge 2017 with 30 sequences each (2019 challenge pages). The schema.org metadata on the home page describes DAVIS 2017 as 150 video sequences, each with several objects and their masks in every frame.
• DAVIS 2017 unsupervised (2019): the same 60 training and 30 validation videos, “reannotated taking into account the unsupervised definition”, plus 30 new Test-Dev 2019 and 30 new Test-Challenge 2019 sequences (2019 unsupervised page).
• Scribbles (2018): three human-drawn scribbles for each object of the semi-supervised Train and Val set (DAVIS 2017 download page).

Classes

DAVIS is not a category benchmark. The 2016 clips span four evenly distributed classes, humans, animals, vehicles and objects, and several actions (§3). For DAVIS 2017 the authors later released an object-categories package: semantic masks for every publicly available frame of the semi-supervised sets, a JSON file with the category of each object and another with each category’s id and super-category. The 2019 semi-supervised rules forbid using these categories as additional information in a challenge entry.

Annotation Types

• Masks. DAVIS 2016 provides, for every frame, a manually created binary mask (§3). DAVIS 2017 stores one indexed PNG per frame in which each pixel holds the id of the object it belongs to, 0 for background; the same format is required of submissions, with each object keeping its id for the whole sequence (2019 unsupervised page).
• Void pixels. The evaluation code treats label 255 as void (DAVIS.VOID_LABEL in davis.py): get_all_masks moves those pixels into a separate void mask and sets them to background before splitting the objects.
• Two annotation sets for the same videos. For DAVIS 2017 Train and Val, the semi-supervised annotations (folder Annotations) and the unsupervised ones (Annotations_unsupervised) differ; the download page says the video sequences are the same “but the annotations are different”. The code chooses the folder by task.
• Resolutions. Images and annotations come at 480p and at full resolution (4k, 1080p and others). The official metrics are computed at 480p; the 2016 authors down-sampled to 480p because several methods could not run at full resolution (§6).
• Attributes. Each 2016 sequence is tagged with challenge attributes. Table 1 of the paper lists 15, eight taken from an earlier tracking benchmark (background clutter, deformation, motion blur, fast motion, low resolution, occlusion, out-of-view, scale variation) and seven new ones (appearance change, edge ambiguity, camera shake, heterogeneous object, interacting objects, dynamic background, shape complexity), several with numeric thresholds, such as fast motion when the centroid moves more than 20 pixels per frame on average. The paper disagrees with itself on the count: its dependency analysis in §6.2 writes the attribute graph’s vertex set as $\{1, \dots, 16\}$ and counts $16^2 = 256$ possible edges, against the 15 rows of Table 1 and the 15 bars of Figure 2. The supplemental material lists the attributes of each sequence.
• Community additions listed on the home page include referring expressions for DAVIS 2016 and 2017 (Khoreva, Rohrbach & Schiele, ACCV 2018), human eye fixations of twenty participants for all frames of DAVIS 2016 Train and Val, and object shadows in 29 videos of DAVIS 2016.

Splits

The 2016 paper subdivided DAVIS into a training and a test set “to provide guidelines for future works” but did not use the partition in its own evaluation, because most evaluated methods were not trained (§3).

For DAVIS 2017 the ground truth of Train and Val is public. Test-Dev 2017 and Test-Challenge 2017 come with images and first-frame annotations only, since semi-supervised methods receive the first-frame masks; the unsupervised Test-Dev 2019 and Test-Challenge 2019 come with images only. Test-Dev accepted an unlimited number of submissions; Test-Challenge was open for about two weeks before each workshop and limited to five submissions in total (2019 and 2020 challenge pages). The 2019 pages invite training or pre-training on datasets other than DAVIS, such as YouTube-VOS, MS COCO or Pascal, and allow DAVIS’s full-resolution images and annotations.

Tasks

The site separates the settings by human input at test time, not during training: “semi-supervised”, which it says is better called human-guided, gives the mask of each object in the first frame, and “unsupervised”, better called human non-guided, gives nothing.

• Semi-supervised (2016 single-object, 2017 onward multi-object): segment each object given its first-frame mask.
• Interactive (teaser track in 2018, full track in 2019 and 2020): a server simulates a user who draws scribbles; the method returns masks for all frames and receives corrective scribbles, for up to 8 interactions with at most 30 seconds per object per interaction. Methods are ranked by the area under the curve of $\mathcal{J}\&\mathcal{F}$ against time and by $\mathcal{J}\&\mathcal{F}$ interpolated at 60 seconds; the 2019 page says that edition used $\mathcal{J}\&\mathcal{F}$ “instead of only Jaccard”. Interactive segmentation covers the single-image counterpart and the DAVIS frames used there.
• Unsupervised (2019, 2020): methods return “a set of object candidates with no overlapping pixels that span through the whole video sequence”, which should contain at least the objects that capture human attention, those most likely to be followed by gaze (2019 unsupervised page).

Derived benchmarks reuse the videos for other tasks: TAP-Vid-DAVIS for point tracking, DAVIS-Moving for motion segmentation, and frame-level sets for interactive segmentation.

Evaluation

The measures are those of the 2016 paper: region similarity $\mathcal{J}$, the Jaccard index or intersection over union of predicted and true masks; contour accuracy $\mathcal{F}$, an F-measure of boundary precision and recall; and a temporal stability measure $\mathcal{T}$, which the DAVIS 2017 code does not implement. The video object segmentation article gives their formulas, the empty-mask conventions, the boundary tolerance of $\lceil 0.008 \times \text{diagonal} \rceil$ pixels, and the mean, recall and decay statistics.

How the official code reads a submission

The evaluation package for DAVIS 2017 is davisvideochallenge/davis2017-evaluation; the description here follows the code at commit ac7c43f (26 February 2023), read line by line. Its README says the same tool scores the Codalab submissions of the semi-supervised and unsupervised challenges.

• Results folder. A submission is one folder per sequence holding one indexed PNG per frame, named like the ground-truth frames. Results.read_masks reads the frames being scored (all of them for the unsupervised task, all but the first and last for the semi-supervised one) and sets the number of predicted objects to the largest id used in any frame; ids that do not occur become empty masks. A missing frame stops the program with a message.
• Ground truth. get_all_masks separates the void pixels (label 255) and counts the annotated objects from the first frame only (np.max(masks[0, ...])), so an object id that first appears later is not scored. This is our reading of the code; we did not check whether any DAVIS sequence has an object that first appears after frame 0.
• Frames. For the semi-supervised task, evaluate drops the first and last frames, as the 2016 paper did: the first is the given input, and some methods did not process the last (§6). For the unsupervised task it keeps every frame.
• Void pixels. The semi-supervised path passes None for the void mask, so void pixels count as background; the unsupervised path passes the void mask, which removes those pixels from $\mathcal{J}$ and from the masks before their boundaries are traced for $\mathcal{F}$.
• Object count. If a semi-supervised submission uses an id higher than the number of annotated objects, or an unsupervised one more than 20 proposals, the program prints a message and exits (sys.exit()). Because the count is the largest id used, a submission with four proposals numbered 1, 2, 3 and 21 is rejected.
• Scoring. For each object, db_statistics gives the mean, recall and decay over frames. The headline $\mathcal{J}\&\mathcal{F}$ in evaluation_method.py is the mean of the objects’ $\mathcal{J}$ means and $\mathcal{F}$ means, averaged over all objects of all sequences, and written to global_results-<set>.csv, with per-object means in per-sequence_results-<set>.csv.
• Codalab script. The repository’s evaluation_codalab.py hard-codes the semi-supervised task and the Test-Dev set, although the README points to it as the script running on the Codalab servers; the repository’s file list at this commit has no version set to the unsupervised task.

The 2019 unsupervised protocol

The 2019 unsupervised page states the rule in words: each annotated object “is matched with one of the N video object proposals” by a bipartite graph matching that maximizes $\mathcal{J}\&\mathcal{F}$; methods are not penalized for detecting more objects than were annotated; and the result is the mean over all matched objects. _evaluate_unsupervised implements it as follows. For $N$ proposals and $K$ annotated objects (with empty proposals added if $N < K$), it computes per-frame $\mathcal{J}$ and $\mathcal{F}$ for every proposal–object pair, averages each over the sequence, and forms

$$M_{nk} = \tfrac{1}{2}\left(\overline{\mathcal{J}}_{nk} + \overline{\mathcal{F}}_{nk}\right),$$

then calls SciPy’s linear_sum_assignment(-M), which solves the assignment problem of the Hungarian algorithm (with a shortest-augmenting-path method, as that article describes), to choose $K$ pairs with the largest total. Only the matched pairs’ per-frame scores go on to the statistics. The 2016 paper had already matched methods that output several segments per frame to the target by bipartite matching, but on $\mathcal{J}$ alone (§5); the 2019 page and the code match on the mean of $\mathcal{J}$ and $\mathcal{F}$.

One detail of the code’s output is easy to miss. linear_sum_assignment returns the matched pairs in proposal order, and evaluate then names them <sequence>_1, <sequence>_2, … by position. In our reading, the per-object table therefore lists the scores in the order of the matched proposals, not of the annotated objects, whenever the two orders differ; the means, and so the headline score, are unaffected.

Example: the unsupervised protocol on a synthetic clip

The script ports the functions the protocol needs from the evaluation package at ac7c43f and runs them on in-memory label maps. The official code itself cannot run here: it imports OpenCV and reads a DAVIS folder, and no dataset file was downloaded. The port replaces cv2.dilate with SciPy’s binary_dilation using the same scikit-image disk, and returns a message where the code calls sys.exit(). For the semi-supervised path the official code takes the largest id over frames 1 to $T-2$ only, while the port takes it over all frames; by construction every id in the submissions below occurs in frames 1–10, so the counts agree. Two checks test the port: the repository’s own unit test test_void_masks, in which a prediction that differs from the ground truth only inside the void region must score $\mathcal{J} = \mathcal{F} = 1$, and a comparison of the disk dilation against a Euclidean distance threshold on 20 random boundary maps.

The clip has 12 frames of $96 \times 128$ pixels and three objects present from the first frame: a disk A moving right, a square B moving left and a bar C moving right, always more than 9 pixels apart, so a proposal that covers one object scores $\mathcal{J} = \mathcal{F} = 0$ against another (the boundary tolerance is 2 pixels). A vertical band of void pixels, label 255 over object and background alike, lies across A’s path from frame 7. The submissions are built from the ground truth:

• S0 the exact objects, with A’s full shape including the pixels under the void band; S1 the same masks with ids permuted;
• S2 A split at its centre column into two proposals; S3 B and C merged under one id;
• S4 S0 plus 17 spurious $3 \times 3$ blobs in the background (20 proposals); S4b the same blobs given A’s id;
• S5 S0 plus 18 blobs (21 proposals); S5b S0 plus one blob with id 21;
• S6 an identity switch: proposal 1 follows A in frames 0–6 and B from frame 7, proposal 2 is empty until frame 7 and follows A from frame 8, and proposal 3 is C;
• S7 S0 with every proposal empty in frame 0.

Each submission is scored five ways: the official unsupervised path (U); the same with greedy matching, which takes the largest remaining entry of $M$ until every object is matched; U without the void mask; U on frames 1–10 only; and the code’s semi-supervised path on the same masks (object $k$ against id $k$, no void, frames 1–10). Greedy matching and the two U variants are our comparisons, not part of the official code. Before running anything we derived the expected results by hand: id permutation and spurious proposals cost nothing; S6’s matrix gives exactly $21/36$ under Hungarian matching and $19/36$ under greedy matching; S7 scores $11/12$ with all frames and 1 without frame 0; S0 scores 1 only when void pixels are ignored. The same derivation shows that a single split or a single merge of disjoint objects cannot make greedy and Hungarian matching differ: a split puts all of an object’s non-zero entries in one column, a merge puts two objects’ entries in one row, and both rules take the largest entry of that column or row. They can differ when one proposal’s assignment changes what is left for another, as in S6, where proposal 1 overlaps A and B in different frames and proposal 2 overlaps A. The example therefore illustrates rules that follow from the code, rather than testing an open question. It runs in about 2 seconds with NumPy 2.3.5, SciPy 1.16.3 and scikit-image 0.25.2.

import os
for v in ("OPENBLAS_NUM_THREADS", "OMP_NUM_THREADS", "MKL_NUM_THREADS"):
    os.environ[v] = "1"
import itertools
import numpy as np
from scipy import ndimage
from scipy.optimize import linear_sum_assignment
from skimage.morphology import disk

# ---- Port of davis2017-evaluation at ac7c43f (davis2017/metrics.py, utils.py, davis.py, results.py,
# evaluation.py). cv2.dilate is replaced by SciPy's binary_dilation with the same disk; sys.exit() is
# replaced by returning a message. The repository's LICENSE file at ac7c43f, in full:
#
# BSD 3-Clause License
#
# Copyright (c) 2020, DAVIS: Densely Annotated VIdeo Segmentation
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
#
# 1. Redistributions of source code must retain the above copyright notice, this
#    list of conditions and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright notice,
#    this list of conditions and the following disclaimer in the documentation
#    and/or other materials provided with the distribution.
#
# 3. Neither the name of the copyright holder nor the names of its
#    contributors may be used to endorse or promote products derived from
#    this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
# ----
def db_eval_iou(annotation, segmentation, void_pixels=None):
    annotation, segmentation = annotation.astype(bool), segmentation.astype(bool)
    void_pixels = np.zeros_like(segmentation) if void_pixels is None else void_pixels.astype(bool)
    inters = np.sum(annotation & segmentation & ~void_pixels, axis=(-2, -1))
    union = np.sum((annotation | segmentation) & ~void_pixels, axis=(-2, -1))
    j = inters / union
    j[np.isclose(union, 0)] = 1                        # nothing outside the void: J = 1
    return j

def seg2bmap(seg):
    seg = seg.astype(bool)
    e, s, se = np.zeros_like(seg), np.zeros_like(seg), np.zeros_like(seg)
    e[:, :-1] = seg[:, 1:]; s[:-1, :] = seg[1:, :]; se[:-1, :-1] = seg[1:, 1:]
    b = seg ^ e | seg ^ s | seg ^ se
    b[-1, :] = seg[-1, :] ^ e[-1, :]; b[:, -1] = seg[:, -1] ^ s[:, -1]; b[-1, -1] = 0
    return b

def f_measure(fg, gt, void=None, bound_th=0.008):
    void = np.zeros_like(fg, bool) if void is None else void.astype(bool)
    bound_pix = bound_th if bound_th >= 1 else np.ceil(bound_th * np.linalg.norm(fg.shape))
    fg_b, gt_b = seg2bmap(fg & ~void), seg2bmap(gt & ~void)   # void pixels removed before the boundary
    se = disk(bound_pix).astype(bool)
    fg_dil, gt_dil = ndimage.binary_dilation(fg_b, se), ndimage.binary_dilation(gt_b, se)
    n_fg, n_gt = fg_b.sum(), gt_b.sum()
    if n_fg == 0 and n_gt > 0: p, r = 1, 0
    elif n_fg > 0 and n_gt == 0: p, r = 0, 1
    elif n_fg == 0 and n_gt == 0: p, r = 1, 1
    else: p, r = (fg_b & gt_dil).sum() / n_fg, (gt_b & fg_dil).sum() / n_gt
    return 0 if p + r == 0 else 2 * p * r / (p + r)

def db_eval_boundary(gt, seg, void=None):
    return np.array([f_measure(seg[t], gt[t], None if void is None else void[t]) for t in range(len(gt))])

def gt_masks(labels):                                  # DAVIS.get_all_masks(seq, True)
    m = labels.astype(int); void = m == 255; m[void] = 0
    n = m[0].max()                                     # objects are counted in the first frame
    return np.stack([m == k for k in range(1, n + 1)]), void

def res_masks(labels):                                 # Results.read_masks
    n = labels.max()                                   # proposals = largest id in any frame
    return np.stack([labels == k for k in range(1, n + 1)])

def greedy(M):                                         # comparison only, not in the official code
    M, rows, cols = M.astype(float).copy(), [], []
    for _ in range(min(M.shape)):
        r, c = np.unravel_index(np.argmax(M), M.shape)
        rows.append(r); cols.append(c); M[r, :] = M[:, c] = -np.inf
    o = np.argsort(rows)
    return np.array(rows)[o], np.array(cols)[o]

def evaluate(gt_labels, res_labels, task="unsupervised", matching="hungarian", use_void=True, trim=False):
    gt, void = gt_masks(gt_labels)
    res = res_masks(res_labels)
    fr = np.arange(1, gt.shape[1] - 1) if trim else np.arange(gt.shape[1])
    if task == "unsupervised" and len(res) > 20:
        return None, f"exit: {len(res)} proposals > 20"
    if task == "semi-supervised" and len(res) > len(gt):
        return None, f"exit: {len(res)} ids > {len(gt)} objects"
    if len(res) < len(gt):                             # missing proposals are padded with empty masks
        res = np.concatenate([res, np.zeros((len(gt) - len(res), *res.shape[1:]), bool)])
    v = void[fr] if use_void else None
    if task == "semi-supervised":                      # object k is scored against id k
        J = np.stack([db_eval_iou(g[fr], r[fr], v) for g, r in zip(gt, res)])
        F = np.stack([db_eval_boundary(g[fr], r[fr], v) for g, r in zip(gt, res)])
        info = None
    else:                                              # every proposal against every object
        Jp = np.array([[db_eval_iou(g[fr], r[fr], v) for g in gt] for r in res])
        Fp = np.array([[db_eval_boundary(g[fr], r[fr], v) for g in gt] for r in res])
        M = (Jp.mean(axis=2) + Fp.mean(axis=2)) / 2    # sequence-mean (J + F) / 2 per pair
        rows, cols = linear_sum_assignment(-M) if matching == "hungarian" else greedy(M)
        J, F, info = Jp[rows, cols], Fp[rows, cols], (M, rows, cols)
    JM, FM = np.nanmean(J, axis=1), np.nanmean(F, axis=1)   # per-object means (db_statistics' M)
    return (JM.mean() + FM.mean()) / 2, (info, JM)

# ---- synthetic clip: three objects, a void band (label 255) ----
H, W, T = 96, 128, 12
yy, xx = np.mgrid[:H, :W]
A = [(yy - 30) ** 2 + (xx - 20 - 4 * t) ** 2 <= 100 for t in range(T)]               # disk, moves right
B = [(yy >= 60) & (yy < 78) & (xx >= 100 - 3 * t) & (xx < 118 - 3 * t) for t in range(T)]  # square, left
C = [(yy >= 84) & (yy < 94) & (xx >= 10 + 2 * t) & (xx < 40 + 2 * t) for t in range(T)]    # bar, right
band = (yy >= 10) & (yy < 50) & (xx >= 56) & (xx < 64)
blobs = [(yy >= r0) & (yy < r0 + 3) & (xx >= c0) & (xx < c0 + 3) for r0 in (2, 8) for c0 in range(70, 126, 6)]
E = np.zeros((H, W), bool)

def labelmap(*tracks):                                 # (id, list of T masks): indexed PNG frames
    L = np.zeros((T, H, W), np.uint8)
    for i, ms in tracks:
        for t in range(T):
            assert not L[t][ms[t]].any(), "proposals overlap"
            L[t][ms[t]] = i
    return L

gt_lab = labelmap((1, A), (2, B), (3, C)); gt_lab[:, band] = 255
still = lambda m: [m] * T
spur = lambda n: [(4 + i, still(blobs[i])) for i in range(n)]
subs = {
    "S0 exact": labelmap((1, A), (2, B), (3, C)),
    "S1 permuted ids": labelmap((3, A), (1, B), (2, C)),
    "S2 A split": labelmap((1, [a & (xx < 20 + 4 * t) for t, a in enumerate(A)]), (2, B), (3, C),
                           (4, [a & (xx >= 20 + 4 * t) for t, a in enumerate(A)])),
    "S3 B+C merged": labelmap((1, A), (2, [b | c for b, c in zip(B, C)])),
    "S4 +17 spurious": labelmap((1, A), (2, B), (3, C), *spur(17)),
    "S4b same, as A": labelmap((1, [a | np.any(blobs[:17], axis=0) for a in A]), (2, B), (3, C)),
    "S5 +18 spurious": labelmap((1, A), (2, B), (3, C), *spur(18)),
    "S5b one id 21": labelmap((1, A), (2, B), (3, C), (21, still(blobs[0]))),
    "S6 switch": labelmap((1, A[:7] + B[7:]), (2, [E] * 8 + A[8:]), (3, C)),
    "S7 frame 0 empty": labelmap((1, [E] + A[1:]), (2, [E] + B[1:]), (3, [E] + C[1:])),
}

# ---- checks on the data and the port ----
tol = int(np.ceil(0.008 * np.hypot(H, W)))
gap = min(ndimage.distance_transform_edt(~P[t])[Q[t]].min() for P, Q in itertools.permutations((A, B, C), 2) for t in range(T))
print(f"boundary tolerance {tol} px; smallest gap between objects {gap:.3f} px; "
      f"objects in frame 0: {sorted(set(np.unique(gt_lab[0]).tolist()) - {0, 255})}")
print("frames in which A meets the void band:", [t for t in range(T) if (A[t] & band).any()])
spots = np.any(blobs[:18], axis=0)
print(f"spurious blobs: smallest gap to an object {min(ndimage.distance_transform_edt(~spots)[P[t]].min() for P in (A, B, C) for t in range(T)):.3f} px, "
      f"touch the band: {bool((spots & band).any())}, pixels in 17 blobs: {int(np.any(blobs[:17], axis=0).sum())}")
g, m, vd = (np.zeros((2, 200, 200), bool) for _ in range(3))     # the repository's test_void_masks
g[:, 100:150, 100:150] = m[:, 50:150, 100:150] = vd[:, 50:100, 100:150] = True
print("official test_void_masks: J =", db_eval_iou(g, m, vd).mean(), " F =", db_eval_boundary(g, m, vd).mean())
rng = np.random.default_rng(0)
maps = [seg2bmap(ndimage.binary_opening(rng.random((H, W)) < 0.5)) for _ in range(20)]
print("disk dilation == (distance to boundary <= 2) on 20 random maps:",
      all(np.array_equal(ndimage.binary_dilation(b, disk(tol).astype(bool)), ndimage.distance_transform_edt(~b) <= tol) for b in maps))

# ---- the 2019 unsupervised protocol against its variants ----
cols = {"U (official)": {}, "greedy": dict(matching="greedy"), "U, no void": dict(use_void=False),
        "U, frames 1-10": dict(trim=True), "semi path": dict(task="semi-supervised", use_void=False, trim=True)}
print(f"\n{'submission':18s}" + "".join(f"{c:>16s}" for c in cols))
out = {}
for name, L in subs.items():
    out[name] = {c: evaluate(gt_lab, L, **kw) for c, kw in cols.items()}
    print(f"{name:18s}" + "".join(f"{v:16.6f}" if v is not None else f"{'-':>16s}" for v, _ in out[name].values()))
    for c, (v, msg) in out[name].items():
        if v is None and c in ("U (official)", "semi path"):
            print(f"{'':18s}{c}: {msg}")

for name in ("S1 permuted ids", "S2 A split", "S3 B+C merged", "S6 switch"):
    (M, r, c), JM = out[name]["U (official)"][1]
    best = max(M[list(p), [0, 1, 2]].sum() for p in itertools.permutations(range(len(M)), 3))
    print(f"\n{name}: (J+F)/2, rows = proposal ids 1..{len(M)}, columns = objects A B C")
    print(np.array2string(M, precision=6, floatmode="fixed"))
    print(f"Hungarian: proposals {(r + 1).tolist()} -> objects {[ 'ABC'[k] for k in c]}; "
          f"sum minus brute-force optimum {M[r, c].sum() - best:.1e}")
    print("per-object J as the code labels it: " +
          ", ".join(f"obj{k + 1} {JM[k]:.6f} (is {'ABC'[c[k]]})" for k in range(3)))

u, gr = out["S6 switch"]["U (official)"][0], out["S6 switch"]["greedy"][0]
print(f"\nS6: U - 21/36 = {u - 21 / 36:.1e}, greedy - 19/36 = {gr - 19 / 36:.1e}")
print(f"S7: U - 11/12 = {out['S7 frame 0 empty']['U (official)'][0] - 11 / 12:.1e}, "
      f"frames 1-10 - 1 = {out['S7 frame 0 empty']['U, frames 1-10'][0] - 1:.1e}")
others = [n for n in subs if n != "S6 switch" and out[n]["U (official)"][0] is not None]
print("largest |greedy - U| outside S6: %.1e" % max(abs(out[n]["greedy"][0] - out[n]["U (official)"][0]) for n in others))
print("U - 1 for S0, S1, S4:", [f"{out[n]['U (official)'][0] - 1:.1e}" for n in ("S0 exact", "S1 permuted ids", "S4 +17 spurious")])

boundary tolerance 2 px; smallest gap between objects 9.220 px; objects in frame 0: [1, 2, 3]
frames in which A meets the void band: [7, 8, 9, 10, 11]
spurious blobs: smallest gap to an object 11.180 px, touch the band: False, pixels in 17 blobs: 153
official test_void_masks: J = 1.0  F = 1.0
disk dilation == (distance to boundary <= 2) on 20 random maps: True

submission            U (official)          greedy      U, no void  U, frames 1-10       semi path
S0 exact                  1.000000        1.000000        0.964624        1.000000        0.967752
S1 permuted ids           1.000000        1.000000        0.964624        1.000000        0.000000
S2 A split                0.864702        0.864702        0.855216        0.866673               -
                  semi path: exit: 4 ids > 3 objects
S3 B+C merged             0.528404        0.528404        0.493028        0.528404        0.494767
S4 +17 spurious           1.000000        1.000000        0.964624        1.000000               -
                  semi path: exit: 20 ids > 3 objects
S4b same, as A            0.846887        0.846887        0.832953        0.847553        0.834956
S5 +18 spurious                  -               -               -               -               -
                  U (official): exit: 21 proposals > 20
                  semi path: exit: 21 ids > 3 objects
S5b one id 21                    -               -               -               -               -
                  U (official): exit: 21 proposals > 20
                  semi path: exit: 21 ids > 3 objects
S6 switch                 0.583333        0.527778        0.549132        0.566667        0.533333
S7 frame 0 empty          0.916667        0.916667        0.881291        1.000000        0.967752

S1 permuted ids: (J+F)/2, rows = proposal ids 1..3, columns = objects A B C
[[0.000000 1.000000 0.000000]
 [0.000000 0.000000 1.000000]
 [1.000000 0.000000 0.000000]]
Hungarian: proposals [1, 2, 3] -> objects ['B', 'C', 'A']; sum minus brute-force optimum 0.0e+00
per-object J as the code labels it: obj1 1.000000 (is B), obj2 1.000000 (is C), obj3 1.000000 (is A)

S2 A split: (J+F)/2, rows = proposal ids 1..4, columns = objects A B C
[[0.565933 0.000000 0.000000]
 [0.000000 1.000000 0.000000]
 [0.000000 0.000000 1.000000]
 [0.594107 0.000000 0.000000]]
Hungarian: proposals [2, 3, 4] -> objects ['B', 'C', 'A']; sum minus brute-force optimum 0.0e+00
per-object J as the code labels it: obj1 1.000000 (is B), obj2 1.000000 (is C), obj3 0.517543 (is A)

S3 B+C merged: (J+F)/2, rows = proposal ids 1..3, columns = objects A B C
[[1.000000 0.000000 0.000000]
 [0.000000 0.581044 0.585212]
 [0.000000 0.000000 0.000000]]
Hungarian: proposals [1, 2, 3] -> objects ['A', 'C', 'B']; sum minus brute-force optimum 0.0e+00
per-object J as the code labels it: obj1 1.000000 (is A), obj2 0.480769 (is C), obj3 0.000000 (is B)

S6 switch: (J+F)/2, rows = proposal ids 1..3, columns = objects A B C
[[0.583333 0.416667 0.000000]
 [0.333333 0.000000 0.000000]
 [0.000000 0.000000 1.000000]]
Hungarian: proposals [1, 2, 3] -> objects ['B', 'A', 'C']; sum minus brute-force optimum 0.0e+00
per-object J as the code labels it: obj1 0.416667 (is B), obj2 0.333333 (is A), obj3 1.000000 (is C)

S6: U - 21/36 = 0.0e+00, greedy - 19/36 = 0.0e+00
S7: U - 11/12 = 0.0e+00, frames 1-10 - 1 = 0.0e+00
largest |greedy - U| outside S6: 0.0e+00
U - 1 for S0, S1, S4: ['0.0e+00', '0.0e+00', '0.0e+00']

The data checks hold: the objects never come within the tolerance of each other or of the blobs, the three ids are present in frame 0, A meets the void band in frames 7–11, the port passes the repository’s void test, and the disk dilation matches the distance threshold exactly. Every printed brute-force comparison confirms that linear_sum_assignment found the optimum.

Every planned outcome held, each as an exact equality where one was derived (all printed differences are 0.0e+00):

• Permuted ids. S1 scores 1 under U, the same as S0: the matching recovers the permutation. The hypothesis held.
• Spurious proposals. S4, with 17 extra proposals, also scores exactly 1, the page’s “we do not penalize” rule; the hypothesis held. Attaching the same 153 pixels to A’s id instead (S4b) lowers U to 0.846887; by construction the blobs now enter A’s union, and in our reading their boundaries, far from A’s, also lower A’s boundary precision. That hypothesis held too. Extra foreground costs nothing only as a separate proposal.
• Greedy against Hungarian. On S6, greedy matching first takes C’s perfect pair, then the largest remaining entry, proposal 1 with A (7/12), which leaves B with only proposal 2, worth 0; it scores $19/36 = 0.527778$. The optimal assignment gives proposal 1 to B (5/12) and proposal 2 to A (4/12) and scores $21/36 = 0.583333$. Both values held exactly, and greedy matched Hungarian on every other submission (largest difference 0.0e+00), as derived. The split and merge rows show why: in S2 the larger half of A (0.594107) wins under both rules and the smaller half is simply unmatched, and in S3 the merged proposal goes to C, whose entry (0.585212) narrowly exceeds B’s (0.581044), while B is matched to an empty padding proposal and scores 0.
• Void pixels. S0 scores exactly 1 under U but 0.964624 without the void mask and 0.967752 on the semi-supervised path, where A’s pixels under the band count as false positives against a background label. The hypothesis held: in the official code the void label only affects the unsupervised path.
• The first frame. S7 loses one frame in twelve for every object, $11/12 = 0.916667$, under U and exactly 1 when frame 0 is dropped; the hypothesis held. An unsupervised method is scored on the first frame, which semi-supervised methods are given.
• Proposal limits. S5 and S5b are both rejected, S5b with only four proposals, because the count is the largest id used; the hypothesis held. The semi-supervised path rejects S2 and S4, which use more ids than there are annotated objects; that hypothesis held as well.
• Per-object labels. In S6 the code’s per-object output lists B’s $\mathcal{J}$ (0.416667) first and A’s (0.333333) second, as derived from its ordering; the hypothesis held.

Two further observations were noted only after the planning script had printed them, so they are reported as such rather than as tested predictions. First, the semi-supervised path gives S1 a score of 0.000000: there, object $k$ is compared with id $k$, which is the semi-supervised protocol working as designed, since methods receive the ids with the first-frame masks at test time. Second, the per-object labels are also reordered in S1, S2 and S3 (in S2, A’s score appears third, behind those of B and C), again with unchanged means.

In our reading, on real submissions the same rules mean that an unsupervised method may output up to 20 tracks with no cost for the extra ones, gains nothing by fragmenting an object, and is judged on whole-sequence identity: a track that switches objects midway is matched to only one of them.

License

As of November 2026 the DAVIS site gives two licences that do not agree. The schema.org metadata embedded in the home page, which describes “DAVIS: Densely Annotated VIdeo Segmentation 2017”, names Creative Commons Attribution-NonCommercial 4.0 (http://creativecommons.org/licenses/by-nc/4.0). The 2019 semi-supervised, interactive and unsupervised challenge pages and the 2020 unsupervised page state instead that “the new annotations in this dataset belong to the organizers of the challenge” and are licensed under a Creative Commons Attribution 4.0 License, linking CC BY 4.0. No page we read states a licence for the videos themselves, so the terms for commercial use of the frames are unclear. The download pages link the archives directly, with no registration step on the pages; submitting to the Codalab servers requires registering there. The evaluation code is under the BSD 3-Clause License (copyright 2020, DAVIS: Densely Annotated VIdeo Segmentation).

Limitations and Bias

• Small. DAVIS 2017 Train has 60 sequences. As the video object segmentation article reports, STM’s authors read the gain from adding YouTube-VOS training data as a sign that DAVIS alone is too small to train a network that generalizes.
• Short clips. The 2016 authors kept clips to 2–4 seconds because of computational cost (§3); in our reading this leaves little room for long occlusions and re-appearances, which later long-video sets, listed in the video object segmentation article, target.
• Salient objects. The 2016 design chose one object, or two connected ones, per clip, and the 2019 unsupervised task asks for at least the objects likely to attract gaze. As the interactive segmentation article reports, Chen et al. (2022) note that the DAVIS frames used to benchmark interactive image segmentation lack small, occluded and non-salient objects, because those frames’ instance masks are merged into one, following DAVIS 2016’s single-object annotation.
• Frozen leaderboards. With the site in maintenance mode, the leaderboards on the site stop at 2020, and later results are reported in papers. The Test-Dev and Test-Challenge ground truth remains withheld; the published comparisons collected in the video object segmentation article are on Val.
• Resolution and tolerance. Scoring at 480p fixes $\mathcal{F}$’s boundary tolerance at 8 pixels ($\lceil 0.008 \times 979.7 \rceil$), which in our reading is lenient for thin structures.

Common Uses

• Training and validating semi-supervised video object segmentation models, reporting $\mathcal{J}\&\mathcal{F}$ on DAVIS 2017 Val and in several papers on DAVIS 2016 Val as well (see the results table in video object segmentation).
• Evaluating frozen self-supervised features by label propagation, as in DINO’s DAVIS-2017 experiment.
• Interactive and unsupervised video segmentation through the 2018–2020 tracks.
• Source videos for derived benchmarks in point tracking, motion segmentation and interactive image segmentation.

Historical Milestones

• 2016. The CVPR paper evaluated twelve methods and two pre-processing baselines at 480p. Among the twelve methods, the unsupervised NLC reached the best mean $\mathcal{J}$, 0.641, closely followed by the semi-supervised FCP at 0.631; an oracle that picks the best MCG object proposal in every frame reached 0.724. The authors observed that semi-supervised methods propagating frame to frame decay over time and that in video, motion is a stronger low-level cue for object presence than colour, so motion saliency scored well above colour saliency (§6.1; Table 2).
• 2017. First DAVIS Challenge, on DAVIS 2017, at a CVPR 2017 workshop in Honolulu; the site reports 22 submissions.
• 2018. Second challenge, in Salt Lake City, with 23 participants, adding the interactive teaser track and releasing the interactive Python package.
• 2019. Third challenge, in Long Beach, with 28 teams across the semi-supervised, interactive and unsupervised tracks. The interactive track was won by Seoung Wug Oh (Yonsei University), with AUC 0.783 and $\mathcal{J}\&\mathcal{F}$@60s 0.791 on the challenge leaderboard, computed by the server over Test-Dev 2017. The unsupervised track’s first-placed entry, listed on the site’s publications page, was UnOVOST by Zulfikar, Luiten and Leibe; its leaderboard table could not be read from the saved page.
• 2020. Last challenge, held virtually with CVPR 2020. The interactive track was won by Yuk Heo (Korea University) with AUC 77.2 and $\mathcal{J}\&\mathcal{F}$@60s 78.7, as the leaderboard prints them; the 2020 leaderboard page does not name the evaluation set.
• After 2020. The site entered maintenance mode (the home page gives no date), moved its result tables to Papers with Code and kept the Codalab evaluation servers running.

Related Datasets

• SegTrack and SegTrack v2, FBMS, and the Berkeley video segmentation set are the predecessors the 2016 paper compares against; its supplemental material tabulates which requirements each meets.
• YouTube-VOS is the larger successor for semi-supervised segmentation, annotated every fifth frame; the video object segmentation article describes it and its overall score $\mathcal{G}$.
• MOSE, LVOS, BURST and SA-V target occlusion, long videos and open-world objects (same article).
• MOTChallenge benchmarks multi-object tracking with boxes rather than masks.

Resources

• DAVIS home page and downloads: davischallenge.org, with the DAVIS 2016 and DAVIS 2017 download pages.
• Evaluation code: davis2017-evaluation (Python, semi-supervised and unsupervised); the DAVIS 2017 page also links a MATLAB repository and the interactive Python package (not read here).
• Later results: the home page points to the Papers with Code task pages for semi-supervised, interactive and unsupervised video object segmentation; the Codalab servers remain the official route for Test-Dev scores.