LVIS

LVIS (Large Vocabulary Instance Segmentation, pronounced “el-vis”) is a dataset of instance segmentation masks for more than 1,000 entry-level object categories, drawn on the images of COCO. Agrim Gupta, Piotr Dollár and Ross Girshick of Facebook AI Research introduced it at CVPR 2019, released version 0.5 later in 2019 and version 1.0 in 2020. Because object categories in natural images follow a long-tailed, roughly Zipfian distribution, many LVIS categories have only a handful of training examples, and the benchmark was built to measure instance segmentation and object detection in that low-shot regime. Its central design idea is the federated dataset: instead of labelling every category in every image, each category is exhaustively annotated on a set of positive images and verified absent on a set of negative images, and evaluation for that category uses only those images. Results are reported as average precision over all categories and separately for rare, common and frequent ones.

Purpose

Gupta, Dollár and Girshick (2019) argued that detectors were evaluated either with few categories (COCO’s 80) or with many examples per category, while the long tail of rare categories is unavoidable: annotating more images keeps uncovering new, rare categories. They followed what they call an evaluation-first design: they fixed the task (COCO-style instance segmentation) and the metric (COCO’s mask AP) before collecting data, kept both for continuity with COCO, and designed the dataset to make that evaluation fair.

A large vocabulary makes fairness hard. COCO’s 80 categories are pairwise disjoint, but among 1,000 categories some overlap (a toy deer is both toy and deer), some are parent and child (vehicle and car), and some are synonyms (backpack and rucksack). If an object carries only one of its valid labels, a detector that predicts another valid label is wrongly penalized. Labelling every one of 164,000 images with every category would remove the problem but forces a yes-or-no decision for every category on every object, with genuine ambiguity, disagreement between annotators and a very large workload. The federated design avoids both.

The other aim was mask quality. The authors argued that COCO’s relatively coarse masks limit how finely predicted mask quality can be told apart, and designed the pipeline to produce masks closer to expert outlines.

Dataset Size

LVIS annotates the COCO 2017 images; the authors had the COCO committee’s permission to label the test images as well. The counts changed as the dataset grew:

Version and source	Images	Categories	Instances
Plan, CVPR 2019 paper	164k	over 1,000	2.2 million (projection)
Analysis set, CVPR 2019 paper	5,000 (COCO val2017, annotated twice)	977 present	11.2 per image on average
v0.5, arXiv version (App. A)	about 82k: 57k train, 5k val, 20k test	1,230	694k train; 51k val
v1.0, official dataset page	about 160k: 100,170 train, 19,809 val, 19,822 test-dev, 19,822 test-challenge	1,203	about 2 million; 1,270,141 train, 244,707 val

The CVPR paper projected 2.2 million masks from the data collected at the time; the arXiv version projected about 2 million after labelling 85k images. The LVIS API README states that v1.0 has 159,623 annotated images, which matches the sum of the four v1.0 splits on the dataset page. The train and validation instance counts add up to 1,514,848 (our arithmetic); the test annotations are not public. All v1.0 counts are as the dataset page lists them in October 2026.

On the 5,000-image analysis set the CVPR paper reports a median of 9 instances per category, an average of 3.4 categories per image (against COCO’s narrower spread) and at most 294 instances in one image. Compared with COCO, LVIS objects tend to be smaller, and objects that fill most of the image are about ten times less frequent.

Classes

Each LVIS category is a WordNet synset, a word sense with a set of synonyms and a definition, so synonyms are never split into separate categories (Gupta et al., 2019); the dataset page describes the categories more cautiously as “loosely based on WordNet synsets”, with definitions mostly derived from WordNet. Each category record carries the synset name, its synonyms, the definition, image and instance counts, and a frequency label.

Vocabulary construction. The authors started from 8.8k WordNet synsets, removed obvious non-candidates such as proper nouns, and kept the intersection with highly concrete common nouns, using the concreteness ratings of Brysbaert, Warriner and Kuperman (2014). They then ran the object-spotting stage (below) on 10k COCO images with autocomplete against this super-vocabulary, repeated the process with the reduced vocabulary, and edited the result by hand. The resulting 1,723 synsets are an upper bound: a category enters the dataset only if annotators actually spot it. After the first half of the v0.5 collection, the authors inspected all 1,415 categories that had appeared and voted on each one’s visual consistency, removing about 18% of the categories and about 10% of the labelled instances (arXiv version, App. A). The v1.0 changelog reports a slight drop from 1,230 to 1,203 categories because of more stringent quality control.

Frequency groups. The arXiv version bins categories by the number of training images in which they appear: rare (1–10 images), common (11–100) and frequent (more than 100), and announces that AP will be reported per bin. The CVPR version has no such bins; its Fig. 9 groups categories by instance count instead (1–20, 21–200, more than 200). In v0.5 roughly 75% of categories appear in at most 100 training images (App. A). For v1.0, Dave et al. (2021, Table 3) count 337 rare, 461 common and 405 frequent categories. The dataset page states that the frequency field is based on the image count in the training set. A comment in the API’s eval.py gives the boundaries as below 10, 10 to 99, and 100 or more, which differs from the paper at the edges; the code itself only reads the frequency label stored in the annotation file, and we did not check which convention the released files follow.

All 80 COCO categories map to LVIS categories; the LVIS API repository ships a mapping from COCO categories to synsets.

Annotation Types

The pipeline has six stages, described in both versions of the paper. Embedded “gold set” questions with known answers monitor annotators in stages 2, 3, 5 and 6.

1. Object spotting. Each image is visited repeatedly. On each visit an annotator marks one object of a category not yet marked, naming it by autocomplete against all synonyms with their definitions, or skips the image; an image skipped three times is retired. Salient objects are spotted first and incidental ones later. The stage is run twice, and a category is kept for an image only if both runs spotted it, so two people must agree on the name. The result is an initial positive set $P_c$ for each category $c$.
2. Exhaustive instance marking. Five annotators check the spotted object against the category’s definition and, if it matches, mark every other instance of the category with a point. Frequent categories are subsampled so that no positive set exceeds 1% of the images.
3. Instance segmentation. One annotator verifies the label of each marked instance and draws its polygon mask. Annotators are trained first and monitored by the gold set and by their average number of vertices per polygon.
4. Segment verification. Up to five annotators rate each mask; a mask is accepted only if four agree it is of high quality and is sent back to stage 3 if two reject it. After four rounds of stages 3 and 4, more than 99% of marked objects have an accepted mask.
5. Full recall verification. Up to five annotators are asked whether any instance of $c$ in the image is still unsegmented. Annotation is accepted as exhaustive when four agree; as soon as two say it is not, the image’s exhaustive flag $e_i^c$ is set to false.
6. Negative sets. Images outside $P_c$ are sampled at random and shown to up to five annotators; if any one reports that $c$ appears, the image is rejected, otherwise it joins the negative set $N_c$, until $N_c$ reaches 1% of the images.

The CVPR paper reports that the exhaustive flag is true in 91% of cases; the missing instances are often in crowds, where delineating many instances is hard. Unlike COCO, which stops at 10–15 instances of a category per image and covers the rest with a crowd region, LVIS tries to segment every instance and then records whether it succeeded.

The released files follow COCO’s JSON layout with additions (dataset page). Each image lists neg_category_ids (categories verified absent) and not_exhaustive_category_ids (categories with at least one unannotated instance), and a coco_url that identifies the COCO split and file. Each annotation has a polygon segmentation, an area and a box; there is no crowd flag.

Mask quality. The CVPR paper measures it twice. In repeatability, stages 2–5 were run twice on the 5,000 analysis images, and the two runs’ instances were matched per image and category: the matched masks agree with an average mask IoU of 0.85, the categories with an average F1 score of 0.87, about half the matched instances have IoU above 0.9 and about three quarters of image–category pairs have a perfect F1. Against experts, 100 instances with area above $32^2$ pixels were drawn at random from each of LVIS, COCO and ADE20K and outlined by two expert annotators with professional image editing tools. Table 1a gives bootstrapped 95% intervals for the mean mask IoU against the experts: 0.90–0.92 for LVIS, 0.83–0.87 for COCO and 0.84–0.88 for ADE20K; the mean boundary quality (boundary F-measure) intervals are 0.87–0.91, 0.77–0.82 and 0.83–0.87. LVIS objects also have more complex boundaries (perimeter divided by the square root of area; Table 1b). In the v0.5 baselines, mask AP is slightly above box AP on LVIS, the reverse of COCO, where mask AP is typically 3 to 4 points below box AP; the authors attribute this to the finer masks and report that degrading the masks programmatically lowered mask AP with almost no change in box AP (arXiv version, App. B.1).

Splits

The CVPR paper’s analysis set is COCO val2017. In v0.5 the train and test images were sampled from the same distribution and annotated in mixed batches, and the val set is the 5,000 analysis images with visually inconsistent categories and categories absent from train removed, which reduced its categories from 977 to 830 (App. A). In v1.0 the validation set grew to about 20k images, and the single test set became two, test-dev for year-round evaluation subject to a usage quota and test-challenge for the annual challenge (changelog). The v1.0 splits no longer follow COCO’s: according to the dataset page, the validation set includes images from COCO train2017 as well as val2017, so a model trained on COCO train2017 has seen some LVIS v1.0 validation images with COCO’s labels (our inference). Test annotations, and with them the membership of each test image in the positive and negative sets, are withheld, so a detector cannot know which categories will be scored on an image and must predict all of them.

The 5,000-image v0.5 val set turned out to be a biased estimate of test performance. Larger evaluation sets contain more rare categories (15.1% of the categories in val against 28.2% in test), and the authors found that AP computed on a smaller evaluation set is biased upward, most strongly for rare categories; for three v0.5 baselines AP dropped by 6.0 to 6.6 points from val to test while their ranking was preserved (App. C, Table 4b). They wrote that they might enlarge the validation set in v1.0, which the changelog confirms was done.

Tasks

The benchmark task is instance segmentation: for a fixed, known vocabulary, output a mask, a category and a confidence score for each instance in a previously unseen image. Box detection is scored with the same protocol on boxes. LVIS is also used to evaluate long-tailed and open-vocabulary detection, where a model trained on other data is evaluated on LVIS’s categories.

Evaluation

The protocol is that of the LVIS API (lvis/eval.py and lvis/results.py, read at commit 7d7f07d) and follows the paper. It keeps COCO’s machinery: greedy matching of score-ranked detections to unclaimed ground truth at IoU thresholds 0.50:0.05:0.95, interpolated precision sampled at 101 recall levels, and averaging over thresholds and categories, as the average precision article describes. What changes is which detections and images are scored:

• Only $P_c \cup N_c$ is evaluated for category $c$. Before matching, the API discards every detection of $c$ on an image that neither contains an annotation of $c$ nor lists $c$ among its negative categories. A detector is neither rewarded nor penalized for predictions about categories whose presence in an image was never established.
• Negative images make false positives count. A detection of $c$ on an image of $N_c$ cannot match anything and is a false positive.
• Not-exhaustive images do not count false positives. On an image whose not_exhaustive_category_ids contains $c$, unmatched detections of $c$ are ignored, while detections matched to annotated instances are true positives and the annotated instances still count toward recall.
• No special treatment of hierarchy. Every category is scored independently (synonyms are already merged into one synset). A detected object is evaluated under every category whose positive set contains its image, so the best strategy is to output the most specific correct category together with its more general ones (canoe and boat), as the paper advises.
• 300 detections per image. LVISResults keeps the 300 highest-scoring detections of each image across all categories, against COCO’s 100; the arXiv version notes that raising the number of detections per image from 100 to 300 and lowering the score threshold from 0.05 to 0 raised a Mask R-CNN baseline from 14.8 to 20.9 mask AP on v0.5 val (Table 3a).
• Summaries. AP averages over all categories with at least one annotated instance in the evaluated images (categories with none are skipped). The API also reports $\mathrm{AP}_{50}$, $\mathrm{AP}_{75}$, AP for small, medium and large objects with COCO’s area limits, average recall at 300 detections, and $\mathrm{AP}_r$, $\mathrm{AP}_c$, $\mathrm{AP}_f$, the AP averaged over the rare, common and frequent categories. Mask AP is the primary metric.

The CVPR paper says the evaluation API, and the arXiv version the challenge evaluation server, returns only the overall AP, not per-category AP, to avoid leaking which categories are present in the test set and because per-category AP is unstable for rare categories; the 2019–2021 challenge results pages nevertheless list $\mathrm{AP}_r$, $\mathrm{AP}_c$ and $\mathrm{AP}_f$ for every team.

The CVPR paper validated the federated design in two ways. Three COCO-trained Mask R-CNN models scored about the same on LVIS’s annotations of the COCO categories as on COCO’s own (for example 41.2 mask AP on COCO’s annotations and 41.7 on LVIS’s for the largest model; Table 2). Subsampling COCO val2017, AP fell by about 2 points as the number of negative images per category grew from 50 to 5,000 with all positives kept, and with 50 negatives the gap between the first and third quartiles over 20 draws stayed below 2 points even with positive sets as small as 5 images (Fig. 8). The arXiv version adds that small positive sets bias AP upward, the effect App. C analyses.

Variance. The best-practices page reports standard deviations over 50 Mask R-CNN models trained on v1.0 with different seeds: 0.20 points for AP (0.10 on COCO), 0.78 for $\mathrm{AP}_r$, 0.25 for $\mathrm{AP}_c$ and 0.07 for $\mathrm{AP}_f$. It recommends reporting the mean and standard deviation over several seeds, treats a 1-point change as possibly significant for AP, $\mathrm{AP}_c$ and $\mathrm{AP}_f$ but not for $\mathrm{AP}_r$, asks not to report the best of several runs and not to train on train and val together, and suggests reporting key results on test-dev through the evaluation server (as of October 2026). The statistical significance article covers the general problem.

Fixed AP. Dave et al. (2021) showed that the per-image cap, shared by all categories, breaks the category independence that AP intends. In their toy example, with a limit of two detections per image, a perfectly calibrated detector scores a higher expected AP by ranking a less confident detection of one class above a more confident one of another. On the LVIS validation set, keeping only the 10,000 top-scoring detections of each category across the whole set before applying the 300-per-image cap raised a Mask R-CNN baseline from 22.6 to 23.8 AP and $\mathrm{AP}_r$ by 2.9 points (Table 1), and raising the cap from 300 to 10,000 per image raised AP from 22.6 to 26.1 and $\mathrm{AP}_r$ from 12.6 to 19.8, while COCO AP was 37.4 at 100 detections per image and 37.5 at every limit from 300 up (Table 2); the text gives the $\mathrm{AP}_r$ endpoint as 19.5. They proposed Fixed AP, with no per-image limit and a budget of 10,000 detections per category over the evaluation set, which keeps 98.5% of the AP of an unlimited budget while increasing file size and evaluation time by about 2×, against 15× for a cap of 5,000 detections per image (Section 4 and Table 4). Under Fixed AP the gains of several long-tail methods shrank: the Federated loss improved the softmax cross-entropy baseline by 2.4 AP under the old protocol and by 0.8 under Fixed AP, while repeat factor sampling and larger backbones kept most of their gains (Table 5). They read this as evidence that many reported improvements came from changing how scores are ranked across categories. They also proposed $\mathrm{AP}^{\text{pool}}$, which pools all categories into one precision–recall curve to reward cross-category calibration.

Boundary AP. The 2021 challenge replaced mask AP with boundary AP and adopted Fixed AP (challenge page). In the API’s lvis_challenge_2021 branch (commit 04e89fd), LVISEval(..., mode="challenge2021") takes the IoU between a detection and a ground-truth mask as the minimum of their mask IoU and their boundary IoU, the IoU of the bands of mask pixels within $d$ of each mask’s contour, with $d$ equal to 2% of the image diagonal (Cheng et al., 2021); the challenge page explains that this makes the measure more sensitive to errors near boundaries and removes mask IoU’s bias towards high values on large objects. As of October 2026, the master branch still defaults to 300 detections per image and mask or box IoU.

Example: federated rules on synthetic masks

The code builds six synthetic $64 \times 64$ scenes for one category, “mug”, drawn as discs with handles: two exhaustively annotated positive images, one positive image with a second, unannotated mug (not exhaustive), two verified negative images (one with a bowl), and one image never examined for mugs that contains a real mug. Eight synthetic mask detections are scored with every image treated as exhaustively labelled (“naive”), and with the federated rules.

import numpy as np

H = W = 64
yy, xx = np.mgrid[0:H, 0:W]

def mug(cx, cy, r):
    """Mask of a synthetic 'mug': a disc with a ring-shaped handle on its right."""
    body = (xx - cx) ** 2 + (yy - cy) ** 2 <= r**2
    d = np.hypot(xx - (cx + r), yy - cy)
    return body | ((d <= 0.6 * r) & (d >= 0.3 * r))

def bowl(cx, cy, r):                          # a distractor: half disc, no handle
    return ((xx - cx) ** 2 + (yy - cy) ** 2 <= r**2) & (yy >= cy)

# Six synthetic 64x64 scenes as lists of object masks; the ground truth for category
# "mug" is what a federated annotation would keep, and each image has a status for "mug".
scenes = {  # image: (status for "mug", objects drawn, which drawn mugs are annotated)
    1: ("positive",       [mug(18, 20, 8), mug(44, 40, 9)],  [0, 1]),
    2: ("positive",       [mug(30, 30, 10)],                 [0]),
    3: ("not exhaustive", [mug(16, 40, 8), mug(46, 18, 7)],  [0]),  # 2nd mug unlabeled
    4: ("negative",       [bowl(30, 34, 12)],                []),   # verified: no mug
    5: ("negative",       [],                                []),
    6: ("unknown",        [mug(32, 28, 9)],                  []),   # never asked about mugs
}
gt = {i: [objs[k] for k in keep] for i, (_, objs, keep) in scenes.items()}

# Synthetic detections of "mug": (image, score, mask), slightly shifted or resized shapes.
dets = [(1, 0.95, mug(19, 20, 8)), (1, 0.60, mug(46, 42, 8)),   # two good hits
        (2, 0.90, mug(31, 30, 10)), (2, 0.40, mug(26, 33, 7)),  # hit and duplicate
        (3, 0.85, mug(16, 41, 8)), (3, 0.80, mug(46, 18, 7)),   # labeled and unlabeled mug
        (4, 0.70, mug(30, 33, 11)),                             # false alarm on the bowl
        (6, 0.75, mug(32, 28, 9))]                              # real mug, unknown image

def iou(a, b):
    return (a & b).sum() / (a | b).sum()

def evaluate(rules, thr):
    """Label each detection TP / FP / ignored / dropped at one IoU threshold."""
    labels, claimed = [], {i: set() for i in gt}
    for k in np.argsort([-s for _, s, _ in dets], kind="stable"):
        img, _, m = dets[k]
        status = scenes[img][0]
        if rules == "federated" and status == "unknown":
            labels.append((k, "dropped")); continue      # image not in P_c or N_c
        ious = [(iou(m, g), j) for j, g in enumerate(gt[img]) if j not in claimed[img]]
        best = max(ious, default=(0.0, -1))
        if best[0] >= thr:
            claimed[img].add(best[1]); labels.append((k, "TP"))
        elif rules == "federated" and status == "not exhaustive":
            labels.append((k, "ignored"))                # unmatched, e_i^c is false
        else:
            labels.append((k, "FP"))
    return labels

def ap101(labels, n_gt):
    tp = np.cumsum([l == "TP" for _, l in labels if l in ("TP", "FP")])
    fp = np.cumsum([l == "FP" for _, l in labels if l in ("TP", "FP")])
    rec, prec = tp / n_gt, tp / np.maximum(tp + fp, 1e-12)
    env = np.maximum.accumulate(prec[::-1])[::-1]        # interpolated precision
    idx = np.searchsorted(rec, np.linspace(0, 1, 101), side="left")
    return np.mean([env[i] if i < len(env) else 0.0 for i in idx])

# Image 3, every 2nd column and 3rd row: '#' annotated mug, 'o' unannotated mug.
view = np.full((H, W), ".")
view[scenes[3][1][1]] = "o"
view[gt[3][0]] = "#"
print("\n".join("".join(r[::2]) for r in view[8:52:3]))
print()

n_gt = sum(len(g) for g in gt.values())
thrs = np.linspace(0.5, 0.95, 10)
print("img  status          score  IoU(best gt)  naive@0.5  federated@0.5")
naive, fed = dict(evaluate("naive", 0.5)), dict(evaluate("federated", 0.5))
for k, (img, s, m) in enumerate(dets):
    best = max([iou(m, g) for g in gt[img]], default=0.0)
    print(f"{img:>3}  {scenes[img][0]:14s}  {s:5.2f}  {best:12.2f}  {naive[k]:>9}  {fed[k]:>13}")
print()
for rules in ("naive", "federated"):
    aps = [ap101(evaluate(rules, t), n_gt) for t in thrs]
    print(f"{rules:9s}  AP@0.5 = {aps[0]:.3f}  AP@[0.5:0.95] = {np.mean(aps):.3f}  "
          f"per threshold: {' '.join(f'{a:.2f}' for a in aps)}")

Output:

................................
.......................o........
.....................ooooooo....
....................ooooooo.o...
....................ooooooooo...
.....................ooooo......
................................
................................
........#.......................
.....#######....................
.....#######.##.................
.....#######..#.................
.....#########..................
.......###......................
................................

img  status          score  IoU(best gt)  naive@0.5  federated@0.5
  1  positive         0.95          0.82         TP             TP
  1  positive         0.60          0.59         TP             TP
  2  positive         0.90          0.86         TP             TP
  2  positive         0.40          0.41         FP             FP
  3  not exhaustive   0.85          0.80         TP             TP
  3  not exhaustive   0.80          0.00         FP        ignored
  4  negative         0.70          0.00         FP             FP
  6  unknown          0.75          0.00         FP        dropped

naive      AP@0.5 = 0.894  AP@[0.5:0.95] = 0.568  per threshold: 0.89 0.89 0.75 0.75 0.75 0.75 0.75 0.13 0.00 0.00
federated  AP@0.5 = 0.950  AP@[0.5:0.95] = 0.579  per threshold: 0.95 0.95 0.75 0.75 0.75 0.75 0.75 0.13 0.00 0.00

Two of the three high-scoring “false positives” in the naive evaluation are correct detections of real mugs: one the annotators missed on image 3, which the not-exhaustive flag turns into an ignored detection, and one on an image nobody checked for mugs, which the federated rules drop. Only the false alarm on the bowl in verified-negative image 4 still counts, so the negative set is what lets LVIS measure false positives at all. The two rules differ only at IoU 0.50 and 0.55: there the detection with score 0.60 is a true positive ranked below the disputed detections, so removing them raises its precision from 4/7 to 4/5. From 0.60 upward that detection no longer matches, no true positive is ranked below the disputed detections, and the two rules agree; ignored false positives only matter when they outrank true positives. Running the official LVISEval from commit 7d7f07d on the same scenes, with the image statuses written as negative and not-exhaustive category lists, gives identical AP, $\mathrm{AP}_{50}$ and $\mathrm{AP}_r$ to six decimals for both rule sets; for that cross-check, the three functions of the COCO API’s compiled mask module that this path calls (area, toBbox and iou) were replaced by NumPy versions working on the masks directly, since pycocotools is not installed in our environment, and OpenCV, imported only by the visualization module, by an empty module.

License

According to the dataset page (as of October 2026), the LVIS annotations and the website are licensed under a Creative Commons Attribution 4.0 License, and all images come from COCO, whose terms of use apply to them. Those terms (as of October 2026) state that the COCO Consortium does not own the images’ copyright, that their use must abide by the Flickr Terms of Use, and that users accept full responsibility for it. The LVIS API is distributed under a two-clause BSD-style license (copyright Agrim Gupta and Ross Girshick).

Limitations and Bias

• Inherited images. LVIS adds no images, so it shares COCO’s image distribution: Flickr photographs of everyday scenes collected with COCO’s searches. As far as we found, neither the LVIS paper nor its website discusses the people who appear in the images; questions of consent and representation are those of COCO’s images.
• A vocabulary of what annotators noticed. Categories enter only if two independent spotting runs name them, so the vocabulary reflects the WordNet super-vocabulary, the concreteness filter and what crowd workers chose to mark in COCO scenes, followed by manual quality control that removed about 18% of v0.5’s candidate categories.
• Noisy tail evaluation. Rare categories have few test instances, so $\mathrm{AP}_r$ varies about four times as much as AP across training seeds (best-practices page), and small evaluation sets bias AP upward (arXiv version, App. C). Comparisons on $\mathrm{AP}_r$ need several runs.
• Incomplete labels by design. Each category’s positive set is capped at, and its negative set aimed at, 1% of the images, so a category is scored on a small fraction of the data, and on not-exhaustive images its false positives go unmeasured. A detector that marks correct objects nobody annotated loses nothing, but neither does one that hallucinates objects on images outside a category’s two sets.
• Version differences. Counts, categories and splits differ between v0.5 and v1.0, and the v1.0 validation set mixes COCO train2017 and val2017 images, so results on the two versions are not directly comparable. Even the paper’s object-center figure changed: the arXiv version notes that the CVPR version plotted Open Images’ validation set, which has more center bias, with an intensity-scaling artifact, and replaces it with the training set.

Common Uses

• Long-tailed instance segmentation and detection, reporting mask AP with $\mathrm{AP}_r$, $\mathrm{AP}_c$ and $\mathrm{AP}_f$ on v1.0 val, often with Mask R-CNN and repeat factor sampling as the baseline.
• Open-vocabulary and zero-shot detection, where models trained on other data are evaluated on LVIS’s categories, as with Grounding DINO; SAM was evaluated on LVIS v1.0 for object proposals and instance segmentation.
• Mask quality. Its finer masks are used as a stricter reference for COCO’s categories than COCO’s own masks, as the Mask R-CNN article reports for PointRend.

Historical Milestones

• 2019, CVPR. The paper introduced the federated design, the six-stage pipeline and the analysis of 5,000 doubly annotated images, and showed that Mask R-CNN trained on 1k COCO images reached only 9.8 mask AP on COCO val2017 against 36.4 with the full training set (Fig. 8c), as evidence that low-shot detection was an open problem.
• 2019, v0.5 and the arXiv version. Mask R-CNN with a ResNet-50-FPN backbone scored 14.8 mask AP on v0.5 val with COCO’s inference settings and 20.9 with 300 detections and no score threshold, with $\mathrm{AP}_r$ of 0.8 and 3.4 (Table 3a). Repeat factor sampling raised $\mathrm{AP}_r$ from 3.2 to 13.4 and AP from 21.0 to 23.2 (means of five runs, Table 3b). Each category $c$ gets a repeat factor $r_c = \max\left(1, \sqrt{t / f_c}\right)$, where $f_c$ is the fraction of training images that contain $c$, and each image is repeated in every epoch according to the largest repeat factor among its categories; with $t = 0.001$, the 829 of 1,230 categories that appear in fewer than 0.1% of images are oversampled. The paper’s text says at one point that categories with $f_c \le t$ are not oversampled, which reverses the formula and the paper’s own count.
• 2019, first LVIS Challenge (ICCV 2019, on v0.5). Teams strangeturtle (28.9 AP on test) and lvlvisis (26.7) were named co-winners; the organizers’ baseline scored 20.5 (challenge page).
• 2020, v1.0 and the second challenge (ECCV 2020, jointly with COCO). Team lvisTraveler won with 41.3 AP and 32.0 $\mathrm{AP}_r$; the organizers’ baseline, Mask R-CNN with a ResNeXt-101-32x8d backbone and classifier retraining, scored 26.7 on test-challenge. External annotations and model ensembles were no longer allowed, except the classification labels of the 1,000-category ImageNet training set (challenge page).
• 2021, Fixed AP and Boundary AP. Dave et al. (2021) published Fixed AP, and the third challenge (ICCV 2021 workshop) scored boundary, fixed AP; team FuXi-Fresher won with 48.1 (challenge page). The challenge pages list no later LVIS challenge (as of October 2026).

Related Datasets

• COCO: the source of all LVIS images and of the AP protocol LVIS extends; its 80 categories map into LVIS.
• Open Images: per the LVIS paper, Open Images V4 adopted a federated design for detection concurrently, with a “group of” label for five or more instances instead of exhaustive masks, and selected images for annotation with classifiers.
• ADE20K: densely annotated scenes with about 3,000 named objects, stuff and parts by a single expert annotator, used in the LVIS mask-quality comparison.
• iNaturalist: nearly 900k images with boxes for 5,000 plant and animal species, which the LVIS paper contrasts with its own entry-level categories and masks.
• ImageNet: its WordNet synsets are the same kind of category identifiers LVIS uses, and its classification labels were the only external annotations allowed in the 2020 challenge.

Resources

• lvisdataset.org: downloads, data format, changelog, license, best practices and the 2019–2021 challenge pages with their results.
• LVIS API: loading, visualization and the official evaluation (LVISEval); the lvis_challenge_2021 branch implements Fixed AP and Boundary AP.
• The LVIS v1.0 evaluation server on EvalAI, linked from the best-practices page, for test-dev results.
• The arXiv version of the paper, with the v0.5 statistics, baselines and val-to-test analysis.