iNaturalist

iNaturalist is a family of species classification datasets made from photographs that members of the iNaturalist citizen-science platform upload as observations of plants, animals and fungi. Grant Van Horn and colleagues at Caltech, Google, Cornell Tech and iNaturalist introduced the first edition, iNat2017, in a CVPR 2018 paper; its competition ran on Kaggle in 2017 as part of the FGVC4 workshop at CVPR. Later editions followed with the FGVC5 (2018), FGVC6 (2019) and FGVC8 (2021) workshops, all documented in the Visipedia inat_comp repository. iNat2017 has 5,089 categories with a training set whose images per class follow how often the community observes each taxon, from 9 to 3,919. iNat2021 has 10,000 species with at least 152 training images each. Each edition is a fine-grained problem with many visually similar species, and the early ones are standard examples of long-tailed training data.

This article covers the editions as datasets: their construction, splits and evaluation designs, the official scorer as its code computes it, and the terms of use. Fine-Grained Image Classification covers iNaturalist as a benchmark for fine-grained methods, and Class Imbalance covers long-tail methods and their results.

Purpose

Van Horn et al. (2018) start from the observation that existing classification datasets had roughly uniform numbers of images per class, while the natural world is “heavily imbalanced” because some species are more abundant and easier to photograph. They built iNat2017 to push classification and detection on data with many imbalanced, fine-grained categories photographed “in the wild”. The paper names further properties: images taken with many kinds of cameras and of varying quality; labels confirmed by several citizen scientists rather than taken from web searches; and a use in biodiversity monitoring. The paper also hoped the dataset could grow with the platform: at the time of writing, with counts given as of November 2017, every 1.7 hours another species passed the threshold of 20 unique observers needed for inclusion.

The 2021 edition had a different aim. Van Horn et al. (2021) present iNat2021 as a testbed for representation learning, including self-supervised learning, on natural-world images. They say it “effectively removed the long tail” of the earlier training sets but keeps enough images per species to study the tail by removing data systematically.

Dataset Size

The editions differ in scale and in how many images each class gets. The table combines the repository’s READMEs with Table 1 of Van Horn et al. (2021). The last three columns are training images per class.

Edition	Classes	Train	Validation	Test	Min	Max	Mean
iNat2017	5,089	579,184	95,986	182,707	9	3,919	114
iNat2018	8,142	437,513	24,426	149,394	2	1,000	54
iNat2019	1,010	265,213	3,030	35,350	16	500	263
iNat2021 mini	10,000	500,000	100,000	500,000	50	50	50
iNat2021	10,000	2,686,843	100,000	500,000	152	300	267

The two iNat2021 variants share the same validation and test sets. The 2019 README gives only the combined training and validation count, 268,243, which equals the sum in the 2021 table. The 2017 paper describes the dataset as 859,000 images in total, with “over 560,000” boxes. Images are JPEGs with a maximum dimension of 800 pixels in 2017–2019 and 500 pixels in 2021. As of November 2026 the 2021 README lists the training images as a 224 GB archive and the mini training set as 42 GB.

Per super-category, iNat2017 is dominated by plants (2,101 classes, 158,407 training images), insects (1,021; 100,479) and birds (964; 214,295). Birds have fewer classes than insects but more than twice as many training images (2017 README).

Classes

• iNat2017. In October 2016 the GBIF export of research-grade iNaturalist observations held 54k taxa at genus level and below. The authors kept taxa with at least 20 observations from at least 20 different observers. This left 5,089 taxa in 13 super-classes, from Plantae to Protozoa (Van Horn et al., 2018, §3.1–3.2). The 2018 README notes that the 2017 categories were “mostly species” and include a few genus, subspecies and variety ranks.
• iNat2018. All 8,142 categories are species, with kingdom, phylum, class, order, family and genus given for each. Every taxonomic name, including the species name, was obfuscated “to hinder participants from performing web searchs”. The real names were released on 23 June 2018, after the workshop.
• iNat2019. Species with too few observations were removed. Of the rest, the 2019 README says only species in genera with at least 10 remaining species were kept, giving 1,010 species in 72 genera: in effect many 10-way fine-grained problems (Van Horn et al., 2021, §3.2). Names were again obfuscated and released on 6 August 2019.
• iNat2021. 10,000 species in eleven “iconic groups” (plants 4,271, insects 2,526, birds 1,486, then fungi, reptiles, mammals, ray-finned fishes, amphibians, mollusks, arachnids, and a catch-all Animalia). The names were released at once. The README’s reason is that with 2.7 million training images and no focus on the long tail, scraping for more data was less of a concern; scraping remained forbidden. Each category entry also carries a common name and the full taxonomy down to genus and specific epithet.

Species and images overlap between editions, and no mapping is provided between them (2018, 2019 and 2021 READMEs).

Annotation Types

• Labels. Each image has one category label, from the community’s consensus on its observation. The annotation files follow the COCO JSON layout (images, categories, annotations, licenses). Each image record carries its own license id and rights_holder.
• Bounding boxes (2017 only). Crowd workers were asked to box every instance of the super-class rather than the species (“Box all Birds”), up to 10 per image. Boxes were drawn on 9 of the 13 super-classes, covering 2,854 classes; plants, fungi, protozoa and chromista have none. The README warns that several boxes in one image share the image’s species label, though they are only guaranteed to share its super-class. For that reason the box validation set is restricted to single-instance images. iscrowd is always 0, and area is approximated as half the box area, since no masks were collected. The paper reports 561,767 boxes on 449,313 training and validation images (Table 2 and §3.3). The 2017 README’s box table gives 496,164 training and 48,736 validation boxes, 544,900 in all, and its per-super-class counts also differ (Insecta 106,304 + 16,732 against the paper’s 125,679). We report the disagreement without resolving it.
• Location and time. On 16 August 2019 the repository added latitude, longitude, date and user id for the 2017 and 2018 training and validation images. iNat2021 gives latitude, longitude, location_uncertainty and date for every image. The 2021 README says the latitude, longitude and uncertainty can be null for private locations or sensitive species, and that location_uncertainty is left raw, with “bogus values such as -80 and 106807033”. The 2021 README says that location and date had been added retroactively to 2017 and 2018, but that in 2021 competitors could use them when building models.

Splits

iNat2017: by observer. For each taxon, observers were sorted by their number of observations, fewest first. The first 40% went to the test split and the rest to “train-val”, so that one person’s camera, location and backgrounds stay within one split for that taxon. One observer can still be in the test split for one taxon and in train-val for another. About 183,000 test images were then sampled at random from the test split, and the rest were held in reserve. Train-val was split again by observer: observers were added to validation until it held more than 30 images or 33% of the taxon’s train-val images. This gives at least 4 and at most about 30 validation images per class and at least 8 training images (Van Horn et al., 2018, §3.2). The test set is divided into a public part of 90,427 images and a private part of 92,280 (Table 2 caption). Crowd workers checked the test images of three super-classes for “non-instance” photos such as footprints, feces and habitat shots. Fewer than 1.1% of the bird and reptile test images were non-instance images; the fraction was higher for mammals, and these images were filtered out of the test set, while training and validation images were not filtered (§3.2.1).

iNat2018 and iNat2019: three validation images per class. iNat2018 has 24,426 validation images for 8,142 species and iNat2019 has 3,030 for 1,010, exactly 3 per class in both. The READMEs do not describe how either edition’s test set was made, or how many test images each class has.

iNat2021: by date. Training and test images were split at a fixed date, and one photographer may have images in both. The authors describe this as building a training set from data submitted over a year ago “to classify the most observed species in the last year” (Van Horn et al., 2021, §3.1). Every species has 10 validation images. The mini training set takes 50 images per species from the full training set. The per-class make-up of the 500,000 test images is not given in the README or the paper’s main text; the paper refers to its supplementary material for the construction, which we did not read.

The test sets were scored on the competitions’ Kaggle leaderboards; the 2021 README notes that test.json has no annotations field and that the test set can only be evaluated by uploading a submission to Kaggle. Results in later papers are frequently given on a validation set instead, such as the iNat2018 validation results that Class Imbalance cites.

Tasks

• Species classification: every edition, one label per image, scored by top-$k$ error (below). See image classification and fine-grained image classification.
• Detection: iNat2017 only. Van Horn et al. (2018) trained Faster R-CNN on the 2,854 boxed classes and reported COCO-style average precision on validation images with a single instance, which the paper uses in place of the test set. See object detection.
• Representation learning and transfer: iNat2021 and its mini set were introduced for pretraining studies. They are paired with NeWT, a separate suite of 164 binary natural-world tasks from the same paper. Other papers report frozen-feature results on iNat2018 or iNat2021 (see transfer learning).

Evaluation

The metric as the READMEs define it. For image $i$ with true label $g_i$ and submitted labels $l_{i1}, \dots, l_{ik}$,

$$e_i = \min_j d(l_{ij}, g_i), \qquad d(x, y) = \begin{cases} 0 & x = y \\ 1 & \text{otherwise,} \end{cases} \qquad \text{score} = \frac{1}{N} \sum_i e_i ,$$

an average over all $N$ test images, with no weighting by class. The number of labels changed with each edition:

• 2017: $k = 5$. The README says several labels are allowed because location and date can be needed to tell some categories apart, and because one photo can show several species (“a bee on a flower”).
• 2018: $k = 3$. Participants were asked to sort the three labels by confidence so that top-1 and top-2 could also be studied.
• 2019: $k = 1$. The README calls this making the metric “more strict”.
• 2021: top-1 on the leaderboard, but five labels required. The README says the organizers would use the other four labels to study top-$k$ error and present it at the workshop. The README lists the cases a top-$k$ metric forgives: several species in one image, noisy labels, and species that look identical but live in different places. Location and date were provided to address the last.

This is accuracy at $k$ subtracted from one. ILSVRC uses the same per-image form (the ImageNet article covers its top-5 and hierarchical variants). In both, the score weights each class by its number of test images.

The scorer as its code computes it. The repository’s eval/top_k_accuracy.py (Grant Van Horn; read at commit 4ea5638) computes public and private accuracy from a submission CSV and a solution CSV:

• It reads the columns id and predicted from the submission and id, expected and usage from the solution. usage must be Public or Private, otherwise it raises an error.
• It splits each predicted string on single spaces. It raises an error if a row has fewer than $k$ labels, and keeps only the first $k$. The order of the labels therefore matters, and extra labels are ignored, which is how a five-label 2021 row can be scored at top-1.
• An image counts as correct if its true label is anywhere among its kept labels. Repeated labels in a row are neither penalized nor rewarded.
• It raises an error if any solution image is missing from the submission, and silently ignores submission images that are not in the solution. A row repeated for the same image is scored twice.
• Public and private accuracies are each a count of correct images divided by the number of images, with an empty set scored 0.

The code and its documentation do not fully agree. The module’s docstring describes the columns as image_id, predictions and image_id, truth, usage. The unit-test file eval/test_top_k_accuracy.py writes files with those headers, so its submission tests would fail with a missing-column error against this scorer; we read this and did not run the file as written. The test file also uses Python 2 idioms (len of a map object). The 2021 README’s submission example uses capitalized Id,Predicted headers, which this scorer does not read. The scorer also uses np.object, which NumPy removed in version 1.24. Kaggle hosted the leaderboards. The example below runs a port of the repository’s scorer, which may or may not be the code Kaggle ran; we make no claim either way.

Reported numbers depend on the evaluation design. The editions’ evaluation sets give each class a very different number of images:

• iNat2017: validation has 4 to about 30 per class. The test set is a random sample of images from 40% of each taxon’s observers, so common taxa contribute more test images, though the paper gives no per-class counts.
• iNat2018 and iNat2019: 3 validation images per class.
• iNat2021: 10 validation images per class.

Per-image accuracy on a set like iNat2017’s counts common classes more. Mean per-class accuracy, as in the paper’s per-super-class Table 4, counts every class once. On sets with the same number of images per class, the two are equal. With 3 or 10 images per class, any one class’s measured accuracy is coarse: with 3 images it can only be 0, 1/3, 2/3 or 1. The example measures this.

Example: a fixed classifier under three evaluation designs

The script makes a synthetic species problem, trains one classifier on a long-tailed training set and leaves it fixed. It then scores the classifier on three kinds of evaluation set:

• “2017-style” (our analogue): each class has test images in proportion to its training images, at iNat2017’s ratio of 182,707 test to 579,184 training images, and at least one. The paper describes an observer split followed by a random sample, not this rule.
• “2018-style”: designed like the iNat2018 and iNat2019 validation sets, 3 per class.
• “2021-style”: designed like the iNat2021 validation set, 10 per class.

Each design is drawn 20 times (seeds 1000–1019), and every image is randomly marked Public or Private with probability one half, as in iNat2017’s two test halves.

Data. Images are 32 × 32 RGB gratings. There are 200 classes: 40 genera, each with its own hue (8 values) and grating orientation (5 values, 36° apart), and 5 species per genus that differ only in grating frequency, 4 to 8 cycles per image width. Species of one genus are therefore the confusable ones, as on iNaturalist. Each image has random orientation (standard deviation 6°), frequency (6% log-normal), phase, contrast (0.5–1), colour (standard deviation 0.05 per channel) and pixel noise (standard deviation 0.6).

Classifier. The features are fixed by hand: the mean colour, plus the share of the grey image’s spectral power in 12 orientation bins and 10 radial bins. The classifier is a nearest class mean on these features, and its top-$k$ labels are the $k$ nearest class means. The training set is long-tailed: class $c$ of rank $r$ gets $\operatorname{round}(300\, r^{-\ln 100 / \ln 200})$ images, from 300 to 3, with the ranks shuffled over classes (seed 0). No network is trained.

Scoring. Each draw is written as Kaggle-style CSVs and scored by the official functions, ported into the script with the full MIT notice. The one change is np.object → object; everything else is as at commit 4ea5638, with the docstrings and comments cut. A vectorized reimplementation scores the same draws.

Checks. The script first runs:

• six of the seven compute_top_k_accuracy tests of the repository’s unit-test file, ported to Python 3 lists (the seventh, which expects an error when labels and predictions differ in length, is not ported);
• six edge cases of evaluate(), whose results we predicted by reading the code;
• a base check: on noise-free, jitter-free images, class means from 3 images per class must classify 3 new images per class perfectly.

Measures. Each class $c$ has a reference accuracy $p_c$, measured on 500 images per class (seed 1). For a draw with $n_c$ images of class $c$, the measured per-class accuracy is $a_c$. Because the classifier is fixed and test images are independent, $n_c a_c$ is binomial with parameters $n_c$ and the class’s true accuracy. The script prints two numbers:

• $\rho = \sum_c (a_c - p_c)^2 / \sum_c \hat v_c$, where $\hat v_c = p_c(1-p_c)\frac{M}{M-1}\left(\frac{1}{n_c} + \frac{1}{M}\right)$ is the expected squared deviation, including the reference set’s own noise ($M = 500$). It should be near 1.
• The share of the spread across classes that is binomial noise: $\operatorname{mean}_c\, p_c(1-p_c)\frac{M}{M-1}/n_c$ divided by the variance of $a_c$ over classes.

Both are averaged over the 20 draws and printed with their range. As a printed column only, the script also gives per-image accuracy (public, private and all) and mean per-class accuracy for draw 0 of each design.

Disclosures. The noise level, the colour weight and the unscaled features were chosen in planning probes so that top-1 accuracy is mid-range and most errors stay within the genus. The probes printed top-1 and top-5 accuracy, the share of errors within the genus, and the spread of per-class top-1 accuracy over 200 images per class with its correlation with training-set size; they did not compute $\rho$ or the share defined above. Every bar below was then written after a planning script had printed the exact binomial model’s values for this set-up. That script simulated 2,000 replications of the 20 draws from the reference accuracies and never drew the example’s test images. The example is therefore an illustration of what that model predicts, not a test of an open question. The hypotheses were:

• H1: the port and the reimplementation agree exactly.
• H2: for every design and $k$, $\rho$ lies inside the exact model’s 0.5–99.5% interval for that condition. The intervals are 0.9585–1.0438, 0.9421–1.0650 and 0.9081–1.0948 for the 2017-style design at $k = 1, 3, 5$; 0.9501–1.0503, 0.9350–1.0714 and 0.9198–1.0889 at 3 per class; and 0.9405–1.0520, 0.9378–1.0707 and 0.9214–1.0781 at 10 per class.
• H3a: at 3 per class, more than half of the spread is noise at every $k$.
• H3b: the share is larger at 3 per class than at 10.
• H3c: the share is larger for the 2017-style design than at 3 per class. The planning script showed this before it was stated.

The script uses only Python 3.13.9 and NumPy 2.3.5, with one BLAS thread. It writes its CSV files to a new temporary folder beside itself and runs in about 12 seconds. Two runs gave identical output.

import os
for v in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS"):
    os.environ[v] = "1"
import csv, random, tempfile
import numpy as np

# ---------------------------------------------------------------------------
# Official scorer: visipedia/inat_comp eval/top_k_accuracy.py, commit 4ea5638
# (Grant Van Horn). Functions copied verbatim except one mechanical port:
# np.object -> object (np.object was removed in NumPy 1.24). Docstrings and
# comments cut. The unit-test helper below is ported from the same repository.
#
# MIT License
#
# Copyright (c) 2017 Visipedia
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.
# ---------------------------------------------------------------------------
def read_submission_file(submission_file, k):
    image_ids = []
    predictions = []
    with open(submission_file) as csvfile:
        reader = csv.DictReader(csvfile)
        for row in reader:

            image_id = row['id']
            preds = row['predicted']

            preds = preds.strip().split(' ')
            if len(preds) < k:
                raise ValueError("Image %s in submission file %s contains "\
                                 "%d predictions when k=%d." % (
                                     image_id, submission_file, len(preds), k))
            preds = preds[:k]

            image_ids.append(image_id)
            predictions.append(preds)

    predictions = np.array(predictions, dtype=object)   # was dtype=np.object
    return image_ids, predictions

def read_solution_file(solution_file):
    public_data = {}
    private_data = {}

    with open(solution_file) as csvfile:
        reader = csv.DictReader(csvfile)
        for row in reader:
            image_id = row['id']
            true_label = row['expected']
            usage = row['usage']

            if usage == 'Public':
                public_data[image_id] = true_label
            elif usage == 'Private':
                private_data[image_id] = true_label
            else:
                raise ValueError("Image %s in solution file %s contains "\
                                 "an unknown usage %s" % (
                                     image_id, solution_file, usage))

    return public_data, private_data

def compute_top_k_accuracy(labels, predictions):
    num_samples = len(labels)

    if num_samples == 0:
        return 0.

    if num_samples != len(predictions):
        raise ValueError("`labels` and `predictions` must have the same "\
                         "number of entries")

    num_correct = 0
    for i in range(num_samples):
        if labels[i] in predictions[i]:
            num_correct += 1
    return num_correct / (1. * num_samples)

def evaluate(submission_file, solution_file, k):
    if k < 1:
        raise ValueError("Bad value for k %d" % (k,))

    submission_image_ids, submission_predictions = read_submission_file(
        submission_file, k)

    public_image_data, private_image_data = read_solution_file(solution_file)
    public_image_ids = set(public_image_data.keys())
    private_image_ids = set(private_image_data.keys())

    all_solution_image_ids = public_image_ids.union(private_image_ids)
    missing_submission_ids = all_solution_image_ids.difference(
        submission_image_ids)
    num_missing_submission_ids = len(missing_submission_ids)
    if num_missing_submission_ids > 0:
        raise ValueError("Submission file %s is missing %d image ids that "\
                         "were found in the solution file %s" % (
                             submission_file, num_missing_submission_ids,
                             solution_file)
                        )

    public_submission_predictions_idxs = []
    public_submission_labels = []
    private_submission_predictions_idxs = []
    private_submission_labels = []

    for idx, image_id in enumerate(submission_image_ids):
        if image_id in public_image_ids:
            public_submission_predictions_idxs.append(idx)
            label = public_image_data[image_id]
            public_submission_labels.append(label)
        elif image_id in private_image_ids:
            private_submission_predictions_idxs.append(idx)
            label = private_image_data[image_id]
            private_submission_labels.append(label)
        else:
            pass

    public_submission_predictions = submission_predictions[
        public_submission_predictions_idxs]
    private_submission_predictions = submission_predictions[
        private_submission_predictions_idxs]

    public_accuracy = compute_top_k_accuracy(public_submission_labels,
                                             public_submission_predictions)
    private_accuracy = compute_top_k_accuracy(private_submission_labels,
                                              private_submission_predictions)

    return public_accuracy, private_accuracy
# ------------------------------ end of official code ------------------------

TMP = tempfile.mkdtemp(dir=os.path.dirname(os.path.abspath(__file__)))

def write_csvs(tag, ids, ranked, truth, public, header=("id", "predicted")):
    sub = os.path.join(TMP, tag + "_sub.csv"); sol = os.path.join(TMP, tag + "_sol.csv")
    with open(sub, "w", newline="") as f:
        w = csv.writer(f); w.writerow(header)
        for i, r in zip(ids, ranked):
            w.writerow([i, " ".join(str(x) for x in r)])
    with open(sol, "w", newline="") as f:
        w = csv.writer(f); w.writerow(["id", "expected", "usage"])
        for i, t, p in zip(ids, truth, public):
            w.writerow([i, t, "Public" if p else "Private"])
    return sub, sol

def our_top_k(ranked, truth, public, k):
    hit = (ranked[:, :k] == truth[:, None]).any(axis=1)
    return hit[public].mean(), hit[~public].mean()

# ---------------------------------------------------------------------------
# Synthetic species: 40 genera (8 hues x 5 grating orientations) of 5 species
# each, the species of a genus differing only in grating frequency (4..8 cycles
# per 32 px). Nuisances: orientation, frequency, phase, contrast, colour, noise.
# ---------------------------------------------------------------------------
G_HUES, G_ORI, S = 8, 5, 5
C = G_HUES * G_ORI * S
H = W = 32
SIGMA = 0.6
yy, xx = np.mgrid[0:H, 0:W].astype(float)

def hue_rgb(h):
    a = 2 * np.pi * h
    return 0.5 + 0.35 * np.array([np.cos(a), np.cos(a - 2*np.pi/3), np.cos(a + 2*np.pi/3)])

def class_params(c):
    g, s = divmod(c, S)
    hue, ori = divmod(g, G_ORI)
    return hue_rgb(hue / G_HUES), np.pi * ori / G_ORI, 4.0 + 1.0 * s

def make_images(c, n, rng, clean=False):
    rgb, th0, f0 = class_params(c)
    j = 0.0 if clean else 1.0
    th = th0 + j * rng.normal(0, np.deg2rad(6), n)
    f = f0 * np.exp(j * rng.normal(0, 0.06, n))
    ph = rng.uniform(0, 2*np.pi, n)
    con = rng.uniform(0.5, 1.0, n)
    u = xx[None] * np.cos(th)[:, None, None] + yy[None] * np.sin(th)[:, None, None]
    gr = con[:, None, None] * np.cos(2*np.pi * f[:, None, None] * u / W + ph[:, None, None])
    col = rgb[None] + j * rng.normal(0, 0.05, (n, 3))
    img = col[:, None, None, :] * (0.6 + 0.4 * gr[..., None])
    return img + j * rng.normal(0, SIGMA, img.shape)

fy = np.fft.fftfreq(H)[:, None] * H
fx = np.fft.fftfreq(W)[None, :] * W
rad = np.hypot(fy, fx)
ang = np.mod(np.arctan2(fy, fx), np.pi)
valid = (rad >= 2) & (rad < 12)
obin = (np.floor(ang / (np.pi / 12)).astype(int) % 12)[valid]
rbin = np.floor(rad - 2).astype(int)[valid]

def features(img):
    """Mean colour (3) + share of spectral power in 12 orientation and 10 radial bins."""
    gray = img.mean(axis=3)
    P = np.abs(np.fft.fft2(gray - gray.mean(axis=(1, 2), keepdims=True))) ** 2
    Pv = P[:, valid]
    tot = Pv.sum(axis=1, keepdims=True)
    O = np.stack([Pv[:, obin == j].sum(1) for j in range(12)], 1) / tot
    R = np.stack([Pv[:, rbin == j].sum(1) for j in range(10)], 1) / tot
    return np.hstack([img.mean(axis=(1, 2)), O, R])

def rank_classes(F, means, k=5):
    d = ((F[:, None, :] - means[None]) ** 2).sum(-1)
    return np.argsort(d, axis=1, kind="stable")[:, :k]

def sample(counts, rng, clean=False):
    F = np.vstack([features(make_images(c, n, rng, clean)) for c, n in enumerate(counts) if n > 0])
    y = np.repeat(np.arange(C), counts)
    return F, y

# ---------------------------------------------------------------------------
print("== Checks of the scorer port and of the data")
# (a) the official unit tests of compute_top_k_accuracy, ported to Python 3 lists
random.seed(0)
def make_accuracy_data(num_classes, num_test_samples, num_correct_predictions, k):
    labels, predictions = [], []
    correct = set(random.sample(range(num_test_samples), num_correct_predictions)) if num_correct_predictions else set()
    for i in range(num_test_samples):
        gt = random.choice(range(num_classes)); labels.append(gt)
        preds = random.sample(range(num_classes), k)
        if i in correct:
            if gt not in preds: preds[0] = gt
        elif gt in preds:
            preds = [p for p in preds if p != gt]; preds.append(gt + 1 % num_classes)
        predictions.append(preds)
    return labels, predictions
cases = [(10, 100, 100, 1, 1.0), (10, 100, 100, 3, 1.0), (10, 10, 5, 5, 0.5),
         (10, 100, 0, 5, 0.0), (5, 100, 100, 5, 1.0)]
ok = []
for nc, n, ncor, k, want in cases:
    lab, pr = make_accuracy_data(nc, n, ncor, k)
    slab, spr = [str(x) for x in lab], [[str(x) for x in p] for p in pr]
    got = [compute_top_k_accuracy(lab, pr), compute_top_k_accuracy(slab, spr),
           compute_top_k_accuracy(np.array(slab), np.array(spr))]
    ok.append(all(g == want for g in got))
ok.append(compute_top_k_accuracy([], []) == 0 and compute_top_k_accuracy(np.array([]), np.array([])) == 0)
print(f"official compute_top_k_accuracy unit-test cases passed: {sum(ok)} of {len(ok)}")

# (b) edge cases of evaluate(), read from the code
ids = ["1", "2", "3", "4"]; truth = ["7", "7", "7", "7"]; pub = np.array([True, True, False, False])
def run(rows, k, header=("id", "predicted"), sol_ids=ids):
    sub = os.path.join(TMP, "edge_sub.csv"); sol = os.path.join(TMP, "edge_sol.csv")
    with open(sub, "w", newline="") as f:
        w = csv.writer(f); w.writerow(header); [w.writerow(r) for r in rows]
    with open(sol, "w", newline="") as f:
        w = csv.writer(f); w.writerow(["id", "expected", "usage"])
        for i, t, p in zip(sol_ids, truth, pub): w.writerow([i, t, "Public" if p else "Private"])
    try:
        return evaluate(sub, sol, k)
    except Exception as e:
        return type(e).__name__
print("duplicates in a row, k=3 (rows '7 7 7', '1 1 7', '1 1 1', '1 2 3'):",
      run([["1", "7 7 7"], ["2", "1 1 7"], ["3", "1 1 1"], ["4", "1 2 3"]], 3))
print("a row with 2 labels when k=3:", run([["1", "7 7 7"], ["2", "1 7"], ["3", "1 2 3"], ["4", "1 2 3"]], 3))
print("2021 README headers 'Id,Predicted':",
      run([["1", "7"], ["2", "7"], ["3", "7"], ["4", "7"]], 1, header=("Id", "Predicted")))
print("image 1 submitted twice (right, then wrong), k=1:",
      run([["1", "7"], ["1", "3"], ["2", "3"], ["3", "7"], ["4", "3"]], 1))
print("image 4 missing from the submission:", run([["1", "7"], ["2", "7"], ["3", "7"]], 1))
print("an extra image 9 not in the solution, k=1:",
      run([["1", "7"], ["2", "3"], ["3", "7"], ["4", "7"], ["9", "7"]], 1))

# (c) generator and base check: noise-free, jitter-free images, 3 training and 3 test per class
params = {tuple(np.round(np.r_[class_params(c)[0], class_params(c)[1:]], 9)) for c in range(C)}
print(f"classes {C}, distinct generating parameters {len(params)}")
rng = np.random.default_rng(10)
Fb, yb = sample(np.full(C, 3), rng, clean=True)
mb = np.stack([Fb[yb == c].mean(0) for c in range(C)])
Ft, yt = sample(np.full(C, 3), rng, clean=True)
rb = rank_classes(Ft, mb)
pb = np.arange(len(yt)) % 2 == 0
sub, sol = write_csvs("base", np.arange(len(yt)), rb, yt, pb)
print("noise-free base check, official top-1 (public, private):", evaluate(sub, sol, 1))

# ---------------------------------------------------------------------------
print("\n== Long-tailed training set and the nearest-class-mean classifier")
rng = np.random.default_rng(0)
alpha = np.log(100) / np.log(C)
n_train = np.round(300 * np.arange(1, C + 1) ** (-alpha)).astype(int)[rng.permutation(C)]
Ftr, ytr = sample(n_train, rng)
means = np.stack([Ftr[ytr == c].mean(0) for c in range(C)])
print(f"training images {n_train.sum()}, per class max {n_train.max()} min {n_train.min()}, "
      f"imbalance {n_train.max() / n_train.min():.0f}")

M = 500
rng = np.random.default_rng(1)
Fr, yr = sample(np.full(C, M), rng)
rr = rank_classes(Fr, means)
KS = (1, 3, 5)
p_ref = {k: (rr[:, :k] == yr[:, None]).any(1).reshape(C, M).mean(1) for k in KS}
err = rr[:, 0] != yr
print(f"reference set: {M} images per class; mean per-class accuracy top-1 {p_ref[1].mean():.4f}, "
      f"top-3 {p_ref[3].mean():.4f}, top-5 {p_ref[5].mean():.4f}")
print(f"top-1 errors whose guess is in the true genus: {(rr[err, 0] // S == yr[err] // S).mean():.3f}")

# ---------------------------------------------------------------------------
designs = {"2017-style (follows training)": np.maximum(1, np.round(n_train * 182707 / 579184)).astype(int),
           "2018-style (3 per class)": np.full(C, 3),
           "2021-style (10 per class)": np.full(C, 10)}
R = 20
res = {}
max_diff = 0.0
for name, n_test in designs.items():
    for r in range(R):
        rng = np.random.default_rng(1000 + r)
        Fte, yte = sample(n_test, rng)
        rte = rank_classes(Fte, means)
        public = rng.random(len(yte)) < 0.5
        sub, sol = write_csvs("t", np.arange(len(yte)), rte, yte, public)
        for k in KS:
            off = evaluate(sub, sol, k)
            ours = our_top_k(rte, yte, public, k)
            max_diff = max(max_diff, abs(off[0] - ours[0]), abs(off[1] - ours[1]))
            hit = (rte[:, :k] == yte[:, None]).any(1)
            a_c = np.bincount(yte, hit, minlength=C) / n_test
            res.setdefault((name, k), []).append((off, hit.mean(), a_c.mean(), a_c))
print(f"\nofficial scorer against our reimplementation, {len(designs)} designs x {R} draws x k in {KS}: "
      f"largest |difference| {max_diff:.1e}")

print(f"\n== Draw 0 of each design: official public | private, all images, mean per-class accuracy")
for name, n_test in designs.items():
    print(f"{name}: {n_test.sum()} test images, {n_test.min()}..{n_test.max()} per class")
    for k in KS:
        (pu, pr), per_img, per_cls, _ = res[(name, k)][0]
        print(f"   top-{k}: public {pu:.4f} | private {pr:.4f}   all {per_img:.4f}   per-class {per_cls:.4f}")

print(f"\n== Spread of measured per-class accuracy against the binomial expectation ({R} draws)")
print("rho = sum_c (a_c - p_c)^2 / sum_c p_c(1-p_c)*M/(M-1)*(1/n_c + 1/M); "
      "share = mean_c p_c(1-p_c)*M/(M-1)/n_c / var_c(a_c)")
for name, n_test in designs.items():
    for k in KS:
        p = p_ref[k]
        expected = (p * (1 - p) * M / (M - 1) * (1 / n_test + 1 / M)).sum()
        noise = (p * (1 - p) * M / (M - 1) / n_test).mean()
        A = np.array([x[3] for x in res[(name, k)]])
        rho = ((A - p) ** 2).sum(1) / expected
        share = noise / A.var(axis=1)
        print(f"{name:30s} top-{k}: rho mean {rho.mean():.3f} (range {rho.min():.3f}-{rho.max():.3f})   "
              f"share mean {share.mean():.3f} (range {share.min():.3f}-{share.max():.3f})")

Output:

== Checks of the scorer port and of the data
official compute_top_k_accuracy unit-test cases passed: 6 of 6
duplicates in a row, k=3 (rows '7 7 7', '1 1 7', '1 1 1', '1 2 3'): (1.0, 0.0)
a row with 2 labels when k=3: ValueError
2021 README headers 'Id,Predicted': KeyError
image 1 submitted twice (right, then wrong), k=1: (0.3333333333333333, 0.5)
image 4 missing from the submission: ValueError
an extra image 9 not in the solution, k=1: (0.5, 1.0)
classes 200, distinct generating parameters 200
noise-free base check, official top-1 (public, private): (1.0, 1.0)

== Long-tailed training set and the nearest-class-mean classifier
training images 2461, per class max 300 min 3, imbalance 100
reference set: 500 images per class; mean per-class accuracy top-1 0.5777, top-3 0.8269, top-5 0.8895
top-1 errors whose guess is in the true genus: 0.678

official scorer against our reimplementation, 3 designs x 20 draws x k in (1, 3, 5): largest |difference| 0.0e+00

== Draw 0 of each design: official public | private, all images, mean per-class accuracy
2017-style (follows training): 778 test images, 1..95 per class
   top-1: public 0.6458 | private 0.6650   all 0.6555   per-class 0.5386
   top-3: public 0.8906 | private 0.9061   all 0.8985   per-class 0.8318
   top-5: public 0.9427 | private 0.9569   all 0.9499   per-class 0.9215
2018-style (3 per class): 600 test images, 3..3 per class
   top-1: public 0.5406 | private 0.5964   all 0.5667   per-class 0.5667
   top-3: public 0.7844 | private 0.8357   all 0.8083   per-class 0.8083
   top-5: public 0.8750 | private 0.9036   all 0.8883   per-class 0.8883
2021-style (10 per class): 2000 test images, 10..10 per class
   top-1: public 0.5692 | private 0.5881   all 0.5785   per-class 0.5785
   top-3: public 0.8113 | private 0.8279   all 0.8195   per-class 0.8195
   top-5: public 0.8715 | private 0.8887   all 0.8800   per-class 0.8800

== Spread of measured per-class accuracy against the binomial expectation (20 draws)
rho = sum_c (a_c - p_c)^2 / sum_c p_c(1-p_c)*M/(M-1)*(1/n_c + 1/M); share = mean_c p_c(1-p_c)*M/(M-1)/n_c / var_c(a_c)
2017-style (follows training)  top-1: rho mean 1.001 (range 0.904-1.102)   share mean 0.805 (range 0.729-0.875)
2017-style (follows training)  top-3: rho mean 1.007 (range 0.787-1.218)   share mean 0.820 (range 0.636-1.024)
2017-style (follows training)  top-5: rho mean 1.017 (range 0.776-1.403)   share mean 0.852 (range 0.554-1.178)
2018-style (3 per class)       top-1: rho mean 0.987 (range 0.838-1.144)   share mean 0.706 (range 0.614-0.828)
2018-style (3 per class)       top-3: rho mean 1.005 (range 0.773-1.299)   share mean 0.698 (range 0.578-0.965)
2018-style (3 per class)       top-5: rho mean 0.983 (range 0.734-1.182)   share mean 0.703 (range 0.549-0.864)
2021-style (10 per class)      top-1: rho mean 1.016 (range 0.787-1.215)   share mean 0.392 (range 0.340-0.454)
2021-style (10 per class)      top-3: rho mean 0.997 (range 0.757-1.311)   share mean 0.402 (range 0.304-0.488)
2021-style (10 per class)      top-5: rho mean 0.987 (range 0.710-1.269)   share mean 0.396 (range 0.285-0.561)

The scorer. The repository’s unit-test cases pass, and every edge case came out as read from the code (all checks held). The duplicate-label row 1 1 7 counts as correct at $k = 3$, and the row 1 1 1 as wrong. A submitted image that appears twice is scored twice: image 1, right then wrong, makes public accuracy 1/3 instead of 1/2. The 2021 README’s capitalized headers raise a KeyError. Over 3 designs, 20 draws, three values of $k$ and both halves, the port and the reimplementation never differ, so H1 held. The noise-free base check scores 1.0 on both halves, and it held.

The classifier. Reference top-1 accuracy averages 0.578 over classes, and 68% of top-1 errors name a species of the true genus. Top-3 reaches 0.827; since about two thirds of top-1 errors name a sister species, in our reading top-3 recovers many of them. Both learnability checks held.

Per-image against per-class (printed column only). On the 2017-style draw, top-1 accuracy over all images is 0.656 but the mean over classes is 0.539. Common classes supply most test images, and in our reading their class means, estimated from many training images, classify better. On the 3- and 10-per-class designs the two columns are identical, as they must be, and close to the reference 0.578. The public and private halves of one draw differ by up to 0.056 at 3 per class (0.5406 against 0.5964 at top-1), with roughly half of the 600 images in each. Class Imbalance and Accuracy discuss which average to report.

How much of the per-class spread is noise.

• H2 held in all nine conditions. $\rho$ averages 0.983 to 1.017, each inside its interval, so the measured per-class accuracies scatter around the reference values as the binomial model predicts.
• H3a held. With 3 images per class the share is 0.706, 0.698 and 0.703 at $k = 1, 3, 5$, and every draw’s value is above one half (the lowest is 0.549). About 70% of the variation among measured per-class accuracies is test-set noise, and about 30% is real difference between classes.
• H3b held. At 10 per class the share falls to 0.392, 0.402 and 0.396 at $k = 1, 3, 5$, still about two fifths.
• H3c held on the means: 0.805 > 0.706, 0.820 > 0.698 and 0.852 > 0.703. The per-draw ranges overlap at every $k$ (0.729–0.875 against 0.614–0.828 at $k = 1$, and 0.554–1.178 against 0.549–0.864 at $k = 5$). In our reading, the 2017-style design has the largest share because its tail classes get a single test image, so their accuracy is 0 or 1. In our reading, the shares above 1 in single draws arise because the variance across classes is itself estimated from one draw.

In our reading, this bears on per-class plots such as the paper’s Fig. 5. That figure shows top-1 public-test accuracy per class against the number of training images, with a wide spread among classes with similar amounts of training data. Some of such a spread can come from how few test images a class has. The paper does not give per-class test counts, so we cannot say how much. The general problem of test-set size is the subject of statistical significance in evaluation (intervals for one accuracy) and benchmarking (uncertainty in leaderboard ranks). The point here is narrower: an edition’s evaluation design fixes how many images each class gets, and with 3 per class, per-class accuracies are mostly noise.

License

The repository’s READMEs give the dataset’s terms of use (as of November 2026). Each begins “By downloading this dataset you agree to the following terms”, and the five terms are:

1. abide by the iNaturalist Terms of Service;
2. use the data “only for non-commercial research and educational purposes”;
3. not distribute the images;
4. the institution makes no representations or warranties about the data;
5. the user accepts full responsibility and indemnifies the institution against claims arising from their use, including claims over copies of copyrighted images.

The institution is the California Institute of Technology in the 2017, 2018 and 2019 terms and Cornell University in the 2021 terms. The READMEs name no licence for the dataset as a whole. Each image record has its own license and rights_holder fields, whose values we did not read, because reading them would mean downloading the annotation files. As of November 2026 the READMEs link the files through the AWS Open Data Program as freely available downloads (update of 15 February 2021). The repository’s code, including the scorer, is under the MIT License (copyright 2017 Visipedia).

Limitations and Bias

• Who photographs what. The images reflect what iNaturalist users observe and where. In our reading, the selection threshold of 20 observers per taxon favours species that many people see and photograph. Van Horn et al. (2021) suggest that their per-group accuracy differences may reflect bias in the data, for example that close-ups of insects may be easier to take than photos of amphibians, rather than intrinsic difficulty.
• Community labels. Labels are the platform’s research-grade consensus, not expert verification of each image. The 2021 README accepts noisy ground truth as “an unfortunate reality of large scale datasets” and cites it as one reason for top-$k$ metrics. Non-instance images (footprints, feces, habitat) were looked for only in the test images of three 2017 super-classes, as Splits describes.
• Ambiguity a single image cannot settle. Some species are told apart by location, date or features not visible in the photo. Top-5 and top-3 scoring in 2017–2018 partly allowed for this. From 2019 the leaderboard metric was top-1, with location and date provided in 2021.
• Super-class boxes. The 2017 boxes do not guarantee that every boxed instance belongs to the image’s species.
• Small per-class evaluation sets. With 3 validation images per class (2018, 2019), per-class accuracies are dominated by sampling noise, as the example shows.
• Overlap between editions. Species and images recur across editions without a mapping. In our reading, a model pretrained on one edition and evaluated on another may have seen some of the evaluation images.
• Split choices. The 2017 observer split keeps one observer’s photos of a taxon out of the other split, while the 2021 date split allows the same photographer on both sides. Results on the two are measured under different notions of novelty.

Common Uses

• Fine-grained and long-tailed classification: iNat2017 and iNat2018 as benchmarks for methods that handle imbalance. Class Imbalance reports class-balanced losses on the iNat2018 validation set.
• Transfer and self-supervised evaluation: iNat2018 or iNat2021 as a target for frozen features. I-JEPA reports linear probes on iNaturalist 2018 alongside CIFAR-100 and Places205.
• Pretraining: the competition rules allow weights from earlier editions to initialize models (2018, 2019 and 2021 READMEs), and Van Horn et al. (2021) compare iNat2021 pretraining with ImageNet pretraining on downstream fine-grained tasks and NeWT.
• Detection: the 2017 boxes. The LVIS article lists iNaturalist among its related datasets, which the LVIS paper contrasts with its own entry-level categories.

Historical Milestones

• 2017. iNat2017 was released on 5 April 2017. The Kaggle submission server opened on 1 June, the deadline was 7 July, and winners were announced on 21 July at FGVC4, with Google as sponsor (2017 README). In the CVPR 2018 paper, the best baseline was an Inception-ResNet-v2 with squeeze-and-excitation blocks, fine-tuned from ImageNet at 299 × 299 with one centre crop. It reached 67.7% top-1 and 87.9% top-5 on the private test set (Table 3). The paper sets this against the same model’s 80.2% and 95.21% on the ILSVRC 2012 validation set. Per super-class on the public test set, reptiles were hardest (45.9% top-1, averaged over species) and protozoa easiest (89.2%; Table 4). Faster R-CNN with an Inception V3 backbone reached 43.5 AP over the 2,854 boxed classes on single-instance validation images (Table 5).
• 2018. iNat2018 (FGVC5) introduced all-species categories, obfuscated names, top-3 scoring and three validation images per class.
• 2019. iNat2019 (FGVC6) narrowed the problem to 72 genera of at least 10 species each and moved to top-1.
• 2021. On 15 February 2021 the 2017–2019 editions became freely downloadable through the AWS Open Data Program; earlier, from August 2020, the files had been in a “requester pays” bucket. iNat2021 (FGVC8) added the date split, the mini training set, immediate species names and per-image location and date. Van Horn et al. (2021, Table 2) report test top-1 accuracy for a ResNet-50 initialized from ImageNet of 0.760 when trained on the full training set and 0.654 when trained on mini, and 0.746 and 0.616 from random initialization. The corresponding validation figures for the ImageNet-initialized models are 76.4% and 65.8%.

Related Datasets

• CUB-200-2011 and NABirds: smaller fine-grained bird datasets that Van Horn et al. (2018, Table 1) compare with iNat2017. NABirds has 555 classes.
• ImageNet: the general benchmark whose metric iNaturalist adapted. The 2018 paper notes that ImageNet contains about 60 bird species and about 120 dog breeds.
• NeWT: 164 binary natural-world tasks released with iNat2021 for evaluating representations (Van Horn et al., 2021).
• LVIS: a long-tailed detection and segmentation dataset of everyday categories.

Resources

• The inat_comp repository, with each edition’s README (details, data links, terms) and the eval scorer.
• The Kaggle competition pages linked from the READMEs, where the leaderboards were hosted: 2017, 2018, 2019, 2021.
• The papers: iNat2017 and iNat2021 with NeWT on CVF Open Access.