Computer Vision in Agriculture

Computer vision in agriculture measures crops and fields from images: how much of a field is covered by plants and how that changes through a season, which crop grows on which parcel, how much grain a region will harvest, where weeds stand among the crop rows, which leaves show disease, and how many fruits or wheat heads a plot carries. The images range from satellite pixels tens to hundreds of metres across, through aerial images at 10 cm, to photographs taken a metre or two above the canopy in which a wheat head spans tens of pixels. The vision tasks are the familiar ones, semantic segmentation, object detection, classification and counting, but the data make them hard in a specific way. Fields, regions and seasons differ in soil, light, crop variety and growth stage, so a model that scores well on images from the fields it was trained on can fail on the next farm or the next year. How a dataset is split, and what that split measures, is therefore as important here as the choice of model.

The sensors and spectral bands behind satellite crop monitoring, including vegetation indices beyond NDVI, are described under multispectral satellite imaging and, in outline, remote sensing. Land cover classification covers mapping classes such as cropland, the spatial autocorrelation that inflates random splits, and design-based accuracy assessment; change detection covers comparing dates. This article treats what is specific to agriculture: the tasks from field to plant, the datasets and the protocols that define their splits, results with their designs, and the shifts between fields and seasons. Drone surveys, planned for ground sampling distance and overlap and stitched into orthomosaics as described under aerial and drone image analysis, supply many of the field-scale images, and robots that weed or harvest use the plant-scale detectors described here.

Problem

The questions differ with the user and the scale.

• Agencies and markets want crop areas and yields over regions before harvest. You et al. (2017) call yield estimation, predicting crop yields before harvest, a central challenge of agricultural monitoring, and note that the weather, soil and land cover data that support it in the United States are often missing in developing countries, while satellite images are available everywhere. In the European Union, farmers declare their crops to receive subsidies, and national agencies check the declarations with field samples or with Earth observation (Rußwurm et al., 2020).
• Growers want to know where something is wrong in a field while there is still time to act. Chiu et al. (2020) motivate their aerial dataset with field conditions such as weed clusters, nutrient deficiency and planter skips that affect the final yield and that timely detection could address.
• Weed control wants each plant located and named. Steininger et al. (2023) contrast applying herbicide to a whole field with treating single plants, or removing weeds mechanically, once crops and weeds can be located and classified.
• Disease diagnosis wants a name for a symptom. Mohanty et al. (2016) argue that the spread of smartphones with good cameras could make image-based diagnosis available on a large scale, and point to smallholder farmers as especially vulnerable to crop disease.
• Breeders want plant traits measured on many plots. Wheat head density is a major yield component that is still counted by hand in breeding trials, with measurement errors around 10% according to the work David et al. (2020) cite; their dataset was built to automate the count.

Visual Data

Scale	Platform and sensor	Pixel size	Example in this article
Region and county	MODIS reflectance and land surface temperature, images every 8 days	500 m crop mask	County yield prediction (You et al., 2017)
Parcel	Sentinel-2 time series, 13 bands (L1C) or 10 bands (L2A), averaged per parcel	10–60 m	BreizhCrops (Rußwurm et al., 2020)
Field	Aircraft-mounted cameras, near infrared, red, green and blue	10, 15 or 20 cm	Agriculture-Vision (Chiu et al., 2020)
Plot and plant	Handheld, cart, mini-vehicle and gantry cameras, RGB, nadir	0.10–0.62 mm	GWHD (David et al., 2020)
Plant	Full-frame SLR, manually captured, 50 mm lens, about 1.1 m above the ground	—	CropAndWeed (Steininger et al., 2023)
Leaf	Single leaves photographed under controlled conditions	—	PlantVillage (Mohanty et al., 2016)

Satellites. Satellite crop monitoring relies on the near-infrared and red contrast of green vegetation and on revisits through the season; the multispectral satellite imaging article covers both. Two practical points recur. A pixel often mixes crop with other cover: You et al. (2017) mask non-cropland with a global crop mask at 500 m but note that it does not say which pixels hold the target crop. And optical time series are interrupted by clouds. BreizhCrops keeps all Sentinel-2 Level-1C acquisitions of 2017 regardless of cloud cover, 51 or 102 observations per parcel, so the models must cope with cloudy outliers, while its Level-2A version keeps only images under 80% cloud cover, atmospherically corrected (Rußwurm et al., 2020). Synthetic aperture radar images through clouds and can fill such gaps.

Aircraft and drones. Agriculture-Vision’s images were taken from aerial vehicles over corn and soybean fields, mainly around Illinois and Iowa, in the growing seasons from 2017 to 2019 (Chiu et al., 2020). The cameras changed every year: two aligned Canon SLRs at 15 cm in 2017, two Nikon D850s at 10 cm with custom filters for near infrared, plus a separate 20 cm blue camera, in 2018, and a proprietary four-band Wide Area Multi-Spectral System at 10 cm in 2019. Whole-field images are large (the largest is 33,571 × 24,351 pixels), and the authors normalize each channel per field image by clipping at bounds set from its 5th and 95th percentiles. Their results also show what the near-infrared channel adds: with a ResNet-101 backbone, test mean intersection over union (mIoU) rose from 39.63 with RGB to 43.66 with near infrared, red, green and blue (their Table 5).

Ground platforms. Close-range images resolve organs. The GWHD sub-datasets were taken from 1.8 to 3 m above the ground, looking straight down, at ground sampling distances from 0.10 to 0.62 mm; heads about 1.5 cm across and 10 cm long are then about 25 to 150 pixels wide (David et al., 2020; the pixel range is our arithmetic). At this scale the geometry matters for counting: David et al. compute that with a ±15° field of view at 2.5 m from the canopy, an error of 10 cm in canopy height changes the sampled area by 8%, and that the top and bottom of a 25 cm thick head layer differ by 21% in sampled area. A density estimate inherits these errors whatever the detector does.

Controlled images. Leaf images on uniform backgrounds are easy to collect and label but differ from field images in background, lighting and viewpoint, which is the subject of the lab-to-field gap below.

Vision Tasks Involved

Crop monitoring and crop type mapping

From satellites, crop monitoring is mostly classification and regression on time series. Crop type mapping assigns a crop to each parcel from how its spectra develop through the season; it is a form of land cover classification, and the parcel-level time-series models described there (pixel-set encoders with temporal attention) apply directly. BreizhCrops frames the task on 768,175 parcels in Brittany with nine classes (barley, wheat, rapeseed, corn, sunflower, orchards, nuts, permanent meadows and temporary meadows) taken from the French Agricultural Land Parcel Information System (Registre Parcellaire Graphique) for 2017, where the labels come from the farmers’ own declarations (Rußwurm et al., 2020). Each Sentinel-2 band is averaged over the parcel’s polygon, so the input is one multivariate time series per parcel rather than an image.

From the air, monitoring becomes segmentation of patterns within a field. Agriculture-Vision annotates nine field anomaly patterns: double plant, drydown, endrow, nutrient deficiency, planter skip, storm damage, water, waterway and weed cluster (Chiu et al., 2020). They are regions rather than objects. Weed clusters range from small patches to large areas, and storm damage was so rare that it is left out of evaluation.

Yield estimation

Yield is usually measured as an average over an area: a county, a field or a plot. You et al. (2017) predict county-level soybean yields, in bushels per acre from the USDA, from MODIS images taken every 8 days from day 49 to day 281 of the year. With fewer than 10,000 labelled examples, they argue that end-to-end training on the images is not feasible and that ImageNet pretraining does not fit multispectral overhead images. They therefore assume that yield does not depend on where in the county a pixel lies, reduce each band of each image to a 32-bin histogram of its cropland pixels, and train a CNN or an LSTM on the stack of histograms over time. A Gaussian process on top models what the network misses and what is correlated in space and time, such as soil. Their error maps show that neighbouring counties have similar errors, which the Gaussian process reduces.

Their design trains on all years before the test year and tests on each year from 2011 to 2015, a season-level split. Averaged over the five years, the CNN with the Gaussian process reached an RMSE of 5.55 bushels per acre, against 7.73 for the best baseline, a decision tree on average NDVI sequences, and 7.96 for ridge regression (their Table 1). The text describes this as a 30% reduction; the table’s averages give 28% (our arithmetic). They also predicted each month from the data available so far, and all models were poor in the early months, before the images contain much information on plant growth.

At plot and plant scale, yield is approached through its components: heads per square metre in wheat, fruits per tree in orchards. These become detection and counting problems.

Weed detection

Weed detection has two steps that are often separated: find the vegetation, then tell crop from weed. The first step is old. Woebbecke et al. (1995) digitized colour slides of weeds on different soils and residues and computed chromatic coordinates; their abstract does not spell out the definition, and the usual one divides each colour channel by the sum of the three,

$$r = \frac{R}{R+G+B}, \qquad g = \frac{G}{R+G+B}, \qquad b = \frac{B}{R+G+B}.$$

Among the indices they tested, the modified hue, the green coordinate and $2g - r - b$ (written ExG, for excess green, in this article) separated weeds from a non-plant background better than the others, in both shaded and sunlit conditions, and the modified hue was the most expensive to compute (abstract of Woebbecke et al., 1995). Because the coordinates are ratios, a change in overall brightness cancels; a change in the colour of the light does not. Agriculture-Vision’s authors list an excess green index and NDVI as popular indices for agricultural images (Chiu et al., 2020), and CropAndWeed’s annotation pipeline first split each image into soil and vegetation by colour thresholding with Otsu’s method (see thresholding) before CNN-based segmentation took over (Steininger et al., 2023).

The second step needs shape, texture and context as well as colour, because crops and weeds are both green. CropAndWeed’s semantic segmentation benchmark shows the difference between the steps (Steininger et al., 2023, Table 5): separating vegetation from soil gave a test mIoU of 87.7, while the two-class crop-or-weed variant gave IoUs of 70.0 for crop, 27.1 for weed and 99.5 for soil (mIoU 65.5). Detection with YOLOv5 reached an average precision (AP) of 71.5 averaged over IoU thresholds from 0.5 to 0.95 when all weeds were merged into one class alongside eight crop classes, and 55.4 with 16 separate weed classes (their Table 3). The authors note that weeds tend to be more frequent but smaller than crops, which makes them the hardest class to annotate and to learn. They also found that a detector for a single crop trained without other plants produced many false positives on the test images, and that adding the other plants as classes improved crop detection.

Disease detection

Mohanty et al. (2016) framed disease detection as image classification of leaf photographs. They trained AlexNet and GoogLeNet, from scratch and by transfer learning from ImageNet, on 54,306 PlantVillage images of leaves of 14 crops, labelled with 38 crop–disease pairs (26 diseases and healthy leaves). The leaves were photographed under controlled conditions, and the authors tested whether the networks learned the backgrounds and lighting by also training on grey-scale images and on images with the background removed; the colour images gave the best results and the segmented ones were close behind. The best model reached 99.35% accuracy on held-out images, and the authors also tested it on images taken elsewhere, with the result discussed under evaluation below. They limit their claim to single leaves, facing up, on a uniform background, and call for new images taken from many angles in settings as realistic as possible, since many diseases do not show on the upper side of leaves.

Fruit and organ detection and counting

Plant organs such as wheat heads, maize tassels, fruits or flowers can be counted by detecting each one and counting the boxes, or by regressing a count or density directly. GWHD labels each visible wheat head with a box (David et al., 2020). The rules decide what counts: partly hidden heads were labelled only if at least one spikelet was visible, heads cut by the image border only if more than 30% of their basal part was visible, and boxes excluded the awns when including them would make the box too large. To label faster, a YOLOv3 model trained on two labelled sub-datasets proposed boxes with confidence above 0.5 for correction, which the authors report increased labelling throughput by a factor of four, and each sub-dataset was then reviewed by two operators from other institutions. The baseline, a Faster R-CNN with a ResNet-34 backbone, reached an mAP at IoU 0.5 of 0.68 on the test continents and a count RMSE of 13.75 heads per image, a relative RMSE of 34% (preprint version).

For orchard fruit, the MinneApple repository sets up separate benchmarks for apple detection, segmentation and counting with evaluation servers. Its README lists, as of November 2019, an AP of 0.438 (IoU 0.50:0.05:0.95) for Faster R-CNN and 0.433 for Mask R-CNN, both with ResNet-50, and a mean counting accuracy of 0.908 for a CNN against 0.816 for a Gaussian mixture model. Separating fruit that touch or overlap is an instance segmentation problem.

Phenotyping in outline

Plant phenotyping measures the traits of many genotypes to support breeding; for wheat heads, David et al. (2020) name density, size, inclination, colour, the presence of awns and sanitary state among them. Image-based phenotyping turns such traits into vision tasks: counting (heads, leaves, plants), segmentation (leaves, canopy cover), detection and tracking over time. GWHD was built for one such trait. Its authors list earlier phenotyping datasets, including the CVPPP datasets for rosette leaf counting and segmentation, and note that few open datasets come from outdoor fields, which practical phenotyping needs (David et al., 2020). Their recommendations for future datasets are as much about acquisition as about labels: image right after flowering, when heads have fully emerged and are still upright; look near nadir with a narrow field of view to limit overlap; sample at least about 1 m² per plot for density estimates; and attach a minimum set of metadata to every session and image, including the camera, its height, the development stage, the genotype and the sowing density.

Common Methods

• Indices and thresholds. ExG on chromatic coordinates, NDVI on near infrared and red, and their variants, thresholded with a fixed value or with Otsu’s method per image, are used for vegetation masks and canopy cover and in annotation tools: CropAndWeed’s first pre-segmentation was a colour threshold, and Agriculture-Vision’s annotation software displayed NDVI alongside RGB and near infrared (Chiu et al., 2020).
• Segmentation networks. Agriculture-Vision’s baselines were DeepLab v3 and v3+ and the authors’ own feature pyramid network model; to feed four-channel images to an ImageNet-pretrained backbone, they copied the first layer’s red-channel weights for the extra channel (Chiu et al., 2020). On the test set their model reached an mIoU of 43.66, against 42.42 for DeepLab v3+ (output stride 16), with IoUs from 24.43 for endrow to 72.55 for background and 30.87 for weed clusters (their Table 4).
• Detectors. Two-stage detectors such as Faster R-CNN served as baselines for wheat heads in GWHD 2020 and the 2021 challenge and for apples in MinneApple; CropAndWeed chose YOLOv5 over RetinaNet after comparing them (Steininger et al., 2023).
• Time-series models. For parcels, BreizhCrops compares a random forest with seven deep models: TempCNN, MS-ResNet, InceptionTime, OmniscaleCNN, LSTM, StarRNN and a transformer (Rußwurm et al., 2020). For yields, You et al. (2017) use histograms of each band over time as the input to CNNs and LSTMs.
• Transfer learning. Pretrained backbones are used even when the input differs from photographs: Mohanty et al. (2016) found transfer learning better than training from scratch in every configuration, and Chiu et al. (2020) call the transfer from RGB photographs of common objects to multispectral farmland images an uncommon but effective one.
• Model-assisted labelling. Labelling is slow, and GWHD’s authors note that usually only a small fraction of the acquired images is labelled (David et al., 2020). Datasets therefore bootstrap their labels from models: GWHD with YOLOv3 proposals, CropAndWeed with threshold and then CNN pre-segmentation corrected by hand.

Example: A Vegetation Index and a Crop/Weed Classifier Across Fields and Seasons

The script builds synthetic top-down images of four fields in two seasons, six images each, at 96 × 96 pixels. Each image has soil, crop plants in rows and weeds scattered between them, with a known label for every pixel. The things that vary are the things that vary on real farms: each field has its own soil colour, its own crop variety (a small colour offset kept in both seasons), and its own weed mix in each season (another colour offset); the second season has larger crop plants and yellower light; and each image has its own exposure (a gain between 0.6 and 1.4). The class colours, offsets and sizes are illustrative values, not measurements. Weeds are drawn smaller than crops, as Steininger et al. (2023) found them to be.

Part A segments vegetation in the images of scene seed 0 in two ways: ExG on chromatic coordinates with an Otsu threshold for each image, and the same index on raw values ($2G - R - B$) with one threshold taken from the first image and applied everywhere. It prints the true and estimated canopy cover and the intersection over union of each mask with the truth. Part B trains a random forest to label each vegetation pixel as crop or weed from six features (the chromatic red and green coordinates, ExG, brightness, the distance to the nearest soil pixel, and the local standard deviation of the green coordinate), using the true vegetation mask so that the classifier is judged on its own. It evaluates four splits, pooling predictions over their folds: pixels assigned to four folds at random, whole images assigned to four folds at random, leave one field out (both seasons of a field held out together), and leave one season out. It reports pixel accuracy and the IoU of the weed class for scene seeds 0 to 4. The splits and parameters were fixed in a plan before the first run. It runs in about 50 seconds with NumPy 2.3.5, SciPy 1.16.3, scikit-image 0.25.2 and scikit-learn 1.7.2, and two runs gave identical output.

import numpy as np
from scipy import ndimage
from skimage.filters import threshold_otsu
from sklearn.ensemble import RandomForestClassifier

H = W = 96
SOIL, CROP, WEED = 0, 1, 2
TINT = {0: np.array([1.0, 1.0, 1.0]), 1: np.array([1.08, 1.0, 0.88])}  # season light

def leaf_plant(lab, rgb, cy, cx, n, lmin, lmax, cls, colour, rng):
    yy, xx = np.mgrid[0:H, 0:W]
    for a in rng.uniform(0, 2 * np.pi, n):
        L = rng.uniform(lmin, lmax)
        my, mx = cy + 0.5 * L * np.sin(a), cx + 0.5 * L * np.cos(a)
        u = (yy - my) * np.sin(a) + (xx - mx) * np.cos(a)      # along the leaf
        v = (yy - my) * np.cos(a) - (xx - mx) * np.sin(a)      # across it
        m = (u / (0.5 * L)) ** 2 + (v / (0.2 * L)) ** 2 <= 1
        lab[m] = cls
        rgb[m] = colour * rng.uniform(0.85, 1.15)              # per-leaf shade

def make_field_data(seed, n_fields=4, n_img=6):
    rng = np.random.default_rng(seed)
    soil_f = np.array([0.40, 0.30, 0.22]) + rng.normal(0, 0.04, (n_fields, 3))
    crop_f = np.array([0.20, 0.42, 0.12]) + rng.normal(0, 0.025, (n_fields, 3))
    weed_fs = np.array([0.24, 0.44, 0.16]) + rng.normal(0, 0.025, (n_fields, 2, 3))
    images = []
    for f in range(n_fields):
        for s in (0, 1):
            for _ in range(n_img):
                tex = 1 + 0.08 * ndimage.gaussian_filter(rng.normal(size=(H, W)), 2) / 0.14
                rgb = soil_f[f] * tex[..., None]
                lab = np.zeros((H, W), int)
                for _ in range(rng.integers(15, 41)):          # weeds first
                    leaf_plant(lab, rgb, rng.uniform(0, H), rng.uniform(0, W),
                               rng.integers(3, 9), 2, 5, WEED, weed_fs[f, s], rng)
                lmin, lmax = (6, 9) if s == 0 else (9, 13)       # growth stage
                off = rng.uniform(0, 32)
                for cx in np.arange(off, W, 32):                 # crop rows
                    for cy in np.arange(rng.uniform(0, 20), H, 20):
                        leaf_plant(lab, rgb, cy + rng.normal(0, 2), cx + rng.normal(0, 1.5),
                                   rng.integers(4, 7), lmin, lmax, CROP, crop_f[f], rng)
                gain = rng.uniform(0.6, 1.4)
                img = np.clip(rgb * gain * TINT[s] + rng.normal(0, 0.01, (H, W, 3)), 0, 1)
                images.append(dict(img=img, lab=lab, field=f, season=s))
    return images

def exg(img):
    c = img / np.maximum(img.sum(axis=-1, keepdims=True), 1e-6)   # chromatic coords
    return 2 * c[..., 1] - c[..., 0] - c[..., 2]

def iou(a, b):
    return (a & b).sum() / max((a | b).sum(), 1)

def features(d):
    img, veg = d["img"], d["lab"] > 0
    c = img / np.maximum(img.sum(axis=-1, keepdims=True), 1e-6)
    g = c[..., 1]
    loc_std = np.sqrt(np.maximum(ndimage.uniform_filter(g ** 2, 5) - ndimage.uniform_filter(g, 5) ** 2, 0))
    dist = ndimage.distance_transform_edt(veg)
    F = np.stack([c[..., 0], g, exg(img), img.sum(axis=-1), dist, loc_std], axis=-1)
    return F[veg], (d["lab"][veg] == WEED).astype(int)

def evaluate(X, y, test_sets, rng, n_tr=15000, n_te=20000):
    tp = fp = fn = correct = total = 0
    for te in test_sets:
        tr = np.flatnonzero(~te)
        te = np.flatnonzero(te)
        tr = rng.choice(tr, min(n_tr, tr.size), replace=False)
        te = rng.choice(te, min(n_te, te.size), replace=False)
        rf = RandomForestClassifier(n_estimators=60, min_samples_leaf=5, random_state=0, n_jobs=1)
        p = rf.fit(X[tr], y[tr]).predict(X[te])
        t = y[te]
        tp += ((p == 1) & (t == 1)).sum(); fp += ((p == 1) & (t == 0)).sum()
        fn += ((p == 0) & (t == 1)).sum(); correct += (p == t).sum(); total += t.size
    return correct / total, tp / (tp + fp + fn)

# Part A: vegetation index and threshold on seed 0
data = make_field_data(0)
first = data[0]["img"]
raw_thr = threshold_otsu(2 * first[..., 1] - first[..., 0] - first[..., 2])
print(f"fixed raw 2G-R-B threshold (Otsu on field 0, season 1, image 1): {raw_thr:.3f}")
print("field season  true cover  ExG+Otsu cover  IoU ExG+Otsu  IoU raw 2G-R-B (fixed thr)  soil raw 2G-R-B")
for f in range(4):
    for s in (0, 1):
        ds = [d for d in data if d["field"] == f and d["season"] == s]
        true = np.concatenate([(d["lab"] > 0).ravel() for d in ds])
        e = np.concatenate([(exg(d["img"]) > threshold_otsu(exg(d["img"]))).ravel() for d in ds])
        r = np.concatenate([((2 * d["img"][..., 1] - d["img"][..., 0] - d["img"][..., 2]) > raw_thr).ravel()
                            for d in ds])
        soil = np.concatenate([(2 * d["img"][..., 1] - d["img"][..., 0] - d["img"][..., 2])[d["lab"] == SOIL]
                               for d in ds])
        print(f"{f:5d} {s + 1:6d}  {true.mean():10.3f}  {e.mean():14.3f}  {iou(true, e):12.3f}  "
              f"{iou(true, r):26.3f}  {soil.mean():15.3f}")

# Part B: crop/weed pixel classifier under four splits, scene seeds 0-4
print("\nseed  pixel acc / weed IoU:  random pixels   random images   leave-field-out   leave-season-out")
res = []
for seed in range(5):
    data = make_field_data(seed)
    Xs, ys, img_id, field, season = [], [], [], [], []
    for k, d in enumerate(data):
        X, y = features(d)
        Xs.append(X); ys.append(y)
        img_id.append(np.full(y.size, k)); field.append(np.full(y.size, d["field"]))
        season.append(np.full(y.size, d["season"]))
    X, y = np.concatenate(Xs), np.concatenate(ys)
    img_id, field, season = map(np.concatenate, (img_id, field, season))
    rng = np.random.default_rng(100 + seed)
    pix_fold = rng.integers(0, 4, y.size)
    img_fold = rng.permutation(np.arange(len(data)) % 4)[img_id]
    splits = {"pix": [pix_fold == k for k in range(4)],
              "img": [img_fold == k for k in range(4)],
              "field": [field == k for k in range(4)],
              "season": [season == k for k in range(2)]}
    row = [evaluate(X, y, splits[k], rng) for k in ("pix", "img", "field", "season")]
    res.append(row)
    print(f"{seed:4d}  " + "   ".join(f"{a:.3f} / {w:.3f}" for a, w in row))
res = np.array(res)
print("range " + "   ".join(f"{res[:, j, 0].min():.3f}-{res[:, j, 0].max():.3f} / {res[:, j, 1].min():.3f}-{res[:, j, 1].max():.3f}"
                           for j in range(4)))

fixed raw 2G-R-B threshold (Otsu on field 0, season 1, image 1): 0.032
field season  true cover  ExG+Otsu cover  IoU ExG+Otsu  IoU raw 2G-R-B (fixed thr)  soil raw 2G-R-B
    0      1       0.130           0.130         1.000                       1.000           -0.071
    0      2       0.233           0.233         1.000                       0.999           -0.060
    1      1       0.145           0.145         0.999                       0.999           -0.079
    1      2       0.240           0.240         1.000                       0.999           -0.080
    2      1       0.139           0.139         1.000                       0.243            0.032
    2      2       0.239           0.236         0.987                       0.508            0.019
    3      1       0.147           0.147         1.000                       0.892           -0.018
    3      2       0.257           0.257         1.000                       0.955           -0.021

seed  pixel acc / weed IoU:  random pixels   random images   leave-field-out   leave-season-out
   0  0.961 / 0.815   0.935 / 0.724   0.781 / 0.416   0.917 / 0.688
   1  0.980 / 0.903   0.978 / 0.894   0.917 / 0.662   0.953 / 0.793
   2  0.932 / 0.693   0.920 / 0.650   0.747 / 0.346   0.834 / 0.423
   3  0.983 / 0.923   0.974 / 0.886   0.879 / 0.554   0.896 / 0.604
   4  0.943 / 0.743   0.921 / 0.661   0.653 / 0.273   0.850 / 0.468
range 0.932-0.983 / 0.693-0.923   0.920-0.978 / 0.650-0.894   0.653-0.917 / 0.273-0.662   0.834-0.953 / 0.423-0.793

A ratio index with a per-image threshold finds the vegetation; a fixed threshold on raw values does not. ExG with Otsu’s threshold recovers the canopy cover of every field and season, with IoUs of 0.987 to 1.000. The fixed threshold on $2G - R - B$ works in fields 0 and 1 but fails in field 2 (IoU 0.243 and 0.508) and loses some accuracy in field 3. The last column shows why. The threshold taken from the first image, 0.032, sits above the soil of field 0, whose raw index averages −0.071 in that season. The soil of field 2 has a raw index averaging 0.032 in the first season, right at the threshold, so much of it is labelled vegetation. Exposure makes it worse, since a raw difference scales with the gain while the chromatic coordinates do not. The near-perfect ExG results reflect the idealized scene, with sharp leaf edges and no shadows or crop residue; Woebbecke et al. (1995) tested their indices on several soils and residues and in shaded and sunlit conditions, which real images add. The same index also gives the canopy cover that phenotyping and monitoring use, here 13–15% in the first season and 23–26% in the second.

The split decides the score. In every scene, random pixel folds give the highest accuracy and weed IoU, random image folds slightly less, and leaving a field out the least, with weed IoU falling from 0.69–0.92 to 0.27–0.66. A pixel in a random fold has neighbours from the same plant in the training set, and an image in a random fold has sister images from the same field and season, with the same crop variety, weed mix and soil; the forest can learn each field’s colours instead of the difference between crop and weed. A held-out field brings a crop colour and a weed colour the forest has never seen, and because those offsets are as large as the difference between the classes, colour alone stops working.

Here a new field is harder than a new season, but that follows from how the scenes were made. The plan expected the season split to be the hardest; the output shows it between the image and field splits in every scene. In these scenes a field keeps its crop variety and soil from one season to the next, so a model tested on the other season has seen them, and only the weed mix, plant size and light change. Which shift is larger on real farms depends on what changes more between fields than between years there; the example shows only that the two splits answer different questions and can rank in either order (our reasoning). The spread across scenes is wide, from 0.653 to 0.917 accuracy for the field split, because each scene draws new field offsets, and four fields are few units from which to estimate generalization, a point land cover classification makes for spatial blocks.

After the first run, two printed items were added to Part A, the fixed threshold and the mean raw index of the soil pixels in each field and season, so that the explanation of the field 2 failure rests on printed numbers. No parameter, data or measurement changed, and the rest of the output is identical to the first run. The first run also showed the leave-one-field-out split below the leave-one-season-out split, against the plan’s hypothesis; the result is reported as printed.

Datasets and Protocols

Dataset	Imagery	Labels	Size	Split protocol	Metric
PlantVillage, as used by Mohanty et al. (2016)	Single leaves, controlled conditions, resized to 256 × 256	38 crop–disease classes	54,306 images	Train/test 80/20 to 20/80, images of the same leaf kept together	Mean F1, accuracy
Agriculture-Vision (2020)	Aerial NRGB, 10–20 cm, 512 × 512 crops	9 field anomaly patterns (8 evaluated)	94,986 images from 3,432 field images	Random 6/2/2 by field image	mIoU
GWHD 2020	Ground RGB, 1024 × 1024 patches, rescaled	Wheat head boxes	4,698 images, 188,445 heads (preprint)	Train Europe and North America, test Australia, Japan, China	mAP at IoU 0.5
GWHD 2021	As GWHD 2020, plus five countries	Relabelled boxes	6,515 images, 275,187 heads, 47 sub-datasets	Global Wheat Challenge 2021 and GlobalWheat-WILDS splits by sub-dataset	Weighted domain accuracy
CropAndWeed (2023)	Ground RGB, about 1.1 m height	Boxes, masks and stem points, 74 species	8,034 images, 111,953 instances	Random 70/15/15	AP (IoU 0.5–0.95), mIoU
BreizhCrops (2020)	Sentinel-2 L1C and L2A time series, parcel means	9 crop classes	768,175 parcels	Train, validation and test by department	Overall accuracy, kappa, F-score

PlantVillage. Many PlantVillage images show the same leaf from several orientations. Mohanty et al. (2016) had the mapping of such images for 41,112 of the 54,306 and kept all images of a leaf on the same side of every split; a random split by image would have put views of one leaf on both sides. They describe the images as openly and freely available through the PlantVillage project. The same-leaf rule removes one form of leakage, but all images still share the controlled set-up, so the held-out test measures performance under the training conditions.

Agriculture-Vision. The 512 × 512 images were cropped around annotations from 3,432 field images, discarding crops more than 90% annotated and, when two crops overlapped by more than 30% IoU, the one with fewer annotated pixels (Chiu et al., 2020). The split was made on the field images first, 6/2/2, and each crop was assigned to the split of its field, giving 56,944 training, 18,334 validation and 19,708 test images with no field in two splits. The annotations were made by five annotators trained by agronomists and reviewed by the agronomists. The challenge subset described on the workshop website contains 21,061 images captured in 2019, with six patterns (cloud shadow, double plant, planter skip, standing water, waterway and weed cluster), boundary maps and masks of valid pixels; access is by contacting the organizers under the dataset’s terms (as of November 2026).

GWHD. The 2020 dataset combined 11 sub-datasets from nine institutions in seven countries, rescaled so that heads had more similar sizes in pixels and cut into 1024 × 1024 patches with roughly 20 to 60 heads each (David et al., 2020, preprint). The authors state the reason for their split directly: random splits are common, but they wanted to test the method on unseen genotypes, environments and observation conditions, so all images from Europe and North America (3,422) form the training set and all images from Australia, Japan and China (1,276) the test set. They also call training on part of a phenotyping trial and testing on the rest a fundamental flaw, because nothing guarantees that the model works on new acquisitions.

The 2021 version redefined a sub-dataset as a consistent set of images from one experimental unit, one acquisition session, one platform and one sensor, and treats each as a domain (David et al., 2021). The 11 original sub-datasets became 19, and new ones, from the original institutions and from five new countries (Norway, Belgium, the United States, Mexico and Sudan), bring the total to 47, with 6,515 images and 275,187 heads in its Table 1. The paper states the additions differently in different places (1,722 images and 81,553 heads in the abstract, 1,722 images and 86,000 heads in the methods, 1,200 images and 120,000 heads in the conclusion); the table’s totals are used here. The 2021 version also responded to problems found in the 2020 Kaggle competition, which attracted 2,245 teams: label noise, an unbalanced test set, and a metric averaged over all test images, which let the largest sub-dataset dominate the score. The paper recommends two splits, both training on the European sub-datasets: the Global Wheat Challenge 2021 split for phenotyping use, and the GlobalWheat-WILDS split for studying distribution shift, which tests on the sub-datasets from Mexico, the United States, Canada and Sudan. The paper gives the licence as MIT; the Zenodo record lists CC BY 4.0 (as of November 2026).

CropAndWeed. Images were taken over four years, from March to July, at several hundred commercial fields in Austria (the Application Set) and on plots sown with chosen species, including rare weeds (the Experimental Set) (Steininger et al., 2023). Each recording session, one site at one time, yields about 20 images at least 3 m apart. The 74 classes are 16 crops and 58 weeds, and the dataset can be mapped to 20 variants, from soil against vegetation to 8 crops and 16 weed groups. Instances smaller than 16 × 16 pixels go to a fallback vegetation class and are ignored in evaluation. All experiments use one random 70/15/15 split; the paper does not say whether images of one session are kept together, and with about 20 images per session the answer affects how far the test set is independent of training (our reading). The README and licence file allow non-commercial use and forbid redistribution of the data (as of November 2026).

BreizhCrops. The parcels are partitioned by the four departments of Brittany, with Côtes-d’Armor and Finistère for training, Ille-et-Vilaine for validation and Morbihan for testing (Rußwurm et al., 2020). On Morbihan, overall accuracy on Level-1C data ranged from 0.77 (random forest and OmniscaleCNN) to 0.80 (LSTM and transformer). Rotating the held-out department (three folds; Ille-et-Vilaine, used for tuning, was not a test fold) gave mean accuracies of 0.71 to 0.80 for Level-1C and 0.73 to 0.81 for Level-2A, with standard deviations up to 0.07 (their Table 2). The authors list regional differences in soil, elevation, temperature and rainfall, and hence in which crops dominate, among the dataset’s challenges.

Evaluation

Splits by field, region and season

Agricultural images come in nested groups: plants in images, images in sessions, sessions in fields, fields in regions and seasons. Any split that puts members of one group on both sides measures performance within groups the model has seen. The published protocols differ in which group they hold out:

• Same leaf: PlantVillage as used by Mohanty et al. (2016).
• Field: Agriculture-Vision, by field image (Chiu et al., 2020).
• Region: GWHD 2020 by continent; GlobalWheat-WILDS by country; BreizhCrops by department.
• Season: You et al. (2017), training on earlier years and testing on later ones.

Which group to hold out follows from how the model will be used (our reasoning). A model for a new farm needs a field-level or region-level test; a forecast for next year needs a season-level test; a model that will only ever run on the trial fields it was trained on can be tested within them. The spatial version of the problem, autocorrelation between nearby pixels and the blocked and buffered cross-validation that addresses it, is treated under land cover classification, with Kattenborn et al.’s drone study of tree species as a measured case; grouped splits in general are under cross-validation and the train, validation and test split.

Results across domains, with their designs

• Lab to field, disease classification. The best PlantVillage models reached 99.35% accuracy on held-out PlantVillage images (Mohanty et al., 2016; abstract and discussion; their Table 1 lists 0.9935 accuracy and a mean F1 of 0.9934 for GoogLeNet with transfer learning on colour images and an 80/20 split, while the results text gives 99.34%). On two small sets of images taken under other conditions, 121 and 119 images found through Bing Image Search and IPM Images and visually verified, the best models reached 31.40% and 31.69% top-1 accuracy over the 38 classes, against 2.63% for random guessing, and the correct class was in the top five for 52.89% and 65.61%. When the crop was given, accuracy within each crop’s diseases was 41.1–54.5% against 17.9–31.4% for random guessing. The test sets are small and the models date from 2016; what carries over is the size of the gap between a test under the training conditions and one outside them (our reading).
• Across countries, wheat head detection. In WILDS, models trained on GlobalWheat-WILDS with standard empirical risk minimization reached an average domain accuracy of 48.4 on the test countries, against 64.8 when images from the test distribution were mixed into a training set of similar or smaller size while the test set was held approximately constant (Koh et al., 2021, Table 1, standard deviations 1.8 and 0.4 over replicates). The authors use this “fixed-test” comparison to show that the drop is due to the shift, not to harder test images. In the Global Wheat Challenge 2021, the winning solution reached a weighted domain accuracy of 0.700 on its test sub-datasets, against 0.492 for a Faster R-CNN reference with the parameters of the 2020 paper (David et al., 2021, Table 2).
• Across departments, crop type mapping. For the same model, accuracy on BreizhCrops’ rotated held-out departments varies with standard deviations of 0.01 to 0.07, the largest for convolutional models (Rußwurm et al., 2020).
• Across years, yield. You et al.’s yearly RMSEs vary from 4.89 to 5.83 bushels per acre for the CNN with the Gaussian process and from 7.40 to 8.13 for the decision tree (their Table 1), with each year predicted from earlier years only.

Metrics

Segmentation benchmarks report per-class IoU and its mean; detection benchmarks report average precision; counting reports RMSE, relative RMSE and $R^2$ per image or plot; yield reports RMSE and mean absolute percentage error over areas. Agriculture adds a metric designed for domains. GWHD 2021 computes, for each image $i$ of domain $d$, an accuracy from the matched heads,

$$\mathrm{AI}_d(i) = \frac{\mathrm{TP}}{\mathrm{TP} + \mathrm{FN} + \mathrm{FP}},$$

and averages it first within each domain and then over the $D$ domains,

$$\mathrm{WDA} = \frac{1}{D} \sum_{d=1}^{D} \frac{1}{n_d} \sum_{i=1}^{n_d} \mathrm{AI}_d(i),$$

where $n_d$ is the number of images in domain $d$ (David et al., 2021). Each domain counts equally, however many images it has, which addresses the 2020 competition’s problem of the largest sub-datasets dominating the mean.

Challenges

• Shift between fields, regions and seasons. Soil, light, crop variety, growth stage, sowing pattern and camera all change, and the results above show drops whenever the test domain is new. It is dataset bias in agricultural form.
• Growth stage. Appearance changes through the season. The GWHD 2021 authors split sub-datasets by session partly because heads are barely visible at emergence and bend and overlap during ripening, which makes labelling more erratic (David et al., 2021).
• Small and similar objects. Weeds at early growth stages are small and green on green; CropAndWeed puts instances under 16 × 16 pixels in a fallback class because annotators could not name them reliably (Steininger et al., 2023).
• Class imbalance. Drydown, nutrient deficiency and weed clusters dominate Agriculture-Vision’s annotated area (Chiu et al., 2020), and a few crops dominate BreizhCrops’ parcels, with sunflower and nuts kept as deliberately rare classes (Rußwurm et al., 2020).
• Label definitions and noise. What counts as a visible head, a weed cluster or a disease symptom is a protocol decision, and labelling noise was one reason GWHD was relabelled in 2021.
• Image size, sparsity and clouds. Whole-field aerial images are tens of thousands of pixels across with sparse annotations, so training uses crops around labels (Chiu et al., 2020); optical satellite series have cloudy gaps and outliers (Rußwurm et al., 2020).

Failure Modes and Risks

• Lab-to-field collapse. A classifier trained on controlled images can lose most of its accuracy on field photographs (PlantVillage, above).
• Learning the setting instead of the plant. A model can key on background, lighting or the colours of particular fields. Mohanty et al. designed their grey-scale and segmented experiments to test for this, and the example shows a classifier that scores 0.93–0.98 on random pixel folds and 0.65–0.92 on a held-out field.
• Threshold drift. Fixed thresholds on indices tuned on one field or day fail when soil, residue or light change, as in Part A of the example.
• Scale errors in counts. Head or plant density depends on the sampled area, which depends on camera height and the reference level in the canopy; David et al. (2020) quantify errors of 8% and 21% from these alone.
• Optimistic splits. Random splits across images from the same field, session or leaf report the within-domain score, which is usually not the number a deployment needs. Dataset papers sometimes do not say how images of one session were split.
• Mismatched test sets. GWHD 2021’s diversity analysis found the 2020 test set biased relative to its training set (David et al., 2021).

Deployment Considerations

• Real-time constraints. On a sprayer or weeding machine, models must keep up with the machine’s travel (our reasoning). CropAndWeed chose an efficient DLA-34 segmentation model so that segmentation can run with detection in real time, and adds stem localization to guide intervention at the plant; given perfect boxes, it placed stem points correctly for 73.1% of plants (Steininger et al., 2023).
• Phones. Mohanty et al. (2016) note that classification takes less than a second on a CPU once a model is trained, which makes on-phone diagnosis feasible; by their own field test, accuracy rather than speed is the limit (our reading).
• Acquisition protocols. Fixing camera height, view angle, field of view and growth stage, as GWHD recommends, should reduce the shift a model must absorb and make counts comparable (our reasoning).
• Metadata. Session, site, date, platform, camera settings, growth stage and genotype make it possible to split by domain, to diagnose failures and to combine datasets; GWHD lists a minimum set (David et al., 2020).
• Timing. Yield and condition estimates are worth most early in the season, when the images hold least information; You et al. (2017) show models improving month by month.
• Licences and access. Agricultural datasets come with different terms: GWHD 2021 is open, CropAndWeed is non-commercial, Agriculture-Vision’s challenge data are released on request under terms, and BreizhCrops builds on parcel data that the French IGN releases under an open licence, by the paper’s account (Rußwurm et al., 2020); dataset terms are as of November 2026.