Ophthalmic Image Analysis

Ophthalmic image analysis applies computer vision to images of the eye, above all to photographs of the retina and to cross-sectional scans of it made by optical coherence tomography (OCT). Its most developed use is screening people with diabetes for diabetic retinopathy, a disease that is graded from visible lesions on standard photographs and whose screening is a high-volume, repetitive reading task. A deep learning system for this task was validated on photographs of several thousand patients in 2016, and in April 2018 an AI system that outputs a screening result without a clinician reading the images was classified by the US Food and Drug Administration (FDA) through a De Novo request. Vessel segmentation, lesion detection and OCT layer segmentation are the other main tasks.

Medical Image Classification covers the classification task in general, including Raghu et al.’s experiments on retinal fundus photographs, and Clinical Validation and Regulation of Medical Imaging AI covers reader studies, prospective and randomized designs and regulation in outline. This article takes them as given and treats what is specific to the eye: the two main sensors in outline, the grading scales that define the labels, the screening systems and their evidence, segmentation of vessels, lesions and retinal layers, and the public datasets with their access terms. It describes methods, evidence and limits for practitioners and researchers; it gives no clinical guidance and draws no clinical conclusions of its own, and reported results are attributed to their authors with the design behind them.

Problem

Screening for diabetic retinopathy looks for retinal damage in people with diabetes before they notice symptoms. Gulshan et al. (2016) summarize the usual management the guidelines they cite recommend: annual screening for people with no or mild retinopathy, repeat examination in six months for moderate disease, and referral to an ophthalmologist for severe or worse retinopathy or for referable diabetic macular edema. Because management changes at moderate severity, they define referable diabetic retinopathy (RDR) as moderate or worse retinopathy, referable macular edema, or both. The vision task follows from that definition: from one or more photographs per eye, decide whether a person crosses the referral threshold. The decision is binary, but the label underneath is an ordinal grade read by people, with the disagreements that this implies.

Abràmoff et al. (2018) state the case for automation in their trial report: fewer than half of US patients with diabetes keep to the recommended schedule of eye examinations, by the source they cite, and imaging in or near primary care with remote reading had been widely studied as a remedy. An automated reader removes the remote reader. Beyond diabetes, the same images are used for macular degeneration, glaucoma and many other conditions, and OCT has become a routine instrument for macular disease, diabetic retinopathy and glaucoma (Aumann et al., 2019).

Visual Data

Fundus photography

A fundus photograph is a colour image of the back of the eye taken through the pupil, showing the optic disc, the macula with the fovea at its centre, and the retinal vessels. The public datasets describe typical acquisitions. DRIVE’s 40 images came from a Canon CR5 non-mydriatic 3CCD camera with a 45° field of view, 8 bits per colour plane and 768 × 584 pixels, with a circular field of view about 540 pixels across (DRIVE challenge page). The 1,200 Messidor images were taken with a Topcon TRC NW6 non-mydriatic retinograph, also with a 45° field, at 1440 × 960, 2240 × 1488 or 2304 × 1536 pixels; 800 were taken after pupil dilation and 400 without (Decencière et al., 2014). Gulshan et al.’s EyePACS validation images were macula-centred, came from several camera models with 45° fields, and about 40% were taken with dilation.

“Non-mydriatic” means the camera can image through an undilated pupil, but dilation is still sometimes needed. In Abràmoff et al.’s pivotal trial, 23.6% of participants needed dilation before the AI system judged their images to be of sufficient quality. The reference standard in that trial used a much larger protocol: four stereoscopic pairs of images per eye, each pair covering 45–60°, taken by certified photographers (the Wisconsin reading centre’s 4W-D protocol), which the authors describe as covering the same retinal area as the older modified 7-field stereo film protocol. The protocol matters for vision: a single 45° image sees a fraction of what the reference standard sees, and stereo pairs let graders judge retinal thickening that a single image cannot show.

Optical coherence tomography

OCT measures the depth at which light is scattered back from tissue, by low-coherence interferometry: light from a broadband source is split between the sample and a reference arm, and the two returns interfere only when their optical path lengths match within the source’s short coherence length (Aumann et al., 2019). One depth profile is an A-scan; scanning the beam sideways gives a cross-sectional B-scan, and stepping the B-scans gives a volume. The first systems moved the reference mirror for every A-scan (time-domain OCT); Fourier-domain systems record the interference spectrum and compute the whole A-scan by a Fourier transform, either with a spectrometer and line camera (spectral-domain OCT, SD-OCT) or with a wavelength-swept source and a single detector (swept-source OCT).

Axial and lateral resolution are decoupled. The axial resolution in air equals the round-trip coherence length of the source; for a source with centre wavelength $\lambda_0$ and spectral bandwidth $\Delta\lambda$ (full width at half maximum), Aumann et al. (2019, Eq. 3.5) give

$$\delta z = \frac{2 \ln 2}{\pi} \, \frac{\lambda_0^2}{\Delta \lambda},$$

so a broader spectrum gives finer depth resolution, independently of the focusing optics and therefore of the small pupil. Ophthalmic systems work near 850 nm or 1050 nm, wavelengths that pass the retinal pigment epithelium so that the choroid can be imaged. Because OCT measures optical delays, axial distances must be divided by the tissue’s refractive index to give physical thicknesses. As an example the chapter gives a Heidelberg SPECTRALIS source with an 880 nm centre wavelength and 40 nm bandwidth and an axial resolution of about 7 µm in the eye; the formula gives about 8.5 µm in air (our arithmetic), which the division by the refractive index brings down. Lateral resolution is set by the focused spot and is typically about 10 µm in retinal OCT, with a depth of focus of about 700 µm. Coherent detection also produces speckle, a granular pattern whose grain size matches the resolution element; averaging repeated scans reduces it, with the signal-to-noise ratio growing with the square root of the number of acquisitions.

The chapter dates the first commercial ophthalmic OCT to 1996, five years after OCT was introduced, and SD-OCT combined with a scanning laser ophthalmoscope for motion tracking to 2006. For vision code, the important properties are that a macular OCT volume is a stack of B-scans whose voxels are much larger between B-scans than within them, that intensities are speckled and depend on the instrument, and that the retina appears as a sequence of near-horizontal layers whose boundaries are the targets of segmentation.

Grading Scales

The labels that screening models learn are grades on published scales, and the scales differ in their categories, in where they put the referral threshold and in what they require the grader to see.

• International Clinical Diabetic Retinopathy (ICDR) scale. Five grades: none, mild, moderate, severe and proliferative diabetic retinopathy. Gulshan et al. (2016) graded their images on this scale, defined referable macular edema in photographs as any hard exudates within one disc diameter of the macula (a proxy for edema when stereoscopic views are unavailable), and called RDR moderate or worse retinopathy, referable macular edema, or both. Raghu et al.’s retinal data used a five-class scale on which the upper three grades counted as referable. The ICDR scale’s own defining publication could not be read for this article; its use is described from the papers that apply it.
• ETDRS severity scale. A finer scale of numbered levels read from standard stereo photographs by trained reading-centre graders. In Abràmoff et al.’s trial, more than mild diabetic retinopathy (mtmDR) was ETDRS level 35 or higher, or diabetic macular edema, in at least one eye; the paper describes levels 10–20 as no or mild retinopathy, 35–47 as moderate and 53 or higher as more than moderate, following the American Academy of Ophthalmology practice pattern it cites.
• English NHS Diabetic Eye Screening Programme. Feature-based grades for each eye: R0 (no retinopathy), R1 (background; for example any microaneurysm or haemorrhage), R2L and R2H (pre-proliferative, lower and higher risk, defined by features such as venous beading, numbers of blot haemorrhages and intraretinal microvascular abnormalities), R3S and R3A (stable treated and active proliferative), maculopathy grades M0 and M1 (for example exudate within one disc diameter of the fovea), OCT grades used in surveillance clinics, and U for an ungradable image set. The guidance names a referral destination for grades such as R2L, R2H and R3A (NHS England, 2025). Graders select individual features on a form and grading software applies rules to produce the grade.
• Dataset-specific rules. Messidor’s retinopathy grade is a count rule rather than a clinical scale: grade 0 has no microaneurysms or haemorrhages, grade 1 has up to five microaneurysms and no haemorrhages, grade 2 has more than five and fewer than fifteen microaneurysms or one to four haemorrhages, without neovascularization, and grade 3 has fifteen or more microaneurysms, five or more haemorrhages, or neovascularization; its macular edema risk grade is based on the distance of hard exudates from the macula (ADCIS, Messidor page).

A model trained on one scale and tested against another inherits the mismatch, and so does any comparison of results across datasets graded differently. Gulshan et al.’s Messidor-2 results, for example, use their own panel’s ICDR grades, not Messidor’s count rule.

Vision Tasks Involved

• Image-level grading and referral. Classify a photograph, an eye or a person into grades or into referable and not referable, usually with an image-quality decision alongside. This is the screening task.
• Image quality assessment. Decide whether an image is gradable. Gulshan et al. trained gradability as one of the network’s outputs; the IDx-DR system has a separate image quality algorithm that gives the operator feedback within seconds (Abràmoff et al., 2018).
• Lesion detection and segmentation. Find microaneurysms, haemorrhages, hard and soft exudates and new vessels, the features from which grades are defined. Lesion Detection treats the task in general.
• Vessel segmentation. Delineate the retinal vasculature, for measurements of vessel width and tortuosity, for registration of images taken at different times, and as a step in other analyses (DRIVE challenge page).
• Anatomical landmarks. Locate the optic disc and the fovea; IDRiD includes optic disc and fovea centre labels (IDRiD dataset page).
• OCT layer and fluid segmentation. Find the boundaries of retinal layers in B-scans, for thickness measurements, and segment fluid and other pathology. Medical Image Segmentation treats segmentation in general.
• OCT referral. Classify a volume into referral categories, as De Fauw et al. (2018) did.

Common Methods

Vessel enhancement and segmentation

Vessels in a fundus photograph are thin, elongated structures darker than the background, a few pixels to a few tens of pixels wide. Classical methods enhance such tubular structures with filters built on second derivatives. The Frangi vesselness filter (Frangi et al., 1998), as implemented in scikit-image 0.25.2, computes at each scale $\sigma$ the eigenvalues $\lambda_1, \lambda_2$ of the Hessian of the Gaussian-smoothed image, ordered so that $|\lambda_1| \le |\lambda_2|$. A dark line on a bright background has a large positive curvature across the line and a small curvature along it. For dark ridges the 2D response is

$$\mathcal{V}_\sigma = \exp\left(-\frac{R_B^2}{2\beta^2}\right)\left(1 - \exp\left(-\frac{S^2}{2\gamma^2}\right)\right), \qquad R_B = \frac{|\lambda_1|}{\lambda_2}, \quad S = \sqrt{\lambda_1^2 + \lambda_2^2},$$

where $R_B$ is small for line-like and large for blob-like structure, $S$ is small in flat, noisy background, and $\beta$ and $\gamma$ set the sensitivity to each. Where $\lambda_2$ is not positive (a bright structure) the response is zero: the implementation clips $\lambda_2$ to $10^{-10}$, so that the first factor underflows. The output is the maximum over scales. The derivatives are not multiplied by $\sigma^2$, so the responses at different scales are not scale-normalized. In scikit-image the defaults are $\beta = 0.5$ and $\gamma$ equal to half the maximum of $S$ over the image; on our reading of the source, $\gamma$ is computed at the first scale and reused for the others. The code comments cite the paper’s equations (13) and (15) for the 2D form; the paper’s text could not be read for this article, so the form is given as implemented.

Supervised methods learn a pixel classifier from manual vessel maps. Staal et al. (2004), the paper Decencière et al. (2014) cite for DRIVE, presented a ridge-based method, and fully convolutional networks such as the U-Net are now common; DRIVE’s challenge page ranks submissions by mean Dice coefficient inside the field-of-view mask.

Lesion-based and image-level grading

Two designs dominate screening systems. The first detects the lesions that define the grades and combines the detections into a decision. IDx-DR follows it: Abràmoff et al. (2018) describe independent detectors for microaneurysms, haemorrhages and exudates, implemented as convolutional networks except for a multiscale feature-bank microaneurysm detector, trained on more than a million lesion patches and fused into a disease-level output by a separately trained algorithm. The authors argue that lesion detectors make the system more robust and less prone to bias across ethnic groups; these are their arguments, and they point to the absence in their trial of significant effects of race, ethnicity and other covariates on sensitivity.

The second design trains one network on whole images against grades. Gulshan et al. (2016) trained Inception-v3, initialized from ImageNet weights (which, they report, sped up training and improved performance), on 128,175 macula-centred images graded 3 to 7 times each by 54 US-licensed ophthalmologists and senior residents. One network made several binary predictions (moderate or worse, severe or worse, referable edema, fully gradable), and an ensemble of 10 networks was averaged. They noted that the network does not explicitly detect lesions. Transfer Learning and Medical Image Classification report Raghu et al.’s finding that ImageNet transfer barely changed accuracy on a large retinal dataset.

Pretraining on large collections of unlabelled retinal images is the newest step. RETFound (Zhou et al., 2023) was pretrained with self-supervised learning on 904,170 colour fundus photographs and 736,442 OCT scans; it is one of the medical foundation models, a broader subject with its own pretraining methods and evaluations.

OCT layer segmentation

Layer boundaries in a B-scan run roughly across the image, and each boundary is a change from bright to dark or dark to bright along the A-scan. Graph methods exploit this. In the shortest-path formulation that Mishra et al. (2020) build on, each pixel is a node connected to its eight neighbours, two buffer columns of minimal weight are added at the sides, and Dijkstra’s algorithm finds the cheapest path across the image, with edge weights

$$w_{ab} = 2 - (g_a + g_b) + w_{\min},$$

where $g_a$ and $g_b$ are the normalized vertical gradients at the two pixels and $w_{\min}$ is the weight in the buffer columns; a second weight map with the inverted gradient serves boundaries of the opposite polarity. Layers are found in order of contrast, the inner limiting membrane and the inner–outer segment junction first, each segmented boundary narrowing the search for the next. Mishra et al. combine this with a U-Net: the network’s probability maps for each boundary enter the edge weights and the regions of interest. They report that graph-based approaches had generally performed best in earlier work, that graph-search frameworks are slow and that shortest-path frameworks lack constraints between surfaces, and they apply the combination to 11 retinal surfaces and to drusen and reticular pseudodrusen in SD-OCT B-scans of eyes with age-related macular degeneration, with healthy eyes used to train the layers these lesions leave largely unaffected.

Semantic segmentation networks label every pixel with its layer or region instead. Kugelman et al. (2022) compared eight U-Net variants (vanilla, dense, attention, squeeze-and-excite, residual, recurrent-residual, U-Net++ and Inception U-Net) on four OCT datasets: healthy children (Spectralis), Stargardt disease (Spectralis), age-related macular degeneration (Bioptigen, four sites) and widefield scans of healthy adults, each split by participant. Scored by an overall Dice coefficient on the test sets, most architectures differed little; adding a third convolution per block improved every architecture slightly, and the authors conclude that the vanilla U-Net is sufficient for this task given the extra complexity and slower speed of the variants. Their reported Dice values are around 99% on the healthy and AMD datasets (Tables 2 and 4). By our reasoning, an overall Dice over regions that include the large vitreous and choroid–sclera areas is dominated by those regions, so boundary errors of a few pixels, which matter for thickness measurement, change it little.

OCT referral from segmentation

De Fauw et al. (2018) applied a deep learning architecture to three-dimensional OCT scans of patients referred to a major eye hospital. From the abstract, the only part of the paper that could be read for this article: after training on 14,884 scans, the system made referral recommendations whose performance reached or exceeded that of experts on a range of sight-threatening retinal diseases, and the tissue segmentations it produced served as a device-independent representation, so that referral accuracy was maintained with segmentations from a different type of device. The architecture, the experts’ comparison and the data set-up are in the paper’s body.

Example: Vessel Enhancement and Evaluation at a Referable Threshold

The script has two parts. Part A applies the Frangi filter to the fundus photograph bundled with scikit-image (skimage.data.retina(), a 1411 × 1411 public-domain image, loaded from the installed package without a network connection). It works on the green channel, a common choice for retinal vessels that we use without measuring the alternatives, builds a field-of-view (FOV) mask from the red channel, and thresholds the vesselness inside the FOV by Otsu’s method. It compares the raw image with a copy whose outside-FOV pixels are filled with the median green value inside the FOV, and reports how many flagged pixels fall in a 15-pixel ring inside the FOV edge, and at which scale the flagged pixels respond most. There is no manual vessel map for this image, so Part A measures where the filter responds, not its accuracy.

Part B builds synthetic 64 × 64 patches of patients with two eyes, each eye with an ICDR-like grade 0–4. The grade distributions are taken from Gulshan et al.’s Table: the enriched development set for the tuning data, the EyePACS-1 set for test A and Messidor-2 for test B. The worse eye’s grade is drawn from the distribution, and the other eye has the same grade with probability 0.7 and otherwise one grade lower. Each patch is a smoothed-noise background with a random illumination gradient, a Poisson number of small dark Gaussian dots that stand in for microaneurysms (mean 0.3, 2, 5, 9 and 14 dots for grades 0–4), a random blur for image quality, and noise. The “classifier” is untrained: it counts dark-dot candidates as local minima of a difference of Gaussians. Referable means grade 2 or worse. Two operating points are fixed on the tuning set, as Gulshan et al. fixed theirs on their development set: the highest score threshold that keeps sensitivity at least 0.97, and the lowest that gives specificity at least 0.98. On the test sets the script reports ROC AUC, sensitivity and specificity with exact Clopper–Pearson 95% intervals, predictive values, sensitivity by grade, patient-level results for the rule “refer if either eye is referable”, and a patient-level bootstrap interval for eye-level sensitivity. It then repeats the data generation for seeds 0–4 and prints the range of test-A sensitivity and specificity at the high-sensitivity point.

The example was planned before it was first run. Two changes followed the first run, and the shown script is the second version. In Part A, a line printing the mean green intensity in the edge ring and in the rest of the FOV was added, so that the result below can be explained from the shown output; no parameter changed. In Part B, the standard deviation of the background texture was lowered from 0.05 to 0.02: in the first run the texture itself produced many dark-dot candidates in every patch, so the count barely separated the grades. The detector, the threshold rules, the lesion counts, the set sizes and the seeds were not changed. Nothing else changed. The script runs in about 20 seconds with NumPy 2.3.5, SciPy 1.16.3, scikit-image 0.25.2 and scikit-learn 1.7.2, and two runs gave identical output.

import numpy as np
from scipy import ndimage as ndi
from scipy.stats import binomtest
from skimage import data, filters
from skimage.feature import peak_local_max
from sklearn.metrics import roc_auc_score

# ---------- Part A: Frangi vesselness on the bundled fundus photograph ----------
rgb = data.retina()                                   # (1411, 1411, 3) uint8, CC0
green = rgb[..., 1] / 255.0
fov = ndi.binary_erosion(rgb[..., 0] > 20, iterations=5)
edge_band = fov & ~ndi.binary_erosion(fov, iterations=15)   # 15-px ring inside the FOV edge
sigmas = (1, 2, 4, 8)

def flagged(img):
    v = filters.frangi(img, sigmas=sigmas, black_ridges=True)
    t = filters.threshold_otsu(v[fov])
    return v, fov & (v > t)

filled = green.copy()
filled[~fov] = np.median(green[fov])                  # remove the dark-to-bright FOV border
print(f"green: FOV median {np.median(green[fov]):.3f}, edge-ring mean {green[edge_band].mean():.3f}, "
      f"rest of FOV {green[fov & ~edge_band].mean():.3f}")
for name, img in [("raw", green), ("filled", filled)]:
    v, m = flagged(img)
    print(f"{name:6s}: flagged {m.sum() / fov.sum():.3f} of FOV, "
          f"{(m & edge_band).sum() / m.sum():.3f} of flagged pixels in the edge ring")
per_scale = np.stack([filters.frangi(filled, sigmas=[s], black_ridges=True) for s in sigmas])
best = per_scale.argmax(0)[m]
print("best scale among flagged pixels:",
      {s: round(float(np.mean(best == i)), 3) for i, s in enumerate(sigmas)})

# ---------- Part B: grading-style evaluation on synthetic patches ----------
DIST = {"dev (Gulshan development)": [.454, .259, .206, .045, .037],
        "test A (EyePACS-1)":        [.825, .096, .062, .006, .011],
        "test B (Messidor-2)":       [.697, .151, .121, .016, .014]}
LAM = [0.3, 2, 5, 9, 14]                              # mean number of dark dots per grade
yy, xx = np.mgrid[:64, :64]

def make_set(n_patients, dist, rng):
    worse = rng.choice(5, n_patients, p=np.array(dist) / sum(dist))
    other = np.where(rng.random(n_patients) < 0.7, worse, np.maximum(worse - 1, 0))
    grades = np.stack([worse, other], 1).ravel()      # two eyes per patient
    patient = np.repeat(np.arange(n_patients), 2)
    imgs = []
    for g in grades:
        bg = ndi.gaussian_filter(rng.normal(size=(64, 64)), 3)
        img = 0.5 + 0.02 * bg / bg.std() + rng.uniform(-0.05, 0.05) * (yy / 63 - 0.5)
        for _ in range(rng.poisson(LAM[g])):
            cy, cx = rng.uniform(4, 60, 2)
            img -= rng.uniform(0.08, 0.2) * np.exp(-((yy - cy)**2 + (xx - cx)**2) / (2 * 1.2**2))
        img = ndi.gaussian_filter(img, rng.uniform(0.5, 2.0))
        imgs.append(img + rng.normal(0, 0.02, img.shape))
    return np.array(imgs), grades, patient

def score(img):                                        # count dark dot candidates
    dog = ndi.gaussian_filter(img, 1) - ndi.gaussian_filter(img, 3)
    return len(peak_local_max(-dog, min_distance=3, threshold_abs=0.02))

def run(seed, verbose):
    rng = np.random.default_rng(seed)
    sets = {k: make_set(n, d, rng) for (k, d), n in zip(DIST.items(), (1000, 1500, 600))}
    out = {}
    for k, (imgs, g, pid) in sets.items():
        s = np.array([score(im) for im in imgs]); y = g >= 2
        out[k] = (s, y, g, pid)
    s, y, _, _ = out["dev (Gulshan development)"]
    cand = np.unique(s)
    t_sens = max(t for t in cand if np.mean(s[y] >= t) >= 0.97)    # high-sensitivity point
    t_spec = min(t for t in cand if np.mean(s[~y] < t) >= 0.98)    # high-specificity point
    res = {}
    for k in list(DIST)[1:]:
        s, y, g, pid = out[k]
        if verbose:
            print(f"\n{k}: {len(y)} eyes, referable prevalence {y.mean():.3f}, AUC {roc_auc_score(y, s):.3f}")
        for name, t in [("high-sens", t_sens), ("high-spec", t_spec)]:
            p = s >= t
            tp, fn, tn, fp = (p & y).sum(), (~p & y).sum(), (~p & ~y).sum(), (p & ~y).sum()
            se, sp = tp / (tp + fn), tn / (tn + fp)
            res[(k, name)] = (se, sp)
            if not verbose:
                continue
            ci_se = binomtest(tp, tp + fn).proportion_ci(method="exact")
            ci_sp = binomtest(tn, tn + fp).proportion_ci(method="exact")
            print(f"  {name} (score >= {t}): sens {se:.3f} [{ci_se.low:.3f}, {ci_se.high:.3f}]"
                  f"  spec {sp:.3f} [{ci_sp.low:.3f}, {ci_sp.high:.3f}]"
                  f"  PPV {tp / (tp + fp):.3f}  NPV {tn / (tn + fn):.3f}")
            if name == "high-sens":
                print("    sensitivity by grade:",
                      {gr: round(float(np.mean(p[g == gr])), 3) for gr in (2, 3, 4)})
                pp, py = p.reshape(-1, 2).any(1), y.reshape(-1, 2).any(1)
                print(f"    patient level (either eye): sens {np.mean(pp[py]):.3f}  spec {np.mean(~pp[~py]):.3f}")
                boot, rb = [], np.random.default_rng(1)
                n_pat = len(y) // 2
                for _ in range(1000):
                    idx = (2 * rb.integers(0, n_pat, n_pat)[:, None] + [0, 1]).ravel()
                    boot.append(np.mean(p[idx][y[idx]]))
                print(f"    eye-level sens, patient bootstrap 95%: [{np.percentile(boot, 2.5):.3f}, "
                      f"{np.percentile(boot, 97.5):.3f}]")
    return res

run(0, verbose=True)
r = [run(seed, verbose=False)[("test A (EyePACS-1)", "high-sens")] for seed in range(5)]
print("\nseeds 0-4, test A high-sens: sens %.3f-%.3f, spec %.3f-%.3f" % (
    min(x[0] for x in r), max(x[0] for x in r), min(x[1] for x in r), max(x[1] for x in r)))

green: FOV median 0.314, edge-ring mean 0.282, rest of FOV 0.328
raw   : flagged 0.032 of FOV, 0.027 of flagged pixels in the edge ring
filled: flagged 0.032 of FOV, 0.085 of flagged pixels in the edge ring
best scale among flagged pixels: {1: 0.089, 2: 0.499, 4: 0.3, 8: 0.113}

test A (EyePACS-1): 3000 eyes, referable prevalence 0.072, AUC 0.861
  high-sens (score >= 2): sens 0.981 [0.953, 0.995]  spec 0.207 [0.192, 0.222]  PPV 0.087  NPV 0.993
    sensitivity by grade: {2: 0.977, 3: 1.0, 4: 1.0}
    patient level (either eye): sens 1.000  spec 0.041
    eye-level sens, patient bootstrap 95%: [0.962, 0.996]
  high-spec (score >= 10): sens 0.219 [0.165, 0.280]  spec 0.994 [0.991, 0.997]  PPV 0.746  NPV 0.943

test B (Messidor-2): 1200 eyes, referable prevalence 0.124, AUC 0.854
  high-sens (score >= 2): sens 0.980 [0.942, 0.996]  spec 0.183 [0.160, 0.207]  PPV 0.145  NPV 0.985
    sensitivity by grade: {2: 0.973, 3: 1.0, 4: 1.0}
    patient level (either eye): sens 1.000  spec 0.039
    eye-level sens, patient bootstrap 95%: [0.954, 1.000]
  high-spec (score >= 10): sens 0.228 [0.163, 0.304]  spec 0.993 [0.986, 0.997]  PPV 0.829  NPV 0.901

seeds 0-4, test A high-sens: sens 0.978-0.990, spec 0.187-0.207

Part A. Filling the outside of the FOV was meant to remove the strong dark-to-bright step at its border, and it did not reduce the flags near the edge: it more than tripled their share. The first line explains why. This photograph darkens towards the rim of the field, so the 15-pixel ring is darker than both the interior and the median used for filling; once the dark outside is replaced by a brighter value, the dimmed rim becomes a dark band between two brighter regions, which is exactly the structure a dark-ridge filter is built to find. The raw image avoids that band but has its own dependence on the border: by our reading of the code, the default $\gamma$ comes from the strongest Hessian response anywhere in the image, the step at the FOV border included, so what lies outside the field can change the response inside it. Masking the field of view is therefore a design decision with consequences for the output, not a formality. In the filled image, half of the flagged pixels respond most at $\sigma = 2$ pixels and most of the rest at $\sigma = 4$, with about a tenth each at the smallest and largest scales, so a single scale would miss much of what the multiscale filter finds; because the responses are not scale-normalized, this split reflects the implementation as well as the vessel widths. Without a manual vessel map, none of this says how accurate the filter is.

Part B. The untrained dot counter ranks eyes moderately well (AUC about 0.86 on both test sets), and the operating points show what that means for screening. At the threshold chosen for 97% sensitivity on the tuning data, the test sensitivity stays near 0.98, but four out of five non-referable eyes are flagged. The misses are all at grade 2, where the number of dots overlaps with grade 1; this is our construction, but it is also where the source’s referral threshold sits. At the high-specificity point the same score finds only about a fifth of referable eyes. The sensitivity and specificity are about the same on the two test sets, while the predictive values follow prevalence: PPV at the high-specificity point rises from 0.75 to 0.83 as the referable prevalence rises from 7.2% to 12.4%. The patient-level rule “refer if either eye is referable” catches every referable patient here but flags almost every patient, because two chances of a false positive compound a low specificity. The patient bootstrap interval for eye-level sensitivity is close to the exact binomial interval in this simulation, so treating the two eyes as independent cost little here; the US special controls quoted below nonetheless require that such independence not be assumed without justification. Over seeds 0–4 the test-A sensitivity at the high-sensitivity point ranged from 0.978 to 0.990 and the specificity from 0.187 to 0.207.

Datasets and Access

Facts are from the official pages and papers, as of November 2026. None of the datasets was downloaded for this article, and no data-use agreement was accepted.

Dataset	Content	Labels	Access terms (official page)
DRIVE	40 colour fundus photographs (768 × 584) from a Dutch screening programme; 33 without and 7 with signs of mild early retinopathy	Manual vessel maps (one for each training image; test maps held by the challenge) and FOV masks; 20 training and 20 test images	Download through the grand-challenge.org site; the download page refused automated access, so its terms could not be read
Messidor	1,200 images from three French ophthalmology departments, acquired before 2007 (Decencière et al., 2014)	Retinopathy grade 0–3 by the count rule above and macular edema risk 0–2; errata list corrected grades and 13 duplicate pairs	Free for research and education; copying, redistribution and unauthorized commercial use prohibited; download link by e-mail after a form
Messidor-2	874 examinations, 1,748 macula-centred images (the paired Messidor images plus 345 examinations from Brest, 2009–2010)	No official grades; third-party grades exist separately	Same terms as Messidor; acknowledgement of LaTIM and the Messidor partners required
EyePACS (Kaggle)	Screening photographs from the EyePACS programme; Gulshan et al. mention a Kaggle competition that used deep learning to predict the retinopathy grade only, and RETFound used 88,702 Kaggle EyePACS images in pretraining	Retinopathy grades (the label scheme could not be checked on the competition pages)	Kaggle competition pages render only in a browser with JavaScript and could not be read
APTOS-2019	Kaggle competition dataset from India	ICDR grades; Zhou et al. (2023) describe the grading information as limited, possibly from a single clinician	Kaggle competition pages (not readable, as above)
IDRiD	Indian dataset from the ISBI 2018 Diabetic Retinopathy: Segmentation and Grading Challenge	81 images with pixel-level lesion maps (microaneurysms, haemorrhages, hard and soft exudates) and optic disc; 516 images (413 training, 103 test) with retinopathy and macular edema grades and optic disc and fovea centres	Listed as open access on IEEE DataPort; no licence is stated on the page
Duke DME OCT	OCT images of eyes with diabetic macular edema, with the manual and automated markings used in Chiu et al. (2015)	Layers and cysts, as the laboratory’s dataset list describes them	Free download; research and educational use; commercialization and redistribution prohibited; citation of the paper required

Two properties of these datasets matter for evaluation. Their grades come from very different reference standards: Gulshan et al.’s validation grades are majority votes of seven or eight ophthalmologists, IDRiD’s are adjudicated consensus grades of two experts by Zhou et al.’s summary, and Messidor’s original grades were made by the medical experts who provided its diagnoses. And several include both eyes or repeat images of a person, which calls for splits by patient, as Medical Image Classification describes. Messidor’s own errata show the kind of label problems that persist in a widely used public set: duplicated images, two pairs of which carry inconsistent grades.

Evaluation and Evidence

Retrospective validation with expert panels

Gulshan et al. (2016) validated their algorithm on two retrospective sets that were not enriched for disease: EyePACS-1, a random sample of 9,963 images from 4,997 patients at EyePACS screening sites in 2015, graded by eight board-certified ophthalmologists, and Messidor-2, 1,748 images from 874 patients, graded by seven. The reference standard was the majority decision of the panel. On fully gradable images (8,788 and 1,745), the algorithm detected RDR with an AUC of 0.991 (95% CI 0.988–0.993) on EyePACS-1 and 0.990 (0.986–0.995) on Messidor-2. At the operating point chosen on the development set to approximate the ophthalmologists’ specificity, sensitivity and specificity were 90.3% and 98.1% on EyePACS-1 and 87.0% and 98.5% on Messidor-2; at the high-sensitivity point they were 97.5% and 93.4%, and 96.1% and 93.9%. Their intervals were exact Clopper–Pearson intervals with individual coverage $\sqrt{0.95} \approx 0.975$, so that sensitivity and specificity hold jointly at 95%. Performance on the tuning set plateaued at about 60,000 training images, and using only one grade per image on the tuning set lowered the measured performance markedly, which led the authors to recommend spending grading effort on the evaluation sets. The abstract gives the EyePACS-1 RDR prevalence as 683 of 8,878 fully gradable images, while the Table, the text and Figures 1 and 2 give 8,788 fully gradable images (683 of 8,788 is the 7.8% that both state); this article uses the Table’s figure. The copy read is the version corrected on 13 December 2016 for a numerical error in Figure 1, and the abstract’s 8,878 remains in it.

The authors list the limits of the design: the reference standard was the majority of graders, so images with subtle findings that most graders miss may be handled less well; the features the network uses are unknown; graders worked in an online tool rather than a clinic; the algorithm was trained only for diabetic retinopathy and macular edema and is not a substitute for a comprehensive eye examination; and further validation was needed against a reference standard not set by experts who took part in the algorithm’s development. These are retrospective, image-level results on curated photographs, and the paper’s own conclusion calls for research on feasibility in clinical settings and on patient outcomes.

A prospective pivotal trial of an autonomous system

Abràmoff et al. (2018) tested IDx-DR, locked before the study, prospectively at 10 US primary care sites between January and July 2017. Its operators were existing clinic staff with no prior ocular imaging experience and four hours of standardized training, using a Topcon NW400 non-mydriatic camera to take one disc-centred and one fovea-centred 45° image per eye. The reference standard was set independently: widefield stereoscopic photographs and macular OCT taken by certified photographers and graded at the Wisconsin Fundus Photograph Reading Center, the photographs on the ETDRS scale by three readers with majority voting and the OCT for centre-involved macular edema. Of 900 enrolled participants, 819 could be fully analysed; mtmDR prevalence was 23.8%, after an enrichment strategy, prespecified in the protocol, recruited higher-risk participants midway. Against prespecified superiority endpoints of 85% sensitivity and 82.5% specificity, discussed beforehand with the FDA, the system reached 87.2% sensitivity (95% CI 81.8–91.2%), 90.7% specificity (88.3–92.7%, corrected for enrichment) and an imageability of 96.1%. The study was funded by IDx; the first author is a shareholder, director and employee of IDx, and an independent contractor held the locked system and results in escrow so that the sponsor had no access during the study. The authors attribute the lower sensitivity than the 97% of an almost identical system on a laboratory dataset to the wider and stereo reference imaging, the higher quality of reference photographs, and the prospective intent-to-screen design, which reduces selection and spectrum bias.

A randomized trial of outcomes

Diagnostic accuracy does not show whether screening changes what happens to patients. Wolf et al. (2024) randomized youth aged 8–21 with type 1 or type 2 diabetes at one academic paediatric diabetes centre (two hospital sites in Baltimore) either to an autonomous AI eye examination at the point of care (81 participants) or to a scripted referral to an eye care provider with education (83). Diabetic eye examination completion within six months, the primary outcome, was 100% (95% CI 95.5–100%) in the AI arm and 22% (14.2–32.4%; 18 of the 82 control participants analysed) in the control arm. Of 25 participants with an abnormal AI result, 16 (64%) completed follow-through with an eye care provider, compared with 22% in the control arm. The system (LumineticsCore, formerly IDx-DR) is not labeled for people under 22, so all images were also read by a retina specialist; the authors list this, and participants’ familiarity with the AI from an earlier study, among the limitations, and the senior author reports being an investor, director and consultant of the manufacturer, which provided the AI interpretations in kind. The trial measured completion and follow-through, not visual outcomes.

Clinical Validation and Regulation of Medical Imaging AI places these designs, retrospective panel-graded validation, a prospective trial against an independent reference standard, and a randomized trial of a process outcome, in the general hierarchy of evidence.

Regulation in the United States

The following is taken only from the FDA’s final order in the Federal Register of 21 January 2022 and the 2025 edition of the Code of Federal Regulations, read on govinfo.gov; it is an outline as of November 2026, not legal analysis, and does not cover other jurisdictions or individual devices’ clearances.

The FDA received IDx, LLC’s request for De Novo classification of IDx-DR on 12 January 2018 and, on 11 April 2018, issued an order classifying the device into class II with special controls; the final order of January 2022 codified the classification as 21 CFR 886.1100 (Food and Drug Administration, 2022). The regulation names a new generic type, the retinal diagnostic software device: a prescription software device that incorporates an adaptive algorithm to evaluate ophthalmic images for diagnostic screening to identify retinal diseases or conditions. As the order explains for De Novo classifications in general, a device not in commercial distribution before 28 May 1976 is automatically in class III until it is classified, and once a type is placed in class I or II, later devices of the same type can use the 510(k) process with the first device as predicate; the order states that this device type is subject to 510(k) premarket notification. The risks the order lists are false positives leading to unnecessary procedures, false negatives delaying evaluation or treatment, and operator failure to provide images that meet the input quality specifications.

The special controls translate the evaluation questions of this article into requirements (21 CFR 886.1100(b)). Clinical performance testing must evaluate sensitivity, specificity, positive and negative predictive value for each endpoint across the range of device outcomes, under anticipated conditions of use; where several samples come from the same patient, the statistics must not assume independence without adequate justification; every metric needs a confidence interval; and the clinical data must evaluate the variability due to the user and to the image acquisition device. The software documentation must describe the expected effect of image acquisition hardware on performance and the minimum hardware specifications. A training program on acquiring quality images is required, with human factors testing of its effect, and a protocol must describe the level of change in technical specifications that could significantly affect safety or effectiveness. The labeling must warn that the device is not to be used to screen for diseases beyond its indicated uses, that it provides a screening diagnosis only and the patient must be advised to receive follow-up care, and that it does not treat the disease, and it must summarize clinical performance with confidence intervals and describe the population and clinical environment in which the device was evaluated. Wolf et al. (2024) describe the device they used as regulated under the De Novo regulation; other devices of the type are not described here because the FDA’s own databases could not be read.

Challenges

• The reference standard. Graders disagree, especially near the referral threshold. On EyePACS-1, ophthalmologists’ mean agreement on referable images was 77.7%, with complete agreement on 19.6% of them, against 97.4% agreement on non-referable images (Gulshan et al., 2016). Scales differ (ICDR, ETDRS, national programmes, dataset rules), and photographs miss what OCT shows: in Abràmoff et al.’s trial only 6 of 19 cases of centre-involved macular edema found on OCT were identified from fundus photographs.
• Image quality and acquisition. Small pupils, cataract and operator skill decide whether an image can be graded. In the pivotal trial, lens opacities increased the number of imaging attempts and the need for dilation, and the 73 participants who could not be analysed differed significantly in lens status from those who could (Abràmoff et al., 2018). Cameras, fields of view and dilation policies vary between datasets.
• Field of view and sampling. A single 45° image covers a fraction of the retina that a reading-centre protocol covers, and the example above shows that even the boundary of the circular field changes what a filter responds to.
• Several images per person. Two eyes, several fields per eye and repeat visits create clusters that a split and a statistical analysis must respect.
• Thin and small targets. Microaneurysms span a few pixels, capillaries are near the resolution limit, and vessel maps depend on the annotator’s confidence threshold (DRIVE’s observers marked pixels they were at least 70% certain were vessel).
• Scope. A screening model trained for diabetic retinopathy is not a general eye examination. Gulshan et al. state that their algorithm may miss lesions it was not trained to identify, and in the pivotal trial 6 of 11 participants with enlarged optic disc cups and 13 of 26 with drusen or pigment epithelium atrophy were not flagged, which the authors call observations only, because the confidence intervals around them include 0%.

Failure Modes and Risks

• Spectrum and selection effects. Laboratory test sets, curated or enriched, give higher sensitivity than intent-to-screen populations, as the pivotal trial’s comparison with the laboratory study illustrates.
• Operating-point drift. A threshold chosen on one population carries its sensitivity and specificity to another only when the score distributions are the same; predictive values change with prevalence even when they are (example above).
• Ungradable images. Images that a system declines to grade must go somewhere; in the pivotal trial, mtmDR prevalence among the 33 participants with insufficient-quality output was 30%, comparable to the analysed group.
• Subgroup differences. Abràmoff et al. found higher specificity in participants over 65 and attribute it to a reflective internal limiting membrane in younger people that can be mistaken for exudates; Wolf et al. note that retinal pigmentation could introduce bias into an AI system.
• Use outside the label. A device evaluated in adults, or for one disease, has no evidence behind it for other populations or conditions, which is why the ACCESS trial added specialist overreading.

Deployment Considerations

Screening systems work at the point of care, often operated by staff without imaging experience, so image-quality feedback, operator training and dilation policies are part of the system, as the special controls require. Outputs must route each person to follow-up, including those with ungradable images. Performance claims carry their design: retrospective panel-graded validation, a prospective trial against reading-centre grading, or a randomized trial of completion and follow-through answer different questions. Locked algorithms, changes to cameras and changes to the population all bear on whether a validated result still holds; the regulation asks manufacturers for a protocol defining which technical changes matter, and Clinical Validation and Regulation of Medical Imaging AI describes monitoring after deployment in general.