Vision for Drones and UAVs

Vision for drones is the computer vision that small unmanned aircraft run on board, or that is applied to the images they record. The aircraft are multirotors such as quadrotors, fixed-wing aircraft and hybrids of the two; the literature calls them drones, unmanned aerial vehicles (UAVs), micro aerial vehicles (MAVs) for the smallest research platforms, and, with their ground station and radio links, unmanned aircraft systems (UAS). Two uses of vision meet on these platforms. The aircraft uses its cameras to fly: to estimate its own motion without external positioning, to see obstacles, to find a place to land, to follow a target or pass through a racing gate. And it carries cameras to observe: detecting and tracking vehicles and people in video, inspecting structures, surveying fields.

This article is the overview of both as the research literature describes them. It covers what an aircraft’s perception has to estimate, the cameras and sensors flown, onboard state estimation, obstacle sensing in outline, tracking and following, vision-based landing, inspection, the detection and tracking benchmarks recorded from drones, drone racing and event cameras, the limits set by compute, payload, power and latency, and the documented failures. A worked example measures how motion during the exposure, from translation and from rotation, affects the detection and localization of small targets seen by a downward-looking camera. Mapping with drone imagery, from ground sampling distance and survey overlap to orthomosaics, inspection datasets and flight rules, is the subject of aerial and drone image analysis, and satellite imagery that of remote sensing. The article describes methods, benchmarks and their documented limits; it gives no guidance on operating aircraft.

Problem

A drone’s perception has two kinds of output. The first serves flight: the aircraft’s own position, velocity and attitude, fast enough for its controller; the geometry of what is around it, so that it does not collide; and the pose of whatever it must reach, such as a landing pad, a gate or a moving target. The second serves the mission: objects detected, tracked or counted in its video, defects found on a structure, measurements of a field. The first kind is robot perception under unusually tight constraints; the second is object detection, tracking and segmentation on images with an unusual geometry.

The sources agree on what makes both hard. Delmerico and Scaramuzza (2018) state the flight requirement as accurate, low-latency state estimation under the computational constraints of embedded hardware, imposed by the power and payload limits of aerial platforms. The UAVDT authors list four difficulties for detection and tracking from drones: high object density from the wide view, small objects from high altitude, fast object motion or rotation from the drone’s own flight and camera rotation, and the need to run in real time on embedded platforms (Du et al., 2018). The VisDrone single-object tracking organizers name abrupt camera motion, small targets and viewpoint change (Du et al., 2019). For racing, Foehn et al. (2021) add low light and motion blur at high speed, with limited onboard computing. Endurance constrains everything: Falanga et al. (2017) note that a multirotor’s propulsion is less efficient than a fixed-wing aircraft’s, so its time in the air is relatively short.

Visual Data

Cameras and sensors on research platforms

The systems described in this article show what is typically flown. They combine small cameras, several of them with global shutters, an inertial measurement unit (IMU), sometimes a range sensor, and a single-board or embedded computer.

System	Cameras	Other sensors	Onboard computer	Size and mass
SVO on a quadrotor (Forster et al., 2014)	One downward-looking global-shutter camera, 752 × 480	–	Odroid-U2 (ARM Cortex-A9, 4 cores, 1.6 GHz)	–
Elevation mapping and landing (Forster et al., 2015)	One downward-looking global-shutter camera, 752 × 480	IMU on the autopilot	1.7 GHz quad-core smartphone processor	Under 450 g, 35 cm frame
Landing on a moving platform (Falanga et al., 2017)	Two 752 × 480 cameras: one forward and tilted 45° down, one downward	IMU, distance sensor	Odroid XU4	1 kg
Event-camera flight (Rosinol Vidal et al., 2018)	DAVIS 240C event camera with frames, 70° lens, looking down	IMU in the sensor	Odroid XU4 (2.0 GHz quad-core)	–
UZH-FPV recording platform (Delmerico et al., 2019)	Stereo fisheye 640 × 480 (186° field of view, global shutter); mDAVIS 346 × 260 event camera, 120° lens	IMUs	Snapdragon Flight board; Up Board for logging	1,110 g
AlphaPilot race drone (Foehn et al., 2021)	Two forward stereo pairs at ±30°, colour global shutter, 1200 × 720 at 60 Hz	IMU at 430 Hz, downward laser rangefinder	NVIDIA Jetson Xavier	About 0.7 m across, 3.4 kg
High-speed flight in the wild (Loquercio et al., 2021)	Intel RealSense D435i stereo depth camera (640 × 480 depth at 30 Hz); RealSense T265 stereo camera running visual-inertial odometry	IMUs in both sensors	NVIDIA Jetson TX2	890 g

A dash means the item is not stated in the passages read for this article.

Two choices recur. Camera direction follows the task: earlier work on MAVs proposed a single downward-looking camera with an IMU for minimal weight and power (as Forster et al., 2014, summarize it), racing drones look forward at the gates, and Falanga et al. (2017) used both, because near the landing platform a downward camera sees mostly the moving platform, so visual odometry on it would estimate motion relative to the platform rather than to the world. Field of view trades resolution for robustness: the UZH-FPV platform carries fisheye cameras, and the journal version of SVO extends semi-direct odometry to fisheye and catadioptric lenses and to multi-camera rigs, which it reports improve resilience to rotations on the spot (Forster et al., 2017).

The recognition benchmarks were recorded with commercial drones: UAVDT’s 1080 × 540 video at 30 frames per second with a DJI Inspire 2 (Du et al., 2018), VisDrone’s with several drone models (VisDrone README). Their contents are described below and in the aerial image analysis article.

Image geometry from a moving aircraft

For a camera looking straight down at flat ground from height $H$, with focal length $f$ in pixels, one pixel covers a ground distance of $H/f$, the ground sampling distance (GSD) whose derivation and use in survey planning are given in the aerial image analysis article. Motion of the aircraft moves the image. Delmerico et al. (2019) argue that the difficulty a visual dataset poses is set by this apparent motion rather than by the vehicle’s speed, and give an example: a downward VGA camera with a 90° lens at 10 m/s sees about 320 pixels per second of optical flow at 10 m altitude but about 32,000 at 10 cm. Their numbers follow the translational flow of a downward camera, $f\,v/H$, with $f = 320$ pixels for that lens (our arithmetic). A rotation at angular rate $\omega$ about an axis parallel to the image adds, for small angles and near the image centre, a flow of about $f\,\omega$ that does not depend on height (our summary of standard camera geometry). Over an exposure of duration $t_e$ the image therefore smears by

$$L \approx t_e\, f \left(\frac{v}{H} + \omega\right) \ \text{pixels},$$

when the two motions act along the same image direction. The translational part falls with altitude and is, in ground units, always $v\,t_e$; the rotational part is the same number of pixels at every altitude, so it grows relative to a target whose pixel size shrinks with altitude (our reasoning from the formula). The example below measures what this does to small targets. For forward-looking cameras, flow varies across the image and vanishes near the focus of expansion, which is why Delmerico et al. report flow distributions rather than one number; the optical flow and obstacle detection articles cover that geometry.

Vision Tasks Involved

State estimation

A drone that cannot rely on satellite positioning or motion capture must estimate its own motion from its sensors, and cameras with an IMU are the usual answer: Delmerico and Scaramuzza (2018) note that the pair meets the power and payload constraints of flying robots. How the estimators work is described in visual odometry, including its visual-inertial form, in SLAM, and for filters in the extended Kalman filter. What is specific to drones is how they were designed and benchmarked.

SVO was built for MAV state estimation. Forster et al. (2014) used direct alignment of small patches for motion estimation, so that no features need to be extracted and matched in every frame, and report 55 frames per second on the Odroid-U2 carried by the quadrotor and more than 300 on a laptop, against 27 and 91 for the PTAM variant they compared with. The pose drives the aircraft’s stabilization and control. The authors credit the high frame rate with robustness to repetitive, high-frequency texture such as asphalt and grass. The later journal version reports mean times of 0.88 to 5.48 ms for the individual components on an NVIDIA Jetson TX1, and states that SVO is most robust with cameras running at 40 to 80 frames per second (Forster et al., 2017). The code is released under GPLv3, with a closed-source edition for commercial use (README, as of November 2026).

VINS-Mono, a sliding-window optimization over pre-integrated IMU measurements and visual features described in the visual odometry article, states in its README that it is primarily designed for state estimation and feedback control of autonomous drones. The README lists features that matter on drones: automatic initialization, online camera–IMU extrinsic calibration, online estimation of the time offset between unsynchronized cameras and IMUs, and support for rolling-shutter cameras given the readout time; it asks for images above 20 Hz and IMU data above 100 Hz with accurate timestamps, and ranks expected performance from a global-shutter camera with a synchronized high-end IMU down to a rolling-shutter camera with an unsynchronized low-rate IMU (as of November 2026). The TRO paper (Qin, Li and Shen, 2018) was not read for this article.

Benchmarks on drone hardware. Delmerico and Scaramuzza (2018) ran six public monocular visual-inertial pipelines on the EuRoC MAV sequences on a laptop and three computers small enough to fly, as robot perception summarizes. The hardware shows the budget: the reference laptop draws nominally 47 W and is described as not feasible on a flying robot; an Intel NUC draws 28 W in a 10 × 10 cm box; the UP Board weighs 78 g and draws 12 W; the ODROID XU4 weighs 59 g and draws 10 W. On the two small boards, authors’ parameter settings had to be cut to run in real time (OKVIS from 400 to 200 keypoints, VINS-Mono from 150 to 100 tracked features), ROVIO could not be made to converge on the UP Board at all, and VINS-Mono with loop closure could not run on the ODROID. The authors also caution that OKVIS’s tolerance of low frame rates may owe something to EuRoC’s low-speed trajectories.

Faster flight. EuRoC’s top speed is 2.3 m/s (Delmerico et al., 2019, Table I). The UZH-FPV dataset was recorded on a 1,110 g racing quadrotor flown by an expert pilot, with more than 10 km of flight in 27 sequences in a hangar and a field, top speeds of 12.8 m/s indoors and 23.4 m/s outdoors, and laser-tracker ground truth. Its authors show the largest optical flow among the datasets they compare. In their qualitative tests, all methods tracked an easy indoor sequence with slow drift, but on a fast outdoor sequence OKVIS, VINS-Mono and Qualcomm’s proprietary VI-SLAM all began to drift after a few turns, which they take as evidence that such sequences are beyond the state of the art of the time (Delmerico et al., 2019).

Obstacle sensing

Finding obstacles from stereo depth, time to contact or learned collision prediction, and turning them into avoidance, is the subject of obstacle detection and avoidance, which treats drones alongside ground robots. Two points from it matter for the overview. Latency, from exposure to command, decides how far the aircraft travels before it reacts. And learned avoidance runs on the aircraft’s small computer: Loquercio et al. (2021) report that their network’s forward pass takes 38.9 ms on the aircraft’s Jetson TX2 and the whole step from sensor reading to plan 41.6 ms. Training such policies in simulation and transferring them to real aircraft is the subject of sim-to-real transfer.

Tracking and following

Following a target from a drone combines single-object tracking in the image with control that keeps the target in view, a loop the visual servoing article analyses. The benchmarks measure the first part on video recorded from drones. UAV123 (Mueller, Smith and Ghanem, 2016), whose paper was not read for this article, is described in the single-object tracking article.

UAVDT has a single-object tracking task on 50 test sequences, scored with the success plot (the fraction of frames whose box overlap exceeds a threshold, ranked by area under the curve) and the precision plot (the fraction of frames whose centre error is within a distance, ranked at 20 pixels). Each sequence is labelled with attributes including camera rotation, small object, object blur and illumination variation (Du et al., 2018). Of 18 trackers evaluated, MDNet scored best overall, 46.4 success and 72.5 precision, against 69.4 and 92.8 for the best results on OTB100; KCF scored 33.1 success on UAV123 and 29.0 on UAVDT. The authors also report speed: the ten most accurate trackers ran far from real time even on a high-end CPU, the most accurate at 0.28 frames per second, and the two trackers that ran in real time on a CPU, Staple-CA and KCF, reached success scores of only 39.5 and 29.0.

VisDrone-SOT2019 extended the 2018 challenge with 25 long sequences (82,644 frames, 12 by day and 13 at night) whose targets are smaller on average than those of the 2018 set (25.5 against 62.5, figures the paper gives without units). The full set has 167 sequences with 188,998 frames, split into 86 training, 11 validation and 60 test sequences, scored by success and precision at 20 pixels. The best of 22 submissions, ED-ATOM, built on the ATOM tracker with a low-light enhancement module, reached 63.5 success and 90.0 precision at about 20 frames per second; on the long-term sequences its success fell to 48.9, and the organizers attribute the drop across submissions to target drift. They list augmentation, feature matching against camera motion and blur, and re-detection after occlusion as directions for drone tracking (Du et al., 2019).

On board, tracking a target that leaves the view needs a motion model. Falanga et al. (2017) tracked a moving landing platform with an extended Kalman filter on a model of a ground vehicle, so that the aircraft could keep predicting the platform’s position when it was not visible, including during the final descent when it left the camera’s view.

Vision-based landing

Landing needs the pose of a landing site relative to the aircraft. Falanga et al. (2017) classify earlier work by how the target is found and the aircraft localized (motion capture, GPS or vision) and note that vision is the most common way to detect the target. Two systems from the same group show two designs.

Markers. Falanga et al. (2017) put a tag on the moving platform: a black cross inside a black circle on a white square. Each downward image is thresholded, the largest white quadrangle is found, and the circle, the inner corners of the cross or the quadrangle’s corners, whichever is visible, give points from which a Perspective-n-Point solution with the known tag size gives the platform’s position; RANSAC rejects outliers. Detection takes about 12 ms per image on the Odroid XU4 and was run at 20 Hz, with visual-inertial odometry at 40 Hz fused with the IMU at 200 Hz. The authors write that other tags, such as AprilTags, would fit their framework. They demonstrated landing on a ground robot moving at 1.2 m/s, which they describe as the first fully autonomous quadrotor landing on a moving target using only onboard sensing and computing.

Landing-site detection without markers. Forster et al. (2015) built a robot-centred elevation map under the aircraft from depth maps triangulated across several images of the single downward camera, updated once per second on a smartphone processor. A cell is a candidate landing spot when the surface within a radius related to the aircraft’s size is flat, measured by the summed squared height differences; cells without a converged depth count as unsafe, and the spot chosen is the one farthest, by a distance transform, from any unsafe cell. Depth did not converge on untextured surfaces, reflecting ones such as water, or moving objects, and those regions stayed empty in the map. Finding the best spot among the 10,000 cells took about 268 ms. Event cameras have also been used for landing on quadrotors with limited computing (Gallego et al., 2022).

Inspection and agriculture

Inspection flights photograph structures such as bridges, power lines and towers at close range to find defects, and agricultural flights survey crops; both are planned for ground sampling distance and overlap and often processed into maps. Their tasks and datasets, such as TTPLA for transmission towers and power lines and CODEBRIM for concrete defects, are described in aerial and drone image analysis, and farm applications in computer vision in agriculture. Factory inspection, where lighting and viewpoint are controlled, is the subject of industrial machine vision. The robotics literature names inspection among the purposes of autonomous MAVs: SVO’s authors list disaster management, industrial inspection and environment conservation (Forster et al., 2014), and the elevation-mapping paper adds agriculture, search and rescue and delivery (Forster et al., 2015).

Object detection from drones

Detection in drone images differs from detection in photographs because objects are small, numerous and seen from above or obliquely, and because the camera moves. The VisDrone benchmark, collected by the AISKYEYE team at Tianjin University with several drone models in 14 cities, under various weather and lighting conditions, covers detection in images, detection in video, single- and multi-object tracking and crowd counting; its README lists agriculture, aerial photography, fast delivery and surveillance among the uses of camera drones that motivate it (VisDrone README, as of November 2026). The image-detection task, its COCO-style protocol and the VisDrone-DET2019 results, and UAVDT’s vehicle detection protocol (AP at IoU 0.7), are described in aerial and drone image analysis. The organizers’ journal paper describing the benchmark (Zhu et al., 2022) could not be read for this article.

Detection in video. VisDrone-VID2019 used 79 sequences with 33,366 frames (56 training clips with 24,198 frames, 7 validation with 2,846 and 16 test with 6,322), captured at different locations for the three subsets, in the same ten categories as the image task. Ranking uses average precision averaged over IoU thresholds from 0.50 to 0.95, with AP50, AP75 and average recall at up to 1, 10, 100 and 500 detections per frame. The best of 13 submitted detectors, DBAI-Det, combining Cascade R-CNN, a feature pyramid network and deformable convolutions, reached 29.22% AP on the test set; the organizers’ baselines ranged from 14.46% (Faster R-CNN) to 18.33% (the video detector FGFA). AP was clearly lower for person, bicycle, tricycle, awning-tricycle and motor than for the other five categories, and the organizers conclude that detectors do poorly when several objects appear together, as with a person riding a bicycle; they also report that all the video detection methods ran at fewer than 10 frames per second (Zhu et al., 2019).

What the UAVDT tests add. Du et al. (2018) evaluated detectors, trackers and multi-object trackers by attribute. Detectors did better at low altitude and in side views, where objects show more detail, and worse in daylight than at night because daytime scenes held more objects and more clutter. Region-based detectors, the most accurate, ran at fewer than 5 frames per second, which the authors call too slow for practical use with constrained computing. Most multi-object trackers did better at lower altitude, where fewer objects are in view, and much worse at night, even though the detections given to them at night had good AP.

Drone racing

Delmerico et al. (2019) present drone racing as a setting in which low-level vision is not yet solved. A race course is a sequence of gates; human pilots fly it through a first-person video link, and autonomous systems must do it with onboard sensing and computing. Three systems from the University of Zurich’s Robotics and Perception Group illustrate the approaches.

Learned waypoints. In Deep Drone Racing, Kaufmann et al. (2018) trained a network with DroNet’s ResNet-based architecture to map one 300 × 200 forward image to a direction in normalized image coordinates and a desired speed, which a minimum-jerk planner and a conventional controller turned into motion. Training labels came from a global minimum-snap trajectory through known gate positions, projected into the image; in the real world, the authors carried the quadrotor through the track by hand rather than let a partially trained network fly. The network ran at about 10 frames per second on board. In simulation the method worked reliably up to 9 m/s, and the authors attribute the decline at higher speeds to body rates too large for the prediction rate. On a real four-gate track of 21 m, it beat a baseline that tracked a global trajectory with visual-inertial odometry, whose drift at high speed degraded it, and it coped with gates moved during flight; a professional and an intermediate human pilot flew the track faster but crashed more.

Gate corners and a filter. For the 2019 AlphaPilot Challenge, Foehn et al. (2021) detected the inner corners of every visible gate with a small five-level U-Net that outputs corner maps and part affinity fields linking corners of the same gate, trained on a dataset of 28,000 real images (24,000 for training) annotated by hand with the help of feature tracking. One inference for both forward cameras costs 3.86 GFLOPs and took 10.5 ms on the Jetson Xavier in half precision with TensorRT, so it ran at the 60 Hz camera rate; on a separate test set of 4,000 images the detector reached 96.4% IoU, with corner precision 0.997 and recall 0.918. An extended Kalman filter fused the detections with ROVIO visual-inertial odometry, which ran at about 35 Hz, and with the rangefinder, estimating a gate map and the odometry’s drift. The state estimate carried about 130 ms of latency, which the system compensated by predicting forward with the IMU. The drone completed the 74 m championship course in 11.36 s, reaching 8 m/s; the authors report 100% success at 5 m/s and 60% at 8 m/s, with crashes at higher speeds when odometry tracking failed while no gate was visible; the team ranked second.

Reinforcement learning. Kaufmann et al. (2023) published “Champion-Level Drone Racing Using Deep Reinforcement Learning” in Nature; the paper could not be read for this article. The supplementary data the first author deposited describes recorded positions of human and autonomous drones in head-to-head races and time trials, and pseudocode for gate detection, estimation of odometry drift, policy training and identification of a residual dynamics model (Zenodo record, as of November 2026).

Event cameras

Event cameras, introduced in the robot perception article, report per-pixel brightness changes asynchronously, with microsecond-scale resolution, high dynamic range and reduced motion blur, but need new algorithms (Gallego et al., 2022). On drones they address the two failures of frame cameras in fast flight and poor light. Rosinol Vidal et al. (2018) fused events, frames and IMU data in one optimization and flew a quadrotor with it on an Odroid XU4, which they describe as the first autonomous quadrotor flight using an event camera for state estimation. The aircraft kept flying circles when the room light was switched off and the frames went black, and at a top speed of 1.68 m/s on a circle 1 m above the floor, about 340 pixels per second of optical flow, the frames blurred while the events still gave feature tracks. Hovering exposed the opposite weakness: with almost no motion, events alone drifted, because vibrations kept changing the motion direction and shortened the feature tracks, while frames held the position. On the Event Camera Dataset, the hybrid improved accuracy by 85% over frames with IMU and by 130% over events with IMU. On the easier UZH-FPV sequence an event-based pipeline did not beat the frame-based ones; its authors call the comparison unfair, because the frame-based methods ran on better hardware, with higher resolution, contrast sensitivity and signal-to-noise ratio and more accurate camera–IMU synchronization (Delmerico et al., 2019).

Example: Blur from Translation and Rotation, and Small Targets

The script renders flat ground with small bright targets of known size and position, as seen by a downward-looking pinhole camera with a focal length of 1,000 pixels at five altitudes, and images it while the aircraft hovers, translates at 10 m/s, rotates at 1 rad/s, or both, with a 5 ms exposure. The blur length follows the formula above: translation gives 5 pixels at 10 m and less higher up, rotation 5 pixels at every altitude. Each 192 × 192 image holds 24 targets of 15 cm diameter (contrast 0.10 on ground of mean 0.30 with weak texture at 4–8 cm and smooth variation at 2–5 m), passes through a Gaussian optical blur of 0.5 pixels and gets Gaussian noise of standard deviation 0.02. Two matched filters, written out, look for the targets after a high-pass: one built for a sharp target, one smeared by the known blur, as could be computed from the aircraft’s own velocity, angular rate and altitude (our design, not taken from a source). Each threshold is set on ten target-free images to give one false alarm per image on average, and a detection counts as a hit within half a target diameter plus one pixel of a target’s position at mid-exposure. The table gives recall over 120 targets per case with the range over the five scenes (seeds 0–4), the RMS localization error of hits in centimetres on the ground, and, as a column for interpretation, the ideal matched-filter signal-to-noise ratio of one target from noise-free renders.

Two questions here have answers the sources leave open, so the code prints both: which instant a detection describes (the mean offset of the blur-aware detector’s hits from the target’s position at the start of the exposure is printed in the last column, next to the mid-exposure error), and whether the detector knows the blur. The aerial and drone image analysis example measures how the ground sampling distance affects recall at a fixed false-alarm rate per area, with a translational smear that it finds negligible, and the exposure article’s example trades blur against noise by changing the exposure time; this one fixes the exposure and asks where the blur comes from.

The plan fixed every parameter before the first run, and the script shown is that first run unchanged. It runs in about ten seconds with NumPy 2.3.5 and SciPy 1.16.3, and two runs gave identical output.

import numpy as np
from scipy import ndimage as ndi
from scipy.signal import fftconvolve

F = 1000.0            # focal length in pixels (downward-looking pinhole camera)
N = 192               # image size in pixels
SS = 4                # supersampling per pixel side
PSF = 0.5             # optical blur, Gaussian sigma in pixels
D = 0.15              # target diameter in metres
CONTRAST, NOISE = 0.10, 0.02
T_EXP = 0.005         # exposure time in seconds
MOTIONS = {"hover": (0.0, 0.0), "translate": (10.0, 0.0),   # (v m/s, omega rad/s)
           "rotate": (0.0, 1.0), "both": (10.0, 1.0)}

def scene(seed, H, targets=True):
    """Sharp supersampled image of flat ground seen from height H, plus target centres (px)."""
    rng = np.random.default_rng(seed)
    gsd = H / F
    c = (np.arange(N * SS) + 0.5) / SS                     # pixel coordinates of samples
    x, y = np.meshgrid(c * gsd, c * gsd)                   # ground coordinates (m)
    img = np.full(x.shape, 0.30)
    for _ in range(3):                                     # low-frequency ground (2-5 m)
        kx, ky = 2 * np.pi / rng.uniform(2, 5, 2)
        img += 0.01 * np.sin(kx * x + rng.uniform(0, 6.3)) * np.sin(ky * y + rng.uniform(0, 6.3))
    for _ in range(3):                                     # fine texture (4-8 cm)
        k, a = 2 * np.pi / rng.uniform(0.04, 0.08), rng.uniform(0, np.pi)
        img += 0.01 * np.sin(k * (x * np.cos(a) + y * np.sin(a)) + rng.uniform(0, 6.3))
    cells = [(r, q) for r in range(4) for q in range(6)]
    cy = 20 + (np.array([r for r, q in cells]) + 0.5) * (N - 40) / 4 + rng.uniform(-6, 6, 24)
    cx = 20 + (np.array([q for r, q in cells]) + 0.5) * (N - 40) / 6 + rng.uniform(-6, 6, 24)
    if targets:
        r_px = D / 2 / gsd
        for ty, tx in zip(cy, cx):
            img[(x / gsd - tx) ** 2 + (y / gsd - ty) ** 2 <= r_px ** 2] += CONTRAST
    return img, np.c_[cy, cx]

def box(L):
    """Normalized box kernel of length L samples with fractional end weights."""
    n = int(np.ceil(L)) | 1                                 # odd length
    k = np.clip(L / 2 - (np.abs(np.arange(n) - n // 2) - 0.5), 0, 1)
    return k / k.sum()

def capture(sharp, L, rng):
    """Blur along x by L pixels during the exposure, integrate pixels, optics, noise."""
    img = ndi.convolve1d(sharp, box(L * SS), axis=1, mode="reflect") if L > 0 else sharp
    img = img.reshape(N, SS, N, SS).mean(axis=(1, 3))
    img = ndi.gaussian_filter(img, PSF)
    return img + rng.normal(0, NOISE, img.shape) if rng is not None else img

def template(d_px, L):
    """Disk of diameter d_px through the PSF, optionally smeared by L px along x."""
    h = int(np.ceil(d_px / 2 + L / 2 + 3 * PSF + 2))
    n = 2 * h + 1                                           # odd size: centre on a pixel
    c = (np.arange(-n * SS // 2, n * SS // 2) + 0.5) / SS
    xx, yy = np.meshgrid(c, c)
    t = ((xx ** 2 + yy ** 2) <= (d_px / 2) ** 2).astype(float)
    if L > 0:
        t = ndi.convolve1d(t, box(L * SS), axis=1, mode="constant")
    t = t.reshape(n, SS, n, SS).mean(axis=(1, 3))
    t = ndi.gaussian_filter(t, PSF, mode="constant")
    t -= t.mean()
    return t / np.linalg.norm(t)

def detect(img, tmpl, d_px):
    """High-pass, correlate with the template, return peaks (row, col, response)."""
    hp = img - ndi.gaussian_filter(img, max(3 * d_px, 6.0))
    r = fftconvolve(hp, tmpl[::-1, ::-1], mode="same")
    m = (r == ndi.maximum_filter(r, size=5))
    m[:8], m[-8:], m[:, :8], m[:, -8:] = False, False, False, False
    ys, xs = np.nonzero(m)
    out = []
    for yy, xx in zip(ys, xs):                              # 3-point parabolic sub-pixel fit
        cy_ = r[yy - 1, xx] - 2 * r[yy, xx] + r[yy + 1, xx]
        cx_ = r[yy, xx - 1] - 2 * r[yy, xx] + r[yy, xx + 1]
        dy = 0.5 * (r[yy - 1, xx] - r[yy + 1, xx]) / cy_ if cy_ < 0 else 0.0
        dx = 0.5 * (r[yy, xx - 1] - r[yy, xx + 1]) / cx_ if cx_ < 0 else 0.0
        out.append((yy + 0.5 + dy, xx + 0.5 + dx, r[yy, xx]))   # pixel centres at k + 0.5
    return np.array(out)

def score(dets, truth, thr, tol):
    """Greedy matching by response; returns hit flags per target and errors (px) of hits."""
    dets = dets[dets[:, 2] >= thr]
    dets = dets[np.argsort(-dets[:, 2])]
    free = np.ones(len(truth), bool)
    err = []
    for yy, xx, _ in dets:
        d = np.hypot(truth[:, 0] - yy, truth[:, 1] - xx)
        d[~free] = np.inf
        j = np.argmin(d)
        if d[j] <= tol:
            free[j] = False
            err.append((yy - truth[j, 0], xx - truth[j, 1]))
    return ~free, np.array(err).reshape(-1, 2)

print(" H(m) GSD(cm) D(px) motion     blur(px)  SNR | static: recall [min-max] err(cm)"
      " | blur-aware: recall [min-max] err(cm) | offset from start (cm)")
for H in (10, 20, 40, 80, 120):
    gsd, d_px = H / F, D / (H / F)
    tol = d_px / 2 + 1
    sharp_t = [scene(s, H) for s in range(5)]
    sharp_n = [scene(s, H, targets=False)[0] for s in range(100, 110)]
    for name, (v, w) in MOTIONS.items():
        L = T_EXP * F * (v / H + w)                         # blur length in pixels
        clean = capture(sharp_t[0][0], L, None) - capture(scene(0, H, False)[0], L, None)
        snr = np.sqrt((clean ** 2).sum() / 24) / NOISE      # ideal matched-filter SNR per target
        row = f"{H:5d} {100*gsd:6.0f} {d_px:6.2f} {name:10s} {L:6.2f} {snr:6.1f}"
        for k, tmpl in enumerate((template(d_px, 0), template(d_px, L))):
            rng = np.random.default_rng(1000 * H + list(MOTIONS).index(name))  # same noise for both
            nulls = np.concatenate([detect(capture(s, L, rng), tmpl, d_px)[:, 2] for s in sharp_n])
            thr = np.sort(nulls)[-10]                       # 1 false alarm per image on average
            hits, errs = [], []
            for img, truth in sharp_t:
                h, e = score(detect(capture(img, L, rng), tmpl, d_px), truth, thr, tol)
                hits.append(h); errs.append(e)
            rec = [h.mean() for h in hits]
            e = np.concatenate(errs)
            rms = np.sqrt((e ** 2).sum(axis=1).mean()) * gsd * 100 if len(e) else np.nan
            row += (f" | {np.concatenate(hits).mean():.2f} [{min(rec):.2f}-{max(rec):.2f}]"
                    f" {rms:6.2f}")
        start = np.hypot(e[:, 0], e[:, 1] + L / 2).mean() * gsd * 100 if len(e) else np.nan
        print(row + f" | {start:6.2f}")

Output:

 H(m) GSD(cm) D(px) motion     blur(px)  SNR | static: recall [min-max] err(cm) | blur-aware: recall [min-max] err(cm) | offset from start (cm)
   10      1  15.00 hover        0.00   63.8 | 1.00 [1.00-1.00]   0.18 | 1.00 [1.00-1.00]   0.18 |   0.15
   10      1  15.00 translate    5.00   60.7 | 1.00 [1.00-1.00]   0.26 | 1.00 [1.00-1.00]   0.32 |   2.52
   10      1  15.00 rotate       5.00   60.7 | 1.00 [1.00-1.00]   0.27 | 1.00 [1.00-1.00]   0.33 |   2.50
   10      1  15.00 both        10.00   56.3 | 1.00 [1.00-1.00]   0.51 | 1.00 [1.00-1.00]   0.39 |   5.00
   20      2   7.50 hover        0.00   30.5 | 1.00 [1.00-1.00]   0.27 | 1.00 [1.00-1.00]   0.27 |   0.24
   20      2   7.50 translate    2.50   29.4 | 1.00 [1.00-1.00]   0.31 | 1.00 [1.00-1.00]   0.30 |   2.43
   20      2   7.50 rotate       5.00   27.3 | 1.00 [1.00-1.00]   0.40 | 1.00 [1.00-1.00]   0.34 |   5.01
   20      2   7.50 both         7.50   24.9 | 1.00 [1.00-1.00]   0.65 | 1.00 [1.00-1.00]   0.44 |   7.49
   40      4   3.75 hover        0.00   13.8 | 1.00 [1.00-1.00]   0.72 | 1.00 [1.00-1.00]   0.72 |   0.65
   40      4   3.75 translate    1.25   13.5 | 1.00 [1.00-1.00]   0.77 | 1.00 [1.00-1.00]   0.77 |   2.60
   40      4   3.75 rotate       5.00   10.7 | 1.00 [1.00-1.00]   1.77 | 1.00 [1.00-1.00]   1.38 |  10.01
   40      4   3.75 both         6.25    9.9 | 1.00 [1.00-1.00]   2.66 | 1.00 [1.00-1.00]   1.61 |  12.55
   80      8   1.88 hover        0.00    5.5 | 0.76 [0.67-0.83]   2.69 | 0.76 [0.67-0.83]   2.69 |   2.22
   80      8   1.88 translate    0.62    5.4 | 0.82 [0.71-0.88]   2.41 | 0.82 [0.71-0.88]   2.41 |   3.44
   80      8   1.88 rotate       5.00    3.6 | 0.05 [0.00-0.08]   8.82 | 0.20 [0.12-0.29]   6.82 |  20.04
   80      8   1.88 both         5.62    3.4 | 0.12 [0.08-0.21]   7.27 | 0.26 [0.17-0.50]   5.49 |  23.42
  120     12   1.25 hover        0.00    2.9 | 0.09 [0.04-0.12]   4.13 | 0.09 [0.04-0.12]   4.13 |   3.49
  120     12   1.25 translate    0.42    2.9 | 0.05 [0.00-0.08]   1.95 | 0.05 [0.00-0.08]   1.94 |   3.46
  120     12   1.25 rotate       5.00    1.7 | 0.00 [0.00-0.00]    nan | 0.01 [0.00-0.04]   5.00 |  25.27
  120     12   1.25 both         5.42    1.7 | 0.00 [0.00-0.00]    nan | 0.02 [0.00-0.04]   6.10 |  34.50

Altitude alone. In the hover rows the ideal SNR falls by about half or a little more with each doubling of altitude: the matched-filter SNR grows with the square root of the target’s signal energy, which scales with its area in pixels, and the optical blur spreads a larger share of a small target’s signal (our reading of the code’s construction). Recall stays at 1.00 down to targets 3.75 pixels across and then collapses: 0.76 at 1.9 pixels and 0.09 at 1.25 pixels, where the ideal SNR is 2.9. Our plan expected “recall near 1 down to about 2 px targets”; the output does not bear that out at this noise level.

Translation keeps its ratio to the target. The translational blur shrinks from 5 to 0.42 pixels with altitude, but it is always 5 cm on the ground, a third of the target, so it stretches the target by the same fraction at every altitude and recall barely moves from the hover rows (the differences at 80 and 120 m are within the spread over scenes, since each motion case draws its own noise). The offset of the detections from the start-of-exposure position is about 2.5 cm wherever detections are reliable, half of $v\,t_e$: a detection describes the middle of the exposure.

Rotation does not. The rotational blur of 5 pixels costs almost nothing at 10–40 m, where targets are 15 to 3.75 pixels wide, but at 80 m it spreads a 1.9-pixel target over about seven pixels, the ideal SNR falls from 5.5 to 3.6, and recall drops from 0.76 in hover to 0.05 with the sharp template and 0.20 with the smeared one. In ground units the same 5 pixels grow with altitude, so the offset from the start of the exposure doubles with each doubling of altitude, 2.5, 5.0, 10.0 and 20.0 cm, which is $\omega\,t_e\,H/2$. With these numbers the two sources of blur are equal at $H = v/\omega = 10$ m; above it the angular rate during the exposure, not the speed, sets the blur (our arithmetic from the formula, confirmed by the printed blur lengths).

Knowing the blur helps, within limits. The smeared template never does worse in recall, quadruples it at 80 m with rotation, and lowers the localization error wherever the blur is longer than the target and both detectors have hits, for example 1.38 against 1.77 cm at 40 m with rotation. It cannot restore the signal-to-noise ratio that the blur took: the ideal SNR column already assumes the right template, and at 120 m it is below 3 in every case. Where the target is much larger than the blur, as at 10 m with translation or rotation alone, the smeared template’s error is slightly larger (0.32–0.33 against 0.26–0.27 cm); in our reading, the elongated correlation peak makes the three-point sub-pixel fit along the blur direction less precise.

The example leaves out much of what a real camera does: blur along one axis only, rotation as a uniform image shift, flat ground, white noise, a global shutter, and synthetic disks found by a matched filter rather than objects found by a learned detector. A rolling shutter, which the VINS-Mono README asks users to model with its readout time, would add a different distortion per row.

Common Methods

The systems above combine a few families of methods.

• Geometric state estimation with inertial fusion. Feature-based or semi-direct visual odometry fused with an IMU in a filter or a sliding-window optimization (SVO with an EKF, ROVIO, VINS-Mono), sometimes with a range sensor added, which Falanga et al. (2017) use to estimate the scale of the visual pose and Foehn et al. (2021) fuse in their filter. The Kalman filter family also predicts states forward to cover latency.
• Hand-designed detection of known structure. Thresholding and polygon fitting for markers, PnP for their pose, with RANSAC against outliers (Falanga et al., 2017).
• Small learned perception with conventional control. Compact networks that output a waypoint, gate corners or collision scores, leaving stability to a classical controller; Kaufmann et al. (2018) argue that this lets the network run at a lower rate than the controller, which onboard computing requires, and report that a baseline regressing body rates directly from images never passed more than one gate in simulation.
• Policies trained in simulation, such as the depth-to-trajectory network of Loquercio et al. (2021), described in the obstacle detection article; transfer from simulation in general is the subject of the sim-to-real article.
• Detectors and trackers adapted to small objects. Multi-scale feature pyramids, cascades and high-resolution representations among the top VisDrone video detectors (Zhu et al., 2019); in tracking, magnifying the input images for small targets, low-light enhancement and re-detection after occlusion (Du et al., 2019). Overhead detection with oriented boxes is the subject of aerial object detection.
• New sensors, chiefly event cameras fused with frames and inertial data (Rosinol Vidal et al., 2018).

Challenges

• Compute, payload and power. Every method competes for a computer that the aircraft must lift and power. Delmerico and Scaramuzza (2018) conclude that there is “no free lunch in visual state estimation”: accuracy and robustness can be bought with computation, and on small boards a balance must be found; the most accurate pipeline, VINS-Mono, comes at a cost the user must weigh against the computing left for navigation, control and other perception. On AlphaPilot’s eight-core Jetson Xavier, four cores served the sensor and drone interfaces, and gate detection, odometry, filtering and planning each had one (Foehn et al., 2021). The general techniques are in edge inference, model compression and quantization.
• Latency. Computing AlphaPilot’s state estimate, chiefly interfacing the cameras and running the odometry, added latency of the order of 130 ms (Foehn et al., 2021); Deep Drone Racing’s 30 Hz prediction rate in simulation was not enough at the body rates of high speed (Kaufmann et al., 2018). Real-time vision pipelines covers budgets and timestamps.
• Small objects. Altitude makes targets small: in UAVDT, most tiny objects at high altitude cover 0.005% of a frame, while at low altitude an object may cover 22.6% in an extreme case (Du et al., 2018). The example shows how quickly a matched filter loses targets below a few pixels.
• Motion blur and apparent motion. Fast flight blurs frames and breaks feature tracking (Kaufmann et al., 2018; Rosinol Vidal et al., 2018); flow near the ground grows as $1/H$ (Delmerico et al., 2019). AlphaPilot’s planner allows a speed limit, among other purposes to limit motion blur (Foehn et al., 2021).
• Altitude and viewpoint change. UAVDT’s detectors did better at low altitude and in side views, and its multi-object trackers at lower altitude and in front and side views rather than bird’s-eye views (Du et al., 2018).
• Light and weather. Most UAVDT multi-object trackers did worse at night or in fog than in daylight, and the authors note that night video offers scarcely any texture and fog removes sharp detail (Du et al., 2018); frames go black in the dark (Rosinol Vidal et al., 2018).
• Vibration. In Deep Drone Racing, a handcrafted gate detector baseline sometimes lost the gate because of platform vibrations (Kaufmann et al., 2018); in hover, vibrations shortened event-based feature tracks (Rosinol Vidal et al., 2018).
• Labels. UAVDT’s 0.84 million boxes took more than ten domain experts two months to draw, and regions of vehicles too small to label are marked as ignored (Du et al., 2018); VisDrone’s ground truth also marks ignored regions, as described in aerial image analysis.

Evaluation

State estimation is scored against motion-capture or laser-tracker ground truth after aligning the trajectories; Delmerico and Scaramuzza (2018) report RMSE after Sim(3) alignment and relative errors over segments of 7 to 35 m, the conventions of the absolute trajectory error and EuRoC MAV articles. On a drone, cost belongs in the result: they report per-frame processing time, CPU and memory load next to accuracy, for every computer, and play the data back in real time, to simulate the live stream a flying robot would have. Delmerico et al. (2019) propose reporting a dataset’s difficulty as the distribution of its optical flow, in pixels, normalized by the image diagonal and normalized by the focal length, rather than by vehicle speed.

Detection and tracking use the protocols of their benchmarks: COCO-style AP over IoU 0.50–0.95 with recall at up to 500 detections per frame (VisDrone-VID), AP at IoU 0.7 (UAVDT), and success AUC with precision at 20 pixels (UAVDT and VisDrone single-object tracking). Results should be compared only within one protocol and test set; the VisDrone test-challenge annotations are withheld, and test-dev annotations are released for papers (VisDrone README, as of November 2026). Speed should be reported with its hardware: the UAVDT and VisDrone papers report frames per second on desktop hardware, not on aircraft.

Closed-loop tasks are scored by outcome: task completion (the fraction of gates passed, with full completion at five laps in simulation) and success rate with best lap time on a real track (Kaufmann et al., 2018); completion time and success rate at each speed (Foehn et al., 2021); or, for landing, demonstrations with recorded trajectories (Falanga et al., 2017; Forster et al., 2015). These depend on the course and conditions as much as on the method.

Failure Modes and Risks

The documented failures include:

• Odometry drift at speed, which degraded VIO-based baselines in Deep Drone Racing and, combined with gates out of view, caused AlphaPilot’s crashes at higher speeds (Kaufmann et al., 2018; Foehn et al., 2021), and drift on fast UZH-FPV sequences (Delmerico et al., 2019).
• No depth where there is no texture: untextured and reflecting surfaces and moving objects left holes in the landing elevation map, which the method treats as unsafe (Forster et al., 2015).
• Ambiguity and occlusion: Deep Drone Racing’s network coped with up to about half of a gate occluded, but with only one edge visible it could not tell top from bottom (Kaufmann et al., 2018).
• Long-term drift of trackers on long drone sequences (Du et al., 2019), and poor tracking under background clutter and large occlusion (Du et al., 2018).
• Appearance outside the training data: Kaufmann et al. (2018) expect performance to degrade when the environment changes substantially beyond what was seen in training.
• Privacy and misuse. Camera drones photograph people from above; the benchmark creators list security and surveillance among drone applications (Du et al., 2018; VisDrone README). This article gives no guidance for such uses, and the privacy and safety points of mapping flights are discussed with aerial image analysis.

Deployment Considerations

The engineering lessons the sources draw concern perception systems, not the operation of aircraft:

• Budget computation for the whole system. State estimation, detection, planning and control share one computer; the choice of estimator depends on how much of it may be used (Delmerico & Scaramuzza, 2018), and network size is chosen against inference time on the target hardware (Foehn et al., 2021).
• Timestamp and compensate. AlphaPilot time-stamped all sensor data globally on reception and predicted the delayed state forward with the IMU (Foehn et al., 2021); VINS-Mono’s README asks for accurate timestamps and offers online camera–IMU time-offset estimation for unsynchronized sensors. The example shows that, for motion at constant speed, a detection describes the middle of its exposure.
• Prefer global-shutter cameras where geometry matters, or model the rolling shutter: VINS-Mono’s README ranks rolling-shutter cameras with unsynchronized IMUs last in expected performance.
• Evaluate on the hardware and speeds of use. Results on desktop computers and slow trajectories do not transfer to flying robots (Delmerico & Scaramuzza, 2018; Delmerico et al., 2019).

Flight rules differ by country. The United States and European Union rules, as their official texts state them, are outlined in the aerial and drone image analysis article.