Obstacle Detection and Avoidance

Obstacle detection and avoidance is the task of finding, in a robot’s sensor data, the things it could hit, and of turning that knowledge into motion that does not hit them. This article is about doing it with cameras on mobile robots and drones: estimating how far away obstacles are, or how soon they will be reached, from stereo pairs, single images or the motion of the image, and predicting collisions directly from pixels with learned models. It is also called collision avoidance, obstacle sensing, or, when the image expansion of an approaching surface is the cue, looming detection. The perception side is the subject here. The planners and controllers that consume it appear only as far as their inputs shape what perception must deliver; how a robot reaches a goal through a building, with maps, planners and learned policies, is the subject of visual navigation, and the sensing and state estimation that every robot needs are surveyed in robot perception.

The task predates deep learning by decades. Moravec’s Stanford Cart triangulated features to plan around obstacles in 1980, as the robot perception article describes; flow-based robots followed in the 1990s, and learned collision predictors trained on driving, crashing or simulated flight in the 2010s. What has not changed is that an obstacle must be detected early enough, at the robot’s speed and with its processing delay, for the avoidance to finish in time.

Problem Definition

At each time step the robot receives images, often with inertial data, and must produce one of two things: a representation of nearby obstacles that a planner can use, or a command that avoids them. The representations differ in what they claim to know:

• Metric geometry. A depth map, point cloud or occupancy grid gives the distance to each visible surface, in metres. It needs stereo, an active sensor, or a learned monocular depth model with known scale.
• Time to contact. For each region of the image, the time until the camera would reach the surface if the current relative motion continued. A single camera can measure it without knowing distance or speed, as the next sections show.
• Collision probability. A learned score that the current view, or a candidate direction of motion, leads to a collision. It has no physical units and is defined by how its training labels were made.

A robot that avoids obstacles reactively works from the current observation, or a short history, and does not keep a map. One that plans keeps a map, usually an occupancy grid built from depth, and plans paths through it, as the map-based pipelines of the visual navigation article do. Both need the same thing from perception: obstacles found reliably, with few false alarms, early enough to act.

Input and Output

• Input: a monocular image stream, a stereo pair, or an RGB-D frame, often with the robot’s velocity or inertial measurements. Some methods need calibration (stereo, metric depth); time to contact from image expansion does not need the focal length (Horn et al., 2007).
• Output: a depth or disparity map, an occupancy grid, a map of time to contact, a collision probability, or directly a steering and speed command. Learned systems such as DroNet output a steering angle and a collision probability for each frame (Loquercio et al., 2018); others output a set of short trajectories with predicted costs (Loquercio et al., 2021).

Variants

• Ground robots and drones. Wheeled robots move in a plane and can stop; multicopters move in three dimensions, fast, with little payload for sensors and computers, so latency and weight dominate their designs (see the aerial robots section of the robot perception article).
• Static and dynamic obstacles. Walls and furniture stay where they are; people, vehicles and other robots move, and their motion must be predicted or reacted to quickly.
• Reactive and map-based avoidance. Reactive methods map the current observation to a command; map-based ones accumulate observations, which rejects noise at the cost of latency (Loquercio et al., 2021).
• Cars. Where a vehicle may drive, as opposed to what it might hit, is the subject of drivable area segmentation, which also covers stereo v-disparity obstacle detection on roads; detecting cars and pedestrians as objects belongs to the autonomous driving perception stack.

Challenges

• Range and resolution. Stereo depth error grows roughly with the square of distance (depth estimation), so an obstacle is measured coarsely when it is first seen and accurately only when it is close.
• Latency. The time from exposure to command decides how far the robot travels before it reacts. Loquercio et al. (2021) derive a theoretical maximum speed for each method from the sensing range, the camera’s frame interval and the processing latency.
• Obstacles near the direction of travel. The obstacle the robot is heading for is the one whose image moves least, because image motion vanishes at the focus of expansion. Coombs et al. (1995) note that an object near the focus of expansion generates less optical flow than an object at the same range off to the side.
• Texture. Stereo matching and optical flow need image structure to match; a blank wall gives neither.
• Thin, transparent and reflective obstacles. Wires, branches and glass return little or misleading signal to cameras and depth sensors (see Documented Limits below).
• Motion blur and lighting. Fast motion blurs the image; moving from indoors to outdoors changes the brightness range abruptly (Loquercio et al., 2021).
• Data for learning. Collisions are what a learned system most needs to see and what is most costly to record on a real robot (Loquercio et al., 2018; Sadeghi & Levine, 2017).

Approaches

Depth from stereo and single images

Many obstacle detectors on robots measure depth and treat anything close enough, above the ground, as an obstacle. Stereo matching gives metric depth wherever matching succeeds, monocular depth estimation gives depth from one image with the scale problems that article describes, and active depth cameras and LiDAR measure it directly. Those articles cover the methods; what matters here is how far the measurement can be trusted. Loquercio et al. (2021) define the sensing range of a stereo camera as the distance at which the triangulation uncertainty stays below 20% of the depth; for their simulated stereo camera (VGA resolution, 0.1 m baseline, 4 mm focal length) that range is 6 m. The depth images that stereo sensors produce in the real world have missing values and noise, and their simulated stereo depth, computed by semi-global matching, reproduced those failure cases (Loquercio et al., 2021, Fig. 5).

Depth is then turned into obstacles. On roads, the ground plane is found in the disparity image and everything that rises above it is a potential obstacle, as the v-disparity method in the drivable area article shows. Indoor robots and drones usually project depth into an occupancy grid or a 3D map; the visual navigation article’s example does this with a depth scan.

Time to contact

The time to contact (TTC), also called time to collision, is the time that would elapse before the camera’s centre of projection reached the surface it is looking at, if the current relative motion continued unchanged (Horn et al., 2007). For a surface at distance $Z$ approaching at speed $-W = -\mathrm{d}Z/\mathrm{d}t$,

$$T = -\frac{Z}{\mathrm{d}Z/\mathrm{d}t}.$$

Neither $Z$ nor $W$ can be recovered from a single camera without extra knowledge, but their ratio can. If an object of size $S$ lies perpendicular to the optical axis and its image has size $s$, the pinhole camera model gives $sZ = fS$, and differentiating gives

$$T = \frac{s}{\mathrm{d}s/\mathrm{d}t},$$

the ratio of the image size to its rate of growth (Horn et al., 2007, Eqs. 1–3). Time to contact is therefore a property of the image alone, measured in frames when the frame interval is the unit of time, and needs no calibration. In the study of perception, Lee (1976) proposed a theory of the visual control of braking based on information about time-to-collision, as its title states; the paper could not be read for this article.

Measuring $s$ and its change directly is hard. Horn et al. (2007) give the arithmetic: at a time to contact of 100 frames, a 100-pixel image grows by 1 pixel per frame, so a 10% accuracy needs the size measured to a tenth of a pixel. Methods instead use the whole image motion, either through optical flow or directly from brightness derivatives.

From flow divergence. When the camera translates without rotating, flow vectors radiate from the focus of expansion (FOE), the image of the direction of travel. For a fronto-parallel surface the flow is $u = -(W/Z)(x - x_0)$, $v = -(W/Z)(y - y_0)$, so it is a uniform expansion about the FOE $(x_0, y_0)$ whose rate is the inverse time to contact (Horn et al., 2007, Eq. 10). The divergence of the flow field, $\partial u/\partial x + \partial v/\partial y$, then equals $2/T$. Coombs et al. (1995) derive the same relation from the flow of a general camera motion, at the image centre and for a surface roughly perpendicular to the line of sight, and stress the property that made it attractive for robots: divergence is theoretically unaffected by camera rotation, which no mechanical stabilization removes completely. Time to contact is then twice the inverse of the divergence.

Directly from brightness derivatives. Horn, Fang and Masaki (2007) skip flow altogether. Substituting the motion field into the brightness constancy constraint $uE_x + vE_y + E_t = 0$ (the starting point of Horn–Schunck flow) gives, for translation along the optical axis towards a fronto-parallel plane,

$$C\,G + E_t = 0, \qquad G = xE_x + yE_y, \qquad C = -\frac{W}{Z} = \frac{1}{T},$$

where $G$ is the “radial gradient” and image coordinates are measured from the principal point. Least squares over a region gives

$$C = -\frac{\sum G E_t}{\sum G^2},$$

a sum of products of image derivatives and coordinates, with no feature detection, tracking or flow (their case A, Eq. 16). When the translation has an unknown sideways component, the constraint becomes $A E_x + B E_y + C G + E_t = 0$ with $A = fU/Z$ and $B = fV/Z$; three linear equations give $A$, $B$ and $C$, and the FOE follows as $x_0 = -A/C$, $y_0 = -B/C$ (case B, Eqs. 18–22). A third case handles a tilted plane approached along the optical axis, and the general case needs non-linear optimization. The focal length is never needed.

Horn et al. tested the method on synthetic sequences made by shifting and magnifying a single image, on stop-motion sequences of toy cars and on video from a car. On the synthetic sequences the estimates were biased towards longer times to contact than the truth, by up to 20% for long times and less than 10% for short ones, and the bias dropped with 2 × 2 block averaging. Near the end of each sequence results became unreliable, which they attribute mainly to temporal aliasing: as with other gradient-based methods, results degrade once the frame-to-frame motion exceeds about half the size of typical texture elements, or, as a rule of thumb, about a pixel. Averaging and sub-sampling enlarge the effective pixel: on 480 × 640 images with the FOE at the centre, 4 × 4 blocks gave reliable estimates down to a time to contact of about 60 frames and 16 × 16 blocks down to less than 15 frames, while long times to contact became less accurate, a trade-off they propose to handle with spatial sub-sampling for short times and temporal sub-sampling for long ones. On the car video, where ground truth was unknown, the estimates agreed with manual estimates from object sizes but tended to be lower, because faster-moving image regions belonging to nearby objects contributed to the sum (Horn et al., 2007).

Flow balancing and the centering response

A second use of flow ignores the direction of travel and compares the two sides of the visual field. When a camera translates past walls, the image speed of a wall point is inversely proportional to its distance, so the side with the faster flow is the nearer one. Kirchner and Srinivasan (1989) trained honeybees to fly through a 12 cm wide corridor lined with gratings. With both walls still, the bees flew along the midline; when one grating moved, their path shifted off centre, and the authors proposed that a bee balances the image speeds seen by its two eyes. This account is described here through Serres et al. (2008), who summarize it and the later corridor experiments.

Robots have used the same comparison of the two sides. Coombs et al. (1995) built a mobile robot that used only optical flow: a narrow central camera (40° field of view) computed normal flow and its divergence in three overlapping windows to estimate time to contact, and a wide peripheral camera (115°) compared the largest flow in the left and right periphery to steer away from the nearer side. Active gaze control kept the cameras rotationally stabilized, so that their motion was close to a pure translation and flow reflected distance. When the central time to contact predicted an imminent collision at a dead end, the robot stopped, turned and resumed wandering. Single estimates were too noisy to act on: a recursive least-squares fit of a linear model to the sequence of estimates, and a rule combining a short predicted time to contact, a decreasing trend and enough contributing flow pixels, served as the collision indicator. The robot wandered around a laboratory containing office furniture and robot and computing equipment, with office chairs as obstacles, at 30 cm/s for up to 20 minutes without collision. Its failures had three causes: a variable delay between detection and response from processes running under Unix, steering towards a blank wall that produced no flow, and its rectangular base brushing bumpers against nearby objects while turning (Coombs et al., 1995).

The insect account itself has been revised. Serres et al. (2008) trained bees in a 95 cm wide corridor, wider than in the earlier studies, and found that bees entering and feeding near one wall flew along that wall rather than centering, and kept doing so when 150 cm of the opposite wall was removed. They conclude that optic-flow balancing cannot explain this and account for both centering and wall-following with a lateral optic flow regulator, a feedback loop that holds the flow on one side constant.

Example: time to contact from image expansion

The script renders approach sequences with known ground truth and estimates time to contact with Horn et al.’s direct method. A pinhole camera ($f = 100$ px, 128 × 128 pixels, principal point at the centre) approaches a fronto-parallel plane textured with scikit-image’s bundled camera image, sampled so that one texel covers one pixel at the start; as the plane comes closer the texture is magnified, as in Horn et al.’s synthetic sequences. Each frame receives Gaussian sensor noise with a standard deviation of one grey level. Derivatives are averaged over a 2 × 2 × 2 cube of pixels from two frames, as Horn et al. did; unlike them, no threshold on $E_t$ is applied. Because Horn et al. report that block averaging changes the range of reliable estimates, every estimate is printed without averaging ($b = 1$) and with 4 × 4 block averaging ($b = 4$). Part 1 sweeps the true time to contact from 99.5 to 6.5 frames. Part 2 holds it at 29.5 frames and varies the motion and the texture: a head-on approach; an FOE 30 pixels right of the centre; a sideways component that puts the FOE at 90 pixels, outside the image, so that lateral image motion dominates; and a weak texture, the same image blurred with a Gaussian of 8 texels. Part 3 is an obstacle detector: a textured square 3 m ahead, in front of a wall at 20 m, with the time to contact estimated in each of 64 blocks of 16 × 16 pixels and blocks below 60 frames flagged. Seeds 0–4 draw the noise. The example runs in about two seconds with NumPy 2.3.5, SciPy 1.16.3 and scikit-image 0.25.2.

import numpy as np
from scipy import ndimage
from skimage import data, color

F, N = 100.0, 128                      # focal length (px), image size (px)
c = (N - 1) / 2                         # principal point at the image centre
ys, xs = np.mgrid[0:N, 0:N].astype(float) - c

cam = data.camera().astype(float)                               # 512 x 512
astro = 255 * color.rgb2gray(data.astronaut())
smooth = ndimage.gaussian_filter(cam, 8)                        # weak texture

def plane(tex, s, Z, X0=0.0):
    """Image of a fronto-parallel plane at depth Z (m), camera shifted by X0 (m).
    One texel of `tex` is s metres; the texture centre lies on the optical axis."""
    tc = (np.array(tex.shape) - 1) / 2
    rows = (ys * Z / F) / s + tc[0]
    cols = (xs * Z / F + X0) / s + tc[1]
    return ndimage.map_coordinates(tex, [rows, cols], order=3, mode="mirror")

def derivatives(E0, E1):
    """Horn-Schunck estimates of Ex, Ey, Et on a 2x2x2 cube of two frames."""
    s = E0 + E1
    Ex = (s[:-1, 1:] - s[:-1, :-1] + s[1:, 1:] - s[1:, :-1]) / 4
    Ey = (s[1:, :-1] - s[:-1, :-1] + s[1:, 1:] - s[:-1, 1:]) / 4
    d = E1 - E0
    Et = (d[:-1, :-1] + d[:-1, 1:] + d[1:, :-1] + d[1:, 1:]) / 4
    return Ex, Ey, Et

def block(E, b):
    return E.reshape(E.shape[0] // b, b, E.shape[1] // b, b).mean(axis=(1, 3))

def ttc(E0, E1, b=1, case="A", region=None):
    """Time to contact in frames from two frames (Horn, Fang & Masaki 2007).
    Case A: FOE at the principal point, C = -sum(G Et) / sum(G^2).
    Case B: unknown FOE, least squares for A, B, C in A Ex + B Ey + C G + Et = 0."""
    E0, E1 = block(E0, b), block(E1, b)
    n = E0.shape[0]
    y, x = np.mgrid[0:n - 1, 0:n - 1] * b + b / 2 + (b - 1) / 2 - c  # cube centres
    Ex, Ey, Et = derivatives(E0, E1)
    Ex, Ey = Ex / b, Ey / b                          # gradients per original pixel
    G = x * Ex + y * Ey
    m = np.ones_like(G, bool) if region is None else region(x, y)
    if case == "A":
        C = -(G[m] * Et[m]).sum() / (G[m] ** 2).sum()
        return 1 / C, np.nan
    M = np.stack([Ex[m], Ey[m], G[m]], axis=1)
    A, B, C = np.linalg.lstsq(M, -Et[m], rcond=None)[0]
    return 1 / C, -A / C                             # TTC, FOE x-coordinate

def pair(tex, s, Z0, W, U, k, seed):
    """Frames k and k+1 of an approach at W m/frame with lateral motion U m/frame."""
    rng = np.random.default_rng(seed)
    f = [plane(tex, s, Z0 - W * t, U * t) for t in (k, k + 1)]
    return [im + rng.normal(0, 1.0, im.shape) for im in f]       # sensor noise

def summary(vals):
    v = np.array(vals)
    return f"{np.median(v):7.1f} ({v.min():6.1f} to {v.max():6.1f})"

seeds = range(5)
print("1. Textured plane, head-on approach: estimated TTC, frames, median (min to max) over seeds 0-4")
print(f"{'true':>6} {'case A, b=1':>26} {'case A, b=4':>26}")
Z0, W = 5.0, 0.05                                  # TTC = 100 frames at frame 0
for true in (99.5, 79.5, 59.5, 39.5, 29.5, 19.5, 14.5, 9.5, 6.5):
    k = int(100 - 0.5 - true)
    fr = [pair(cam, Z0 / F, Z0, W, 0.0, k, sd) for sd in seeds]
    row = [summary([ttc(*p, b=b)[0] for p in fr]) for b in (1, 4)]
    print(f"{true:6.1f} {row[0]:>26} {row[1]:>26}")

print("\n2. True TTC 29.5 frames: estimated TTC, median (min to max) over seeds 0-4; FOE x in px (true in brackets)")
cases = [("textured, head-on", cam, 0.0, 0.0),
         ("textured, FOE at +30 px", cam, 0.30 * W, 30.0),
         ("textured, FOE at +90 px (lateral)", cam, 0.90 * W, 90.0),
         ("weak texture, head-on", smooth, 0.0, 0.0)]
k = int(100 - 0.5 - 29.5)
for name, tex, U, foe in cases:
    fr = [pair(tex, Z0 / F, Z0, W, U, k, sd) for sd in seeds]
    print(f"{name} [FOE {foe:+.0f}]")
    for b in (1, 4):
        a = [ttc(*p, b=b, case="A")[0] for p in fr]
        bb = [ttc(*p, b=b, case="B") for p in fr]
        print(f"  b={b}  A: {summary(a)}   B: {summary([t for t, _ in bb])}"
              f"   B's FOE x: {np.median([x0 for _, x0 in bb]):+7.1f}")

print("\n3. Obstacle 3 m ahead (TTC 29.5 at the pair midpoint) in front of a wall at 20 m (TTC 199.5)")
Zo, Zb, h, W3 = 3.0, 20.0, 0.96, 0.1
def scene(t):
    Zt_o, Zt_b = Zo - W3 * t, Zb - W3 * t
    obst = (np.abs(xs * Zt_o / F) < h) & (np.abs(ys * Zt_o / F) < h)
    return np.where(obst, plane(cam, Zo / F, Zt_o), plane(astro, Zb / F, Zt_b)), obst
truth = scene(0)[1].reshape(8, 16, 8, 16).mean(axis=(1, 3)) > 0.5  # 16x16 blocks
for sd in seeds:
    rng = np.random.default_rng(sd)
    E0, E1 = [scene(t)[0] + rng.normal(0, 1.0, (N, N)) for t in (0, 1)]
    grid = np.full((8, 8), np.nan)
    for i in range(8):
        for j in range(8):
            r = lambda x, y, i=i, j=j: ((y + c) // 16 == i) & ((x + c) // 16 == j)
            grid[i, j] = ttc(E0, E1, region=r)[0]
    det = (grid > 0) & (grid < 60)
    if sd == 0:
        print("per-block TTC (case A, b=1), seed 0:")
        for row in grid:
            print("  " + " ".join(f"{v:6.0f}" for v in row))
        print(f"whole-image TTC, seed 0: {ttc(E0, E1)[0]:.1f}")
    print(f"seed {sd}: blocks flagged (0 < TTC < 60): {det.sum():2d}, "
          f"obstacle blocks found {np.sum(det & truth)}/16, background blocks flagged {np.sum(det & ~truth)}/48")

1. Textured plane, head-on approach: estimated TTC, frames, median (min to max) over seeds 0-4
  true                case A, b=1                case A, b=4
  99.5   115.1 ( 115.0 to  115.2)   147.2 ( 146.3 to  147.8)
  79.5    84.6 (  84.5 to   84.7)    96.7 (  96.3 to   96.8)
  59.5    60.1 (  60.0 to   60.1)    66.4 (  66.3 to   66.5)
  39.5    39.7 (  39.6 to   39.7)    43.9 (  43.8 to   43.9)
  29.5    29.5 (  29.4 to   29.5)    31.5 (  31.4 to   31.5)
  19.5    20.3 (  20.1 to   20.3)    20.0 (  19.9 to   20.1)
  14.5    62.4 (  61.0 to   63.6)    14.8 (  14.5 to   15.2)
   9.5   339.8 ( 315.0 to  353.4)    10.6 (  10.4 to   11.1)
   6.5   397.4 ( 349.6 to  413.7)     7.7 (   7.7 to    8.1)

2. True TTC 29.5 frames: estimated TTC, median (min to max) over seeds 0-4; FOE x in px (true in brackets)
textured, head-on [FOE +0]
  b=1  A:    29.5 (  29.4 to   29.5)   B:    30.5 (  30.5 to   30.6)   B's FOE x:    -0.2
  b=4  A:    31.5 (  31.4 to   31.5)   B:    32.4 (  32.3 to   32.5)   B's FOE x:    -1.2
textured, FOE at +30 px [FOE +30]
  b=1  A:    69.5 (  69.3 to   69.6)   B:    30.4 (  30.4 to   30.5)   B's FOE x:   +29.8
  b=4  A:    78.5 (  78.1 to   78.7)   B:    33.2 (  33.2 to   33.2)   B's FOE x:   +31.2
textured, FOE at +90 px (lateral) [FOE +90]
  b=1  A:    19.5 (  19.5 to   19.5)   B:    32.7 (  32.6 to   32.7)   B's FOE x:   +79.7
  b=4  A:    15.8 (  15.8 to   15.9)   B:    29.2 (  29.0 to   29.3)   B's FOE x:   +90.6
weak texture, head-on [FOE +0]
  b=1  A:    91.1 (  89.1 to   92.6)   B:   104.7 ( 101.8 to  106.5)   B's FOE x:   -20.6
  b=4  A:    29.7 (  29.3 to   30.0)   B:    30.0 (  29.5 to   30.5)   B's FOE x:    -0.2

3. Obstacle 3 m ahead (TTC 29.5 at the pair midpoint) in front of a wall at 20 m (TTC 199.5)
per-block TTC (case A, b=1), seed 0:
     485    429    491    676    246    218    217    561
     353    104     40     32     54    177    231    595
     321     64     29     32     30     29    198    214
     215     76     33     33     32     31    130    255
     208     54     38     36     79     30    214    210
     229    196     32     33     34     31    169    223
     234    255    207    206    214    143    231    215
     236    249    245    258    267    230    212    227
whole-image TTC, seed 0: 141.5
seed 0: blocks flagged (0 < TTC < 60): 19, obstacle blocks found 15/16, background blocks flagged 4/48
seed 1: blocks flagged (0 < TTC < 60): 19, obstacle blocks found 15/16, background blocks flagged 4/48
seed 2: blocks flagged (0 < TTC < 60): 19, obstacle blocks found 15/16, background blocks flagged 4/48
seed 3: blocks flagged (0 < TTC < 60): 19, obstacle blocks found 15/16, background blocks flagged 4/48
seed 4: blocks flagged (0 < TTC < 60): 21, obstacle blocks found 16/16, background blocks flagged 5/48

Part 1 reproduces the trade-off Horn et al. describe. Without averaging, the estimate is within 5% of the truth from 59.5 down to 19.5 frames and too long by 16% at 99.5 frames, the direction of bias they report. At 14.5 frames it collapses: the true expansion moves points at the image edge by $64/14.5 \approx 4.4$ pixels per frame, the linear brightness constraint no longer holds, and the method returns times to contact four to sixty times too long, which would make an obstacle seem far away in time. With 4 × 4 averaging, the effective pixel is four times larger, so the 9.8 pixels of edge motion at 6.5 frames are about 2.5 averaged pixels, and the estimate is within 20% there; the price is a larger bias at long times, 48% at 99.5 frames. The narrow ranges show that noise of one grey level hardly matters for the full-contrast texture: the errors are systematic rather than random, and in our reading they are set by the geometry and the sampling.

Part 2 shows what each assumption costs. Case A assumes the FOE at the principal point. When it lies 30 pixels to the side, part of the expansion appears as a sideways shift that $G$ does not model, and case A more than doubles the time to contact; case B, which estimates the FOE, recovers the time to contact within 4% and the FOE within 0.2 pixels at full resolution (13% and 1.2 pixels with averaging). When lateral motion dominates, case A errs the other way, giving 19.5 and 15.8 frames for a true 29.5, while case B stays within 11% without averaging and within about 1% with it, placing the FOE at 79.7 and 90.6 pixels. The weak texture is a different failure. Without averaging, both cases return three to three and a half times the true time to contact. In our reading this is the familiar effect of noise in a regressor: the blurred texture has weak gradients, so the noise in $G$ is large relative to them; it enlarges the denominator $\sum G^2$ but averages out of the numerator, which pulls $C$ towards zero and the time to contact towards infinity. Averaging 4 × 4 blocks divides the noise by four while leaving the smooth texture almost untouched, and the estimates return to within 2%. A surface with no texture at all, where $\sum G^2$ would be zero for any block size, gives no estimate; that is the blank wall on which Coombs et al.’s robot failed.

Part 3 turns the estimator into a detector. The obstacle’s blocks read about 30 frames and the wall’s about 200, but noisier, because at a time to contact of 200 frames the wall moves by a small fraction of a pixel per frame. The whole-image estimate of 141.5 frames is the warning: summed over the image, the wall dominates, and the obstacle straight ahead nearly disappears into a reassuring average. In our reading the wall wins because it covers three quarters of the pixels and the outer ones, where $G$ is largest because it grows with distance from the centre, all belong to it. Per block, 15 of the 16 obstacle blocks are found with seeds 0–3 and all 16 with seed 4. The missed block (79 frames with seed 0) touches the image centre, where $x$ and $y$, and with them $G$, are smallest; in our reading that weak signal is why it fails, although the three other blocks touching the centre were found. It is the FOE problem of the Challenges section in miniature. The background blocks flagged with seed 0 all border the obstacle. By construction the obstacle’s edges lie on block boundaries in the first frame and move outward by about a pixel in the second, so the neighbouring blocks contain an occluding edge that violates the single-plane model. The example leaves out camera rotation, which Coombs et al. removed by gaze stabilization, independently moving obstacles, motion blur, and textures other than two bundled photographs.

Learned collision prediction

Learning-based methods replace explicit geometry with a network that maps images to collision scores or commands. Their central difficulty is data: a network that must predict collisions needs examples of them.

Learning from crashes. The paper by Gandhi, Pinto and Gupta (2017), titled “Learning to Fly by Crashing”, could not be read for this article, and it is described here only as DroNet’s authors used it. DroNet’s authors write that collision trajectories, which expert human pilots avoid, are needed for a robot to learn how to behave in dangerous situations, citing Gandhi et al., and they implemented Gandhi et al.’s strategy of flying in the direction that minimizes the predicted collision probability as a baseline, with DroNet’s architecture and collision data, calling it a strong baseline designed for narrow indoor spaces (Loquercio et al., 2018).

DroNet. Loquercio et al. (2018) avoided crashing drones by learning from ground vehicles. DroNet is an 8-layer residual network (ResNet) that takes a single 200 × 200 grey-scale frame and outputs a steering angle and a collision probability from two heads on a shared trunk. Steering was learned from more than 70,000 images of the Udacity car-driving dataset; collision probability from about 32,000 images in 137 sequences recorded with a GoPro on a bicycle handlebar, starting far from an obstacle and ending very close to it, with frames labelled manually as 0 (far) or 1 (very close). The authors note that the collision output is not a Bayesian probability but an index of the network’s uncertainty, and conjecture that the network judges distance to the objects in view. The outputs drive the drone directly: forward speed is a low-pass-filtered version of $(1 - p_t)V_{\max}$, so the drone slows as the collision probability $p_t$ rises and stops when it approaches 1, and the steering output sets the yaw. On the test images of their collision dataset, DroNet reached a collision-classification accuracy of 95.4% and an F1 score of 0.901, against 96.6% and 0.921 for a ResNet-50 that is about 80 times larger, running at 20 frames per second on an Intel Core i7 CPU that received the drone’s images over Wi-Fi (Loquercio et al., 2018, Table I). In flight tests measured by the distance travelled before stopping or colliding, DroNet went farthest in five of six environments, including an indoor garage visually different from its training data, while the collision-minimizing baseline went farther in an indoor corridor (31 m against 27 m; Table II). The authors expect the policy to fail where the “line-like” features it attends to are missing, as in a forest without a visible path, where it did (Loquercio et al., 2018). The official code, collision dataset and trained weights are linked from the project page and repository (as of November 2026).

Learning in simulation. Sadeghi and Levine (2017) trained a collision-avoidance policy for indoor flight without a single real image. CAD2RL divides the image into a 41 × 41 grid of bins, each standing for a flight direction through it, and a fully convolutional network predicts, for every bin, a value learned from Monte Carlo rollouts in simulated hallways: the long-horizon chance of flying that way without collision. It was first pretrained to predict whether a 1 m ray cast in each direction hit an obstacle, which amounts to thresholding the depth map at 1 m, and then refined with the rollouts, because free space 1 m ahead does not decide whether a collision follows later, for example in a narrow dead end. Textures, lighting and furniture placement were randomized so heavily that the network had to rely on geometry rather than appearance. In real hallway-following tests in two Berkeley buildings, it collided 6 times in 164 m and 4 times in 55 m, against 13 and 7 for the free-space predictor alone over similar distances (Sadeghi & Levine, 2017, Table I). Randomizing simulation to bridge the gap to the real world is the technique of domain randomization; sim-to-real transfer describes CAD2RL’s randomized hallways and its ablations in detail.

From depth to trajectories. Loquercio et al. (2021) moved the learned part from pixels to depth. Their network takes a stereo depth image, the drone’s estimated velocity and attitude and a desired flight direction, and outputs several short trajectories with predicted costs, the cheapest of which is executed; it was trained in simulation to imitate a sampling-based planner with privileged access to the full point cloud and the drone’s exact state. Depth was chosen as an input abstract enough for simulated and real observations to look alike, and training on simulated stereo depth taught the policy the artefacts of real depth sensors, such as missing values and discretization. In 31 real-world runs in forests and on snowy mountain terrain, with a 40 m straight or 6 m-radius circular reference path that ran through obstacles, the drone completed every run at average speeds of 3 and 5 m/s, 8 of 10 at 7 m/s and 60% at 10 m/s. A simulated experiment, avoiding a pole 6 m ahead, isolates perception. With ground-truth depth all methods succeeded up to 5 m/s; with stereo depth, a map-based planner failed from 5 m/s on, because filtering noisy depth over two or three observations delayed obstacles entering its map, a reactive motion-primitive planner lost 30% success at 7 m/s, and the learned policy lost 10% at 10 m/s and nothing at lower speeds (Loquercio et al., 2021). Learning control from images in general, beyond collision avoidance, is the subject of visuomotor policy learning, and systems that map sensors directly to driving commands are discussed under end-to-end autonomous driving.

From perception to avoidance

The avoidance step decides how much perception must deliver, and in what form.

Potential fields. Khatib (1986) treated collision avoidance as a low-level control problem rather than a planning one: the robot moves in a field of forces in which the goal is an attractive pole and obstacles are repulsive surfaces. The attractive potential is $\tfrac{1}{2}k_p(\mathbf{x} - \mathbf{x}_d)^2$ for goal $\mathbf{x}_d$, and the repulsive one, built on an obstacle description $f(\mathbf{x}) = 0$, is

$$U_O(\mathbf{x}) = \begin{cases} \dfrac{1}{2}\,\eta\left(\dfrac{1}{f(\mathbf{x})} - \dfrac{1}{f(\mathbf{x}_0)}\right)^{2} & \text{if } f(\mathbf{x}) \le f(\mathbf{x}_0), \\[1ex] 0 & \text{if } f(\mathbf{x}) > f(\mathbf{x}_0), \end{cases}$$

so that the repulsion grows without bound at the obstacle’s surface and vanishes outside a region of influence (his Eq. 16). The command is the negative gradient of the summed potentials. Khatib applied it to a PUMA 560 arm and extended it to moving obstacles with time-varying fields, demonstrated with visual sensing, and noted that the application to mobile robots is straightforward. What the field needs from perception is a distance to each nearby obstacle surface, updated as fast as the controller runs. In our reading of the construction, attractive and repulsive forces can also cancel at points other than the goal.

Histograms, windows and primitives. The vector field histogram of Borenstein and Koren (1991) and the dynamic window approach of Fox, Burgard and Thrun (1997) are two other collision-avoidance methods, named here by their titles; neither paper could be read for this article. Among the methods compared by Loquercio et al. (2021), a reactive baseline selected among predefined motion primitives using the current depth image and a cost for collision and progress, and a map-based planner built a map over several frames; the first had low latency but could not represent every manoeuvre and was sensitive to noise in a single observation, the second rejected noise at the cost of latency.

Commands from learned scores. DroNet’s speed law, which scales speed by one minus the collision probability, is the simplest coupling: the perception output is the controller’s input. CAD2RL chooses the image bin with the highest predicted value.

Datasets and Benchmarks

There is no single benchmark for vision-based obstacle avoidance comparable to those of detection or depth estimation; in our reading of the sources here, each group evaluates on its own data, simulators and test runs, which is one reason results rarely compare across papers.

• Collision prediction data. DroNet’s collision dataset (about 32,000 bicycle images in 137 sequences, manually labelled far or close) is public, as are its trained weights; steering comes from the Udacity driving dataset, split by the authors into training, validation and testing experiments (Loquercio et al., 2018, and the repository README).
• Simulated environments. Loquercio et al. (2021) evaluate in four simulated environments: a forest of trees about 0.6 m in diameter placed by a Poisson process with an intensity of one tree per 25 m² in a 30 × 60 m region, a wall with a gap 0.8–1.0 m wide at a random offset, a disaster zone and a city street, with 10 runs per speed per environment. Sadeghi and Levine (2017) built randomized synthetic hallways.
• Glass. The Glass Detection Dataset (GDD) of Mei et al. (2020) has 3,916 images with pixel-level glass masks, 2,827 indoor and 1,089 outdoor, split randomly into 2,980 for training and 936 for testing. Their GDNet reached an IoU of 87.63% on the GDD test set (Table 1).
• Depth. Obstacle detectors that rely on depth inherit the depth benchmarks described in depth estimation and stereo matching; driving datasets such as KITTI cover road scenes.

Evaluation Metrics

• Perception accuracy. Collision classifiers are scored by accuracy and the F1 score (Loquercio et al., 2018); glass and free-space segmentation by intersection over union and related pixel measures (Mei et al., 2020); time-to-contact estimators by their error against ground truth on synthetic or stop-motion sequences, since real video rarely has it (Horn et al., 2007).
• Distance or time before failure. DroNet reports the average distance travelled before stopping or colliding; CAD2RL reports collisions per metre and per minute and the average collision-free distance and time (Sadeghi & Levine, 2017).
• Success rate. Loquercio et al. (2021) count a run as successful if the drone reaches within 5 m of the goal without crashing, and report success against speed, which exposes the effect of latency.
• Latency. Processing time per frame belongs with accuracy: Loquercio et al. (2021) report 65.2 ms for the map-based baseline and 19.1 ms for the reactive one on a desktop CPU, and 41.6 ms from sensor reading to plan for their own method on the drone’s onboard computer.

These scores depend on speed, environment and protocol as much as on the method. The results cited here are measurements under the conditions their authors describe; this article draws no conclusions from them about the safety of any deployed robot or vehicle.

Documented Limits

• Textureless surfaces. Flow needs texture: Coombs et al.’s robot collided with a blank wall it was steering towards, and the authors state that the system fails in the absence of texture (Coombs et al., 1995). CAD2RL collided more in a building with narrower hallways and glossy, textureless walls, as well as stronger air currents (Sadeghi & Levine, 2017). Stereo matching fails on the same surfaces; Loquercio et al. (2021) name snow-covered terrain as a source of very noisy depth.
• Glass and reflections. In a qualitative test, CAD2RL crashed into a reflective glass door that mirrored the hallway behind it, a structure absent from its training (Sadeghi & Levine, 2017). Mei et al. (2020) motivate glass detection with exactly this risk, noting that the content seen through glass resembles what lies behind it. The robot perception article describes a legged robot whose sensors missed a transparent step.
• Thin obstacles. Loquercio et al. (2021) list thin branches and thick vegetation among the structures that make high-speed flight in forests hard; branches, wires and poles cover few pixels and give stereo and flow little to match.
• Motion blur and speed. At 10 m/s, Loquercio et al. (2021) report that aerodynamics, battery voltage drops and motion blur widen the gap between simulation and reality, and their real-world success rate fell to 60%, with failures near openings narrower than a metre. Two failures at 7 m/s happened when obstacles entered the field of view late because the drone was turning fast.
• Large image motion. Gradient-based time to contact fails when image motion exceeds about a pixel per frame unless the image is averaged and sub-sampled (Horn et al., 2007), as the example’s collapse at short times to contact shows.
• Obstacles near the focus of expansion. The obstacle straight ahead produces the least flow (Coombs et al., 1995); Coombs et al. kept their gaze aligned with the heading for this reason.
• Mixed depths. A time to contact computed over a region containing several surfaces is a weighted mixture; Horn et al. (2007) found nearby objects pulling their car-video estimates down, and the example’s whole-image estimate hid the obstacle behind the wall.
• Domain shift for learned models. DroNet drifted at indoor intersections and failed in a forest without a path (Loquercio et al., 2018); CAD2RL found particularly novel situations confusing (Sadeghi & Levine, 2017).
• Timing. Variable delays between detection and response caused collisions in Coombs et al.’s system, and map-based filtering of noisy depth added the latency that made Loquercio et al.’s baseline detect obstacles too late (Coombs et al., 1995; Loquercio et al., 2021).

Applications

Obstacle detection is part of every mobile robot that moves among things it could hit: indoor service and warehouse robots, inspection and mapping drones, planetary rovers and cars. Loquercio et al. (2018) list surveillance, construction monitoring, delivery and emergency response as applications that need drones to navigate while avoiding obstacles, and Loquercio et al. (2021) search and rescue, logistics, security, infrastructure, entertainment and agriculture. How obstacle perception fits into a full robot is described in robot perception and visual navigation, and on roads in autonomous driving perception.

Open Problems

• Transparent, reflective and thin obstacles remain hard for every camera-based method, and dedicated detectors such as GDNet are trained on still images rather than evaluated in closed-loop avoidance.
• Speed. Loquercio et al. (2021) report that even their privileged expert often failed to find collision-free trajectories at 10 m/s or more, and they suggest event cameras, which report brightness changes per pixel with low latency, as one way to reduce perception latency (see robot perception for these sensors).
• Comparable evaluation. Without shared closed-loop benchmarks, methods are compared on their authors’ own courses, simulators and speeds.
• Calibrated uncertainty. DroNet’s authors suggest adding a measure of uncertainty so that a system could switch to a safety mode when needed (Loquercio et al., 2018); a learned collision score is not by itself a probability.
• Combining reactive and planned avoidance. Loquercio et al. (2021) see combining a short-horizon learned controller with a long-term planner as a major opportunity, and Coombs et al. (1995) saw a local map as the logical extension of a purely reactive system, while noting that it introduces new problems.