Robot Perception

Robot perception is the part of a robot that turns what its sensors measure into the estimates it needs to act: where the robot is and how it is moving, what the space around it looks like, which objects are there and where, and whether its last action had the effect it expected. It draws on most of computer vision, from depth estimation and object detection to SLAM, but under conditions that a photo or a video clip does not impose. The robot is part of the scene it observes and moves the camera itself; its estimates must be metric and expressed in frames it can act in; they must arrive while they still describe the world; and they feed back into actions that change the next image. The field is also called robotic perception, and in work on learning agents it overlaps with embodied vision.

This article is the hub for the robotics category. It describes the problem and its history from SRI’s Shakey and the Stanford Cart, what robots sense and how each sensor works in principle, the perception–action loop, state estimation in outline, and the main robot settings: mobile robots, manipulators, aerial robots, legged robots and autonomous vehicles. Each subject is summarized in a paragraph and linked to the article that treats it. A worked example simulates a camera that follows a moving target and shows how latency and prediction decide whether the loop keeps the target in view.

Problem

A robot acts through a loop: it senses, updates what it believes about itself and the world, chooses an action, executes it, and senses again. Perception’s share of that loop is to keep those beliefs current, in a form the next stage can use. What it must estimate depends on the robot and the task, but the outputs fall into a few kinds:

• The robot’s own state. Its pose in a map or a world frame, its velocity, and the configuration of its joints. A mobile robot needs its position to plan a route; an arm needs the pose of its gripper relative to the object it reaches for.
• The geometry of the surroundings. Free space and obstacles, the shape of the ground, surfaces that can be stepped on, grasped or pushed. It is usually stored as an occupancy grid, an elevation map, a point cloud or a mesh.
• Objects and their states. What the objects are, where they are in 3D, how they are oriented, how they move, and for some tasks where they can be grasped or how they will deform.
• Contact and its outcome. Whether the gripper is touching an object, with how much force, and whether it is slipping; whether the foot has found ground.

Three features separate this from perception for images alone. The first is uncertainty that the robot must carry with it. Thrun (2000) traces it to sensor limitations, noise and environments that are partly unpredictable, and states as his central conjecture that a robot that carries a notion of its own uncertainty, and acts on it, will do better than one that does not. The second is time. An estimate is used after it is computed, and in that interval the robot and the world have moved; the example below measures the cost. The third is that perception and action are coupled. The robot decides where to look and how to move, which changes what it will see next. Bajcsy, Aloimonos and Tsotsos (2018) define an agent as an active perceiver if it knows why it wishes to sense, then chooses what to perceive and determines how, when and where to perceive it; they build the definition on Bajcsy’s seminal paper of 1988 (with an earlier version in 1985) and on Aloimonos, Weiss and Bandyopadhyay (1988).

History

Shakey. SRI International describes Shakey, the subject of research at its Artificial Intelligence Center from 1966 to 1972, as the first mobile robot able to perceive and reason about its surroundings (SRI). Nilsson’s (1969) report on the project’s state in early 1969 shows how perception fitted into the system. The vehicle carried a vidicon television camera and an optical range-finder on a movable head, and “cat-whisker” touch sensors around its perimeter that told it when it bumped into something; radio links carried commands and telemetry between it and a time-shared SDS-940 computer, and video from the camera. Nilsson grouped the abilities a robot needs into problem-solving, modelling and perception, and gave perception two jobs: to supply new information about the world, and to check the predicted consequences of actions against reality while a plan executes. The vision system digitized pictures as arrays of 120 × 120 pixels with 16 intensity levels, found intensity changes by digital differentiation, fitted line segments and joined them into a line drawing, and traced the boundary of the floor in it. Following the camera ray through each boundary point to where it met the floor gave a polygon of floor known to be empty, which was entered into the robot’s map, a hierarchy of grid cells (Nilsson, 1969). Bajcsy et al. (2018) name J. Martin Tenenbaum’s 1970 Stanford doctoral dissertation, in which the computer controlled the camera’s parameters as part of recognition, as perhaps the earliest on active perception.

The Stanford Cart. Hans Moravec’s 1980 thesis at Stanford describes a card-table-sized cart with a TV camera, controlled over a radio link, that gained its knowledge of the world entirely from images. At each stop it took about nine pictures while sliding its camera along a 50 cm track, picked about 30 distinctive features in one picture with an interest operator, found them in the other pictures by correlation, and computed their 3D positions from their apparent motion; a navigator then planned a path that avoided them, and the plan changed as new obstacles appeared. Moravec called the system moderately reliable but very slow: the cart moved about a metre every ten to fifteen minutes, and several 20 m indoor courses took about five hours each. An outdoor run in which it collided with the third of three obstacles was less successful, and he names harsh lighting and the movement of shadows during the cart’s slow progress as major reasons (Moravec, 1980). Feature detection, matching and triangulation from a moving camera remain the core of visual odometry.

Probabilistic robotics. By 2000, Thrun described the probabilistic approach, which represents what the robot knows as probability densities, as the dominant paradigm across a wide range of robotic problems, with roots he traces to the early 1960s. Its tools were Bayes filters for localization, among them the Kalman filter, historically the most popular for tracking position, and sample-based Monte Carlo localization, and probabilistic methods for mapping. One example he gives is CMU’s robot Xavier, which travelled more than 230 km through the university’s hallways over several years using Markov localization with a discrete, topological representation of the belief (Thrun, 2000). Cadena et al. (2016) call the period from 1986 to 2004 the classical age of SLAM, in which the main probabilistic formulations were introduced (extended Kalman filters, Rao-Blackwellized particle filters and maximum-likelihood estimation), and the period from 2004 to 2015 the algorithmic-analysis age, which studied observability, convergence and consistency, exploited sparsity for efficient solvers and produced the main open-source libraries. They argued that the field was entering a third, robust-perception age.

Learning. Learned models now perform many of the perception components, from object detection and segmentation to grasp detection, as the linked articles describe, and some systems learn the mapping from sensors to actions as a whole. Miki et al. (2022), for example, trained in simulation a controller for a quadrupedal robot that combines its terrain map with its joint and inertial sensing, then ran it on the real robot. Learned agents that perceive and act, the simulators they are trained in and the transfer from simulation to real robots are the subject of embodied AI.

What Robots Sense

Robot sensors are usually divided into exteroceptive sensors, which measure the world (cameras, depth sensors, LiDAR, touch), and proprioceptive sensors, which measure the robot’s own body (joint encoders, motor currents, wheel odometry, and the inertial measurement unit, which measures the body’s motion). Thrun (2000) makes a related distinction from the estimator’s point of view: data that describe the momentary situation, such as camera images and laser scans, are observations, and data about change, such as motor commands and odometer readings, are action data.

Sensor	What it measures	Principle	Typical weaknesses
Camera	Light intensity and colour per pixel	Projection through a lens onto an image sensor	No direct depth; motion blur; low light and glare
Event camera	Per-pixel brightness changes, asynchronously	Each pixel reports a change when it happens	Unconventional output that needs new algorithms
Stereo or active depth camera	Depth per pixel	Triangulation between two views, a projected pattern, or the time of flight of light	Texture-less, dark, reflective and transparent surfaces
LiDAR	Range along laser beams	Time of flight of laser pulses (or a frequency-modulated continuous wave), swept over the scene	Sparse; rain, fog and dust; little appearance
IMU	Angular rate and specific force	Gyroscopes and accelerometers	Drifts when integrated over time
Joint encoders, wheel odometry	Joint angles, wheel rotation	Measuring the rotation of shafts	Wheel slip; drift without external references
Force/torque and tactile sensors	Contact forces and contact geometry	Deformation of an elastic element, or a camera viewing a deforming skin	Only what is touched

Cameras. Digital cameras are small and light for the information they give: appearance, which identifies objects and places, at high resolution. They give depth only through geometry or inference, and they lose information to motion blur, low light and high contrast, as Moravec’s outdoor runs showed. How the sensor forms the image is described in image sensor, and recovering the camera’s intrinsic parameters and its pose relative to the robot in camera calibration. Event cameras report brightness changes at each pixel as they happen instead of whole frames at a fixed rate. Gallego et al. (2022) list their attractions as temporal resolution of the order of microseconds, a dynamic range of 140 dB against 60 dB for conventional cameras, low power consumption and reduced motion blur, which makes them interesting for fast robots, while noting that their output needs new processing methods.

Depth sensors. Depth can come from two cameras by stereo vision, from a camera and a projected pattern, or from the round-trip time of emitted light; the depth estimation article compares these active and passive methods, and monocular depth estimation infers depth from one image with a learned model. Each fails on some surfaces. Miki et al. (2022) summarize the failures they met on legged robots: stereo matching needs texture and struggles with low-texture surfaces and over- or underexposed regions; time-of-flight cameras often miss dark surfaces and become noisy in sunlight; light-based sensors produce artefacts on reflective surfaces; and no depth sensor can tell soft vegetation from rigid ground.

LiDAR. A LiDAR measures range along each laser beam, most often by timing pulses, and sweeps its beams to build a point cloud, giving accurate metric geometry by night as by day. Its article describes the sensor, its calibration and its failures in rain, fog and dust.

Inertial sensors. An inertial measurement unit (IMU) measures angular rate and acceleration at hundreds of samples per second; EuRoC’s visual-inertial sequences, recorded on a small drone, pair 20 Hz stereo cameras with an IMU at 200 Hz (EuRoC MAV). Integrated alone, its readings drift; combined with a camera, as the visual odometry article describes, it bridges short visual failures and, for a single camera, makes metric scale observable. Delmerico and Scaramuzza (2018) note that cameras and IMUs meet the power and payload constraints of flying robots, which is why visual-inertial odometry is a popular choice of state estimator for them.

Proprioception. Encoders on joints and wheels tell the robot how its body is configured and how far its wheels have turned. Forward kinematics turns joint angles into the pose of the gripper or the camera mounted on it; integrating wheel rotations gives odometry. Cadena et al. (2016) recall that early mobile robots obtained odometry by integrating wheel encoders and that the estimate quickly drifted, becoming unusable after a few metres, which was one of the main motivations for SLAM. Proprioception is precise about the body and blind to the world, and legged robots show how far it can go: Miki et al. (2022) note that the most robust and general locomotion controllers before theirs relied on proprioception alone, at the cost of speed, since the robot had to feel out the terrain with its feet before adapting its gait.

Force and touch. A force/torque sensor at the wrist measures the forces of contact, and tactile sensors on the fingers measure where and how an object is touched. Lambeta et al. (2020) describe vision-based tactile sensors, which measure contact as changes in the image that a camera records of a deformable elastomer skin, and which offer high spatial resolution and sensitivity at low cost, though in bulkier housings than other designs; their DIGIT sensor is small enough to mount on the fingers of a multi-fingered hand. Touch is where computer vision reaches inside the gripper.

No sensor covers every condition, so robots carry several and fuse them. How measurements from different sensors are put into one frame and one time base is described for cars in sensor fusion for autonomous driving, and the general idea in multimodal fusion.

The Perception–Action Loop

The loop has the same stages on every robot: sensors capture data; perception turns them into estimates of state; a planner or policy chooses an action; controllers execute it on motors; and the result shows up in the next measurements. Three properties of the loop shape how perception is built.

Frames. Measurements arrive in the frame of the sensor that made them, and actions are commanded in the frames of the robot’s joints or base. Perception must know the transforms between them: the camera’s intrinsic parameters, the pose of the camera on the robot (for a camera on an arm, found by hand–eye calibration), and the robot’s own pose, which changes as it moves. Errors in any of these transforms place correctly detected objects in the wrong place.

Time. Each stage takes time, and a result describes the moment its image was exposed, not the moment it is used. Real-time vision pipelines covers frame budgets, queues, dropped frames and timestamps, and edge inference the limits of the computers robots carry. For a robot the cost of latency is direct. A delayed estimate can be corrected in part by predicting it forward with a motion model, and a feedback loop that acts on delayed measurements without such a correction can become unstable. The example below shows both effects.

Coupling. Because the robot acts on what it sees, perception can be organized around the action. In visual servoing, the controller drives the error between measured and desired visual features to zero, frame after frame; in its image-based form the features are image measurements and no 3D model of the object is needed. The camera can ride on the robot’s hand or watch the robot from a fixed place. In active perception, the robot chooses what, where and when to sense; Barrow and Popplestone (1971), as Bajcsy et al. (2018) quote them, already suggested that a recognition system could simply shift its viewpoint and walk round an object to gather more information and resolve ambiguities. In learned visuomotor policies, a network maps images directly to motor commands, and perception is no longer a separate module with its own output; vision-based robotic manipulation describes these policies for manipulation.

The classic architecture is modular: perception produces an explicit estimate of state, such as a map, poses and object lists, and planning and control consume it. Nilsson’s (1969) Shakey and Moravec’s (1980) cart are early examples, and many of the systems in the linked articles are still built this way. In our reading, the appeal is that each module can be tested on its own and its estimates inspected, while learned end-to-end systems give up that structure in exchange for learning from data what matters for the task. Some systems combine the two, such as the legged-robot controller described below, which learns how to combine an explicit terrain map with the robot’s own sensing.

State Estimation

A robot never observes its state directly; it infers it from a stream of measurements and from its own commands. Thrun (2000) writes the robot’s belief about its pose $s_t$ at time $t$ as the posterior given all observations $o$ and action data $a$ so far and a map $m$, and derives, under the Markov assumption that the past is independent of the future given the current state, the recursive update of a Bayes filter

$$b_t(s_t) = \eta_t \, p(o_t \mid s_t, m) \int p(s_t \mid a_{t-1}, s_{t-1}, m) \, b_{t-1}(s_{t-1}) \, ds_{t-1},$$

where $\eta_t$ normalizes the result. The integral predicts the state forward with a motion model, using the robot’s commands or odometry; the factor in front corrects the prediction with a sensor model, the likelihood of the new observation given each possible state. The estimators that robots use are implementations of this update with different representations of the belief:

• The Kalman filter represents the belief as a Gaussian, which is exact for linear models with Gaussian noise; Thrun (2000) calls it historically the most popular approach for tracking a robot’s position, though one that does poorly when the robot does not know roughly where it is. The extended Kalman filter linearizes nonlinear motion and sensor models around the current estimate.
• The particle filter represents the belief by samples, which can hold several hypotheses at once; applied to localization in a known map it is Monte Carlo localization, which can start from samples spread over all possible poses when the robot does not know where it is (Thrun, 2000).
• Optimization over a window or a whole trajectory solves for many states at once as a maximum a posteriori problem. Cadena et al. (2016) describe this maximum a posteriori formulation, often written as a factor graph, as the de facto standard for SLAM; measurements such as odometry, inertial data, landmark observations and loop closures each add a factor.

Three estimation problems recur on almost every robot that moves. Odometry estimates the robot’s motion incrementally, from wheels, from an IMU or from images, and drifts without bound. Visual odometry estimates the motion of a camera from consecutive images, and visual-inertial odometry adds an IMU. Localization estimates the robot’s pose in a map it already has. SLAM builds the map and localizes in it at the same time, and uses recognized revisits, loop closures, to remove accumulated drift. Cadena et al. (2016) put the relation compactly: without loop closures, SLAM reduces to odometry, and visual-inertial navigation is SLAM with the loop-closure module switched off. They also note that the map lets a robot predict and validate future measurements, which they see as a defence against perceptual aliasing, where different places look alike. The SLAM article covers filtering and graph-based methods, keyframe and dense visual SLAM, LiDAR SLAM and loop closure.

Objects in the scene are estimated in the same way. A tracker predicts each object’s motion with a model such as constant velocity and corrects it with new detections, as in multi-object tracking; the pose of an object to be grasped is estimated by fitting its model to depth data, for example with iterative closest point, or with the learned estimators described in the manipulation article.

Robot Settings

The same components appear in every kind of robot, but the weight on each differs with how the robot moves, what it carries and what it must do.

Mobile robots

Wheeled robots indoors, in warehouses, hospitals and homes, are where robot perception began, with Shakey and the Stanford Cart. Cadena et al. (2016) consider 2D mapping of an indoor environment by a robot with wheel encoders and a laser scanner, to an accuracy better than 10 cm and with a low failure rate, as largely solved, and vision-based SLAM on slowly moving robots such as Mars rovers and domestic robots as a mature research field. For the robust-perception age they saw coming, the same authors listed four requirements: robust performance with a low failure rate over extended periods, including fail-safe mechanisms; high-level understanding beyond geometry, such as semantics, physics and affordances; awareness of the available sensing and computing resources; and task-driven perception that keeps what the task needs and adapts the map to it. Visual navigation covers how a robot reaches a goal from what it sees, by map-based pipelines or by learned policies, with point-goal, object-goal and instruction-following tasks, the success and SPL metrics, and the gap between results in simulation and on real robots.

Manipulators

A robot arm needs to know where objects are relative to its gripper, accurately enough to grasp, place or insert them. The camera is either fixed above the workspace or mounted on the wrist, and in both cases it must be calibrated to the arm. Robotic grasping covers how grasps are found in depth images and point clouds, from analytic force-closure criteria to learned grasp detectors, and why reported success rates mean little without their objects, clutter and trial protocol. Vision-based robotic manipulation covers the rest of the task: 6D object pose estimation and its benchmarks, pick-and-place, keypoint and affordance representations, assembly and insertion, deformable objects, and learned visuomotor policies. Touch takes over where vision is occluded by the hand itself; Lambeta et al. (2020) note that much earlier work on learning manipulation relied purely on vision or proprioception, and that the difficulty of turning high-resolution tactile signals into features for control had held touch back. In factories, arms guided by vision are part of production cells with fixed lighting and triggered cameras, the setting of industrial machine vision.

Aerial robots

A multicopter moves fast in six degrees of freedom, cannot stop to think as Moravec’s cart did, and carries little weight and power. Delmerico and Scaramuzza (2018) state the requirement as accurate and low-latency state estimation for stable flight under the computational constraints of embedded hardware, and benchmarked six public visual-inertial pipelines on a laptop and on single-board computers typical of flying robots, using EuRoC’s flights. They found a trade-off rather than a winner: the most efficient pipeline on the most constrained boards sacrificed accuracy and robustness on hard trajectories, the most accurate one needed a much larger share of the computer, and a third sat between them. Cadena et al. (2016) list fast robot dynamics among the conditions under which SLAM still fails. Event cameras, with their low latency and reduced motion blur, have been flown on quadrotors with limited onboard computing for autonomous landing and flight (Gallego et al., 2022).

Legged robots

A walking robot must perceive the terrain it is about to step on: its height, its slope and whether it will bear weight. Miki et al. (2022) describe the usual representation, an elevation map built from depth sensor data around the robot, and its weaknesses: it depends on the estimated pose of the robot, and it is corrupted by occlusion and moving objects. Their controller for the quadruped ANYmal reads samples from the elevation map together with its joint and inertial sensing, through a recurrent encoder trained in simulation that learns how far to trust the map. They tested it with active stereo cameras in some experiments and LiDAR in others. The robot completed a 2.2 km hiking route with 120 m of elevation gain on the Etzel mountain in Switzerland in 78 minutes, without a fall, against 76 minutes suggested by a hiking planner. When the map misled it, as with a foam block that looked solid or a transparent acrylic step that the sensors missed, the controller revised its estimate of the terrain on contact (Miki et al., 2022). Legged robots are a clear case of fusing exteroception and proprioception, in which the robot’s body is a sensor of last resort.

Autonomous vehicles

A self-driving car is a mobile robot that moves fast among other road users. Autonomous driving perception is the overview of its perception stack, from Dickmanns’s vision-guided van and the DARPA challenges to learned 3D detectors and benchmarks. Its components have their own articles: sensor fusion for autonomous driving combines cameras, LiDAR and radar; bird’s-eye-view perception turns camera images into a metric top-down grid around the vehicle; and lane detection finds the lanes the vehicle drives in. Range and latency requirements grow with speed, and the driving overview works through that arithmetic.

Example

The script simulates the simplest perception–action loop: a camera on a pan joint follows a bright target that moves back and forth across its view. Each frame at 30 Hz is rendered as a noisy 24 × 160 image; perception finds the target’s column from the peak of the image’s column sums, and the controller sets the pan angle for the next frame. Angles are measured in pixels to keep the arithmetic simple. Processing adds a latency of $L$ frames: the result for frame $k$ is available only when frame $k+L$ is captured, and it moves the camera one frame later still. Four controllers are compared:

• Image feedback turns the camera by half the target’s offset in the last processed image, a proportional controller on the image error that ignores how the camera has moved since that image was taken.
• World, raw adds the pan angle at the time the image was taken, known from the joint encoder, to the image offset, which gives the target’s direction in the world, and points the camera there.
• Kalman, filtered smooths those world measurements with a constant-velocity Kalman filter and points the camera at the filtered position.
• Kalman, predicted uses the filter’s velocity to predict where the target will be when the camera gets there, $L+1$ frames after the image was taken.

For the image-feedback controller the script also prints the largest root magnitude $|z|$ of its characteristic equation $z^{L+1} - z^L + 0.5 = 0$, which follows from writing the controller as a difference equation in the pan error; the loop is stable when every root lies inside the unit circle. The filter’s noise settings are assumptions chosen for this target, not tuned values from a source. Measurement noise comes from a seeded random generator, and the script reports the mean and range over seeds 0 to 4. It runs with NumPy 2.3.5.

import numpy as np

H, W, DT, N = 24, 160, 1 / 30, 450       # image size, 30 Hz frames, 15 s
C = (W - 1) / 2                           # image centre column
t = np.arange(N) * DT
x_true = 120 * np.sin(2 * np.pi * t / 6) + 40 * np.sin(2 * np.pi * t / 2.3)  # target bearing (px)
cols = np.arange(W)

def render(u, rng):
    """A bright blob at column u on a noisy 24 x 160 frame."""
    rows = np.arange(H)[:, None]
    img = np.exp(-((rows - H / 2) ** 2 + (cols - u) ** 2) / (2 * 2.0 ** 2))
    return img + rng.normal(0, 0.2, img.shape)

def measure(img):
    """Blob column by peak of the column profile with parabolic refinement."""
    prof = img.sum(axis=0)
    k = int(np.argmax(prof))
    if prof[k] < 2.5 or k in (0, W - 1):
        return None                      # target not visible
    a, b, c = prof[k - 1], prof[k], prof[k + 1]
    return k + 0.5 * (a - c) / (a - 2 * b + c)

class CVKalman:
    """Constant-velocity Kalman filter on the target's bearing in the world."""
    def __init__(self, q=300.0 ** 2, r=0.3 ** 2):
        self.F = np.array([[1, DT], [0, 1]])
        self.Q = q * np.array([[DT**4 / 4, DT**3 / 2], [DT**3 / 2, DT**2]])
        self.r, self.s, self.P = r, None, None
    def step(self, z):
        if self.s is None:
            if z is not None:
                self.s, self.P = np.array([z, 0.0]), np.diag([1.0, 1e4])
            return
        self.s, self.P = self.F @ self.s, self.F @ self.P @ self.F.T + self.Q
        if z is not None:
            S = self.P[0, 0] + self.r
            K = self.P[:, 0] / S
            self.s = self.s + K * (z - self.s[0])
            self.P = self.P - np.outer(K, self.P[0])

def run(strategy, L, seed):
    rng = np.random.default_rng(seed)
    pan = np.zeros(N + 1)
    meas = [None] * N                     # world bearing measured in each frame
    offs = [None] * N                     # image offset measured in each frame
    kf, hold = CVKalman(), 0.0
    for j in range(N):
        u = measure(render(x_true[j] - pan[j] + C, rng))
        if u is not None:
            offs[j] = u - C
            meas[j] = u - C + pan[j]      # pan angle known from the joint encoder
        i = j - L                         # newest frame whose result has arrived
        if i < 0:
            pan[j + 1] = pan[j]
            continue
        if strategy == "image feedback":
            pan[j + 1] = pan[j] + 0.5 * (offs[i] if offs[i] is not None else 0.0)
            continue
        if strategy == "world, raw":
            hold = meas[i] if meas[i] is not None else hold
            pan[j + 1] = hold
            continue
        kf.step(meas[i])
        if kf.s is None:
            pan[j + 1] = pan[j]
        elif strategy == "Kalman, filtered":
            pan[j + 1] = kf.s[0]
        else:                             # "Kalman, predicted": aim where it will be
            pan[j + 1] = kf.s[0] + kf.s[1] * (L + 1) * DT
    err = (x_true - pan[:N])[30:]         # offset from image centre, after 1 s
    return np.sqrt(np.mean(err ** 2)), np.mean(np.abs(err) > C)

strategies = ["image feedback", "world, raw", "Kalman, filtered", "Kalman, predicted"]
v = np.gradient(x_true, DT) * DT          # target motion per frame (px)
print(f"target motion per frame: RMS {np.sqrt(np.mean(v**2)):.1f} px, max {np.abs(v).max():.1f} px;"
      f" image half-width {C:.1f} px")
print("RMS offset from image centre (px): mean over seeds 0-4 (min-max); % frames out of view")
print("latency     |z|" + "".join(f"{s:>24}" for s in strategies))
for L in (0, 1, 2, 3, 4, 6):
    poly = np.zeros(L + 2)                # image feedback: z^(L+1) - z^L + 0.5 = 0
    poly[0], poly[1], poly[-1] = 1, poly[1] - 1, poly[-1] + 0.5
    line = f"{L} fr {1000 * L * DT:3.0f} ms {np.abs(np.roots(poly)).max():4.2f}"
    for s in strategies:
        rms, out = np.array([run(s, L, seed) for seed in range(5)]).T
        cell = f"{rms.mean():.1f} ({rms.min():.1f}-{rms.max():.1f}) {100 * out.mean():.0f}%"
        line += cell.rjust(24)
    print(line)

target motion per frame: RMS 3.9 px, max 7.8 px; image half-width 79.5 px
RMS offset from image centre (px): mean over seeds 0-4 (min-max); % frames out of view
latency     |z|          image feedback              world, raw        Kalman, filtered       Kalman, predicted
0 fr   0 ms 0.50        7.6 (7.6-7.7) 0%        3.9 (3.9-3.9) 0%        3.9 (3.9-4.0) 0%        1.0 (1.0-1.1) 0%
1 fr  33 ms 0.71        7.7 (7.7-7.8) 0%        7.7 (7.7-7.7) 0%        7.7 (7.7-7.7) 0%        1.7 (1.6-1.8) 0%
2 fr  67 ms 0.94        7.9 (7.8-7.9) 0%     11.5 (11.4-11.5) 0%     11.5 (11.5-11.5) 0%        2.6 (2.5-2.7) 0%
3 fr 100 ms 1.02  105.5 (48.2-276.7) 26%     15.3 (15.2-15.3) 0%     15.3 (15.2-15.3) 0%        3.7 (3.6-3.7) 0%
4 fr 133 ms 1.06 397.5 (155.8-617.2) 90%     18.9 (18.9-18.9) 0%     19.0 (19.0-19.0) 0%        5.1 (5.0-5.1) 0%
6 fr 200 ms 1.08 321.8 (138.8-434.7) 85%     26.3 (26.2-26.3) 0%     26.3 (26.3-26.3) 0%        8.6 (8.5-8.6) 0%

The raw world controller’s error is close to the distance the target travels while the loop catches up: its RMS offset is about the target’s RMS motion per frame, 3.9 pixels, times $L+1$ frames, a little less at the longer latencies. Measurement noise contributes little, as the narrow spread over seeds shows; the error is lag. That is why the Kalman filter used only as a smoother gains nothing: it removes noise that hardly exists, and its filtered position is as old as the raw one.

Prediction is what helps. Pointing the camera where the filter expects the target to be when the command takes effect cuts the error by a factor of three to four and a half at every latency. What remains is the target’s acceleration, which a constant-velocity model does not predict, and it grows with the prediction horizon, from 1.0 pixel with no extra latency to 8.6 pixels at 200 ms. A better motion model would shrink it further; no model removes it for a target that can change course without warning.

The image-feedback controller fails differently. With up to two frames of latency it lags by about 8 pixels, because a gain of 0.5 closes only half the remaining error each frame. At three frames (100 ms) the largest root of its characteristic equation passes 1, the loop oscillates with growing amplitude, and the target leaves the view in a quarter of the frames; at four frames it is out of view in 90% of them. The controller keeps correcting an error that its own earlier commands have already corrected, because it does not know they have. The world-frame controllers avoid this by using the joint encoder: proprioception tells perception where the camera was pointing when each image was taken. The same issue, a loop gain too high for its delay, appears in the stability analysis of visual servoing. The example leaves out exposure time and the dynamics of a real motor, which add delay, and motion blur, which adds error, so its numbers are optimistic.

Challenges

• Uncertainty everywhere. Sensors are noisy and limited, the world is partly unpredictable and models are approximate (Thrun, 2000). Every estimate needs an uncertainty that downstream modules can use.
• Latency and compute. The robot carries its computer, with limited power, weight and heat, and every stage of perception competes for it within a frame budget. Flying robots show the trade-off most sharply (Delmerico & Scaramuzza, 2018).
• Surfaces that defeat sensors. Low texture, difficult lighting, reflective and transparent surfaces, fog, dust, snow, water and vegetation each defeat at least one sensor (Miki et al., 2022). The Stanford Cart already lost an outdoor run to harsh lighting and moving shadows (Moravec, 1980).
• Drift and aliasing. Odometry drifts without bound, and places that look alike can make a robot believe it is somewhere else; maps and loop closures address both, at the risk of a wrong closure (Cadena et al., 2016).
• Dynamic and changing worlds. People, vehicles and moved objects violate the static-world assumption on which many estimators rely (Thrun, 2000), and a map built last month may no longer match the building.
• Calibration over time. The transforms between cameras, other sensors and the robot’s body can change in service, and errors in them misplace everything that is perceived correctly.
• From simulation and data to the real robot. Learned perception and policies are often trained in simulation or on data from other robots and places, and how well they transfer is an open question that the visual navigation and manipulation articles discuss for their tasks.

Evaluation

Robot perception is evaluated at two levels.

Components are measured against ground truth on recorded data. Trajectories from odometry and SLAM are compared with motion-capture or laser-tracker ground truth by the absolute trajectory error, on datasets such as EuRoC MAV for drones and TUM RGB-D for handheld and robot-mounted depth cameras; driving datasets such as KITTI add odometry, detection and tracking benchmarks. Object pose estimates are scored on benchmarks with their own pose errors, described in the manipulation article, and detectors and segmenters by the metrics of object detection and semantic segmentation. For robots, compute and timing belong with accuracy: Delmerico and Scaramuzza (2018) report per-frame processing time and CPU and memory load next to pose error, because, they argue, results on desktop or laptop computers do not represent how the algorithms perform on a flying robot’s onboard computer.

Tasks are measured on the robot or in simulation: the success rate and SPL of navigation episodes, the success rate of grasps over a stated object set and protocol, the completion of a locomotion course. These scores depend on the conditions as much as on the method, as the grasping and navigation articles show with examples. Cadena et al. (2016) make the same point about SLAM: whether it is solved can only be asked for a given combination of robot (its motion and sensors), environment and required performance, including accuracy, success rate, latency and operating time.

Neither level measures safety. The results cited in this article are measurements on benchmark data or on test runs under conditions their authors describe, and this article draws no conclusions from them about the safety of deployed robots.

Failure Modes and Risks

• Lost track. A tracker or a localizer that loses its target or its pose makes every later decision on wrong premises; a filter that represents only one hypothesis may not recover without help (Thrun, 2000).
• Wrong loop closures. Perceptual aliasing can join two different places in the map; Cadena et al. (2016) see the metric map itself as the defence, since it lets spurious loop closures be discarded.
• Phantom and missed obstacles. Snow, vegetation and water can appear as obstacles or be missing from the map, and transparent or soft obstacles are misjudged until the robot touches them (Miki et al., 2022).
• Latency instability. A loop that acts on delayed measurements without accounting for its own motion oscillates, as the example shows.
• Silent calibration errors. A camera knocked out of alignment still produces sharp images and confident detections, in the wrong place.
• Overreliance on one modality. A system that depends on a single sensor fails with it; Miki et al. (2022) trained their controller to rely on what its legs feel when the terrain map is wrong.
• Privacy. Robots in homes, hospitals and streets record people who have not agreed to it. Driving datasets such as nuScenes and Argoverse blur faces and licence plates before release, as the driving overview describes; robot data collected elsewhere raises the same question.

Deployment Considerations

• Design from the task. Required accuracy, range, speed and environment set the sensors, compute and models; designing a computer vision system walks through that process from requirements to maintenance.
• Budget time end to end. Measure the latency from exposure to actuation, timestamp every measurement at capture, and predict estimates forward to the time they are used (see real-time vision pipelines).
• Fit the compute. Models and estimators must run on the robot’s own processors within their power and thermal limits; edge inference and model compression cover how. Delmerico and Scaramuzza (2018) show that the choice of estimator depends on how much of the computer it may use.
• Combine sensors that fail differently. Exteroception and proprioception, cameras and depth, vision and touch, each covering the others’ blind spots (Miki et al., 2022; Lambeta et al., 2020).
• Keep calibration current. Recheck intrinsic, extrinsic and hand–eye calibration in service, not only at installation; see camera calibration.
• Plan for failure. Cadena et al. (2016) list fail-safe mechanisms, and self-tuning that adapts the system’s parameters to the scenario, among the requirements for robust perception.
• Test where the robot will work. Benchmark scores rank methods; how a robot performs depends on its own environment, lighting and objects, which is why the task-level results above are reported with their conditions.