Active Perception

Active perception is the view that an agent should control how it acquires data, choosing where to look, how to set its sensors and when to sense, according to what it already believes and what it is trying to do, instead of processing whatever data happens to arrive. A robot that cannot tell whether a cup has a handle walks around it; a camera that sees a blurred edge refocuses; an arm that cannot find a knife moves the bowl that hides it. The idea is also called active vision when the sensor is a camera, active sensing, and, in Ballard’s version, animate vision. Next-best-view planning, active object recognition and learned exploration are its main algorithmic forms. Robot perception introduces the coupling of perception and action in general, and visual navigation describes exploration as part of reaching goals in unseen buildings; this article covers the concept itself, its history, the information-theoretic formulation of view selection and the methods built on it.

Definition

Bajcsy’s technical report of March 1988, marked as submitted for publication to the IEEE that year (the longer Proceedings of the IEEE paper was not read for this article), states the problem of active sensing as one of controlling the strategies applied to data acquisition, depending on the current state of interpretation and on the goal or task. She separates the term from “active sensors” such as radar, sonar or structured light, which emit energy and measure what returns: active sensing in her sense can use passive sensors, such as ordinary cameras, employed actively by purposefully changing their parameters. She also argues that this is not simply control theory, because the feedback acts on processed data (extracted features, including relations between them) and depends on prior models of the world that mix numerical and symbolic information (Bajcsy, 1988, technical report). The report opens with the observation, “We do not just see, we look.”

Bajcsy, Aloimonos and Tsotsos (2018) restate the idea three decades later. An agent is an active perceiver, in their definition, if it knows why it wishes to sense and then chooses what to perceive and determines how, when and where to achieve that perception. They call why, what, how, when and where the active pentuple and give each a role:

Element	Meaning in Bajcsy et al. (2018)
Why	The agent’s current state generates expectations, and these determine its next actions; inductive inference of any form (Bayesian, analogical, predictive) supplies them
What	Scene selection: which subset of the sensory field an expectation applies to
How	Mechanical alignment of the body, sensor alignment (pose, focus, exposure) and priming of the interpretation system
When	Temporal selection: when an expected event is valid and for how long
Where	Viewpoint selection: the pose of the agent and of the sensor from which the target can be sensed

The why is what distinguishes the definition. Nearly every agent controls one of the other four, so the authors require that the agent determine the why of its behaviour dynamically and then control at least one of the others; a fixed, pre-specified control loop does not qualify (Bajcsy et al., 2018).

History

Accommodation and early robot vision (1970–1971). Bajcsy et al. (2018) name J. Martin Tenenbaum’s Stanford dissertation of 1970, on a vision system whose camera parameters the computer controlled as part of recognition, as perhaps the earliest dissertation on active perception, and quote Barrow and Popplestone’s (1971) suggestion, which the robot perception article also cites, that a recognition program could walk round an object to resolve ambiguities. Bajcsy et al. add that Barrow and Popplestone gave no strategy or method for doing it, and that the community let their words fade.

Active perception and active vision (1985–1991). Bajcsy’s 1988 paper, whose perspective had appeared in an earlier version in 1985 (Bajcsy et al., 2018), made the control of data acquisition a research programme. The technical report lays out an active vision system as levels of control, each with its feedback parameters and stopping condition: lighting, aperture and gross focus; focus and zoom on one object; thresholds of low-level edge and region modules; vergence of a stereo pair; segmentation of the depth map into surfaces; several views of the scene integrated into a 3D description; and semantic matching against models. Its main worked example is Krotkov’s agile camera system at Pennsylvania, two cameras on an 11-degree-of-freedom platform with controlled focus, zoom, aperture, vergence and lighting, in which range from focus and range from stereo check each other (Bajcsy, 1988, technical report).

The same report cites a paper titled “Active Vision” by Aloimonos and Bandyopadhyay at the first International Conference on Computer Vision (June 1987); in our reading it is an earlier version of the paper of the same title, with Weiss as a further author, in the first volume of the International Journal of Computer Vision (1988). As Bajcsy’s report summarizes the conference paper, the paper argued that problems that are ill-posed, nonlinear or unstable for a passive observer become well-posed, linear or stable for an active one, and showed this for shape from shading, shape from contour, shape from texture, structure from motion and optical flow, assuming an observer that moves with known motion. Bajcsy wrote that this result was unsurprising, since more measurements taken in a controlled way should make a problem better posed, and placed her own emphasis elsewhere: on modelling sensors, objects and environment and on control strategies for a purpose. Bajcsy et al. (2018) quote the 1988 journal paper’s definition of an active observer as one that controls the geometric parameters of its sensory apparatus in order to manipulate the constraints underlying what it observes.

Ballard’s animate vision (1991) tied active vision to behaviour. Bajcsy et al. (2018) quote his list of what gaze control buys a behaving robot: it can move cameras closer, refocus and search; make programmed camera movements; segment regions of interest before recognizing them; fixate a point in the world while moving and use it as a coordinate frame; servo relative to that frame; and use object-centred coordinates for spatial memory.

Robot heads (late 1980s–1990s). These ideas were built into motorized camera heads: the Pennsylvania head of Krotkov’s dissertation, with pan, tilt, focus and vergence; the Rochester head built from 1985 under Brown and Ballard, a fast binocular system mounted on a robot arm; the KTH head with three degrees of freedom for each eye; and heads at Oxford, Harvard, Surrey, Grenoble, NIST, Illinois and Aalborg. Bajcsy et al. (2018) note that the engineering effort to build and maintain such systems was large, which limited how far the methods spread, although the work led to the binocular heads now common on humanoid robots.

Next-best-view planning (from 1985). In parallel, robotics and range sensing posed view selection as an optimization problem for building 3D models: given the partial model so far, where should the sensor go next? Connolly’s ICRA paper of 1985 is an early instance that both Isler et al. (2016) and Delmerico et al. (2018) cite, and they describe it as counting the unknown voxels a candidate view would see (Delmerico et al., 2018). The section on next-best-view planning below follows this line to probabilistic maps.

Complexity and attention. Tsotsos showed in 1989 that basic visual matching is NP-complete, and Ye and Tsotsos later proved the sensor planning problem, choosing robot actions that maximize the probability of finding a target within a cost limit, to be NP-hard (Bajcsy et al., 2018). Both results are reasons to use knowledge of the task to limit what is sensed and processed. Bajcsy et al. (2018) describe active perception as a subset of attentional processing, and note that methods for choosing what to attend to within an image had received much more work than methods for choosing which image to take, with blanket sensing of the whole environment, for example by lidar, often preferred instead.

Learned exploration (from about 2017). Reinforcement learning with deep networks made it possible to learn where to look from experience: curiosity rewards for exploring without external reward (Pathak et al., 2017), look-around policies that complete a scene or object from a few glimpses (Jayaraman & Grauman, 2018), and modular agents that explore whole buildings (Chaplot et al., 2020). These form the later sections of this article.

Intuition

A single image is a projection: depth, the far side of objects, everything outside the field of view and anything hidden by an occluder are missing from it. A passive system must infer them from priors. An active system can often measure them instead, by moving to see the far side, changing focus to measure depth, or waiting for an expected event. Bajcsy’s report puts the point as a matter of economy: information that one view lacks must be recovered laboriously, while more views can resolve the problem more easily (Bajcsy, 1988, technical report).

Bajcsy et al. (2018) give a second argument, from reliability. Suppose a trained detector finds knives with 90% success. For searching a photo collection that is an excellent result: nine of ten returned images contain a knife. For a robot about to grasp and cut, acting on the wrong object one time in ten is not acceptable. The detector is better used as an attention mechanism that suggests where a knife may be, after which the robot fixates, segments and verifies, and if no knife is found it moves, takes another view, or moves the bowl that may hide it.

The same reasoning applies to the choice of view. Some views are much more informative than others: a view of a mug from the side cannot show what is inside it (Jayaraman & Grauman, 2018). Choosing well requires predicting what a view would reveal before taking it, which is what the formal treatment makes precise.

Formal Definition

Let $X$ be the unknown state of the world relevant to the task (an object’s identity and pose, an occupancy map, the location of a target), and let the agent hold a belief $p(X \mid z_{1:t})$ after observations $z_{1:t}$, maintained by a Bayes filter or a map. A sensing action $a$ (a viewpoint, a gaze direction, a focus or exposure setting) produces an observation $z$ with likelihood $p(z \mid X, a)$. Active perception chooses $a$.

Decision-theoretic form. Bajcsy’s report, following Hager and Mintz, writes a controllable measurement as

$$z_i = H(u_i, p) + V(u_i, p),$$

with control vector $u_i$, unknown parameters $p$ and noise $V$ that itself depends on the control. A sequence of controls $u = (u_1, \ldots, u_n)$ is judged by a loss that adds the estimation loss to the cost of acquiring $n$ samples,

$$l(p, \delta_n, n, u) = l_d(p, \delta_n) + c(n, u),$$

where $\delta_n$ is the estimator. For a sensor asked to reach a tolerance $\varepsilon$, the report proposes a 0–1 loss, zero when $|\delta_n(z) - p| < \varepsilon$ and one otherwise, and distinguishes a batch procedure, which fixes $n$ in advance, from a sequential one, which decides after each measurement whether enough data have been taken (Bajcsy, 1988, technical report).

Information gain. A common choice of utility is the expected reduction in uncertainty about $X$, measured by entropy:

$$\mathrm{IG}(a) = H(X \mid z_{1:t}) - \mathbb{E}_{z \sim p(z \mid z_{1:t}, a)} \left[ H(X \mid z_{1:t}, z) \right],$$

which is the mutual information between $X$ and the observation that action $a$ would produce. The next action trades gain against the cost $c(a)$ of moving or sensing,

$$a^{*} = \arg\max_{a} \; \mathrm{IG}(a) - \lambda \, c(a).$$

Choosing one action at a time this way is greedy and myopic. Planning a whole sequence is harder: the sensor planning problem for visual search is NP-hard (Bajcsy et al., 2018), so practical systems use one-step or short-horizon approximations.

Volumetric information gain. For 3D reconstruction the state is an occupancy map whose cells are treated as independent, so the map’s entropy is the sum of cell entropies $H(x) = -P_o(x) \log P_o(x) - (1 - P_o(x)) \log (1 - P_o(x))$, where $P_o(x)$ is the probability that cell $x$ is occupied. Isler et al. (2016) predict the gain of a candidate view $v$ by casting its rays $\mathcal{R}_v$ through the map and summing a per-cell volumetric information $I$ over the cells $\mathcal{X}_r$ that each ray traverses:

$$G_v = \sum_{r \in \mathcal{R}_v} \sum_{x \in \mathcal{X}_r} I(x).$$

In their occlusion-aware form, $I(x) = P_v(x) \, H(x)$, where the probability that cell $x_n$ on a ray is visible is the probability that all cells before it are free,

$$P_v(x_n) = \prod_{i=1}^{n-1} \bigl(1 - P_o(x_i)\bigr).$$

Other choices of $I$ count unknown cells, frontier cells between free and unknown space, or cells behind already observed surfaces. In our reading, $G_v$ is a heuristic stand-in for $\mathrm{IG}(v)$: it credits a view with the whole entropy of every cell it is likely to see, as if one observation made each cell certain. The authors combine it with movement cost as $U_v = G_v / \sum G - C_v / \sum C$, normalizing gains and costs over the candidate views, take the view that maximizes $U_v$, and stop when no candidate’s gain exceeds a threshold (Isler et al., 2016).

Properties

• Purpose decides the measure. What counts as information depends on the task: completing a model, recognizing an object, finding a target, or covering a building. Delmerico et al. (2018) found that the ability of a gain formulation to reduce map entropy is not well correlated with its effectiveness at completing the object’s surface. Isler et al. (2016) note that entropy also falls when free space around the object is observed, and Delmerico et al. that lowering a voxel’s entropy requires observing it repeatedly, which works against observing new parts of the object.
• Myopic choices can be poor. A greedy rule picks the view with the largest immediate gain; it cannot know that a less informative view now opens better views later, and its gain estimate is only as good as its model of what is hidden.
• Active is not always cheaper. Tsotsos (1992), as Bajcsy et al. (2018) describe it, showed that an active, interpretation-guided sequence of images can cost far less than blind analysis of a single image, because evidence accumulated image by image narrows the hypotheses, but only under certain conditions; active vision is not always more efficient than passive vision.
• Diminishing returns. In Delmerico et al.’s simulated reconstructions of 11 object models from 48 candidate views, the other formulations typically matched the best ones after 4 to 6 views, and beyond 6 to 10 views, depending on the model, surface completion was asymptotic; random views did reasonably well on average but with a much larger variance than any gain formulation.
• Evaluation needs a world to act in. A fixed dataset cannot answer the question an active system asks next. Bajcsy et al. (2018) note that active perception needs a live environment or a scene generator that renders on demand for given imaging geometry, and that such systems can be evaluated by their performance on the tasks they serve.

Active Vision and Gaze Control

The first form of active perception controlled the camera itself. Bajcsy et al. (2018) list the reasons for active camera control, among them: to fixate or track, to see a region hidden by occlusion, to see more of the surroundings, to compensate for a foveated sensor’s uneven resolution, to increase resolution by zooming or moving closer, to change the depth of interest by vergence, to focus, to adjust depth of field and exposure, to disambiguate a degenerate view by moving or by changing the lighting, and to reach a view that is characteristic of an object. Each is a choice of the how, when or where in their pentuple.

Focus is the simplest example of an active measurement: a lens focused on a point determines its distance, and Krotkov’s work, described in Bajcsy’s report, modelled the depth of field to predict the precision of range from focus, opened the aperture to make the depth of field shallow, and verified focus ranging and stereo ranging against each other. Fixation and tracking connect active vision to visual servoing, which closes a control loop on image features to move a camera or a hand. Selecting regions within an image that is already captured is the subject of attention models and of salient object detection; active perception adds the choice of which image to capture.

Next-Best-View Planning

Next-best-view (NBV) planning chooses the next sensor pose for building a model of an object or scene. Isler et al. (2016) describe the task as an iteration of three largely independent parts: building the 3D model, planning the view, and moving the sensor. Following the survey of Scott and colleagues, which they cite, they divide methods into model-based ones, which assume an approximate model of the object in advance, and non-model-based ones, which estimate everything online, and the latter into surface-based methods, which examine the boundaries of a partial mesh, and volumetric methods, which keep an occupancy map.

Volumetric methods have become popular because visibility is easy to compute in a grid: casting rays from a candidate pose through the map simulates the image the camera would take (Isler et al., 2016). 3D reconstruction and 3D representations describe the voxel maps themselves. Delmerico et al. (2018) group the gain measures on such maps into counting metrics, such as the number of unknown voxels (attributed to Connolly in 1985 and to later work) or of frontier voxels bordering free and unknown space, introduced by Yamauchi for exploration in 1997, and probabilistic metrics based on entropy.

Isler et al. (2016) proposed several formulations: the occlusion-aware entropy above; an unobserved-voxel variant that restricts that visibility-weighted entropy to cells not yet observed; rear-side variants that reward rays expected to strike the back of an already observed surface; and a proximity count that rewards unknown cells close to observed surfaces. In their simulations, with 48 candidate views on a cylinder topped by a half-sphere around the Stanford bunny, the Stanford dragon and a teapot, and 20 views per trial, the rear-side, proximity and combined formulations had a slight advantage in surface coverage, while entropy reduction was less discriminative and proceeded at comparable rates for all formulations. They concluded that weighting entropy by visibility likelihood did not improve object-centred reconstruction unless something else focused the gain on the object. Delmerico et al. (2018) extended the comparison to 11 models with an idealized depth sensor and 20 random starting views per model. Vasquez-Gomez et al.’s area factor and their own proximity count did best during the first few views, each with the best area under the coverage curve on 4 of the 11 models, while Kriegel et al.’s average entropy was most effective at reducing map entropy.

The term “next best view” is also used in structure from motion, where COLMAP chooses which already captured image to register next. That is a choice of processing order, not of sensing: the images exist before the choice is made.

Active Recognition and Object Search

In active recognition the goal is a label rather than a model, and a view is valuable if it separates the remaining hypotheses. Bajcsy’s report makes the number $N$ of interpretations consistent with the data a feedback signal for further processing: $N = 1$ means a unique interpretation, $N = 0$ means that nothing in the current data matches (for a search, that the object is not there, at least for the moment), and $N > 1$ leaves the question of what to do next (Bajcsy, 1988, technical report). Bajcsy et al. (2018) describe Wilkes and Tsotsos (1992), whose behaviour-based controller set the camera’s distance and viewing angles depending on the state of recognition, and Dickinson et al. (1994), who used aspect graphs, which record how an object’s appearance changes with viewpoint, to decide how to change the view.

Object search adds the where of the target. Ye and Tsotsos, after proving the sensor planning problem NP-hard, defined a search strategy that found objects in unknown environments and was implemented on several robots, including the PLAYBOT wheelchair and the ASIMO humanoid (Bajcsy et al., 2018). In embodied AI, object-goal navigation is the learned descendant of this problem; the visual navigation article describes semantic exploration methods for it.

Jayaraman and Grauman (2018) note that the active object recognition systems before theirs were supervised and specific to one recognition task. They instead learn a task-independent look-around policy by reinforcement learning, rewarding the agent for actions that reduce its uncertainty about the unobserved parts of its environment, measured by how well it can complete its internal model of all views at the end of its budget. A recurrent network aggregates the glimpses so far and chooses the next camera motion. They train and test it on panoramic scenes from SUN360, seen through a 45° field of view, and on CAD models from ModelNet that the agent rotates, and report that policies learned this way transfer to recognition tasks and to object categories not seen in training.

Exploration with Learned Policies

Exploration is active perception with coverage as the goal: see as much of an unknown environment as possible within a budget.

Curiosity. Pathak et al. (2017) make exploration its own reward. Their intrinsic curiosity module learns a feature space $\phi$ by training an inverse model to predict the action $a_t$ from $\phi(s_t)$ and $\phi(s_{t+1})$, then trains a forward model to predict $\phi(s_{t+1})$ from $\phi(s_t)$ and $a_t$; the reward is the forward model’s error, $r^i_t = \frac{\eta}{2} \lVert \hat{\phi}(s_{t+1}) - \phi(s_{t+1}) \rVert_2^2$. Because the features need encode only what predicts the agent’s actions, the reward ignores parts of the image the agent can neither control nor be affected by. They cite Schmidhuber’s example of an agent that watches a television screen showing white noise: if novelty or prediction error is measured on raw images, every state stays novel and the agent can stall there. With curiosity as its only reward, their agent crossed over 30% of Level-1 of Super Mario Bros, and in the VizDoom maze it learned to walk along corridors instead of bumping into walls.

Look-around policies. Jayaraman and Grauman’s policies, above, are exploration over viewing directions of a scene or object rather than positions in a building.

Active Neural SLAM. Chaplot et al. (2020) define exploration, following earlier work, as maximizing coverage, the area known to be traversable, within a fixed budget. Their modular agent, whose parts the visual navigation article describes, has a global policy trained by reinforcement learning with the increase in coverage as reward, which picks long-term goals on the map. On the Gibson validation scenes, after training on 10 million frames, it covered 94.8% of the explorable area (32.7 m²) in episodes of 1,000 steps, against 78.9% (24.9 m²) for the best end-to-end reinforcement learning baseline. Replacing the learned global policy with classical frontier-based exploration, which sends the agent to the nearest boundary between explored free space and unknown space, gave comparable coverage in small scenes but about 10% less in large ones; the authors observed that the frontier policy spent much time in corners and small spaces behind furniture (their Tables 1 and 2, arXiv version). The visual navigation article describes the same agent’s transfer to point-goal navigation.

Active Perception in Embodied AI

In embodied AI active perception is part of the problem rather than an add-on: the embodied AI article reports that Habitat’s authors name it among the things that action by an embodied agent brings to the fore. In our reading, its simulators supply what Bajcsy et al. (2018) said active perception needs, an environment that renders a new view for whatever pose the agent chooses, although the same authors describe active perception systems as rejecting simulated worlds and warn that a simulated framework’s lack of real environments may be a problem. Embodied question answering, in which an agent must find an object before it can answer a question about it, exploration and object-goal navigation all measure an agent by what it achieves after choosing its own observations. In our reading, the classical concepts survive in modern agents under other names: a coverage reward is an information measure, a global policy that picks long-term goals on a map is a learned next-best-view planner, and an exploration budget is the cost term of the decision-theoretic formulation.

Recognition took a different route. The Nature deep learning review forecast vision systems that learn by reinforcement where to look; that article, in its own reading, observes that the main gains after 2015 came instead from models that see the whole image in one pass. Active perception and whole-image recognition are not exclusive: a robot can use a strong passive detector to propose and an active strategy to verify, as in Bajcsy et al.’s knife example.

Examples

The script compares view-selection policies on two synthetic objects in a 72 × 72 occupancy grid: a convex ellipse, and a C-shaped ring whose inner wall is visible only through a 50° opening. Thirty-six candidate views lie on a circle around the object, every 10°, each looking at the centre with a 60° field of view and 64 rays. A view is simulated by ray casting: each ray stops at the first truly occupied cell, cells before it are recorded as misses and the stopping cell as a hit, and the map’s log-odds are updated once per cell per view. Five policies choose views, never repeating one:

• sweep: the next view around the circle, a fixed order;
• spread: a fixed order that ignores the map and takes the view farthest in angle from all views so far;
• random: a random order, seeded by the trial index;
• IG entropy: the occlusion-aware expected entropy $G_v$ of Isler et al. (2016);
• IG unknown: the number of never-observed cells each ray passes before reaching a cell the map believes occupied, a counting metric.

Every policy starts from each of the 36 views in turn, and the script reports the mean over these 36 trials. It scores two things: surface coverage, the fraction of the object’s surface cells (occupied cells with a free neighbour) observed at least once, and the map’s entropy within a disc of radius 24 cells around the object as a percentage of its initial value. How informative one observation is depends on the sensor model, which no source fixes for this setting, so the script runs two: a soft sensor (a first hit takes a cell’s occupancy probability from 0.5 to 0.7, a first miss to 0.4) and a confident one (0.97 and 0.12, the occupied and free thresholds Delmerico et al. used in their simulations, taken here as the probabilities after one observation). Log-odds are clamped to $[-2, 3.5]$. The script runs with NumPy 2.3.5 in about 15 seconds.

import numpy as np

N, C = 72, 35.5                          # 72 x 72 occupancy grid, centre
yy, xx = np.mgrid[0:N, 0:N] + 0.5 - C    # cell-centre coordinates
rr, ang = np.hypot(xx, yy), np.arctan2(yy, xx)
ROI = (rr <= 24).ravel()                 # region whose entropy is scored

def ellipse():                           # convex object
    t = np.deg2rad(30)
    u, v = xx * np.cos(t) + yy * np.sin(t), -xx * np.sin(t) + yy * np.cos(t)
    return (u / 13) ** 2 + (v / 8) ** 2 <= 1

def c_shape():                           # ring with a 50 degree opening
    return (rr >= 6) & (rr <= 13) & (np.abs(ang) >= np.deg2rad(25))

def surface(occ):                        # occupied cells with a free 4-neighbour
    f, nb = ~occ, np.zeros_like(occ)
    nb[1:] |= f[:-1]; nb[:-1] |= f[1:]; nb[:, 1:] |= f[:, :-1]; nb[:, :-1] |= f[:, 1:]
    return occ & nb

# 36 candidate views on a circle of radius 32, each looking at the centre and
# casting 64 rays over a 60 degree field of view; a ray is its list of grid cells
V, R, DUMMY = 36, 64, N * N              # DUMMY pads short rays: free, never updated
seqs = []
for v in range(V):
    a = np.deg2rad(10 * v)
    for b in a + np.pi + np.linspace(-np.pi / 6, np.pi / 6, R):
        s = np.arange(0, 80, 0.25)
        px, py = 32 * np.cos(a) + s * np.cos(b) + C, 32 * np.sin(a) + s * np.sin(b) + C
        ok = (px >= 0) & (px < N) & (py >= 0) & (py < N)
        cells = py[ok].astype(int) * N + px[ok].astype(int)
        seqs.append(cells[np.r_[True, cells[1:] != cells[:-1]]])
L = max(len(q) for q in seqs)
RAYS = np.array([np.r_[q, np.full(L - len(q), DUMMY)] for q in seqs]).reshape(V, R, L)

SENSORS = {"soft": (0.7, 0.4), "confident": (0.97, 0.12)}   # P(occupied | hit), P(occupied | miss)

def observe(m, occ, v):
    """Cast view v's rays against the true object and update the map m (log-odds)."""
    L_HIT, L_MISS = (np.log(q / (1 - q)) for q in SENSORS[SENSOR])
    cells = RAYS[v]
    o = occ[cells]
    first = np.where(o.any(1), o.argmax(1), L)          # first occupied cell per ray
    hits = np.unique(cells[first < L, first[first < L]])
    miss = np.setdiff1d(cells[np.arange(L) < first[:, None]], np.r_[hits, DUMMY])
    m["lo"][miss] = np.clip(m["lo"][miss] + L_MISS, -2, 3.5)
    m["lo"][hits] = np.clip(m["lo"][hits] + L_HIT, -2, 3.5)
    m["seen"][miss] = m["seen"][hits] = m["hit"][hits] = True

def H(p):                                # binary entropy in bits
    p = np.clip(p, 1e-12, 1 - 1e-12)
    return -(p * np.log2(p) + (1 - p) * np.log2(1 - p))

def gain_entropy(m, cand):
    """Expected visible entropy: P(cell visible) x H(cell), summed over the rays."""
    p = 1 / (1 + np.exp(-m["lo"])); p[DUMMY] = 0.0
    pc = p[RAYS[cand]]
    vis = np.cumprod(np.concatenate([np.ones(pc.shape[:2] + (1,)), 1 - pc[..., :-1]], -1), -1)
    return (vis * H(pc)).sum((1, 2))

def gain_unknown(m, cand):
    """Number of never-observed cells in front of the first cell believed occupied."""
    cells = RAYS[cand]
    block = (m["lo"][cells] > 0) & m["seen"][cells]
    front = np.cumsum(block, -1) == 0
    return (~m["seen"][cells] & front & (cells != DUMMY)).sum((1, 2))

def run(policy, occ, start, rng):
    m = {"lo": np.zeros(N * N + 1), "seen": np.zeros(N * N + 1, bool),
         "hit": np.zeros(N * N + 1, bool)}
    order = [start]
    if policy == "random":
        order += [v for v in rng.permutation(V) if v != start]
    observe(m, occ, start)
    cov, ent = [], []
    for k in range(1, V + 1):
        cov.append(m["hit"][:-1][SURF].mean())
        ent.append(H(1 / (1 + np.exp(-m["lo"][:-1])))[ROI].sum())
        if k == V:
            break
        cand = np.array([v for v in range(V) if v not in order])
        if policy == "sweep":                           # fixed order around the circle
            nxt = (start + k) % V
        elif policy == "spread":                        # fixed order, as far as possible from visited views
            d = np.abs((cand[:, None] - np.array(order)[None, :] + V / 2) % V - V / 2).min(1)
            nxt = cand[np.argmax(d)]
        elif policy == "random":
            nxt = order[k]
        else:
            g = gain_entropy(m, cand) if policy == "IG entropy" else gain_unknown(m, cand)
            nxt = cand[np.argmax(g)]
        if policy != "random":
            order.append(nxt)
        observe(m, occ, nxt)
    return np.array(cov), np.array(ent)

policies = ["sweep", "spread", "random", "IG entropy", "IG unknown"]
K = [1, 2, 3, 4, 6, 8]
for SENSOR, (name, obj) in [(s, o) for s in SENSORS for o in [("ellipse", ellipse()), ("C-shape", c_shape())]]:
    occ = np.r_[obj.ravel(), False]
    SURF = surface(obj).ravel()
    res = {p: [run(p, occ, s, np.random.default_rng(s)) for s in range(V)] for p in policies}
    full = res["sweep"][0][0][-1]                      # coverage after all 36 views
    h0 = ROI.sum()                                     # initial entropy: 1 bit per cell
    print(f"{name}, {SENSOR} sensor: {SURF.sum()} surface cells, coverage with all 36 views {full:.3f}")
    print(f"{'':11}{'coverage after k views (mean of 36 starts)':>46}"
          f"{'views to 95%':>19}{'entropy % at k':>18}")
    print(f"{'policy':11}" + "".join(f"{k:>7}" for k in K) + f"{'mean (min-max)':>21}{'4':>8}{'8':>8}")
    for p in policies:
        cov = np.array([c for c, e in res[p]]); ent = np.array([e for c, e in res[p]])
        need = (cov >= 0.95 * full).argmax(1) + 1
        print(f"{p:11}" + "".join(f"{cov[:, k - 1].mean():7.3f}" for k in K)
              + f"{need.mean():9.1f} ({need.min():2d}-{need.max():2d})"
              + f"{100 * ent[:, 3].mean() / h0:8.1f}{100 * ent[:, 7].mean() / h0:8.1f}")
    print()

ellipse, soft sensor: 56 surface cells, coverage with all 36 views 1.000
               coverage after k views (mean of 36 starts)       views to 95%    entropy % at k
policy           1      2      3      4      6      8       mean (min-max)       4       8
sweep        0.346  0.375  0.403  0.431  0.486  0.542     23.5 (20-27)    88.2    78.9
spread       0.346  0.692  0.844  0.996  0.998  1.000      4.0 ( 4- 4)    91.2    74.8
random       0.346  0.554  0.719  0.838  0.920  0.972      6.2 ( 3-10)    90.5    75.9
IG entropy   0.346  0.609  0.943  0.948  0.979  0.979      4.6 ( 3- 6)    89.6    74.7
IG unknown   0.346  0.666  0.777  0.903  0.997  1.000      5.0 ( 4- 7)    91.0    75.6

C-shape, soft sensor: 100 surface cells, coverage with all 36 views 0.940
               coverage after k views (mean of 36 starts)       views to 95%    entropy % at k
policy           1      2      3      4      6      8       mean (min-max)       4       8
sweep        0.217  0.245  0.271  0.297  0.349  0.402     27.6 (24-35)    91.7    84.8
spread       0.217  0.434  0.585  0.731  0.787  0.819     17.1 ( 9-35)    94.1    83.3
random       0.217  0.364  0.502  0.595  0.710  0.776     20.2 ( 8-32)    93.5    83.2
IG entropy   0.217  0.424  0.626  0.643  0.684  0.714     26.5 (25-31)    93.1    81.6
IG unknown   0.217  0.427  0.581  0.707  0.814  0.858     13.1 ( 7-26)    94.3    83.0

ellipse, confident sensor: 56 surface cells, coverage with all 36 views 1.000
               coverage after k views (mean of 36 starts)       views to 95%    entropy % at k
policy           1      2      3      4      6      8       mean (min-max)       4       8
sweep        0.346  0.375  0.403  0.431  0.486  0.542     23.5 (20-27)    74.7    65.8
spread       0.346  0.692  0.844  0.996  0.998  1.000      4.0 ( 4- 4)    61.1    58.4
random       0.346  0.554  0.719  0.838  0.920  0.972      6.2 ( 3-10)    64.6    59.3
IG entropy   0.346  0.615  0.943  0.949  0.970  0.985      5.0 ( 3- 9)    70.7    67.7
IG unknown   0.346  0.666  0.777  0.903  0.997  1.000      5.0 ( 4- 7)    60.5    58.4

C-shape, confident sensor: 100 surface cells, coverage with all 36 views 0.940
               coverage after k views (mean of 36 starts)       views to 95%    entropy % at k
policy           1      2      3      4      6      8       mean (min-max)       4       8
sweep        0.217  0.245  0.271  0.297  0.349  0.402     27.6 (24-35)    79.4    70.4
spread       0.217  0.434  0.585  0.731  0.787  0.819     17.1 ( 9-35)    67.6    58.8
random       0.217  0.364  0.502  0.595  0.710  0.776     20.2 ( 8-32)    69.1    61.5
IG entropy   0.217  0.424  0.626  0.643  0.684  0.711     26.5 (25-31)    73.5    70.1
IG unknown   0.217  0.427  0.581  0.707  0.814  0.858     13.1 ( 7-26)    65.9    58.9

The fixed sweep is the worst order on both objects because neighbouring views 10° apart see almost the same cells; it needs most of the circle to cover the surface. For the convex ellipse, choosing views well is a matter of geometry alone: four views 90° apart see the whole outline, and the spread order, which never looks at the map, reaches 95% coverage in exactly four views from every start. Neither information-gain policy reaches 95% sooner. The C-shape is different. Its inner wall can be seen only from the few views facing the opening, and a fixed order reaches them only by chance, which is why spread needs anywhere from 9 to 35 views depending on where it starts. The unknown-cell count steers toward views that can still see unobserved cells and covers the surface in 13.1 views on average, against 17.1 for spread and 20.2 for random. Random views are acceptable on average; on the ellipse their range over starts is wider than that of either information-gain policy, which echoes the larger variance Delmerico et al. (2018) found for random views, while on the C-shape the fixed spread order varies as much.

The entropy policy is the surprise: on the C-shape it covers less surface than random after eight views and needs almost as many views as the sweep. The printed tables do not show why; the following is our reasoning from the gain function in the code. A free cell observed once keeps much of its entropy, 0.97 bit at probability 0.4 under the soft sensor and 0.53 bit at probability 0.12 under the confident one, while the visibility product, a factor of $(1 - 0.4)$ or $(1 - 0.12)$ for each observed free cell a ray crosses, shrinks along the ray. Already-seen free space near the camera can then carry much of a view’s predicted gain, which would favour views that look through it again rather than views that reach unseen surface. Under the soft sensor the entropy policy has the lowest entropy on the C-shape after eight views (81.6%) and the worse coverage of the two information-gain policies, as repeated looks at free space would produce. Under the confident sensor a second miss can move a free cell only from 0.12 to the clamp at 0.119, so repeated looks would change almost nothing, and the policy trails every policy but the sweep on both scores. The other four policies cover the same surface under both sensors: with a geometrically perfect sensor a cell is only ever hit or only ever missed, so their choices, which depend only on which cells have been seen and on the sign of their log-odds, are the same. The soft-sensor result is consistent with what Delmerico et al. (2018) and Isler et al. (2016) report from their 3D simulations: entropy reduction and surface coverage are different targets, and an occlusion-weighted entropy gain does not by itself focus on the object.

The first run of the script printed only the soft-sensor tables. After that run, a diagnostic script (not shown) that broke the entropy policy’s predicted gain down by cell type suggested that its result depends on the miss probability, a sensor model the sources leave open, so the confident sensor was added as a second case; the soft-sensor tables did not change. The example is two-dimensional, has a geometrically perfect sensor, knows the region that contains the object, ignores the cost of moving between views, and selects greedily one view at a time, so it shows the mechanics of the gain functions, not the performance of any published planner.

Common Misconceptions

• “Active perception means using active sensors.” Lidar, structured light and time-of-flight cameras are called active sensors because they emit energy, as in the 3D reconstruction article’s section on active sensing. Bajcsy’s active sensing is about control: a passive camera moved and refocused purposefully is active in her sense (Bajcsy, 1988, technical report), and by the same definition a lidar spinning in a fixed pattern is not.
• “Any moving camera is active.” A scan along a pre-planned path, or a camera that sweeps regardless of what it sees, controls where it senses but not why. Bajcsy et al. (2018) exclude fixed, pre-specified control loops from their definition.
• “More information gain always means better results.” Gain is measured against a model of uncertainty. A measure that rewards reducing entropy can spend views on empty space and repeated observations while the object stays incomplete, as both the Delmerico et al. (2018) comparison and the example above show.
• “Attention and active perception are the same thing.” Selecting regions inside an image already captured is attention; choosing which image to capture is active perception. Bajcsy et al. (2018) regard active perception as a subset of attentive processing, and Jayaraman and Grauman (2018) draw the same line between saliency, which prioritizes regions of captured data, and choosing a new observation.
• “Next-best-view always means moving a sensor.” In incremental structure from motion the next best view is the next image to register from a fixed collection, a processing order rather than an action.
• “Active is always more efficient.” It can reduce the total cost of interpretation sharply, but only under some conditions (Tsotsos, 1992, as described by Bajcsy et al., 2018), and moving a sensor costs time and energy that a passive system does not pay.

Where It Is Used

• 3D reconstruction and inspection. Next-best-view planners choose sensor poses for reconstructing objects and scenes with depth cameras, stereo or multi-view stereo, trading information gain against motion cost (Isler et al., 2016; Delmerico et al., 2018).
• Exploration and mapping. Frontier-based and learned exploration build maps of unknown buildings for SLAM and visual navigation; Chaplot et al. (2020) place their agent among active SLAM methods, which control the camera for map building.
• Manipulation. Finding, verifying and grasping objects in clutter, Bajcsy et al.’s kitchen example, uses viewpoint changes and actions that move occluders; robotic grasping and vision-based manipulation describe the downstream tasks.
• Recognition and search. Active object recognition and object search choose views to separate hypotheses or to find a target, from Wilkes and Tsotsos’s viewpoint control to learned look-around policies (Jayaraman & Grauman, 2018).
• Embodied agents. Navigation, embodied question answering and instruction following in embodied AI all require agents to choose their own observations; vision-and-language navigation and vision for drones and other aerial robots are further settings.
• Camera control. Gaze control, fixation and tracking on robot heads and pan-tilt cameras, and the visual feedback loops of visual servoing.