Aperture Problem

The aperture problem is the ambiguity in motion seen through a small window: when the window contains only a straight edge or a set of parallel stripes, any motion along the edge leaves the image inside the window unchanged. Only the component of motion perpendicular to the edge, the normal flow, can be measured. It affects every local motion measurement, whether by a gradient-based method, a correlation window, or a direction-selective neuron, and every method that estimates optical flow must resolve it with information from outside the window.

Definition

The aperture problem is the fact that the motion of a one-dimensional intensity pattern, such as an edge, a line, or a grating, is determined only up to its component along the direction of intensity change, unless the aperture also contains the pattern’s ends or other structure.

Intuition

Look at a long diagonal pole through a hole in a sheet of cardboard as the pole slides past. Whether it moves horizontally, vertically, or along its own length, the visible section looks like the same diagonal line moving perpendicular to itself, because motion along the line produces no visible change.

The ambiguity disappears as soon as the window contains something that breaks the symmetry: the end of the pole, a knot in the wood, or a second edge at a different angle. The problem is therefore not the size of the window as such, but whether it contains intensity variation in more than one direction.

Formal Definition

Under brightness constancy, linearized for small motion, each pixel gives one linear equation in the two components of the flow $\mathbf{w} = (u, v)$:

$$\nabla I^\top \mathbf{w} + I_t = 0.$$

Decompose $\mathbf{w}$ into a component along the unit gradient direction $\mathbf{n} = \nabla I / \lVert \nabla I \rVert$ and a component along the perpendicular, tangential direction $\mathbf{t}$:

$$\mathbf{w} = w_n\, \mathbf{n} + w_t\, \mathbf{t}.$$

Since $\nabla I^\top \mathbf{t} = 0$, the equation reduces to $\lVert \nabla I \rVert\, w_n + I_t = 0$, which determines $w_n$ (the scalar normal flow in optical flow) and says nothing about $w_t$. The vector

$$\mathbf{w}_n = -\frac{I_t}{\lVert \nabla I \rVert^2}\, \nabla I$$

is the normal flow. It is the shortest flow vector consistent with the measurement, and the true flow is the normal flow plus an arbitrary multiple of $\mathbf{t}$.

The constraint line

In velocity space, the plane with axes $u$ and $v$, the constraint equation describes a straight line: the set of all motions consistent with the measurement at that pixel. The line is perpendicular to the gradient, and the normal flow is its closest point to the origin.

If two measurements come from differently oriented edges of the same moving surface, their lines intersect in a single point, the true velocity. This construction is the intersection of constraints. With many noisy measurements the lines do not meet exactly, and the velocity is estimated as the point closest to all of them in the least-squares sense.

Properties

Classifying local structure

Collecting the constraints from all pixels in a window $\Omega$ and solving by least squares leads to the normal equations

$$\mathbf{M}\, \mathbf{w} = -\sum_{\Omega} I_t\, \nabla I, \qquad \mathbf{M} = \sum_{\Omega} \nabla I\, \nabla I^\top.$$

The $2 \times 2$ matrix $\mathbf{M}$, known as the structure tensor or second-moment matrix, summarizes how the gradient directions in the window are distributed. Its eigenvalues $\lambda_1 \ge \lambda_2 \ge 0$ classify the window:

• Uniform region, $\lambda_1 \approx \lambda_2 \approx 0$. Every velocity is consistent with the data, a more severe ambiguity than the aperture problem.
• Edge or grating, $\lambda_1 \gg \lambda_2 \approx 0$. All constraint lines are parallel. The eigenvector of $\lambda_1$ is the edge normal, and only the normal flow is determined. This is the aperture problem.
• Corner or texture, both eigenvalues large. Constraint lines with different orientations intersect, and the full velocity is determined.

The same matrix is used to select trackable points (feature tracking): a point is trackable exactly when the aperture problem does not arise in its window.

Uncertainty is anisotropic

If the temporal derivatives carry independent noise of variance $\sigma^2$ and the spatial gradients are treated as exact, the covariance of the least-squares velocity is $\sigma^2 \mathbf{M}^{-1}$. Near an edge this uncertainty ellipse is elongated along the edge. The aperture problem is the limiting case of a continuum: a window with weak texture along an edge is not strictly ambiguous, but its tangential velocity is much less reliable than its normal velocity.

It also applies to matching

A correlation or block-matching search along an edge finds a valley of equally good matches rather than a single minimum. In rectified stereo, where correspondences are searched along horizontal scanlines, edges parallel to the scanlines produce the same ambiguity in disparity.

Resolving the Ambiguity

Every method that recovers full flow adds an assumption that links measurements made at different places.

• Local aggregation. Assume the flow is constant, or affine, within a window and solve the overdetermined system by least squares, as the Lucas–Kanade method does. It succeeds where $\mathbf{M}$ is well conditioned and fails, or should report low confidence, where it is not.
• Global smoothness. Horn and Schunck added a penalty on spatial variation of the flow and minimized it with the constraint over the entire image. Velocities measured reliably at corners and in texture propagate along edges and into uniform regions, at the cost of over-smoothing at motion boundaries.
• Smoothness along contours. Hildreth proposed choosing, among all velocity fields consistent with the normal flow measured along a contour, the one that varies least along the contour. For a translating curved contour this is the true motion, since a constant field has no variation, and Hildreth reported that the method’s predictions are consistent with human motion perception.
• Intersection of constraints. Intersect the constraint lines of differently oriented components of the same surface, as described above.
• Feature-based motion. Track only points where the problem does not arise, such as corners, line endings, and blobs, and infer the motion elsewhere from them.

Examples

• Moving bar. A long horizontal bar moving diagonally appears, in the middle of its length, to move straight up or down. Only its two ends reveal the horizontal component.
• Barber pole. Diagonal stripes on a rotating cylinder move horizontally, yet appear to move along the pole’s long axis. Most visible stripe ends lie along the pole’s long edges and move along them, and perception follows these terminators rather than the stripes’ normal flow, as Wallach proposed.
• Plaids. Two superimposed gratings at different orientations can be seen as two gratings sliding over each other or as a single plaid moving coherently in the direction predicted by the intersection of constraints. Adelson and Movshon studied when the coherent percept occurs.
• Lane markings. In driving video, a long continuous lane line gives almost no information about motion along it, while dashed markings and their ends do.

Perception and History

The aperture problem was first studied in human vision. Wallach’s 1935 experiments on lines moving behind apertures of different shapes showed that perceived direction depends on the aperture’s shape: behind a rectangular aperture, lines appear to move along its longer side, which Wallach attributed to the line endings at the aperture’s borders. Wuerger, Shapley, and Rubin published an English translation with commentary in 1996. Marr and Ullman analyzed early motion detection computationally: a local measurement at an oriented contour constrains the direction of motion only to within 180°, so the direction must be recovered in a second stage that combines local constraints. Adelson and Movshon’s plaid experiments, with physiological work on primate visual cortex, made the problem central to theories of motion integration.

Common Misconceptions

• “A larger window solves the aperture problem.” It helps only if the larger window contains structure at other orientations, such as the corners of a square whose side fills a small window. A larger window on a long straight edge is no better, and windows that span motion boundaries create new errors.
• “Normal flow is wrong.” It is the correct, complete answer to what a single edge can tell you. It is wrong only when interpreted as the full motion.
• “The aperture problem is specific to differential methods.” It is a property of the image data. Correlation, block matching, and learned features all face it on one-dimensional structure, although learned methods hide it behind large receptive fields.

Practical Example

A sinusoidal grating, with intensity varying along a direction 60° from horizontal, moves one pixel to the right. The least-squares system over the whole image is rank-deficient, and its minimum-norm solution is the normal flow. Adding a perpendicular grating makes the structure tensor full rank.

import numpy as np

y, x = np.mgrid[0:128, 0:128].astype(float)
true_w = np.array([1.0, 0.0])  # true motion (u, v) in pixels per frame

def grating(theta, shift):
    """Sinusoidal stripes whose intensity varies along the direction theta."""
    n = np.array([np.cos(theta), np.sin(theta)])
    return np.sin(0.3 * ((x - shift[0]) * n[0] + (y - shift[1]) * n[1]))

def analyze(make_image):
    I0, I1 = make_image(np.zeros(2)), make_image(true_w)
    Iy, Ix = np.gradient((I0 + I1) / 2)
    It = I1 - I0
    Ix, Iy, It = Ix[1:-1, 1:-1], Iy[1:-1, 1:-1], It[1:-1, 1:-1]  # drop one-sided borders
    A = np.stack([Ix.ravel(), Iy.ravel()], axis=1)
    eigenvalues = np.linalg.eigvalsh(A.T @ A) / A.shape[0]
    w = np.linalg.lstsq(A, -It.ravel(), rcond=1e-3)[0]  # minimum-norm solution
    print(f"eigenvalues {np.round(eigenvalues, 4)}   estimated w = {np.round(w, 2)}")

theta = np.deg2rad(60)
n = np.array([np.cos(theta), np.sin(theta)])
print("normal flow of single grating:", np.round((true_w @ n) * n, 2))

analyze(lambda s: grating(theta, s))                                 # edge-like: one direction
analyze(lambda s: grating(theta, s) + grating(theta + np.pi / 2, s))  # plaid: two directions

Output:

normal flow of single grating: [0.25 0.43]
eigenvalues [0.     0.0439]   estimated w = [0.25 0.44]
eigenvalues [0.0434 0.0439]   estimated w = [ 1.01 -0.  ]

With one grating, an eigenvalue is zero and the estimate is the normal flow, not the true motion $(1, 0)$. With the plaid, both eigenvalues are large and the true motion is recovered, as in the intersection of constraints.

Where It Is Used

• Point selection. Trackers select features where both eigenvalues of $\mathbf{M}$ are large (feature tracking).
• Drift along edges. A tracked point that ends up on an edge, for example because its corner is occluded, can slide along the edge while the residual stays small. Monitoring the conditioning of $\mathbf{M}$ detects this.
• Confidence estimates. The anisotropic uncertainty $\sigma^2 \mathbf{M}^{-1}$ can be passed to later stages, such as a Kalman filter or bundle adjustment, so that measurements are trusted only in informative directions.
• Motion estimation with parametric models. Estimating a global motion, such as a homography, pools constraints over the whole image and is rarely affected, because natural images contain edges at many orientations.