Motion & Tracking

Optical Flow

The apparent motion of image content between two frames, represented as a two-dimensional displacement at every pixel.

intermediate

Optical flow is the apparent motion of image content between two frames of a video. It describes, for every pixel, where that part of the image moved. Optical flow is one of the basic ways computer vision represents motion, and it underlies video stabilization, action recognition, visual odometry, and frame interpolation.

This article explains what optical flow is and what its properties are. Computing it from a pair of frames is a separate problem, optical flow estimation, with methods ranging from Lucas–Kanade and Horn–Schunck to deep networks such as RAFT.

Definition

Optical flow is a field of two-dimensional vectors, one per pixel, giving the displacement of the image content at that pixel from one frame to the next. It describes how brightness patterns move in the image, which is not necessarily how objects move in the world.

Intuition

Imagine drawing a small arrow at every pixel of a video frame, pointing to where that bit of the image will be in the next frame. A car driving to the right is covered in arrows pointing right. The static background has arrows of zero length. If the camera itself moves forward, every arrow points away from the spot the camera is heading toward, and arrows near the edges of the frame are longer.

That collection of arrows is the optical flow. It is usually drawn either as a sparse grid of arrows or as a color image in which the hue shows the direction of motion and the brightness or saturation shows its speed.

Formal Definition

For two frames I(x,t)I(\mathbf{x}, t) and I(x,t+1)I(\mathbf{x}, t + 1), the optical flow is a vector field

w(x)=(u(x),v(x))\mathbf{w}(\mathbf{x}) = \big(u(\mathbf{x}), v(\mathbf{x})\big)

that maps each pixel x=(x,y)\mathbf{x} = (x, y) in the first frame to its corresponding position x+w(x)\mathbf{x} + \mathbf{w}(\mathbf{x}) in the second. The components uu and vv are horizontal and vertical displacements, measured in pixels per frame.

Correspondence is usually defined through brightness constancy: a point keeps the same intensity as it moves.

I(x,y,t)=I(x+u,y+v,t+1)I(x, y, t) = I(x + u, y + v, t + 1)

For small displacements, a first-order Taylor expansion gives the optical flow constraint equation:

Ixu+Iyv+It=0I_x u + I_y v + I_t = 0

Here IxI_x and IyI_y are the spatial image gradients and ItI_t is the temporal change in intensity. In continuous time, u=dx/dtu = \mathrm{d}x/\mathrm{d}t and v=dy/dtv = \mathrm{d}y/\mathrm{d}t are image velocities rather than displacements.

Properties

Optical flow is not the motion field

The motion field is the projection of the true 3D motion of scene points onto the image. Optical flow is what can be observed from changing brightness. The two usually agree, but not always:

  • A smooth, uniformly colored sphere rotating in place has a non-zero motion field but produces almost no visible change, so its optical flow is close to zero.
  • A static scene under a moving light source has a zero motion field, but the changing shading produces non-zero optical flow.

Only normal flow is observable locally

The constraint equation is one equation in two unknowns. At a single pixel, it determines only the component of motion along the gradient direction, called the normal flow:

wn=−It∥∇I∥w_n = \frac{-I_t}{\lVert \nabla I \rVert}

Motion along an edge produces no change in intensity and cannot be observed locally. This is the aperture problem: seen through a small window, a moving straight edge looks the same whatever its motion along the edge. Recovering the full flow always requires an additional assumption, such as constant flow within a neighborhood or smoothness across the image.

Flow is undefined in some places

Pixels that become occluded between frames have no corresponding point in the second frame, and newly revealed pixels have none in the first. In regions without texture, any displacement fits the data equally well. Flow is also discontinuous at motion boundaries, where surfaces moving differently meet.

Flow depends on depth and camera motion

In a static scene viewed by a moving camera, the flow at each pixel splits into two parts:

  • a rotational component, caused by the camera turning, which does not depend on scene depth;
  • a translational component, caused by the camera moving, whose magnitude is inversely proportional to the depth of the point.

When the camera translates without rotating, all flow vectors radiate from a single point, the focus of expansion, which marks the direction of travel. Nearby objects produce longer vectors than distant ones. This is why optical flow carries information about both camera motion and scene structure.

Direction and units matter

Flow from frame tt to t+1t + 1 (forward flow) is not simply the negative of flow from t+1t + 1 to tt (backward flow), because the two are sampled on different pixel grids and differ at occlusions. Flow magnitudes are measured in pixels per frame, so they depend on image resolution and frame rate.

Dense and sparse flow

Flow can be represented at every pixel (dense flow) or only at selected points, usually corners where both gradient directions are present (sparse flow). Sparse flow at well-textured points is the basis of feature tracking.

Examples

  • Camera moving forward. Flow vectors point outward from the focus of expansion and grow toward the edges of the frame.
  • Camera panning. A camera rotating about its vertical axis produces nearly uniform horizontal flow across the whole image, independent of depth.
  • Moving object, static camera. Flow is non-zero only on the object, with sharp discontinuities at its boundary.
  • The barber pole. The stripes on a rotating barber pole move horizontally, yet they appear to move along the pole. Locally, only the motion perpendicular to the diagonal stripes is visible, and the stripe ends along the pole’s long edges move vertically, so the apparent motion differs from the true motion.

Common Misconceptions

  • “Optical flow is the motion of objects.” It is the motion of image brightness patterns. It differs from true motion under lighting changes, on textureless surfaces, and at specular highlights.
  • “Every pixel has a correct flow vector.” Occluded and newly revealed pixels have no correspondence, and textureless regions have no unique answer.
  • “Longer vectors mean faster objects.” Flow magnitude depends on depth and camera motion as well as object speed. A slow object near the camera can produce more flow than a fast one far away.
  • “Flow appears only where something moves.” A moving camera produces flow everywhere, even in a completely static scene.

Practical Example

Computing dense flow with Farnebäck’s method in OpenCV and visualizing it as a color image, with hue for direction and brightness for speed:

import cv2
import numpy as np

previous = cv2.imread("frame_000.png", cv2.IMREAD_GRAYSCALE)
current = cv2.imread("frame_001.png", cv2.IMREAD_GRAYSCALE)

flow = cv2.calcOpticalFlowFarneback(
    previous, current, None,
    pyr_scale=0.5, levels=3, winsize=15,
    iterations=3, poly_n=5, poly_sigma=1.2, flags=0,
)

# flow has shape (height, width, 2): the (u, v) displacement of each pixel.
magnitude, angle = cv2.cartToPolar(flow[..., 0], flow[..., 1])

hsv = np.zeros((*previous.shape, 3), dtype=np.uint8)
hsv[..., 0] = angle * 180 / np.pi / 2  # hue: direction of motion
hsv[..., 1] = 255
hsv[..., 2] = cv2.normalize(magnitude, None, 0, 255, cv2.NORM_MINMAX)  # brightness: speed
visualization = cv2.cvtColor(hsv, cv2.COLOR_HSV2BGR)

Where It Is Used

  • Feature tracking. Sparse flow at corner points, typically with the Lucas–Kanade method, follows features across frames (feature tracking).
  • Visual odometry and structure from motion. The way flow varies across the image constrains camera motion and scene depth.
  • Video stabilization. Estimated global motion is smoothed and compensated to remove camera shake.
  • Action recognition. Two-stream networks use flow as a second input alongside appearance.
  • Video frame interpolation and slow motion. Intermediate frames are synthesized by moving pixels part of the way along their flow vectors.
  • Motion segmentation. Regions that share consistent flow are grouped into independently moving objects.
  • Video compression. Codecs use block motion vectors, a coarse form of flow, to predict each frame from its neighbors.

The classic estimation methods include Lucas–Kanade, which assumes constant flow in a small window, Horn–Schunck, which assumes smooth flow across the image, and Farnebäck’s polynomial expansion. Modern learning-based methods such as RAFT are trained on large datasets and handle occlusions, large motions, and textureless regions considerably better. They are evaluated on benchmarks such as Middlebury, MPI Sintel, and KITTI, typically using the endpoint error between estimated and ground-truth flow vectors.

Related

  • Feature Tracking

    How distinctive image points are selected and followed across video frames, using the classic KLT tracker as the main example.

  • Optical Flow Estimation

    Computing a dense field of pixel displacements between two video frames, from classical variational methods to learned networks such as RAFT.

  • Brightness Constancy Assumption

    The assumption that a scene point keeps the same image intensity as it moves between frames, which turns motion estimation into an intensity-matching problem.

  • Aperture Problem

    Why motion seen through a small window is ambiguous along edges, so that only the component of motion across an edge can be measured locally.

  • Motion Estimation

    Recovering how image content, objects, or the camera moved from a sequence of images, from per-pixel flow to global and 3D motion.

  • MPI Sintel

    An optical flow benchmark rendered from the open-source animated film Sintel, with long sequences, large motions, motion blur, and atmospheric effects.

References

  1. Horn, B. K. P. & Schunck, B. G. (1981). Determining Optical Flow. Artificial Intelligence, 17(1–3), 185–203.
  2. Lucas, B. D. & Kanade, T. (1981). An Iterative Image Registration Technique with an Application to Stereo Vision. Proceedings of the 7th International Joint Conference on Artificial Intelligence (IJCAI), 674–679.
  3. Barron, J. L., Fleet, D. J. & Beauchemin, S. S. (1994). Performance of Optical Flow Techniques. International Journal of Computer Vision, 12(1), 43–77.
  4. Baker, S., Scharstein, D., Lewis, J. P., Roth, S., Black, M. J. & Szeliski, R. (2011). A Database and Evaluation Methodology for Optical Flow. International Journal of Computer Vision, 92(1), 1–31.
  5. Farnebäck, G. (2003). Two-Frame Motion Estimation Based on Polynomial Expansion. Scandinavian Conference on Image Analysis (SCIA), 363–370.