Scene Flow

Scene flow is the three-dimensional motion of every visible point of a scene between two moments in time. It is to the 3D world what optical flow is to the image plane: a dense field of displacement vectors, but in metres along $X$, $Y$ and $Z$ rather than in pixels along two image axes. Sundar Vedula, Simon Baker, Peter Rander, Robert Collins and Takeo Kanade named and defined it at ICCV 1999. Estimating it needs depth as well as motion, so the input is stereo video, RGB-D frames from a depth camera, or a pair of lidar point clouds. The output tells a robot or a car which parts of the world are moving, in which direction, and how fast, independently of how far away they are.

Motion Estimation introduces scene flow as one variant of the motion problem; this article covers its relation to optical flow and disparity, its benchmarks and the main families of methods.

Problem Definition

Vedula et al. (1999) defined scene flow as the three-dimensional motion field of points in the world, and observed that any optical flow is simply the projection of the scene flow onto the image plane of a camera. For a point $\mathbf{x}(t)$ on a moving, possibly non-rigid surface, imaged by a fixed camera $i$ at image position $\mathbf{u}_i(t)$, the two flows are linked by the chain rule:

$$\frac{d\mathbf{u}_i}{dt} = \frac{\partial \mathbf{u}_i}{\partial \mathbf{x}}\,\frac{d\mathbf{x}}{dt},$$

where $\partial \mathbf{u}_i / \partial \mathbf{x}$ is the $2 \times 3$ Jacobian of the camera’s projection and $d\mathbf{x}/dt$ is the scene flow. Each camera’s optical flow therefore gives two linear constraints on the three unknowns of the scene flow at a point. Inverting the relation is impossible from one camera without knowing the surface, because the projection loses the depth direction.

In the discrete two-frame form used by the benchmarks below, the task is: given observations at times $t$ and $t+1$, estimate for each point $\mathbf{X}_1 \in \mathbb{R}^3$ seen at time $t$ its displacement $\mathbf{s} = \mathbf{X}_2 - \mathbf{X}_1$, where $\mathbf{X}_2$ is the same physical point at $t+1$. Methods that work from images usually also estimate $\mathbf{X}_1$ itself, so stereo scene flow is a joint estimate of geometry and motion. Mayer et al. (2016) put it this way: for a stereo pair at two times, scene flow gives each visible point its 3D position and its 3D motion vector, and these 3D quantities can only be computed when the camera intrinsics and extrinsics are known.

Several assumptions recur:

• Calibrated sensors. Metric 3D motion requires known intrinsics and baseline, or a depth sensor.
• Camera-relative motion. Measured from a moving sensor, a static point has non-zero scene flow, the inverse of the sensor’s own motion. In KITTI the cameras move with the car, so the static background moves relative to them; Menze and Geiger (2015) note that it moves rigidly with respect to the observer and treat it as one more rigidly moving object.
• Visibility. Points occluded in some of the images have no measurement there; methods must either leave them out or infer their motion from a prior.

Input and Output

The output can be written in several equivalent forms, and the choice matters for how methods are built and evaluated.

• 3D vectors. One vector $\mathbf{s} \in \mathbb{R}^3$ per pixel of a reference image, or per point of a point cloud. This is the form of Vedula et al., of FlowNet3D and of the point-cloud benchmarks.
• Image-space components. For a rectified stereo rig, the four numbers $(d_1, d_2, u, v)$ at each reference pixel: the disparity at $t$, the disparity at $t+1$ of the same point, and its optical flow. Huguet and Devernay (2007) estimate exactly these four unknowns per pixel. Mayer et al. (2016) describe the same camera-independent split into optical flow, disparity and disparity change $d_2 - d_1$, and note that it is complete: given camera parameters, the 3D points and their motion follow. KITTI 2015 evaluates scene flow in this form.
• Rigid motions. A rigid-body transform $\mathbf{T} \in SE(3)$ per pixel, segment or object, from which 3D displacements follow. SphereFlow (Hornáček et al., 2014) outputs a dense field of 6-degree-of-freedom motions rather than the usual 3D translations; Object Scene Flow (Menze and Geiger, 2015) assigns each superpixel to one of a few rigidly moving objects; RAFT-3D (Teed and Deng, 2021) estimates an $SE(3)$ transform at every pixel.

The inputs are two stereo pairs (four images), two RGB-D frames (two images with registered depth maps), or two point clouds with no correspondence between their points.

Relation to Optical Flow and Disparity

For a rectified stereo pair with focal length $f$ in pixels, baseline $B$ and principal point $(c_x, c_y)$, a reference pixel $(x, y)$ with disparity $d_1$ back-projects to

$$\mathbf{X}_1 = \frac{fB}{d_1}\left(\frac{x - c_x}{f},\ \frac{y - c_y}{f},\ 1\right)^\top,$$

and the same point at $t+1$, seen at $(x + u, y + v)$ with disparity $d_2$, back-projects to $\mathbf{X}_2$ by the same formula. The scene flow is $\mathbf{s} = \mathbf{X}_2 - \mathbf{X}_1$. Read the other way, a known scene flow induces the optical flow

$$u = f\,\frac{X_1 + s_X}{Z_1 + s_Z} - f\,\frac{X_1}{Z_1}, \qquad v = f\,\frac{Y_1 + s_Y}{Z_1 + s_Z} - f\,\frac{Y_1}{Z_1},$$

and the disparity change

$$d_2 - d_1 = fB\left(\frac{1}{Z_1 + s_Z} - \frac{1}{Z_1}\right) \approx -\frac{fB}{Z_1^2}\,s_Z .$$

Three consequences follow from these formulas (they are our reasoning, not a quoted result):

• Motion in depth is seen only through the disparity change. The optical flow of a point moving straight along its viewing ray is zero; only $d_2 - d_1$ reveals the motion. Conversely, motion parallel to the image plane leaves the disparity unchanged and appears only in the optical flow.
• Depth motion is hard to measure far away. The disparity change shrinks with $1/Z^2$. A point at 40 m moving 0.4 m towards a camera with $fB = 350$ pixel-metres changes its disparity by less than a tenth of a pixel, well below the precision of most stereo matchers.
• Errors in the two disparities add. Since $s_Z = Z_2 - Z_1$, its error combines the depth errors at both times. With the first-order depth error $\Delta Z \approx Z^2 \Delta d / (fB)$ derived in Stereo Vision, independent disparity errors of standard deviation $\sigma_1$ and $\sigma_2$ give a depth-direction scene-flow error of about $\sqrt{(Z_1^2 \sigma_1)^2 + (Z_2^2 \sigma_2)^2}\,/\,(fB)$, which grows with the square of distance.

Mayer et al. (2016) make the related point that reconstructing 3D motion from flow and disparities is sensitive to noise, because a small error in the optical flow can cause a large error in the 3D motion vector, and that when only optical flow and the two disparities are given, as in KITTI 2015, scene flow can be reconstructed only for surface points visible in both the left and the right image.

The relation also explains why scene flow is not just optical flow plus stereo. Estimating flow in each view and disparity at each time separately gives four quantities that need not describe one consistent 3D motion, and each is subject to its own aperture problem. The authors’ submitted manuscript of the TPAMI version of Vedula et al. argues that the fundamental limitation of optical flow, that only normal flow can be computed from image measurements without regularization, applies to scene flow too, however many cameras are used, except at points with more than one-dimensional structure such as corners. Vogel et al. (2013) call this the 3D equivalent of the aperture problem. Every method therefore adds a prior: smoothness of the 3D motion, local rigidity, piecewise planar geometry, a small number of moving objects, or a learned prior.

Variants

• Multi-view scene flow. Many calibrated cameras watch a non-rigid scene, as in the 51-camera CMU Virtualized Reality dome used by Vedula et al. (1999).
• Stereo scene flow. Two consecutive stereo pairs of a calibrated rig, the setting of KITTI 2015 and of most driving work.
• RGB-D scene flow. Two frames from a depth camera such as the Kinect, which gives depth directly. Menze and Geiger (2015) note that the Kinect works well indoors and, being interested in outdoor driving, focused on appearance-based methods instead.
• Point-cloud scene flow. Two lidar sweeps or other point sets; the flow is estimated for each point of the first cloud.
• Monocular scene flow. A single camera sees motion in depth only up to scale. RAFT-3D’s authors describe a method by Yang and Ramanan that predicts optical expansion, the change in an object’s apparent size, and combines it with optical flow to obtain scene flow up to a scale that lidar, stereo or monocular depth estimation can fix.
• Multi-frame scene flow. Using more than two time steps; the KITTI 2015 data include a multi-view extension with 20 frames per scene, and the leaderboard marks methods that use more than two temporally adjacent images.

Challenges

• Small depth changes at long range. The disparity change of a distant object is tiny, as derived above.
• Large displacements. Objects close to a moving vehicle move far in the image. Menze and Geiger reported motions of up to 500 pixels near the image boundaries in their annotations, and found that none of the existing optical flow or scene flow methods coped with the most extreme motions in their benchmark.
• Textureless and reflective surfaces. Menze and Geiger named textureless and reflecting surfaces as a second source of failure, because stereo matching and optical flow estimation often both fail there.
• Occlusion. With four images, a point can be hidden in any of them; disocclusions at depth discontinuities are common in driving scenes.
• Non-rigid motion. Pedestrians and cyclists violate the rigid-object priors that work well for cars. KITTI 2015 masks them out of the ground truth.
• Ground truth. No sensor measures dense 3D motion in complex real scenes, so benchmarks rely on rendering or on laborious annotation.

Approaches

Scene flow from optical flows

Vedula et al. (1999) computed dense, non-rigid scene flow from optical flow and organized the task into three scenarios by how much is known about the scene’s structure:

1. Complete knowledge of the structure, including surface normals and the rate of change of the depth maps: one optical flow suffices.
2. Only stereo correspondences: the two constraints per camera are stacked into a linear system $B\,\frac{d\mathbf{x}}{dt} = U$ with $2N$ equations for $N$ cameras, solved in the least-squares sense with a singular value decomposition. The system is degenerate only when the point and all camera centres are collinear. They used optical flows from 15 cameras so that every point was seen by at least two or three.
3. No knowledge of the surface: the inconsistency of many optical flows constrains where the surface can be, which they used to recover scene structure from flows in all 51 dome cameras. That flow-based reconstruction recovered structure only where the scene actually moved.

Vogel et al. (2013) describe this approach as two independent steps, flow in each view and then a 3D fit, without requiring that the flows be projections of one 3D motion.

Variational stereo scene flow

The next family estimates disparity and flow jointly from a stereo sequence by minimizing an energy with a data term and a smoothness term, in the tradition of Horn and Schunck’s variational optical flow, solved coarse to fine. Huguet and Devernay (2007) parametrized the scene flow by the optical flow $(u, v)$ in the left image and the disparities $d$ and $d'$ at the two times, and wrote

$$E(u, v, d, d') = E_\text{data} + \alpha E_\text{smooth},$$

where the data term sums four robust brightness- and gradient-constancy terms, one for each relation among the four images: flow in the left view, flow in the right view, stereo at $t$, and stereo at $t+1$. The smoothness term penalizes the gradients of the flow, of the disparity and of the disparity change $d' - d$. Their method estimated the 3D reconstruction at $t$ and the scene flow in a single optimization, and they wrote that earlier variational methods had estimated the two separately.

Vogel et al. (2013) summarize the alternatives that followed. Wedel et al. precomputed the disparity at each time step and then estimated the optical flow in the reference view and the disparity differences, a decoupled approach whose 2D regularizer, in Vogel et al.’s words, encourages smooth projections rather than smooth 3D scene flow. Basha et al. parametrized the problem by depth and a 3D motion vector relative to a reference view, and Vogel et al.’s own earlier work replaced the total variation prior on 3D motion with a prior that encourages locally rigid motion.

Piecewise rigid models

Driving scenes consist mostly of a static background and a few vehicles, each moving rigidly, and piecewise rigid methods build that structure into the representation.

Piecewise Rigid Scene Flow (PRSF; Vogel, Schindler and Roth, ICCV 2013) represents the scene as a collection of planar, rigidly moving segments. Each segment has a 3D plane and a rigid motion, nine unknowns, which is why the estimation starts from superpixels rather than pixels. A large set of candidate moving planes is generated, and scene flow becomes a labelling problem: assign each segment a moving plane, then refine the pixel-to-segment assignment to repair superpixel boundaries, with occlusion reasoning at both levels. Inference uses fusion moves with QPBO. On the training set of the KITTI stereo and flow data of the time, the authors’ per-pixel variant beat all three of their baselines, which included implementations of two recent 3D scene flow methods, by 33% to 44% on average, depending on the proposal set, and on the official KITTI optical flow benchmark their method ranked first among 28 published methods at the time of writing. They also noted the limits of that benchmark for scene flow: it had ground truth only for disparity and 2D flow, and independently moving objects usually had none. A journal version (IJCV, 2015) was not read for this article.

Object Scene Flow (OSF; Menze and Geiger, CVPR 2015) goes one step further and assumes a small number of rigidly moving objects. Each superpixel $i$ carries a plane $\mathbf{n}_i \in \mathbb{R}^3$ (with $\mathbf{n}_i^\top \mathbf{x} = 1$ for points on the plane) and an index $k_i$ of the object it belongs to; each object carries a rigid motion $\mathbf{o} \in SE(3)$. A superpixel is warped into each of the other three images by the homography induced by its plane and by a transform $[\mathbf{R}|\mathbf{t}]$ that combines the target camera’s extrinsic calibration with its object’s motion,

$$\mathbf{q} = \mathbf{K}\left(\mathbf{R}(\mathbf{o}) - \mathbf{t}(\mathbf{o})\,\mathbf{n}^\top\right)\mathbf{K}^{-1}\mathbf{p},$$

where $\mathbf{K}$ is the calibration matrix. The energy of a discrete-continuous conditional random field sums stereo, flow and cross data terms (Census matching costs plus sparse feature matches) and smoothness terms on depth, orientation and motion boundaries between adjacent superpixels. The scene flow inside a superpixel is fixed by four numbers (three for the plane, one object index) plus the parameters of a few objects. Inference uses max-product particle belief propagation with five objects, 30 shape and ten motion particles, and 50 iterations; the authors’ MATLAB and C++ implementation took about 50 minutes per scene, and a reduced setting about 120 seconds. Besides scene flow, OSF outputs a segmentation of the scene into independently moving objects.

RGB-D scene flow

With a depth camera, the depth at both times is measured rather than estimated, and methods concentrate on the motion. Menze and Geiger list several RGB-D methods that appeared after the Kinect. SphereFlow (Hornáček et al., 2014) matches patches defined not as square windows of pixels but as the 3D points inside spheres in world space, related by 6-degree-of-freedom rigid motions. This lets it avoid both brightness constancy and local planarity; it obtains an initial correspondence field with an adaptation of PatchMatch and then uses α-expansion for occlusion handling and regularization. Teed and Deng (2021) describe an alternating method by Jaimez et al. for RGB-D pairs that iterates between grouping pixels into rigidly moving clusters and fitting a motion to each cluster, and present RAFT-3D’s soft grouping as a differentiable version of the same idea.

Learned image-based methods

Mayer et al. (2016) built FlyingThings3D precisely to train networks for disparity, optical flow and scene flow. Their SceneFlowNet interleaves the weights of a pretrained FlowNet and two pretrained DispNets (one for each time step) into one network and fine-tunes it to output flow, disparity and disparity change. On the FlyingThings3D test set they reported endpoint errors of 10.99 pixels for flow, 2.21 for disparity and 0.80 for disparity change (Table 5; Table 4 of the same paper gives 0.79 for disparity change), and found that solving the joint task gave better results on each component than the separately trained networks.

Later stereo methods often combine learned components with explicit rigid-motion reasoning. RAFT-3D’s authors describe several that use object detection or instance segmentation networks to split the scene into potentially moving objects and then fit a rigid motion to each, and point out two limits: the segmentation networks need instance-level supervision, and they do not generalize to object types they were not trained on.

RAFT-3D (Teed and Deng, CVPR 2021) carries the design of RAFT over to scene flow. It takes two image–depth pairs; for stereo input, the depths come from a separate stereo network (GA-Net in their KITTI experiments). With an augmented projection $\pi$ that maps a 3D point to pixel coordinates and inverse depth, $(x, y, 1/Z)$, each pixel $\mathbf{x}_i$ is mapped by its own rigid transform $\mathbf{T}_i$:

$$\mathbf{x}'_i = \pi\!\left(\mathbf{T}_i\,\pi^{-1}(\mathbf{x}_i)\right).$$

The first two components of $\mathbf{x}'_i - \mathbf{x}_i$ are the optical flow and the third is the change in inverse depth. As in RAFT, features of the two images form an all-pairs correlation volume, and a convolutional GRU reads from it at the current correspondences. Instead of a flow update, the GRU predicts rigid-motion embeddings, per-pixel vectors whose distances define soft affinities between pixels, and revisions of the flow and inverse depth with confidences. A differentiable Dense-SE3 layer then applies one Gauss–Newton step that makes each pixel’s transform agree with the revised correspondences of the neighbours it is grouped with. The embeddings receive no direct supervision; the loss is on the optical flow and inverse-depth change induced by the estimated transforms, plus a smaller loss on the GRU’s revisions. The model has 45M parameters, 40M of them in a pretrained ResNet-50 context encoder.

Point-cloud methods

Lidar gives accurate 3D points without any stereo matching, but they are sparse and unordered, and the two sweeps sample different surface points. FlowNet3D (Liu, Qi and Guibas, CVPR 2019) learns scene flow directly from two point clouds. It builds on PointNet++-style set convolution layers for hierarchical point features, and adds a flow embedding layer that correlates the two clouds: for each point $\mathbf{x}_i$ of the first cloud with feature $\mathbf{f}_i$, it gathers the points $\mathbf{y}_j$ of the second cloud within a radius $r$, with features $\mathbf{g}_j$, and aggregates

$$\mathbf{e}_i = \max_{\{j \,:\, \|\mathbf{y}_j - \mathbf{x}_i\| \le r\}} h\!\left(\mathbf{f}_i, \mathbf{g}_j, \mathbf{y}_j - \mathbf{x}_i\right),$$

where $h$ is a learned function and the maximum is taken element-wise. Since exact correspondences rarely exist, the layer learns to weigh the candidate flow vectors $\mathbf{y}_j - \mathbf{x}_i$. Set upconv layers propagate the embeddings back to all points. Training uses a smooth $L_1$ loss on ground-truth flow with a cycle-consistency term that asks the flow predicted backwards from the moved points to cancel the forward flow.

PointPWC-Net (Wu et al., ECCV 2020) transfers the coarse-to-fine design of PWC-Net to point clouds: a four-level feature pyramid, upsampling of the coarse flow, warping of the first cloud by it, and a learnable point-based cost volume computed on the input points and processed by PointConv, which avoids a dense cost volume on a 3D grid. The authors’ ablation on FlyingThings3D shows the warping layer and their cost volume each reducing the 3D endpoint error substantially (from 0.0984 m to 0.0588 m with warping; 0.0741 m for FlowNet3D’s MLP-and-max-pooling embedding against 0.0588 m for theirs).

Self-supervised learning

Annotated real scene flow is scarce, so point-cloud methods can be trained without labels by asking the warped first cloud to match the second. PointPWC-Net’s self-supervised loss has three parts: a Chamfer distance between the warped first cloud $P_w = P + \mathbf{S}$ and the second cloud $Q$,

$$\ell_C = \sum_{\mathbf{p} \in P_w} \min_{\mathbf{q} \in Q} \|\mathbf{p} - \mathbf{q}\|_2^2 + \sum_{\mathbf{q} \in Q} \min_{\mathbf{p} \in P_w} \|\mathbf{p} - \mathbf{q}\|_2^2,$$

a smoothness term that asks neighbouring points to move alike, and a Laplacian regularization that asks the local shape of the warped cloud to match that of the second cloud. In their ablation, the Chamfer distance alone gave a 3D endpoint error of 0.2112 m on FlyingThings3D, adding smoothness 0.1304 m, and adding the Laplacian term 0.1213 m.

Mittal, Okorn and Held (CVPR 2020), described by PointPWC-Net’s authors as concurrent work, use two losses. A nearest-neighbour loss pulls each moved point toward its nearest neighbour in the second cloud, an approximation of the true correspondence borrowed from iterative closest point. A cycle-consistency loss predicts the reverse flow and asks the round trip to return to the start. They found the plain cycle loss unstable without labels and anchored the reverse pass at a point halfway between each moved point and its nearest neighbour ($\lambda = 0.5$), which also removes the degenerate zero-flow solution. With these losses they trained on unlabelled nuScenes lidar data before training on KITTI.

Datasets and Benchmarks

KITTI 2015 scene flow

Menze and Geiger (2015) created the first realistic scene flow benchmark with independently moving objects from the KITTI raw recordings: 400 dynamic scenes, 200 for training and 200 for testing, each with four colour images (two stereo pairs). Earlier evaluations had used synthetic spheres and simple street scenes, or the static scenes of the Middlebury and KITTI 2012 stereo and flow benchmarks. The ground truth is semi-dense:

• Static scene. Seven lidar scans were corrected for the rolling shutter, registered by combining GPS/IMU with ICP in a non-linear least-squares optimization, and accumulated; points on moving objects were removed using the KITTI 3D box annotations.
• Moving vehicles. Each moving vehicle was re-inserted by fitting one of 16 CAD models to the accumulated lidar points, to semi-global matching disparities as a weak prior, and to 5 to 10 manually annotated 2D–3D correspondences per object, estimating its pose, scale and rigid motion. The fitted models were projected into all four images.
• Excluded regions. Non-rigid objects such as pedestrians and cyclists, and erroneous laser regions, were masked by hand.

The authors found the ground truth accurate to at least 3 pixels for most parts, but less accurate for very large motions at the image boundaries. On 13 March 2017 the benchmark fixed small errors in the flow ground truth of the moving objects; the KITTI page notes that error numbers decreased slightly as a result and asks authors to report the corrected numbers.

The benchmark evaluates four maps at each reference pixel with ground truth: the disparity at $t$ (D1), the disparity at $t+1$ mapped to the reference pixel (D2), and the two flow components (Fl). A disparity or flow estimate counts as an outlier when its error exceeds both 3 pixels and 5% of the true value, a combination Menze and Geiger chose to stay faithful to the annotation errors. For flow, the error is the endpoint error:

$$\text{outlier}(\hat{q}, q) = \left[\,\|\hat{q} - q\| > 3\ \text{px}\ \wedge\ \|\hat{q} - q\| > 0.05\,\|q\|\,\right].$$

A pixel is a scene flow outlier (SF) when it is an outlier in any of D1, D2 and Fl. Each rate is reported over background pixels (bg), foreground pixels on moving vehicles (fg), and all pixels with ground truth (all), with variants over all or only non-occluded ground-truth pixels; the ranking uses the percentage of erroneous pixels, and sparser submissions are completed by background interpolation. (The legend of the KITTI page, read in November 2026, defines SF as outliers in “either D0, D1 or Fl”, labelling the two disparity maps differently from the D1 and D2 used everywhere else on the page.)

Results as reported on the 200 test scenes, all pixels with ground truth (SF-all):

Method	Source	D1-all	D2-all	Fl-all	SF-all
Huguet and Devernay (2007)	Menze and Geiger (2015), Table 2	26.38	57.08	49.64	67.08
SGM + LDOF	Menze and Geiger (2015), Table 2	6.83	28.56	40.15	44.12
PRSF	Menze and Geiger (2015), Table 2	6.24	12.69	14.39	16.85
OSF	Menze and Geiger (2015), Table 2	5.79	7.77	8.37	10.63
OSF	KITTI leaderboard (after the 2017 fix), as listed by Teed and Deng (2021)	5.79	7.77	7.83	10.23
DRISF	Teed and Deng (2021), Table 3	2.55	4.04	4.73	6.31
RAFT-3D	Teed and Deng (2021), Table 3	1.81	3.67	4.29	5.77

The two OSF rows differ only in the flow and scene flow columns (Fl-fg 22.18 in the paper, 18.92 on the leaderboard), consistent with the 2017 correction of the foreground flow ground truth; the leaderboard read in November 2026 lists the same values as Teed and Deng’s table. The baselines show what joint, piecewise rigid estimation buys: combining good semi-global matching disparities with a separately computed optical flow left 44% of pixels wrong as scene flow, mostly through the flow and the second disparity. RAFT-3D was fine-tuned on KITTI for 50k iterations from a FlyingThings3D model, and its authors stress that, unlike DRISF, it used no instance segmentation supervision. For current standings, see the official leaderboard.

KITTI also serves as a lidar benchmark, but under protocols the benchmark does not define. FlowNet3D evaluated on the 150 of the 200 training scenes for which raw point clouds could be traced, with ground points removed, and with ground points kept fine-tuned on 100 of them and tested on the other 50; PointPWC-Net evaluated on 142 training scenes with ground points below 0.3 m removed, following HPLFlowNet; Mittal et al. trained on 100 of the 150 scenes and tested on the other 50, with ground points removed. Numbers from these protocols are not comparable with each other or with the image-based leaderboard.

FlyingThings3D

FlyingThings3D is the largest of three synthetic datasets that Mayer et al. (2016) rendered with Blender, together with Monkaa and Driving, to train networks for disparity, optical flow and scene flow. Its scenes have a textured ground plane, 200 randomly shaped static background objects, and between 5 and 20 ShapeNet objects with random textures; the camera and the objects move along randomized linear 3D trajectories. Training and test objects come from disjoint ShapeNet categories (32,872 and 3,055 models). The paper’s text speaks of about 25,000 stereo frames, and its Table 1 lists 21,818 training and 4,248 test frames at $960 \times 540$, rendered in a clean pass and a final pass with blur and other effects. Because the renderer knows every point, the ground truth is complete, including disparity change and occluded regions, in forward and backward directions.

The dataset is used for scene flow in two ways that give incomparable numbers. Image-based methods such as RAFT-3D use the full images (RAFT-3D excludes pixels at infinity and flows above 250 pixels). Point-cloud methods convert disparity and flow into 3D points and 3D flow, and two preprocessings are in use: FlowNet3D’s (20,000 training and 2,000 test examples, randomly sub-sampled from the rendered stereo frames) and the one PointPWC-Net uses following HPLFlowNet (19,640 training and 3,824 evaluation pairs, with 8,192 points per cloud in PointPWC-Net). The same network can score very differently under the two: FlowNet3D reported a 3D endpoint error of 0.1694 m on its own split, while PointPWC-Net’s table lists FlowNet3D at 0.1136 m on HPLFlowNet’s data.

Evaluation Metrics

Image-space measures follow KITTI: the D1, D2, Fl and SF outlier rates above, with endpoint error for the flow component. They inherit the strengths and weaknesses of pixel thresholds. With $fB = 350$ pixel-metres, as in the example below, a 3-pixel disparity error moves a point at 5 m by about 0.2 m in depth but a point at 40 m to anywhere between about 30 and 61 m, so the SF rate weighs near and far errors very differently in metric terms (our arithmetic from $Z = fB/d$).

3D measures compare estimated and true 3D vectors directly:

• 3D endpoint error (EPE3D): $\|\hat{\mathbf{s}} - \mathbf{s}\|_2$ averaged over points, in metres.
• Accuracy and outlier rates. PointPWC-Net, following HPLFlowNet, reports Acc3DS (EPE3D below 0.05 m or relative error below 5%), Acc3DR (below 0.1 m or 10%) and Outliers3D (above 0.3 m or relative error above 10%); FlowNet3D’s KITTI table uses an outlier ratio of “0.3 m or 5%”. RAFT-3D reports threshold accuracies $\delta_{3D} < 0.05$ and $\delta_{3D} < 0.10$ next to the 2D and 3D endpoint errors.
• 2D measures for 3D methods. Point-cloud papers also project the flow into the image and report a 2D endpoint error and a KITTI-style 2D accuracy.

Because the thresholds and protocols differ between papers, a result is only meaningful together with its dataset, preprocessing, point count and measure definition.

Example: Scene Flow from Stereo and Optical Flow

The example builds scene flow the classical way, from a disparity map at each time and an optical flow, on a rendered stereo sequence whose 3D motion is known exactly. Four textured planes at 8, 15, 25 and 40 m, textured with crops of images bundled with scikit-image, all move by the same translation $(0.1, 0.05, -0.4)$ m in front of a static background at 100 m; the rig has $f = 700$ pixels and $B = 0.5$ m. Disparity comes from sum-of-absolute-differences block matching with parabolic sub-pixel refinement in each stereo pair, and optical flow from scikit-image’s TV-L1 solver on the left images. The second disparity is mapped to the reference pixel through the estimated flow, as Menze and Geiger describe for methods that output it in the second frame. The script then reports, for each plane, KITTI’s outlier rates and PointPWC-Net’s 3D measures on the pixels visible in all four images.

import numpy as np
from scipy.ndimage import map_coordinates, uniform_filter
from skimage import data
from skimage.color import rgb2gray
from skimage.registration import optical_flow_tvl1

# Rectified stereo rig (static cameras): focal length f (px), baseline B (m)
f, B = 700.0, 0.5
H, W = 240, 720
cx, cy = W / 2, H / 2
t_motion = np.array([0.1, 0.05, -0.4])          # 3D translation of every object (m)

def gray(img):
    g = rgb2gray(img) if img.ndim == 3 else img / 255.0
    return g.astype(np.float64)

textures = [gray(data.camera()), gray(data.coffee()), gray(data.astronaut()),
            gray(data.chelsea()), gray(data.rocket())]

# Four textured fronto-parallel planes (depth at time t, image centre in px)
# and a static textured background at 100 m. Each plane is about 130 x 110 px.
planes = [dict(Z=25.0, xc=100), dict(Z=8.0, xc=270),
          dict(Z=15.0, xc=450), dict(Z=40.0, xc=620)]
for k, p in enumerate(planes):
    s = p["Z"] / f                               # texel size (m): ~1 px at time t
    p.update(X0=(p["xc"] - 65 - cx) * s, Y0=(cy - 55 - cy) * s,
             Wm=130 * s, Hm=110 * s, s=s, tex=textures[k])
Z_bg = 100.0

def render(cam_x, time):
    """Ray-cast one view: cam_x = camera centre (0 left, B right), time 0 or 1."""
    ys, xs = np.mgrid[0:H, 0:W].astype(np.float64)
    depth = np.full((H, W), Z_bg)
    label = np.full((H, W), -1)
    sb = Z_bg / f
    Xb = cam_x + (xs - cx) * Z_bg / f
    Yb = (ys - cy) * Z_bg / f
    img = map_coordinates(textures[4], [Yb / sb + 200, Xb / sb + 300], order=1, mode="wrap")
    for k, p in enumerate(planes):
        off = t_motion * time
        Z = p["Z"] + off[2]
        X = cam_x + (xs - cx) * Z / f
        Y = (ys - cy) * Z / f
        u = (X - p["X0"] - off[0]) / p["s"]
        v = (Y - p["Y0"] - off[1]) / p["s"]
        hit = (u >= 0) & (u <= 130) & (v >= 0) & (v <= 110) & (Z < depth)
        val = map_coordinates(p["tex"], [v + 100, u + 100], order=1, mode="reflect")
        img[hit], depth[hit], label[hit] = val[hit], Z, k
    return img, depth, label

rng = np.random.default_rng(0)
views = {}
for name, cam_x, time in [("L0", 0, 0), ("R0", B, 0), ("L1", 0, 1), ("R1", B, 1)]:
    img, depth, label = render(cam_x, time)
    views[name] = (np.clip(img + rng.normal(0, 1 / 255, img.shape), 0, 1), depth, label)

def block_matching(left, right, dmax=64, win=7):
    cost = np.full((dmax + 1, H, W), np.inf)
    for d in range(dmax + 1):
        diff = np.full((H, W), 1.0)
        diff[:, d:] = np.abs(left[:, d:] - right[:, :W - d])
        cost[d] = uniform_filter(diff, win)
    d0 = np.argmin(cost, axis=0)
    dc = np.clip(d0, 1, dmax - 1)
    c_m, c_0, c_p = (np.take_along_axis(cost, (dc + o)[None], 0)[0] for o in (-1, 0, 1))
    denom = c_m - 2 * c_0 + c_p
    sub = np.where(denom > 0, 0.5 * (c_m - c_p) / np.where(denom > 0, denom, 1), 0.0)
    return dc + np.clip(sub, -0.5, 0.5)

L0, depth0, lab0 = views["L0"]
L1, depth1, lab1 = views["L1"]
disp0 = block_matching(L0, views["R0"][0])
disp1 = block_matching(L1, views["R1"][0])
flow = optical_flow_tvl1(L0.astype(np.float32), L1.astype(np.float32))   # (row, col)
v_est, u_est = flow[0].astype(np.float64), flow[1].astype(np.float64)

ys, xs = np.mgrid[0:H, 0:W].astype(np.float64)
# Second-frame disparity mapped to the reference pixel through the estimated flow
d2_est = map_coordinates(disp1, [ys + v_est, xs + u_est], order=1, mode="nearest")

def backproject(x, y, d):
    Z = f * B / np.maximum(d, 0.5)
    return np.stack([(x - cx) * Z / f, (y - cy) * Z / f, Z])

def project(P, cam_x=0.0):
    return f * (P[0] - cam_x) / P[2] + cx, f * P[1] / P[2] + cy

# Ground truth from the known geometry and motion
P1 = backproject(xs, ys, f * B / depth0)
P2 = P1 + t_motion[:, None, None] * (lab0 >= 0)
x2, y2 = project(P2)
u_gt, v_gt = x2 - xs, y2 - ys
d1_gt, d2_gt = f * B / P1[2], f * B / P2[2]
sf_gt = P2 - P1
# Sanity check: scene flow composed from ground-truth disparities and flow
sf_check = backproject(xs + u_gt, ys + v_gt, d2_gt) - backproject(xs, ys, d1_gt)
print("max |GT composition - known motion| (m): %.1e"
      % np.abs(sf_check - sf_gt)[:, lab0 >= 0].max())

# Pixels visible in all four images at the same point
def visible(lbl, x, y, k):
    xi, yi = np.round(x).astype(int), np.round(y).astype(int)
    ok = (xi >= 0) & (xi < W) & (yi >= 0) & (yi < H)
    out = np.zeros_like(ok)
    out[ok] = lbl[yi[ok], xi[ok]] == k
    return out

# Estimated scene flow from (d1, flow, d2)
sf_est = backproject(xs + u_est, ys + v_est, d2_est) - backproject(xs, ys, disp0)

print("\nplane  Z(m)  d1(px)  d2-d1(px)  |flow|(px)  D1%   D2%   Fl%   SF%   "
      "EPE3D(m)  RMS dZ  pred dZ  Acc3DS%  Out3D%  medEPE  med|dZ|  pred med")
for k, p in enumerate(planes):
    m = lab0 == k
    xr0, yr0 = project(P1, B)
    xr1, yr1 = project(P2, B)
    m &= visible(views["R0"][2], xr0, yr0, k) & visible(lab1, x2, y2, k) \
        & visible(views["R1"][2], xr1, yr1, k)
    e_d1 = np.abs(disp0 - d1_gt)[m]
    e_d2 = np.abs(d2_est - d2_gt)[m]
    e_fl = np.hypot(u_est - u_gt, v_est - v_gt)[m]
    mag_fl = np.hypot(u_gt, v_gt)[m]
    D1 = (e_d1 > 3) & (e_d1 > 0.05 * d1_gt[m])
    D2 = (e_d2 > 3) & (e_d2 > 0.05 * d2_gt[m])
    Fl = (e_fl > 3) & (e_fl > 0.05 * mag_fl)
    SF = D1 | D2 | Fl
    err = sf_est[:, m] - sf_gt[:, m]
    epe = np.linalg.norm(err, axis=0)
    rel = epe / np.linalg.norm(sf_gt[:, m], axis=0)
    acc3ds = (epe < 0.05) | (rel < 0.05)
    out3d = (epe > 0.3) | (rel > 0.1)
    # First-order prediction of the depth-direction error from the measured disparity errors
    Z1, Z2 = p["Z"], p["Z"] + t_motion[2]
    s1, s2 = np.std(disp0[m] - d1_gt[m]), np.std(d2_est[m] - d2_gt[m])
    pred = np.hypot(Z1**2 * s1 / (f * B), Z2**2 * s2 / (f * B))
    # Same prediction from a robust scale (1.4826 x MAD); median |N(0, s)| = 0.6745 s
    mad = lambda e: 1.4826 * np.median(np.abs(e - np.median(e)))
    r1, r2 = mad(disp0[m] - d1_gt[m]), mad(d2_est[m] - d2_gt[m])
    pred_med = 0.6745 * np.hypot(Z1**2 * r1 / (f * B), Z2**2 * r2 / (f * B))
    print(f"{k:5d} {Z1:5.0f} {d1_gt[m].mean():7.2f} {(d2_gt - d1_gt)[m].mean():10.3f} "
          f"{mag_fl.mean():11.2f} {100*D1.mean():5.1f} {100*D2.mean():5.1f} "
          f"{100*Fl.mean():5.1f} {100*SF.mean():5.1f} {epe.mean():9.3f} "
          f"{np.sqrt(np.mean(err[2]**2)):7.3f} {pred:8.3f} {100*acc3ds.mean():8.1f} "
          f"{100*out3d.mean():7.1f} {np.median(epe):7.3f} {np.median(np.abs(err[2])):8.3f} "
          f"{pred_med:9.3f}")
print("true scene flow magnitude: %.3f m" % np.linalg.norm(t_motion))

Output (NumPy 2.3.5, SciPy 1.16.3, scikit-image 0.25.2; about 2 seconds):

max |GT composition - known motion| (m): 7.1e-15

plane  Z(m)  d1(px)  d2-d1(px)  |flow|(px)  D1%   D2%   Fl%   SF%   EPE3D(m)  RMS dZ  pred dZ  Acc3DS%  Out3D%  medEPE  med|dZ|  pred med
    0    25   14.00      0.228        2.06   5.0   6.1   0.0   7.6     4.391  29.599    6.912      9.1    92.7   0.187    0.176     0.145
    1     8   43.75      2.303        6.68   0.4   0.9   7.4   8.4     0.090   1.327    0.253     78.8    30.7   0.032    0.028     0.017
    2    15   23.33      0.639        7.69   1.3   1.9   0.0   2.3     0.546   9.097    1.296     15.9    87.9   0.097    0.095     0.061
    3    40    8.75      0.088        4.49   0.1   0.1   0.0   0.1     0.734  10.077    1.467      7.3    94.0   0.238    0.224     0.206
true scene flow magnitude: 0.415 m

The first line checks the bookkeeping: scene flow composed from the true disparities and true flow reproduces the known motion to rounding error, so every error below comes from the estimates.

The two families of measures rank the planes in opposite orders. By KITTI’s image-space criterion the farthest plane is the easiest: only 0.1% of its pixels are scene flow outliers, because its true disparity change of under a tenth of a pixel and its flow of a few pixels are well inside the 3-pixel tolerance. By the 3D criteria it is the hardest: its median 3D error is more than half of the 0.415 m motion, 94% of its pixels are 3D outliers, and fewer than one in ten meets the 5 cm or 5% accuracy test. The nearest plane is the reverse: it has the most image-space outliers, mostly flow errors (the script does not break them down further), yet the smallest 3D errors. The median 3D error rises with distance, from about 3 cm at 8 m to about 24 cm at 40 m, and nearly all of it is in the depth direction, as the comparison of the median 3D error with the median depth-direction error shows.

The means tell a different story: they do not rise steadily with distance. The plane at 25 m has a mean 3D error of 4.4 m, larger than that of the plane at 40 m, and an RMS depth-direction error of about 30 m, far above its median of 0.19 m. A few percent of its pixels get grossly wrong disparities (5% D1 and 6% D2 outliers). The script does not locate them; weakly textured parts of its camera crop and the plane’s borders, where the matching window also covers the background, are possible causes (our conjecture). Since depth is inversely proportional to disparity, a disparity that is too small by several pixels produces a depth error of tens of metres. The first-order prediction from the standard deviation of the disparity errors is inflated by the same outliers and does not match the RMS error, because a linearization cannot follow a reciprocal through gross errors. The prediction from a robust scale of the disparity errors gets the growth with distance right but falls short of the measured medians on every plane, by 8% to 39%, which suggests that the disparity errors at the two times and the flow errors are not the small, independent Gaussian errors the formula assumes. Heavy-tailed errors of this kind are one reason to read outlier rates and medians next to mean errors. The example composes independent estimates and tests no rigid-object prior; the KITTI baselines above show how much such composition loses against joint, piecewise rigid estimation.

After the first run, which printed only the columns up to Out3D% and showed means dominated by the gross errors on the 25 m plane, three print-only columns were added: the median 3D error, the median absolute depth-direction error, and the prediction from a robust scale (1.4826 times the median absolute deviation of the disparity errors). The scene, the estimators and the other columns are unchanged, and their values are identical to the first run’s. Each run of the script prints the same output.

Applications

• Autonomous driving. Scene flow separates moving obstacles from the static world and gives their 3D velocity, which Menze and Geiger name as input to scene understanding, obstacle avoidance and path planning; their KITTI benchmark was built for this setting. See Autonomous Driving Perception.
• Robotics. RAFT-3D’s authors list path planning, collision avoidance, virtual reality and motion modelling as uses of detailed 3D motion.
• Lidar processing. FlowNet3D’s authors demonstrate scan registration and motion segmentation from point-cloud scene flow.
• Dynamic multi-view capture. Vedula et al. proposed scene flow as a predictor for stereo in the next frame and for dynamic rendering, such as slow-motion replays, and for modelling human actions, in a multi-camera dome.

Open Problems

• Motion in depth at range. Image-only systems measure the depth motion of distant objects poorly, as the example shows, while lidar measures depth directly.
• Non-rigid and unseen objects. Rigid priors work well for vehicles, but pedestrians, cyclists and deformable objects violate them, and KITTI 2015 has no ground truth for them. Methods built on instance segmentation are limited to the object classes they were trained on, as Teed and Deng point out.
• Real ground truth. KITTI 2015, the real benchmark used by the methods above, has 200 test scenes with semi-dense, CAD-based labels for vehicles, so the learned methods above were trained mainly on synthetic data such as FlyingThings3D and fine-tuned, if at all, on a small real set. Self-supervised losses reduce the need for labels but depend on assumptions such as nearest-neighbour correspondence and local smoothness.
• Comparable evaluation for point clouds. Lidar results still lack a common KITTI protocol, as described above.
• Occlusions and multiple frames. The methods described here use two time steps, and the motion of points hidden in some of the four images must come from a prior. Longer sequences offer more observations of each point; the KITTI leaderboard marks the methods that use them.