PWC-Net

PWC-Net is a convolutional network for optical flow estimation that builds three classical principles into its architecture: a pyramid, warping and a cost volume, which give it its name. Deqing Sun, Xiaodong Yang, Ming-Yu Liu and Jan Kautz of NVIDIA presented it at CVPR 2018. Instead of an image pyramid it learns a feature pyramid; at each level it warps the second image’s features towards the first with the flow estimated at the coarser level, compares the two feature maps in a small cost volume, and lets a convolutional estimator turn the cost volume into flow. A context network of dilated convolutions refines the final estimate. The network has 8.75 million parameters against 162.49 million for FlowNet2, and at publication, by its authors’ account, it had the lowest error of published methods on the MPI Sintel final pass and the KITTI 2015 flow benchmark. A journal extension in IEEE TPAMI (“Models Matter, So Does Training”, 2020) kept the architecture and showed how much of the gain came from the training procedure.

Problem Addressed

By 2017, learned flow had two reference points. FlowNet showed that a network could predict flow end to end, and FlowNet2 stacked several FlowNets with image warping between them to reach the accuracy of the best classical methods. Sun et al. pointed to the cost of that design: FlowNet2’s 162.49 million parameters need 638.5 MB of memory, its sub-networks have to be trained one after another because large models overfit, and the whole training takes at least 14 days by their estimate. At the other end, SpyNet (Ranjan and Black, 2017) used a spatial pyramid and warped the second image towards the first, so that a small network only had to estimate the remaining motion at each level; it had 1.2 million parameters but, in PWC-Net’s comparison, performed on par with FlowNetC and below FlowNetS and FlowNet2.

The authors asked whether a model could be both smaller and more accurate, and argued that SpyNet’s gap came from using classical principles only partly. Traditional methods extract features that are invariant to shadows and lighting instead of comparing raw intensities, and stereo matching had shown that a cost volume is a more discriminative representation than raw images or features. A full cost volume, comparing every pixel with every other, is too expensive for real-time flow; a pyramid avoids it, because a search range of a few pixels at each level adds up to a large displacement at full resolution once the levels are linked by warping. This is the classical coarse-to-fine scheme, which the authors describe as the most popular framework for optical flow, rebuilt with learned components.

Architecture

Figure 3 of the paper sets PWC-Net beside the traditional energy-minimization approach. Both work from the coarsest pyramid level to the finest; at each level, the traditional method warps the second image with the upsampled flow and minimizes an energy, while PWC-Net warps features, computes a cost volume and processes it with a network. The warping and cost volume layers have no learnable parameters, which keeps the model small. The full network uses a 7-level pyramid and stops at level 2, a quarter of the input resolution, whose flow is upsampled bilinearly to full size. The supplement gives the layer-by-layer design (its Figures 4 to 6).

Feature pyramid extractor

The two input images $I_1$ and $I_2$ pass through the same network, a Siamese structure in the supplement’s words, which builds an $L$ -level pyramid of features $\mathbf{c}_t^l$ with level 0 the image itself, $\mathbf{c}_t^0 = I_t$ . Each level halves the resolution of the one below with convolutions: in the supplement’s Figure 4, two $3 \times 3$ convolutions per level, the first with stride 2, each followed by a leaky ReLU. From level 1 to level 6 the numbers of channels are 16, 32, 64, 96, 128 and 196 in the CVPR text; the supplement’s figure and the TPAMI manuscript give 192 for level 6, while the official PyTorch code uses 196.

Learning the features instead of using an image pyramid is the first departure from classical practice, motivated by raw images varying with shadows and lighting. The TPAMI manuscript tests it directly, with models trained on Flying Chairs: replacing the feature pyramid with an image pyramid raised the error on every set (Chairs test 2.95 against 2.00; Sintel training clean 4.42 against 3.33, final 5.58 against 4.59), which its text sums up as about a 40% loss in accuracy.

Warping layer

At level $l$ , the features of the second image are warped towards the first with the flow from level $l + 1$ , upsampled by a factor of 2:

\mathbf{c}_w^l(\mathbf{x}) = \mathbf{c}_2^l\big(\mathbf{x} + \mathrm{up}_2(\mathbf{w}^{l+1})(\mathbf{x})\big),

where $\mathbf{x}$ is the pixel index (Eq. 1 of the paper). At the top level there is no coarser flow, so the upsampled flow is zero and the warped features are the second image’s features. The warp uses bilinear interpolation, with gradients to both the features and the flow, as in FlowNet2’s warping layer, so the whole network can be trained end to end; the general operation is described under image warping. After warping, the two feature maps differ only by the motion that the coarser levels missed, so the next comparison can be local. The authors add that for non-translational motion warping compensates for some geometric distortion and puts image patches at the right scale.

Two details are left to the implementation. The TPAMI manuscript writes the warp with an explicit factor, $\mathbf{x} + 2 \times \mathrm{up}_2(\mathbf{w}^{l+1})(\mathbf{x})$ , because flow measured in coarser-level pixels must be doubled at the finer level. The CVPR paper instead scales the flow by constants (see Training), giving the factor only for level 2, 5 (= 20/4), and the official PyTorch code multiplies the upsampled flow by 0.625, 1.25, 2.5 and 5.0 before warping at levels 5 to 2. The code also upsamples the flow and the estimator’s features with learned transposed convolutions, and sets warped values to zero where the sample falls outside the image.

Cost volume layer

The matching cost between a pixel $\mathbf{x}_1$ of the first image’s features and a pixel $\mathbf{x}_2$ of the warped second image’s features is their correlation, the normalized inner product of Eq. 2:

\mathbf{cv}^l(\mathbf{x}_1, \mathbf{x}_2) = \frac{1}{N} \big(\mathbf{c}_1^l(\mathbf{x}_1)\big)^{\top} \mathbf{c}_w^l(\mathbf{x}_2),

where $N$ is the length of the feature vector. This is the correlation of FlowNetC with single-pixel patches. What differs is where it is computed: only within a range of $d$ pixels, $|\mathbf{x}_1 - \mathbf{x}_2|_\infty \le d$ , at every level, a “partial” cost volume. A one-pixel motion at the top level corresponds to $2^{L-1}$ pixels at full resolution, so $d$ can be small; the experiments use $d = 4$ at every level. The paper writes the volume’s size as $d^2 \times H^l \times W^l$ , but a range of $\pm d$ in each direction gives $(2d + 1)^2$ displacements: the supplement’s Figure 5 shows 81 cost channels for $d = 4$ , and the official code computes $(2 \cdot 4 + 1)^2$ . The code also applies a leaky ReLU to the cost volume.

The cost volume’s range is a property of each level, so a modest search covers large motion. By the authors’ account, a range of 2 at every level already handles motion of up to 200 pixels at input resolution. FlowNetC, by contrast, builds one volume at a single resolution with a large range (displacement up to 20 at one-eighth resolution, 441 channels); the cost volume article places both among other matching volumes.

Optical flow estimator

At each level, a five-layer convolutional network receives the cost volume, the first image’s features and the upsampled flow, and outputs the flow $\mathbf{w}^l$ at that level. Its layers have 128, 128, 96, 64 and 32 channels at every level, and each level has its own weights. At the top level, the estimator receives only the cost volume, computed directly between the two images’ features. At level 2 in the supplement’s Figure 5, the input to the first convolution has 115 channels: 81 from the cost volume, 32 from the features and 2 from the upsampled flow.

The estimator can be enhanced with DenseNet connections, in which every layer receives both the output of its predecessor and that predecessor’s own input, so that each layer sees all earlier outputs. The full PWC-Net uses them; PWC-Net-small drops them. In the official code, each layer’s output is concatenated with its input before the next layer.

Context network

Classical flow methods often post-process the flow with contextual information, such as median or bilateral filtering. PWC-Net’s counterpart is a context network that enlarges the receptive field of each output at the final level. It takes the estimated flow and the features of the estimator’s second-last layer, and is a feed-forward stack of seven $3 \times 3$ convolutions with dilation constants 1, 2, 4, 8, 16, 1 and 1, from bottom to top. Dilated convolutions widen the receptive field without a large computational cost. In the supplement’s Figure 6, the context network is used only at level 2; its last layer outputs a flow correction that is added to the estimator’s flow. The supplement also compared plain convolutions with the dilated ones: the plain version was worse on Flying Chairs and Sintel and slightly better on KITTI.

Training

Loss

The network is trained with the multi-scale loss of FlowNet, summed over the levels from the output level $l_0$ to the top level $L$ :

\mathcal{L}(\Theta) = \sum_{l = l_0}^{L} \alpha_l \sum_{\mathbf{x}} \big| \mathbf{w}_\Theta^l(\mathbf{x}) - \mathbf{w}_\mathrm{GT}^l(\mathbf{x}) \big|_2 + \gamma |\Theta|_2 ,

where $\mathbf{w}_\Theta^l$ is the predicted flow at level $l$ , $\mathbf{w}_\mathrm{GT}^l$ its supervision signal, $|\cdot|_2$ the L2 norm, and the last term regularizes the parameters (Eq. 3). The weights are $\alpha_6 = 0.32$ , $\alpha_5 = 0.08$ , $\alpha_4 = 0.02$ , $\alpha_3 = 0.01$ and $\alpha_2 = 0.005$ , and $\gamma = 0.0004$ ; with $l_0 = 2$ the coarsest levels carry the largest weights. The ground truth is “scaled by 20” and downsampled to each level, without rescaling the vectors per level, so the upsampled flow has to be rescaled before warping (by 5 = 20/4 at level 2, in the paper’s example). The official PyTorch README makes the direction explicit: the ground truth was divided by 20 in training, and the estimated flow must be multiplied by 20. The network therefore predicts the flow divided by 20, in full-resolution pixels at every level, which the code’s warping factors $20/2^l$ convert to pixels of level $l$ . In the official Caffe training file, the per-level losses are L1Loss layers with an L2 norm per location and the same weights, and the solver is Adam with weight decay 0.0004.

For fine-tuning, the loss becomes robust:

\mathcal{L}(\Theta) = \sum_{l = l_0}^{L} \alpha_l \sum_{\mathbf{x}} \Big( \big| \mathbf{w}_\Theta^l(\mathbf{x}) - \mathbf{w}_\mathrm{GT}^l(\mathbf{x}) \big| + \epsilon \Big)^{q} + \gamma |\Theta|_2 ,

with the L1 norm, $q < 1$ to penalize outliers less, and a small constant $\epsilon$ (Eq. 4); the Sintel fine-tuning uses $\epsilon = 0.01$ and $q = 0.4$ .

Schedule

Training follows the dataset order that FlowNet 2.0 introduced. The models are first trained on Flying Chairs in Caffe with the $S_\text{long}$ schedule, starting at a learning rate of $10^{-4}$ and halving it at 0.4M, 0.6M, 0.8M and 1M iterations, with FlowNet2’s data augmentation, $448 \times 384$ crops and batches of 8. They are then fine-tuned on FlyingThings3D with the $S_\text{fine}$ schedule, $768 \times 384$ crops and batches of 4, leaving out image pairs whose motion exceeds 1,000 pixels. The released solver lists one more halving at 1.2M iterations, which is also its last iteration. Unlike FlowNet2, the whole network is trained end to end from scratch; Table 7 of the paper puts its training at 4.8 days, against 4 for FlowNetS, 6 for FlowNetC and more than 14 for FlowNet2.

For the benchmarks the models were fine-tuned again. On MPI Sintel, training used $768 \times 384$ crops with horizontal flips, no additive noise and batches of 4, in two schemes: PWC-Net-ft used the clean and final passes throughout, and PWC-Net-ft-final used only the final pass for the second half, because DCFlow, the best published method then, had learned its features from the final pass alone. On KITTI, training used $896 \times 320$ crops with less rotation, zoom and squeeze; because the ground truth is semi-dense, the quarter-resolution prediction was upsampled and compared with the full-resolution ground truth, excluding invalid pixels.

The order of datasets mattered here as it had for FlowNet2 (Table 6 of the paper). After Flying Chairs alone, PWC-Net’s average endpoint error on the Sintel training set was 3.33 (clean) and 4.59 (final); after Chairs and Things, 2.55 and 3.93; fine-tuning on Sintel brought its own training error to 1.70 and 2.21 and also improved KITTI 2015 (Fl-all from 33.67% to 24.35%). Training on Sintel alone fitted Sintel nearly as well (1.86 and 2.31) but transferred worse to Chairs and KITTI.

Inference

PWC-Net takes two RGB images and returns a dense flow field in one forward pass, at a quarter of the input resolution before bilinear upsampling. In the official PyTorch release, images are fed in BGR order and divided by 255 without further normalization, because the Caffe model was trained that way.

Size and speed are the paper’s main argument. Its Table 7, measured on an NVIDIA Pascal Titan X:

Model	Parameters (M)	Memory (MB)	Training (days)	Forward (ms)	Backward (ms)
FlowNetS	38.67	154.5	4	11.40	16.71
FlowNetC	39.17	156.4	6	21.69	48.67
FlowNet2	162.49	638.5	>14	84.80	78.96
SpyNet	1.2	9.7	–	–	–
PWC-Net	8.75	41.1	4.8	28.56	44.37
PWC-Net-small	4.08	22.9	4.1	20.76	28.44

The TPAMI manuscript adds that the timings are for $1024 \times 448$ images. The text describes PWC-Net as about 17 times smaller and about 2 times faster in forward inference than FlowNet2, and at least 3 times faster to train. The table’s own numbers give somewhat larger ratios: 162.49 against 8.75 million parameters is about 18.6 times, and 84.80 against 28.56 ms of forward time about 3 times, while the training ratio, more than 14 against 4.8 days, is about 2.9 for the 14-day lower bound. The 28.56 ms match the “about 35 frames per second” at Sintel resolution ( $1024 \times 436$ ) in the abstract, and the Sintel table lists 0.03 s per frame for PWC-Net against 0.12 s for FlowNet2. PWC-Net-small, without DenseNet connections, is about 5% less accurate but 40% faster, by the paper’s account.

Performance

MPI Sintel

Average endpoint error (EPE) from Table 1 of the CVPR paper; values in parentheses are on data the model was fine-tuned on:

Method	Train clean	Train final	Test clean	Test final	Time (s)
MRFlow	1.83	3.59	2.53	5.38	480
FlowFieldsCNN	–	–	3.78	5.36	23.0
DCFlow	–	–	3.54	5.12	8.60
SpyNet-ft	(3.17)	(4.32)	6.64	8.36	0.16
FlowNet2	2.02	3.14	3.96	6.02	0.12
FlowNet2-ft	(1.45)	(2.01)	4.16	5.74	0.12
PWC-Net-small-ft	(2.27)	(2.45)	5.05	5.32	0.02
PWC-Net	2.55	3.93	–	–	0.03
PWC-Net-ft	(1.70)	(2.21)	3.86	5.13	0.03
PWC-Net-ft-final	(2.02)	(2.08)	4.39	5.04	0.03

On the final pass, PWC-Net-ft-final had the lowest error of the methods published at the time, the first time, the authors note, that an end-to-end method beat well-engineered classical methods there, and it was the fastest of the top methods. On the clean pass it was less accurate than classical approaches such as MRFlow; the authors attribute this to classical methods using image edges to refine motion boundaries, which align perfectly with motion boundaries in the clean pass but are blurred and noisy in the final pass, and suggest PWC-Net may suit real images better. PWC-Net had higher training error but lower test error than FlowNet2, which they read as a more appropriate capacity for the task.

Their Table 2 breaks the test error down by region. PWC-Net did relatively better than FlowNet2 in regions of large motion and away from motion boundaries (final pass: 31.28 against 35.54 for speeds above 40 pixels per frame), which the authors link to training only on data with large motion; on the clean pass FlowNet2 was better for slow pixels (0.60 against 0.70 below 10 pixels) and near boundaries. The table does not say which fine-tuned variant it shows. Failures include small, fast objects, such as a character’s arm in the “Market 5” sequence, the classical weakness of coarse-to-fine schemes, and thin structures rarely seen in training, such as bicycle wheels in the supplement’s KITTI results.

KITTI

Selected entries of Table 3 (test sets; Fl-Noc and Fl-all are percentages of outlier pixels in non-occluded and all regions):

Method	KITTI 2012 AEPE	KITTI 2012 Fl-Noc	KITTI 2015 Fl-all
SDF	2.3	3.80%	11.01%
MirrorFlow	2.6	4.38%	10.29%
SpyNet-ft	4.7	12.31%	35.07%
FlowNet2-ft	1.8	4.82%	10.41%
PWC-Net-ft	1.7	4.22%	9.60%

PWC-Net-ft had the lowest Fl-all of published two-frame flow methods on KITTI 2015, and the second-lowest Fl-Noc on KITTI 2012 after SDF, which assumes a rigid background that suits the static scenes of the 2012 set. On KITTI 2015’s foreground pixels (all-pixel evaluation), FlowNet2 (8.75%) and PWC-Net (9.31%) both beat the best published scene flow method, ISF (10.29%), while scene flow methods, which use stereo, were far better on the static background (Table 4). The supplement’s screenshots, taken on 15 November 2017, show PWC-Net first among optical flow methods on Sintel final and KITTI 2015 and second on KITTI 2012. The FlowNet2 entry of 10.41% matches the KITTI leaderboard screenshot but differs from the 11.48% in FlowNet 2.0’s own paper. For current standings, see the official Sintel and KITTI leaderboards.

Ablations

Table 5 of the CVPR paper removes or resizes components, with models trained on Flying Chairs and evaluated by EPE on the Chairs test set and the Sintel training set:

Warping. Without warping layers, Sintel final error rose from 4.59 to 5.30 and KITTI 2015 Fl-all from 41.79% to 44.87%, “a significant loss of accuracy” in the authors’ words. The network still produced reasonable results, because a range of 4 at the coarse levels already captures most motions.
Search range. Ranges of 2, 4 and 6 gave similar Sintel errors (final pass 4.50, 4.59 and 4.60). A larger range lowered the training error and the KITTI endpoint error, likely because KITTI’s displacements are larger, but the range of 2 gave the lowest Fl-all on both KITTI sets, which the authors read as the network ignoring pixels with extreme motion and concentrating on small motions. The TPAMI manuscript adds a range of 0, that is, no cost volume, which was consistently worse (final pass 5.09).
Capacity. A three-layer feature extractor per level and a seven-layer estimator were better than the defaults, a one-layer extractor or a four-layer estimator worse. Deeper estimators sometimes got stuck in poor local minima, detectable from the validation error after a few thousand iterations and fixed by restarting from another random initialization.
Context network and DenseNet. Removing the context network increased errors on both training and validation sets. Removing the DenseNet connections increased the Chairs error but lowered the Sintel errors after Chairs training; after fine-tuning on FlyingThings3D, DenseNet gave lower errors. The supplement’s Table 1a prints the “No context” and “No DenseNet” rows the other way round, and its Table 1b differs slightly from the fine-tuned columns of Table 5c; the main text supports Table 5c’s assignment. Table 5c also lists the fine-tuned full model at 2.34, 2.60 and 3.95, against 2.30, 2.55 and 3.93 for the same Chairs-then-Things model in Table 6.
Residual estimator. An estimator that predicts a flow increment and adds it to the upsampled flow was slightly better (final pass 4.43).

Strengths

Size. About 9 million parameters, under a tenth of FlowNet2’s, and 41 MB of memory, against a FlowNet2 footprint the authors judged unsuited to mobile and embedded devices.
Speed. About 35 frames per second at Sintel resolution on a Pascal Titan X, faster than FlowNet2 and far faster than the classical methods it outperformed.
Large displacements with a small search. Linking per-level cost volumes by warping covers large motions at a fraction of the cost of a full-range volume.
End-to-end training. One network trained from scratch, unlike FlowNet2’s sequentially trained stack.
Accuracy on realistic data. At publication, the lowest error on Sintel’s final pass and KITTI 2015 among published flow methods.

Limitations

Coarse-to-fine failures. Small, fast-moving objects and thin structures are lost at coarse levels, and the finer levels search only a few pixels around the warped estimate; the authors show failures of both kinds.
Clean-pass accuracy. On Sintel’s clean pass it stayed behind classical methods that use image edges to sharpen motion boundaries.
Dependence on training. Results depend heavily on the data schedule and fine-tuning; the TPAMI study found that much of the improvement over earlier networks came from training, not architecture (see Variants).
Implementation details outside the paper. The learned upsampling of flow and features, the warping factors at levels other than 2 and the extra convolution per level in the default PyTorch model are documented only in the code, and the paper’s cost-volume size ( $d^2$ ) does not match its own figure.
Licence. The official code and models are released under CC BY-NC-SA 4.0, which excludes commercial use (as of November 2026).

Historical Importance

PWC-Net showed that the principles of classical variational flow, pyramids, warping and cost volumes, could be built into a network that was both smaller and more accurate than the stacked FlowNet2, reversing the trade-off between accuracy and size that FlowNet2 and SpyNet had suggested. By its authors’ account, it was the first end-to-end method to beat the engineered classical methods on the Sintel final pass. A variant fine-tuned on several datasets, PWC-Net ROB, ranked first in the optical flow leaderboard of the Robust Vision Challenge 2018, which scores one parameter setting across Middlebury, KITTI, MPI Sintel and HD1K. When RAFT appeared (Teed and Deng, 2020), its authors counted PWC-Net among the leading networks that had inherited the coarse-to-fine strategy, and built RAFT to avoid that cascade’s weaknesses: it keeps a single resolution, computes correlations between all pairs of pixels, and refines one flow field iteratively.

Variants

PWC-Net-small (2018). The same design without DenseNet connections in the estimator: 4.08 million parameters, about 5% less accurate and 40% faster.
PWC-Net-ft and PWC-Net-ft-final (2018). The Sintel fine-tuning schemes described under Training; PWC-Net-ft-final is the entry with 5.04 on Sintel final.
“Models Matter, So Does Training” (TPAMI, 2020). The extension, read here in its September 2018 manuscript on NVIDIA Research’s site, keeps the architecture unchanged across the first arXiv version (September 2017), the CVPR version (November 2017) and the manuscript itself, and studies training. Horizontal flips, dropping additive Gaussian noise and “disrupting” the learning rate during fine-tuning improved the Sintel final test error by about 10% between the first two versions. Retrained with PWC-Net’s procedure, including the robust loss, FlowNetC (called FlowNetC+) reached 5.47 on the Sintel final test set, against 8.51 for the fine-tuned FlowNetC in the 2015 FlowNet paper (as the FlowNet article reports it) and 5.74 for the fine-tuned FlowNet2; the manuscript’s “about 56%” and “about 5% more accurate” match these ratios taken as old error over new (our arithmetic). FlowNetC+ remained about 8% less accurate than PWC-Net-ft-final (5.47 against 5.04); the manuscript calls it three times PWC-Net’s size, although Table 7’s counts (39.17 against 8.75 million parameters) give about 4.5. Fixing an input bug that had misread 44 of the 200 KITTI 2015 training sequences improved KITTI 2015 Fl-all from 9.60% to 7.90% and KITTI 2012 Fl-Noc from 4.22% to 3.41%. Fine-tuning on Sintel, KITTI and HD1K together gave PWC-Net ROB (Sintel final 4.90) and, with a different batch mix, PWC-Net+ (final 4.60, clean 3.45). The manuscript also adds ablations on the image pyramid, the cost volume and the number of levels, and a comparison table of FlowNetC, FlowNet2, SpyNet and PWC-Net: 3-level or image pyramids against PWC-Net’s 6-level feature pyramid, image warping against feature warping, a single-level volume with a large range against volumes at several levels with a small range. The published TPAMI version was not compared with this manuscript.
LiteFlowNet (Hui, Tang and Loy, CVPR 2018). A concurrent network from the Chinese University of Hong Kong built on the same ideas: a two-stream encoder (NetC) producing pyramidal features, and a decoder (NetE) that warps the second image’s features and estimates flow coarse to fine. At each level it runs a cascade of two small networks, a descriptor-matching unit on a short-range cost volume (radius 3 at levels 6 to 4, radius 6 at levels 3 and 2, where matching is done on a grid with stride 2) and a sub-pixel refinement unit, followed by a flow regularization layer, a feature-driven local convolution whose per-pixel filters depend on the features, the flow and an occlusion estimate. It has 5.37 million parameters, about 30 times fewer than FlowNet2, and runs 1.36 times faster (90.25 against 122.39 ms on a GTX 1080). It is trained stage-wise, level by level, on the Chairs → Things3D schedule. Its fine-tuned version reported 6.09 on the Sintel final test set and 10.24% Fl-all on KITTI 2015; disabling feature warping increased the error by 33% on KITTI 2012 and 25% on KITTI 2015 in its ablation. The PWC-Net TPAMI manuscript notes that an updated LiteFlowNet with similar data augmentation came close to PWC-Net, as further evidence that training matters.

Implementations

The official repository, NVlabs/PWC-Net, contains the Caffe models, training prototxt files and solvers, a warping layer in CUDA, the retrained PWC-Net+ models with their training files, and a PyTorch port, which its README says almost matches the Caffe model (Sintel final training EPE 2.31 against 2.29). The PyTorch code was written for Python 2.7 and PyTorch 0.2 with a custom CUDA correlation package, and its default model, PWCDCNet, uses three convolutions per pyramid level where the paper describes two (the two-convolution version is kept as PWCDCNet_old). The repository’s TensorFlow folder points to an implementation by Phil Ferriere. Code and models are licensed under CC BY-NC-SA 4.0; the latest commit dates from August 2022 (as of November 2026).

Example: a pyramid of partial cost volumes without learning

The NumPy code below (NumPy 2.3.5, SciPy 1.16.3, scikit-image 0.25.2) keeps PWC-Net’s skeleton but replaces every learned part with something fixed, to isolate what the pyramid, warping and partial cost volume do. The features are $5 \times 5$ patches of the image at each level ( $N = 25$ channels), and the convolutional estimator is replaced by winner-take-all: each pixel adds the displacement with the highest correlation (Eq. 2) to the upsampled flow. Because PWC-Net learns its features, the code runs two fixed substitutes: patches normalized to zero mean and unit length, and raw standardized intensities. The data are 8 pairs made from scikit-image’s bundled grass texture: the second frame is a $192 \times 192$ crop, and the first is sampled at $\mathbf{x} + \mathbf{w}(\mathbf{x})$ for a smooth flow field $\mathbf{w}$ , a translation of 0 to 35 pixels plus a smooth variation of up to 4 pixels per component, so the ground truth is exact. Five methods are compared: single-scale cost volumes with $d = 4$ (A) and $d = 40$ (B, enough to cover every motion here), the coarsest level of a 5-level pyramid alone, upsampled (C), and the full pyramid with feature warping and $d = 4$ (D, PWC-Net’s range) or $d = 2$ (E). The table reports mean EPE by true displacement, overall, and the share of pixels with an error above 3 pixels, with the number of cost-volume channels computed per full-resolution pixel. After the first run, two things were changed in what is printed: the column with the share of pixels above 3 pixels was added, because the means alone cannot show whether errors are spread evenly or concentrated in a few badly wrong pixels, and the channel count is printed with one decimal, because the coarsest-level count (81/256) had printed as 0. The data, methods and EPE columns did not change. The script takes under a minute, mostly for B.

import numpy as np
from scipy import ndimage
from skimage import data

S = data.grass().astype(float)                      # 512 x 512 bundled texture
S = (S - S.mean()) / S.std()
H = W = 192
Y, X = np.mgrid[0:H, 0:W].astype(float)
I2 = S[160:160 + H, 160:160 + W]                    # second frame


def make_pair(k):
    """Frame 1 sampled from the source at x + w(x), so I1(x) = I2(x + w(x)) exactly."""
    rng = np.random.default_rng(k)
    angle = rng.uniform(0, 2 * np.pi)
    w = np.zeros((H, W, 2))
    for c in range(2):
        f = ndimage.gaussian_filter(rng.standard_normal((H, W)), 24)
        w[..., c] = 4 * f / np.abs(f).max()          # smooth field, |component| <= 4 px
    w += 5 * k * np.array([np.cos(angle), np.sin(angle)])
    I1 = ndimage.map_coordinates(S, [Y + 160 + w[..., 1], X + 160 + w[..., 0]], order=3)
    return I1, w


def features(img, normalize):
    """5x5 patch vectors (N = 25 channels), a fixed stand-in for learned features."""
    p = np.pad(img, 2, mode="edge")
    f = np.stack([p[dy:dy + img.shape[0], dx:dx + img.shape[1]]
                  for dy in range(5) for dx in range(5)])
    if normalize:
        f = f - f.mean(0)
        f = f / (np.linalg.norm(f, axis=0) + 1e-6)
    return f


def warp(f, flow):
    """Bilinear backward warp of every feature channel: f(x + flow(x)), zero outside."""
    h, w = f.shape[1:]
    yy, xx = np.mgrid[0:h, 0:w].astype(float)
    coords = [yy + flow[..., 1], xx + flow[..., 0]]
    return np.stack([ndimage.map_coordinates(c, coords, order=1, cval=0.0) for c in f])


def cost_volume_argmax(f1, f2, d):
    """Partial cost volume (Eq. 2: (1/N) f1(x)^T f2(x + delta), |delta|_inf <= d), then argmax."""
    N, h, w = f1.shape
    pad = np.pad(f2, ((0, 0), (d, d), (d, d)))
    best = np.full((h, w), -np.inf)
    arg = np.zeros((h, w, 2))
    for dy in range(-d, d + 1):
        for dx in range(-d, d + 1):
            cost = np.einsum("nij,nij->ij", f1, pad[:, d + dy:d + dy + h, d + dx:d + dx + w]) / N
            better = cost > best
            best[better] = cost[better]
            arg[better] = (dx, dy)
    return arg


def pyramid(img, levels):
    out = [img]
    for _ in range(levels - 1):
        out.append(ndimage.gaussian_filter(out[-1], 1.0)[::2, ::2])
    return out


def upsample(flow, shape):
    """Bilinear x2 upsampling of the flow field; vectors doubled for the finer grid."""
    zoom = (shape[0] / flow.shape[0], shape[1] / flow.shape[1])
    return np.stack([ndimage.zoom(flow[..., c], zoom, order=1) for c in range(2)], -1) * zoom[0]


def single_scale(I1, I2, d, normalize):
    return cost_volume_argmax(features(I1, normalize), features(I2, normalize), d)


def pwc_style(I1, I2, d, normalize, levels=5, warping=True):
    P1, P2 = pyramid(I1, levels), pyramid(I2, levels)
    flow = np.zeros(P1[-1].shape + (2,))
    for lev in reversed(range(levels)):
        if flow.shape[:2] != P1[lev].shape:
            flow = upsample(flow, P1[lev].shape)
        if not warping and lev < levels - 1:
            continue                                  # coarsest estimate only, upsampled
        f1, f2 = features(P1[lev], normalize), features(P2[lev], normalize)
        f2w = warp(f2, flow) if warping else f2
        flow = flow + cost_volume_argmax(f1, f2w, d)
    return flow


def channels(d, levels=1):
    return sum((2 * d + 1) ** 2 / 4 ** l for l in range(levels))


if __name__ == "__main__":
    bins = [0, 4, 8, 16, 32, np.inf]
    methods = [("A single scale, d=4", lambda a, b, n: single_scale(a, b, 4, n), channels(4)),
               ("B single scale, d=40", lambda a, b, n: single_scale(a, b, 40, n), channels(40)),
               ("C coarsest level only", lambda a, b, n: pwc_style(a, b, 4, n, warping=False), channels(4) / 256),
               ("D pyramid+warp, d=4", lambda a, b, n: pwc_style(a, b, 4, n), channels(4, 5)),
               ("E pyramid+warp, d=2", lambda a, b, n: pwc_style(a, b, 2, n), channels(2, 5))]
    pairs = [make_pair(k) for k in range(8)]
    valid = []
    for I1, w in pairs:
        tx, ty = X + w[..., 0], Y + w[..., 1]
        v = (tx >= 2) & (tx <= W - 3) & (ty >= 2) & (ty <= H - 3)
        v[:8], v[-8:], v[:, :8], v[:, -8:] = False, False, False, False
        valid.append(v)
    mags = np.concatenate([np.hypot(w[..., 0], w[..., 1])[v] for (_, w), v in zip(pairs, valid)])
    print("pixels per |w| bin:", [int(((mags >= lo) & (mags < hi)).sum()) for lo, hi in zip(bins, bins[1:])])
    print(f"{'method':24s}{'features':>11s}{'ch/px':>7s} |  0-4   4-8  8-16 16-32  32+  |  all  >3px")
    for name, fn, ch in methods:
        for normalize in (True, False):
            errs = np.concatenate([np.linalg.norm(fn(I1, I2, normalize) - w, axis=-1)[v]
                                   for (I1, w), v in zip(pairs, valid)])
            cells = [errs[(mags >= lo) & (mags < hi)].mean() for lo, hi in zip(bins, bins[1:])]
            print(f"{name:24s}{'normalized' if normalize else 'raw':>11s}{ch:7.1f} |"
                  + "".join(f"{c:6.2f}" for c in cells) + f"  | {errs.mean():5.2f} {(errs > 3).mean():5.0%}")

pixels per |w| bin: [31334, 33085, 45696, 92766, 25187]
method                     features  ch/px |  0-4   4-8  8-16 16-32  32+  |  all  >3px
A single scale, d=4      normalized   81.0 |  0.44  6.32 11.98 24.43 34.97  | 17.18   83%
A single scale, d=4             raw   81.0 |  1.91  6.17 11.80 24.46 34.94  | 17.33   87%
B single scale, d=40     normalized 6561.0 |  3.16  3.41  3.56  4.12  4.24  |  3.79   11%
B single scale, d=40            raw 6561.0 | 26.54 27.45 29.28 30.98 34.14  | 29.87   90%
C coarsest level only    normalized    0.3 |  1.41  5.78  9.50  9.46 14.26  |  8.36   80%
C coarsest level only           raw    0.3 |  2.69  8.84 16.21 16.79 23.45  | 14.32   83%
D pyramid+warp, d=4      normalized  107.9 |  0.71  1.60  7.58  6.83 12.72  |  6.03   34%
D pyramid+warp, d=4             raw  107.9 | 13.20 18.70 25.66 26.17 28.18  | 23.43   92%
E pyramid+warp, d=2      normalized   33.3 |  0.55  0.74  2.60  2.64  7.08  |  2.56   13%
E pyramid+warp, d=2             raw   33.3 |  3.13  4.85  5.29  6.47  8.59  |  5.77   50%

The single-scale volume with $d = 4$ (A) shows the limit that motivates the design: below 4 pixels it is as precise as integer matching allows, and above that its error grows with the displacement, because the true match has no channel. A full-range volume (B) removes that limit, at 6,561 channels per pixel, 61 times the pyramid with $d = 4$ and nearly 200 times the one with $d = 2$ . The plan expected B with normalized features to be “accurate in every bin”; it was not, and about one pixel in nine is more than 3 pixels off. Our reading: every extra candidate is another chance for a distractor in this texture to score higher than the true match. The coarsest level alone (C) has the reach but not the precision: one pixel there spans 16 at full resolution. Warping and refining at the finer levels with $d = 2$ (E) gives a lower mean error than the full search in every bin below 32 pixels and overall, with a similar share of pixels off by more than 3 pixels (13% against 11%), at 33 channels per pixel; only above 32 pixels is its error higher.

The plan expected the pyramid with PWC-Net’s range of 4 (D) to match B “within a few tenths of a pixel in every bin”, and the range of 2 (E) to be “similar to D”; instead D did worse than E in every bin, with a third of the pixels more than 3 pixels off. Our reading, not a claim of the paper: with winner-take-all, a wider window at the coarse levels admits more false matches, and once a coarse level picks a wrong displacement, the finer levels search only $\pm d$ around the warped estimate and cannot return to the true one, so errors made at the top are inherited by the full-resolution field. PWC-Net’s estimator does not take an argmax; it is a network that sees the whole cost volume together with the features and the upsampled flow of neighbouring pixels, and the context network adds a wider spatial view, which is what this untrained version lacks. The paper’s own ablation found the search range to matter little on Sintel and the range of 2 to give the lowest KITTI outlier rates, which points the same way, but its networks were trained and ours are not. Raw intensity patches have a higher overall error than normalized ones for every method, and fail outright for B and D, as the raw dot product did in FlowNet’s correlation example: in our reading, without normalization the score grows with patch contrast as well as with agreement, so the features before a correlation have to be learned into a form where the raw product is a useful matching score. The pyramid’s weakness for small, fast objects appears in the coarse-to-fine estimation example, with a dense Lucas–Kanade estimator.