FlowNet

FlowNet is a family of convolutional networks that predict optical flow directly from a pair of images. Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg and colleagues at the University of Freiburg and the Technical University of Munich introduced it at ICCV 2015 in two versions: FlowNetSimple (FlowNetS), which stacks the two frames as input channels of one network, and FlowNetCorr (FlowNetC), which processes each frame separately and compares their features in a correlation layer. Both are encoder–decoder networks: a contracting part that reduces resolution, and an expanding part that refines the flow back towards full resolution. They were trained on Flying Chairs, a synthetic dataset created for them. FlowNet 2.0 (Ilg et al., CVPR 2017) stacked several FlowNets with image warping between them, trained them with a schedule of datasets, and added a network for small displacements, which brought learned flow to the accuracy of the best classical methods of the time. This article covers the family; the task, its benchmarks and later networks are described under optical flow estimation.

Problem Addressed

By 2015 the most accurate optical flow methods were variational energy minimizations, often combined with descriptor matching for large displacements, such as DeepFlow and EpicFlow. EpicFlow and DeepFlow took 16 to 17 s per Sintel frame on a CPU, by the timings in the FlowNet paper. Convolutional networks had begun to make per-pixel predictions in semantic segmentation and depth estimation, but Dosovitskiy et al. pointed out that flow differs from those tasks: a network has to find correspondences between two images, not only describe one. Whether a generic network could learn this was open, so the authors built two architectures, one generic and one with an explicit matching operation.

The second obstacle was data. A network has to learn the task from examples, and the benchmarks with ground truth were small: 8 training pairs for Middlebury, 194 for KITTI and 1,041 for MPI Sintel, as the paper’s Table 1 lists them. The authors traded realism for quantity and generated Flying Chairs, 22,872 image pairs with exact flow.

FlowNet 2.0 addressed what remained. Its authors wrote that classical methods still defined the state of the art, and that FlowNet could not compete with variational methods on small displacements and on real-world data, where its flow fields were noisy.

Architecture

The networks of 2015 take two images and output a two-channel flow field at a quarter of the input resolution, which is then upsampled. Both versions share the same overall design: nine convolutional layers in the contracting part, six of them with stride 2 (the paper’s simplest form of pooling) and a ReLU after each layer, no fully connected layers, so any image size is accepted, and the same expanding part. Filter sizes fall from $7 \times 7$ in the first layer to $5 \times 5$ in the next two and $3 \times 3$ from the fourth layer on, while the number of feature maps roughly doubles after each stride-2 layer, from 64 to 1024 in the paper’s Figure 2 for a $384 \times 512$ input.

FlowNetSimple

FlowNetS concatenates the two RGB images into a six-channel input and lets a generic stack of convolutions decide how to extract motion. In Figure 2 of the paper, layers conv1 to conv6 reduce a $384 \times 512$ input to a $6 \times 8$ map with 1024 channels; layers conv3_1, conv4_1 and conv5_1 add stride-1 convolutions between the downsampling steps. That this plain design learned flow at all was, in the authors’ words, surprising: they had added the correlation layer because it was not clear a standard network would manage.

FlowNetCorr and the correlation layer

FlowNetC runs the two images through two separate but identical streams (conv1 to conv3), a Siamese structure in FlowNet 2.0’s words, so that each image is first described on its own, and then compares the two feature maps $\mathbf{f}_1, \mathbf{f}_2$ with a correlation layer. The correlation of a patch centred at $\mathbf{x}_1$ in the first map with a patch centred at $\mathbf{x}_2$ in the second is

$$c(\mathbf{x}_1, \mathbf{x}_2) = \sum_{\mathbf{o} \in [-k, k] \times [-k, k]} \big\langle \mathbf{f}_1(\mathbf{x}_1 + \mathbf{o}), \, \mathbf{f}_2(\mathbf{x}_2 + \mathbf{o}) \big\rangle ,$$

for a square patch of size $K = 2k + 1$. The paper notes that this is one step of a convolution in which data is convolved with other data instead of a filter, so the layer has no trainable weights. The official Caffe layer also divides the sum by $K^2 c$, the number of multiplied elements, for $c$ channels.

Comparing every position with every other position would cost $w^2 h^2$ patch comparisons for a $w \times h$ map, which the authors judged intractable for training. The layer therefore has three parameters besides $k$:

• Maximum displacement $d$. For each $\mathbf{x}_1$, only positions $\mathbf{x}_2$ within $d$ in each direction are compared, a neighbourhood of size $D = 2d + 1$.
• Stride $s_1$ subsamples the positions $\mathbf{x}_1$ in the first map.
• Stride $s_2$ subsamples the displacements within the neighbourhood.

The relative displacements are arranged as channels, so the output is a $w \times h$ map whose channels are the correlations for each displacement; the paper writes its size as $w \times h \times D^2$. FlowNetC uses $k = 0$ (single-pixel “patches”), $d = 20$, $s_1 = 1$ and $s_2 = 2$. With $s_2 = 2$, the official layer forms a grid of $2 \lfloor d / s_2 \rfloor + 1 = 21$ displacements per axis, giving the 441 channels drawn in Figure 2, so $D^2$ in the text is the count before the stride is applied. A $1 \times 1$ convolution of the first image’s features (conv_redir, 32 channels) is concatenated with the correlation output, giving 473 channels, and conv3_1 to conv6 continue as in FlowNetS. The cost volume article places this layer among other matching volumes.

The correlation is computed on conv3 features, which are at one-eighth of the input resolution ($48 \times 64$ for a $384 \times 512$ input in Figure 2). It follows that $d = 20$ feature pixels corresponds to about 160 input pixels, sampled every 16 input pixels because of $s_2$; this is our reading of the figure’s sizes, not a number the paper states. The authors suggested that the limited range of the correlation is one reason why FlowNetC did worse than FlowNetS on large displacements, and that the range could be increased at a computational cost.

Expanding part

The expanding part, which the paper calls refinement, is built from “upconvolutional” layers, an unpooling followed by a convolution (a transposed convolution in today’s terms; the paper notes that such layers are often called deconvolutional although they perform a convolution). At each step it upconvolves the current feature maps, predicts a coarse flow, and concatenates three things: the upconvolved features, the feature maps of the same resolution from the contracting part, and the upsampled coarser flow prediction. The authors credit this with keeping both the high-level information of coarse maps and the fine local information of early layers; unlike earlier fully convolutional designs, they upconvolve whole feature maps, not only the coarse prediction. Four such steps, from upconv5 (512 channels) to upconv2 (64 channels) in Figure 3, bring the prediction to one quarter of the input resolution ($96 \times 128$). Further learned refinement did not significantly improve results over bilinear upsampling to full resolution, which is what the networks use. The general pattern is described under encoder–decoder networks.

Variational refinement

As an option, marked “+v” in the results, the authors replaced the final bilinear upsampling with the variational method of Brox and Malik without its matching term: starting at quarter resolution, 20 coarse-to-fine iterations bring the flow to full resolution, followed by 5 more iterations there, with the smoothness weight reduced at detected image boundaries. It costs about a second per frame and smooths the noisy output of the networks.

FlowNet 2.0: stacking with warping

FlowNet 2.0 stacks networks. The first network in a stack receives the two images $I_1, I_2$; each later network receives $I_1$, $I_2$ and the previous estimate $\mathbf{w}_i = (u_i, v_i)^\top$. With warping, the second image is also resampled with bilinear interpolation according to the current flow,

$$\tilde I_{2,i}(x, y) = I_2(x + u_i, \, y + v_i),$$

and the next network additionally receives $\tilde I_{2,i}$ and the brightness error $e_i = \lVert \tilde I_{2,i} - I_1 \rVert$, so that it only has to estimate the remaining increment, the same idea as warping in classical coarse-to-fine estimation. Bilinear interpolation makes the warp differentiable, so a stack can be trained end to end (the derivation is in the supplementary material). The official release implements it as a FlowWarp layer, in addition to the Correlation layer of FlowNetC.

Ilg et al. tested the options with two FlowNetS networks (Table 2 of the paper). Stacking without warping improved the error on the Chairs test set but worsened it on the Sintel training set, a sign of overfitting. Stacking with warping always helped, and the best Sintel result came from freezing the first network and training only the second, after warping. Applying one network repeatedly with the same weights did not improve results, so the stack uses networks with different weights, trained one after another.

The first network, the bootstrap network, may be a FlowNetC or a FlowNetS; FlowNetC is used only in this position, because the inputs of later networks were too diverse for its Siamese structure. Networks can also be thinned: a FlowNetS with 3/8 of the channels in every layer was a good trade-off between speed and accuracy. The naming uses one letter per network, upper case for full width and lower case for 3/8 width, so FlowNet2-CSS is one FlowNetC followed by two FlowNetS, and FlowNet2-css is the thin version.

FlowNet 2.0: small displacements and fusion

The authors found that FlowNet could not reliably estimate small motions. Comparing training data with UCF101 videos, they observed that Sintel and Chairs are dominated by fast movement, while displacements in UCF101 are mostly below one pixel. They generated ChairsSDHom, a dataset in the style of Flying Chairs with very small displacements, a displacement histogram closer to UCF101’s, and some homogeneous or gradient-only backgrounds. Two networks use it:

• FlowNet2-CSS-ft-sd is FlowNet2-CSS fine-tuned as a whole on a mixture of Things3D and ChairsSDHom, with a non-linearity applied to the error to down-weight large displacements.
• FlowNet2-SD is a modified FlowNetS for small displacements: the first layer has no stride 2, the $7 \times 7$ and $5 \times 5$ kernels at the start are replaced by several $3 \times 3$ kernels, and convolutions between the upconvolutions smooth the estimate, since noise is a problem for subpixel motion.

A small fusion network combines the two. It receives both flows, their magnitudes and the brightness errors after warping, contracts the resolution twice by a factor of 2 and expands again, and, unlike the original FlowNet, outputs full resolution. The whole system is called FlowNet2. The paper leaves the exact layers of FlowNet2-SD and of the fusion network, and the form of the error non-linearity, to its supplementary material.

Training

Data and loss

The 2015 networks were trained on Flying Chairs, whose construction (Flickr backgrounds and rendered chairs moved by random 2D affine transformations, with a displacement histogram tuned to resemble Sintel’s) is described in the Flying Chairs article. The authors split it into 22,232 training and 640 test pairs. They found data augmentation crucial even at this size, and applied it on the GPU during training: the same strong geometric transformation to both images plus a smaller relative one between them, with translation sampled within ±20% of the image width, rotation within ±17° and scaling between 0.9 and 2.0 (the paper does not say whether these ranges belong to the shared or the relative transformation), additive Gaussian noise with $\sigma$ up to 0.04, and changes of contrast, per-channel colour, gamma and brightness. In their analysis, training without augmentation raised the error on Sintel by roughly 2 pixels.

The training loss is the endpoint error, the Euclidean distance between predicted and true flow vectors averaged over pixels:

$$\mathcal{L} = \frac{1}{N} \sum_{\mathbf{x}} \big\lVert \hat{\mathbf{w}}(\mathbf{x}) - \mathbf{w}(\mathbf{x}) \big\rVert_2 .$$

The paper does not state how the coarser flow predictions of the expanding part enter the loss.

Optimization (2015)

The networks were trained with a modified Caffe, using Adam with $\beta_1 = 0.9$ and $\beta_2 = 0.999$ and mini-batches of 8 image pairs, small because every pixel acts as a training sample. The learning rate started at $10^{-4}$ and was halved every 100k iterations after the first 300k. FlowNetC suffered exploding gradients at that rate, so its training started at $10^{-6}$ and rose to $10^{-4}$ over the first 10k iterations. At test time, FlowNetC’s input images were upscaled by a factor of 1.25, FlowNetS’s not at all.

Networks marked “+ft” were fine-tuned on the Sintel training set, clean and final passes together, at a learning rate of $10^{-6}$ for several thousand iterations: the number of iterations was chosen on a split of 908 training and 133 validation pairs, then the networks were fine-tuned on all pairs for that number. KITTI was not used for fine-tuning, because it is small and its ground truth is sparse.

Schedules and dataset order (2017)

Ilg et al. defined three learning-rate schedules (Figure 3 of their paper):

• $S_\text{short}$, 600k iterations, the 2015 schedule with two changes: it starts at $10^{-4}$ without a warm-up, holds that rate for 300k iterations and then halves it every 100k.
• $S_\text{long}$, 1.2M iterations; in the figure it holds $10^{-4}$ for 400k iterations and then halves the rate every 200k.
• $S_\text{fine}$, a fine-tuning schedule with smaller rates; in the figure it runs for 500k iterations after $S_\text{long}$, starting at $10^{-5}$.

They trained FlowNetS and FlowNetC on Chairs, on FlyingThings3D (Things3D, which they describe as a three-dimensional version of Chairs with 22k renderings of ShapeNet objects moving in front of static 3D backgrounds, with true 3D motion and lighting), on an equal mixture, and on Chairs followed by Things3D. Measured by endpoint error on the Sintel training set’s clean pass (their Table 1):

Network	Data	$S_\text{short}$	$S_\text{long}$	$S_\text{fine}$
FlowNetS	Chairs	4.45	4.24	4.21
FlowNetS	Things3D	–	5.07	4.50
FlowNetS	mixed	–	4.52	4.10
FlowNetS	Chairs → Things3D	–	4.24	3.79
FlowNetC	Chairs	3.77	–	–
FlowNetC	Chairs → Things3D	–	3.58	3.04

Three findings followed. Longer training helped. The more realistic Things3D alone gave worse results than Chairs alone, and Chairs followed by Things3D beat the mixture and either dataset alone; the authors conjectured that the simpler data first teaches matching of colours before priors about 3D motion and lighting can mislead the network, and related the effect to curriculum learning. FlowNetC beat FlowNetS, the opposite of the 2015 paper’s conclusion: Ilg et al. reproduced the 2015 FlowNetS result but obtained a much better FlowNetC, and concluded that the two had not been trained under exactly the same conditions in 2015. Changes of data and schedule alone improved the 2015 FlowNetS result by about 25% and FlowNetC by about 30%. The Flying Chairs article summarizes how this Chairs-then-Things order became the standard pretraining of later networks.

Every network of a FlowNet 2.0 stack is trained with this Chairs → Things3D schedule ($S_\text{long}$, then $S_\text{fine}$) while the networks below it are kept fixed. The README of the official release gives the training crop sizes: $448 \times 320$ with batches of 8 for Flying Chairs and ChairsSDHom, and $768 \times 384$ with batches of 4 for FlyingThings3D.

Inference

All FlowNets take two RGB images and return a dense flow field; there is no iteration or post-processing unless variational refinement is added. In 2015, on an NVIDIA GTX Titan, FlowNetS took 0.08 s and FlowNetC 0.15 s per frame pair at Sintel’s full resolution, and about 1 s with variational refinement, against 16 to 65 s on a CPU for EpicFlow, DeepFlow and LDOF and 0.2 s for EPPM on a GPU. The abstract states 5 to 10 frames per second and the introduction up to 10 pairs per second; 0.08 s per pair in Table 2 corresponds to 12.5 pairs per second.

FlowNet 2.0 timings (GTX 1080, Table 4 of its paper) range from 7 ms for FlowNet2-s, which the authors describe as running at 140 frames per second with nearly the accuracy of the original FlowNet, to 123 ms for the full FlowNet2, about 8 frames per second. On the same GPU, the 2015 FlowNetS and FlowNetC took 18 and 32 ms.

Performance

FlowNet (2015)

Dosovitskiy et al. report average endpoint error on Sintel (clean and final passes, training and test sets), KITTI (the 2012 flow benchmark, under which FlowNet 2.0’s Table 4 files these numbers), Middlebury and the Flying Chairs test pairs. Selected entries of their Table 2:

Method	Sintel clean test	Sintel final test	KITTI 2012 test	Middlebury test	Chairs test
EpicFlow	4.12	6.29	3.8	0.39	2.94
DeepFlow	5.38	7.21	5.8	0.42	3.53
EPPM	6.49	8.38	9.2	0.33	–
LDOF	7.56	9.12	12.4	0.56	3.47
FlowNetS	7.42	8.43	–	–	2.71
FlowNetS+ft	6.96	7.76	9.1	–	3.04
FlowNetS+ft+v	6.16	7.22	7.6	0.47	3.03
FlowNetC	7.28	8.81	–	–	2.19
FlowNetC+ft	6.85	8.51	–	–	2.27
FlowNetC+ft+v	6.08	7.88	–	0.50	2.67

The networks trained only on Flying Chairs beat LDOF on Sintel, and the fine-tuned FlowNetS beat EPPM, the competing real-time method, on Sintel’s final pass and on KITTI while running more than twice as fast (the paper says twice as fast). They did not reach EpicFlow or DeepFlow except on the Chairs test pairs, although FlowNetS+ft+v was on par with DeepFlow on Sintel’s final pass (7.22 against 7.21), as the authors note, where both networks beat every classical method and variational refinement made results worse. FlowNetC was better than FlowNetS on Chairs and Sintel clean, FlowNetS better on Sintel final and KITTI; the authors read this as FlowNetC adapting more closely to its training data, which lacks Sintel final’s blur and fog. On Sintel final, FlowNetS+ft had an endpoint error of 43.3 pixels on pixels moving at least 40 pixels, against 48 for FlowNetC+ft. Raw network output on Middlebury, where displacements are mostly below 10 pixels, was poor (training-set errors of 1.09 and 1.15, against 0.31 for EpicFlow) and improved to about 0.33 with variational refinement. A network trained only on Sintel, with aggressive augmentation, had an error on Sintel roughly 1 pixel higher than one pretrained on Chairs and fine-tuned on Sintel.

FlowNet 2.0 (2017)

Stacking improved accuracy steadily (Table 3 of Ilg et al., Sintel training clean pass): from 3.04 for FlowNet2-C to 2.20 for FlowNet2-CS and 2.10 for FlowNet2-CSS, about 30% better than FlowNet2-C and about 50% better than the original FlowNetC. Two thin networks beat one full network while being faster and smaller: FlowNet2-ss (11M weights) did better than FlowNet2-S (38M), and FlowNet2-cs (11M) better than FlowNet2-C (38M).

Selected test-set entries of their Table 4 (average endpoint error unless marked):

Method	Sintel clean test	Sintel final test	KITTI 2012 test	KITTI 2015 test Fl-all	Middlebury test
EpicFlow	4.12	6.29	3.8	27.10%	0.39
FlowFields	3.75	5.81	3.5	–	0.33
FlowNet2	3.96	6.02	–	–	0.52
FlowNet2-ft-sintel	4.16	5.74	–	–	–
FlowNet2-ft-kitti	–	–	1.8	11.48%	–

On Sintel, FlowNet2 outperformed DeepFlow and EpicFlow and was on par with FlowFields; fine-tuning on Sintel slightly degraded the clean-pass result and put the final pass on par with DeepDiscreteFlow, the best published method then. On KITTI, fine-tuning on the combined KITTI 2012 and 2015 training sets reduced the error by a factor of about 3, and the authors reported the best endpoint error among non-stereo methods on KITTI 2012 and first rank on KITTI 2015 at the time; Fl-all counts pixels whose error exceeds both 3 pixels and 5% of the true flow. On Middlebury, FlowNet2 was comparable to classical methods on the training set (0.35) but, in the authors’ word, unexpectedly much worse on the test set (0.52). Used as input to other methods, FlowNet2’s flow matched classical flow: on the FBMS-59 training set, a trajectory-based motion segmentation reached an F-measure of 79.92% with FlowNet2 against 56.87% with FlowNetS (evaluated on 28 of the 29 sequences, because the optimization did not converge on one), and the temporal stream of a two-stream action recognition network reached 79.51% accuracy against 55.27% with FlowNetS (Table 5).

Two inconsistencies between sources are worth noting. FlowNet 2.0’s Table 4 lists the fine-tuned FlowNetS at 7.52 on Sintel final test, whereas the 2015 paper’s Table 2 gives 7.76 for FlowNetS+ft (and 7.52 for that network on the KITTI training set). PWC-Net’s paper (Sun et al., 2018) lists the KITTI-fine-tuned FlowNet2 on the KITTI 2015 test set at an Fl-all of 10.41%, against 11.48% in FlowNet 2.0’s own table; the source of the difference is not stated. For current standings, see the official Sintel and KITTI leaderboards.

Strengths

• Speed. A single forward pass gave real-time flow in 2015, and the FlowNet 2.0 family covers 8 to 140 frames per second with one design.
• Large displacements and fine detail. FlowNet 2.0’s authors list mastering large displacements and fine motion detail among FlowNet’s advantages; on the Chairs test pairs, the 2015 paper shows the networks finding small objects with large displacements that EpicFlow missed.
• Learning from synthetic data. Networks trained only on unrealistic chairs transferred to Sintel and KITTI, and FlowNet 2.0’s schedule showed how to combine synthetic datasets.
• Specialization. Fine-tuning adapts a network to a domain, as FlowNet2-ft-kitti did for driving scenes.

Limitations

• Accuracy (2015). The original networks stayed behind the best classical methods on Sintel, KITTI and Middlebury, and their output was noisy and non-smooth in large smooth background regions.
• Small and subpixel motion. FlowNet failed on small displacements, and FlowNet 2.0 needed a separate network and dataset for them; its authors still read the Middlebury test result as imperfect subpixel accuracy.
• Bounded search range. FlowNetC compares features only within its maximum displacement, and its authors linked this range to its weaker large-displacement results.
• Size and training cost. By PWC-Net’s count, FlowNet2 has 162.49 million parameters and a memory footprint of 638.5 MB, against 38.67M for FlowNetS and 39.17M for FlowNetC, and its sub-networks must be trained one after another, at least 14 days by PWC-Net’s estimate.
• Domain gap. Results depend on the training data and its order; KITTI’s projective motion differed strongly from Flying Chairs, and benchmark accuracy relied on fine-tuning to each target set.

Historical Importance

FlowNet (2015) showed that optical flow, a correspondence problem, could be learned end to end by a convolutional network, and that synthetic training data with little resemblance to real scenes was enough. FlowNet 2.0’s authors called it a paradigm shift, as did PWC-Net’s. Correlating learned features, as FlowNetC did, remained central to later flow networks such as PWC-Net and RAFT, and pretraining on synthetic data, extended by FlowNet 2.0’s Chairs-then-Things schedule, was adopted by later networks, RAFT among them.

FlowNet 2.0 (2017) closed the accuracy gap to classical methods at a fraction of their runtime and brought warping, a classical idea, into a stack of learned stages. PWC-Net’s authors named its size, memory footprint and sequential training as drawbacks, and built pyramid, warping and cost volume into one network with 17 times fewer parameters; RAFT later dropped coarse-to-fine stages altogether.

Variants

• FlowNetS and FlowNetC (2015): the two original architectures, with “+v” (variational refinement) and “+ft” (Sintel fine-tuning) versions.
• FlowNet2-S, -C and thin versions (2017): single networks retrained with the Chairs → Things3D schedule, at full or 3/8 width.
• Stacks (2017): FlowNet2-CS, -CSS, -css, -ss and others, trading accuracy for speed.
• FlowNet2-SD, FlowNet2-CSS-ft-sd and FlowNet2 (2017): the small-displacement network, the stack fine-tuned on small displacements, and their fusion.
• Fine-tuned FlowNet2: FlowNet2-ft-sintel and FlowNet2-ft-kitti for the two benchmarks.

Successors include coarse-to-fine networks such as LiteFlowNet and PWC-Net (Sun et al., 2018), which warps learned features and builds a local cost volume at each level of a feature pyramid, and RAFT, which replaces local correlation with an all-pairs volume and recurrent refinement.

Implementations

The official release of FlowNet 2.0, lmb-freiburg/flownet2, is a fork of Caffe with the Correlation, FlowWarp and loss layers in CUDA. The trained networks are downloaded by a script in the repository (models/download-models.sh) from the Freiburg group’s server, and the README states that the code is for research purposes only, with commercial use requiring the authors’ consent (as of November 2026; the latest commit dates from July 2019). NVIDIA’s flownet2-pytorch is a PyTorch reimplementation of FlowNet2S, FlowNet2C, FlowNet2CS, FlowNet2CSS, FlowNet2SD and FlowNet2, with its own CUDA kernels for the correlation and resampling layers, under the Apache 2.0 license (as of November 2026).

The NumPy code below (NumPy 2.3.5, SciPy 1.16.3, scikit-image 0.25.2) implements the correlation layer of FlowNetC as Eq. 1 defines it, for a single channel, and applies it without any training to scikit-image’s bundled gravel texture and a copy translated by a known whole number of pixels. The features are the standardized intensities, the patch is $5 \times 5$ ($k = 2$), and each pixel takes the displacement of its highest-scoring channel. The cases vary the maximum displacement $d$ and the displacement stride $s_2$. The script was changed after its first run in three ways: a cosine variant, which divides each patch product by both patch norms and is not FlowNet’s layer, was added beside the raw dot product, because the first run found the raw product right for far fewer pixels inside the range than the plan had expected, and a score that ignores patch magnitude separates the effect of the range and stride from that of the scoring; the patch sum became a patch mean, the normalization of the official layer (this does not change the winner); and a line printing the channel count for FlowNetC’s own setting was added. The raw dot-product columns did not change. It runs in a few seconds.

import numpy as np
from scipy.ndimage import uniform_filter
from skimage import data

img = data.gravel().astype(float)                  # 512 x 512 bundled texture
img = (img - img.mean()) / img.std()

def correlation(f1, f2, k, d, s2, cosine=False):
    """FlowNetC's correlation (Eq. 1, c = 1 channel): for each displacement o on a grid
    with spacing s2 inside [-d, d]^2, sum f1(x + p) * f2(x + o + p) over a (2k+1)^2 patch.
    cosine=True divides by both patch norms (a normalized variant, not FlowNet's layer)."""
    offs = [(dy, dx) for dy in range(-(d // s2) * s2, d + 1, s2)
                     for dx in range(-(d // s2) * s2, d + 1, s2)]
    H, W = f1.shape
    out = np.empty((len(offs), H - 2 * d, W - 2 * d))
    for i, (dy, dx) in enumerate(offs):
        prod = f1[d:H - d, d:W - d] * f2[d + dy:H - d + dy, d + dx:W - d + dx]
        out[i] = uniform_filter(prod, 2 * k + 1, mode="constant")
        if cosine:
            e1 = uniform_filter(f1[d:H - d, d:W - d] ** 2, 2 * k + 1, mode="constant")
            e2 = uniform_filter(f2[d + dy:H - d + dy, d + dx:W - d + dx] ** 2, 2 * k + 1, mode="constant")
            out[i] /= np.sqrt(e1 * e2) + 1e-12
    return out, np.array(offs)

m = 48                                             # crop margin, larger than any shift
f1 = img[m:-m, m:-m]
print("channels for FlowNetC's d=20, s2=2:", len(correlation(f1[:50, :50], f1[:50, :50], 0, 20, 2)[1]))
print(" d s2 channels | shift (u,v) |   dot product      |   cosine")
print("               |             | correct  mean EPE  | correct  mean EPE")
for d, s2 in [(4, 1), (4, 2), (8, 2)]:
    for u, v in [(3, 0), (4, 2), (6, 2), (7, 5), (10, 0)]:
        f2 = img[m - v:img.shape[0] - m - v, m - u:img.shape[1] - m - u]   # content moves by (u, v)
        cells = []
        for cosine in (False, True):
            corr, offs = correlation(f1, f2, k=2, d=d, s2=s2, cosine=cosine)
            best = offs[corr.argmax(0)][..., ::-1]     # (dx, dy) of the winning channel
            err = np.hypot(best[..., 0] - u, best[..., 1] - v)[8:-8, 8:-8]
            cells.append(f"{np.mean(err == 0):6.1%}  {err.mean():6.2f}")
        print(f"{d:2d} {s2:2d} {len(offs):8d} | ({u:2d},{v:2d})     | " + "    | ".join(cells))

channels for FlowNetC's d=20, s2=2: 441
 d s2 channels | shift (u,v) |   dot product      |   cosine
               |             | correct  mean EPE  | correct  mean EPE
 4  1       81 | ( 3, 0)     |  25.5%    2.65    | 100.0%    0.00
 4  1       81 | ( 4, 2)     |  34.2%    2.84    | 100.0%    0.00
 4  1       81 | ( 6, 2)     |   0.0%    5.60    |   0.0%    4.97
 4  1       81 | ( 7, 5)     |   0.0%    8.25    |   0.0%    8.09
 4  1       81 | (10, 0)     |   0.0%   10.14    |   0.0%   10.07
 4  2       25 | ( 3, 0)     |   0.0%    3.19    |   0.0%    1.83
 4  2       25 | ( 4, 2)     |  47.2%    2.68    | 100.0%    0.00
 4  2       25 | ( 6, 2)     |   0.0%    5.64    |   0.0%    5.07
 4  2       25 | ( 7, 5)     |   0.0%    8.23    |   0.0%    8.05
 4  2       25 | (10, 0)     |   0.0%   10.17    |   0.0%   10.10
 8  2       81 | ( 3, 0)     |   0.0%    5.80    |   0.0%    2.96
 8  2       81 | ( 4, 2)     |  26.0%    5.45    | 100.0%    0.00
 8  2       81 | ( 6, 2)     |  26.1%    5.95    | 100.0%    0.00
 8  2       81 | ( 7, 5)     |   0.0%    7.74    |   0.0%    4.69
 8  2       81 | (10, 0)     |   0.0%   10.36    |   0.0%    9.24

The channel count reproduces the 441 channels of FlowNetC’s correlation output, and the table shows what the two parameters do. A displacement outside $[-d, d]$ has no channel, so no pixel can find it, whatever the scoring: with $d = 4$, shifts of 6, 7 and 10 pixels are never recovered, and the winner’s error grows with the shift. With $s_2 = 2$ only even displacements exist, so (3, 0) and (7, 5) are missed even inside the range, while (4, 2) and (6, 2) are on the grid; the stride roughly halves the channels per axis (from 9 to 5 at $d = 4$) or doubles the range for the same count. Within these limits the cosine score recovers every on-grid shift on this texture, but the raw dot product, which is what the layer computes, picks the right channel for only a quarter to a half of the pixels, because the product grows with the magnitude of the second image’s patch as well as with its agreement, so a nearby patch with larger values can outscore the true match. This is our reading of the result, not a claim of the paper: FlowNetC correlates learned features, and its layers before and after the correlation have to learn a representation in which the raw product is a useful matching score.