[Paper note] FlowNet: Learning Optical Flow with Convolutional Networks

paper

code

Highlight

First paper to use trained CNN for optical flow estimation

Introduce novel correlation layer

Refine network by upsampling

Model

FlowNetSimple: concatenate two consecutive images.

FlowNetCorr: use correlation layer

Correlation layer

Calculated between two feature maps

c(x1,x2)=∑o∈[−k,k]×[−k,k]<f1(x1+o),f2(x2+o)>

See model picture for an illustration

Refinement



Concatenate the upsampled flow prediction and conv feature map

Experiment

Datasets:

Middlebury

KITTI

Sintel

Flying Chairs (proposed, auto generated)

Loss function: endpoint error – Euclidean distance between the predicted flow vector and GT.

Conclusion

FlowNet performs a little worse than other OF algorithm, but obviously faster.

Network trained on Flying Chairs (auto generated) data has good generalization ability on natural scenes.