Models Matter, So Does Training: An Empirical Study of CNNs for Optical Flow Estimation

TL;DR

This paper introduces PWC-Net, a compact and efficient CNN for optical flow estimation that outperforms larger models, and provides an analysis of training strategies that significantly improve accuracy.

Contribution

The paper presents PWC-Net, a small yet effective CNN model for optical flow, and demonstrates how training improvements can enhance existing models' performance.

Findings

PWC-Net is 17 times smaller and twice as fast as FlowNet2, with 11% higher accuracy.

Retraining FlowNetC with PWC-Net's training protocol improves its accuracy by 56%.

Training protocol enhancements increase PWC-Net's accuracy by up to 20% on multiple datasets.

Abstract

We investigate two crucial and closely related aspects of CNNs for optical flow estimation: models and training. First, we design a compact but effective CNN model, called PWC-Net, according to simple and well-established principles: pyramidal processing, warping, and cost volume processing. PWC-Net is 17 times smaller in size, 2 times faster in inference, and 11\% more accurate on Sintel final than the recent FlowNet2 model. It is the winning entry in the optical flow competition of the robust vision challenge. Next, we experimentally analyze the sources of our performance gains. In particular, we use the same training procedure of PWC-Net to retrain FlowNetC, a sub-network of FlowNet2. The retrained FlowNetC is 56\% more accurate on Sintel final than the previously trained one and even 5\% more accurate than the FlowNet2 model. We further improve the training procedure and increase the accuracy of PWC-Net on Sintel by 10\% and on KITTI 2012 and 2015 by 20\%. Our newly trained model parameters and training protocols will be available on https://github.com/NVlabs/PWC-Net