Learning to Extract Motion from Videos in Convolutional Neural Networks

TL;DR

This paper introduces a CNN-based method for extracting dense optical flow from videos, emphasizing invariance properties and efficient training, aiming to replace traditional algorithms in motion analysis tasks.

Contribution

The paper presents a CNN architecture derived from signal processing principles with rotation invariance and reduced parameters, trained on limited data for effective motion estimation.

Findings

Achieves comparable performance to classical methods on Middlebury benchmark.

Requires significantly less training data than existing CNN-based motion estimation methods.

Outputs a distributed representation capable of modeling multiple motions and textures.

Abstract

This paper shows how to extract dense optical flow from videos with a convolutional neural network (CNN). The proposed model constitutes a potential building block for deeper architectures to allow using motion without resorting to an external algorithm, \eg for recognition in videos. We derive our network architecture from signal processing principles to provide desired invariances to image contrast, phase and texture. We constrain weights within the network to enforce strict rotation invariance and substantially reduce the number of parameters to learn. We demonstrate end-to-end training on only 8 sequences of the Middlebury dataset, orders of magnitude less than competing CNN-based motion estimation methods, and obtain comparable performance to classical methods on the Middlebury benchmark. Importantly, our method outputs a distributed representation of motion that allows representing multiple, transparent motions, and dynamic textures. Our contributions on network design and rotation invariance offer insights nonspecific to motion estimation.