Using phase instead of optical flow for action recognition

TL;DR

This paper introduces a novel Eulerian phase-based motion representation for action recognition, learned via deep networks, offering an alternative to traditional optical flow methods.

Contribution

It proposes a new end-to-end trainable phase extraction module using complex filters resembling Gabor filters for Eulerian motion representation.

Findings

Eulerian phase-based features improve action recognition accuracy.

The proposed method outperforms optical flow on UCF101 dataset.

The phase extraction module is versatile and can be integrated into various architectures.

Abstract

Currently, the most common motion representation for action recognition is optical flow. Optical flow is based on particle tracking which adheres to a Lagrangian perspective on dynamics. In contrast to the Lagrangian perspective, the Eulerian model of dynamics does not track, but describes local changes. For video, an Eulerian phase-based motion representation, using complex steerable filters, has been successfully employed recently for motion magnification and video frame interpolation. Inspired by these previous works, here, we proposes learning Eulerian motion representations in a deep architecture for action recognition. We learn filters in the complex domain in an end-to-end manner. We design these complex filters to resemble complex Gabor filters, typically employed for phase-information extraction. We propose a phase-information extraction module, based on these complex filters, that can be used in any network architecture for extracting Eulerian representations. We experimentally analyze the added value of Eulerian motion representations, as extracted by our proposed phase extraction module, and compare with existing motion representations based on optical flow, on the UCF101 dataset.