Ordered Pooling of Optical Flow Sequences for Action Recognition

TL;DR

This paper proposes an ordered pooling method for optical flow sequences to improve action recognition in videos, demonstrating that flow-based summaries outperform RGB frames and match state-of-the-art accuracy.

Contribution

It introduces a novel ordered optical flow pooling technique that captures action dynamics more effectively than RGB-based methods.

Findings

Flow summaries outperform RGB frames in action recognition accuracy.

The method achieves comparable results to state-of-the-art on UCF101 and HMDB datasets.

Optical flow-based representations improve dynamic information capture.

Abstract

Training of Convolutional Neural Networks (CNNs) on long video sequences is computationally expensive due to the substantial memory requirements and the massive number of parameters that deep architectures demand. Early fusion of video frames is thus a standard technique, in which several consecutive frames are first agglomerated into a compact representation, and then fed into the CNN as an input sample. For this purpose, a summarization approach that represents a set of consecutive RGB frames by a single dynamic image to capture pixel dynamics is proposed recently. In this paper, we introduce a novel ordered representation of consecutive optical flow frames as an alternative and argue that this representation captures the action dynamics more effectively than RGB frames. We provide intuitions on why such a representation is better for action recognition. We validate our claims on standard benchmark datasets and demonstrate that using summaries of flow images lead to significant improvements over RGB frames while achieving accuracy comparable to the state-of-the-art on UCF101 and HMDB datasets.