Rank Pooling for Action Recognition

TL;DR

This paper introduces a function-based temporal pooling method called rank pooling for action recognition, capturing video dynamics by learning to rank frame features, leading to improved recognition accuracy across various benchmarks.

Contribution

The paper presents a novel rank pooling technique that models temporal evolution in videos, providing a robust and interpretable representation for action recognition.

Findings

Rank pooling improves recognition accuracy by 7-10% over average pooling.

The method is compatible with existing appearance and motion features.

Rank pooling is fast, easy to implement, and effective across diverse action recognition tasks.

Abstract

We propose a function-based temporal pooling method that captures the latent structure of the video sequence data - e.g. how frame-level features evolve over time in a video. We show how the parameters of a function that has been fit to the video data can serve as a robust new video representation. As a specific example, we learn a pooling function via ranking machines. By learning to rank the frame-level features of a video in chronological order, we obtain a new representation that captures the video-wide temporal dynamics of a video, suitable for action recognition. Other than ranking functions, we explore different parametric models that could also explain the temporal changes in videos. The proposed functional pooling methods, and rank pooling in particular, is easy to interpret and implement, fast to compute and effective in recognizing a wide variety of actions. We evaluate our method on various benchmarks for generic action, fine-grained action and gesture recognition. Results show that rank pooling brings an absolute improvement of 7-10 average pooling baseline. At the same time, rank pooling is compatible with and complementary to several appearance and local motion based methods and features, such as improved trajectories and deep learning features.