Action Representation Using Classifier Decision Boundaries

TL;DR

This paper introduces SVM pooling, a novel method for action recognition that leverages classifier decision boundaries to select the most relevant features from video segments, improving accuracy over traditional pooling methods.

Contribution

The paper proposes a new pooling scheme based on learning decision hyperplanes that identify key features for action recognition, advancing beyond heuristic aggregation methods.

Findings

Achieved state-of-the-art results on HMDB and UCF101 datasets.

Demonstrated robustness of hyperplane-based features for video classification.

Provided an efficient joint optimization framework for hyperplane and classifier learning.

Abstract

Most popular deep learning based models for action recognition are designed to generate separate predictions within their short temporal windows, which are often aggregated by heuristic means to assign an action label to the full video segment. Given that not all frames from a video characterize the underlying action, pooling schemes that impose equal importance to all frames might be unfavorable. In an attempt towards tackling this challenge, we propose a novel pooling scheme, dubbed SVM pooling, based on the notion that among the bag of features generated by a CNN on all temporal windows, there is at least one feature that characterizes the action. To this end, we learn a decision hyperplane that separates this unknown yet useful feature from the rest. Applying multiple instance learning in an SVM setup, we use the parameters of this separating hyperplane as a descriptor for the video. Since these parameters are directly related to the support vectors in a max-margin framework, they serve as robust representations for pooling of the CNN features. We devise a joint optimization objective and an efficient solver that learns these hyperplanes per video and the corresponding action classifiers over the hyperplanes. Showcased experiments on the standard HMDB and UCF101 datasets demonstrate state-of-the-art performance.