Beyond Spatial Pyramid Matching: Space-time Extended Descriptor for Action Recognition

TL;DR

This paper proposes a space-time extended descriptor for action recognition in videos, integrating spatio-temporal location into feature encoding to improve efficiency and accuracy over traditional spatio-temporal pyramids.

Contribution

It introduces a simple, efficient spatio-temporal location encoding method that outperforms or matches existing pyramid-based approaches in action recognition tasks.

Findings

Achieves comparable or better results than spatio-temporal pyramid methods.

Reduces dimensionality and overfitting risks in video feature encoding.

Demonstrates effectiveness across multiple benchmark datasets.

Abstract

We address the problem of generating video features for action recognition. The spatial pyramid and its variants have been very popular feature models due to their success in balancing spatial location encoding and spatial invariance. Although it seems straightforward to extend spatial pyramid to the temporal domain (spatio-temporal pyramid), the large spatio-temporal diversity of unconstrained videos and the resulting significantly higher dimensional representations make it less appealing. This paper introduces the space-time extended descriptor, a simple but efficient alternative way to include the spatio-temporal location into the video features. Instead of only coding motion information and leaving the spatio-temporal location to be represented at the pooling stage, location information is used as part of the encoding step. This method is a much more effective and efficient location encoding method as compared to the fixed grid model because it avoids the danger of over committing to artificial boundaries and its dimension is relatively low. Experimental results on several benchmark datasets show that, despite its simplicity, this method achieves comparable or better results than spatio-temporal pyramid.