Deep hierarchical pooling design for cross-granularity action recognition

TL;DR

This paper presents a hierarchical pooling method for action recognition that captures multiple temporal granularities, improving localization and robustness across different video lengths, validated on the UCF-101 dataset.

Contribution

A novel tree-structured hierarchical pooling design that adaptively combines temporal features at multiple granularities for improved action recognition.

Findings

Effective in capturing multi-scale temporal information

Video-length agnostic and resilient to misalignments

Achieves state-of-the-art results on UCF-101

Abstract

In this paper, we introduce a novel hierarchical aggregation design that captures different levels of temporal granularity in action recognition. Our design principle is coarse-to-fine and achieved using a tree-structured network; as we traverse this network top-down, pooling operations are getting less invariant but timely more resolute and well localized. Learning the combination of operations in this network -- which best fits a given ground-truth -- is obtained by solving a constrained minimization problem whose solution corresponds to the distribution of weights that capture the contribution of each level (and thereby temporal granularity) in the global hierarchical pooling process. Besides being principled and well grounded, the proposed hierarchical pooling is also video-length agnostic and resilient to misalignments in actions. Extensive experiments conducted on the challenging UCF-101 database corroborate these statements.