MGSampler: An Explainable Sampling Strategy for Video Action Recognition

TL;DR

This paper introduces MGSampler, an explainable, motion-guided frame sampling method for video action recognition that adaptively selects informative frames, improving over fixed strategies across multiple benchmarks.

Contribution

We propose a novel motion-guided sampling strategy that is flexible, explainable, and effective, enhancing video action recognition performance across various datasets and models.

Findings

Outperforms fixed sampling strategies on five benchmarks.

Generalizes well across different video architectures.

Effectively captures salient motion information for sampling.

Abstract

Frame sampling is a fundamental problem in video action recognition due to the essential redundancy in time and limited computation resources. The existing sampling strategy often employs a fixed frame selection and lacks the flexibility to deal with complex variations in videos. In this paper, we present a simple, sparse, and explainable frame sampler, termed as Motion-Guided Sampler (MGSampler). Our basic motivation is that motion is an important and universal signal that can drive us to adaptively select frames from videos. Accordingly, we propose two important properties in our MGSampler design: motion sensitive and motion uniform. First, we present two different motion representations to enable us to efficiently distinguish the motion-salient frames from the background. Then, we devise a motion-uniform sampling strategy based on the cumulative motion distribution to ensure the sampled frames evenly cover all the important segments with high motion salience. Our MGSampler yields a new principled and holistic sampling scheme, that could be incorporated into any existing video architecture. Experiments on five benchmarks demonstrate the effectiveness of our MGSampler over the previous fixed sampling strategies, and its generalization power across different backbones, video models, and datasets.