OCSampler: Compressing Videos to One Clip with Single-step Sampling

TL;DR

OCSampler introduces a single-step, instance-specific video condensation method that efficiently selects informative frames for recognition, outperforming previous sequential sampling approaches in accuracy and speed.

Contribution

The paper presents a novel single-step learning paradigm for video frame sampling, enabling efficient and accurate video recognition with fewer frames and reduced computational cost.

Findings

Achieves 76.9% mAP on ActivityNet

Runs at 123.9 Videos/sec on a single GPU

Outperforms previous methods in accuracy and efficiency

Abstract

In this paper, we propose a framework named OCSampler to explore a compact yet effective video representation with one short clip for efficient video recognition. Recent works prefer to formulate frame sampling as a sequential decision task by selecting frames one by one according to their importance, while we present a new paradigm of learning instance-specific video condensation policies to select informative frames for representing the entire video only in a single step. Our basic motivation is that the efficient video recognition task lies in processing a whole sequence at once rather than picking up frames sequentially. Accordingly, these policies are derived from a light-weighted skim network together with a simple yet effective policy network within one step. Moreover, we extend the proposed method with a frame number budget, enabling the framework to produce correct predictions in high confidence with as few frames as possible. Experiments on four benchmarks, i.e., ActivityNet, Mini-Kinetics, FCVID, Mini-Sports1M, demonstrate the effectiveness of our OCSampler over previous methods in terms of accuracy, theoretical computational expense, actual inference speed. We also evaluate its generalization power across different classifiers, sampled frames, and search spaces. Especially, we achieve 76.9% mAP and 21.7 GFLOPs on ActivityNet with an impressive throughput: 123.9 Videos/s on a single TITAN Xp GPU.