GCF-Net: Gated Clip Fusion Network for Video Action Recognition

TL;DR

The paper introduces GCF-Net, a novel network that explicitly models inter-clip dependencies and selects relevant clips to significantly improve video action recognition accuracy with minimal additional computation.

Contribution

GCF-Net is a new model that enhances existing video classifiers by modeling inter-clip relationships and selecting key clips for better video-level understanding.

Findings

Boosts accuracy of existing classifiers by over 11% on Kinetics-600.

Explicitly models inter-clip dependencies to improve feature representation.

Achieves significant performance gains with minimal computational overhead.

Abstract

In recent years, most of the accuracy gains for video action recognition have come from the newly designed CNN architectures (e.g., 3D-CNNs). These models are trained by applying a deep CNN on single clip of fixed temporal length. Since each video segment are processed by the 3D-CNN module separately, the corresponding clip descriptor is local and the inter-clip relationships are inherently implicit. Common method that directly averages the clip-level outputs as a video-level prediction is prone to fail due to the lack of mechanism that can extract and integrate relevant information to represent the video. In this paper, we introduce the Gated Clip Fusion Network (GCF-Net) that can greatly boost the existing video action classifiers with the cost of a tiny computation overhead. The GCF-Net explicitly models the inter-dependencies between video clips to strengthen the receptive field of local clip descriptors. Furthermore, the importance of each clip to an action event is calculated and a relevant subset of clips is selected accordingly for a video-level analysis. On a large benchmark dataset (Kinetics-600), the proposed GCF-Net elevates the accuracy of existing action classifiers by 11.49% (based on central clip) and 3.67% (based on densely sampled clips) respectively.