Condensing a Sequence to One Informative Frame for Video Recognition

TL;DR

This paper introduces a novel method called Informative Frame Synthesis (IFS) that condenses videos into a single informative frame capturing essential spatio-temporal information for efficient video recognition.

Contribution

The paper proposes an end-to-end trainable IFS architecture with multiple objectives and regularizers, enabling effective video-to-image synthesis for recognition tasks.

Findings

IFS outperforms baseline methods in video recognition accuracy.

It improves performance on both 2D and 3D image-based networks.

IFS achieves comparable results to state-of-the-art methods with less computation.

Abstract

Video is complex due to large variations in motion and rich content in fine-grained visual details. Abstracting useful information from such information-intensive media requires exhaustive computing resources. This paper studies a two-step alternative that first condenses the video sequence to an informative "frame" and then exploits off-the-shelf image recognition system on the synthetic frame. A valid question is how to define "useful information" and then distill it from a video sequence down to one synthetic frame. This paper presents a novel Informative Frame Synthesis (IFS) architecture that incorporates three objective tasks, i.e., appearance reconstruction, video categorization, motion estimation, and two regularizers, i.e., adversarial learning, color consistency. Each task equips the synthetic frame with one ability, while each regularizer enhances its visual quality. With these, by jointly learning the frame synthesis in an end-to-end manner, the generated frame is expected to encapsulate the required spatio-temporal information useful for video analysis. Extensive experiments are conducted on the large-scale Kinetics dataset. When comparing to baseline methods that map video sequence to a single image, IFS shows superior performance. More remarkably, IFS consistently demonstrates evident improvements on image-based 2D networks and clip-based 3D networks, and achieves comparable performance with the state-of-the-art methods with less computational cost.