Bidirectional Multirate Reconstruction for Temporal Modeling in Videos

TL;DR

This paper introduces an unsupervised bidirectional multirate reconstruction method for temporal modeling in videos, effectively handling motion speed variations and improving performance in event detection and captioning tasks.

Contribution

It proposes a novel multirate visual recurrent model with bidirectional reconstruction for unsupervised temporal learning in videos, addressing motion speed variance.

Findings

Achieves 10.4% improvement in event detection on MEDTest-13

Sets new state-of-the-art in video captioning on YouTube2Text

Effective in modeling temporal information with untrimmed videos

Abstract

Despite the recent success of neural networks in image feature learning, a major problem in the video domain is the lack of sufficient labeled data for learning to model temporal information. In this paper, we propose an unsupervised temporal modeling method that learns from untrimmed videos. The speed of motion varies constantly, e.g., a man may run quickly or slowly. We therefore train a Multirate Visual Recurrent Model (MVRM) by encoding frames of a clip with different intervals. This learning process makes the learned model more capable of dealing with motion speed variance. Given a clip sampled from a video, we use its past and future neighboring clips as the temporal context, and reconstruct the two temporal transitions, i.e., present$\rightarrow$past transition and present$\rightarrow$future transition, reflecting the temporal information in different views. The proposed method exploits the two transitions simultaneously by incorporating a bidirectional reconstruction which consists of a backward reconstruction and a forward reconstruction. We apply the proposed method to two challenging video tasks, i.e., complex event detection and video captioning, in which it achieves state-of-the-art performance. Notably, our method generates the best single feature for event detection with a relative improvement of 10.4% on the MEDTest-13 dataset and achieves the best performance in video captioning across all evaluation metrics on the YouTube2Text dataset.