Describing Videos by Exploiting Temporal Structure

TL;DR

This paper introduces a novel video description method that leverages 3-D CNNs for capturing short-term dynamics and a temporal attention mechanism for selecting relevant segments, achieving state-of-the-art results on multiple datasets.

Contribution

It combines 3-D CNN representations with a temporal attention mechanism to improve video description quality, addressing both local and global temporal structures.

Findings

Outperforms previous methods on BLEU and METEOR metrics.

Effective in selecting relevant temporal segments for description.

Demonstrates scalability on larger datasets.

Abstract

Recent progress in using recurrent neural networks (RNNs) for image description has motivated the exploration of their application for video description. However, while images are static, working with videos requires modeling their dynamic temporal structure and then properly integrating that information into a natural language description. In this context, we propose an approach that successfully takes into account both the local and global temporal structure of videos to produce descriptions. First, our approach incorporates a spatial temporal 3-D convolutional neural network (3-D CNN) representation of the short temporal dynamics. The 3-D CNN representation is trained on video action recognition tasks, so as to produce a representation that is tuned to human motion and behavior. Second we propose a temporal attention mechanism that allows to go beyond local temporal modeling and learns to automatically select the most relevant temporal segments given the text-generating RNN. Our approach exceeds the current state-of-art for both BLEU and METEOR metrics on the Youtube2Text dataset. We also present results on a new, larger and more challenging dataset of paired video and natural language descriptions.