Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks

TL;DR

This paper introduces a hierarchical RNN framework for video paragraph captioning, generating coherent multi-sentence descriptions by modeling inter-sentence dependencies and focusing on visual elements.

Contribution

It proposes a novel hierarchical RNN architecture with sentence and paragraph generators, improving video captioning performance on benchmark datasets.

Findings

Outperforms state-of-the-art methods with BLEU@4 scores of 0.499 and 0.305

Uses temporal and spatial attention mechanisms for better visual focus

Demonstrates significant improvements on YouTubeClips and TACoS-MultiLevel datasets

Abstract

We present an approach that exploits hierarchical Recurrent Neural Networks (RNNs) to tackle the video captioning problem, i.e., generating one or multiple sentences to describe a realistic video. Our hierarchical framework contains a sentence generator and a paragraph generator. The sentence generator produces one simple short sentence that describes a specific short video interval. It exploits both temporal- and spatial-attention mechanisms to selectively focus on visual elements during generation. The paragraph generator captures the inter-sentence dependency by taking as input the sentential embedding produced by the sentence generator, combining it with the paragraph history, and outputting the new initial state for the sentence generator. We evaluate our approach on two large-scale benchmark datasets: YouTubeClips and TACoS-MultiLevel. The experiments demonstrate that our approach significantly outperforms the current state-of-the-art methods with BLEU@4 scores 0.499 and 0.305 respectively.