Show, Tell and Summarize: Dense Video Captioning Using Visual Cue Aided Sentence Summarization

TL;DR

This paper introduces a novel dense video captioning framework that partitions videos into events, generates segment descriptions, and then summarizes these into a single sentence using a hierarchical LSTM with visual cues.

Contribution

The work proposes a new division-and-summarization framework with a hierarchical LSTM and attention mechanism for dense video captioning, improving the summarization of event descriptions.

Findings

Effective dense captioning demonstrated on ActivityNet dataset

Hierarchical LSTM with visual cues outperforms existing methods

Summarization achieves more coherent and comprehensive descriptions

Abstract

In this work, we propose a division-and-summarization (DaS) framework for dense video captioning. After partitioning each untrimmed long video as multiple event proposals, where each event proposal consists of a set of short video segments, we extract visual feature (e.g., C3D feature) from each segment and use the existing image/video captioning approach to generate one sentence description for this segment. Considering that the generated sentences contain rich semantic descriptions about the whole event proposal, we formulate the dense video captioning task as a visual cue aided sentence summarization problem and propose a new two stage Long Short Term Memory (LSTM) approach equipped with a new hierarchical attention mechanism to summarize all generated sentences as one descriptive sentence with the aid of visual features. Specifically, the first-stage LSTM network takes all semantic words from the generated sentences and the visual features from all segments within one event proposal as the input, and acts as the encoder to effectively summarize both semantic and visual information related to this event proposal. The second-stage LSTM network takes the output from the first-stage LSTM network and the visual features from all video segments within one event proposal as the input, and acts as the decoder to generate one descriptive sentence for this event proposal. Our comprehensive experiments on the ActivityNet Captions dataset demonstrate the effectiveness of our newly proposed DaS framework for dense video captioning.