Bidirectional Attentive Fusion with Context Gating for Dense Video Captioning

TL;DR

This paper introduces a bidirectional proposal method and an attentive fusion approach with context gating for dense video captioning, significantly improving event localization and description accuracy by leveraging both past and future contexts.

Contribution

It proposes a novel bidirectional proposal mechanism and an attentive fusion with context gating, enhancing dense video captioning performance over previous methods.

Findings

Outperforms state-of-the-art on ActivityNet Captions dataset

Achieves over 100% relative improvement in Meteor score

Demonstrates the effectiveness of bidirectional context and attentive fusion

Abstract

Dense video captioning is a newly emerging task that aims at both localizing and describing all events in a video. We identify and tackle two challenges on this task, namely, (1) how to utilize both past and future contexts for accurate event proposal predictions, and (2) how to construct informative input to the decoder for generating natural event descriptions. First, previous works predominantly generate temporal event proposals in the forward direction, which neglects future video context. We propose a bidirectional proposal method that effectively exploits both past and future contexts to make proposal predictions. Second, different events ending at (nearly) the same time are indistinguishable in the previous works, resulting in the same captions. We solve this problem by representing each event with an attentive fusion of hidden states from the proposal module and video contents (e.g., C3D features). We further propose a novel context gating mechanism to balance the contributions from the current event and its surrounding contexts dynamically. We empirically show that our attentively fused event representation is superior to the proposal hidden states or video contents alone. By coupling proposal and captioning modules into one unified framework, our model outperforms the state-of-the-arts on the ActivityNet Captions dataset with a relative gain of over 100% (Meteor score increases from 4.82 to 9.65).