Unifying Event Detection and Captioning as Sequence Generation via Pre-Training

TL;DR

This paper introduces a unified sequence generation framework for dense video captioning that combines event detection and captioning, leveraging pre-training to improve inter-task association and temporal consistency.

Contribution

It proposes a novel unified pre-training and fine-tuning approach that models event detection as sequence generation, enhancing inter-task interaction and temporal dependency modeling.

Findings

Outperforms state-of-the-art on ActivityNet dataset

Benefits from large-scale video-text pre-training

Detects more diverse and consistent events

Abstract

Dense video captioning aims to generate corresponding text descriptions for a series of events in the untrimmed video, which can be divided into two sub-tasks, event detection and event captioning. Unlike previous works that tackle the two sub-tasks separately, recent works have focused on enhancing the inter-task association between the two sub-tasks. However, designing inter-task interactions for event detection and captioning is not trivial due to the large differences in their task specific solutions. Besides, previous event detection methods normally ignore temporal dependencies between events, leading to event redundancy or inconsistency problems. To tackle above the two defects, in this paper, we define event detection as a sequence generation task and propose a unified pre-training and fine-tuning framework to naturally enhance the inter-task association between event detection and captioning. Since the model predicts each event with previous events as context, the inter-dependency between events is fully exploited and thus our model can detect more diverse and consistent events in the video. Experiments on the ActivityNet dataset show that our model outperforms the state-of-the-art methods, and can be further boosted when pre-trained on extra large-scale video-text data. Code is available at \url{https://github.com/QiQAng/UEDVC}.