Exploring Temporal Event Cues for Dense Video Captioning in Cyclic Co-learning

TL;DR

This paper introduces MCCL, a dense video captioning model that uses cyclic co-learning and concept detection to improve event localization and description in untrimmed videos, achieving state-of-the-art results.

Contribution

The paper proposes a novel cyclic co-learning framework that integrates weakly supervised concept detection with captioning for enhanced dense video captioning.

Findings

Achieves state-of-the-art performance on ActivityNet Captions.

Demonstrates effectiveness of cyclic co-learning in video captioning.

Improves semantic perception and event localization accuracy.

Abstract

Dense video captioning aims to detect and describe all events in untrimmed videos. This paper presents a dense video captioning network called Multi-Concept Cyclic Learning (MCCL), which aims to: (1) detect multiple concepts at the frame level, using these concepts to enhance video features and provide temporal event cues; and (2) design cyclic co-learning between the generator and the localizer within the captioning network to promote semantic perception and event localization. Specifically, we perform weakly supervised concept detection for each frame, and the detected concept embeddings are integrated into the video features to provide event cues. Additionally, video-level concept contrastive learning is introduced to obtain more discriminative concept embeddings. In the captioning network, we establish a cyclic co-learning strategy where the generator guides the localizer for event localization through semantic matching, while the localizer enhances the generator's event semantic perception through location matching, making semantic perception and event localization mutually beneficial. MCCL achieves state-of-the-art performance on the ActivityNet Captions and YouCook2 datasets. Extensive experiments demonstrate its effectiveness and interpretability.