PIC 4th Challenge: Semantic-Assisted Multi-Feature Encoding and Multi-Head Decoding for Dense Video Captioning

TL;DR

This paper introduces a semantic-assisted encoding-decoding framework for dense video captioning that leverages concept detection and semantic supervision to improve localization and caption quality, achieving state-of-the-art results.

Contribution

The novel integration of semantic concept detection and supervision into the dense video captioning framework enhances performance over previous methods.

Findings

Significant improvements on YouMakeup dataset

High performance in PIC 4th Challenge MDVC task

Effective fusion of semantic and visual features

Abstract

The task of Dense Video Captioning (DVC) aims to generate captions with timestamps for multiple events in one video. Semantic information plays an important role for both localization and description of DVC. We present a semantic-assisted dense video captioning model based on the encoding-decoding framework. In the encoding stage, we design a concept detector to extract semantic information, which is then fused with multi-modal visual features to sufficiently represent the input video. In the decoding stage, we design a classification head, paralleled with the localization and captioning heads, to provide semantic supervision. Our method achieves significant improvements on the YouMakeup dataset under DVC evaluation metrics and achieves high performance in the Makeup Dense Video Captioning (MDVC) task of PIC 4th Challenge.