GEM-VPC: A dual Graph-Enhanced Multimodal integration for Video Paragraph Captioning

TL;DR

This paper presents GEM-VPC, a novel multimodal graph-enhanced framework for video paragraph captioning that effectively captures key events and themes, improving performance on benchmark datasets.

Contribution

It introduces a dual graph structure and a node selection module to better utilize multimodal signals and external knowledge in video captioning.

Findings

Achieves superior performance on benchmark datasets.

Effectively models key events and themes in videos.

Enhances decoding efficiency with node selection.

Abstract

Video Paragraph Captioning (VPC) aims to generate paragraph captions that summarises key events within a video. Despite recent advancements, challenges persist, notably in effectively utilising multimodal signals inherent in videos and addressing the long-tail distribution of words. The paper introduces a novel multimodal integrated caption generation framework for VPC that leverages information from various modalities and external knowledge bases. Our framework constructs two graphs: a 'video-specific' temporal graph capturing major events and interactions between multimodal information and commonsense knowledge, and a 'theme graph' representing correlations between words of a specific theme. These graphs serve as input for a transformer network with a shared encoder-decoder architecture. We also introduce a node selection module to enhance decoding efficiency by selecting the most relevant nodes from the graphs. Our results demonstrate superior performance across benchmark datasets.