Discriminative Latent Semantic Graph for Video Captioning

TL;DR

This paper introduces a novel framework for video captioning that enhances object proposals, extracts high-level visual semantics, and validates generated sentences, leading to significant improvements over existing methods.

Contribution

It proposes a joint framework with a Conditional Graph, Latent Proposal Aggregation, and a Discriminative Language Validator for improved video captioning.

Findings

Significant improvements on MVSD and MSR-VTT datasets.

Enhanced BLEU-4 and CIDEr scores.

Effective preservation of key semantic concepts.

Abstract

Video captioning aims to automatically generate natural language sentences that can describe the visual contents of a given video. Existing generative models like encoder-decoder frameworks cannot explicitly explore the object-level interactions and frame-level information from complex spatio-temporal data to generate semantic-rich captions. Our main contribution is to identify three key problems in a joint framework for future video summarization tasks. 1) Enhanced Object Proposal: we propose a novel Conditional Graph that can fuse spatio-temporal information into latent object proposal. 2) Visual Knowledge: Latent Proposal Aggregation is proposed to dynamically extract visual words with higher semantic levels. 3) Sentence Validation: A novel Discriminative Language Validator is proposed to verify generated captions so that key semantic concepts can be effectively preserved. Our experiments on two public datasets (MVSD and MSR-VTT) manifest significant improvements over state-of-the-art approaches on all metrics, especially for BLEU-4 and CIDEr. Our code is available at https://github.com/baiyang4/D-LSG-Video-Caption.