Support-set based Multi-modal Representation Enhancement for Video Captioning

TL;DR

This paper introduces the SMRE framework for video captioning, leveraging support-sets and semantic space transformations to enhance multi-modal representations and improve descriptive accuracy.

Contribution

The novel SMRE model effectively mines semantic associations between samples, surpassing existing methods in video captioning performance.

Findings

Achieves state-of-the-art results on MSVD and MSR-VTT datasets.

Effectively models semantic relationships between video samples.

Enhances multi-modal interaction through self-supervised learning.

Abstract

Video captioning is a challenging task that necessitates a thorough comprehension of visual scenes. Existing methods follow a typical one-to-one mapping, which concentrates on a limited sample space while ignoring the intrinsic semantic associations between samples, resulting in rigid and uninformative expressions. To address this issue, we propose a novel and flexible framework, namely Support-set based Multi-modal Representation Enhancement (SMRE) model, to mine rich information in a semantic subspace shared between samples. Specifically, we propose a Support-set Construction (SC) module to construct a support-set to learn underlying connections between samples and obtain semantic-related visual elements. During this process, we design a Semantic Space Transformation (SST) module to constrain relative distance and administrate multi-modal interactions in a self-supervised way. Extensive experiments on MSVD and MSR-VTT datasets demonstrate that our SMRE achieves state-of-the-art performance.