Multi-Granularity and Multi-modal Feature Interaction Approach for Text Video Retrieval

TL;DR

This paper introduces a multi-granularity and multi-modal feature interaction approach for text-video retrieval, leveraging detailed text and audio features to improve alignment and retrieval accuracy.

Contribution

It proposes novel MGFI and CMFI modules that enhance video-text and audio-text feature interactions, addressing limitations of existing methods.

Findings

Outperforms state-of-the-art on MSR-VTT, MSVD, DiDeMo datasets.

Effectively utilizes multi-granularity text and audio features.

Improves alignment accuracy between video and text representations.

Abstract

The key of the text-to-video retrieval (TVR) task lies in learning the unique similarity between each pair of text (consisting of words) and video (consisting of audio and image frames) representations. However, some problems exist in the representation alignment of video and text, such as a text, and further each word, are of different importance for video frames. Besides, audio usually carries additional or critical information for TVR in the case that frames carry little valid information. Therefore, in TVR task, multi-granularity representation of text, including whole sentence and every word, and the modal of audio are salutary which are underutilized in most existing works. To address this, we propose a novel multi-granularity feature interaction module called MGFI, consisting of text-frame and word-frame, for video-text representations alignment. Moreover, we introduce a cross-modal feature interaction module of audio and text called CMFI to solve the problem of insufficient expression of frames in the video. Experiments on benchmark datasets such as MSR-VTT, MSVD, DiDeMo show that the proposed method outperforms the existing state-of-the-art methods.