Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Loss

TL;DR

This paper introduces a multi-stream corpus alignment network with a dual softmax loss to improve video-text retrieval, effectively addressing heterogeneity issues and achieving state-of-the-art results on multiple benchmarks.

Contribution

It proposes CAMoE with Mixture-of-Experts for multi-perspective video representations and a novel Dual Softmax Loss to enhance similarity matching in video-text retrieval.

Findings

Achieves SOTA performance on MSR-VTT, MSVD, and LSMDC benchmarks.

Surpasses previous methods by around 4.6% R@1 on MSR-VTT.

Demonstrates the effectiveness of CAMoE and DSL individually and combined.

Abstract

Employing large-scale pre-trained model CLIP to conduct video-text retrieval task (VTR) has become a new trend, which exceeds previous VTR methods. Though, due to the heterogeneity of structures and contents between video and text, previous CLIP-based models are prone to overfitting in the training phase, resulting in relatively poor retrieval performance. In this paper, we propose a multi-stream Corpus Alignment network with single gate Mixture-of-Experts (CAMoE) and a novel Dual Softmax Loss (DSL) to solve the two heterogeneity. The CAMoE employs Mixture-of-Experts (MoE) to extract multi-perspective video representations, including action, entity, scene, etc., then align them with the corresponding part of the text. In this stage, we conduct massive explorations towards the feature extraction module and feature alignment module. DSL is proposed to avoid the one-way optimum-match which occurs in previous contrastive methods. Introducing the intrinsic prior of each pair in a batch, DSL serves as a reviser to correct the similarity matrix and achieves the dual optimal match. DSL is easy to implement with only one-line code but improves significantly. The results show that the proposed CAMoE and DSL are of strong efficiency, and each of them is capable of achieving State-of-The-Art (SOTA) individually on various benchmarks such as MSR-VTT, MSVD, and LSMDC. Further, with both of them, the performance is advanced to a big extend, surpassing the previous SOTA methods for around 4.6\% R@1 in MSR-VTT.