SAVE: Speech-Aware Video Representation Learning for Video-Text Retrieval

TL;DR

SAVE introduces a speech-aware video representation learning method that enhances video-text retrieval by effectively incorporating speech and audio features, outperforming previous audiovisual models across multiple benchmarks.

Contribution

The paper proposes a novel speech-aware video representation learning approach with a dedicated speech branch and early vision-audio alignment, improving upon state-of-the-art audiovisual methods.

Findings

SAVE outperforms AVIGATE by +4.1% on MSRVTT-9k

SAVE improves retrieval metrics on five benchmarks

Effective speech embedding enhances video-text retrieval performance

Abstract

For video-text retrieval, the use of CLIP has been a de facto choice. Since CLIP provides only image and text encoders, this consensus has led to a biased paradigm that entirely ignores the sound track of videos. While several attempts have been made to reintroduce audio -- typically by incorporating an audio encoder and fusing its output with visual features -- these methods face two challenges: ineffective representation of speech content and suboptimal vision-audio fusion. To address these issues jointly, we propose SAVE, a Speech Aware Video rEpresentation learning method. SAVE improves upon AVIGATE, a SOTA audiovisual method, with a dedicated speech branch for more effective speech embedding. Furthermore, we introduce soft-ALBEF for early vision-audio alignment that facilitates fusion. Extensive experiments on five benchmarks show that SAVE compares favorably against the SOTA, outperforming AVIGATE by +4.1% on MSRVTT-9k, +1.9% on MSRVTT-7k, +2.5% on VATEX, +9.8% on Charades, and +2.1% on LSMDC, in light of the SumR metric.