GAIS: Frame-Level Gated Audio-Visual Integration with Semantic Variance-Scaled Perturbation for Text-Video Retrieval

TL;DR

GAIS introduces a novel audio-visual retrieval framework with frame-level gating and semantic-aware regularization, significantly improving text-video retrieval accuracy by better aligning multimodal features.

Contribution

The paper proposes GAIS, a new framework combining fine-grained temporal fusion and semantic variance-scaled perturbation for enhanced multimodal alignment in text-video retrieval.

Findings

Outperforms strong baselines on multiple datasets.

Achieves better retrieval metrics with efficient computation.

Enhances multimodal representation quality.

Abstract

Text-to-video retrieval requires precise alignment between language and temporally rich audio-video signals. However, existing methods often emphasize visual cues while underutilizing audio semantics or relying on coarse fusion strategies, resulting in suboptimal multimodal representations. We introduce GAIS, a retrieval framework that strengthens multimodal alignment from both representation and regularization perspectives. First, a Frame-level Gated Fusion (FGF) module adaptively integrates audio-visual features under textual guidance, enabling fine-grained temporal selection of informative frames. Second, a Semantic Variance-Scaled Perturbation (SVSP) mechanism regularizes the text embedding space by controlling perturbation magnitude in a semantics-aware manner. These two modules are complementary: FGF minimizes modality gaps through selective fusion, while SVSP improves embedding stability and discrimination. Extensive experiments on MSR-VTT, DiDeMo, LSMDC, and VATEX demonstrate that GAIS consistently outperforms strong baselines across multiple retrieval metrics while maintaining notable computational efficiency.