Rebalancing Contrastive Alignment with Bottlenecked Semantic Increments in Text-Video Retrieval

TL;DR

This paper introduces GARE, a novel framework for text-video retrieval that mitigates optimization tension caused by modality gaps and noisy negatives, leading to improved alignment accuracy and robustness.

Contribution

GARE employs a learnable, pair-specific increment guided by a Taylor expansion and a semantic gap-conditioned neural module, enhancing contrastive learning in text-video retrieval.

Findings

Consistently improves alignment accuracy across four benchmarks.

Enhances robustness against noisy negatives and modality gaps.

Demonstrates effectiveness of gap-aware tension mitigation.

Abstract

Recent progress in text-video retrieval has been largely driven by contrastive learning. However, existing methods often overlook the effect of the modality gap, which causes anchor representations to undergo in-place optimization (i.e., optimization tension) that limits their alignment capacity. Moreover, noisy hard negatives further distort the semantics of anchors. To address these issues, we propose GARE, a Gap-Aware Retrieval framework that introduces a learnable, pair-specific increment $\Delta_{ij}$ between text $t_i$ and video $v_j$, redistributing gradients to relieve optimization tension and absorb noise. We derive $\Delta_{ij}$ via a multivariate first-order Taylor expansion of the InfoNCE loss under a trust-region constraint, showing that it guides updates along locally consistent descent directions. A lightweight neural module conditioned on the semantic gap couples increments across batches for structure-aware correction. Furthermore, we regularize $\Delta$ through a variational information bottleneck with relaxed compression, enhancing stability and semantic consistency. Experiments on four benchmarks demonstrate that GARE consistently improves alignment accuracy and robustness, validating the effectiveness of gap-aware tension mitigation. Code is available at https://github.com/musicman217/GARE-text-video-retrieval.