CVA: Context-aware Video-text Alignment for Video Temporal Grounding

TL;DR

CVA is a comprehensive framework that improves video-text alignment for temporal grounding by combining novel data augmentation, contrastive loss, and a hierarchical transformer architecture, achieving state-of-the-art results.

Contribution

The paper introduces three innovative components—QCD, CBD loss, and CTE—that together enhance the robustness and accuracy of video-text temporal grounding.

Findings

Achieves approximately 5-point improvement in Recall@1 scores.

Outperforms existing methods on QVHighlights and Charades-STA benchmarks.

Demonstrates robustness to irrelevant background context.

Abstract

We propose Context-aware Video-text Alignment (CVA), a novel framework to address a significant challenge in video temporal grounding: achieving temporally sensitive video-text alignment that remains robust to irrelevant background context. Our framework is built on three key components. First, we propose Query-aware Context Diversification (QCD), a new data augmentation strategy that ensures only semantically unrelated content is mixed in. It builds a video-text similarity-based pool of replacement clips to simulate diverse contexts while preventing the ``false negative" caused by query-agnostic mixing. Second, we introduce the Context-invariant Boundary Discrimination (CBD) loss, a contrastive loss that enforces semantic consistency at challenging temporal boundaries, making their representations robust to contextual shifts and hard negatives. Third, we introduce the Context-enhanced Transformer Encoder (CTE), a hierarchical architecture that combines windowed self-attention and bidirectional cross-attention with learnable queries to capture multi-scale temporal context. Through the synergy of these data-centric and architectural enhancements, CVA achieves state-of-the-art performance on major VTG benchmarks, including QVHighlights and Charades-STA. Notably, our method achieves a significant improvement of approximately 5 points in Recall@1 (R1) scores over state-of-the-art methods, highlighting its effectiveness in mitigating false negatives.