Progressively Guide to Attend: An Iterative Alignment Framework for Temporal Sentence Grounding

TL;DR

This paper introduces IA-Net, an iterative alignment framework for temporal sentence grounding that progressively refines vision-language feature alignment through multiple reasoning steps, improving accuracy over existing single-step methods.

Contribution

The paper proposes an iterative alignment network with multi-step reasoning, feature padding, and calibration modules to enhance vision-language alignment in TSG.

Findings

Outperforms state-of-the-art on three benchmarks.

Effectively captures fine-grained cross-modal relations.

Robustly refines temporal boundary predictions.

Abstract

A key solution to temporal sentence grounding (TSG) exists in how to learn effective alignment between vision and language features extracted from an untrimmed video and a sentence description. Existing methods mainly leverage vanilla soft attention to perform the alignment in a single-step process. However, such single-step attention is insufficient in practice, since complicated relations between inter- and intra-modality are usually obtained through multi-step reasoning. In this paper, we propose an Iterative Alignment Network (IA-Net) for TSG task, which iteratively interacts inter- and intra-modal features within multiple steps for more accurate grounding. Specifically, during the iterative reasoning process, we pad multi-modal features with learnable parameters to alleviate the nowhere-to-attend problem of non-matched frame-word pairs, and enhance the basic co-attention mechanism in a parallel manner. To further calibrate the misaligned attention caused by each reasoning step, we also devise a calibration module following each attention module to refine the alignment knowledge. With such iterative alignment scheme, our IA-Net can robustly capture the fine-grained relations between vision and language domains step-by-step for progressively reasoning the temporal boundaries. Extensive experiments conducted on three challenging benchmarks demonstrate that our proposed model performs better than the state-of-the-arts.