EagleNet: Energy-Aware Fine-Grained Relationship Learning Network for Text-Video Retrieval

TL;DR

EagleNet introduces a novel energy-aware, fine-grained relationship learning network that enhances text-video retrieval by capturing frame contextual information and improving cross-modal alignment.

Contribution

The paper proposes a new FRL mechanism and EAM to generate context-aware text embeddings and model interaction energy, advancing text-video retrieval accuracy.

Findings

EagleNet outperforms existing methods on MSRVTT, DiDeMo, MSVD, and VATEX datasets.

The energy-aware matching improves the modeling of real text-video pair distributions.

Replacing softmax contrastive loss with sigmoid loss stabilizes training and enhances performance.

Abstract

Text-video retrieval tasks have seen significant improvements due to the recent development of large-scale vision-language pre-trained models. Traditional methods primarily focus on video representations or cross-modal alignment, while recent works shift toward enriching text expressiveness to better match the rich semantics in videos. However, these methods use only interactions between text and frames/video, and ignore rich interactions among the internal frames within a video, so the final expanded text cannot capture frame contextual information, leading to disparities between text and video. In response, we introduce Energy-Aware Fine-Grained Relationship Learning Network (EagleNet) to generate accurate and context-aware enriched text embeddings. Specifically, the proposed Fine-Grained Relationship Learning mechanism (FRL) first constructs a text-frame graph by the generated text candidates and frames, then learns relationships among texts and frames, which are finally used to aggregate text candidates into an enriched text embedding that incorporates frame contextual information. To further improve fine-grained relationship learning in FRL, we design Energy-Aware Matching (EAM) to model the energy of text-frame interactions and thus accurately capture the distribution of real text-video pairs. Moreover, for more effective cross-modal alignment and stable training, we replace the conventional softmax-based contrastive loss with the sigmoid loss. Extensive experiments have demonstrated the superiority of EagleNet across MSRVTT, DiDeMo, MSVD, and VATEX. Codes are available at https://github.com/draym28/EagleNet.