An Empirical Study of Excitation and Aggregation Design Adaptions in CLIP4Clip for Video-Text Retrieval

TL;DR

This paper introduces novel excitation and aggregation modules to improve video-text retrieval performance in CLIP4Clip models by generating more discriminative video representations, outperforming existing methods on benchmark datasets.

Contribution

The paper proposes new excitation and aggregation design modules that enhance frame feature representation in CLIP4Clip for better video-text retrieval accuracy.

Findings

Achieved state-of-the-art results on MSR-VTT, ActivityNet, and DiDeMo datasets.

Outperformed baseline CLIP4Clip by up to 9.5% in R@1 metric.

Demonstrated the effectiveness of excitation and aggregation modules in discriminative video representation.

Abstract

CLIP4Clip model transferred from the CLIP has been the de-factor standard to solve the video clip retrieval task from frame-level input, triggering the surge of CLIP4Clip-based models in the video-text retrieval domain. In this work, we rethink the inherent limitation of widely-used mean pooling operation in the frame features aggregation and investigate the adaptions of excitation and aggregation design for discriminative video representation generation. We present a novel excitationand-aggregation design, including (1) The excitation module is available for capturing non-mutuallyexclusive relationships among frame features and achieving frame-wise features recalibration, and (2) The aggregation module is applied to learn exclusiveness used for frame representations aggregation. Similarly, we employ the cascade of sequential module and aggregation design to generate discriminative video representation in the sequential type. Besides, we adopt the excitation design in the tight type to obtain representative frame features for multi-modal interaction. The proposed modules are evaluated on three benchmark datasets of MSR-VTT, ActivityNet and DiDeMo, achieving MSR-VTT (43.9 R@1), ActivityNet (44.1 R@1) and DiDeMo (31.0 R@1). They outperform the CLIP4Clip results by +1.2% (+0.5%), +4.5% (+1.9%) and +9.5% (+2.7%) relative (absolute) improvements, demonstrating the superiority of our proposed excitation and aggregation designs. We hope our work will serve as an alternative for frame representations aggregation and facilitate future research.