Renmin University of China at TRECVID 2022: Improving Video Search by Feature Fusion and Negation Understanding

TL;DR

This paper presents novel techniques for video search enhancement, including feature fusion and negation understanding, achieving top-tier results in TRECVID 2022 by combining diverse features and addressing negation cues effectively.

Contribution

Introduces Lightweight Attentional Feature Fusion and Bidirectional Negation Learning for improved video retrieval, leveraging diverse features and negation cues in a unified framework.

Findings

Achieved second place in TRECVID 2022 AVS with infAP of 0.262.

LAFF outperforms multi-head self-attention in feature fusion.

BNL effectively models negation cues in video search.

Abstract

We summarize our TRECVID 2022 Ad-hoc Video Search (AVS) experiments. Our solution is built with two new techniques, namely Lightweight Attentional Feature Fusion (LAFF) for combining diverse visual / textual features and Bidirectional Negation Learning (BNL) for addressing queries that contain negation cues. In particular, LAFF performs feature fusion at both early and late stages and at both text and video ends to exploit diverse (off-the-shelf) features. Compared to multi-head self attention, LAFF is much more compact yet more effective. Its attentional weights can also be used for selecting fewer features, with the retrieval performance mostly preserved. BNL trains a negation-aware video retrieval model by minimizing a bidirectionally constrained loss per triplet, where a triplet consists of a given training video, its original description and a partially negated description. For video feature extraction, we use pre-trained CLIP, BLIP, BEiT, ResNeXt-101 and irCSN. As for text features, we adopt bag-of-words, word2vec, CLIP and BLIP. Our training data consists of MSR-VTT, TGIF and VATEX that were used in our previous participation. In addition, we automatically caption the V3C1 collection for pre-training. The 2022 edition of the TRECVID benchmark has again been a fruitful participation for the RUCMM team. Our best run, with an infAP of 0.262, is ranked at the second place teamwise.