Learning Partially-Decorrelated Common Spaces for Ad-hoc Video Search

TL;DR

This paper introduces LPD, a novel method for ad-hoc video search that learns multiple partially decorrelated feature spaces to better capture visual diversity and improve retrieval accuracy.

Contribution

LPD proposes feature-specific common spaces with de-correlation loss and a fair multi-space triplet loss, advancing the diversity and effectiveness of video search models.

Findings

LPD outperforms existing methods on TRECVID AVS benchmarks.

LPD enhances diversity in search results.

Visualizations show increased feature space separation.

Abstract

Ad-hoc Video Search (AVS) involves using a textual query to search for multiple relevant videos in a large collection of unlabeled short videos. The main challenge of AVS is the visual diversity of relevant videos. A simple query such as "Find shots of a man and a woman dancing together indoors" can span a multitude of environments, from brightly lit halls and shadowy bars to dance scenes in black-and-white animations. It is therefore essential to retrieve relevant videos as comprehensively as possible. Current solutions for the AVS task primarily fuse multiple features into one or more common spaces, yet overlook the need for diverse spaces. To fully exploit the expressive capability of individual features, we propose LPD, short for Learning Partially Decorrelated common spaces. LPD incorporates two key innovations: feature-specific common space construction and the de-correlation loss. Specifically, LPD learns a separate common space for each video and text feature, and employs de-correlation loss to diversify the ordering of negative samples across different spaces. To enhance the consistency of multi-space convergence, we designed an entropy-based fair multi-space triplet ranking loss. Extensive experiments on the TRECVID AVS benchmarks (2016-2023) justify the effectiveness of LPD. Moreover, diversity visualizations of LPD's spaces highlight its ability to enhance result diversity.