Beyond Simple Edits: Composed Video Retrieval with Dense Modifications

TL;DR

This paper introduces Dense-WebVid-CoVR, a large dataset for composed video retrieval with dense modifications, and proposes a new model that achieves state-of-the-art results by integrating visual and textual information through Cross-Attention fusion.

Contribution

The paper presents a novel large-scale dataset for fine-grained composed video retrieval and a new model that effectively aligns dense textual modifications with target videos.

Findings

Achieved 71.3% Recall@1 in visual+text retrieval setting.

Dataset contains 1.6 million samples with dense modification texts.

Model outperforms existing methods on all evaluation metrics.

Abstract

Composed video retrieval is a challenging task that strives to retrieve a target video based on a query video and a textual description detailing specific modifications. Standard retrieval frameworks typically struggle to handle the complexity of fine-grained compositional queries and variations in temporal understanding limiting their retrieval ability in the fine-grained setting. To address this issue, we introduce a novel dataset that captures both fine-grained and composed actions across diverse video segments, enabling more detailed compositional changes in retrieved video content. The proposed dataset, named Dense-WebVid-CoVR, consists of 1.6 million samples with dense modification text that is around seven times more than its existing counterpart. We further develop a new model that integrates visual and textual information through Cross-Attention (CA) fusion using grounded text encoder, enabling precise alignment between dense query modifications and target videos. The proposed model achieves state-of-the-art results surpassing existing methods on all metrics. Notably, it achieves 71.3\% Recall@1 in visual+text setting and outperforms the state-of-the-art by 3.4\%, highlighting its efficacy in terms of leveraging detailed video descriptions and dense modification texts. Our proposed dataset, code, and model are available at :https://github.com/OmkarThawakar/BSE-CoVR