CoVA: Text-Guided Composed Video Retrieval for Audio-Visual Content

TL;DR

This paper introduces CoVA, a new task and benchmark for retrieving videos based on combined visual and audio modifications specified by text, addressing limitations of existing visual-only retrieval methods.

Contribution

It proposes AVT, a novel multimodal fusion method, and AV-Comp, a benchmark dataset for audio-visual composed video retrieval, advancing multimodal retrieval research.

Findings

AVT outperforms unimodal fusion baselines.

AV-Comp enables evaluation of audio-visual retrieval.

The dataset includes cross-modal change queries.

Abstract

Composed Video Retrieval (CoVR) aims to retrieve a target video from a large gallery using a reference video and a textual query specifying visual modifications. However, existing benchmarks consider only visual changes, ignoring videos that differ in audio despite visual similarity. To address this limitation, we introduce Composed retrieval for Video with its Audio CoVA, a new retrieval task that accounts for both visual and auditory variations. To support this, we construct AV-Comp, a benchmark consisting of video pairs with cross-modal changes and corresponding textual queries that describe the differences. We also propose AVT Compositional Fusion (AVT), which integrates video, audio, and text features by selectively aligning the query to the most relevant modality. AVT outperforms traditional unimodal fusion and serves as a strong baseline for CoVA. Examples from the proposed dataset, including both visual and auditory information, are available at https://perceptualai-lab.github.io/CoVA/.