X-Aligner: Composed Visual Retrieval without the Bells and Whistles

TL;DR

X-Aligner introduces a novel cross-attention based framework leveraging Vision Language Models for improved composed video retrieval, achieving state-of-the-art results and strong zero-shot generalization.

Contribution

The paper proposes X-Aligner, a new multimodal fusion framework with a progressive cross-attention module and two-stage training, enhancing video retrieval performance.

Findings

Achieves 63.93% Recall@1 on Webvid-CoVR-Test

Outperforms existing CoVR methods in retrieval accuracy

Demonstrates strong zero-shot generalization on CIR datasets

Abstract

Composed Video Retrieval (CoVR) facilitates video retrieval by combining visual and textual queries. However, existing CoVR frameworks typically fuse multimodal inputs in a single stage, achieving only marginal gains over initial baseline. To address this, we propose a novel CoVR framework that leverages the representational power of Vision Language Models (VLMs). Our framework incorporates a novel cross-attention module X-Aligner, composed of cross-attention layers that progressively fuse visual and textual inputs and align their multimodal representation with that of the target video. To further enhance the representation of the multimodal query, we incorporate the caption of the visual query as an additional input. The framework is trained in two stages to preserve the pretrained VLM representation. In the first stage, only the newly introduced module is trained, while in the second stage, the textual query encoder is also fine-tuned. We implement our framework on top of BLIP-family architecture, namely BLIP and BLIP-2, and train it on the Webvid-CoVR data set. In addition to in-domain evaluation on Webvid-CoVR-Test, we perform zero-shot evaluations on the Composed Image Retrieval (CIR) data sets CIRCO and Fashion-IQ. Our framework achieves state-of-the-art performance on CoVR obtaining a Recall@1 of 63.93% on Webvid-CoVR-Test, and demonstrates strong zero-shot generalization on CIR tasks.