VeRVE: Versatile Retrieval for Videos via Unified Embeddings

TL;DR

VeRVE is a versatile, MLLM-based video retrieval framework that unifies corpus, moment, and multimodal query handling, surpassing existing methods in zero-shot and reranked retrieval tasks.

Contribution

Introduces VeRVE, a unified MLLM-based model trained with LoRA that achieves state-of-the-art zero-shot and reranked video retrieval performance.

Findings

Outperforms other MLLM-based methods on zero-shot video retrieval.

Achieves competitive zero-shot moment retrieval results.

State-of-the-art results for zero-shot composed video retrieval.

Abstract

Modern video retrieval systems are expected to handle diverse tasks ranging from corpus-level retrieval, fine-grained moment localization to flexible multimodal querying. Specialized architectures achieve strong retrieval performance by training modality-specific encoders on massive datasets, but they lack the ability to process composed multimodal queries. In contrast, multimodal LLM (MLLM)-based methods support rich multimodal search but their retrieval performance remains well below that of specialized systems. We present VeRVE, an MLLM-based versatile video retrieval framework that integrates corpus and moment-level retrieval capabilities while accommodating composed multimodal queries within a single architecture. We use contrastive alignment of visual and textual embeddings generated using a shared MLLM backbone to facilitate efficient embedding-based candidate search. Our embedding model, trained efficiently using low-rank adaptation (LoRA) on 700K paired visual-text data samples, surpasses other MLLM-based methods on zero-shot video retrieval tasks. Additionally, we demonstrate that the same model can be adapted without further training to achieve competitive results on zero-shot moment retrieval, and state of the art results for zero-shot composed video retrieval. With additional training for reranking candidates identified in the embedding-based search, our model substantially outperforms existing MLLM-based retrieval systems and achieves retrieval performance comparable to state of the art specialized models.