ViLL-E: Video LLM Embeddings for Retrieval

TL;DR

ViLL-E is a unified VideoLLM architecture that enhances retrieval tasks by combining generative and contrastive training, achieving state-of-the-art performance and zero-shot capabilities in video understanding.

Contribution

Introduces ViLL-E, a novel VideoLLM with a unique embedding mechanism and a three-stage training process for improved retrieval and localization.

Findings

Improves temporal localization by an average of 7% over other VideoLLMs.

Enhances video retrieval accuracy up to 4% over dual encoder models.

Achieves state-of-the-art zero-shot performance in composed video retrieval and long text retrieval.

Abstract

Video Large Language Models (VideoLLMs) excel at video understanding tasks where outputs are textual, such as Video Question Answering and Video Captioning. However, they underperform specialized embedding-based models in Retrieval tasks, such as Text-toVideo Retrieval and Moment Retrieval. We introduce ViLL-E (Video-LLM-Embed), a unified VideoLLM architecture endowed with a novel embedding generation mechanism that allows the model to "think longer" for complex videos and stop early for easy ones. We train this model with a three-stage training methodology combining generative and contrastive learning: initial large-scale pre-training with video-caption pairs; followed by continual training on a smaller, detailed-caption dataset; and concluding with task-specific fine-tuning on a novel multi-task dataset covering Video QA, Temporal Localization, Video Retrieval, and Video-Text Matching. Our model significantly improves temporal localization (on avg. 7% over other VideoLLMs) and video retrieval (up to 4% over dual encoder models), achieving performance comparable to state-of-the-art specialized embedding models while remaining competitive on VideoQA tasks. Furthermore, our joint contrastive-generative training unlocks new zero-shot capabilities, significantly outperforming state-of-the-art methods in composed video retrieval (+5% over SotA) and retrieval from long text (+2% over SotA).