Q2E: Query-to-Event Decomposition for Zero-Shot Multilingual Text-to-Video Retrieval

TL;DR

Q2E introduces a zero-shot, multilingual method that decomposes complex queries to improve text-to-video retrieval by leveraging parametric knowledge from LLMs and VLMs, enhancing accuracy across modalities.

Contribution

This work presents a novel query decomposition approach that improves zero-shot multilingual text-to-video retrieval by integrating multimodal knowledge and entropy-based fusion.

Findings

Q2E outperforms state-of-the-art baselines on diverse datasets.

Decomposing queries improves understanding of complex real-world events.

Audio integration significantly boosts retrieval performance.

Abstract

Recent approaches have shown impressive proficiency in extracting and leveraging parametric knowledge from Large-Language Models (LLMs) and Vision-Language Models (VLMs). In this work, we consider how we can improve the identification and retrieval of videos related to complex real-world events by automatically extracting latent parametric knowledge about those events. We present Q2E: a Query-to-Event decomposition method for zero-shot multilingual text-to-video retrieval, adaptable across datasets, domains, LLMs, or VLMs. Our approach demonstrates that we can enhance the understanding of otherwise overly simplified human queries by decomposing the query using the knowledge embedded in LLMs and VLMs. We additionally show how to apply our approach to both visual and speech-based inputs. To combine this varied multimodal knowledge, we adopt entropy-based fusion scoring for zero-shot fusion. Through evaluations on two diverse datasets and multiple retrieval metrics, we demonstrate that Q2E outperforms several state-of-the-art baselines. Our evaluation also shows that integrating audio information can significantly improve text-to-video retrieval. We have released code and data for future research.