Bidirectional Likelihood Estimation with Multi-Modal Large Language Models for Text-Video Retrieval

TL;DR

This paper introduces BLiM, a bidirectional likelihood estimation framework with candidate prior normalization for improved text-video retrieval, effectively reducing bias and enhancing relevance detection in large-scale multi-modal datasets.

Contribution

The paper proposes a novel bidirectional likelihood estimation method with a training-free candidate prior normalization to mitigate bias in multi-modal large language model-based retrieval.

Findings

BLiM outperforms previous models by 6.4 R@1 on average across benchmarks.

Candidate Prior Normalization effectively reduces candidate prior bias.

The approach enhances relevance detection in various multi-modal tasks.

Abstract

Text-Video Retrieval aims to find the most relevant text (or video) candidate given a video (or text) query from large-scale online databases. Recent work leverages multi-modal large language models (MLLMs) to improve retrieval, especially for long or complex query-candidate pairs. However, we observe that the naive application of MLLMs, i.e., retrieval based on candidate likelihood, introduces candidate prior bias, favoring candidates with inherently higher priors over those more relevant to the query. To this end, we propose a novel retrieval framework, Bidirectional Likelihood Estimation with MLLM (BLiM), which leverages both query and candidate likelihoods by training the model to generate text from a given video as well as video features from a given text. Furthermore, we introduce Candidate Prior Normalization (CPN), a simple yet effective training-free score calibration module designed to mitigate candidate prior bias in candidate likelihood. On four Text-Video Retrieval benchmarks, our BLiM equipped with CPN outperforms previous state-of-the-art models by 6.4 R@1 on average, effectively alleviating candidate prior bias and emphasizing query-candidate relevance. Our in-depth analysis across various multi-modal tasks beyond retrieval highlights the broad applicability of CPN which enhances visual understanding by reducing reliance on textual priors. Code is available at https://github.com/mlvlab/BLiM.