RETLLM: Training and Data-Free MLLMs for Multimodal Information Retrieval

TL;DR

RETLLM introduces a training- and data-free framework for multimodal information retrieval using large language models, leveraging prompting and a coarse-then-fine scoring pipeline to outperform fine-tuned models.

Contribution

This work presents a novel, training- and data-free approach to MMIR that utilizes prompt-based scoring with a coarse-then-fine pipeline and visual enhancement modules.

Findings

RETLLM outperforms fine-tuned models on MMIR benchmarks.

The coarse-then-fine scoring pipeline improves retrieval accuracy.

Visual enhancement during reasoning boosts multimodal retrieval performance.

Abstract

Multimodal information retrieval (MMIR) has gained attention for its flexibility in handling text, images, or mixed queries and candidates. Recent breakthroughs in multimodal large language models (MLLMs) boost MMIR performance by incorporating MLLM knowledge under the contrastive finetuning framework. However, they suffer from pre-training inconsistency and require large datasets. In this work, we introduce a novel framework, RetLLM, designed to query MLLMs for MMIR in a training- and data-free manner. Specifically, we formulate MMIR as a similarity score generation task and prompt MLLMs to directly predict retrieval scores in a coarse-then-fine pipeline. At the coarse stage, a top-k filtering strategy builds a small yet high-quality candidate pool for each query, enabling MLLMs to focus on semantically relevant candidates. Subsequently, the retrieval score is predicted by feeding both the query and candidate into MLLMs at the fine stage. Importantly, we propose a visual enhancement module during reasoning to help MLLMs re-pick forgotten visuals, improving retrieval. Extensive experiments on MMIR benchmarks show that RetLLM outperforms fine-tuned models. Ablation studies further verify each component. Our work demonstrates that MLLMs can achieve strong MMIR performance without any training, highlighting their inherent multimodal reasoning ability in a simple, scalable framework. We release our code at: https://github.com/alivecat05/RETLLM