Windsock is Dancing: Adaptive Multimodal Retrieval-Augmented Generation

TL;DR

Windsock introduces adaptive retrieval and modality selection strategies for multimodal LLMs, significantly improving response quality and efficiency by dynamically deciding when and what to retrieve and how to utilize external knowledge.

Contribution

The paper presents Windsock, a novel query-dependent module for adaptive retrieval and modality selection, along with DANCE instruction tuning for improved knowledge utilization in MLLMs.

Findings

Response quality improved by 17.07%.

Retrieval times reduced by 8.95%.

Enhanced robustness against noise.

Abstract

Multimodal Retrieval-Augmented Generation (MRAG) has emerged as a promising method to generate factual and up-to-date responses of Multimodal Large Language Models (MLLMs) by incorporating non-parametric knowledge from external knowledge bases. However, existing MRAG approaches suffer from static retrieval strategies, inflexible modality selection, and suboptimal utilization of retrieved information, leading to three critical challenges: determining when to retrieve, what modality to incorporate, and how to utilize retrieved information effectively. To address these challenges, we introduce Windsock, a query-dependent module making decisions on retrieval necessity and modality selection, effectively reducing computational overhead and improving response quality. Additionally, we propose Dynamic Noise-Resistance (DANCE) Instruction Tuning, an adaptive training strategy that enhances MLLMs' ability to utilize retrieved information while maintaining robustness against noise. Moreover, we adopt a self-assessment approach leveraging knowledge within MLLMs to convert question-answering datasets to MRAG training datasets. Extensive experiments demonstrate that our proposed method significantly improves the generation quality by 17.07% while reducing 8.95% retrieval times.