Mixture-of-Retrieval Experts for Reasoning-Guided Multimodal Knowledge Exploitation

TL;DR

The paper introduces MoRE, a framework enabling multimodal models to dynamically interact with multiple retrieval experts for improved knowledge exploitation and reasoning, demonstrated by significant performance gains.

Contribution

MoRE allows MLLMs to learn to selectively engage with diverse retrieval experts based on reasoning needs, enhancing knowledge utilization and reasoning accuracy.

Findings

Achieves over 7% performance improvement on QA benchmarks.

Demonstrates effective dynamic coordination of heterogeneous retrieval experts.

Validates robustness and adaptability in reasoning-driven knowledge retrieval.

Abstract

Multimodal Retrieval-Augmented Generation (MRAG) has shown promise in mitigating hallucinations in Multimodal Large Language Models (MLLMs) by incorporating external knowledge. However, existing methods typically adhere to rigid retrieval paradigms by mimicking fixed retrieval trajectories and thus fail to fully exploit the knowledge of different retrieval experts through dynamic interaction based on the model's knowledge needs or evolving reasoning states. To overcome this limitation, we introduce Mixture-of-Retrieval Experts (MoRE), a novel framework that enables MLLMs to collaboratively interact with diverse retrieval experts for more effective knowledge exploitation. Specifically, MoRE learns to dynamically determine which expert to engage with, conditioned on the evolving reasoning state. To effectively train this capability, we propose Stepwise Group Relative Policy Optimization (Step-GRPO), which goes beyond sparse outcome-based supervision by encouraging MLLMs to interact with multiple retrieval experts and synthesize fine-grained rewards, thereby teaching the MLLM to fully coordinate all experts when answering a given query. Experimental results on diverse open-domain QA benchmarks demonstrate the effectiveness of MoRE, achieving average performance gains of over 7% compared to competitive baselines. Notably, MoRE exhibits strong adaptability by dynamically coordinating heterogeneous experts to precisely locate relevant information, validating its capability for robust, reasoning-driven expert collaboration. All codes and data are released on https://github.com/OpenBMB/MoRE.