Mediator-Guided Multi-Agent Collaboration among Open-Source Models for Medical Decision-Making

TL;DR

This paper introduces MedOrch, a mediator-guided multi-agent framework that enhances medical decision-making by enabling diverse vision-language models to collaborate via an LLM mediator, improving accuracy without additional training.

Contribution

The paper presents MedOrch, a novel framework that leverages an LLM mediator to facilitate effective collaboration among heterogeneous open-source VLMs for medical multimodal tasks.

Findings

Collaboration among VLM agents surpasses individual performance.

The approach achieves superior results on five medical VQA benchmarks.

No additional model training is required for improved performance.

Abstract

Complex medical decision-making involves cooperative workflows operated by different clinicians. Designing AI multi-agent systems can expedite and augment human-level clinical decision-making. Existing multi-agent researches primarily focus on language-only tasks, yet their extension to multimodal scenarios remains challenging. A blind combination of diverse vision-language models (VLMs) can amplify an erroneous outcome interpretation. VLMs in general are less capable in instruction following and importantly self-reflection, compared to large language models (LLMs) of comparable sizes. This disparity largely constrains VLMs' ability in cooperative workflows. In this study, we propose MedOrch, a mediator-guided multi-agent collaboration framework for medical multimodal decision-making. MedOrch employs an LLM-based mediator agent that enables multiple VLM-based expert agents to exchange and reflect on their outputs towards collaboration. We utilize multiple open-source general-purpose and domain-specific VLMs instead of costly GPT-series models, revealing the strength of heterogeneous models. We show that the collaboration within distinct VLM-based agents can surpass the capabilities of any individual agent. We validate our approach on five medical vision question answering benchmarks, demonstrating superior collaboration performance without model training. Our findings underscore the value of mediator-guided multi-agent collaboration in advancing medical multimodal intelligence.