MedRoute: RL-Based Dynamic Specialist Routing in Multi-Agent Medical Diagnosis

TL;DR

MedRoute introduces a dynamic multi-agent framework with reinforcement learning to emulate clinical specialist collaboration, improving diagnostic accuracy over static approaches in multimodal medical diagnosis.

Contribution

This work presents MedRoute, a flexible multi-agent system with RL-based specialist routing and a moderator, closely mirroring real clinical workflows.

Findings

Outperforms state-of-the-art baselines in diagnostic accuracy

Demonstrates effectiveness on text and image-based medical datasets

Validates the adaptability of dynamic specialist selection

Abstract

Medical diagnosis using Large Multimodal Models (LMMs) has gained increasing attention due to capability of these models in providing precise diagnoses. These models generally combine medical questions with visual inputs to generate diagnoses or treatments. However, they are often overly general and unsuitable under the wide range of medical conditions in real-world healthcare. In clinical practice, diagnosis is performed by multiple specialists, each contributing domain-specific expertise. To emulate this process, a potential solution is to deploy a dynamic multi-agent LMM framework, where each agent functions as a medical specialist. Current approaches in this emerging area, typically relying on static or predefined selection of various specialists, cannot be adapted to the changing practical scenario. In this paper, we propose MedRoute, a flexible and dynamic multi-agent framework that comprises of a collaborative system of specialist LMM agents. Furthermore, we add a General Practitioner with an RL-trained router for dynamic specialist selection, and a Moderator that produces the final decision. In this way, our framework closely mirrors real clinical workflows. Extensive evaluations on text and image-based medical datasets demonstrate improved diagnostic accuracy, outperforming the state-of-the-art baselines. Our work lays a strong foundation for future research. Code and models are available at https://github.com/UCF-CRCV/MedRoute/.