MedChat: A Multi-Agent Framework for Multimodal Diagnosis with Large Language Models

TL;DR

MedChat introduces a multi-agent system combining specialized vision models and role-specific language models to improve multimodal medical diagnosis, reduce hallucinations, and facilitate interactive clinical reporting.

Contribution

The paper presents MedChat, a novel multi-agent framework that integrates multiple specialized LLMs and vision models for more reliable and interpretable medical diagnosis.

Findings

Enhanced diagnostic accuracy with multi-agent collaboration

Reduced hallucination and increased interpretability

Interactive reporting tailored for clinical review

Abstract

The integration of deep learning-based glaucoma detection with large language models (LLMs) presents an automated strategy to mitigate ophthalmologist shortages and improve clinical reporting efficiency. However, applying general LLMs to medical imaging remains challenging due to hallucinations, limited interpretability, and insufficient domain-specific medical knowledge, which can potentially reduce clinical accuracy. Although recent approaches combining imaging models with LLM reasoning have improved reporting, they typically rely on a single generalist agent, restricting their capacity to emulate the diverse and complex reasoning found in multidisciplinary medical teams. To address these limitations, we propose MedChat, a multi-agent diagnostic framework and platform that combines specialized vision models with multiple role-specific LLM agents, all coordinated by a director agent. This design enhances reliability, reduces hallucination risk, and enables interactive diagnostic reporting through an interface tailored for clinical review and educational use. Code available at https://github.com/Purdue-M2/MedChat.