A Multi-Agent System for Complex Reasoning in Radiology Visual Question Answering

TL;DR

This paper presents a multi-agent system for radiology visual question answering that enhances reasoning, accuracy, and interpretability in analyzing chest X-ray images, addressing current challenges in factual correctness and cross-modal alignment.

Contribution

The paper introduces a novel multi-agent architecture specifically designed for complex reasoning in RVQA, improving over existing multimodal large language model approaches.

Findings

Outperforms strong MLLM baselines in RVQA tasks

Demonstrates improved factual accuracy and interpretability

Effective in challenging, model-disagreement filtered cases

Abstract

Radiology visual question answering (RVQA) provides precise answers to questions about chest X-ray images, alleviating radiologists' workload. While recent methods based on multimodal large language models (MLLMs) and retrieval-augmented generation (RAG) have shown promising progress in RVQA, they still face challenges in factual accuracy, hallucinations, and cross-modal misalignment. We introduce a multi-agent system (MAS) designed to support complex reasoning in RVQA, with specialized agents for context understanding, multimodal reasoning, and answer validation. We evaluate our system on a challenging RVQA set curated via model disagreement filtering, comprising consistently hard cases across multiple MLLMs. Extensive experiments demonstrate the superiority and effectiveness of our system over strong MLLM baselines, with a case study illustrating its reliability and interpretability. This work highlights the potential of multi-agent approaches to support explainable and trustworthy clinical AI applications that require complex reasoning.