Do Mixed-Vendor Multi-Agent LLMs Improve Clinical Diagnosis?

TL;DR

This paper demonstrates that mixed-vendor multi-agent LLM systems significantly improve clinical diagnosis accuracy by leveraging diverse models to overcome shared biases, outperforming single-vendor approaches.

Contribution

It introduces and evaluates mixed-vendor multi-agent frameworks, showing they outperform single-vendor systems in clinical diagnosis tasks.

Findings

Mixed-vendor teams achieve higher recall and accuracy.

Vendor diversity pools complementary biases for better diagnosis.

Mixed-vendor configurations outperform state-of-the-art models.

Abstract

Multi-agent large language model (LLM) systems have emerged as a promising approach for clinical diagnosis, leveraging collaboration among agents to refine medical reasoning. However, most existing frameworks rely on single-vendor teams (e.g., multiple agents from the same model family), which risk correlated failure modes that reinforce shared biases rather than correcting them. We investigate the impact of vendor diversity by comparing Single-LLM, Single-Vendor, and Mixed-Vendor Multi-Agent Conversation (MAC) frameworks. Using three doctor agents instantiated with o4-mini, Gemini-2.5-Pro, and Claude-4.5-Sonnet, we evaluate performance on RareBench and DiagnosisArena. Mixed-vendor configurations consistently outperform single-vendor counterparts, achieving state-of-the-art recall and accuracy. Overlap analysis reveals the underlying mechanism: mixed-vendor teams pool complementary inductive biases, surfacing correct diagnoses that individual models or homogeneous teams collectively miss. These results highlight vendor diversity as a key design principle for robust clinical diagnostic systems.