EyeAgent: An Agentic AI System for Multimodal Clinical Decision Support in Ophthalmology

TL;DR

EyeAgent is a novel agentic AI system that integrates multimodal ophthalmic tools and large language models to enhance diagnostic accuracy, interpretability, and collaboration in ophthalmology clinical decision support.

Contribution

It introduces the first agentic AI framework for ophthalmology that dynamically orchestrates multiple tools and reasoning engines for comprehensive clinical decision support.

Findings

Diagnostic accuracy improved from 69.71% to 80.79% with full tool integration.

Achieved 93.7% tool selection accuracy in real-world cases.

Enhanced diagnostic and report quality scores by over 18% with AI assistance.

Abstract

Artificial intelligence has shown promise in medical imaging, yet most existing systems lack flexibility, interpretability, and adaptability - challenges especially pronounced in ophthalmology, where diverse imaging modalities are essential. We present EyeAgent, the first agentic AI framework for comprehensive and interpretable clinical decision support in ophthalmology. Using a large language model (DeepSeek-V3) as its central reasoning engine, EyeAgent interprets user queries and dynamically orchestrates 53 validated ophthalmic tools across 23 imaging modalities for diverse tasks including classification, segmentation, detection, image/report generation, and quantitative analysis. Stepwise ablation analysis demonstrated a progressive improvement in diagnostic accuracy, rising from a baseline of 69.71% (using only 5 general tools) to 80.79% when the full suite of 53 specialized tools was integrated. In an expert rating study on 200 real-world clinical cases, EyeAgent achieved 93.7% tool selection accuracy and received expert ratings of more than 88% across accuracy, completeness, safety, reasoning, and interpretability. In human-AI collaboration, EyeAgent matched or exceeded the performance of senior ophthalmologists and, when used as an assistant, improved overall diagnostic accuracy by 18.51% and report quality scores by 19%, with the greatest benefit observed among junior ophthalmologists. These findings establish EyeAgent as a scalable and trustworthy AI framework for ophthalmology and provide a blueprint for modular, multimodal, and clinically aligned next-generation AI systems.