How Well Can Modern LLMs Act as Agent Cores in Radiology Environments?

TL;DR

This paper introduces RadA-BenchPlat, a comprehensive evaluation platform for large language models acting as agent cores in radiology, revealing current capabilities and limitations in complex clinical tasks and tool integration.

Contribution

The paper presents RadA-BenchPlat, a new benchmarking platform with synthetic radiology data, and demonstrates how prompt engineering and automated tool building can improve LLM performance in radiology tasks.

Findings

Claude-3.7-Sonnet achieves 67.1% task completion in routine settings.

Prompt engineering strategies improve complex task performance by 48.2%.

Automated tool building achieves 65.4% success rate.

Abstract

We introduce RadA-BenchPlat, an evaluation platform that benchmarks the performance of large language models (LLMs) act as agent cores in radiology environments using 2,200 radiologist-verified synthetic patient records covering six anatomical regions, five imaging modalities, and 2,200 disease scenarios, resulting in 24,200 question-answer pairs that simulate diverse clinical situations. The platform also defines ten categories of tools for agent-driven task solving and evaluates seven leading LLMs, revealing that while models like Claude-3.7-Sonnet can achieve a 67.1% task completion rate in routine settings, they still struggle with complex task understanding and tool coordination, limiting their capacity to serve as the central core of automated radiology systems. By incorporating four advanced prompt engineering strategies--where prompt-backpropagation and multi-agent collaboration contributed 16.8% and 30.7% improvements, respectively--the performance for complex tasks was enhanced by 48.2% overall. Furthermore, automated tool building was explored to improve robustness, achieving a 65.4% success rate, thereby offering promising insights for the future integration of fully automated radiology applications into clinical practice. All of our code and data are openly available at https://github.com/MAGIC-AI4Med/RadABench.