Reliability of Gemini 2.5 Pro, ChatGPT 4.1, DeepSeek V3, and Claude Opus 4 in generating standardized CMR protocols

TL;DR

This study evaluates how well four AI models can generate standardized CMR protocols, finding that they show moderate to substantial agreement with expert guidelines.

Contribution

The study introduces a novel evaluation of LLMs for generating pathology-adapted CMR protocols under SCMR guidelines.

Findings

Gemini 2.5 Pro achieved the highest concordance with SCMR guidelines at 71.5%.

LLMs showed substantial agreement for mandatory CMR sequences (Fleiss κ ∈ [0.69, 0.74]).

Automation of CMR protocols could improve access to advanced cardiac diagnostics in primary healthcare.

Abstract

Artificial intelligence (AI) and large language models (LLMs) are increasingly integrated into radiology, offering new possibilities for advanced imaging techniques, including cardiovascular magnetic resonance (CMR). This proof-of-concept study assessed four high-performing LLMs (Gemini 2.5 Pro, ChatGPT 4.1, DeepSeek V3, and Claude Opus 4) on their ability to generate CMR protocols for 140 hypothetical cardiac cases. AI-generated protocols were compared against a reference standard established by a consensus between two experienced cardiovascular radiologists, following the Society for Cardiovascular Magnetic Resonance (SCMR) recommendations. Descriptive statistics were used to quantify the concordance of LLM-generated sequences with the SCMR guidelines. Statistical agreement was measured using Cohen and Fleiss κ statistics. Gemini 2.5 Pro achieved the highest concordance, aligning with the SCMR guidelines in 71.5% of all evaluated scenarios. Overall, LLMs showed moderate agreement with the SCMR protocols, with Gemini 2.5 Pro again performing best (Cohen κ = 0.55). Agreement was substantial for mandatory CMR sequences (Fleiss κ ∈ [0.69, 0.74]) and predominantly fair for optional sequences. The tested LLMs demonstrate a potential to generate efficient and pathology-adapted CMR protocols. Under expert supervision, this capability could streamline the imaging workflow and help extend CMR to primary healthcare centers through protocol automation.

The potential of Gemini 2.5 Pro, ChatGPT 4.1, DeepSeek V3, and Claude Opus 4 to suggest pathology-adapted CMR protocols could improve imaging throughput and help to expand access to advanced cardiac diagnostics in primary healthcare centers.

The tested large language models show potential for generating CMR protocols.Substantial agreement on mandatory CMR sequences promises more efficient examinations.Automation of CMR protocols could help to improve access to this advanced technique outside major medical institutions.

The tested large language models show potential for generating CMR protocols.

Substantial agreement on mandatory CMR sequences promises more efficient examinations.

Automation of CMR protocols could help to improve access to this advanced technique outside major medical institutions.

Figures