Clinically Grounded Agent-based Report Evaluation: An Interpretable Metric for Radiology Report Generation

TL;DR

This paper introduces ICARE, an interpretable, agent-based evaluation framework for radiology report generation that uses question-answering to assess clinical accuracy and consistency, aligning better with expert judgment than existing metrics.

Contribution

ICARE is a novel, clinically grounded evaluation method employing large language models and dynamic questioning to provide transparent, interpretable report quality assessment.

Findings

ICARE aligns more closely with expert judgments than prior metrics.

The framework is sensitive to clinical content and reproducible.

It reveals interpretable error patterns in report evaluation.

Abstract

Radiological imaging is central to diagnosis, treatment planning, and clinical decision-making. Vision-language foundation models have spurred interest in automated radiology report generation (RRG), but safe deployment requires reliable clinical evaluation of generated reports. Existing metrics often rely on surface-level similarity or behave as black boxes, lacking interpretability. We introduce ICARE (Interpretable and Clinically-grounded Agent-based Report Evaluation), an interpretable evaluation framework leveraging large language model agents and dynamic multiple-choice question answering (MCQA). Two agents, each with either the ground-truth or generated report, generate clinically meaningful questions and quiz each other. Agreement on answers captures preservation and consistency of findings, serving as interpretable proxies for clinical precision and recall. By linking scores to question-answer pairs, ICARE enables transparent, and interpretable assessment. Clinician studies show ICARE aligns significantly more with expert judgment than prior metrics. Perturbation analyses confirm sensitivity to clinical content and reproducibility, while model comparisons reveal interpretable error patterns.