CTest-Metric: A Unified Framework to Assess Clinical Validity of Metrics for CT Report Generation

TL;DR

This paper introduces CTest-Metric, a comprehensive framework for evaluating the clinical validity of metrics used in CT radiology report generation, addressing the need for standardized assessment in medical AI.

Contribution

It presents the first unified framework with modules for testing metric robustness, generalizability, and correlation with clinician judgments, enhancing reproducibility and benchmarking.

Findings

GREEN Score correlates best with expert ratings (Spearman~0.70)

Lexical metrics are highly sensitive to stylistic variations

BERTScore-F1 is least affected by factual errors

Abstract

In the generative AI era, where even critical medical tasks are increasingly automated, radiology report generation (RRG) continues to rely on suboptimal metrics for quality assessment. Developing domain-specific metrics has therefore been an active area of research, yet it remains challenging due to the lack of a unified, well-defined framework to assess their robustness and applicability in clinical contexts. To address this, we present CTest-Metric, a first unified metric assessment framework with three modules determining the clinical feasibility of metrics for CT RRG. The modules test: (i) Writing Style Generalizability (WSG) via LLM-based rephrasing; (ii) Synthetic Error Injection (SEI) at graded severities; and (iii) Metrics-vs-Expert correlation (MvE) using clinician ratings on 175 "disagreement" cases. Eight widely used metrics (BLEU, ROUGE, METEOR, BERTScore-F1, F1-RadGraph, RaTEScore, GREEN Score, CRG) are studied across seven LLMs built on a CT-CLIP encoder. Using our novel framework, we found that lexical NLG metrics are highly sensitive to stylistic variations; GREEN Score aligns best with expert judgments (Spearman~0.70), while CRG shows negative correlation; and BERTScore-F1 is least sensitive to factual error injection. We will release the framework, code, and allowable portion of the anonymized evaluation data (rephrased/error-injected CT reports), to facilitate reproducible benchmarking and future metric development.