CLEAR: A Clinically-Grounded Tabular Framework for Radiology Report Evaluation

TL;DR

CLEAR is a novel evaluation framework for radiology reports that uses attribute-level comparison and expert labels to provide more detailed, clinically interpretable assessments of report quality.

Contribution

The paper introduces CLEAR, a clinically-grounded, attribute-level evaluation framework for radiology reports, along with the CLEAR-Bench dataset annotated by radiologists.

Findings

CLEAR achieves high accuracy in attribute extraction.

CLEAR metrics strongly align with clinical judgment.

The framework offers a comprehensive evaluation of report quality.

Abstract

Existing metrics often lack the granularity and interpretability to capture nuanced clinical differences between candidate and ground-truth radiology reports, resulting in suboptimal evaluation. We introduce a Clinically-grounded tabular framework with Expert-curated labels and Attribute-level comparison for Radiology report evaluation (CLEAR). CLEAR not only examines whether a report can accurately identify the presence or absence of medical conditions, but also assesses whether it can precisely describe each positively identified condition across five key attributes: first occurrence, change, severity, descriptive location, and recommendation. Compared to prior works, CLEAR's multi-dimensional, attribute-level outputs enable a more comprehensive and clinically interpretable evaluation of report quality. Additionally, to measure the clinical alignment of CLEAR, we collaborate with five board-certified radiologists to develop CLEAR-Bench, a dataset of 100 chest X-ray reports from MIMIC-CXR, annotated across 6 curated attributes and 13 CheXpert conditions. Our experiments show that CLEAR achieves high accuracy in extracting clinical attributes and provides automated metrics that are strongly aligned with clinical judgment.