HeadCT-ONE: Enabling Granular and Controllable Automated Evaluation of Head CT Radiology Report Generation

TL;DR

HeadCT-ONE introduces a domain-specific, ontology-normalized metric for more flexible and precise evaluation of head CT report generation, improving semantic matching and clinical relevance assessment.

Contribution

It presents a novel ontology-based normalization approach that enhances report evaluation by enabling controllable weighting and better semantic equivalence detection.

Findings

Improves detection of semantically equivalent reports

Better distinguishes normal from abnormal reports

Aligns with radiologists' assessment of errors

Abstract

We present Head CT Ontology Normalized Evaluation (HeadCT-ONE), a metric for evaluating head CT report generation through ontology-normalized entity and relation extraction. HeadCT-ONE enhances current information extraction derived metrics (such as RadGraph F1) by implementing entity normalization through domain-specific ontologies, addressing radiological language variability. HeadCT-ONE compares normalized entities and relations, allowing for controllable weighting of different entity types or specific entities. Through experiments on head CT reports from three health systems, we show that HeadCT-ONE's normalization and weighting approach improves the capture of semantically equivalent reports, better distinguishes between normal and abnormal reports, and aligns with radiologists' assessment of clinically significant errors, while offering flexibility to prioritize specific aspects of report content. Our results demonstrate how HeadCT-ONE enables more flexible, controllable, and granular automated evaluation of head CT reports.