Evaluating Automated Radiology Report Quality through Fine-Grained Phrasal Grounding of Clinical Findings

TL;DR

This paper introduces a novel evaluation method for radiology reports that combines fine-grained textual analysis with visual localization to better assess report quality and detect factual errors.

Contribution

It develops a new evaluation metric that integrates phrasal grounding of clinical findings with textual analysis for more accurate report quality assessment.

Findings

The new metric outperforms existing textual metrics in detecting factual errors.

It demonstrates robustness and sensitivity on a MIMIC-derived dataset.

The approach effectively combines textual and visual information for report evaluation.

Abstract

Several evaluation metrics have been developed recently to automatically assess the quality of generative AI reports for chest radiographs based only on textual information using lexical, semantic, or clinical named entity recognition methods. In this paper, we develop a new method of report quality evaluation by first extracting fine-grained finding patterns capturing the location, laterality, and severity of a large number of clinical findings. We then performed phrasal grounding to localize their associated anatomical regions on chest radiograph images. The textual and visual measures are then combined to rate the quality of the generated reports. We present results that compare this evaluation metric with other textual metrics on a gold standard dataset derived from the MIMIC collection and show its robustness and sensitivity to factual errors.