Quality Control for Radiology Report Generation Models via Auxiliary Auditing Components

TL;DR

This paper introduces a modular auditing framework using auxiliary components to evaluate and improve the reliability of AI-generated radiology reports, focusing on clinical accuracy and diagnostic relevance.

Contribution

The proposed framework employs disease-classifiers as auxiliary components to audit report quality, enhancing reliability assessment of AI-generated radiology reports.

Findings

Higher F1 scores for reliable reports with auxiliary auditing.

Confidence scores of auxiliary classifiers improve audit accuracy.

Effective identification of clinically accurate reports.

Abstract

Automation of medical image interpretation could alleviate bottlenecks in diagnostic workflows, and has become of particular interest in recent years due to advancements in natural language processing. Great strides have been made towards automated radiology report generation via AI, yet ensuring clinical accuracy in generated reports is a significant challenge, hindering deployment of such methods in clinical practice. In this work we propose a quality control framework for assessing the reliability of AI-generated radiology reports with respect to semantics of diagnostic importance using modular auxiliary auditing components (AC). Evaluating our pipeline on the MIMIC-CXR dataset, our findings show that incorporating ACs in the form of disease-classifiers can enable auditing that identifies more reliable reports, resulting in higher F1 scores compared to unfiltered generated reports. Additionally, leveraging the confidence of the AC labels further improves the audit's effectiveness.