Uncovering Knowledge Gaps in Radiology Report Generation Models through Knowledge Graphs

TL;DR

This paper introduces ReXKG, a knowledge graph-based evaluation system for radiology report generation models, revealing their understanding gaps and providing insights to enhance AI performance in clinical settings.

Contribution

The paper presents a novel knowledge graph construction and three metrics for evaluating radiology report generation models, addressing limitations of existing evaluation methods.

Findings

AI models show gaps in understanding radiological image details

Knowledge graphs reveal differences between AI-generated and human reports

Evaluation metrics identify specific areas for model improvement

Abstract

Recent advancements in artificial intelligence have significantly improved the automatic generation of radiology reports. However, existing evaluation methods fail to reveal the models' understanding of radiological images and their capacity to achieve human-level granularity in descriptions. To bridge this gap, we introduce a system, named ReXKG, which extracts structured information from processed reports to construct a comprehensive radiology knowledge graph. We then propose three metrics to evaluate the similarity of nodes (ReXKG-NSC), distribution of edges (ReXKG-AMS), and coverage of subgraphs (ReXKG-SCS) across various knowledge graphs. We conduct an in-depth comparative analysis of AI-generated and human-written radiology reports, assessing the performance of both specialist and generalist models. Our study provides a deeper understanding of the capabilities and limitations of current AI models in radiology report generation, offering valuable insights for improving model performance and clinical applicability.