CT-GRAPH: Hierarchical Graph Attention Network for Anatomy-Guided CT Report Generation

TL;DR

This paper introduces CT-GRAPH, a hierarchical graph attention network that models anatomical relationships in CT scans to improve automated radiology report generation, significantly outperforming existing methods.

Contribution

The paper presents a novel hierarchical graph attention network that explicitly encodes anatomical structures for enhanced CT report generation, integrating pretrained features with a large language model.

Findings

Achieves 7.9% higher F1 score than state-of-the-art methods.

Effectively models fine-grained organ relationships.

Utilizes pretrained 3D medical feature encoders for improved accuracy.

Abstract

As medical imaging is central to diagnostic processes, automating the generation of radiology reports has become increasingly relevant to assist radiologists with their heavy workloads. Most current methods rely solely on global image features, failing to capture fine-grained organ relationships crucial for accurate reporting. To this end, we propose CT-GRAPH, a hierarchical graph attention network that explicitly models radiological knowledge by structuring anatomical regions into a graph, linking fine-grained organ features to coarser anatomical systems and a global patient context. Our method leverages pretrained 3D medical feature encoders to obtain global and organ-level features by utilizing anatomical masks. These features are further refined within the graph and then integrated into a large language model to generate detailed medical reports. We evaluate our approach for the task of report generation on the large-scale chest CT dataset CT-RATE. We provide an in-depth analysis of pretrained feature encoders for CT report generation and show that our method achieves a substantial improvement of absolute 7.9\% in F1 score over current state-of-the-art methods. The code is publicly available at https://github.com/hakal104/CT-GRAPH.