Self adaptive global-local feature enhancement for radiology report generation

TL;DR

This paper introduces AGFNet, a novel framework that dynamically fuses global and anatomy-specific features to generate more accurate radiology reports, outperforming existing methods on benchmark datasets.

Contribution

The paper proposes a self-adaptive fusion mechanism for combining multi-granularity features, enhancing radiology report generation from chest X-ray images.

Findings

Achieved state-of-the-art performance on IU X-Ray and MIMIC-CXR datasets.

Effectively leverages multi-grained image and text information.

Improves report accuracy by focusing on important anatomical regions.

Abstract

Automated radiology report generation aims at automatically generating a detailed description of medical images, which can greatly alleviate the workload of radiologists and provide better medical services to remote areas. Most existing works pay attention to the holistic impression of medical images, failing to utilize important anatomy information. However, in actual clinical practice, radiologists usually locate important anatomical structures, and then look for signs of abnormalities in certain structures and reason the underlying disease. In this paper, we propose a novel framework AGFNet to dynamically fuse the global and anatomy region feature to generate multi-grained radiology report. Firstly, we extract important anatomy region features and global features of input Chest X-ray (CXR). Then, with the region features and the global features as input, our proposed self-adaptive fusion gate module could dynamically fuse multi-granularity information. Finally, the captioning generator generates the radiology reports through multi-granularity features. Experiment results illustrate that our model achieved the state-of-the-art performance on two benchmark datasets including the IU X-Ray and MIMIC-CXR. Further analyses also prove that our model is able to leverage the multi-grained information from radiology images and texts so as to help generate more accurate reports.