FITA: Fine-grained Image-Text Aligner for Radiology Report Generation

TL;DR

FITA introduces a novel approach for precise fine-grained alignment of image features and text in radiology report generation, utilizing saliency maps, triplet training, and contrastive loss to improve report accuracy.

Contribution

The paper proposes a new fine-grained alignment method with three innovative modules, enhancing the accuracy of image-text correspondence in radiology reports.

Findings

Outperforms existing methods on benchmark datasets

Achieves more accurate and detailed report generation

Effectively aligns symptoms with abnormal regions

Abstract

Radiology report generation aims to automatically generate detailed and coherent descriptive reports alongside radiology images. Previous work mainly focused on refining fine-grained image features or leveraging external knowledge. However, the precise alignment of fine-grained image features with corresponding text descriptions has not been considered. This paper presents a novel method called Fine-grained Image-Text Aligner (FITA) to construct fine-grained alignment for image and text features. It has three novel designs: Image Feature Refiner (IFR), Text Feature Refiner (TFR) and Contrastive Aligner (CA). IFR and TFR aim to learn fine-grained image and text features, respectively. We achieve this by leveraging saliency maps to effectively fuse symptoms with corresponding abnormal visual regions, and by utilizing a meticulously constructed triplet set for training. Finally, CA module aligns fine-grained image and text features using contrastive loss for precise alignment. Results show that our method surpasses existing methods on the widely used benchmark