CRRG-CLIP: Automatic Generation of Chest Radiology Reports and Classification of Chest Radiographs

TL;DR

The paper introduces CRRG-CLIP, an end-to-end model that automates chest radiology report generation and radiograph classification, improving efficiency and accuracy in medical imaging analysis.

Contribution

It presents a novel combined approach using Faster R-CNN, GPT-2, and CLIP for simultaneous report generation and classification of chest radiographs.

Findings

Generation module performs comparably to high-performance baselines.

Outperforms GPT-4o on BLEU-2, BLEU-3, BLEU-4, and ROUGE-L metrics.

Classification module surpasses state-of-the-art in AUC and Accuracy.

Abstract

The complexity of stacked imaging and the massive number of radiographs make writing radiology reports complex and inefficient. Even highly experienced radiologists struggle to maintain accuracy and consistency in interpreting radiographs under prolonged high-intensity work. To address these issues, this work proposes the CRRG-CLIP Model (Chest Radiology Report Generation and Radiograph Classification Model), an end-to-end model for automated report generation and radiograph classification. The model consists of two modules: the radiology report generation module and the radiograph classification module. The generation module uses Faster R-CNN to identify anatomical regions in radiographs, a binary classifier to select key regions, and GPT-2 to generate semantically coherent reports. The classification module uses the unsupervised Contrastive Language Image Pretraining (CLIP) model, addressing the challenges of high-cost labelled datasets and insufficient features. The results show that the generation module performs comparably to high-performance baseline models on BLEU, METEOR, and ROUGE-L metrics, and outperformed the GPT-4o model on BLEU-2, BLEU-3, BLEU-4, and ROUGE-L metrics. The classification module significantly surpasses the state-of-the-art model in AUC and Accuracy. This demonstrates that the proposed model achieves high accuracy, readability, and fluency in report generation, while multimodal contrastive training with unlabelled radiograph-report pairs enhances classification performance.