Replace and Report: NLP Assisted Radiology Report Generation

TL;DR

This paper introduces a novel template-based method for generating detailed radiology reports from chest X-ray images, significantly outperforming existing models by combining image classification, transformer-based description generation, and template replacement.

Contribution

It presents the first approach that generates abnormal findings as small sentences and replaces them in a normal report template, improving report quality and accuracy.

Findings

Outperforms state-of-the-art models by 25-48% in BLEU-1, ROUGE-L, METEOR, and CIDEr scores.

Effective use of a multi-step process combining image classification, transformer models, and rule-based template replacement.

Demonstrated on IU Chest X-ray and MIMIC-CXR datasets with superior results.

Abstract

Clinical practice frequently uses medical imaging for diagnosis and treatment. A significant challenge for automatic radiology report generation is that the radiology reports are long narratives consisting of multiple sentences for both abnormal and normal findings. Therefore, applying conventional image captioning approaches to generate the whole report proves to be insufficient, as these are designed to briefly describe images with short sentences. We propose a template-based approach to generate radiology reports from radiographs. Our approach involves the following: i) using a multilabel image classifier, produce the tags for the input radiograph; ii) using a transformer-based model, generate pathological descriptions (a description of abnormal findings seen on radiographs) from the tags generated in step (i); iii) using a BERT-based multi-label text classifier, find the spans in the normal report template to replace with the generated pathological descriptions; and iv) using a rule-based system, replace the identified span with the generated pathological description. We performed experiments with the two most popular radiology report datasets, IU Chest X-ray and MIMIC-CXR and demonstrated that the BLEU-1, ROUGE-L, METEOR, and CIDEr scores are better than the State-of-the-Art models by 25%, 36%, 44% and 48% respectively, on the IU X-RAY dataset. To the best of our knowledge, this is the first attempt to generate chest X-ray radiology reports by first creating small sentences for abnormal findings and then replacing them in the normal report template.