Comparative Analysis of Abstractive Summarization Models for Clinical Radiology Reports

TL;DR

This paper compares various advanced abstractive summarization models on radiology reports to identify their strengths and limitations in generating concise impressions from detailed findings, aiding healthcare professionals.

Contribution

It provides a comprehensive analysis of multiple large language models for medical report summarization using diverse evaluation metrics.

Findings

ChatGPT-4 and BART-base perform best on ROUGE scores.

Pointer Generator Network shows strengths in medical terminology retention.

Model performances vary significantly across different evaluation metrics.

Abstract

The findings section of a radiology report is often detailed and lengthy, whereas the impression section is comparatively more compact and captures key diagnostic conclusions. This research explores the use of advanced abstractive summarization models to generate the concise impression from the findings section of a radiology report. We have used the publicly available MIMIC-CXR dataset. A comparative analysis is conducted on leading pre-trained and open-source large language models, including T5-base, BART-base, PEGASUS-x-base, ChatGPT-4, LLaMA-3-8B, and a custom Pointer Generator Network with a coverage mechanism. To ensure a thorough assessment, multiple evaluation metrics are employed, including ROUGE-1, ROUGE-2, ROUGE-L, METEOR, and BERTScore. By analyzing the performance of these models, this study identifies their respective strengths and limitations in the summarization of medical text. The findings of this paper provide helpful information for medical professionals who need automated summarization solutions in the healthcare sector.