shs-nlp at RadSum23: Domain-Adaptive Pre-training of Instruction-tuned LLMs for Radiology Report Impression Generation

TL;DR

This paper improves instruction-tuned large language models for radiology report impression generation by domain-adaptive pre-training with medical data, achieving state-of-the-art zero-shot performance in radiology summarization.

Contribution

It introduces a domain-adaptive pre-training approach for instruction-tuned LLMs using large-scale medical text data, enhancing medical understanding and task performance.

Findings

Outperforms other adaptation methods in zero-shot IMPRESSIONS generation

Ranks 1st in Radiology Report Summarization at BioNLP 2023

Demonstrates improved medical report understanding with domain-specific pre-training

Abstract

Instruction-tuned generative Large language models (LLMs) like ChatGPT and Bloomz possess excellent generalization abilities, but they face limitations in understanding radiology reports, particularly in the task of generating the IMPRESSIONS section from the FINDINGS section. They tend to generate either verbose or incomplete IMPRESSIONS, mainly due to insufficient exposure to medical text data during training. We present a system which leverages large-scale medical text data for domain-adaptive pre-training of instruction-tuned LLMs to enhance its medical knowledge and performance on specific medical tasks. We show that this system performs better in a zero-shot setting than a number of pretrain-and-finetune adaptation methods on the IMPRESSIONS generation task, and ranks 1st among participating systems in Task 1B: Radiology Report Summarization at the BioNLP 2023 workshop.