HC-LLM: Historical-Constrained Large Language Models for Radiology Report Generation

TL;DR

This paper introduces HC-LLM, a novel framework that enhances large language models with the ability to generate radiology reports by effectively incorporating longitudinal historical data, improving accuracy and consistency in disease progression analysis.

Contribution

The paper presents a new method for integrating historical longitudinal data into LLMs for radiology report generation, addressing sequence dependency challenges and improving diagnostic accuracy.

Findings

Achieves state-of-the-art results on Longitudinal-MIMIC dataset.

Performs well without historical data during testing.

Easily adaptable to other multimodal large models.

Abstract

Radiology report generation (RRG) models typically focus on individual exams, often overlooking the integration of historical visual or textual data, which is crucial for patient follow-ups. Traditional methods usually struggle with long sequence dependencies when incorporating historical information, but large language models (LLMs) excel at in-context learning, making them well-suited for analyzing longitudinal medical data. In light of this, we propose a novel Historical-Constrained Large Language Models (HC-LLM) framework for RRG, empowering LLMs with longitudinal report generation capabilities by constraining the consistency and differences between longitudinal images and their corresponding reports. Specifically, our approach extracts both time-shared and time-specific features from longitudinal chest X-rays and diagnostic reports to capture disease progression. Then, we ensure consistent representation by applying intra-modality similarity constraints and aligning various features across modalities with multimodal contrastive and structural constraints. These combined constraints effectively guide the LLMs in generating diagnostic reports that accurately reflect the progression of the disease, achieving state-of-the-art results on the Longitudinal-MIMIC dataset. Notably, our approach performs well even without historical data during testing and can be easily adapted to other multimodal large models, enhancing its versatility.