Enhanced Contrastive Learning with Multi-view Longitudinal Data for Chest X-ray Report Generation

TL;DR

This paper introduces a novel contrastive learning approach that leverages multi-view and longitudinal chest X-ray data, along with report supervision, to improve automated radiology report generation accuracy.

Contribution

It proposes a multi-view longitudinal contrastive learning method with a tokenized absence encoding to handle missing data, enhancing report generation performance.

Findings

Achieved 2.3% BLEU-4 improvement on MIMIC-CXR

Achieved 5.5% F1 score improvement on MIMIC-ABN

Achieved 2.7% F1 RadGraph improvement on Two-view CXR

Abstract

Automated radiology report generation offers an effective solution to alleviate radiologists' workload. However, most existing methods focus primarily on single or fixed-view images to model current disease conditions, which limits diagnostic accuracy and overlooks disease progression. Although some approaches utilize longitudinal data to track disease progression, they still rely on single images to analyze current visits. To address these issues, we propose enhanced contrastive learning with Multi-view Longitudinal data to facilitate chest X-ray Report Generation, named MLRG. Specifically, we introduce a multi-view longitudinal contrastive learning method that integrates spatial information from current multi-view images and temporal information from longitudinal data. This method also utilizes the inherent spatiotemporal information of radiology reports to supervise the pre-training of visual and textual representations. Subsequently, we present a tokenized absence encoding technique to flexibly handle missing patient-specific prior knowledge, allowing the model to produce more accurate radiology reports based on available prior knowledge. Extensive experiments on MIMIC-CXR, MIMIC-ABN, and Two-view CXR datasets demonstrate that our MLRG outperforms recent state-of-the-art methods, achieving a 2.3% BLEU-4 improvement on MIMIC-CXR, a 5.5% F1 score improvement on MIMIC-ABN, and a 2.7% F1 RadGraph improvement on Two-view CXR.