Privacy-Preserving Chest X-ray Report Generation via Multimodal Federated Learning with ViT and GPT-2

TL;DR

This paper introduces a federated learning framework using ViT and GPT-2 for privacy-preserving chest X-ray report generation, achieving comparable or better results than centralized models while maintaining data confidentiality.

Contribution

It proposes a novel multimodal federated learning approach with a new aggregation strategy, L-FedAvg, for medical report generation without sharing raw data.

Findings

Krum aggregation outperforms other strategies in report quality metrics

Federated learning matches or exceeds centralized model performance

The framework ensures data privacy while generating clinically relevant reports

Abstract

The automated generation of radiology reports from chest X-ray images holds significant promise in enhancing diagnostic workflows while preserving patient privacy. Traditional centralized approaches often require sensitive data transfer, posing privacy concerns. To address this, the study proposes a Multimodal Federated Learning framework for chest X-ray report generation using the IU-Xray dataset. The system utilizes a Vision Transformer (ViT) as the encoder and GPT-2 as the report generator, enabling decentralized training without sharing raw data. Three Federated Learning (FL) aggregation strategies: FedAvg, Krum Aggregation and a novel Loss-aware Federated Averaging (L-FedAvg) were evaluated. Among these, Krum Aggregation demonstrated superior performance across lexical and semantic evaluation metrics such as ROUGE, BLEU, BERTScore and RaTEScore. The results show that FL can match or surpass centralized models in generating clinically relevant and semantically rich radiology reports. This lightweight and privacy-preserving framework paves the way for collaborative medical AI development without compromising data confidentiality.