Cross-Modal Vertical Federated Learning for MRI Reconstruction

TL;DR

This paper introduces Fed-CRFD, a novel federated learning framework that improves MRI reconstruction across hospitals with different imaging modalities by disentangling features and aligning shared representations, addressing practical cross-modal challenges.

Contribution

The paper proposes a new federated learning method for cross-modal MRI reconstruction that effectively handles modality differences and limited overlapping data, which was not addressed in prior work.

Findings

Outperforms state-of-the-art MRI reconstruction methods on two datasets.

Effectively mitigates domain shift caused by different modalities.

Leverages overlapping multi-modal data to enhance reconstruction quality.

Abstract

Federated learning enables multiple hospitals to cooperatively learn a shared model without privacy disclosure. Existing methods often take a common assumption that the data from different hospitals have the same modalities. However, such a setting is difficult to fully satisfy in practical applications, since the imaging guidelines may be different between hospitals, which makes the number of individuals with the same set of modalities limited. To this end, we formulate this practical-yet-challenging cross-modal vertical federated learning task, in which shape data from multiple hospitals have different modalities with a small amount of multi-modality data collected from the same individuals. To tackle such a situation, we develop a novel framework, namely Federated Consistent Regularization constrained Feature Disentanglement (Fed-CRFD), for boosting MRI reconstruction by effectively exploring the overlapping samples (individuals with multi-modalities) and solving the domain shift problem caused by different modalities. Particularly, our Fed-CRFD involves an intra-client feature disentangle scheme to decouple data into modality-invariant and modality-specific features, where the modality-invariant features are leveraged to mitigate the domain shift problem. In addition, a cross-client latent representation consistency constraint is proposed specifically for the overlapping samples to further align the modality-invariant features extracted from different modalities. Hence, our method can fully exploit the multi-source data from hospitals while alleviating the domain shift problem. Extensive experiments on two typical MRI datasets demonstrate that our network clearly outperforms state-of-the-art MRI reconstruction methods. The source code will be publicly released upon the publication of this work.