Learning Across Decentralized Multi-Modal Remote Sensing Archives with Federated Learning

TL;DR

This paper introduces a federated learning framework for decentralized multi-modal remote sensing image classification, effectively integrating different data modalities across clients without data sharing.

Contribution

It proposes a novel multi-modal federated learning framework with modules for fusion, alignment, and mutual information maximization, addressing modality heterogeneity in remote sensing data.

Findings

Outperforms traditional iterative model averaging in experiments

Effectively aligns multi-modal representations across clients

Enhances classification accuracy in decentralized multi-modal RS archives

Abstract

The development of federated learning (FL) methods, which aim to learn from distributed databases (i.e., clients) without accessing data on clients, has recently attracted great attention. Most of these methods assume that the clients are associated with the same data modality. However, remote sensing (RS) images in different clients can be associated with different data modalities that can improve the classification performance when jointly used. To address this problem, in this paper we introduce a novel multi-modal FL framework that aims to learn from decentralized multi-modal RS image archives for RS image classification problems. The proposed framework is made up of three modules: 1) multi-modal fusion (MF); 2) feature whitening (FW); and 3) mutual information maximization (MIM). The MF module performs iterative model averaging to learn without accessing data on clients in the case that clients are associated with different data modalities. The FW module aligns the representations learned among the different clients. The MIM module maximizes the similarity of images from different modalities. Experimental results show the effectiveness of the proposed framework compared to iterative model averaging, which is a widely used algorithm in FL. The code of the proposed framework is publicly available at https://git.tu-berlin.de/rsim/MM-FL.