FedRSClip: Federated Learning for Remote Sensing Scene Classification Using Vision-Language Models

TL;DR

FedRSCLIP introduces a federated learning framework for remote sensing image classification using CLIP, employing prompt learning and alignment constraints to address data heterogeneity and communication challenges.

Contribution

This work is the first to adapt Vision-Language Models for federated remote sensing classification with a novel dual-prompt mechanism and alignment constraints.

Findings

FedRSCLIP outperforms baseline methods in classification accuracy.

The dual-prompt mechanism effectively balances global and local model adaptation.

The proposed framework reduces communication costs while maintaining high performance.

Abstract

Remote sensing data is often distributed across multiple institutions, and due to privacy concerns and data-sharing restrictions, leveraging large-scale datasets in a centralized training framework is challenging. Federated learning offers a promising solution by enabling collaborative model training across distributed data sources without requiring data centralization. However, current Vision-Language Models (VLMs), which typically contain billions of parameters, pose significant communication challenges for traditional federated learning approaches based on model parameter updates, as they would incur substantial communication costs. In this paper, we propose FedRSCLIP, the first federated learning framework designed for remote sensing image classification based on a VLM, specifically CLIP. FedRSCLIP addresses the challenges of data heterogeneity and large-scale model transmission in federated environments by introducing Prompt Learning, which optimizes only a small set of tunable parameters. The framework introduces a dual-prompt mechanism, comprising Shared Prompts for global knowledge sharing and Private Prompts for client-specific adaptation. To maintain semantic coherence between shared and private prompts, we propose the Dual Prompt Alignment Constraint to balance global consistency and local adaptability across diverse client distributions. Additionally, to enhance cross-modal representation learning, we introduce the Cross-Modal Feature Alignment Constraint to align multimodal features between text and image prompts. To validate the effectiveness of our proposed model, we construct a Fed-RSIC dataset based on three existing remote sensing image classification datasets, specifically designed to simulate various federated learning configurations. Experimental results demonstrate the effectiveness and superiority of FedRSCLIP in remote sensing image classification.