Global and Local Prompts Cooperation via Optimal Transport for Federated Learning

TL;DR

This paper introduces FedOTP, a federated learning framework that uses optimal transport to effectively balance global consensus and local personalization in prompt learning for visual-language models, especially under data heterogeneity.

Contribution

It proposes a novel federated prompt learning method utilizing optimal transport to address data heterogeneity and enhance collaboration among clients.

Findings

FedOTP outperforms state-of-the-art methods on heterogeneous datasets.

The method effectively balances global and local features via optimal transport.

Prompts focus on core image regions, improving personalization.

Abstract

Prompt learning in pretrained visual-language models has shown remarkable flexibility across various downstream tasks. Leveraging its inherent lightweight nature, recent research attempted to integrate the powerful pretrained models into federated learning frameworks to simultaneously reduce communication costs and promote local training on insufficient data. Despite these efforts, current federated prompt learning methods lack specialized designs to systematically address severe data heterogeneities, e.g., data distribution with both label and feature shifts involved. To address this challenge, we present Federated Prompts Cooperation via Optimal Transport (FedOTP), which introduces efficient collaborative prompt learning strategies to capture diverse category traits on a per-client basis. Specifically, for each client, we learn a global prompt to extract consensus knowledge among clients, and a local prompt to capture client-specific category characteristics. Unbalanced Optimal Transport is then employed to align local visual features with these prompts, striking a balance between global consensus and local personalization. By relaxing one of the equality constraints, FedOTP enables prompts to focus solely on the core regions of image patches. Extensive experiments on datasets with various types of heterogeneities have demonstrated that our FedOTP outperforms the state-of-the-art methods.