Federated Cross-Modal Style-Aware Prompt Generation

TL;DR

FedCSAP introduces a novel federated prompt generation method that leverages multi-scale visual features and style indicators to improve vision-language model performance across diverse, decentralized datasets while preserving data privacy.

Contribution

It is the first to incorporate multi-scale features and style-aware prompts in federated learning for vision-language models, enhancing robustness and generalization.

Findings

Outperforms existing federated prompt methods in accuracy

Effectively handles non-IID class distributions

Improves generalization to unseen classes

Abstract

Prompt learning has propelled vision-language models like CLIP to excel in diverse tasks, making them ideal for federated learning due to computational efficiency. However, conventional approaches that rely solely on final-layer features miss out on rich multi-scale visual cues and domain-specific style variations in decentralized client data. To bridge this gap, we introduce FedCSAP (Federated Cross-Modal Style-Aware Prompt Generation). Our framework harnesses low, mid, and high-level features from CLIP's vision encoder alongside client-specific style indicators derived from batch-level statistics. By merging intricate visual details with textual context, FedCSAP produces robust, context-aware prompt tokens that are both distinct and non-redundant, thereby boosting generalization across seen and unseen classes. Operating within a federated learning paradigm, our approach ensures data privacy through local training and global aggregation, adeptly handling non-IID class distributions and diverse domain-specific styles. Comprehensive experiments on multiple image classification datasets confirm that FedCSAP outperforms existing federated prompt learning methods in both accuracy and overall generalization.