Inclusive Data Representation in Federated Learning: A Novel Approach Integrating Textual and Visual Prompt

TL;DR

This paper introduces TPFL and ATPFL, innovative federated learning methods that integrate visual and textual prompts to better represent local data and improve model robustness amid data heterogeneity.

Contribution

The paper proposes a novel multimodal prompt tuning framework for federated learning, addressing data heterogeneity and communication efficiency challenges.

Findings

TPFL outperforms baseline models in data representation.

ATPFL enhances global knowledge and model robustness.

Contrastive learning improves model performance.

Abstract

Federated Learning (FL) is often impeded by communication overhead issues. Prompt tuning, as a potential solution, has been introduced to only adjust a few trainable parameters rather than the whole model. However, current single-modality prompt tuning approaches fail to comprehensively portray local clients' data. To overcome this limitation, we present Twin Prompt Federated learning (TPFL), a pioneering solution that integrates both visual and textual modalities, ensuring a more holistic representation of local clients' data characteristics. Furthermore, in order to tackle the data heterogeneity issues, we introduce the Augmented TPFL (ATPFL) employing the contrastive learning to TPFL, which not only enhances the global knowledge acquisition of client models but also fosters the development of robust, compact models. The effectiveness of TPFL and ATPFL is substantiated by our extensive evaluations, consistently showing superior performance compared to all baselines.