Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method

TL;DR

This paper provides a theoretical framework for prompt-based federated learning with vision-language models, analyzing signal and noise evolution, and introduces a prompt portfolio approach inspired by portfolio optimization to improve generalization and personalization.

Contribution

It develops a novel theoretical analysis framework for prompt-based federated learning and proposes a prompt portfolio method inspired by portfolio optimization principles.

Findings

Prompt portfolio improves federated learning performance.

Theoretical analysis links task relevance to learning performance.

Empirical results validate the theoretical claims.

Abstract

Integrating pretrained vision-language foundation models like CLIP into federated learning has attracted significant attention for enhancing generalization across diverse tasks. Typically, federated learning of vision-language models employs prompt learning to reduce communication and computational costs, i.e., prompt-based federated learning. However, there is limited theoretical analysis to understand the performance of prompt-based federated learning. In this work, we construct a theoretical analysis framework for prompt-based federated learning via feature learning theory. Specifically, we monitor the evolution of signal learning and noise memorization in prompt-based federated learning, demonstrating that performance can be assessed by the ratio of task-relevant to task-irrelevant coefficients. Furthermore, we draw an analogy between income and risk in portfolio optimization and the task-relevant and task-irrelevant terms in feature learning. Leveraging inspiration from portfolio optimization that combining two independent assets will maintain the income while reducing the risk, we introduce two prompts: global prompt and local prompt to construct a prompt portfolio to balance the generalization and personalization. Consequently, we showed the performance advantage of the prompt portfolio and derived the optimal mixing coefficient. These theoretical claims have been further supported by empirical experiments.