Generalized and Personalized Federated Learning with Black-Box Foundation Models via Orthogonal Transformations

TL;DR

FedOT is a federated learning framework that enables effective personalization and generalization with black-box foundation models by using orthogonal transformations, ensuring privacy and robustness in heterogeneous environments.

Contribution

Introduces FedOT, a novel FL method that preserves black-box FM privacy while balancing personalization and generalization through orthogonal transformations.

Findings

Outperforms baseline FL methods across benchmarks.

Effectively mitigates gradient conflicts in heterogeneous data.

Demonstrates robustness and scalability through extensive validation.

Abstract

Federated Learning (FL) facilitates decentralized model training while preserving data privacy. However, achieving both robust generalization and effective personalization simultaneously in heterogeneous (non-IID) environments remains a formidable challenge. Furthermore, the widespread adoption of proprietary Foundation Models (FMs) introduces a critical requirement for dual privacy: (a) protecting sensitive client data and (b) securing the server's valuable intellectual property. This mandates strictly black-box access to the FM. To address these multifaceted challenges, we introduce FedOT, a novel FL framework optimized for black-box FMs. FedOT employs a shared global task-dependent classifier while facilitating local adaptation through client-specific orthogonal transformations applied externally to the FM embeddings. This architecture inherently guarantees that the FM's internal parameters remain inaccessible and unmodified. By enforcing orthogonality, FedOT effectively mitigates gradient conflicts across diverse clients, which is theoretically bounded, preserves the semantic integrity of the FM representations, and achieves robust performance under significant data heterogeneity. The synergy of global and local parameters optimally balances generalization and personalization, markedly outperforming baseline FL methods across diverse benchmarks. Extensive empirical analysis, including rigorous multi-seed validation and scalability assessments, substantiates the robustness, efficiency, and superior performance of FedOT.