Principled Federated Domain Adaptation: Gradient Projection and Auto-Weighting

TL;DR

This paper introduces a theoretical framework and novel aggregation methods for Federated Domain Adaptation, addressing domain shift and data scarcity by improving server aggregation rules and automatically balancing source and target gradients.

Contribution

It presents a new theoretical analysis, a lightweight aggregation rule called FedGP, and an auto-weighting scheme for optimal gradient combination in federated domain adaptation.

Findings

FedGP significantly improves target performance under domain shift.

Auto-weighting scheme enhances the effectiveness of aggregation rules.

Theoretical metrics and theorems provide insights into FDA performance.

Abstract

Federated Domain Adaptation (FDA) describes the federated learning (FL) setting where source clients and a server work collaboratively to improve the performance of a target client where limited data is available. The domain shift between the source and target domains, coupled with limited data of the target client, makes FDA a challenging problem, e.g., common techniques such as federated averaging and fine-tuning fail due to domain shift and data scarcity. To theoretically understand the problem, we introduce new metrics that characterize the FDA setting and a theoretical framework with novel theorems for analyzing the performance of server aggregation rules. Further, we propose a novel lightweight aggregation rule, Federated Gradient Projection ($\texttt{FedGP}$), which significantly improves the target performance with domain shift and data scarcity. Moreover, our theory suggests an $\textit{auto-weighting scheme}$ that finds the optimal combinations of the source and target gradients. This scheme improves both $\texttt{FedGP}$ and a simpler heuristic aggregation rule. Extensive experiments verify the theoretical insights and illustrate the effectiveness of the proposed methods in practice.