Communication-Efficient Federated Group Distributionally Robust Optimization

TL;DR

This paper proposes communication-efficient algorithms for federated group distributionally robust optimization, significantly reducing communication costs while maintaining robustness across diverse data distributions in federated learning.

Contribution

Introduces three novel algorithms for federated group DRO that lower communication complexity and improve practical performance with Adam-type updates.

Findings

Algorithms achieve lower communication complexity ($O(1/psilon^4)$ and $O(1/psilon^3)$).

Demonstrated effectiveness on NLP and computer vision tasks.

Potential for Adam-type updates to outperform SGD in practical scenarios.

Abstract

Federated learning faces challenges due to the heterogeneity in data volumes and distributions at different clients, which can compromise model generalization ability to various distributions. Existing approaches to address this issue based on group distributionally robust optimization (GDRO) often lead to high communication and sample complexity. To this end, this work introduces algorithms tailored for communication-efficient Federated Group Distributionally Robust Optimization (FGDRO). Our contributions are threefold: Firstly, we introduce the FGDRO-CVaR algorithm, which optimizes the average top-K losses while reducing communication complexity to $O(1/\epsilon^4)$, where $\epsilon$ denotes the desired precision level. Secondly, our FGDRO-KL algorithm is crafted to optimize KL regularized FGDRO, cutting communication complexity to $O(1/\epsilon^3)$. Lastly, we propose FGDRO-KL-Adam to utilize Adam-type local updates in FGDRO-KL, which not only maintains a communication cost of $O(1/\epsilon^3)$ but also shows potential to surpass SGD-type local steps in practical applications. The effectiveness of our algorithms has been demonstrated on a variety of real-world tasks, including natural language processing and computer vision.