Federated Minimax Optimization with Client Heterogeneity

TL;DR

This paper introduces a federated minimax optimization framework addressing client heterogeneity, proposing normalization of client updates to improve convergence, with theoretical analysis and experimental validation showing improved complexity results.

Contribution

It presents a novel federated minimax algorithm with normalization to handle heterogeneity, along with convergence analysis under general conditions, advancing the state-of-the-art in federated minimax optimization.

Findings

Normalized client updates improve convergence.

Theoretical analysis covers nonconvex-concave and nonconvex-nonconcave functions.

Experimental results validate the theoretical improvements.

Abstract

Minimax optimization has seen a surge in interest with the advent of modern applications such as GANs, and it is inherently more challenging than simple minimization. The difficulty is exacerbated by the training data residing at multiple edge devices or \textit{clients}, especially when these clients can have heterogeneous datasets and local computation capabilities. We propose a general federated minimax optimization framework that subsumes such settings and several existing methods like Local SGDA. We show that naive aggregation of heterogeneous local progress results in optimizing a mismatched objective function -- a phenomenon previously observed in standard federated minimization. To fix this problem, we propose normalizing the client updates by the number of local steps undertaken between successive communication rounds. We analyze the convergence of the proposed algorithm for classes of nonconvex-concave and nonconvex-nonconcave functions and characterize the impact of heterogeneous client data, partial client participation, and heterogeneous local computations. Our analysis works under more general assumptions on the intra-client noise and inter-client heterogeneity than so far considered in the literature. For all the function classes considered, we significantly improve the existing computation and communication complexity results. Experimental results support our theoretical claims.