Federated Composite Saddle Point Optimization

TL;DR

This paper introduces FeDualEx, a novel federated primal-dual algorithm designed for saddle point problems with composite objectives, addressing constraints and non-smooth regularization in federated learning.

Contribution

It presents the first federated algorithm for composite saddle point optimization, with convergence analysis and empirical validation, including stochastic rates for the sequential version.

Findings

FeDualEx effectively handles constrained and non-smooth problems in federated settings.

The algorithm converges under challenging composite saddle point conditions.

Stochastic rates are established for the sequential FeDualEx, not previously available.

Abstract

Federated learning (FL) approaches for saddle point problems (SPP) have recently gained in popularity due to the critical role they play in machine learning (ML). Existing works mostly target smooth unconstrained objectives in Euclidean space, whereas ML problems often involve constraints or non-smooth regularization, which results in a need for composite optimization. Addressing these issues, we propose Federated Dual Extrapolation (FeDualEx), an extra-step primal-dual algorithm, which is the first of its kind that encompasses both saddle point optimization and composite objectives under the FL paradigm. Both the convergence analysis and the empirical evaluation demonstrate the effectiveness of FeDualEx in these challenging settings. In addition, even for the sequential version of FeDualEx, we provide rates for the stochastic composite saddle point setting which, to our knowledge, are not found in prior literature.