Federated Empirical Risk Minimization via Second-Order Method

TL;DR

This paper introduces a second-order interior point method for federated empirical risk minimization, achieving efficient communication complexity and enhancing privacy-preserving machine learning across distributed data sources.

Contribution

The paper proposes a novel interior point method tailored for federated ERM problems, with theoretical analysis of its communication complexity per iteration.

Findings

Communication complexity per iteration is O(d^{3/2})

Applicable to various convex ERM problems in federated settings

Improves efficiency and privacy in distributed machine learning

Abstract

Many convex optimization problems with important applications in machine learning are formulated as empirical risk minimization (ERM). There are several examples: linear and logistic regression, LASSO, kernel regression, quantile regression, $p$-norm regression, support vector machines (SVM), and mean-field variational inference. To improve data privacy, federated learning is proposed in machine learning as a framework for training deep learning models on the network edge without sharing data between participating nodes. In this work, we present an interior point method (IPM) to solve a general ERM problem under the federated learning setting. We show that the communication complexity of each iteration of our IPM is $\tilde{O}(d^{3/2})$, where $d$ is the dimension (i.e., number of features) of the dataset.