On Second-order Optimization Methods for Federated Learning

TL;DR

This paper evaluates second-order optimization methods in federated learning, finding that FedAvg performs well and proposing a new second-order method with a global line search to improve convergence.

Contribution

The paper compares second-order methods with FedAvg in federated learning and introduces a novel second-order update with a global line search.

Findings

FedAvg performs competitively against second-order methods under fair metrics.

A new second-order method with a global line search is proposed.

Second-order methods can be effectively adapted for federated learning.

Abstract

We consider federated learning (FL), where the training data is distributed across a large number of clients. The standard optimization method in this setting is Federated Averaging (FedAvg), which performs multiple local first-order optimization steps between communication rounds. In this work, we evaluate the performance of several second-order distributed methods with local steps in the FL setting which promise to have favorable convergence properties. We (i) show that FedAvg performs surprisingly well against its second-order competitors when evaluated under fair metrics (equal amount of local computations)-in contrast to the results of previous work. Based on our numerical study, we propose (ii) a novel variant that uses second-order local information for updates and a global line search to counteract the resulting local specificity.