An Accelerated Second-Order Method for Distributed Stochastic Optimization

TL;DR

This paper introduces an accelerated second-order method for distributed stochastic optimization that reduces communication complexity and improves convergence rates by leveraging statistical similarity and cubic-regularized Newton's method.

Contribution

It proposes a novel inexact accelerated cubic-regularized Newton's method tailored for distributed stochastic optimization, achieving lower communication complexity bounds.

Findings

Achieves lower communication complexity than existing methods.

Provides convergence rate bounds for the original stochastic problem.

Demonstrates improved performance with increased parallelization.

Abstract

We consider distributed stochastic optimization problems that are solved with master/workers computation architecture. Statistical arguments allow to exploit statistical similarity and approximate this problem by a finite-sum problem, for which we propose an inexact accelerated cubic-regularized Newton's method that achieves lower communication complexity bound for this setting and improves upon existing upper bound. We further exploit this algorithm to obtain convergence rate bounds for the original stochastic optimization problem and compare our bounds with the existing bounds in several regimes when the goal is to minimize the number of communication rounds and increase the parallelization by increasing the number of workers.