Decentralized Inexact Cubic Newton Method with Consensus Procedure

TL;DR

This paper introduces a decentralized second-order optimization method that efficiently combines consensus procedures with cubic Newton steps, achieving near-optimal iteration complexity with minimal communication overhead.

Contribution

It develops a novel Decentralized Cubic Newton method with theoretical guarantees and practical implementation for high-dimensional generalized linear models.

Findings

Matches iteration complexity of exact cubic Newton methods

Requires only polylogarithmic communication rounds for consensus

Practical implementation for high-dimensional models

Abstract

Distributed optimization is widely used in large-scale and privacy-preserving machine learning, where each agent stores a local objective and communicates only with its neighbors in a connected network. We study decentralized second-order optimization and focus on consensus procedures that approximately average local iterates, gradients, and Hessians through neighbor-to-neighbor communications. We propose a general Decentralized Cubic Newton method for convex optimization under $L_1$-smoothness of gradients and $L_2$-Lipschitz continuity of Hessians, and develop a theory that accurately tracks the inaccuracies caused by consensus and by disagreement between local iterates. Under these assumptions, the method matches the iteration complexity of the exact Cubic Newton method and requires only additional polylogarithmic communication-round overhead to reach the necessary consensus accuracy. We further propose an Accelerated Decentralized Cubic Newton method for strongly convex objectives and show that it matches the iteration complexity of the exact Accelerated Cubic Newton method, again with only additional polylogarithmic communication-round overhead. Finally, although the general method requires exchanging full $d \times d$ Hessian matrices, we show how it can be implemented for generalized linear models by transmitting only vectors, making the approach substantially more practical in high dimensions.