DINGO: Distributed Newton-Type Method for Gradient-Norm Optimization

TL;DR

DINGO is a communication-efficient distributed Newton-type algorithm that optimizes the gradient norm, applicable to a wide range of functions, demonstrating strong theoretical guarantees and empirical performance.

Contribution

The paper introduces DINGO, a novel distributed Newton-type method that is hyper-parameter insensitive, versatile, and applicable beyond convex functions.

Findings

DINGO guarantees strict reduction in gradient norm regardless of hyper-parameter choices.

Empirical results show DINGO outperforms existing methods in effectiveness and stability.

DINGO is applicable to arbitrary data distributions and non-convex functions.

Abstract

For optimization of a sum of functions in a distributed computing environment, we present a novel communication efficient Newton-type algorithm that enjoys a variety of advantages over similar existing methods. Similar to Newton-MR, our algorithm, DINGO, is derived by optimization of the gradient's norm as a surrogate function. DINGO does not impose any specific form on the underlying functions, and its application range extends far beyond convexity. In addition, the distribution of the data across the computing environment can be arbitrary. Further, the underlying sub-problems of DINGO are simple linear least-squares, for which a plethora of efficient algorithms exist. Lastly, DINGO involves a few hyper-parameters that are easy to tune. Moreover, we theoretically show that DINGO is not sensitive to the choice of its hyper-parameters in that a strict reduction in the gradient norm is guaranteed, regardless of the selected hyper-parameters. We demonstrate empirical evidence of the effectiveness, stability and versatility of our method compared to other relevant algorithms.