On Convergence Analysis of Network-GIANT: An approximate Hessian-based fully distributed optimization algorithm

TL;DR

This paper provides a detailed convergence analysis of Network-GIANT, a distributed optimization algorithm that uses approximate Hessian information, demonstrating its faster linear convergence compared to first-order methods under certain conditions.

Contribution

It explicitly characterizes the global linear convergence rate of Network-GIANT and derives bounds on step size and optimality gap, explaining its faster convergence.

Findings

Network-GIANT achieves faster linear convergence than first-order methods.

Explicit spectral radius-based convergence rate depends on problem and graph parameters.

Numerical experiments confirm theoretical convergence properties.

Abstract

In this paper, we present a detailed convergence analysis of a recently developed approximate Newton-type fully distributed optimization method for smooth, strongly convex local loss functions, called Network-GIANT, which has been empirically illustrated to show faster linear convergence properties while having the same communication complexity (per iteration) as its first order distributed counterparts. By using consensus based parameter updates, and a local Hessian based descent direction at the individual nodes with gradient tracking, we first explicitly characterize a global linear convergence rate for Network-GIANT, which can be computed as the spectral radius of a $3 \times 3$ matrix dependent on the Lipschitz continuity ($L$) and strong convexity ($\mu$) parameters of the objective functions, and the spectral norm ($\sigma$) of the underlying undirected graph represented by a doubly stochastic consensus matrix. We provide an explicit bound on the step size parameter $\eta$, below which this spectral radius is guaranteed to be less than $1$. Furthermore, we derive a mixed linear-quadratic inequality based upper bound for the optimality gap norm, which allows us to conclude that, under small step size values, asymptotically, as the algorithm approaches the global optimum, it achieves a locally linear convergence rate of $1-\eta(1 -\frac{\gamma}{\mu})$ for Network-GIANT, provided the Hessian approximation error $\gamma$ (between the harmonic mean of the local Hessians and the global hessian (the arithmetic mean of the local Hessians) is smaller than $\mu$. This asymptotically linear convergence rate of $\approx 1-\eta$ explains the faster convergence rate of Network-GIANT for the first time. Numerical experiments are carried out with a reduced CovType dataset for binary logistic regression over a variety of graphs to illustrate the above theoretical results.