Compressed Gradient Methods with Hessian-Aided Error Compensation

TL;DR

This paper introduces Hessian-aided error compensation for gradient compression in optimization, providing theoretical convergence guarantees and demonstrating improved accuracy and efficiency, especially for quadratic problems.

Contribution

It presents a novel Hessian-based error compensation method that avoids error accumulation and offers strong convergence guarantees for stochastic gradient descent.

Findings

Hessian-aided error compensation prevents error accumulation in quadratic problems.

The method improves convergence accuracy in stochastic gradient descent.

Diagonal Hessian approximation still achieves similar convergence benefits.

Abstract

The emergence of big data has caused a dramatic shift in the operating regime for optimization algorithms. The performance bottleneck, which used to be computations, is now often communications. Several gradient compression techniques have been proposed to reduce the communication load at the price of a loss in solution accuracy. Recently, it has been shown how compression errors can be compensated for in the optimization algorithm to improve the solution accuracy. Even though convergence guarantees for error-compensated algorithms have been established, there is very limited theoretical support for quantifying the observed improvements in solution accuracy. In this paper, we show that Hessian-aided error compensation, unlike other existing schemes, avoids the accumulation of compression errors on quadratic problems. We also present strong convergence guarantees of Hessian-based error compensation for stochastic gradient descent. Our numerical experiments highlight the benefits of Hessian-based error compensation, and demonstrate that similar convergence improvements are attained when only a diagonal Hessian approximation is used.