HesScale: Scalable Computation of Hessian Diagonals

TL;DR

HesScale introduces a scalable method for efficiently approximating Hessian diagonals, enabling second-order optimization in large models without significant computational overhead.

Contribution

HesScale provides a novel, computationally efficient approach to approximate Hessian diagonals, facilitating scalable second-order optimization.

Findings

HesScale achieves high approximation accuracy.

It has the same complexity as backpropagation.

Enables scalable second-order optimization.

Abstract

Second-order optimization uses curvature information about the objective function, which can help in faster convergence. However, such methods typically require expensive computation of the Hessian matrix, preventing their usage in a scalable way. The absence of efficient ways of computation drove the most widely used methods to focus on first-order approximations that do not capture the curvature information. In this paper, we develop HesScale, a scalable approach to approximating the diagonal of the Hessian matrix, to incorporate second-order information in a computationally efficient manner. We show that HesScale has the same computational complexity as backpropagation. Our results on supervised classification show that HesScale achieves high approximation accuracy, allowing for scalable and efficient second-order optimization.