Kronecker-factored Quasi-Newton Methods for Deep Learning

TL;DR

This paper extends Kronecker-factored quasi-Newton methods to convolutional neural networks, improving efficiency and convergence, and demonstrates superior performance over first-order methods in deep learning tasks.

Contribution

It introduces a Kronecker-factored Hessian approximation for CNNs, with improved memory and time efficiency, and provides convergence analysis and empirical validation.

Findings

Outperforms first-order SOTA methods in CNN and MLP autoencoder tasks.

Achieves comparable results to second-order SOTA methods.

Reduces memory and per-iteration time complexity.

Abstract

Second-order methods have the capability of accelerating optimization by using much richer curvature information than first-order methods. However, most are impractical for deep learning, where the number of training parameters is huge. In Goldfarb et al. (2020), practical quasi-Newton methods were proposed that approximate the Hessian of a multilayer perceptron (MLP) model by a layer-wise block diagonal matrix where each layer's block is further approximated by a Kronecker product corresponding to the structure of the Hessian restricted to that layer. Here, we extend these methods to enable them to be applied to convolutional neural networks (CNNs), by analyzing the Kronecker-factored structure of the Hessian matrix of convolutional layers. Several improvements to the methods in Goldfarb et al. (2020) are also proposed that can be applied to both MLPs and CNNs. These new methods have memory requirements comparable to first-order methods and much less per-iteration time complexity than those in Goldfarb et al. (2020). Moreover, convergence results are proved for a variant under relatively mild conditions. Finally, we compared the performance of our new methods against several state-of-the-art (SOTA) methods on MLP autoencoder and CNN problems, and found that they outperformed the first-order SOTA methods and performed comparably to the second-order SOTA methods.