EA-CG: An Approximate Second-Order Method for Training Fully-Connected Neural Networks

TL;DR

This paper introduces EA-CG, a memory-efficient approximate second-order training method for fully-connected neural networks that leverages conjugate gradient techniques to reduce computational complexity while maintaining effectiveness.

Contribution

The paper presents a novel approximate Hessian and a CG-based method for efficient second-order training of FCNNs, applicable to any twice-differentiable activation and criterion functions.

Findings

Effective in reducing training time and memory usage.

Achieves comparable performance to exact second-order methods.

Applicable to a wide range of FCNN architectures.

Abstract

For training fully-connected neural networks (FCNNs), we propose a practical approximate second-order method including: 1) an approximation of the Hessian matrix and 2) a conjugate gradient (CG) based method. Our proposed approximate Hessian matrix is memory-efficient and can be applied to any FCNNs where the activation and criterion functions are twice differentiable. We devise a CG-based method incorporating one-rank approximation to derive Newton directions for training FCNNs, which significantly reduces both space and time complexity. This CG-based method can be employed to solve any linear equation where the coefficient matrix is Kronecker-factored, symmetric and positive definite. Empirical studies show the efficacy and efficiency of our proposed method.