A Mini-Block Fisher Method for Deep Neural Networks

TL;DR

The paper introduces a mini-block Fisher (MBF) method for training deep neural networks, combining second-order information with computational efficiency, and demonstrates its effectiveness and convergence properties.

Contribution

It proposes a novel mini-block Fisher preconditioning method that balances second-order accuracy with computational efficiency using GPU parallelism.

Findings

MBF improves training efficiency over first-order methods.

MBF enhances generalization in autoencoder and CNN tasks.

An idealized MBF converges linearly.

Abstract

Deep neural networks (DNNs) are currently predominantly trained using first-order methods. Some of these methods (e.g., Adam, AdaGrad, and RMSprop, and their variants) incorporate a small amount of curvature information by using a diagonal matrix to precondition the stochastic gradient. Recently, effective second-order methods, such as KFAC, K-BFGS, Shampoo, and TNT, have been developed for training DNNs, by preconditioning the stochastic gradient by layer-wise block-diagonal matrices. Here we propose a "mini-block Fisher (MBF)" preconditioned gradient method, that lies in between these two classes of methods. Specifically, our method uses a block-diagonal approximation to the empirical Fisher matrix, where for each layer in the DNN, whether it is convolutional or feed-forward and fully connected, the associated diagonal block is itself block-diagonal and is composed of a large number of mini-blocks of modest size. Our novel approach utilizes the parallelism of GPUs to efficiently perform computations on the large number of matrices in each layer. Consequently, MBF's per-iteration computational cost is only slightly higher than it is for first-order methods. The performance of our proposed method is compared to that of several baseline methods, on both autoencoder and CNN problems, to validate its effectiveness both in terms of time efficiency and generalization power. Finally, it is proved that an idealized version of MBF converges linearly.