An Improved Empirical Fisher Approximation for Natural Gradient Descent

TL;DR

This paper introduces an improved empirical Fisher (iEF) method for natural gradient descent that enhances approximation quality, convergence, and robustness in deep learning optimization compared to existing EF and sampled Fisher methods.

Contribution

The paper proposes a novel iEF method addressing EF's limitations, providing better Fisher approximation and optimization performance in deep learning.

Findings

iEF outperforms EF and sampled Fisher in approximation quality.

Applying exact iEF as an optimizer yields strong convergence and generalization.

iEF demonstrates robustness to damping choices across tasks and training stages.

Abstract

Approximate Natural Gradient Descent (NGD) methods are an important family of optimisers for deep learning models, which use approximate Fisher information matrices to pre-condition gradients during training. The empirical Fisher (EF) method approximates the Fisher information matrix empirically by reusing the per-sample gradients collected during back-propagation. Despite its ease of implementation, the EF approximation has its theoretical and practical limitations. This paper investigates the inversely-scaled projection issue of EF, which is shown to be a major cause of its poor empirical approximation quality. An improved empirical Fisher (iEF) method is proposed to address this issue, which is motivated as a generalised NGD method from a loss reduction perspective, meanwhile retaining the practical convenience of EF. The exact iEF and EF methods are experimentally evaluated using practical deep learning setups. Optimisation experiments show that applying exact iEF directly as an optimiser provides strong convergence and generalisation. Additionally, under a novel empirical evaluation framework, the proposed iEF method shows consistently better approximation quality to exact Natural Gradient updates than both the EF and the more expensive sampled Fisher methods, meanwhile demonstrating the superior property of being robust to the choice of damping across tasks and training stages. Improving existing approximate NGD optimisers with iEF is expected to lead to better convergence and robustness. Furthermore, the iEF method also serves as a better approximation method to the Fisher information matrix itself, which enables the improvement of a variety of Fisher-based methods, not limited to the scope of optimisation.