FAdam: Adam is a natural gradient optimizer using diagonal empirical Fisher information

TL;DR

This paper provides a theoretical foundation for Adam as a natural gradient method, introduces FAdam with corrections and enhancements, and demonstrates its superior performance across multiple domains including language models and speech recognition.

Contribution

It establishes a mathematical connection between Adam and natural gradient descent, proposes corrections to Adam, and introduces FAdam with improved performance.

Findings

FAdam outperforms Adam in various tasks.

Theoretical insights clarify Adam's limitations.

Enhanced FAdam achieves state-of-the-art results in ASR.

Abstract

This paper establishes a mathematical foundation for the Adam optimizer, elucidating its connection to natural gradient descent through Riemannian and information geometry. We provide an accessible and detailed analysis of the diagonal empirical Fisher information matrix (FIM) in Adam, clarifying all detailed approximations and advocating for the use of log probability functions as loss, which should be based on discrete distributions, due to the limitations of empirical FIM. Our analysis uncovers flaws in the original Adam algorithm, leading to proposed corrections such as enhanced momentum calculations, adjusted bias corrections, adaptive epsilon, and gradient clipping. We refine the weight decay term based on our theoretical framework. Our modified algorithm, Fisher Adam (FAdam), demonstrates superior performance across diverse domains including LLM, ASR, and VQ-VAE, achieving state-of-the-art results in ASR.