On the Performance Analysis of Momentum Method: A Frequency Domain Perspective

TL;DR

This paper introduces a frequency domain analysis framework for momentum-based optimizers, revealing how adjusting momentum affects gradient filtering and proposing a new dynamic optimizer that improves training performance.

Contribution

It provides a novel frequency domain perspective on momentum methods, offering insights into optimal momentum coefficient selection and introducing the FSGDM optimizer.

Findings

High-frequency gradients are undesirable in late training stages.

Preserving initial gradients and amplifying low-frequency components improves performance.

FSGDM outperforms traditional momentum optimizers in experiments.

Abstract

Momentum-based optimizers are widely adopted for training neural networks. However, the optimal selection of momentum coefficients remains elusive. This uncertainty impedes a clear understanding of the role of momentum in stochastic gradient methods. In this paper, we present a frequency domain analysis framework that interprets the momentum method as a time-variant filter for gradients, where adjustments to momentum coefficients modify the filter characteristics. Our experiments support this perspective and provide a deeper understanding of the mechanism involved. Moreover, our analysis reveals the following significant findings: high-frequency gradient components are undesired in the late stages of training; preserving the original gradient in the early stages, and gradually amplifying low-frequency gradient components during training both enhance performance. Based on these insights, we propose Frequency Stochastic Gradient Descent with Momentum (FSGDM), a heuristic optimizer that dynamically adjusts the momentum filtering characteristic with an empirically effective dynamic magnitude response. Experimental results demonstrate the superiority of FSGDM over conventional momentum optimizers.