Does Momentum Change the Implicit Regularization on Separable Data?

TL;DR

This paper investigates how momentum in optimization algorithms influences implicit regularization, showing that momentum-based methods still converge to low-complexity solutions similar to vanilla gradient descent on separable data.

Contribution

It provides the first theoretical analysis demonstrating that momentum-based optimization algorithms converge to the L2 max-margin solution on separable data.

Findings

Gradient descent with momentum converges to the L2 max-margin solution.

Stochastic and adaptive variants like SGDM and Adam also converge to the max-margin solution.

Numerical experiments support the theoretical convergence results.

Abstract

The momentum acceleration technique is widely adopted in many optimization algorithms. However, there is no theoretical answer on how the momentum affects the generalization performance of the optimization algorithms. This paper studies this problem by analyzing the implicit regularization of momentum-based optimization. We prove that on the linear classification problem with separable data and exponential-tailed loss, gradient descent with momentum (GDM) converges to the L2 max-margin solution, which is the same as vanilla gradient descent. That means gradient descent with momentum acceleration still converges to a low-complexity model, which guarantees their generalization. We then analyze the stochastic and adaptive variants of GDM (i.e., SGDM and deterministic Adam) and show they also converge to the L2 max-margin solution. Technically, to overcome the difficulty of the error accumulation in analyzing the momentum, we construct new potential functions to analyze the gap between the model parameter and the max-margin solution. Numerical experiments are conducted and support our theoretical results.