The Implicit Bias of Adam on Separable Data

TL;DR

This paper investigates the implicit bias of the Adam optimizer in linear logistic regression, showing it converges to a maximum $ ext{l}_ ext{infty}$-margin classifier on separable data within polynomial time, clarifying its theoretical properties.

Contribution

It provides the first theoretical analysis of Adam's implicit bias, demonstrating convergence to a maximum margin classifier on linearly separable data.

Findings

Adam converges to a maximum $ ext{l}_ ext{infty}$-margin classifier.

Convergence occurs within polynomial time for diminishing learning rates.

Theoretical distinction between Adam and gradient descent is clarified.

Abstract

Adam has become one of the most favored optimizers in deep learning problems. Despite its success in practice, numerous mysteries persist regarding its theoretical understanding. In this paper, we study the implicit bias of Adam in linear logistic regression. Specifically, we show that when the training data are linearly separable, Adam converges towards a linear classifier that achieves the maximum $\ell_\infty$-margin. Notably, for a general class of diminishing learning rates, this convergence occurs within polynomial time. Our result shed light on the difference between Adam and (stochastic) gradient descent from a theoretical perspective.