On a Conjecture Regarding the Adam Optimizer

TL;DR

This paper investigates the theoretical foundations of the Adam optimizer, disproves a key conjecture, and proposes a modified version that can underpin future analyses of Adam's effectiveness.

Contribution

It refutes Bock's conjecture about Adam and introduces a generalized version that can replace it in theoretical explanations.

Findings

Bock's conjecture is false.

A modified, generalized conjecture is proven.

The new conjecture supports Adam's theoretical analysis.

Abstract

Why does the Adam optimizer work so well in deep-learning applications? Adam's originators, Kingma and Ba, presented a mathematical argument that was meant to help explain its success, but Bock and colleagues have since reported that a key piece is missing from that argument $-$ an unproven lemma which we will call Bock's conjecture. Here we show that this conjecture is false, but we prove a modified version of it $-$ a generalization of a result of Reddi and colleagues $-$ which can take its place in analyses of Adam.