ICLR Reproducibility Challenge Report (Padam : Closing The Generalization Gap Of Adaptive Gradient Methods in Training Deep Neural Networks)

TL;DR

This paper reproduces and reviews the PADAM optimizer, which aims to improve the generalization of adaptive gradient methods by bridging the gap with SGD, and discusses future research directions.

Contribution

It reproduces the PADAM optimizer, evaluates its performance, and provides insights and future directions for improving adaptive gradient methods.

Findings

PADAM improves generalization over traditional adaptive methods.

The partially adaptive parameter p influences optimizer performance.

Reproduction confirms PADAM's effectiveness in training deep neural networks.

Abstract

This work is a part of ICLR Reproducibility Challenge 2019, we try to reproduce the results in the conference submission PADAM: Closing The Generalization Gap of Adaptive Gradient Methods In Training Deep Neural Networks. Adaptive gradient methods proposed in past demonstrate a degraded generalization performance than the stochastic gradient descent (SGD) with momentum. The authors try to address this problem by designing a new optimization algorithm that bridges the gap between the space of Adaptive Gradient algorithms and SGD with momentum. With this method a new tunable hyperparameter called partially adaptive parameter p is introduced that varies between [0, 0.5]. We build the proposed optimizer and use it to mirror the experiments performed by the authors. We review and comment on the empirical analysis performed by the authors. Finally, we also propose a future direction for further study of Padam. Our code is available at: https://github.com/yashkant/Padam-Tensorflow