On Empirical Comparisons of Optimizers for Deep Learning

TL;DR

This paper shows that hyperparameter tuning protocols critically influence optimizer comparison results in deep learning, revealing that adaptive methods generally outperform momentum-based optimizers when properly tuned.

Contribution

It demonstrates the importance of hyperparameter search spaces in optimizer comparisons and highlights that inclusion relationships between optimizers are practically significant.

Findings

Adaptive gradient methods outperform momentum and gradient descent when hyperparameters are properly tuned.

Changing hyperparameter search spaces can invert optimizer rankings.

Proper tuning and search space design are crucial for fair optimizer benchmarking.

Abstract

Selecting an optimizer is a central step in the contemporary deep learning pipeline. In this paper, we demonstrate the sensitivity of optimizer comparisons to the hyperparameter tuning protocol. Our findings suggest that the hyperparameter search space may be the single most important factor explaining the rankings obtained by recent empirical comparisons in the literature. In fact, we show that these results can be contradicted when hyperparameter search spaces are changed. As tuning effort grows without bound, more general optimizers should never underperform the ones they can approximate (i.e., Adam should never perform worse than momentum), but recent attempts to compare optimizers either assume these inclusion relationships are not practically relevant or restrict the hyperparameters in ways that break the inclusions. In our experiments, we find that inclusion relationships between optimizers matter in practice and always predict optimizer comparisons. In particular, we find that the popular adaptive gradient methods never underperform momentum or gradient descent. We also report practical tips around tuning often ignored hyperparameters of adaptive gradient methods and raise concerns about fairly benchmarking optimizers for neural network training.