DeepOBS: A Deep Learning Optimizer Benchmark Suite

TL;DR

DeepOBS is an open-source Python benchmarking suite for deep learning optimizers, providing standardized, reproducible evaluation protocols and a wide range of realistic problems to facilitate fair comparison and research advancement.

Contribution

It introduces DeepOBS, a comprehensive benchmarking tool that automates evaluation of deep learning optimizers on diverse, realistic problems with baseline results for fair comparison.

Findings

Provides baseline results for popular optimizers on benchmark problems

Automates benchmarking process for reproducibility and fairness

Includes diverse real-world deep learning tasks like ImageNet and language modeling

Abstract

Because the choice and tuning of the optimizer affects the speed, and ultimately the performance of deep learning, there is significant past and recent research in this area. Yet, perhaps surprisingly, there is no generally agreed-upon protocol for the quantitative and reproducible evaluation of optimization strategies for deep learning. We suggest routines and benchmarks for stochastic optimization, with special focus on the unique aspects of deep learning, such as stochasticity, tunability and generalization. As the primary contribution, we present DeepOBS, a Python package of deep learning optimization benchmarks. The package addresses key challenges in the quantitative assessment of stochastic optimizers, and automates most steps of benchmarking. The library includes a wide and extensible set of ready-to-use realistic optimization problems, such as training Residual Networks for image classification on ImageNet or character-level language prediction models, as well as popular classics like MNIST and CIFAR-10. The package also provides realistic baseline results for the most popular optimizers on these test problems, ensuring a fair comparison to the competition when benchmarking new optimizers, and without having to run costly experiments. It comes with output back-ends that directly produce LaTeX code for inclusion in academic publications. It supports TensorFlow and is available open source.