DeepOBS: A Deep Learning Optimizer Benchmark Suite
Frank Schneider, Lukas Balles, Philipp Hennig

TL;DR
DeepOBS is an open-source Python benchmarking suite for deep learning optimizers, providing standardized, reproducible evaluation protocols and a wide range of realistic problems to facilitate fair comparison and research advancement.
Contribution
It introduces DeepOBS, a comprehensive benchmarking tool that automates evaluation of deep learning optimizers on diverse, realistic problems with baseline results for fair comparison.
Findings
Provides baseline results for popular optimizers on benchmark problems
Automates benchmarking process for reproducibility and fairness
Includes diverse real-world deep learning tasks like ImageNet and language modeling
Abstract
Because the choice and tuning of the optimizer affects the speed, and ultimately the performance of deep learning, there is significant past and recent research in this area. Yet, perhaps surprisingly, there is no generally agreed-upon protocol for the quantitative and reproducible evaluation of optimization strategies for deep learning. We suggest routines and benchmarks for stochastic optimization, with special focus on the unique aspects of deep learning, such as stochasticity, tunability and generalization. As the primary contribution, we present DeepOBS, a Python package of deep learning optimization benchmarks. The package addresses key challenges in the quantitative assessment of stochastic optimizers, and automates most steps of benchmarking. The library includes a wide and extensible set of ready-to-use realistic optimization problems, such as training Residual Networks for…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Neural Network Applications · Machine Learning and Data Classification · Machine Learning and Algorithms
