Fast Optimizer Benchmark

TL;DR

The paper introduces the Fast Optimizer Benchmark (FOB), a versatile and user-friendly tool for evaluating deep learning optimizers across various domains, supporting integration with existing HPO tools and offering comprehensive visualization features.

Contribution

The paper presents FOB, a new benchmarking tool with modular design, YAML configuration, SLURM support, and compatibility with hyperparameter optimization workflows.

Findings

FOB enables efficient comparison of optimizers across multiple domains.

The tool supports easy integration into custom training pipelines.

Demonstrated use case shows its effectiveness in optimizer evaluation.

Abstract

In this paper, we present the Fast Optimizer Benchmark (FOB), a tool designed for evaluating deep learning optimizers during their development. The benchmark supports tasks from multiple domains such as computer vision, natural language processing, and graph learning. The focus is on convenient usage, featuring human-readable YAML configurations, SLURM integration, and plotting utilities. FOB can be used together with existing hyperparameter optimization (HPO) tools as it handles training and resuming of runs. The modular design enables integration into custom pipelines, using it simply as a collection of tasks. We showcase an optimizer comparison as a usage example of our tool. FOB can be found on GitHub: https://github.com/automl/FOB.