Unit Tests for Stochastic Optimization

TL;DR

This paper introduces a set of unit tests designed to evaluate the robustness and generality of stochastic optimization algorithms on simplified, well-understood problems, providing a quick assessment tool for algorithm reliability.

Contribution

The paper presents an open-source, extensible testing framework for stochastic optimization algorithms, enabling rapid evaluation on isolated difficulties to assess robustness.

Findings

Initial tests show varying robustness among established algorithms.

The framework effectively distinguishes algorithms based on their ability to handle specific challenges.

Open-source tool facilitates widespread adoption and further development.

Abstract

Optimization by stochastic gradient descent is an important component of many large-scale machine learning algorithms. A wide variety of such optimization algorithms have been devised; however, it is unclear whether these algorithms are robust and widely applicable across many different optimization landscapes. In this paper we develop a collection of unit tests for stochastic optimization. Each unit test rapidly evaluates an optimization algorithm on a small-scale, isolated, and well-understood difficulty, rather than in real-world scenarios where many such issues are entangled. Passing these unit tests is not sufficient, but absolutely necessary for any algorithms with claims to generality or robustness. We give initial quantitative and qualitative results on numerous established algorithms. The testing framework is open-source, extensible, and easy to apply to new algorithms.