ZOOpt: Toolbox for Derivative-Free Optimization

TL;DR

ZOOpt is a user-friendly Python toolbox that offers efficient derivative-free optimization methods, including parallel and distributed solutions, tailored for high-dimensional, noisy machine learning problems like hyper-parameter tuning.

Contribution

The paper introduces ZOOpt, a versatile and scalable toolbox for derivative-free optimization, integrating parallel and distributed computing for complex machine learning tasks.

Findings

Supports high-dimensional, noisy optimization problems

Enables efficient parallel and distributed optimization

Aims for practical application in real-world machine learning

Abstract

Recent advances in derivative-free optimization allow efficient approximation of the global-optimal solutions of sophisticated functions, such as functions with many local optima, non-differentiable and non-continuous functions. This article describes the ZOOpt (Zeroth Order Optimization) toolbox that provides efficient derivative-free solvers and is designed easy to use. ZOOpt provides single-machine parallel optimization on the basis of python core and multi-machine distributed optimization for time-consuming tasks by incorporating with the Ray framework -- a famous platform for building distributed applications. ZOOpt particularly focuses on optimization problems in machine learning, addressing high-dimensional and noisy problems such as hyper-parameter tuning and direct policy search. The toolbox is maintained toward a ready-to-use tool in real-world machine learning tasks.