The ensmallen library for flexible numerical optimization

TL;DR

The ensmallen library offers a flexible, efficient C++ framework for diverse numerical optimization tasks, supporting various objective functions and optimizers, with empirical evidence of superior performance.

Contribution

This paper introduces the ensmallen library, a versatile C++ framework that simplifies implementing and customizing a wide range of optimization algorithms.

Findings

Outperforms other frameworks in empirical tests

Supports many types of objective functions and optimizers

Facilitates easy implementation of new algorithms

Abstract

We overview the ensmallen numerical optimization library, which provides a flexible C++ framework for mathematical optimization of user-supplied objective functions. Many types of objective functions are supported, including general, differentiable, separable, constrained, and categorical. A diverse set of pre-built optimizers is provided, including Quasi-Newton optimizers and many variants of Stochastic Gradient Descent. The underlying framework facilitates the implementation of new optimizers. Optimization of an objective function typically requires supplying only one or two C++ functions. Custom behavior can be easily specified via callback functions. Empirical comparisons show that ensmallen outperforms other frameworks while providing more functionality. The library is available at https://ensmallen.org and is distributed under the permissive BSD license.