A generic and fast C++ optimization framework

TL;DR

This paper introduces a flexible, efficient C++ optimization framework for machine learning that simplifies the addition of new algorithms and objective functions, supporting diverse problem types and enhancing rapid prototyping.

Contribution

It presents a generic, easy-to-extend optimization framework for mlpack, enabling quick implementation of new optimizers and objective functions with minimal code.

Findings

Supports more objective function types than other frameworks

Enables optimizations not possible in other libraries

Facilitates rapid prototyping and experimentation

Abstract

The development of the mlpack C++ machine learning library (http://www.mlpack.org/) has required the design and implementation of a flexible, robust optimization system that is able to solve the types of arbitrary optimization problems that may arise all throughout machine learning problems. In this paper, we present the generic optimization framework that we have designed for mlpack. A key priority in the design was ease of implementation of both new optimizers and new objective functions to be optimized; therefore, implementation of a new optimizer requires only one method and implementation of a new objective function requires at most four functions. This leads to simple and intuitive code, which, for fast prototyping and experimentation, is of paramount importance. When compared to optimization frameworks of other libraries, we find that mlpack's supports more types of objective functions, is able to make optimizations that other frameworks do not, and seamlessly supports user-defined objective functions and optimizers.