Optimization with Trained Machine Learning Models Embedded

TL;DR

This paper discusses embedding trained machine learning models into optimization problems, highlighting the challenges and potential for specialized solution strategies to improve efficiency in large-scale NLPs.

Contribution

It reviews recent reformulation approaches and emphasizes the need for developing more efficient methods to handle evolving ML architectures in optimization.

Findings

Homogeneous structures in ML models can be exploited for optimization

Recent reformulations using MIP and reduced space improve solvability

Further research is needed for scalable solutions with new ML architectures

Abstract

Trained ML models are commonly embedded in optimization problems. In many cases, this leads to large-scale NLPs that are difficult to solve to global optimality. While ML models frequently lead to large problems, they also exhibit homogeneous structures and repeating patterns (e.g., layers in ANNs). Thus, specialized solution strategies can be used for large problem classes. Recently, there have been some promising works proposing specialized reformulations using mixed-integer programming or reduced space formulations. However, further work is needed to develop more efficient solution approaches and keep up with the rapid development of new ML model architectures.