Universal Approximation of Parametric Optimization via Neural Networks with Piecewise Linear Policy Approximation

TL;DR

This paper establishes theoretical foundations for approximating optimal policies in parametric optimization using neural networks with ReLU activations, focusing on piecewise linear policy approximation and its properties.

Contribution

It provides the first formal analysis of piecewise linear policy approximation via neural networks for parametric optimization, including conditions for universal approximation and strategies for feasibility.

Findings

Neural networks can approximate piecewise linear policies with guarantees on generalization.

Conditions are derived for the universal approximation of parametric optimization policies.

Strategies are proposed to enhance the feasibility of approximate solutions.

Abstract

Parametric optimization solves a family of optimization problems as a function of parameters. It is a critical component in situations where optimal decision making is repeatedly performed for updated parameter values, but computation becomes challenging when complex problems need to be solved in real-time. Therefore, in this study, we present theoretical foundations on approximating optimal policy of parametric optimization problem through Neural Networks and derive conditions that allow the Universal Approximation Theorem to be applied to parametric optimization problems by constructing piecewise linear policy approximation explicitly. This study fills the gap on formally analyzing the constructed piecewise linear approximation in terms of feasibility and optimality and show that Neural Networks (with ReLU activations) can be valid approximator for this approximation in terms of generalization and approximation error. Furthermore, based on theoretical results, we propose a strategy to improve feasibility of approximated solution and discuss training with suboptimal solutions.