Linearization of ReLU Activation Function for Neural Network-Embedded Optimization: Optimal Day-Ahead Energy Scheduling

TL;DR

This paper develops and compares four linearization methods for ReLU activation functions to enable efficient neural network-embedded optimization in power system energy scheduling.

Contribution

It introduces novel linearization techniques for ReLU functions, facilitating the integration of neural networks into optimization models for energy scheduling.

Findings

Four linearization methods are developed and analyzed.

The methods effectively linearize ReLU, simplifying neural network-embedded optimization.

The approaches improve computational efficiency in energy scheduling problems.

Abstract

Recently, neural networks have been widely applied in the power system area. They can be used for better predicting input information and modeling system performance with increased accuracy. In some applications such as battery degradation neural network-based microgrid day-ahead energy scheduling, the input features of the trained learning model are variables to be solved in optimization models that enforce limits on the output of the same learning model. This will create a neural network-embedded optimization problem; the use of nonlinear activation functions in the neural network will make such problems extremely hard to solve if not unsolvable. To address this emerging challenge, this paper investigated different methods for linearizing the nonlinear activation functions with a particular focus on the widely used rectified linear unit (ReLU) function. Four linearization methods tailored for the ReLU activation function are developed, analyzed and compared in this paper. Each method employs a set of linear constraints to replace the ReLU function, effectively linearizing the optimization problem, which can overcome the computational challenges associated with the nonlinearity of the neural network model. These proposed linearization methods provide valuable tools for effectively solving optimization problems that integrate neural network models with ReLU activation functions