CDiNN -Convex Difference Neural Networks

TL;DR

This paper introduces CDiNN, a neural network architecture that models functions as differences of convex functions, enabling efficient convex optimization techniques for applications like optimal control.

Contribution

The paper proposes CDiNN, a novel neural network architecture that learns functions as differences of convex functions, addressing limitations of ICNNs in modeling simple dynamic structures.

Findings

CDiNN can be trained to approximate functions as differences of convex functions.

Optimal inputs can be obtained via difference of convex optimization with convergence guarantees.

Each optimization iteration reduces to a linear programming problem.

Abstract

Neural networks with ReLU activation function have been shown to be universal function approximators and learn function mapping as non-smooth functions. Recently, there is considerable interest in the use of neural networks in applications such as optimal control. It is well-known that optimization involving non-convex, non-smooth functions are computationally intensive and have limited convergence guarantees. Moreover, the choice of optimization hyper-parameters used in gradient descent/ascent significantly affect the quality of the obtained solutions. A new neural network architecture called the Input Convex Neural Networks (ICNNs) learn the output as a convex function of inputs thereby allowing the use of efficient convex optimization methods. Use of ICNNs for determining the input for minimizing output has two major problems: learning of a non-convex function as a convex mapping could result in significant function approximation error, and we also note that the existing representations cannot capture simple dynamic structures like linear time delay systems. We attempt to address the above problems by introduction of a new neural network architecture, which we call the CDiNN, which learns the function as a difference of polyhedral convex functions from data. We also discuss that, in some cases, the optimal input can be obtained from CDiNN through difference of convex optimization with convergence guarantees and that at each iteration, the problem is reduced to a linear programming problem.