A Universal Approximation Result for Difference of log-sum-exp Neural Networks

TL;DR

This paper proves that a specific class of neural networks, called Difference-LSE networks, can universally approximate continuous functions over convex sets and are particularly useful for optimization-based design tasks.

Contribution

It introduces Difference-LSE networks as universal approximators with a difference-of-convex structure, enabling efficient optimization in design applications.

Findings

Difference-LSE networks are universal approximators of continuous functions.

They can be optimized efficiently using difference-of-convex algorithms.

Application demonstrated in designing a diet for a type-2 diabetes patient.

Abstract

We show that a neural network whose output is obtained as the difference of the outputs of two feedforward networks with exponential activation function in the hidden layer and logarithmic activation function in the output node (LSE networks) is a smooth universal approximator of continuous functions over convex, compact sets. By using a logarithmic transform, this class of networks maps to a family of subtraction-free ratios of generalized posynomials, which we also show to be universal approximators of positive functions over log-convex, compact subsets of the positive orthant. The main advantage of Difference-LSE networks with respect to classical feedforward neural networks is that, after a standard training phase, they provide surrogate models for design that possess a specific difference-of-convex-functions form, which makes them optimizable via relatively efficient numerical methods. In particular, by adapting an existing difference-of-convex algorithm to these models, we obtain an algorithm for performing effective optimization-based design. We illustrate the proposed approach by applying it to data-driven design of a diet for a patient with type-2 diabetes.

Figures

Tables

Table 1. TABLE I: Prediction errors over 𝒟 valid subscript 𝒟 valid {\mathcal{D}}_{\mathrm{valid}}

Method	Mean Sq.	Mean Rel.	Max Abs.	Max Rel.	r2
	$\left[\mathrm{mg}/\mathrm{dL}\right]$	$[-]$	$\left[\mathrm{mg}/\mathrm{dL}\right]$	$[-]$	$[-]$
$\mathrm{DLSE}$	$0.0624$	$0.0006$	$1.1043$	$0.0043$	$0.9999$
$\mathrm{FFNN}$	$2.3145$	$0.0041$	$7.9480$	$0.0311$	$0.9959$

Equations

$$f(\mathbf{x})=\log \left(\sum _{k=1}^K{b}_k\exp (⟨{\mathbf{\alpha }}^{(k)},\mathbf{x}⟩)\right),$$

$$f_T(\mathbf{x})=T\log \left(\sum _{k=1}^K{b}_k^{1/T}\exp (⟨{\mathbf{\alpha }}^{(k)},\mathbf{x}/T⟩)\right),$$

$$f_T(\mathbf{x})=T\log \left(\sum _{k=1}^K\exp (⟨{\mathbf{\alpha }}^{(k)},\mathbf{x}/T⟩+{\beta }_k/T)\right),$$

$$f_T^{(\overrightarrow{\mathbf{\alpha }},\mathbf{\beta })}(\mathbf{x})=T{f}_1^{(\overrightarrow{\mathbf{\alpha }},\mathbf{\beta }/T)}(\mathbf{x}/T).$$

$$f_{0}(\mathbf{x})\doteq\max_{1\leqslant k\leqslant K}\big{(}\langle\bm{\alpha}^{(k)},\mathbf{x}\rangle+\beta_{k}\big{)}.$$

$$f_0(\mathbf{x})⩽f_T(\mathbf{x})⩽T\log K+f_0(\mathbf{x}),$$

$$\psi (\mathbf{x})=\sum _{k=1}^K{c}_k{\mathbf{x}}^{{\mathbf{\alpha }}^{(k)}}.$$

$${\psi }_T(\mathbf{x})=(\psi ({\mathbf{x}}^{1/T}){)}^T$$

$$\exp \left(f\left(\log (\mathbf{z})\right)\right)$$

$$|f_T(\mathbf{x})-\varphi (\mathbf{x})|⩽\epsilon ,\text{ for all }\mathbf{x}\in \mathcal{K}.$$

$$|\phi (\mathbf{\xi })-\phi ({\mathbf{\xi }}^{\prime })|⩽\Vert \mathbf{\xi }-{\mathbf{\xi }}^{\prime }{\Vert }_{\infty }$$

$$g_{T}(\mathbf{x})=T\log\left(\sum_{k=1}^{K}\exp\Big{(}\langle\bm{\gamma}^{(k)},\mathbf{x}/T\rangle+\delta_{k}/T\Big{)}\right),$$

$$|f_T(\mathbf{x})-g_T(\mathbf{x})|$$

$$\doteq \underset{1⩽k⩽K}{\max }\Vert {\mathbf{\alpha }}^{(k)}-{\mathbf{\gamma }}^{(k)}\Vert R+\underset{1⩽k⩽K}{\max }|{\beta }_k-{\delta }_k|,$$

$$|g(\mathbf{x})-g_T(\mathbf{x})|$$

$$\varphi (x)=\max (0,\min (x,1)).$$

$$\max (0,x)$$

$$⩽T\log 2+\max (0,x),$$

$$\max (0,x-1)$$

$$⩽T\log 2+\max (0,x-1).$$

$$f_{T}\doteq T(\log(1+\exp(x/T)\big{)}-\log\big{(}1+\exp({(x-1)}/T)))$$

$$-T\log 2+f_T(x)⩽\varphi (x)⩽T\log 2+f_T(x),\forall x\in {\mathbb{R}}^n.$$

$$\varphi (x)=ax+b+\sum _{1⩽i⩽K}{\alpha }_i\max (0,x-{\gamma }_i)$$

$$I^{+}$$

$$I^{-}$$

$${\varphi }^{\pm }(x)=\sum _{i\in I^{\pm }}|{\alpha }_i|\max (0,x-{\gamma }_i)$$

$$\varphi (x)={\varphi }^{+}(x)-{\varphi }^{-}(x).$$

$$a_i$$

$$b_i$$

$${\varphi }^{\pm }(x)=\underset{i\in I^{\pm }}{\max }(a_{i}x-b_i).$$

$$f_T^{\pm }(x)=T\log \left(\sum _{i\in I^{\pm }}\exp ((a_{i}x-b_i)/T)\right)$$

$$f_T-T\log |I^{-}|⩽f⩽f_T+T\log |I^{+}|.$$

$$\mathcal{D}=\{({\mathbf{x}}_i,y_i){\}}_{i=1}^m$$

Full text

I Introduction

I-A Motivation

A well-known and compelling property of feedforward neural network ($\mathrm{FFNN}$) models is that they are capable of approximating any continuous function over a compact set. Indeed, classical results in, e.g., [1, 2], show that, given any non-constant, bounded and continuous function, there exists a $\mathrm{FFNN}$ with a single hidden layer that can approximate it over a compact set. This universal approximation capability, together with the development of efficient algorithms to tune the network weights, paved the way to the efficient application of artificial neural networks in several frameworks, such as circuit design [3], control and identification of nonlinear systems [4], optimization over graphs [5], and many others.

However, when the goal of a neural network model is to construct a surrogate model for describing, and then optimizing, a complex input-output relation, it is of crucial importance that the structure of the model be well tailored for the subsequent numerical optimization phase. This is not usually the case for generic $\mathrm{FFNN}$. Indeed, if the input-output model does not satisfy certain properties (such as, for instance, convexity [6]), then designing the input so that the output is minimized, possibly under additional design constraints, can be an extremely difficult task.

In [7], we showed that $\mathrm{LSE}_{T}$-networks, that is, $\mathrm{FFNN}$ with exponential activation functions in the inner layer and logarithmic activation function in the output neuron, parametrized by a positive “temperature” parameter $T>0$, provide a smooth convex model capable of approximating any convex function over a convex, compact set. Maps in the $\mathrm{LSE}_{T}$ class are precisely log-Laplace transforms of nonnegative measures with finite support, a remarkable class of maps enjoying smoothness and strict convexity properties. We showed in particular that if the data to be approximated satisfy some convexity assumptions, such network structures can be readily exploited to perform data-driven design by using convex optimization tools. Nevertheless, most real-life input-output maps are of nonconvex nature, hence while they might still be approximated via a convex model, such an approximation may not yield a desirable accuracy.

I-B Contribution

The purpose of this paper is to propose a new type of neural network model, here named Difference-LSE network ($\mathrm{DLSE}_{T}$), which is constructed by taking the difference of the outputs of two $\mathrm{LSE}_{T}$ networks. First, we prove that $\mathrm{DLSE}_{T}$ networks guarantee universal approximation capabilities (thus overcoming the limitations of plain convex $\mathrm{LSE}_{T}$ networks), see 2. By using a logarithmic transformation, $\mathrm{DLSE}_{T}$ networks map to a family of ratios of generalized posynomials functions, which we show to be subtraction free universal approximators of positive functions over compact subsets of the positive orthant. Subtraction free expressions are fundamental objects in algebraic complexity, studied in particular in [8]. It is a result of independent interest that subtraction free expressions provide universal approximators.

Moreover, we show that Difference-LSE network are of practical interest, as they have a structure which is amenable to effective optimization over the inputs by using “DC-programming” methods, as discussed in Section VI.

Training and subsequent optimization of $\mathrm{DLSE}_{T}$ networks has been implemented in a numerical Matlab toolbox111See https://github.com/Corrado-possieri/DLSE_neural_networks. named DLSE_Neural_Network that we made publicly available. The theoretical results in the paper are illustrated by an example dealing with data-driven design of a diet for a patient with type 2 diabetes.

I-C Related work

Similar to our previous work [7] on which it builds, the present paper is inspired by ideas from tropical geometry and max-plus algebra. The class of functions in $\mathrm{LSE}$ that we study here plays a key role in Viro’s patchworking methods [9, 10] for real curves. We note that the application of tropical geometry to neural networks is an emerging topic: at least two recent works have used tropical methods to provide combinatorial estimates, in terms of Newton polytopes, of the “classifying power” of neural networks with piecewise affine functions, see [11], [12]. Other related results concern the “zero-temperature” ($T=0$) limit of the approximation problem that we consider, i.e., the representation of piecewise linear functions by elementary expression involving min, max, and affine terms [13, 14], and the approximation of functions by piecewise linear functions. In particular, Th. 4.3 of [15] shows that any continuous function can be approximately arbitrarily well on a compact domain by a difference of piecewise linear convex functions. A related approximation result, still concerning differences of structured piecewise linear convex functions, has appeared in [16]. In contrast, our results provide approximation by differences of a family of smooth structured convex functions. This smoothing is essential when deriving universal approximation results by subtraction free rational expresssions.

II Notation and technical preliminaries

Let $\mathbb{N}$, $\mathbb{Z}$, $\mathbb{R}$, $\mathbb{R}_{\geqslant 0}$, and $\mathbb{R}_{>0}$ denote the set of natural, integer, real, nonnegative real, and positive real numbers, respectively. Given a function $\phi:\mathbb{R}^{n}\rightarrow\mathbb{R}\cup\{+\infty\}$, we define its domain as $\operatorname{dom}\phi\doteq\{\mathbf{x}\in\mathbb{R}^{n}:\phi(\mathbf{x})<+\infty\}$. If the function $\phi:\mathbb{R}^{n}\rightarrow\mathbb{R}\cup\{+\infty\}$ is differentiable at point $\mathbf{x}$, we denote by $\nabla\phi(\mathbf{x})$ its gradient at $\mathbf{x}$.

II-A Log-Sum-Exp functions

Following [7], we define $\mathrm{LSE}$ (Log-Sum-Exp) as the class of functions $f:\mathbb{R}^{n}\rightarrow\mathbb{R}$ that can be written as

[TABLE]

for some $K\in\mathbb{N}$, $b_{k}\in\mathbb{R}_{>0}$, \bm{\alpha}^{(k)}=[\begin{array}[]{ccc}\alpha_{1}^{(k)}&\cdots&\alpha_{n}^{(k)}\end{array}]^{\top}\in\mathbb{R}^{n}, $k=1,\dots,K$, where \mathbf{x}=[\begin{array}[]{ccc}x_{1}&\cdots&x_{n}\end{array}]^{\top} is a vector of variables. Further, given $T\in\mathbb{R}_{>0}$, we define $\mathrm{LSE}_{T}$ as the class of functions $f_{T}:\mathbb{R}^{n}\rightarrow\mathbb{R}$ that can be written as

[TABLE]

for some $K\in\mathbb{N}$, $b_{k}\in\mathbb{R}_{>0}$, and $\bm{\alpha}^{(k)}\in\mathbb{R}^{n}$, $k=1,\dots,K$. By letting $\beta_{k}\doteq\log b_{k}$, $k=1,\ldots,K$, we have that functions in the family $\mathrm{LSE}_{T}$ can be equivalently parameterized as

[TABLE]

where the $\beta_{k}$s have no sign restrictions. It may sometimes be convenient to highlight the full parameterization of $f_{T}$, in which case we shall write $f_{T}^{(\overrightarrow{\bm{\alpha}},\bm{\beta})}$, where $\overrightarrow{\bm{\alpha}}=(\bm{\alpha}^{(1)},\ldots,\bm{\alpha}^{(K)})$, and $\bm{\beta}=(\beta_{1},\ldots,\beta_{K})$. It can be readily observed that, for any $T>0$, the following property holds:

[TABLE]

The maps in $\mathrm{LSE}_{T}$ are special instances of the log-Laplace transforms of nonnegative measures, studied in [17]. In particular, the maps in $\mathrm{LSE}_{T}$ are smooth, they are convex (this is an easy consequence of Cauchy-Schwarz inequality), and they are even strictly convex if the vectors $\overrightarrow{\bm{\alpha}}^{(k)}$ constitute an affine generating family of $\mathbb{R}^{n}$, see [7, Prop. 1]. Maps of this kind play a key role in tropical geometry, in the setting of Viro’s patchworking method [9], dealing with the degeneration of real algebraic curves to a piecewise linear limit. We note in this respect that the family of functions $(f_{T})_{T>0}$ given by (3) converges uniformly on $\mathbb{R}^{n}$, as $T\to 0^{+}$, to the function

[TABLE]

Actually, the following inequality holds for all $T>0$

[TABLE]

see [7] for details and background.

II-B Posynomials and $\mathrm{GPOS}_{T}$ functions

Given $c_{k}>0$ and $\bm{\alpha}^{(k)}\in\mathbb{R}^{n}$, a positive monomial is a product of the form $c_{k}\mathbf{x}^{\bm{\alpha}^{(k)}}=c_{k}x_{1}^{\alpha_{1}^{(k)}}x_{2}^{\alpha_{2}^{(k)}}\cdots x_{n}^{\alpha_{n}^{(k)}}$. A posynomial is a finite sum of positive monomials,

[TABLE]

We denote by $\mathrm{POS}$ the class of functions $\psi:\mathbb{R}_{>0}^{n}\rightarrow\mathbb{R}_{>0}$ of the form (6). Posynomials are log-log-convex functions, meaning that the log of a posynomial $\psi$ is convex in the log of its argument, see, e.g., Section II.B of [7]. We denote by $\mathrm{GPOS}_{T}$ the class of functions that can be expressed as

[TABLE]

for some $T>0$ and $\psi\in\mathrm{POS}$. These functions are log-log-convex, and they form a subset of the so-called generalized posynomial functions, see, e.g., Section II.B of [7]. It is observed in Proposition 3 of [7] that $\mathrm{LSE}_{T}$ and $\mathrm{GPOS}_{T}$ functions are related by a one-to-one correspondence. That is, for any $f(\mathbf{x})\in\mathrm{LSE}_{T}$ and $\psi(\mathbf{z})\in\mathrm{GPOS}_{T}$ it holds that

[TABLE]

III A universal approximation theorem

III-A Preliminary: approximation of convex functions

We start by recalling a key result of [7], stating that functions in $\mathrm{LSE}_{T}$ are universal smooth approximators of convex functions, see Theorem 2 in [7].

Theorem 1** (Universal approximators of convex functions, [7]).**

Let $\phi$ be a real valued continuous convex function defined on a compact convex subset ${\mathcal{K}}\subset\mathbb{R}^{n}$. Then, for all $\varepsilon>0$ there exist $T>0$ and a function $f_{T}\in\mathrm{LSE}_{T}$ such that

[TABLE]

We now extend the above result by showing that it actually holds for the restricted class of $\mathrm{LSE}_{T}$ with rational parameters. This extension will allow us to apply our results to the approximation by subtraction free expressions.

Definition 1**.**

A function $f_{T}\in\mathrm{LSE}_{T}$ has rational parameters if $T>0$ is a rational number and $f_{T}=f_{T}^{(\overrightarrow{\bm{\alpha}},\bm{\beta})}$ is of the form (3) where the vectors $\bm{\alpha}^{(1)},\ldots,\bm{\alpha}^{(K)}$ have rational entries, and $\beta_{1},\dots,\beta_{K}$ are rational numbers. We shall also say that $f_{T}$ is a $\varepsilon$-approximation of $f$ on $\mathcal{K}$ when (8) holds.

The following corollary holds.

Corollary 1** (LSE approximation with rational parameters).**

Under the hypotheses of 1, for all $\varepsilon>0$ there exists a rational $T>0$ and a function $f_{T}\in\mathrm{LSE}_{T}$ with rational parameters such that (8) holds. Moreover, $T$ may be chosen of the form $1/p$ where $p$ is a positive integer.

Proof.

First, inspecting the proof of Theorem 2 in [7], one obtains that $\phi$ can be approximated uniformly by a map $f_{T}\in\mathrm{LSE}_{T}$, for all $T>0$ small enough, hence we can always assume that $T$ is of the form $1/p$ for some positive integer $p$. It then remains to be proved that the approximation result still holds if also $\bm{\alpha}^{(1)},\ldots,\bm{\alpha}^{(K)}$ and $\beta_{1},\dots,\beta_{K}$ are rational. To this end, let us study the effect of a perturbation of these parameters on the map $f_{T}$. Observe that the map $\varphi:\mathbb{R}^{K}\to\mathbb{R}$, $\bm{\xi}=(\xi_{1},\dots,\xi_{K})\mapsto T\log(\sum_{k=1}^{K}\exp(\xi_{k}/T))$ satisfies

[TABLE]

for all $\bm{\xi}\in\mathbb{R}^{K}$, where $\|\cdot\|_{\infty}$ is the sup-norm. This follows from the fact that $\varphi$ is order preserving and commutes with the addition of a constant, see e.g. [18] and Section 2 of [19]. It follows from (9) that if $f_{T}$ is as in (3), and if

[TABLE]

then, letting $R\doteq\max_{\mathbf{x}\in\mathcal{K}}\|\mathbf{x}\|$, $\overrightarrow{\bm{\gamma}}=(\bm{\gamma}^{(1)},\ldots,\bm{\gamma}^{(K)})$, and $\bm{\delta}=(\delta_{1},\ldots,\delta_{K})$, we get

[TABLE]

for all $\mathbf{x}\in\mathcal{K}$. Hence, choosing $(\overrightarrow{\bm{\gamma}},\bm{\delta})$ to be a rational approximation of $(\overrightarrow{\bm{\alpha}},\bm{\beta})$ such that $\kappa((\overrightarrow{\bm{\alpha}},\bm{\beta}),(\overrightarrow{\bm{\gamma}},\bm{\delta}))\leqslant\varepsilon$, and supposing that $f_{T}$ is a $\varepsilon$-approximation of $\phi$ on $\mathcal{K}$, we deduce that $g_{T}$ is a $2\varepsilon$-approximation of $\phi$ on $\mathcal{K}$, from which the statement of the corollary follows. ∎

III-B Approximation of general continuous functions

This section contains our main result on universal approximation of continuous functions. To this end, we first define the class of functions that can be expressed as the difference of two functions in $\mathrm{LSE}_{T}$.

Definition 2** ($\mathrm{DLSE}_{T}$ functions).**

We say that a function $\phi:\mathbb{R}^{n}\to\mathbb{R}$ belongs to the $\mathrm{DLSE}_{T}$ class, if $\phi=g_{T}-h_{T}$, for some $g_{T},h_{T}\in\mathrm{LSE}_{T}$. Further, we say that $\phi$ has rational parameters, if $g_{T}$ and $h_{T}$ have rational parameters.

The following result shows that any continuous function can be approximated uniformly by a function in a $\mathrm{DLSE}_{T}$ class.

Theorem 2** (Universal approximation property of $\mathrm{DLSE}_{T}$).**

Let $\phi$ be a real-valued continuous function defined on a compact, convex subset ${\mathcal{K}}\subset\mathbb{R}^{n}$. Then, for any $\varepsilon>0$ there exist a function $f_{T}\in\mathrm{DLSE}_{T}$ with rational parameters, for some $T=1/p$ where $p$ is a positive integer, such that $|f_{T}(\mathbf{x})-\phi(\mathbf{x})|\leqslant\varepsilon$, $\forall\mathbf{x}\in{\mathcal{K}}$.

Proof.

A classical result of convex analysis states that any continuous function $\phi$ defined on a compact convex subset $\mathcal{K}$ of $\mathbb{R}^{n}$ can be written as the difference $g-h$ where $g,h$ are continuous, convex functions defined on $\mathcal{K}$, see, e.g., Proposition 2.2 of [20]. Then, by 1, for all $\varepsilon>0$, we can find a rational $T^{\prime}>0$ and a function $g_{T^{\prime}}\in\mathrm{LSE}_{T^{\prime}}$ with rational parameters such that $|g(\mathbf{x})-g_{T^{\prime}}(\mathbf{x})|\leqslant\varepsilon/2$ holds for all $\mathbf{x}\in\mathcal{K}$. Similarly, we can find a rational $T^{\prime\prime}>0$ and a function $h_{T^{\prime\prime}}\in\mathrm{LSE}_{T^{\prime\prime}}$ with rational parameters such that $|h(\mathbf{x})-h_{T^{\prime\prime}}(\mathbf{x})|\leqslant\varepsilon/2$ holds for all $\mathbf{x}\in\mathcal{K}$. Hence, by taking any rational $T>0$ such that $T^{\prime}$ and $T^{\prime\prime}$ are integer multiples of $T$, it follows from the nesting property in Lemma 1 of [7] that $g_{T^{\prime}}$ and $h_{T^{\prime\prime}}$ both belong to $\mathrm{LSE}_{T}$. Thus, there exist a rational $T>0$ and $g_{T},h_{T}\in\mathrm{LSE}_{T}$ such that, for all $\mathbf{x}\in\mathcal{K}$,

[TABLE]

Summing these conditions we obtain that $|(g(\mathbf{x})-h(\mathbf{x}))-(g_{T}(\mathbf{x})-h_{T}(\mathbf{x}))|\leqslant\varepsilon$, for all $\mathbf{x}\in\mathcal{K}$. The claim then immediately follows by recalling that $g(\mathbf{x})-h(\mathbf{x})=\phi(\mathbf{x})$ for all $\mathbf{x}\in\mathcal{K}$, and letting $f_{T}\doteq g_{T}-h_{T}$, whence $f_{T}\in\mathrm{DLSE}_{T}$. ∎

The following explicit example illustrates the approximation of a non-convex and nondifferentiable function by a function in $\mathrm{LSE}_{T}$.

Example 1*.*

Consider

[TABLE]

Observe that $\phi(x)=\max(0,x)-\max(0,x-1)$, which is indeed a difference of two nonsmooth convex functions. By using (5), we can approximate each term of this difference by a function in $\mathrm{LSE}$ as

[TABLE]

It follows that the map

[TABLE]

is in $\mathrm{DLSE}_{T}$ and satisfies the following uniform approximation property of $\phi$:

[TABLE]

Example 2*.*

The previous explicit approximation carries over to a continuous piecewise affine function $\phi$ of a single real variable, as follows. By piecewise affine, we mean that $\mathbb{R}$ can be covered by finitely many intervals in such a way that $\phi$ is affine over each of these intervals. Then, $\phi$ can be written in a unique way as

[TABLE]

where $a,b$ are real parameters, $\gamma_{1}<\dots<\gamma_{K}$ are the nondifferentiability points of $\phi$, and $\alpha_{i}=\phi^{\prime}(\gamma_{i}^{+})-\phi^{\prime}(\gamma_{i}^{-})$ is the jump of the derivative of $\phi$ at point $\gamma_{i}$. Another way to get insight of (10) is to make the following observation: the function $\phi$ has a second derivative in the distribution sense, $\phi^{\prime\prime}=\sum_{1\leqslant i\leqslant K}\alpha_{i}\delta_{\gamma_{i}}$; then (10) is gotten by integrating twice the latter expression of $\phi^{\prime\prime}$. Possibly after subtracting to $\phi$ an affine function, we will always assume that $a=b=0$.

Then, setting

[TABLE]

and

[TABLE]

we write

[TABLE]

Let

[TABLE]

and note that

[TABLE]

Then, setting $f_{T}\doteq f_{T}^{+}-f_{T}^{-}$ where

[TABLE]

and using (5), we get

[TABLE]

III-C Data approximation

Consider a collection ${\mathcal{D}}$ of $m$ data-points,

[TABLE]

where $y_{i}=\phi(\mathbf{x}_{i})$, $i=1,\ldots,m$, and $\phi$ is an unknown function. The following universal data approximation result holds.

Corollary 2** (Universal data approximation).**

Given a collection of data ${\mathcal{D}}$ as in (11), for any $\varepsilon>0$ there exists ${T}>0$ and a function $d_{T}\in\mathrm{DLSE}_{T}$ with rational coefficients such that

[TABLE]

Proof.

Let $\mathcal{K}\doteq\mbox{co}\{\mathbf{x}_{i},\ldots,\mathbf{x}_{m}\}$ be the convex hull of the input data points. Consider a triangulation of the input points $\mathbf{x}_{i}$: recall that such a triangulation consists of a finite collection of simplices $(\Delta_{r})_{r\in R}$, satisfying the following properties: (i) the vertices of these simplices are taken among the points $\mathbf{x}_{1},\dots,\mathbf{x}_{m}$; (ii) each point $\mathbf{x}_{i}$ is the vertex of at least one simplex; (iii) the interiors of theses simplices have pairwise empty intersections, and (iv) the union of these simplices is precisely $\mathcal{K}$. Then, there is a unique continuous function, $f$, affine on each simplex $\Delta_{r}$, and such that $f(\mathbf{x}_{i})=y_{i}$ for $1\leqslant i\leqslant m$. Observe that $\mathcal{K}$ is convex and compact by construction. Now, a direct application of 2 shows that for any $\varepsilon>0$ there exists ${T}>0$ and a function $d_{T}\in\mathrm{DLSE}_{T}$ with rational coefficients such that

[TABLE]

which concludes the proof. ∎

IV Positive functions on the positive orthant

In this section we discuss approximation results for functions taking positive values on the open positive orthant. A particular case of this class of functions is given by log-log-convex functions, whose uniform approximation by means of $\mathrm{GPOS}_{T}$ functions was discussed in Corollary 1 of [7]. We shall first extend this result to functions with rational parameters, and then provide a universal approximation result for continuous positive functions over the open positive orthant.

IV-A Uniform approximation results

The following preliminary definitions are instrumental for our purposes: a subset ${\mathcal{R}}\subset\mathbb{R}_{>0}^{n}$ will be said to be log-convex if its image by the map that takes the logarithm entry-wise is convex. We shall say that a function $\psi_{T}\in\mathrm{GPOS}_{T}$ has rational parameters if it can be written as in (7) with $\psi$ given by (6), in such a way that $T$, the entries of the vectors $\bm{\alpha}^{(1)}$, …, $\bm{\alpha}^{(K)}$, and the scalars $\log c_{1},\dots,\log c_{K}$ are rational numbers. The following corollary extends Corollary 1 of [7].

Corollary 3** (Universal approximators of log-log-convex functions).**

Let $\ell$ be a log-log-convex function defined on a compact, log-convex subset ${\mathcal{R}}$ of $\mathbb{R}_{>0}^{n}$. Then, for all $\tilde{\varepsilon}>0$ there exist a function $\psi_{T}\in\mathrm{GPOS}_{T}$ with rational parameters, for some $T=1/p$ where $p$ is a positive integer, such that, for all $\mathbf{x}\in\mathcal{R}$,

[TABLE]

Proof.

By using the log-log transformation, define $\tilde{\ell}(\mathbf{q})\doteq\log(\ell(\exp(\mathbf{q})))$. Since $\ell(\mathbf{x})$ is log-log-convex in $\mathbf{x}$, $\tilde{\ell}(\mathbf{q})$ is convex in $\mathbf{q}=\log\mathbf{x}$. Furthermore, the set $\mathcal{K}\doteq\log(\mathcal{R})$ is convex and compact since the set $\mathcal{R}$ is log-convex and compact. Thus, by 1, for all ${\varepsilon}>0$, there exist $T=1/p$ where $p$ is a positive integer, and a function $f_{T}\in\mathrm{LSE}_{T}$ with rational coefficients such that $|f_{T}(\mathbf{q})-\tilde{\ell}(\mathbf{q})|\leqslant{\varepsilon}$ for all $\mathbf{q}\in\mathcal{K}$. From this point on, the proof follows the very same lines as the proof of Corollary 1 of [7]. ∎

We next state an approximation result for functions on the positive orthant. The derivation of 3 from 3 is similar to the derivation of 2 from 1 and thus we omit its proof.

Theorem 3** (Universal approximators of functions on the open orthant).**

Let $\ell$ be a continous positive function defined on a compact log-convex subset ${\mathcal{R}}\subset\mathbb{R}_{>0}^{n}$. Then, for all $\tilde{\varepsilon}>0$ there exist two functions $\psi_{T},\psi^{\prime}_{T}\in\mathrm{GPOS}_{T}$ with rational parameters, for some $T=1/p$ where $p$ is a positive integer, such that, for all $\mathbf{x}\in\mathcal{R}$,

[TABLE]

IV-B Universal approximation by subtraction-free expressions

We next derive from 3 an approximation result by subtraction-free expressions. The latter are an important subclass of rational expressions, studied in [8]. Subtraction-free expressions are well formed expressions in several commutative variables $x_{1},\dots,x_{n}$, defined using the operations $+,\times,/$ and using positive constants, but not using subtraction. Formally, a subtraction-free expression in the variables $x_{1},\dots,x_{n}$ is a term produced by the context-free grammar rule

[TABLE]

where $C$ can take the value of any positive constant. For instance, $E_{1}\doteq(x_{1}+x_{2}^{3})/(2x_{1}+3x_{2}/(x_{1}+x_{2}))$ is a subtraction-free expression, whereas $E_{2}\doteq x_{1}^{2}-x_{1}x_{2}+x_{2}^{2}$ is not a subtraction-free expression, owing to the presence of the $-$ sign. Note that $E_{2}$, thought of as a formal rational fraction, coincides with $E_{3}\doteq(x_{1}^{3}+x_{2}^{3})/(x_{1}+x_{2})$ which is subtraction free, i.e., an expression which is not subtraction-free may well have an equivalent subtraction-free expression. However, there are rational fractions, and even polynomials, like $E_{4}\doteq(x_{1}-x_{2})^{2}$, without subtraction equivalent free expressions, because any subtraction-free expression must take positive values on the interior of the positive cone, whereas $E_{4}$ vanishes on the line $x_{1}=x_{2}$. Important examples of subtraction-free expressions arise from series-parallel composition rules for resistances. More advanced examples, coming from algebraic combinatorics, are discussed in [8].

Corollary 4** (Approximation by subtraction-free expressions).**

Let $\ell$ be a continous positive function defined on a compact log-convex subset ${\mathcal{R}}\subset\mathbb{R}_{>0}^{n}$. Then, for all $\tilde{\varepsilon}>0$ there exist positive integers $p,q$ and a subtraction-free expression $E$ in $n$ variables $y_{1},\dots,y_{n}$ such that the function

[TABLE]

in which $x_{i}^{1/q}$ is substituted to the variable $y_{i}$, satisfies, for all $\mathbf{x}\in\mathcal{R}$,

[TABLE]

Proof.

3 shows that (14) holds with $f=\psi_{T}/\psi_{T}^{\prime}$ where $T=1/p$ for some positive integer $p$, and $\psi_{T},\psi^{\prime}_{T}$ are functions in $\mathrm{GPOS}_{T}$ with rational parameters, i.e.,

[TABLE]

where the vectors $\bm{\alpha}^{(k)}$ and $(\bm{\alpha}^{\prime})^{(k)}$ have rational entries. Denoting by $q$ the least common multiple of the denominators of the entries of the vectors $\bm{\alpha}^{(k)}p$ and $(\bm{\alpha}^{\prime})^{(k)}p$, we see that $\psi_{T}(\mathbf{x})/\psi_{T}(\mathbf{x})$ is precisely of the form $E(x_{1}^{1/q},\dots,x_{n}^{1/q})^{1/p}$ where $E$ is a subtraction-free rational expression. ∎

IV-C Approximation of positive data

Consider a collection $\mathcal{L}$ of $m$ data pairs,

[TABLE]

where $\mathbf{z}_{i}\in\mathbb{R}_{>0}^{n}$, $w_{i}\in\mathbb{R}_{>0}$, $i=1,\dots,m$, with $w_{i}=\ell(\mathbf{z}_{i})$, $i=1,\dots,m$, where $\ell:\mathbb{R}_{>0}^{n}\rightarrow\mathbb{R}_{>0}$ is an unknown function. The data in $\mathcal{L}$ is referred to as positive data. The following proposition is now an immediate consequence of 3, where $\mathcal{R}$ can be taken as the log-convex hull of the input data points222For given $\mathbf{z}_{1},\ldots,\mathbf{z}_{m}\in\mathbb{R}_{>0}^{n}$, we define their log-convex hull as the set of vectors $\mathbf{z}=\prod_{i=1}^{m}\mathbf{z}_{i}^{\xi_{i}}$, where $\xi_{i}\in[0,1]$, $i=1,\dots,m$, and $\sum_{i=1}^{m}\xi_{i}=1$..

Proposition 1**.**

Given positive data $\mathcal{L}\doteq\{(\mathbf{z}_{i},w_{i})\}_{i=1}^{m}$, for any $\tilde{\varepsilon}>0$ there exist a rational $T>0$ and two functions $\psi_{T},\psi^{\prime}_{T}\in\mathrm{GPOS}_{T}$ with rational parameters such that

[TABLE]

V $\mathrm{DLSE}_{T}$ networks

Functions in $\mathrm{DLSE}_{T}$ can be modeled through a feedforward neural network ($\mathrm{FFNN}$) architecture, composed of two $\mathrm{LSE}_{T}$ networks in parallel, whose outputs are fused via an output difference node, see Fig. 1.

It may sometimes be convenient to highlight the full parameterization of the input-output function $d_{T}(\cdot)$ synthesized by the $\mathrm{DLSE}_{T}$ network, in which case we shall write

[TABLE]

where $\overrightarrow{\bm{\alpha}}=(\bm{\alpha}^{(1)},\ldots,\bm{\alpha}^{(K)})$, $\overrightarrow{\bm{\gamma}}=(\bm{\gamma}^{(1)},\ldots,\bm{\gamma}^{(K)})$, $\bm{\beta}=(\beta_{1},\ldots,\beta_{K})$, and $\bm{\delta}=(\delta_{1},\dots,\delta_{k})$ are the parameter vectors of the two $\mathrm{LSE}_{T}$ components.

Each $\mathrm{LSE}_{T}$ component has $n$ input nodes, one hidden layer with $K$ nodes, and one output node. The activation function of the hidden nodes is $s\mapsto(\exp(s/T))$, and the activation of the output node of each $\mathrm{LSE}_{T}$ component is $s\mapsto T\log(s)$. Each node in the hidden layer of the first $\mathrm{LSE}_{T}$ component network computes a term of the form $s_{k}=\langle\bm{\alpha}^{(k)},\mathbf{x}\rangle+\beta_{k}$, where the $i$-th entry $\alpha^{(k)}_{i}$ of $\bm{\alpha}^{(k)}$ represents the weight between node $k$ and input $x_{i}$, and $\beta_{k}$ is the bias term of node $k$. Each node $k$ of the first $\mathrm{LSE}_{T}$ network thus generates activations

[TABLE]

We consider the weights from the inner nodes to the output node to be unitary, whence the output node of the first $\mathrm{LSE}_{T}$ network computes $s=\sum_{k=1}^{K}a_{k}$ and then, according to the output activation function, the output layer returns the value

[TABLE]

An identical reasoning applies to the output of the second component $\mathrm{LSE}_{T}$ network. The overall output realizes a $\mathrm{DLSE}_{T}$ function which, by 2, allows us to approximate any continuous function over a compact convex domain. Similarly, by 2, we can approximate data ${\mathcal{D}}=\{(\mathbf{x}_{i},y_{i})\}_{i=1}^{m}$ via a $\mathrm{DLSE}_{T}$ network, to any given precision.

Theorem 4**.**

Given a collection of data ${\mathcal{D}}\doteq\{(\mathbf{x}_{i},y_{i})\}_{i=1}^{m}$, for each $\varepsilon>0$ there exists a $\mathrm{DLSE}_{T}$ neural network such that

[TABLE]

where $d_{T}$ is the input-output function of the network.

V-A Training $\mathrm{DLSE}_{T}$ networks

By using the scaling property (4) of $\mathrm{LSE}_{T}$ functions, it can be noticed that a simpler $\mathrm{LSE}$ network structure can be used to implement a $\mathrm{DLSE}_{T}$ neural network, as shown in Fig. 2.

As a matter of fact, given the parameter vectors $\overrightarrow{\bm{\alpha}}$, $\overrightarrow{\bm{\gamma}}$, $\bm{\beta}$, $\bm{\delta}$, it can be easily derived that

[TABLE]

and hence dealing with a $\mathrm{DLSE}_{T}$ neural network corresponds to deal with a $\mathrm{DLSE}$ neural network whose input and output have been rescaled. Each of the two $\mathrm{LSE}$ components of the network realizes an input-output map of the form

[TABLE]

The gradients of this function with respect to its parameters $\bm{\alpha}^{(i)}$, $\beta_{i}$ are, for $i=1,\ldots,K$,

[TABLE]

Thus, by using the chain rule, we have that

[TABLE]

Given a dataset ${\mathcal{D}}$ as in (11), these gradients can be used to train a $\mathrm{DLSE}_{T}$ network by using classical algorithms such as the Levenberg-Marquardt algorithm [21], the Fletcher-Powell conjugate gradient [22], or the stochastic gradient descent algorithm [23]. In numerical practice, one may fix the parameter $T$ and $K$ and train the network with respect to the parameters $\overrightarrow{\bm{\alpha}}$, $\bm{\beta}$, $\overrightarrow{\bm{\gamma}}$ and $\bm{\delta}$ by using one of the methods mentioned above, until a satisfactory cross-validated fit is found. A suitable initial value for the $T$ parameter may be set, for instance, to the inverse mid output range ${2}/{|\max(y_{i})-\min(y_{i})|}$. Alternatively, $T$ can be considered as a trainable variable as well, and computed by the training algorithm alongside with the other parameters.

The following example illustrates the application of the Levenberg-Marquardt algorithm to a simple case.

Example 3*.*

Consider the function $\phi:[-2,2]\rightarrow\mathbb{R}$,

[TABLE]

Such a function, which is clearly nonconvex, has been used to generate the dataset ${\mathcal{D}}=\{(x_{i},y_{i})\}_{i=1}^{100}$, where each $x_{i}$ has been taken uniformly at random in $[-2,2]$ and $y_{i}=\phi(x_{i})$, $i=1,\dots,100$. The Levenberg-Marquardt algorithm has been used to train a $\mathrm{DLSE}_{T}$ network fitting such data with $K=10$ and $T={2}/|\max(y_{i})-\min(y_{i})|$. Fig. 3 depicts the output of the $\mathrm{DLSE}_{T}$ network and of its two $\mathrm{LSE}_{T}$ components $f_{T}^{(\overrightarrow{\bm{\alpha}},\bm{\beta})}(\mathbf{x})$ and $f_{T}^{(\overrightarrow{\bm{\gamma}},\bm{\delta})}(\mathbf{x})$.

As shown in Fig. 3, although the function $\phi$ is nonconvex, it is well approximated by a $\mathrm{DLSE}_{T}$ network. Indeed, the data represented in Fig. 3 are approximated by the trained $\mathrm{DLSE}_{T}$ network with a mean square error of $4.4\cdot 10^{-5}$.

VI Non-convex optimization via $\mathrm{DLSE}_{T}$ networks

In view of the results established in Sections III and V, $\mathrm{DLSE}_{T}$ networks can be efficiently used to compute a difference of convex (DC) approximate decomposition of any continuous function over a compact set. Indeed, by using the tools described in Section V, given any continuous function $\phi(\cdot)$ defined on a convex compact set $\mathcal{K}\subset\mathbb{R}^{n}$ (or, more generally, a dataset generated through any function $\phi$) and $\varepsilon\in\mathbb{R}_{>0}$, we can determine $g_{T},h_{T}\in\mathrm{LSE}_{T}$ such that

[TABLE]

Once functions $g_{T}(\mathbf{x})$ and $h_{T}(\mathbf{x})$ are determined via training on available data, we have a surrogate model $d_{T}=g_{T}-h_{T}\simeq\phi$ that we can use for solving approximately design problems of the form $\min_{\mathbf{x}\in\mathcal{K}}\phi(\mathbf{x})$, by substituting the surrogate function $d_{T}$ in place of the (possibly unknown) function $\phi$. The resulting surrogate design problem

[TABLE]

involves the minimization of the difference of two convex $\mathrm{LSE}_{T}$ functions. An approximate solution to this problem can be computed by means of a specific and effective algorithm named Difference-of-Convex Algorithm (DCA), which is described in [24, 25, 26, 27]. We next tailor the DCA to our specific context in the following Algorithm 1, denoted by DLSEA.

The DLSEA returns a candidate optimal solution to problem (19), see [27]. In words, this algorithm starts from a given initial point $\bm{\chi}^{(0)}\in\mathcal{K}$ and iterates until convergence Step 3, in which the gradient of $h_{T}=f_{T}^{(\overrightarrow{\bm{\gamma}},\bm{\delta})}$ is computed as $\mathbf{v}^{(\varkappa)}=\nabla h_{T}(\bm{\chi}^{(\varkappa)})$, and Step 4, in which a local approximation of problem (19) is solved. We observe that the function $g_{T}(\mathbf{x})-\langle\mathbf{x},\mathbf{v}^{(\varkappa)}\rangle$ minimized in Step 4 is (except for a constant term that does not affect the minimization) equal to the difference between the convex function $g_{T}(\mathbf{x})$ and the linearization of $h_{T}$ around the current solution $\bm{\chi}^{(\varkappa)}$. Therefore, the problem to be solved in Step 4 is a convex minimization problem, which can be solved globally and efficiently via standard tools.

Example 4*.*

The DLSEA has been used to find the global minimum of the function $\phi$ given in (16) on the compact convex set $\mathcal{K}=[-2,2]$. Namely, by exploiting the approximate DC decomposition of $\phi$ determined in Example 3 and letting $\bm{\chi}^{(0)}=0$, we obtained, with $4$ iterations of the DLSEA, the approximate solution $\hat{\mathbf{x}}^{\star}=-0.2381$ to problem (19), that is close to the actual solution $\mathbf{x}^{\star}=-0.2379$.

VII Application: diet design for type 2 diabetes

Type 2 diabetes mellitus is a chronic disease that affects the way the human body processes glucose. It is characterized by reduced sensitivity of tissues to insulin, a hormone produced by pancreatic beta cells that promotes the absorption of glucose from blood into liver, fat and skeletal muscle cells [28].

The main objective of this section is to show how $\mathrm{DLSE}_{T}$ networks and the DLSEA can be used to design a diet based on 5 meals for a patient with type 2 diabetes, with the aim of minimizing the maximal concentration of glucose in blood, while guaranteeing that a sufficient amount of glucose is administrated (namely, exactly $185\,\mathrm{g}$). In order to pursue this objective, the meal model for the glucose-insulin system given in [29] has been used to simulate the time-behavior of plasma glucose concentration with breakfast at 8 a.m. (containing $x_{1}\,\mathrm{g}$ of glucose), mid-morning snack at 11 a.m. (containing $x_{2}\,\mathrm{g}$ of glucose), lunch at 1 p.m. (containing $x_{3}\,\mathrm{g}$ of glucose), mid-afternoon snack at 5 p.m. (containing $x_{4}\,\mathrm{g}$ of glucose), and dinner at 8 p.m. (containing $x_{5}\,\mathrm{g}$ of glucose). This model has been used to generate synthetic data: $10^{3}$ points \mathbf{x}^{(i)}=[\begin{array}[]{cccc}x_{1}^{(i)}&\cdots&x_{5}^{(i)}\end{array}]^{\top} have been picked uniformly at random from the set

[TABLE]

and the meal model has been used to determine the corresponding maximum of glucose concentration $y^{(i)}$. Thus, the training dataset ${\mathcal{D}}_{\mathrm{train}}=\{(\mathbf{x}^{(i)},y^{(i)})\}_{i=1}^{1000}$ has been used to train a $\mathrm{DLSE}_{T}$ network with temperature parameter $T=12.98\,\cdot 10^{-3}$ (that is ${2}/{|\max(y_{i})-\min(y_{i})|}$) and with $K=30$ nodes in each of its two $\mathrm{LSE}_{T}$ components. In order to evaluate the prediction capabilities of this network, after the training, we generated, by using the same method as above, a validation dataset ${\mathcal{D}}_{\mathrm{valid}}=\{(\check{\mathbf{x}}^{(i)},\check{y}^{(i)})\}_{i=1}^{100}$, and we compared the outputs of the $\mathrm{DLSE}_{T}$ model $d_{T}(\check{\mathbf{x}}^{(i)})$ with $\check{y}^{(i)}$, $i=1,\dots,100$. For comparison purposes, a classical $\mathrm{FFNN}$ with symmetric sigmoid activation function for the hidden layer (with $60$ nodes) and linear activation function for the output layer has been trained on the same dataset ${\mathcal{D}}_{\mathrm{train}}$ and its prediction performance has been evaluated over ${\mathcal{D}}_{\mathrm{valid}}$. Table I summarizes the prediction errors of the two models.

As shown by Table I, the $\mathrm{DLSE}_{T}$ model has the best performance with respect to all the error metrics. Furthermore, differently form classical $\mathrm{FFNN}$, it is readily amenable to efficient optimization via the DLSEA. Indeed, we applied such an algorithm to the $\mathrm{DLSE}_{T}$ model and we obtained the optimal diet that minimizes the maximal concentration of glucose in blood, while guaranteeing that $185\,\mathrm{g}$ of glucose are administrated. Fig. 4 depicts the optimal diet and the corresponding time-behavior of the plasma glucose concentration. The optimal diet is such that the corresponding maximal plasma glucose concentration in $24\,\mathrm{h}$ is $253.06\,\mathrm{mg/dL}$.

VIII Conclusions

In this paper, we showed that a neural network whose output is the difference of the outputs of two feedforward neural networks with exponential activation function in the hidden layer and logarithmic activation function in the output node is an universal approximator of continuous functions over compact convex sets. By using a logarithmic transformation, such networks maps to a class of subtraction free ratios of generalized posynomials, which we showed to be universal approximators of positive functions over the positive orthant.

The main advantage of $\mathrm{DLSE}_{T}$ networks with respect to classical $\mathrm{FFNN}$ is that they are readily amenable to effective optimization-based design. In particular, by adapting the DCA given in [27] to our context, we derived an ad-hoc algorithm for optimizing $\mathrm{DLSE}_{T}$ models which has proved to be efficient in the considered test cases.