Online learning of neural networks based on a model-free control algorithm

TL;DR

This paper proposes a novel model-free control law for online training of neural networks, framing weight tuning as a feedback control problem, demonstrated through promising numerical results and classification examples.

Contribution

It introduces a new model-free control approach for online neural network training, offering an alternative to traditional methods.

Findings

Effective online weight adjustment demonstrated

Numerical results show promising learning dynamics

Classifier example confirms approach viability

Abstract

We explore the possibilities of using a model-free-based control law in order to train artificial neural networks. In the supervised learning context, we consider the problem of tuning the synaptic weights as a feedback control tracking problem where the control algorithm adjusts the weights online according to the input-output training data set of the neural network. Numerical results illustrate the dynamical learning process and an example of classifier that show very promising properties of our proposed approach.

Equations

$$\left\{\begin{array}[]{l}\dot{\boldsymbol{x}}=f(\boldsymbol{x},u)\\ y=g(\boldsymbol{x})\end{array}\right.$$

$$u_k={\mathcal{C}}_{\pi }^{\{K_p,K_i,k_{\alpha },k_{\beta }\}}(y_k,y_k^{\ast })={\Psi }_k.{\int }_0^t{K}_i(y_k^{\ast }-y_{k-1})d\tau$$

$${\Psi }_k={\Psi }_{k-1}+K_p(k_{\alpha }{e}^{-k_{\beta }k}-y_{k-1}),$$

$$\left(\begin{array}{ccc}3& 0.5& 8\\ 4& 7& 4.5\\ 1& 9& 3\end{array}\right)\left(\begin{array}{c}{x}_1\\ x_2\\ x_3\end{array}\right)=\left(\begin{array}{c}7.95\\ 6.30\\ 3.80\end{array}\right)$$

$$\mathbf{x}\mapsto \mathbf{y}:\mathbf{A}\mathbf{x},$$

$$\begin{array}[]{c}x_{j}=\mathcal{C}_{\pi\,j}^{\{K_{p\,j},K_{i\,j},k_{\alpha\,j},k_{\beta\,j}\}}(y_{j},b_{j})\\ \end{array}$$

$$E(x_1,x_2,\cdots ,x_n,y,W_1,W_2,\cdots ,W_q)=0$$

$$\begin{array}[]{c}W_{1}=\mathcal{C}_{\pi}^{\{K_{p\,1},K_{i\,1},k_{\alpha\,1},k_{\beta\,1}\}}(y,y^{train}),\\ W_{2}=\mathcal{C}_{\pi}^{\{K_{p\,2},K_{i\,2},k_{\alpha\,2},k_{\beta\,2}\}}(y,y^{train}),\\ \vdots\\ W_{q}=\mathcal{C}_{\pi}^{\{K_{p\,q},K_{i\,q},k_{\alpha\,q},k_{\beta q}\}}(y,y^{train}).\\ \end{array}$$

Full text

1 Introduction

Training a neural network consists in tuning its internal weights in order to learn a mapping function from inputs to outputs and eventually examine what the model predicts [1]. Besides classical tuning techniques (see e.g. [2] and a survey in [3] that presents tuning methods to model complex manufacturing processes), some connections between adaptive control and optimization methods have been pointed out recently in [4, 5] that highlight a certain equivalence between using tools from the adaptive control field and solving problems in the machine learning field. In this line of thinking, the motivation of this work is to propose a strategy to tune neural networks using the so-called model-free control algorithm in the context of supervised learning.

The model-free control methodology, originally proposed by [6], has been designed to control a priori any ”unknown” dynamical system in a ”robust” manner, and can be considered as an alternative to standard PI and PID control [7] as it does not need any prior knowledge of the plant to control. Its usefulness has been demonstrated through successful applications111See e.g. the references in [6, 8, 9] and the references therein for an overview of the applications., and in particular, an application dedicated to the supply chain management [9] has been recently proposed. A derivative-free-based version of this control algorithm has been proposed by the author in [10], for which some interesting capabilities of online optimization have been highlighted.

At the intersection between control, optimization and machine learning, in this work, we consider the training of a neural network as a tracking control problem, where the proposed ”para-model” control technique [10] is experimented as a derivative-free learning algorithm to tune the weights of the network in order to fit online the training data.

The paper is organized as follow. Section 2 reviews the para-model approach. In Section 3, a preliminary example illustrates how a model-free-based distributed control could be implemented in order to control multiple systems. Section 4 presents the application of the para-model control to train a simple neural network and numerical results are presented in Section 5 to illustrate the dynamical evolution of the learning process as well as an example of classifier. Section 6 gives some concluding remarks.

2 Principle of the para-model control

Consider a nonlinear SISO dynamical system $f:u\mapsto y$ to control

[TABLE]

where $f$ is the function describing the behavior of a nonlinear system and $\boldsymbol{x}\in I\!\!R$ is the state vector; the para-model control is an application $\mathcal{C}_{\pi}:(y,y^{*})\mapsto u$ whose purpose is to control the output $y$ of (1) following an output reference $y^{*}$. In simulation, the system (1) is controlled in its ”original formulation” without any modification or linearization.

For any discrete moment $t_{k},\,k\in I\!\!N^{*}$, one defines the discrete controller $\mathcal{C}_{\pi}:(y,y^{*})\mapsto u$ as an integrator associated to a numerical series $(\Psi_{k})_{k\in I\!\!N}$ such as symbolically

[TABLE]

with the recursive term

[TABLE]

where $y^{\ast}$ is the output (or tracking) reference trajectory; $K_{p}$ and $K_{i}$ are real positive tuning gains; $\varepsilon_{k-1}=y^{\ast}_{k}-y_{k-1}$ is the tracking error; $k_{\alpha}e^{-k_{\beta}k}$ is an initialization function where $k_{\alpha}$ and $k_{\beta}$ are real positive constants; practically, the integral part is discretized using e.g. Riemann sums.

Define the set of the $\mathcal{C}_{\pi}$-parameters of the controller as the set of the tuning coefficients $\{K_{p},K_{i},k_{\alpha},k_{\beta}\}$222An interesting property that has been observed with para-model control throughout the overall applications is the relative flexibility of the $\mathcal{C}_{\pi}$-parameters to obtain good tracking performances while ”prototyping” a new process to control. In particular, we highlight the case of the experimental validation [11] for which no mathematical representative model of the nonlinear process was available and the control has been tested under several working conditions using indeed the $\mathcal{C}_{\pi}$-parameters adjusted for the corresponding simplified simulation.. The implementation of the control scheme is depicted in Fig. 1 where $\mathcal{C}_{\pi}$ is the proposed para-model controller.

In the next section, an example is presented to illustrate how model-free-based distributed control can be implemented in order to introduce the methodology to train neural networks by controlling the corresponding neural weights.

3 Example of distributed model-free-based control : an amazing way to solve $Ax=b$

To illustrate the properties of the proposed para-model algorithm, consider the following linear system $\boldsymbol{A}\boldsymbol{x}=\boldsymbol{b}$ to solve

[TABLE]

where we denote $\boldsymbol{x^{*}}=\begin{pmatrix}x_{1}^{*}&x_{2}^{*}&x_{3}^{*}\end{pmatrix}^{T}$ the solution of (3). Considering the controlled sub-system derived from (3)

[TABLE]

the goal is to solve the system (3) as a tracking problem in such manner that in the sub-system (4), the controlled $\boldsymbol{y}$ tracks $\boldsymbol{y^{*}}=\boldsymbol{b}$. Hence, if $\boldsymbol{y}$ is kept ”close” to $\boldsymbol{b}$, then the controlled $\boldsymbol{x}$ is ”close” to the solution $\boldsymbol{x^{*}}$.

Each variable $x_{j},j=1...3$ of (4) is driven by an autonomous $\mathcal{C}_{\pi\,j}$ controller, with respect to the tracking reference $b_{j},\,j=1...3$ such as ideally $|\boldsymbol{y}-\boldsymbol{b}|\rightarrow 0$ in a finite time. The associated control law $\mathcal{C}_{\pi\,j}$, that is associated to each variable $x_{j},j=1...3$, reads

[TABLE]

where the set of parameters ${\{K_{p\,j},K_{i\,j},k_{\alpha\,j},k_{\beta\,j}\}}$ is associated to the $j$th $\mathcal{C}_{\pi}$ controller.

Figure 2 illustrates the evolution of the controlled $\boldsymbol{x}$ versus the iterations that converges to the solution $\boldsymbol{x^{*}}$.

4 Application to the training of neural networks

4.1 Problem statement

In the context of supervised learning, let us consider a neural network described as a ”black-box” model $E$

[TABLE]

that is composed of $n$ inputs $x_{1},x_{2},\cdots,x_{n}$; an output $y$; $q$ synaptic weights $W_{1},W_{2},\cdots,W_{q}$ and a sigmoid activation function of the form $y=\tanh(.)$ that defines the output of each neuron (node).

Given training data $x_{1}^{train},$ $x_{2}^{train},\cdots,x_{n}^{train}$ and $y^{train}$ associated respectively to the inputs and to the output of $E$, we assume that the algorithm (2) updates each synaptic weight such as

[TABLE]

and therefore, allows ”configuring” the neural network (updates of the $W_{i}$ for all $i=1...q$) in such manner that asymptotically, the output $y$ remains ”as close as possible” to $y^{train}$. Since the neural network does not include any internal dynamic, a filter is associated to each $W_{i}$ in order to include a dynamic regarding the proper use of the $\mathcal{C}_{\pi}$ controllers (Fig. 1).

Remark 1:

Depending on the expected closed loop transient dynamic, a possible choice of the $\mathcal{C}_{\pi}$-parameters is to consider e.g. a decrease of the control amplification gains according to the $q$th node i.e. $K_{p\,q+1}<K_{p\,q},\,K_{i\,q+1}<K_{i\,q}$ in order to obtain a good dynamic response regarding possible changes of the model $E$ and the rejection of external disturbances, like changes in the training data set.

4.2 Simple example of training

To illustrate our proposed training strategy, consider a three-node network333Such small network is still mathematically interesting to investigate [12]., depicted in Fig. 3 including two inputs $x_{1}$ and $x_{2}$ and an output $y$.

The strategy (7) is applied to calculate online the weights $W_{1},W_{2},\cdots,W_{7}$ given the training values $x_{1}^{train},$ $x_{2}^{train}$ and $y^{train}$ (the latter corresponds to the output reference). A first order filter (with a small time constant) is added to include a dynamic to each controller.

5 Numerical results

To present some preliminary properties, the following test bench have been performed considering the initial set of training data $x_{1}^{train}=0.2$, $x_{2}^{train}=0.6$ and $y^{train}=0.55$. The $\mathcal{C}_{\pi}$-parameters have not been optimized regarding the transient responses and the $W_{i}$ are bounded such as $|W_{i}|\leq 1$ for all $i=1...7$. All $W_{i},\,i=1...7$ are initialized to zero.

Evolution of online modifications of the network topology and the training data

In formula (2), set $K_{p}=1$, $K_{i}=1/100$, $k_{\alpha}=333/2$ and $k_{\beta}=40$ including a first order filter with a time constant of $10^{-5}$ s; the simulation time-step is $10^{-5}$ s. Figure 4 shows respectively the evolution of the weights and the controlled output $y$, when the network is subjected to an arbitrary change of its topology (the weight $W_{7}$ is for example forced to zero at an arbitrary time) as well as arbitrary changes of the training data.

As a result, a great tracking of the output $y$ has been observed despite the different changes of the training data as well as the topology of the network, which is referred to as the ”Dropout” concept in e.g. [13, 14].

A classifier example

Consider training the three-node network as a classifier with the following data training set

[TABLE]

where $y^{train}$ is the boolean test of $(x_{1}^{train}+x_{2}^{train})<0.8$.

The following table illustrates a simple classification test and the resulting average of all output values $\overline{y}=0.19$ defines the output partition of the classifier (i.e. classify the particular input values that produce a ”0” in output and vice versa).

[TABLE]

The set of data is properly classified according to the boolean comparison with $\overline{y}$. Remark that since the proposed control-based training algorithm deals with dynamical systems and sweeps the training data through low pass filtering, the partition of the classifier via $\overline{y}$ corresponds indeed to the ’filtered’ averaged value of the output training data.

6 Conclusion and perspectives

This paper presented an application of the model-free-based control methodology in the field of artificial neural networks. Encouraging results show promising tracking performances taking into account online modifications of the training data set as well as modifications of the topology of the studied network. Further works will include the formalization of our proposed approach (based e.g. on the implicit framework proposed in [15]), as well as as investigations regarding the application of our proposed algorithm to large scale neural networks including specific networks used e.g. in decision support systems [16].