GRP Model for Sensorimotor Learning

TL;DR

This paper introduces the GRP model, a modular neural architecture inspired by physiology, that learns sub-task policies and switching mechanisms from demonstrations, enabling transfer of control from brain to spinal cord in sensorimotor tasks.

Contribution

The paper presents a novel modular architecture, the GRP model, which learns sub-task policies and switching mechanisms from unsegmented demonstrations, inspired by physiological neural structures.

Findings

GRP model successfully learns sub-task policies from demonstrations.

The model enables automatic switching between control policies.

Transfer of swing leg control from brain to spinal cord demonstrated.

Abstract

Learning from complex demonstrations is challenging, especially when the demonstration consists of different strategies. A popular approach is to use a deep neural network to perform imitation learning. However, the structure of that deep neural network has to be ``deep" enough to capture all possible scenarios. Besides the machine learning issue, how humans learn in the sense of physiology has rarely been addressed and relevant works on spinal cord learning are rarer. In this work, we develop a novel modular learning architecture, the Generator and Responsibility Predictor (GRP) model, which automatically learns the sub-task policies from an unsegmented controller demonstration and learns to switch between the policies. We also introduce a more physiological based neural network architecture. We implemented our GRP model and our proposed neural network to form a model the transfers the swing leg control from the brain to the spinal cord. Our result suggests that by using the GRP model the brain can successfully transfer the target swing leg control to the spinal cord and the resulting model can switch between sub-control policies automatically.

Tables

Table 1. TABLE I : Control Parameters Values

parameter	value	parameter	value
$k_p^{\alpha }$	110	$k^i$	23
$k_d^{\alpha }$	8.5	$k^{ii}$	4
$k^{stp}$	250	${\dot{\alpha }}_{max}$	10
$k^{ext}$	200	${\alpha }_{thr}$	${\alpha }_{tgt}+8$

Equations

$$e_G^k$$

$$r_{RP}^k$$

$${\gamma }_{epi}$$

$$e_{RP}^k$$

$$G_{total}$$

$$=r_G\sum _{k=1}^m{r}_{RP}^k=r_G$$

$$b^k$$

$${\pi }^k$$

$$J_G^k$$

$$J_{RP}^k$$

$$-\frac{\partial J_G^k}{\partial W_{ii}^k}$$

$$-\frac{\partial J_G^k}{\partial W_{ij}^k}$$

$$-\frac{\partial J_{RP}^k}{\partial R_{ii}^k}$$

$$-\frac{\partial J_{RP}^k}{\partial R_{ij}^k}$$

$${\tau }_k^i=\{\begin{array}{ll}{k}^i\dot{\alpha }& \dot{\alpha }\le 0\\ 0& \dot{\alpha }>0\end{array}$$

$${\tau }_k^{ii}=\{\begin{array}{ll}-k^{ii}{\dot{\Phi }}_k& {\dot{\Phi }}_k\le 0\\ -k^{ii}{\dot{\Phi }}_k(\alpha -{\alpha }_{tgt}({\dot{\Phi }}_k+\dot{\alpha })& {\dot{\Phi }}_k\le 0\&{\dot{\Phi }}_k>-\dot{\alpha }\\ 0& \mathrm{otherwise}\end{array}$$

$${\tau }_k^{iii}=\{\begin{array}{ll}-k^{stp}({\alpha }_{thr}-\alpha )(1-\frac{\dot{\alpha }}{{\dot{\alpha }}_{max}})& \alpha \le {\alpha }_{thr}\&\dot{\alpha }<{\dot{\alpha }}_{max}\\ 0& \mathrm{otherwise}\end{array}$$

$${\tau }^k$$

$${\tau }_{out}$$

Full text

I INTRODUCTION

There are two distinct strategies to learn a complex task from the demonstration. The first one is to use a single deep neural network which has been widely studied in the machine learning field and implemented on computer graphics and robotics. With this approach, a single neural network can learn highly dynamic skills in simulation[1][2][3] and can act as the initial policy for further training that leads to deployment on real robots [4][5]. However, using a single monolithic policy to present a structured activity or cyclic phase structure can be challenging since a single network does not make explicit the sub-structure and encapsulates all contexts[6][7]. Alternatively, instead of using a monolithic controller that includes everything, a modular strategy which consists of multiple controllers and each one only is responsible for a small portion of the control. This approach has been introduced in the study of supervised learning for a mixture of demonstration data[8][9]. These works include multiple experts’ network and a classification network to split the input space into which expert are specialize according to the experts’ outputs. A similar idea has been proposed which uses a directed graphical model and latent variables[6].

Although a variety of learning from demonstration works have been proposed, specific models that relate to human learning in physiology level is relatively rare[10], and the majority of these works are focusing on the function of the brain. Gomi and Kawato[11] combined mixture of experts supervised learning with feedback-error learning[12]. Wolpert and Kawato, based on the idea that the brain contains multiple pairs of forward and inverse model, introduced the modular selection and identification for motor control (MOSAIC) model [13]. Runbot’s study was focusing on learning in the spinal cord level. Runbot investigated in the variation of gains of the neuron as gait changes using Differential Hebbian Learning[14].

Though learning occurs in the spinal cord has not been studied widely, the spinal cord plays an essential role in legged locomotion tasks[15][16]. Remarkable work on mesencephalic cats and parlayed four leg mammals[17][18] provides direct evidence for verifying that animals can generate adaptive leg behaviors in the absence of brain planning. These observations further suggest that the spinal cord generates part of the leg control in animal legged locomotion.

In this paper, we developed a model describing the transfer of a simple legged locomotion task: swing leg control from the brain to the spinal cord. The GRP model is an online learning model inspired by the previous work on modular architecture. We proposed a new format of the neural network which is more aligned with the biology findings in the interaction between two neurons. Then, our target swing leg controller is introduced. We tested our GRP model by learning the target swing leg control policy. The result shows that our proposed model can learn complicated controllers, and also capable of learning switch between different learned controller. Finally, we discussed our model in the physiological implications of our model and its connection to robot learning.

II MODELS

The feedback error learning[19][20] inspires our learning model. In the previous works, the brain uses feedback error to learn the desired command from the cerebellum. We combined the idea of feedback error learning with the modular model to proposed the GRP model. We used the GRP model to represent the transfers of a swing leg control from the brain to the spinal cord. Moreover, we introduce a new form of neural network structure which uses multiplication as the interaction between two neurons. Then we present our physical swing leg model.

II-A Learning Model: GRP Model

To model the transfer of control between the brain and the spinal cord, we introduce an online learning model: Generator and Responsibility Predictor(GRP) model, which is shown in Figure 1. The model is composed of multiple layers that are parallel to each other. Every layer contains a Generator and a Responsibility Predictor(RP). The generators learn from the reference control signal $r_{G}$ provided by the brain in our setting. The RP estimates its corresponding Generator ’responsibility’ $\pi^{k}$ where k indicates the index of the layer. The responsibility can be interpreted as the weight of the Generator under the following constraint,

$\displaystyle\sum^{m}_{k=1}\pi^{k}=1,\ 0\leq\pi^{k}\leq 1$

RP in layer k is trained with a reference responsibility signal $r^{k}_{RP}$ generated by a normalization function. This normalization function takes the difference between the reference control signal and generator output $e^{k}_{G}$,

[TABLE]

the intuition for the normalization function is to output large values when $|e^{k}_{G}|$ is small and produce small values when $|e^{k}_{G}|$ is large. $\gamma_{epi}$ is the regularization term which is a positive value and increases after every learning episode during training, the growth rate $\beta$ is a constant value larger than one. The RP error $e^{k}_{PR}$ is then sent backward for training the RP.

The total output $G_{total}$ of the online learning model is,

[TABLE]

this guarantees that while transferring control from the brain to the spinal cord, the human is still performing locomotion in the targeting gaits. The Generator and PR are both neural network structured which will be introduced in the following section. Their inputs in our setting are sensory data, and we will discuss them in Section IV.

II-B Neural Network Structure

Instead of using a classic neural network structure, we introduce an alternative neural network structure which enhances the biological plausibility. While class neural network models interaction between two neurons as sum operation, the typical interaction between neurons known as Presynaptic Inhibition can be better represented by multiplication. It is reasonable to assume that when one of the neurons does not sense anything, this neuron will not affect other neurons. Thus, we model the network with the interaction between two networks as the multiplication of the exponential of the input value. The network structure is shown in Figure 2. The input data x first be sent to two tunnels and get its inverse -x in one of the channel. These two signals then pass a threshold function. This process intends to model the positive and negative part of sensory data are sensed separately by two neurons, thus provide two inputs $x_{i}$ and $x_{i+1}$ (one of them is a positive value, while the other must be 0) for neuron network’s main part.

The output of the Generator $G^{k}$ is

$\displaystyle G^{k}=\sum^{n}_{i=1}W^{k}_{ii}x_{i}(\prod^{n}_{j=1,j\neq i}e^{W^{k}_{ij}x_{j}})$

where $W^{k}_{ij}$ is the weight of the $j$th input that corresponds to the $i$th input of the $k$th Generator. For RP’s output $\pi^{k}$, we add a Sigmoid activation function to ensure the predicted responsibility value is between [0, 1].

[TABLE]

here, $R^{k}$ is the weight of the $j$th input to the $i$th input in the $k$th layer of RP.

II-C Brain Control Transfer

In the proposed GRP model, the target control $r_{G}$ is provided by the brain. Refer to Section III for a detailed description of the target control. The error between the reference control and the Generator $e^{k}_{G}$ is sent backward to the Generator for learning. Meanwhile, the RP error $e^{k}_{RP}$ is sent back to RP for the same purpose. The weights of networks are learned via a gradient-based method using the appropriate loss function. $J^{k}_{G}$ and $J^{k}_{RP}$ are the loss functions for the Generator and the RP respectively,

[TABLE]

where $\sum_{ij}(W^{k}_{ij})^{2})$ and $\sum_{ij}(R^{k}_{ij})^{2})$ are regularization terms, and $\lambda$ is a constant. The gradient for each weight can be computed analytically,

[TABLE]

here, $x_{i}$ and $x_{j}$ are the input data for neuron $i$ and $j$.

During learning we use the reference responsibility signals to regulate our learning rate.

$\displaystyle\mu^{k}=r^{k}_{RP}\mu$

here, $\mu$ is the constant learning rate, $\mu^{k}$ is the learning rate for the $k$th layer of the Generator at the current time step. This ensures the Generator would not learn when it takes no responsibility.

II-D Physical Model: Swing Leg Model

We used a classic double pendulum model as our physic model. The thigh and shank of this model are represented as rods of length $l_{t}$ and $l_{s}$ ($l_{t}=50cm$, $l_{s}=50cm$). Point masses $m_{t}$ and $m_{s}$ are attached to the middle point of each rod. The inertial properties are based on anthropomorphic data from a human body with the height and weight being 180cm and 80kg respectively[21] ($m_{t}=7.3kg$, $m_{s}=4.3kg$). The hip is connected to the origin of the world frame, and the joint angles $\Phi_{h}$ and $\Phi_{k}$ are measured as shown in Figure 3. The applied hip and knee torque $\tau_{h}$ and $\tau_{k}$ are added to hip and knee respectively. The leg angle $\alpha$ can be calculated as $\alpha=\Phi_{h}-{\Phi_{k}/2}$, the current leg length is calculated as $l=2l_{t}\sin{\Phi_{k}/2}$. This model was simulated in Simulink.

III Target Swing Leg Controller

The target swing leg controller contains three natural control tasks. Starting from the ground level at the initial configuration (leg angle $\alpha=\alpha_{0}$), the first task is to flex the leg to at least the clearance length $l_{clr}$. Second, the control focus shifts to advancing the swing leg to the target angle $\alpha_{tgt}$. And the final task is to extend the leg until ground contact. Although a conventional state feedback controller can execute this sequence of control, this controller takes advantage of the passive dynamics that the swing leg provides to learn the required torques. Moreover, this controller separates the control of the hip and the knee as much as possible. As a consequence, the controller is structured around the functionally distinct hip and knee joint controllers. Overall, this swing leg control is composed of one hip control policy and three knee control policies.

III-A Hip Control

The primary task of the hip controller is to move the leg to the target leg angle $\alpha_{tgt}$. The hip torque $\tau^{\alpha}_{h}$ is given as:

$\displaystyle\tau^{\alpha}_{h}=k^{\alpha}_{p}(\alpha_{tgt}-\alpha)-k^{\alpha}_{d}\dot{\alpha}$

Beside angle control, the hip controller receives an additional term $\tau^{add}_{h}$:

$\displaystyle\tau_{h}=\tau^{\alpha}_{h}+\tau^{add}_{h}$

from the knee controller during the leg extension phase. The purpose for $\tau^{add}_{h}$ will be discussed in the following section.

III-B Knee Control

The primary purpose of the knee controller is to regulate the leg length. As mentioned above, the controller separates knee control into three natural control tasks. Each task is assigned with an individual control policy. A detailed analysis of this controller is presented in [22].

III-B1 Phase 1

The first control task is to flexing the leg in passing a minimum clearance $l_{clr}$. The dynamics shows that while the Coriolis, centrifugal and gravitational terms always tend to extend the knee, negative hip acceleration tends to flex the knee. If the negative hip acceleration passes a threshold, no torque $\tau_{k}$ is required to flex the knee. Otherwise, we add an adaptive flexion control.

[TABLE]

III-B2 Phase 2

Once the leg length has shorten pass the clearance length $l_{clr}$, the knee controller then is tasked with holding the knee, and the leg angle is only controlled by the aforementioned hip control. This task is realized when the knee flexes and modulated when the knee extends,

[TABLE]

III-B3 Phase 3

Once the leg passes the threshold $\alpha_{thr}=\alpha_{tgt}+\Delta\alpha_{thr}$, the primary objective for the knee control switch to stopping the swing and extending leg to hit the ground. This is achieved by using two functional components. The first component generates a stopping knee-flexion torque inspired by nonlinear contact models.

[TABLE]

The stopping torque works well only if the coupling with the hip motion is canceled. Thus, we apply an a compensation torque on the hip control.

$\displaystyle\tau^{add}_{h}=-2\tau^{iii}_{k}$

The second functional component activates when the leg has slowed down to $\dot{\alpha}=0$, a knee extension torque is added to the knee to land the leg to the ground.

$\displaystyle\tau^{iii^{\prime}}_{k}=\tau^{iii}_{k}+k^{ext}(l_{0}-l)$

One thing that needs to be mention is that the last component does not necessarily needs to be activated. The swing of the leg might be terminated before the last component got activated.

IV EXPERIMENT AND RESULTS

To exterminate the validity of our method for transferring swing leg control from the brain to the spinal cord, we used our model to learn from different trajectories generated from [22]. We tested with different numbers of Generator/Responsibility Predictor to verify that our structure is able to learn complex tasks. The inputs of the Generator and Responsibility Predictor are all sensory data, in our setting, we choose 5 variables as our system’s input: $[(\alpha-\alpha_{tgt});\ \Phi_{h};\ \dot{\Phi}_{h};\ \Phi_{k};\ \dot{\Phi}_{k}]$. Since our network is sensing positive and negative values separately (see Figure 2), the inputs to the network are 8 variables: $[(\alpha-\alpha_{tgt})^{+};\ (\alpha-\alpha_{tgt})^{-};\ \Phi_{h};\ \dot{\Phi}^{+}_{h};\ \dot{\Phi}^{-}_{h};\ \Phi_{k};\ \dot{\Phi}^{+}_{k};\ \dot{\Phi}^{-}_{k}]$ ($\Phi_{h}$ and $\Phi_{k}$ can never be negative value, thus no $\Phi^{-}_{h}$ and $\Phi^{-}_{k}$).

IV-A Target Control Parameters

We used 40 different trajectories generated by a target controller for learning. The initial leg configuration is set to be the same for all trajectories: $\Phi_{h,0}=220\ \deg$ and $\Phi_{k,0}=175\ \deg$ ($\alpha_{0}=132.5\ \deg$). While the target leg angle is from $\alpha_{tgt}=50\ \deg$ to $\alpha_{tgt}=85\ \deg$ which includes the typical human landing leg angles. Initial hip velocity $\dot{\Phi_{h,0}}$ is set to be from -4 to 0 $rads^{-1}$ while the initial knee velocity is from -7 to -1 $rads^{-1}$. The clearance leg length is $l_{clr}=5cm$. The control gains are manually tuned, all the parameters are listed in TABLE I.

IV-B Transfer of Control in the Neural System

The total output of the neural network is defined as model’s output when removing the reference control signal and reference responsibility, and it can be calculated as,

[TABLE]

here, $k$ is the index of the neural network. We modeled the transfer of hip control and knee control separately.

IV-B1 Hip Control

With used a pair of networks to train for the hip control initially. As shown in Figure 4, using a single network, the prediction (blue) overlaps with the target control output (red), we can conclude that the system learned the target control and this indicates the control transferred to the spinal cord. The resulting RP is equal to 1 during the control that is because, in this setting, there is only one pair of networks activated, the corresponding Generator should always be in charge of the command, which means at any time the responsibility should output $100\%$. We also tested using three pairs of networks to model hip control transfer. After training, the total output fits the target well and slightly better than the previous setting. Although there are three paralleled Generator/RPs the resulting control only activated 2 of them in the test swing while one of them remained silent. We will discuss this formally in the knee control section.

IV-B2 Knee Control

For the target knee control, it consists of three distinct control objectives and each objective is formed by an individual control strategy. Thus, we can conclude that the target hip control is harder to learn. We started with using a single pair of networks, from the result (Figure 5) we can see that due to the complexity of the target control, the final result cannot match the target entirely. Then, we used multiple pairs of networks to learn the target control. As the number of networks increased, the prediction can fit the target control output better. In the three pairs of networks setting, the GRP model learned three controllers and learned to switch between them. When using five and seven pairs of networks, in Figure 5, the GRP in both of the settings found 4 primary controllers activated during the test. According to the resulting figure, the last two settings, even though they have five and seven Generators respectively, they only enabled four of them. Moreover, these two settings learned to switch controller at the same time steps. Combining the results of hip learning and knee learning, we can conclude that the GRP model can select the number of network pairs automatically instead of activating all the layers.

IV-C Identified Controller

Looking into the learned weights distribution of networks, we can find a controller that explainable. Extracting the weights of the first responsible Generator of knee control in the multiple layers setting, we get weights distribution Figure 6. From the weights distribution, we can identify a ’passive’ Generator since those weights are close to 0 which means given any inputs the output of this generator close to 0.

IV-D Resulting Spinal Cord Control Performance

We tested the overall performance of the resulting model by removing all the reference signals. We used the resulting single pair network for hip control and 3 pairs of networks for knee control. The performance is defined by the average error and the maximum error. The error here is the absolute value of $\alpha_{tgt}-\alpha_{end}$ which is the distance between the target leg angle and the actual final leg angle $\alpha_{end}$. We sampled 20 trajectories using the same range of initial condition and target angle as the training data, with an average of $4.8\deg$ error and a maximum error of $9.4\deg$ (the target controller has an average error of $2.5\deg$ and a maximum error of $6.2\deg$).

V DISCUSSION

V-A Transfer of Control in the Neural System

Our result suggests a framework where the neural system could be used to transfer the control from brain to spinal cord level. However, some points need to be clearly stated.

Firstly, the model assumes the learning occurs in human which requires the neurons to be able to compute the error and propagate the error back to the reflex synapses for learning. While there is evidence showing the existence of back-propagation within neurons, there is no prove showing that every neuron has such ability. If the neurons connected between brain and the spinal cord is not able to do back-propagation our approach would be physiological implausible.

The second point is that, although we enhance our physiological implausibility by creating a norm form of neural network structure, this structure still need improvement. During sensing, neurons have threshold to filter the low sensing data. This effect can be represented by adding a bias in our current structure. The bias can also be comprised in our learning process as variables.

Moreover, although our network considers the relationship between two neurons as multiplication in order to model the Presynaptic Inhibition which is commonly found in neuron systems, our swing leg model is a simple double pendulum which can extend to a muscle-skeletal model[23][24]. Specifically, the knee and ankle torques are generated by related Hill-type muscles, e.g. gastrocnemius. Each muscle produces a forces as a function of the muscle’s current stimulation, the muscle length and the muscle velocity [25]. By investigating the network relationship between each muscle’s stimulation and positive force/length/velocity feedback from different muscles, we might be able to understand how this adaptation shapes the controller structure in the muscle level.

V-B Learning Framework

Beside the biological plausibility, the structure, and its learning algorithm, although learned the target controller in our setting and can learn to switch control between sub-controls, there are a few points that are worth to be addressed which might inspire new ideas in the robot learning community.

First of all, by increasing the number of pairs of networks, the capability of the system will increase correspondingly, which indicates the ability to learn any control using demonstrations. Plus, there is no doubt that the network structure in our setting can be replaced by other forms like multi-layer perceptions (MLP). For a more complex task which composes a series of sub-tasks, our learning algorithm can distinguish different sub-tasks, this shows the potential of this method in AI reasoning field, as a comparison, current state-of-the-art techniques, using deep neural networks to fit the target control suffers from low explainability. These identified different controllers can be stored in a ’skill library’; those stored controllers can be used separately to achieving complex tasks that consist of several learned sub-tasks. Besides, each controller can be optimized independently using individual well-designed cost functions.

Besides that, our method is based on the assumption that each generator network is initialized with different weights if two sets of weights are equal, then at each step since the error between the output and the target are identical, they will have the same update, which leads to having the weights. Besides the situation as mentioned earlier, in a more general case, where two sets of weight are similar, this would lead to a slow learning process. In other words, the initialization weights are crucial to our algorithm. Since in our setting, we use a simple gradient descent method to optimize the error between prediction and target, in the MOSAIC model, a similar issue has been alleviated using the EM algorithm and hidden Markov model (HMM) based learning[26], thus, we can try to ease this issue by replacing gradient descent with other optimization methods.

VI CONCLUSION

We proposed a novel neural network architecture that uses multiplication as the primary relationship between neurons to imitate the Presynaptic Inhibition effect in neural systems. To use multiple simple neural network structure to learn from an unsegmented complex control policy, we used a predefined multi-phases swing leg control. We introduced a modulate model which is composed by several groups of generator and responsibility predictors. The generator predicts the current output and the responsibility predictor generates the weight of its corresponding generator. Using this model, we modeled the transfer of a swing leg control from the brain to the spinal cord level. We demonstrated that this model can learn when to switch on/off the generator and is able to automatically select the number of Generator it uses. We discussed our work in the context of both physiology and robot learning. For physiology, we can further extend this work by using more physical models like the muscle model which might ignite new physiology findings. For robot learning, we show the potential of this model in AI reasoning. We also discussed some improvements that can be implemented to our model.