Learning Sensory-Motor Associations from Demonstration

TL;DR

This paper introduces a method for learning reactive robot behaviors from human demonstrations using a reactive programming approach, enabling adaptable and robust sensor-motor associations represented in human-readable scripts.

Contribution

It presents a novel approach combining reactive programming with sensor-motor association learning from minimal demonstrations, enhancing robot adaptability.

Findings

Behaviors learned from a single demonstration.

Sensor and motor primitives improve robustness.

Reactive programming enables human-readable behavior scripts.

Abstract

We propose a method which generates reactive robot behavior learned from human demonstration. In order to do so, we use the Playful programming language which is based on the reactive programming paradigm. This allows us to represent the learned behavior as a set of associations between sensor and motor primitives in a human readable script. Distinguishing between sensor and motor primitives introduces a supplementary level of granularity and more importantly enforces feedback, increasing adaptability and robustness. As the experimental section shows, useful behaviors may be learned from a single demonstration covering a very limited portion of the task space.

Tables

Table 1. TABLE I: Motor Primitive descriptions

Primitive	Description
resource: wheels rotation
turn_toward	rotates body to face the target
turn_stop	angular velocity of the robot set to 0
resource: wheels translation
go_toward	moves toward the target, keeping a safety distance
go_stop	linear velocity set to 0
resource: head
look_at	moves to keep the target object in the center of FoV
resource: arm
point_toward	moves left arm to point toward the target
waving	waves the left arm
arm_freeze	stops moving the left arm

Equations

$$\mathcal{D}=\{x_t{\}}_{t=1:T},$$

$$f_{\theta }^{(1)}(x_t,z_t)=\{\begin{array}{ll}{a}_1(x_t,z_t)& \text{if}{e}_{{\theta }_1}(x_t)>0\\ ⋮& \\ a_n(x_t,z_t)& \text{if}{e}_{{\theta }_n}(x_t)>0\end{array},$$

$$f_{\theta }^{(2)}(x_t,z_t)=\sum _i{e}_{{\theta }_i}(x_t)a_i(x_t,z_t).$$

$$L(\theta )=\frac{1}{T}\sum _t\Vert x_{t+1}-f_{\theta }^{(2)}(x_t,z_t){\Vert }^2\forall x_t\in \mathcal{D}.$$

$$p(o^{i}_{t}|w^{i}_{t})\propto\exp\bigg{(}-\frac{|\phi_{i}(x^{i}_{t})-\phi_{i}(x_{t})|^{2}}{2\sigma_{i}}\bigg{)}$$

Full text

I Introduction

Behavior orchestration is the offline setup of activation rules of software primitives so that a robot autonomously performs a desired behavior online. In this paper, we propose a new method to generate behavior orchestration from human demonstration. The novelty of the proposed method is that it does not only learn the activation rules of primitives: it also learns how information should be exchanged between sensor primitives and motor primitives, i.e. it learns which sensory-motor associations should be performed by a robot to imitate a demonstrated behavior.

When composing a behavior from primitives, most methods encode the rules for transiting from one sub-behavior to the other. This may result in learning state transitions of a state machines or in generating a behavior tree in which edges encode the logic, and module activation relies on traversing the tree during run-time (see section II). While these methods use one type of primitive, we describe in this paper a method which distinguishes between sensor and motor primitives. The system does not only learn the activation rules of primitives, but also the associations that should be performed between sensor and motor primitives.

For example, a “look at” motor primitive could be associated to a human detection sensor primitive, creating the sensor-motor coupling required for the robot to look at a moving human. The same “look at” primitive could be associated with a ball detection sensor primitive, resulting in the robot looking at a moving ball. The demonstrator may also during demonstration alternate between looking at the ball and looking at her/his human partner. The goal of our approach is to find which sensor-motor associations were performed by the demonstrator, and the underlying rules the demonstrator applied to activate these associations.

Furthermore, when parsing the data collected during a human demonstration, the proposed method does not attempt to map it to a sequence of actions the robot may replicate. Instead, it detects which sensory-motor associations the demonstrator performed and then encodes the demonstrated behavior into non-sequential rules of activation of sensor-motor associations (e.g. “look at the ball when close to it”).

Finally, it organizes the behavior hierarchically, i.e., in a dynamic behavior tree.

The proposed approach has two particularities:

• •

It relieves the algorithm of generalizing observed task-space trajectories. For example, if the demonstrator directed herself toward an object, the algorithm will only learn that the sensory-motor loop between the detection of this object and the wheels of the robot is of interest. During run-time, the trajectory resulting from the activation of primitives when the robot directs itself toward the object will be different than the one observed during demonstration. It will nevertheless matches the intent of directing itself toward the object.

• •

The method generates behaviors that do not encode explicit goals to achieve, e.g. “first grasp the bottle then fill the glass”. Instead, sequences are implicitly encoded, e.g. “if no bottle in hand, activate primitives to grasp the bottle; if bottle in hand, activate the primitives to pour in the glass”. This example shows the generalization benefits: if the bottle is removed from the hand of the robot, an explicitly encoded sequence would be interrupted and require re-planning. The applied rule-based approach does not suffer from such difficulty.

The output of the proposed algorithm is an executable script that can be interpreted by Playful. Playful is an interpreter for behavior orchestration using the reactive programming paradigm. A Playful script consists of a series of declarative statements. Each statement relates a target (the output of a sensor primitive) to both a motor primitive (such as “look at”) and an activation rule (such as “when close to the robot”). The order of the statements in the script does not matter. Instead, at run-time, all rules of activation are continuously evaluated, and the motor primitives are activated or deactivated accordingly. Furthermore, the scripting language allows to group statements hierarchically to form new reusable higher-level primitives. This results in behavior trees, in which the activation status of branches are monitored online. Playful is further described in section III, but from this short overview, it can be understood that a Playful script encodes sensory-motor associations as presented above. It can be noted, that if in this paper the algorithm outputs a Playful script, the proposed approach is general and could be applied for usage with other reactive programming interpreters.

Because it distinguishes between sensor and motor primitives, the proposed method introduces a supplementary level of granularity: primitives are no longer the building blocks available to the algorithm for imitating the human demonstration. Instead, sensor-motor associations are. If the library of primitives has $n$ sensor primitives and $m$ motor primitives, then there are $n\times m$ sensor-motor associations available to the learning algorithm. Encoding the behavior using a high number of possible associations allows for the learning of rich interactive tasks. Furthermore, the sensor-motor associations implement feedbacks that by nature increase adaptability and robustness. Consequently, the demonstrated behavior is encoded in a fashion that grasps the intent and motion of the demonstrator beyond the sequence of high-level actions she or he performed.

In our experiments (section V), we consider an example where a human attempts to attract the attention of another human passing by, combining head and arm movements with walking. The scripts, which are reported in the paper, are produced from a single demonstration and remain compact while capturing the essence of the performed demonstration.

II Related work

II-A Learning from Demonstration

Learning from human teachers [1], has been a subject of study in the robotics community for decades. It raises multiple questions, including how humans should convey the demonstration and how to design learning algorithms [2]. LfD algorithms are classified as 1) Behavior Cloning (BC) or Inverse Reinforcement Learning (IRL), depending on whether the policy is encoded directly or through a reward function, and as 2) Task or Motion Control learning, depending on the nature of the state-action space. In this work we consider high-level LfD by encoding the policy directly into an executable Playful script, which describes a dynamic Behavior Tree as presented in Section II-B.

In high-level BC, the focus is to learn multi-step policies that combine sequences of primitive actions. Generally the corresponding approaches are broken into three parts:

1. segmentation of the demonstrations by identifying substructures, 2) learning the primitive actions and 3) learning the transitions between primitives. Typically the segmentation is performed using Hidden Markov Models (HMMs). For instance in [3], demonstrations are segmented using a beta-process HMM, and the primitives are learned using Dynamic Movement Primitives [4]. In this paper, we rather consider fixed pre-programmed primitives, divided between sensor and motor primitives. Instead of considering a sequence of active action primitives, we consider a sequence active sensor-motor couplings, where multiple couplings can be active simultaneously. This approach leads to more reactive behaviors and the specification is more modular.

The sequencing of primitive actions can also be learned hierarchically by encoding them in a Hierarchical Task Network (HTNs). In recent work [5], authors have described the encoding of the task structure by having a demonstrator narrating the task. This approach however does not allow for the activation of multiple primitives simultaneously.

II-B Behavior Trees

Behavior Trees (BTs) yield modularity and reactivity, by organizing the switching structure of hybrid dynamical systems. BTs have initially emerged in the computer gaming field as a replacement to Finite State Machines (FSM) for encoding the behavior of non players characters. In [6], Colledanchise and Örgen have formalized BTs and shown that they generalize the sequential behavior composition, the subsumption architecture and the decision trees.

In a sense, BTs are conceptually close to HTNs, which have been used for planning for three decades [7]. However BTs implement dynamic behaviors which are closely related to supervision in robotic systems. Thus they have been applied for high-level supervision of manipulation tasks in [8, 9], and an algorithm has been recently proposed to synthesize BTs using high-level planning [10], connecting high and low-level reasoning.

In this work we learn Playful scripts [11], which differs from the standard BT framework [12] by:

dividing action primitives in sensor and motor primitives 2. 2.

encoding the associations between these sensor and motor primitives 3. 3.

evaluating the BT at a fixed frequency, leading to activation and/or deactivation of its branches, resulting in higher behavioral reactivity. This differs from the previous aforementioned approaches in which logic is encoded in edges, and module activation relies on traversing the tree during operation.

II-C Learning Behavior Trees from Demonstration

Despite the various works on learning from human demonstration, to the best of our knowledge only few works have been carried in learning hybrid controllers from demonstration. The earliest work was performed using a subsumption architecture in [13]. However, this work focuses on navigation behaviors and does not learn a hierarchical grouping of primitives.

The closest to our approach are [14, 6, 12, 15]. In these works a BT is learned from demonstration by exploiting the equivalence between BTs and decision trees [6]. In [14], the C4.5 algorithm is used to first learn a decision tree. The tree is first flattened, and then simplified using a greedy algorithm. In [15], the authors extend this work by taking into account any logic using the UC Berkeley Expresso algorithm [16]. Our work shares similarities with these works, however because we separate sensor and motor primitives and produce a Playful script, we use a dedicated algorithm to convert the decision tree.

III Learning a Playful Script

III-1 Playful Framework

Playful is a software for the orchestration of robot behaviors [11]. It allows developers to compose applications via a list of declarative statements such as:

This playful script has a robot running ball detection (line 1), searching for a ball (line 2), looking at the one it detected (line 3) and attempt grasp when the ball is close to the robot (line 4). Sensory-motor associations between the ball detection sensor primitive and the motor primitives (look_at and grasp) are specified via the targeting keyword.

Note that the motor primitives are not associated directly to the sensor primitive, rather they are associated with the corresponding target. A target is a discretized sensory information item which data is continuously updated by the sensor primitive. A target is used as an association proxy between a sensor and a motor primitive, e.g look_at is associated to ball_detection via the ball target. In the rest of the paper, we use indifferently target-motor association and sensor-motor association.

Playful is based on reactive programming, i.e. the order of statements is of no importance, and no sequence of action is explicitly programmed. The three statements are continuously evaluated, and their primitives (here head_search, look_at and grasp) are activated when their evaluations (here, seen and close) return true. A priority is used to solve conflicts, for example head_search and look_at can not be active simultaneously because they share the resource head. Here, the higher priority of 2 for look_at takes precedence.

Evaluations, i.e. the activation rules, correspond to python code, e.g., close returns true if the distance between the robot and the ball is below a given threshold.

Playful scripts encode behavior trees in which nodes may be primitives that correspond to python code communicating with the robot middleware; but may also correspond to other Playful scripts. For example, in the grasping scenario of Listing 1, grasp could be a Playful script consisting of two statements, e.g. move_arm and move_hand.

Thus, the status of evaluations and priorities results in the activation and deactivation of branches at run-time, shaping the behavior of the robot. For a complete formalized description of Playful, we invite readers to refer to [11].

III-2 Problem Statement

Given a list of Playful sensor and motor primitives, we propose the automated generation of a Playful script based on data collected during human demonstration.

At the start of each learning from demonstration phase, the learner is given a set of observations (i.e., LfD dataset):

[TABLE]

where $t$ is a time index and $x_{t}$ encodes positions and velocities of the humans and objects in the scene.

Flat tree

The state dynamics $x_{t+1}=f_{\theta}(x_{t},z_{t})$ resulting from the execution of a Playful script, can be expressed by the following function:

[TABLE]

where $z_{t}$ is the Playful memory at time index $t$, $a_{i}$ is the result of the activation of the $i$th leaf node (i.e., sensory-motor primitive association in each branch), and $e_{\theta_{i}}:X\to\{0,1\}$ are the binary conditional-evaluation maps parametrized by some vector $\theta_{i}$.

We will subsequently call this representation, the flat tree, i.e. the raw list of sensory-motor association without any hierarchical grouping. Note, that we have relaxed the definition of $f_{\theta}$ by leaving out the priority mechanism.

Resources

The simplified definition of a Playful behavior in Equation 2 only holds for the case where evaluations $e_{\theta_{i}}$ are disjoint. This is not the case in practice, however when the leaf nodes $a_{i}$ control a given resource, e.g., specific DoFs such as the arm, the base or the head, the actions are limited to separate dimensions of the range space of $f_{\theta}$. In this case, the flat tree can be expressed as follows:

[TABLE]

Learning Loss

Given the priority simplification, learning a Playful script amounts to finding for each resource the parameters $\theta$ that minimize the classic LfD mean squared-error loss:

[TABLE]

The overall behavior is the union of all flat trees found for each resource, and a hierarchical grouping of nodes can be performed based on activation pattern similarities of the leaf nodes $a_{i}$. Note that there is always an equivalent flat tree corresponding to a hierarchical tree. The composition of a tree with several layers is beneficial to simplify the output script and provide reusable behaviors.

IV Learning Reactive Programs from Demonstrations

Algorithm 1 sketches our LfD method. Intuitively, the core of the algorithm can be viewed as being separated in the 3 flowing phases:

detect the most likely sequence of target-primitive activations based on a probabilistic inverse model, 2. 2.

train evaluations based on state-activation pairs (i.e, input-output) using supervised learning, 3. 3.

factorize the flat tree and generate the program.

IV-A Target-primitive Activation Detection

The first step of LRPD (i.e., line 3 and 5 of Algorithm 1), is to detect which target-primitive associations are active at each time step $t$ of our dataset $\mathcal{D}$. To do this we map the state space of a Hidden Markov Model (HMM) to activation patterns $w^{i}:t\mapsto\{0,1\}$ at each time step $t$ of target-primitive associations $i$ (e.g., ball-look_at in the example of Section III-1).

We set the uniform transition probabilities and model the observation probability at time $t$ of one such association $o^{i}_{t}$ by the following Gaussian emission densities:

[TABLE]

where $x^{i}_{t}$ is the state we would observe if the $i$th target-primitive was active. $\phi_{i}$ is some non-linear function of the state defined for the target-primitive $i$, (e.g., angular velocity when turning, heading direction when moving forward). $\sigma_{i}$ is a standard deviation term, which is parameter of the algorithm. We make use of the Viterbi algorithm to recover the most likely sequence of activation $\mathcal{W}$ for all the combinations of targets and primitives.

IV-B Binary Classifiers

Once the data points $x_{t}$ are labeled with target-primitive activation $w_{t}^{i}$, evaluation functions $e_{\theta_{i}}$ can be trained to activate the target-primitive for each time index $t$. See line 6 of Algorithm 1. In general, any binary classification method could be used (e.g., support vector machine, neural network). However, in our experiments, we define a non-linear feature $\phi_{d}:X\to\mathbb{R}$, which models the human-robot interaction distance, and simply identify intervals of its co-domain.

IV-C Factorization of the Tree

The last step of the algorithm consists of unifying the flat trees $\mathcal{T}_{flat}$ defined for each resource and grouping the nodes in hierarchies.

Grouping of nodes is performed by measuring similarity of their conditional evaluation functions $e_{\theta}$. Similarity between $e_{\theta}$ is difficult to define in the general case. In this work we simply compare the subset of the domain of $e_{\theta}$ where the evaluation is true. This matching is performed up to some error $\Delta$, which trades off complexity for fidelity.

V Experimental Results

We used the proposed approach to generate scripts that have a mobile humanoid robot (Softbank Robotics Pepper [17]) applying strategies to attract passing visitors to a stand. Each demonstration was the result of capturing two humans (i.e., a demonstrator to imitate, and a visitor).

In this section, we provide implementation details, and show how the learned robot behaviors capture successfully the different strategies applied by the demonstrators.

V-A Experimental setup

V-A1 Robot

We made use of a Pepper robot from Softbank Robotics, which is 1.2 meters in height and offers a total of 20 DoFs, 17 for the body and 3 for the base (see Figure 2). The base is omnidirectional and allows for holonomic navigation. Pepper is equipped with an IMU, which coupled with the wheel’s encoders, provides odometry. The odometry was used to update the position of the stand relative to the robot during all the experiments.

In this study, we used two of the three cameras available on the robot. The first is an RGB camera with a native resolution of 640*480 positioned in the forehead. The second is an ASUS Xtion3D sensor located in one of its eyes. We used these cameras in combination with OpenPose [18] for 3D human detection. The other sensors of the robot (sonars, infrared sensors, touch sensors and bumpers) were not used.

V-A2 Available Playful Primitives

Previously and independently of this study, we implemented a set of basic Playful primitives for Pepper, which are listed in Table I. Each primitive is associated with a resource, i.e. the robotic joint required for its activation. Primitives associated to the same resource may not activate at the same time. Because of the holonomic capabilities of the mobile base, wheels rotation and wheels translation refers to two separated resources.

Out of the eight primitives, four of them (turn_toward, go_toward, look_at and point_toward) have to be associated to a target for activation. In this study, three targets are set to be tracked by the Playful backend: the visitor (human detected using the cameras), the stand, and a virtual fixed position in front of it (tracked using odometry). This results in a total number of 16 possible primitives or target-primitives association that may be activated.

For the application of the algorithm presented in Section IV, an inverse model $\phi_{i}$ for each primitive has been defined to implement observation probabilities $p(o_{t}^{i}|w_{t}^{i})$. These models express the likelihood of observing a certain change in the state of the world given that the motor-coupling $w_{t}^{i}$ is active (e.g., for look at we consider the angle between target and the demonstrator field of view). All $\phi_{i}$ were normalized before training.

V-A3 Human demonstration

Motion capture was performed using a Vicon tracking device. Each dataset $\mathcal{D}$ captured the (fixed) position of the stand, the position of the visitor, the pose of the body of the demonstrator, the pose of the head of the demonstrator, and motion of the left hand of the demonstrator using a stick equipped with reflective markers.

Two participants (i.e, demonstrators) were recruited and given the instructions to try to attract a passing visitor to a stand. We required that he/she should 1) not touch persons or objects, 2) only use the left arm, and keep the right arm inactive and 3) apply common courtesy (e.g. not trying to attract to the stand by using physical constrain). Apart from these limitations, subjects were instructed to move freely in the experimental space. The subjects were not informed of the objective of the experiment.

The experimental space consisted of a corridor of 5 meters in length and 3 meters of width. The stand, a small pulpit, was located at the end and side of the corridor. The demonstrator starts at the stand, and waits for the visitor to enter the corridor. The visitor then enters the corridor, walking straight, before diverging toward the stand when at around 2 meters from it. Data recording stops when the visitor reaches the stand.

The demonstrations are shown in the associated video111https://youtu.be/-8pUC0rN-fQ. It can be seen that the two demonstrators apply different strategies. Demonstrator 1 goes in front of the stand while actively pointing towards it. Demonstrator 2 goes toward the visitor before backing toward the stand.

V-A4 Algorithm run

Algorithm 1 was applied, computing the most likely sequence of activation of target-primitive associations. These are presented for demonstrator 1 is represented in Figure 1 as a function of decreasing distance between the visitor and the stand. As the visitor approached, the demonstrator started to redirect himself toward the stand while pointing at it. From this sequence, overlapping target-primitives are then grouped together using a similarity score implementing a tolerance of 0.3 meter (dotted squares in Figure 1). These groups, labeled A and B, corresponded to branches in the subsequently generated Playful scripts. Related evaluations $e_{\theta}$ consisted of the boundary in terms of distance between the visitor and the stand observed during activations.

V-B Analysis of the learned scripts

The resulting Playful script is shown in Listing 2. It encodes this demonstrated behavior. For example, when the visitor is far (distance superior to 5.1), the branch A activates targeting the visitor, i.e. the robot is commanded to turn toward the visitor while looking at it (line 3). Similarly, when the visitor gets closer (distance below 2.7), the branch B activates, commanding the robot to look at the stand while turning and pointing towards it (line 7).

The same procedure was applied for demonstrator 2, resulting in Listing 3. This script reflects the different strategy applied by the second demonstrator, for example first approaching the visitor (line 4) before backing toward the stand (line 3).

V-B1 Implementation and result

To the generated Playful script were added two nodes:

• •

A higher priority security node, that has the robot safely deactivating itself when its battery is low, or its head is touched

• •

A lower priority default behavior node, that has the robot moving randomly the arms and the head, while going back to the stand. This node provides a default simple set of primitives that activate when the visitor is not detected.

The generated scripts were not edited in any other way, and directly executed on the robot, as shown in Figure 2. The resulting behaviors can also be seen on the support video, where it can be observed that the robot replicates with fidelity the high level strategy of each demonstrators as described in the previous section (V-A4).

V-C Analysis of the learned behavior

Figure 3.a displays in blue the demonstration performed by the second user in the video attachment. The arrows represent his body orientation. The trajectory of the passing visitor is represented in red. Changes in intensity (from light to dark) represent passing time. The visitor first goes toward the stand before walking away. The demonstrator first goes toward the visitor before reverting toward the stand (i.e., green dot), his body always turning toward the visitor.

Figures 3.b,c,d show three executions of the learned script on the Pepper robot shown on Figure 2. In (3.b), the visitor has a behavior similar as during the demonstration. In (3.c), the visitor passes by without visiting the stand. In (3.d), the visitor first pass-by the stand, then comes back to it and finally leaves.

The behavior performed by the robot is robust to the visitor diverging from the demonstrated behavior. In the three cases, the robot properly either goes toward the visitor or back toward the stand. In addition, similarly to the demonstrator, the robot keeps his body turned toward the visitor at all time, which shows how our approach is able to capture the “spirit” underlying the task using a single demonstration.

VI Conclusion

We propose a new method of learning from demonstration, which encodes the learned behavior as sensory-motor associations organized hierarchically. Because of the feedback it enforces, the method can robustly generate Playful scripts based on a single demonstration. A direct benefit of encoding behavior into Playful script is the liveliness and behavioral reactivity displayed by the robot during runtime. The described algorithm is first order as it does not encode rules for generating an explicit sequence of actions suitable to reach an explicit goal.

Next we will investigate how this approach can be used to learn evaluations end-to-end from the raw kinematic states directly and how it could work alongside another learning system which allows for higher level planning.