Learning and Composing Primitive Skills for Dual-arm Manipulation

TL;DR

This paper introduces a novel learning framework for dual-arm robots that learns primitive skills from demonstrations and composes them to perform complex tasks, enhancing autonomous generalization in manipulation.

Contribution

It proposes a modular, learning-based model using dynamic movement primitives for dual-arm manipulation, enabling skill learning and composition from human demonstrations.

Findings

Successfully taught iCub to perform dual-arm pick-and-place tasks.

Framework generalizes to novel manipulation scenarios without additional demonstrations.

Demonstrates improved autonomous manipulation capabilities.

Abstract

In an attempt to confer robots with complex manipulation capabilities, dual-arm anthropomorphic systems have become an important research topic in the robotics community. Most approaches in the literature rely upon a great understanding of the dynamics underlying the system's behaviour and yet offer limited autonomous generalisation capabilities. To address these limitations, this work proposes a modelisation for dual-arm manipulators based on dynamic movement primitives laying in two orthogonal spaces. The modularity and learning capabilities of this model are leveraged to formulate a novel end-to-end learning-based framework which (i) learns a library of primitive skills from human demonstrations, and (ii) composes such knowledge simultaneously and sequentially to confront novel scenarios. The feasibility of the proposal is evaluated by teaching the iCub humanoid the basic skills to succeed on simulated dual-arm pick-and-place tasks. The results suggest the learning and generalisation capabilities of the proposed framework extend to autonomously conduct undemonstrated dual-arm manipulation tasks.

Equations

$$\tau \dot{z}$$

$$\tau \dot{x}$$

$$\tau \dot{\mathbf{\eta }}$$

$$\tau \dot{\mathbf{q}}$$

$$\left[\begin{array}{c}{\dot{\mathbf{y}}}_L\\ {\dot{\mathbf{y}}}_R\end{array}\right]={\mathbf{G}}^T\sum _{j=1}^J{w}_j{\dot{\mathbf{y}}}_{o_j}+\sum _{k=1}^K{w}_k\left[\begin{array}{c}{\dot{\mathbf{y}}}_{C_{L,k}}\\ {\dot{\mathbf{y}}}_{C_{R,k}}\end{array}\right],$$

Full text

1 INTRODUCTION

Complex manipulation tasks can be achieved by endowing anthropomorphic robots with dual-arm manipulation capabilities. Bi-manual arrangements extend the systems competences to efficiently perform tasks involving large objects or assembling multi-component elements without external assistance. These systems not only deal with the challenges of single-arm manipulators, such as trajectory planning and environmental interaction, but also require an accurate synchronisation between arms to avoid breaking or exposing the handled object to stress.

Traditional approaches have addressed the aforementioned challenges by means of control and planning-based methods [18]. These methods depend upon an excellent understanding of the exact model underlying the system’s and task’s dynamics, which are commonly approximated to make the calculations computationally tractable [13]. On top of that, some of these methods lack scalability and generalisation capabilities, involving hand-defining all possible scenarios and actions [5, 3]. All these issues have motivated the use of more natural techniques for robot programming, such as \acfLbD, in which a human movement is recorded to be later reproduced by a robot.

Despite the encouraging possibilities offered by adopting human knowledge for robot control, teaching complex systems, such as dual-arm manipulators, to respond and adapt to a broad case of scenarios remains an open challenge. Particularly, it is expected from a dual-arm system to generalise the provided demonstrations to confront novel scenarios in (a) the task space to deal with the changing requirements about trajectory planning and environmental interaction, and (b) the relative space to ensure the essential synchronisation between arms [12]. However, current learning-based architectures in the literature pursuing autonomy and robustness against the dynamic and unpredictable real-world environments are limited to single-arm arrangements [15, 17, 4]. Contrarily, learning-based frameworks for dual-arm robots do not generalise to undemonstrated states, thus being limited to highly controlled scenarios [21, 7, 19].

This paper presents a novel learning-based framework which endows a dual-arm system with a real-time and generalisable method for manipulation in undemonstrated environments (see Figure 1). The framework models a dual-arm manipulator with a set of \aclpDMP laying in two orthogonal spaces to tackle the task’s requirements separately from the synchronisation constraints. The modularity of the \acpDMP is leveraged to (i) create a library of primitive skills from human demonstrations, and (ii) exploit primitive skills simultaneously and sequentially to create complex behaviours. The potential of the proposal is demonstrated in simulation after recording skills with the iCub humanoid through kinaesthetic guiding. The results suggest the proposal’s suitability to endow a dual-arm robot with the necessary learning and generalisation capabilities to autonomously address novel manipulation tasks.

2 DUAL-ARM SYSTEM MODELISATION

This paper pursues an end-to-end learning-based framework which endows a dual-arm system with enhanced generalisation capabilities, meets the synchronisation constraints, and is easily programmable by non-robotics-experts. This work addresses all these requirements by means of learnable and composable primitive skills represented as \acfpDMP [9]. This section firstly overviews \acpDMP and its use in the literature. It then introduces the proposed typology of actions in a dual-arm system, which allows leveraging the strengths of a \acDMP-based modelisation in the dual-arm context.

2.1 Dynamic Movement Primitives

\acp

DMP are a versatile tool for modelling and learning complex motions. They describe the dynamics of a primitive skill as a spring-damper system under the effect of a virtual external force called coupling term. This coupling term allows for learning and reproducing any dynamical behaviour, i.e. primitive skill. Importantly, (a) coupling terms can be learnt from human demonstrations, (b) they can be efficiently learned and generated, (c) a unique demonstration is already generalisable, (d) convergence to the goal is guaranteed, and (e) their representation is translation and time-invariant. Because of all these properties, \acpDMP are adapted to constitute the fundamental building blocks of this work. Next follows an introduction about \acpDMP and their usage to encode positional and orientational dynamics, and an overview of some coupling terms in the literature.

2.1.1 Positional Dynamics.

Let the positional state of a one-\acDoF system be defined by its position, linear velocity and acceleration. Then, the system’s state transition is defined with non-linear differential equations as:

[TABLE]

where $\tau$ is a scaling factor for time, $x$ is the system’s position, $z$ and $\dot{z}$ respectively are the scaled velocity and accelaration, $\alpha_{x}$ and $\beta_{x}$ are constants defining the positional system’s dynamics, $g_{x}$ is the model’s attractor, and ${f{{}_{{x}}}(\bm{\cdot})}$ is the coupling term. The coupling term applying at multiple \acpDoF at once is defined as $\mathbf{f}{{}_{{x}}}(\bm{\cdot})$. The system will converge to $g_{x}$ with critically damped dynamics and null velocity when ${\tau>0}$, ${\alpha_{x}>0}$, $\beta_{x}>0$ and ${\beta_{x}=\alpha_{x}/4}$ [9].

2.1.2 Orientational Dynamics.

A possible representation of orientations is the unit quaternion ${\mathbf{q}\in{}^{4}=\mathbb{S}^{3}}$ [20]. They encode orientations of a system as a whole, thus ensuring the stability of the orientational dynamics integration. Let the current orientational state of a system be defined by its orientation, angular velocity and acceleration. Then, the orientational state transition is described by the following non-linear differential equations:

[TABLE]

where $\mathbf{q}$ is the system’s orientation, $\bm{\eta}$ and $\bm{\dot{\eta}}$ respectively are the scaled angular velocity and acceleration, $\alpha_{q}$ and $\beta_{q}$ are constants defining the system’s dynamics, $\mathbf{g}_{q}\in\mathbb{S}^{3}$ is the model’s attractor, and ${\mathbf{f}{{}_{{q}}}(\bm{\cdot})\in{}^{4}}$ is the coupling term. The operators $\log(\cdot)$, $\ast$, and $\bar{\mathbf{q}}$ denote the logarithm, multiplication and conjugate operations for quaternions, respectively.

2.1.3 Coupling Terms.

Coupling terms describe the system’s behaviour, thus being useful to learn and retrieve any primitive skill. They are commonly used to encode the positional [9] and orientational [20] dynamics of a motion. Coupling terms are modelled in each dimension as a weighted linear combination of non-linear \acpRBF distributed along the trajectory. Thus, learning a certain movement relies on finding the weights of the \acpRBF which closely reproduce a demonstrated skill.

More complex behaviours may be achieved by exploiting an additional coupling term simultaneously with the motion-encoding one. This approach has been used to avoid joint limits and constraining the robot’s workspace via repulsive forces pushing the system away from these limits [6]. Coupling terms have also been leveraged for obstacle avoidance with an analytic biologically-inspired approach describing how humans steer around obstacles [8, 17]. Another use is for environmental and self-interaction purposes by means of a controller tracking a desired force profile [7]. To the best of the authors’ knowledge, the practice of using coupling terms simultaneously has been limited to two primitive skills acting on the same frame or space [15]. Contrarily, this work further exploits the \acDMP modularity to describe a dual-arm system in two orthogonal spaces with the purpose of facing complex scenarios by composing multiple coupling terms.

2.2 Dual-arm Primitive Skills Taxonomy

Skills for single-arm manipulation have been well analysed in the robotics community. While some of this knowledge can be extrapolated for a dual-arm manipulator as a whole, their complexity resides in the arms interaction. In the context of manipulation via a dual-arm system, a possible classification of any primitive skill falls into two groups: (a) absolute skills, which imply a change of configuration of the manipulated object in the Cartesian or absolute space $\mathcal{S}_{a}$, e.g. move or turn an object in a particular manner, and (b) relative skills, which exert an action on the manipulated object in the object or relative space $\mathcal{S}_{r}$, e.g. opening a bottle’s screw cap, or hold a parcel employing force contact.

Each type of primitive skill uniquely produces movement in its space since they lay in orthogonal spaces such that $\mathcal{S}_{a}\perp\mathcal{S}_{r}$. It is natural to expect from a dual-arm system to simultaneously carry out, at least, one absolute and one relative skill to accomplish a task. Let us analyse the task of moving a bottle to a particular position while opening its screw cap. Both end-effectors synchronously move to reach the desired configuration (absolute skill). At the same time, the left end-effector is constrained to hold the bottle upright (relative skill), while the right end-effector unscrews the cap (relative skill).

2.3 Dual-arm DMP-based Modelisation

Given the variety of primitive skills that a dual-arm system can execute, this work seeks to model the robotic platform in a generalisable yet modular fashion, which accounts for both absolute and relative skills. To this aim, let us consider the closed kinematic chain depicted in Figure 2 operating in a \ac3D workspace ${\mathcal{W}={}^{3}\times\mathrm{SO}(3)}$. Each arm $i$, where $i=\{L,\;R\}$, interacts with the same object $\mathcal{O}$. In this context, the absolute skill explains the movement of the object $\mathcal{O}$ in the workspace $\mathcal{W}=\mathcal{S}_{a}$, while the relative skill describes the actions of each end-effector $i$ in $\mathcal{S}_{r}$, i.e. with respect to the object’s reference frame $\{\mathcal{O}\}$. Note that $\{\mathcal{O}\}$ is the centre of the closed-chain dual-arm system.

The state of the closed-chain dual-arm system in the workspace can be described by the position/orientation, linear/angular velocities and accelerations of $\{\mathcal{O}\}$ in $\mathcal{S}_{a}$. As introduced previously, the system’s state transition is subjected to its modelled dynamics. Figure 2 illustrates the proposed modelisation of the system’s dynamics in $\mathcal{S}_{a}$ as a set of \acpDMP acting between the objects’s frame $\{\mathcal{O}\}$ and its goal configuration $\mathbf{g}_{o}$, which accounts for a desired goal position $\mathbf{g}_{o_{x}}\in{}^{3}$ and orientation $\mathbf{g}_{o_{q}}\in{}^{4}$. Therefore, three positional \acpDMP as in (1)-(2) and one orientational \acDMP as in (3)-(4) are required to encode the system’s dynamics in the absolute space ${\mathcal{S}_{a}={}^{3}\times\mathrm{SO}(3)}$.

In the relative space $\mathcal{S}_{r}$, the dynamics of each end-effector are modelled as \acpDMP referenced to the objects’s frame $\{\mathcal{O}\}$. Since ${\mathcal{S}_{r}={}^{3}\times\mathrm{SO}(3)}$, each end-effector dynamic’s in the relative frame is described by three positional \acpDMP as in (1)-(2) and one orientational \acDMP as in (3)-(4).

Any action referenced to the object’s frame can be projected to the end-effectors using the grasping geometry $\mathbf{G}$ of the manipulated object. This allows computing the required end-effector control commands to achieve a particular absolute task. A detailed explanation of this transformation can be found in [12].

3 LEARNING-BASED DUAL-ARM MANIPULATION

To endow a dual-arm manipulator with autonomy and robustness in novel scenarios while being easily programmable and customisable by non-robotics-experts, this work has decomposed and modelled the system’s dynamics and synchronisation constraints as primitive skills lying in the system’s absolute and relative space. Leveraging the formulated modelisation, this work proposes the framework schematised in Figure 3 which creates and manages a library of primitive skills. The framework has two components: (i) a learning module that learns a set of primitive skills from human demonstrations, and (ii) a manager module that combines simultaneously and sequentially these primitives to address a wide range of complex tasks in unfamiliar environments.

3.1 Library Generation

A primitive skill is represented by its coupling term and frame of reference, i.e. either absolute or relative. Learning coupling terms only requires a human demonstrator teaching the characteristic skill. As previously introduced, different coupling terms might be better formulated with different mathematical representations, e.g. a weighted combination of non-linear \acpRBF to encode the dynamics of a task, an analytical obstacle avoidance expression, or among others, a force profile to control the environmental interaction.

The modularity offered by the proposed \acDMP-based formulation and its use in two different spaces tackles the hindrance and ambiguity arising when demonstrating all features of a dual-arm task in an all-at-once fashion. This means that instead of learning a task as a whole, the framework harvests a collection of primitive skills. Creating a repertoire of skills referred to as a library, allows the demonstrator to teach in a one-at-a-time fashion, i.e. to focus on one feature of the demonstration at a time [4]. Moreover, this modular library can be employed for movement recognition purposes, where a demonstrated skill can be compared against the existing ones in the library. If the observed behaviour does not match any existing primitive, it is identified as a new skill and can be added to the framework’s library [10].

3.2 Attaching Semantics

The framework needs additional information to successfully conduct a dual-arm manipulation task. Let us consider the robotic task of opening a bottle’s screw cap, where the system needs to select a proper sequence of primitive skills in order to succeed (see Figure 3). This is first a grasping, where each end-effector holds a different component of the bottle, then a synchronous turning referenced in the system’s relative space and finally, a placing and releasing primitives. Therefore, in order to ease this action selection, it is essential to attach a semantic description to each primitive skill.

Semantic labels bridge the gap between the low-level continuous representation of primitives and the high-level description of actions and their influence on objects. An approach to tackle the object affordances challenge consists in combining features from the object and their surroundings to infer on a suitable grasp-action based on their purpose of use [1, 2]. The combination of such elements builds the relationship between context, actions and effects that provide a cognitive reasoning of an object affordance.

3.3 Library Management

Each coupling term stored in the framework’s library represents a particular absolute or relative primitive skill. Reproducing a skill consists in using its coupling term as $\mathbf{f}{{}_{{x}}}(\bm{\cdot})$ or $\mathbf{f}{{}_{{q}}}(\bm{\cdot})$ in (1)-(4). This computation retrieves the skill’s required accelerations, which can be integrated over time to obtain the skill’s velocities $\dot{\mathbf{y}}_{o}$ for an absolute primitive or $\dot{\mathbf{y}}_{C_{i}}$ for the end-effector $i$ relative primitive.

The individual retrieval of primitives already accounts for the inner \acDMP generalisation capabilities, such as different start and goal configurations, as well as obstacle locations. However, these primitive skills need to be combined to generate more complex movements, such as a pick-and-place task of a bottle accounting for the presence of unexpected obstacles (absolute space), while opening the bottle’s screw cap considering the exerted force (relative space). The presented framework addresses this prerequisite by simultaneously combining different absolute and relative skills as:

[TABLE]

where ${\dot{\mathbf{y}}_{i}\in{}^{6}}$ describes the linear and angular velocity commands of the ${i=\{L,\;R\}}$ end-effector satisfying the set of activated primitive skills, ${\mathbf{G}\in{}^{6\times 12}}$ is the global grasp map of the two end-effectors grasp matrices as described in [13], and ${\dot{\mathbf{y}}_{o_{j}}\in{}^{6}}$ and ${\dot{\mathbf{y}}_{C_{i,k}}\in{}^{6}}$ are the velocities of the ${j\in[1,\;J]}$ absolute and ${k\in[1,\;K]}$ relative primitive skill stored in the library. Absolute and relative skill selection is conducted with the weights $w_{j}$ and $w_{k}$, respectively.

The resulting framework does not only combines skills simultaneously, but also sequentially. This allows the execution of a complex task composed of a sequence of primitives. To do so, a primitive skill is executed by initialising it with the full state (pose, velocities and accelerations) of its predecessor primitive skill. Such an initialisation avoids abrupt jumps in the system’s state.

4 RESULTS and EVALUATION

The proposed framework has been evaluated on the iCub humanoid robot. Particularly, the real platform has been used to load the framework’s library with a set of primitive skills learnt from human demonstrations. These skills have been employed in simulation to conduct dual-arm pick-and-place tasks of a parcel in novel scenarios, demonstrating the proposal’s potential for humanoid robots.

4.1 Experimental Platform

iCub is an open source humanoid robot with 53 \acpDoF [11] (see Figure 4). The most relevant ones in this work are the three-\acpDoF on the torso, the two seven-\acpDoF arms equipped with a torque sensor on the shoulder, and the two nine-\acpDoF anthropomorphic hands with tactile sensors in the fingertips and palm.

iCub operates under \acsYARP. The deployment of the proposed framework on the iCub platform is schematised in Figure 4. Mainly, four big functional modules can be distinguished: (i) the proposed framework described in this paper (blue blocks), (ii) the real/simulated platform with its visual perception, joint sensors and actuators (magenta blocks), (iii) the end-effectors control via the built-in \acYARP Cartesian controller [16] and an ad-hoc external torso controller (green blocks), and (iv) the \acsHRI interface to parameterise the desired start and goal configurations for the task, and retrieve the robot’s status (red blocks).

4.2 Learning Primitive Skills from Demonstration

For the system to succeed on the dual-arm pick-and-place of a parcel task in novel environments, the framework’s library needs to be loaded with the absolute primitive skills of (i) pick-and-place dynamics on a horizontal surface, (ii) rotational motion around the z-axis, and (iii) obstacle avoidance. Moreover, since the parcel has to be grasped by lateral contact of both end-effectors, the library also requires a relative skill to ensure grasp maintenance, i.e. prevention of contact separation. All these primitive skills have been demonstrated via kineasthetic guiding on the real iCub humanoid robot. To this aim, all joints have been set in gravity compensation, allowing the demonstrator to physically manoeuvre the robot through each primitive. Figure 4 depicts the kineasthetic teaching of obstacle avoidance and grasp maintenance primitives.

During the demonstrations, proprioception information is retrieved via \acsYARP ports to learn the coupling terms $\mathbf{f}{{}_{{x}}}(\bm{\cdot})$ and $\mathbf{f}{{}_{{q}}}(\bm{\cdot})$ in (1)-(4) characterising the different skills. For the pick-and-place and rotational dynamics, the coupling terms are encoded as a weighted linear combination of non-linear \acpRBF distributed along the trajectory as in [9]. The obstacle avoidance is learnt by finding the best-fitting parameters of the biologically-inspired formulation as in [17]. Finally, the grasp maintenance skill is learnt by setting the parcel’s grasping geometry as a pose tracking reference as in [7].

4.3 Experiments on Simulated iCub Humanoid

The evaluation of the framework on the pick-and-place setup has been conducted on a simulated iCub robotic platform. Particularly, the four primitive skills previously learnt and loaded in the framework’s library are simultaneously and sequentially combined to conduct three consecutive dual-arm pick-and-place task in novel environments (see Figure 5).

Given an initial random configuration laying on the table and within iCub’s workspace (see Figure 5), the first action consists of grasping the parcel. This is achieved by retrieving the parcel’s configuration, then use the learnt parcel’s geometry to compute the grasping points, and finally approach them laterally via the middle-setpoints displayed as red and blue prisms for the right and left end-effector, respectively (see Figure 5-Figure 5). From this stage on, the grasp maintenance skill ensures that both end-effectors are in flat contact with the box to avoid undesired slippage.

The following three consecutive movements require picking-and-placing the parcel between different configurations laying on the central, right and left side of iCub’s workspace. The former pick-and-place does not require avoiding any obstacle, thus the built-in \acpDMP generalisation capabilities are sufficient to address this task (see Figure 5-Figure 5). However, the two latter pick-and-place tasks involve adapting the learnt dynamics to address novel scenarios. When the obstacle (red sphere) is collinear with the start and goal positions, i.e. below the demonstrated task (green trajectory), the iCub humanoid circumnavigates the obstacle from the top (see Figure 5-Figure 5). Instead, for an obstacle located forward the demonstration, the framework guides the system through a collision-free trajectory near iCub’s chest (see Figure 5-Figure 5).

The experimental evaluation conducted with the simulated iCub humanoid robot has demonstrated various of the aforementioned framework’s features. Having a repertoire of primitive skills available in the framework’s library allows exploiting them simultaneously and sequentially to confront complex tasks in novel scenarios. The reported case is one of the 16 successful experiments out of a total of 20 trials. In all cases, the robot had to accomplish the three consecutive dual-arm pick-and-place tasks with different start and goal locations, while avoiding novel obstacles and ensuring grasp maintenance. Failure in any of these tasks made the trial unsuccessful. Interestingly, in the four failed trials one of iCub’s forearms collided with the obstacle. This is because the biologically-inspired obstacle avoidance formulation only considers the carried object and should be extended to the object-arm space. The flexibility of the proposed framework could be leveraged to integrate in its library a potential field-inspired approach for obstacle avoidance which also checks for link collisions [14].

5 FINAL REMARKS and FUTURE WORK

This work has presented a novel end-to-end learning-based framework which endows a dual-arm manipulator with real-time and generalisable manipulation capabilities. The framework is built upon the proposed extension of the \acDMP-based modelisation for dual-arm systems, which considers two different frames to reference the movement generation, force interaction and constraints requirements. Based on this arrangement, the proposed framework is twofold: (i) learns from human demonstrations to create a library of primitive skills, and (ii) combines such knowledge simultaneously and sequentially to confront novel scenarios.

The suitability of the proposed approach has been demonstrated in a dual-arm pick-and-place setting, where the iCub humanoid first learnt a repertoire of primitive skills from human demonstrations and then composed such knowledge to successfully generalise to novel scenarios. The framework is not restricted to the presented experimental evaluation nor platform. Any system capable of learning from demonstrations can benefit from this work. Moreover, the framework’s modularity allows loading to its library any primitive skill that might be required for dual-arm manipulation purposes.

Future work will significantly extend the library of primitive skills such that more challenging dual-arm manipulation behaviours can be addressed within the framework. In this regard, imminent efforts will focus on learning force-dependant primitive skills or other actions requiring complex synchronisation between end-effectors, such as the opening of a bottle’s screw cap or succeeding in the peg-in-a-hole tasks.

ACKNOWLEDGMENTS

This work has been partially supported by ORCA Hub EPSRC (EP/R026173/1) and consortium partners.