Modulating Reservoir Dynamics via Reinforcement Learning for Efficient Robot Skill Synthesis

TL;DR

This paper introduces a reservoir computing framework combined with reinforcement learning to enable efficient robot skill learning and online modulation, allowing robots to generate diverse movements beyond initial demonstrations.

Contribution

The work presents a novel RC-based Learning from Demonstration method that incorporates RL for dynamic context modulation, enabling out-of-distribution movement generation without retraining the reservoir.

Findings

Effective generation of reaching movements with obstacle avoidance.

RL-driven context modulation extends action repertoire.

Efficient learning with low-dimensional context and no iterative training.

Abstract

A random recurrent neural network, called a reservoir, can be used to learn robot movements conditioned on context inputs that encode task goals. The Learning is achieved by mapping the random dynamics of the reservoir modulated by context to desired trajectories via linear regression. This makes the reservoir computing (RC) approach computationally efficient as no iterative gradient descent learning is needed. In this work, we propose a novel RC-based Learning from Demonstration (LfD) framework that not only learns to generate the demonstrated movements but also allows online modulation of the reservoir dynamics to generate movement trajectories that are not covered by the initial demonstration set. This is made possible by using a Reinforcement Learning (RL) module that learns a policy to output context as its actions based on the robot state. Considering that the context dimension is typically low, learning with the RL module is very efficient. We show the validity of the proposed model with systematic experiments on a 2 degrees-of-freedom (DOF) simulated robot that is taught to reach targets, encoded as context, with and without obstacle avoidance constraint. The initial data set includes a set of reaching demonstrations which are learned by the reservoir system. To enable reaching out-of-distribution targets, the RL module is engaged in learning a policy to generate dynamic contexts so that the generated trajectory achieves the desired goal without any learning in the reservoir system. Overall, the proposed model uses an initial learned motor primitive set to efficiently generate diverse motor behaviors guided by the designed reward function. Thus the model can be used as a flexible and effective LfD system where the action repertoire can be extended without new data collection.