MP3: Movement Primitive-Based (Re-)Planning Policy

TL;DR

This paper presents MP3, a deep reinforcement learning approach that integrates movement primitives for smooth, adaptable robot motion planning capable of online replanning in complex environments.

Contribution

It extends previous MP-based RL methods by enabling online replanning and adaptation during motion execution in stochastic, sparse reward domains.

Findings

Improved performance in sparse reward settings

Effective online replanning in stochastic environments

Enhanced adaptability over prior MP-RL approaches

Abstract

We introduce a novel deep reinforcement learning (RL) approach called Movement Primitive-based Planning Policy (MP3). By integrating movement primitives (MPs) into the deep RL framework, MP3 enables the generation of smooth trajectories throughout the whole learning process while effectively learning from sparse and non-Markovian rewards. Additionally, MP3 maintains the capability to adapt to changes in the environment during execution. Although many early successes in robot RL have been achieved by combining RL with MPs, these approaches are often limited to learning single stroke-based motions, lacking the ability to adapt to task variations or adjust motions during execution. Building upon our previous work, which introduced an episode-based RL method for the non-linear adaptation of MP parameters to different task variations, this paper extends the approach to incorporating replanning strategies. This allows adaptation of the MP parameters throughout motion execution, addressing the lack of online motion adaptation in stochastic domains requiring feedback. We compared our approach against state-of-the-art deep RL and RL with MPs methods. The results demonstrated improved performance in sophisticated, sparse reward settings and in domains requiring replanning.