Learning Setup Policies: Reliable Transition Between Locomotion Behaviours

TL;DR

This paper introduces a novel training method for setup policies that enable smooth transitions between pre-trained DRL locomotion behaviors, significantly improving success rates over complex terrains.

Contribution

The authors develop a new approach for training setup policies that effectively bridge pre-trained DRL policies, enhancing modular control for diverse terrains.

Findings

Setup policies increased success rate on jump terrain from 51.3% to 82.2%.

Success rate on obstacle sequences improved from 1.9% to 71.2%.

Method outperforms existing transition learning approaches.

Abstract

Dynamic platforms that operate over many unique terrain conditions typically require many behaviours. To transition safely, there must be an overlap of states between adjacent controllers. We develop a novel method for training setup policies that bridge the trajectories between pre-trained Deep Reinforcement Learning (DRL) policies. We demonstrate our method with a simulated biped traversing a difficult jump terrain, where a single policy fails to learn the task, and switching between pre-trained policies without setup policies also fails. We perform an ablation of key components of our system, and show that our method outperforms others that learn transition policies. We demonstrate our method with several difficult and diverse terrain types, and show that we can use setup policies as part of a modular control suite to successfully traverse a sequence of complex terrains. We show that using setup policies improves the success rate for traversing a single difficult jump terrain (from 51.3% success rate with the best comparative method to 82.2%), and traversing a random sequence of difficult obstacles (from 1.9% without setup policies to 71.2%).