Hierarchical Reinforcement Learning of Locomotion Policies in Response to Approaching Objects: A Preliminary Study

TL;DR

This paper investigates hierarchical reinforcement learning for legged robot locomotion in response to approaching objects, demonstrating improved learning efficiency in a simulated environment with partial observability.

Contribution

It introduces a hierarchical reinforcement learning framework for reactive locomotion policies in robots responding to dynamic objects, inspired by animal behavior.

Findings

Hierarchical RL improves learning efficiency.

Partial observability affects the learning process.

Simulation results support the proposed approach.

Abstract

Animals such as rabbits and birds can instantly generate locomotion behavior in reaction to a dynamic, approaching object, such as a person or a rock, despite having possibly never seen the object before and having limited perception of the object's properties. Recently, deep reinforcement learning has enabled complex kinematic systems such as humanoid robots to successfully move from point A to point B. Inspired by the observation of the innate reactive behavior of animals in nature, we hope to extend this progress in robot locomotion to settings where external, dynamic objects are involved whose properties are partially observable to the robot. As a first step toward this goal, we build a simulation environment in MuJoCo where a legged robot must avoid getting hit by a ball moving toward it. We explore whether prior locomotion experiences that animals typically possess benefit the learning of a reactive control policy under a proposed hierarchical reinforcement learning framework. Preliminary results support the claim that the learning becomes more efficient using this hierarchical reinforcement learning method, even when partial observability (radius-based object visibility) is taken into account.