Reinforcement Learning for Ballbot Navigation in Uneven Terrain

TL;DR

This paper introduces an open-source MuJoCo simulation for ballbot navigation on uneven terrain, demonstrating that classical RL methods can effectively learn navigation policies with reasonable data efficiency, surpassing traditional control approaches.

Contribution

It provides the first open-source RL-compatible simulation environment for ballbot navigation and shows RL's effectiveness in this domain with minimal data.

Findings

RL policies successfully navigate uneven terrain

Effective training achieved within four to five hours

Open-source code available for further research

Abstract

Ballbot (i.e. Ball balancing robot) navigation usually relies on methods rooted in control theory (CT), and works that apply Reinforcement learning (RL) to the problem remain rare while generally being limited to specific subtasks (e.g. balance recovery). Unlike CT based methods, RL does not require (simplifying) assumptions about environment dynamics (e.g. the absence of slippage between the ball and the floor). In addition to this increased accuracy in modeling, RL agents can easily be conditioned on additional observations such as depth-maps without the need for explicit formulations from first principles, leading to increased adaptivity. Despite those advantages, there has been little to no investigation into the capabilities, data-efficiency and limitations of RL based methods for ballbot control and navigation. Furthermore, there is a notable absence of an open-source, RL-friendly simulator for this task. In this paper, we present an open-source ballbot simulation based on MuJoCo, and show that with appropriate conditioning on exteroceptive observations as well as reward shaping, policies learned by classical model-free RL methods are capable of effectively navigating through randomly generated uneven terrain, using a reasonable amount of data (four to five hours on a system operating at 500hz). Our code is made publicly available.