Learning to Run with Potential-Based Reward Shaping and Demonstrations from Video Data

TL;DR

This paper introduces a method that uses human running videos to shape reward functions in reinforcement learning, significantly improving humanoid robot running speed and overcoming sub-optimal human movement patterns.

Contribution

The paper presents a novel approach combining potential-based reward shaping with video data to enhance reinforcement learning for humanoid locomotion.

Findings

Video-based reward shaping doubles running speed in 12 hours of training

The approach outperforms the original human videos in running behavior

Combines techniques from top NIPS competition approaches

Abstract

Learning to produce efficient movement behaviour for humanoid robots from scratch is a hard problem, as has been illustrated by the "Learning to run" competition at NIPS 2017. The goal of this competition was to train a two-legged model of a humanoid body to run in a simulated race course with maximum speed. All submissions took a tabula rasa approach to reinforcement learning (RL) and were able to produce relatively fast, but not optimal running behaviour. In this paper, we demonstrate how data from videos of human running (e.g. taken from YouTube) can be used to shape the reward of the humanoid learning agent to speed up the learning and produce a better result. Specifically, we are using the positions of key body parts at regular time intervals to define a potential function for potential-based reward shaping (PBRS). Since PBRS does not change the optimal policy, this approach allows the RL agent to overcome sub-optimalities in the human movements that are shown in the videos. We present experiments in which we combine selected techniques from the top ten approaches from the NIPS competition with further optimizations to create an high-performing agent as a baseline. We then demonstrate how video-based reward shaping improves the performance further, resulting in an RL agent that runs twice as fast as the baseline in 12 hours of training. We furthermore show that our approach can overcome sub-optimal running behaviour in videos, with the learned policy significantly outperforming that of the running agent from the video.