Model-Based Action Exploration for Learning Dynamic Motion Skills

TL;DR

This paper introduces a model-based exploration method in reinforcement learning that uses a forward dynamics model to generate more effective exploratory actions, improving learning efficiency and solution quality in complex control tasks.

Contribution

It proposes a novel approach that leverages a learned forward dynamics model to generate better exploratory actions in continuous control tasks, addressing scalability and safety issues.

Findings

Enhanced learning speed in robotic tasks

Higher quality motion control solutions

Effective exploration in high-dimensional action spaces

Abstract

Deep reinforcement learning has achieved great strides in solving challenging motion control tasks. Recently, there has been significant work on methods for exploiting the data gathered during training, but there has been less work on how to best generate the data to learn from. For continuous action domains, the most common method for generating exploratory actions involves sampling from a Gaussian distribution centred around the mean action output by a policy. Although these methods can be quite capable, they do not scale well with the dimensionality of the action space, and can be dangerous to apply on hardware. We consider learning a forward dynamics model to predict the result, ($x_{t+1}$), of taking a particular action, ($u$), given a specific observation of the state, ($x_{t}$). With this model we perform internal look-ahead predictions of outcomes and seek actions we believe have a reasonable chance of success. This method alters the exploratory action space, thereby increasing learning speed and enables higher quality solutions to difficult problems, such as robotic locomotion and juggling.