Learning Locomotion Skills Using DeepRL: Does the Choice of Action Space Matter?

TL;DR

This paper investigates how different action representations in deep reinforcement learning affect the efficiency and quality of learning locomotion skills, highlighting the importance of action space choice.

Contribution

It systematically compares four action parameterizations in deep RL for locomotion, revealing their impact on learning speed, robustness, and motion quality.

Findings

Higher-level action parameterizations improve learning robustness.

Choice of action space significantly affects policy quality.

Local feedback in action spaces influences learning efficiency.

Abstract

The use of deep reinforcement learning allows for high-dimensional state descriptors, but little is known about how the choice of action representation impacts the learning difficulty and the resulting performance. We compare the impact of four different action parameterizations (torques, muscle-activations, target joint angles, and target joint-angle velocities) in terms of learning time, policy robustness, motion quality, and policy query rates. Our results are evaluated on a gait-cycle imitation task for multiple planar articulated figures and multiple gaits. We demonstrate that the local feedback provided by higher-level action parameterizations can significantly impact the learning, robustness, and quality of the resulting policies.