Teaching a Robot to Walk Using Reinforcement Learning

TL;DR

This paper explores reinforcement learning techniques, specifically deep Q-learning and augmented random search, to teach a simulated bipedal robot to walk, demonstrating ARS's superior performance in solving complex locomotion tasks.

Contribution

The study compares deep Q-learning and ARS for robotic walking, showing ARS's effectiveness in achieving optimal policies in a complex simulation environment.

Findings

ARS successfully solves the BipedalWalker-v3 problem.

Deep Q-learning often converges prematurely to suboptimal policies.

Naive policies serve as benchmarks for evaluating learning algorithms.

Abstract

Classical control techniques such as PID and LQR have been used effectively in maintaining a system state, but these techniques become more difficult to implement when the model dynamics increase in complexity and sensitivity. For adaptive robotic locomotion tasks with several degrees of freedom, this task becomes infeasible with classical control techniques. Instead, reinforcement learning can train optimal walking policies with ease. We apply deep Q-learning and augmented random search (ARS) to teach a simulated two-dimensional bipedal robot how to walk using the OpenAI Gym BipedalWalker-v3 environment. Deep Q-learning did not yield a high reward policy, often prematurely converging to suboptimal local maxima likely due to the coarsely discretized action space. ARS, however, resulted in a better trained robot, and produced an optimal policy which officially "solves" the BipedalWalker-v3 problem. Various naive policies, including a random policy, a manually encoded inch forward policy, and a stay still policy, were used as benchmarks to evaluate the proficiency of the learning algorithm results.