DeepQ Stepper: A framework for reactive dynamic walking on uneven terrain

TL;DR

This paper introduces DeepQ Stepper, a reinforcement learning-based framework enabling reactive, dynamic walking on uneven terrain by approximating 3D capture regions for biped robots, improving robustness and versatility.

Contribution

The paper presents a novel 3D reactive stepper that learns approximate capture regions using reinforcement learning, allowing for effective walking on complex terrains with full robot dynamics.

Findings

Successfully learned stepping strategies on simplified and full robot models.

Achieved higher performance by considering complete robot dynamics.

Handled non-convex terrains and obstacles with constant computational cost.

Abstract

Reactive stepping and push recovery for biped robots is often restricted to flat terrains because of the difficulty in computing capture regions for nonlinear dynamic models. In this paper, we address this limitation by using reinforcement learning to approximately learn the 3D capture region for such systems. We propose a novel 3D reactive stepper, The DeepQ stepper, that computes optimal step locations for walking at different velocities using the 3D capture regions approximated by the action-value function. We demonstrate the ability of the approach to learn stepping with a simplified 3D pendulum model and a full robot dynamics. Further, the stepper achieves a higher performance when it learns approximate capture regions while taking into account the entire dynamics of the robot that are often ignored in existing reactive steppers based on simplified models. The DeepQ stepper can handle non convex terrain with obstacles, walk on restricted surfaces like stepping stones and recover from external disturbances for a constant computational cost.