Robust Quadrupedal Locomotion via Risk-Averse Policy Learning

TL;DR

This paper introduces a risk-averse reinforcement learning approach for quadrupedal robots that improves robustness against terrain uncertainties and external disturbances by modeling environmental uncertainty with distributional value functions.

Contribution

It presents a novel risk-sensitive RL framework using quantile regression to enhance legged robot robustness in uncertain environments.

Findings

Improved robustness against external disturbances.

Effective handling of abrupt terrain changes.

Validated on both simulation and real robot.

Abstract

The robustness of legged locomotion is crucial for quadrupedal robots in challenging terrains. Recently, Reinforcement Learning (RL) has shown promising results in legged locomotion and various methods try to integrate privileged distillation, scene modeling, and external sensors to improve the generalization and robustness of locomotion policies. However, these methods are hard to handle uncertain scenarios such as abrupt terrain changes or unexpected external forces. In this paper, we consider a novel risk-sensitive perspective to enhance the robustness of legged locomotion. Specifically, we employ a distributional value function learned by quantile regression to model the aleatoric uncertainty of environments, and perform risk-averse policy learning by optimizing the worst-case scenarios via a risk distortion measure. Extensive experiments in both simulation environments and a real Aliengo robot demonstrate that our method is efficient in handling various external disturbances, and the resulting policy exhibits improved robustness in harsh and uncertain situations in legged locomotion. Videos are available at https://risk-averse-locomotion.github.io/.