Learning Risk-Aware Quadrupedal Locomotion using Distributional Reinforcement Learning

TL;DR

This paper introduces a novel risk-aware reinforcement learning method for quadrupedal robots that explicitly models and adjusts risk sensitivity in locomotion, enhancing safety in hazardous environments.

Contribution

It proposes Distributional Proximal Policy Optimization (DPPO), a new reinforcement learning approach that incorporates risk metrics for safer robot locomotion without extra reward tuning.

Findings

Emergent risk-sensitive behaviors in simulation

Successful real-world implementation on ANYmal robot

Adjustable risk preference parameter for behavior control

Abstract

Deployment in hazardous environments requires robots to understand the risks associated with their actions and movements to prevent accidents. Despite its importance, these risks are not explicitly modeled by currently deployed locomotion controllers for legged robots. In this work, we propose a risk sensitive locomotion training method employing distributional reinforcement learning to consider safety explicitly. Instead of relying on a value expectation, we estimate the complete value distribution to account for uncertainty in the robot's interaction with the environment. The value distribution is consumed by a risk metric to extract risk sensitive value estimates. These are integrated into Proximal Policy Optimization (PPO) to derive our method, Distributional Proximal Policy Optimization (DPPO). The risk preference, ranging from risk-averse to risk-seeking, can be controlled by a single parameter, which enables to adjust the robot's behavior dynamically. Importantly, our approach removes the need for additional reward function tuning to achieve risk sensitivity. We show emergent risk sensitive locomotion behavior in simulation and on the quadrupedal robot ANYmal. Videos of the experiments and code are available at https://sites.google.com/leggedrobotics.com/risk-aware-locomotion.