Risk-Aware Reinforcement Learning for Mobile Manipulation

TL;DR

This paper introduces a novel approach for mobile manipulation robots to learn risk-aware visuomotor policies using distributional reinforcement learning and imitation learning, enabling safer and more reliable operation in unstructured environments.

Contribution

It is the first to learn risk-aware visuomotor policies conditioned on egocentric depth with adjustable risk sensitivity and transfer these behaviors via imitation learning.

Findings

Policies exhibit risk-aware behaviors with improved worst-case performance.

The method enables reactive whole-body motions in unmapped environments.

Risk-sensitive policies outperform risk-neutral counterparts in safety metrics.

Abstract

For robots to successfully transition from lab settings to everyday environments, they must begin to reason about the risks associated with their actions and make informed, risk-aware decisions. This is particularly true for robots performing mobile manipulation tasks, which involve both interacting with and navigating within dynamic, unstructured spaces. However, existing whole-body controllers for mobile manipulators typically lack explicit mechanisms for risk-sensitive decision-making under uncertainty. To our knowledge, we are the first to (i) learn risk-aware visuomotor policies for mobile manipulation conditioned on egocentric depth observations with runtime-adjustable risk sensitivity, and (ii) show risk-aware behaviours can be transferred through Imitation Learning (IL) to a visuomotor policy conditioned on egocentric depth observations. Our method achieves this by first training a privileged teacher policy using Distributional Reinforcement Learning (DRL), with a risk-neutral distributional critic. Distortion risk-metrics are then applied to the critic's predicted return distribution to calculate risk-adjusted advantage estimates used in policy updates to achieve a range of risk-aware behaviours. We then distil teacher policies with IL to obtain risk-aware student policies conditioned on egocentric depth observations. We perform extensive evaluations demonstrating that our trained visuomotor policies exhibit risk-aware behaviour (specifically achieving better worst-case performance) while performing reactive whole-body motions in unmapped environments, leveraging live depth observations for perception.