Adaptive Risk-Tendency: Nano Drone Navigation in Cluttered Environments with Distributional Reinforcement Learning

TL;DR

This paper introduces a distributional reinforcement learning approach enabling nano drones to adaptively navigate cluttered environments by estimating uncertainty and adjusting risk-tendency, improving safety and performance.

Contribution

It proposes a novel framework using tail conditional variance and EWAF to adapt risk-tendency in reinforcement learning for drone navigation.

Findings

Adaptive risk-tendency varies across states

Achieves better performance than risk-neutral or risk-averse policies

Effective in both simulation and real-world scenarios

Abstract

Enabling the capability of assessing risk and making risk-aware decisions is essential to applying reinforcement learning to safety-critical robots like drones. In this paper, we investigate a specific case where a nano quadcopter robot learns to navigate an apriori-unknown cluttered environment under partial observability. We present a distributional reinforcement learning framework to generate adaptive risk-tendency policies. Specifically, we propose to use lower tail conditional variance of the learnt return distribution as intrinsic uncertainty estimation, and use exponentially weighted average forecasting (EWAF) to adapt the risk-tendency in accordance with the estimated uncertainty. In simulation and real-world empirical results, we show that (1) the most effective risk-tendency vary across states, (2) the agent with adaptive risk-tendency achieves superior performance compared to risk-neutral policy or risk-averse policy baselines.