Automatic Risk Adaptation in Distributional Reinforcement Learning

TL;DR

This paper introduces ARA, a method for dynamically adapting risk levels in distributional RL using RND error, leading to significantly fewer failures and better generalization in various environments.

Contribution

The paper proposes a novel automatic risk adaptation technique that dynamically adjusts risk levels in distributional RL based on environment feedback.

Findings

Reduced failure rates by up to 7 times

Improved generalization performance by up to 14%

Effective in both known and unknown environments

Abstract

The use of Reinforcement Learning (RL) agents in practical applications requires the consideration of suboptimal outcomes, depending on the familiarity of the agent with its environment. This is especially important in safety-critical environments, where errors can lead to high costs or damage. In distributional RL, the risk-sensitivity can be controlled via different distortion measures of the estimated return distribution. However, these distortion functions require an estimate of the risk level, which is difficult to obtain and depends on the current state. In this work, we demonstrate the suboptimality of a static risk level estimation and propose a method to dynamically select risk levels at each environment step. Our method ARA (Automatic Risk Adaptation) estimates the appropriate risk level in both known and unknown environments using a Random Network Distillation error. We show reduced failure rates by up to a factor of 7 and improved generalization performance by up to 14% compared to both risk-aware and risk-agnostic agents in several locomotion environments.