Risk Sensitive Model-Based Reinforcement Learning using Uncertainty Guided Planning

TL;DR

This paper introduces a risk-sensitive model-based reinforcement learning method that uses uncertainty-guided planning with bootstrap ensembles to promote safe actions in high-risk environments, balancing risk and reward.

Contribution

It presents a novel uncertainty-guided cross-entropy planning approach that penalizes high-variance predictions, enhancing safety without explicit constraints.

Findings

Agent identifies uncertain regions during planning

Actions are guided to low-uncertainty areas

Trade-off observed between risk and reward

Abstract

Identifying uncertainty and taking mitigating actions is crucial for safe and trustworthy reinforcement learning agents, especially when deployed in high-risk environments. In this paper, risk sensitivity is promoted in a model-based reinforcement learning algorithm by exploiting the ability of a bootstrap ensemble of dynamics models to estimate environment epistemic uncertainty. We propose uncertainty guided cross-entropy method planning, which penalises action sequences that result in high variance state predictions during model rollouts, guiding the agent to known areas of the state space with low uncertainty. Experiments display the ability for the agent to identify uncertain regions of the state space during planning and to take actions that maintain the agent within high confidence areas, without the requirement of explicit constraints. The result is a reduction in the performance in terms of attaining reward, displaying a trade-off between risk and return.