Risk-Averse Bayes-Adaptive Reinforcement Learning

TL;DR

This paper introduces a risk-averse reinforcement learning framework that optimizes the CVaR in Bayes-adaptive MDPs, effectively managing uncertainty and outperforming existing methods.

Contribution

It formulates CVaR optimization in Bayes-adaptive MDPs as a stochastic game and proposes an approximate solution using Monte Carlo tree search and Bayesian optimization.

Findings

Our approach outperforms baseline methods in experiments.

The method effectively manages both parametric and stochastic uncertainties.

Risk-averse policies improve decision-making under uncertainty.

Abstract

In this work, we address risk-averse Bayes-adaptive reinforcement learning. We pose the problem of optimising the conditional value at risk (CVaR) of the total return in Bayes-adaptive Markov decision processes (MDPs). We show that a policy optimising CVaR in this setting is risk-averse to both the parametric uncertainty due to the prior distribution over MDPs, and the internal uncertainty due to the inherent stochasticity of MDPs. We reformulate the problem as a two-player stochastic game and propose an approximate algorithm based on Monte Carlo tree search and Bayesian optimisation. Our experiments demonstrate that our approach significantly outperforms baseline approaches for this problem.