Parametric Return Density Estimation for Reinforcement Learning

TL;DR

This paper introduces a parametric approach to estimate the density of returns in reinforcement learning, enabling risk-sensitive decision-making by extending Bellman equations and TD-learning to handle various return distributions.

Contribution

It extends the Bellman equation and TD-learning to estimate return densities, providing a unified framework for risk-sensitive RL using parametric distributions.

Findings

Algorithms for Gaussian, Laplace, and skewed Laplace distributions are effective.

The method enables risk-sensitive and robust RL paradigms.

Numerical experiments validate the approach.

Abstract

Most conventional Reinforcement Learning (RL) algorithms aim to optimize decision-making rules in terms of the expected returns. However, especially for risk management purposes, other risk-sensitive criteria such as the value-at-risk or the expected shortfall are sometimes preferred in real applications. Here, we describe a parametric method for estimating density of the returns, which allows us to handle various criteria in a unified manner. We first extend the Bellman equation for the conditional expected return to cover a conditional probability density of the returns. Then we derive an extension of the TD-learning algorithm for estimating the return densities in an unknown environment. As test instances, several parametric density estimation algorithms are presented for the Gaussian, Laplace, and skewed Laplace distributions. We show that these algorithms lead to risk-sensitive as well as robust RL paradigms through numerical experiments.