Quantile-Based Deep Reinforcement Learning using Two-Timescale Policy Gradient Algorithms

TL;DR

This paper introduces novel deep reinforcement learning algorithms, QPO and QPPO, that optimize the quantile of cumulative rewards using two-timescale policy gradient methods, outperforming existing baselines.

Contribution

The paper proposes the first neural network-based algorithms for quantile optimization in deep RL, utilizing two-timescale updates for improved performance.

Findings

QPO and QPPO outperform baseline algorithms under the quantile criterion.

QPPO achieves higher efficiency with multiple updates per episode.

Algorithms effectively optimize the quantile of cumulative rewards.

Abstract

Classical reinforcement learning (RL) aims to optimize the expected cumulative reward. In this work, we consider the RL setting where the goal is to optimize the quantile of the cumulative reward. We parameterize the policy controlling actions by neural networks, and propose a novel policy gradient algorithm called Quantile-Based Policy Optimization (QPO) and its variant Quantile-Based Proximal Policy Optimization (QPPO) for solving deep RL problems with quantile objectives. QPO uses two coupled iterations running at different timescales for simultaneously updating quantiles and policy parameters, whereas QPPO is an off-policy version of QPO that allows multiple updates of parameters during one simulation episode, leading to improved algorithm efficiency. Our numerical results indicate that the proposed algorithms outperform the existing baseline algorithms under the quantile criterion.