Quantile-Based Policy Optimization for Reinforcement Learning

TL;DR

This paper introduces Quantile-Based Policy Optimization (QPO) and QPPO, novel algorithms for reinforcement learning that optimize the quantile of cumulative rewards, providing a new approach beyond expected reward maximization.

Contribution

The paper proposes the first policy gradient algorithms specifically designed for quantile optimization in deep reinforcement learning, with convergence guarantees.

Findings

QPO and QPPO outperform baseline algorithms on quantile-based RL tasks.

The algorithms converge to the global optimal policy under certain conditions.

Numerical experiments validate the effectiveness of the proposed methods.

Abstract

Classical reinforcement learning (RL) aims to optimize the expected cumulative rewards. In this work, we consider the RL setting where the goal is to optimize the quantile of the cumulative rewards. We parameterize the policy controlling actions by neural networks and propose a novel policy gradient algorithm called Quantile-Based Policy Optimization (QPO) and its variant Quantile-Based Proximal Policy Optimization (QPPO) to solve deep RL problems with quantile objectives. QPO uses two coupled iterations running at different time scales for simultaneously estimating quantiles and policy parameters and is shown to converge to the global optimal policy under certain conditions. Our numerical results demonstrate that the proposed algorithms outperform the existing baseline algorithms under the quantile criterion.