Tilted Quantile Gradient Updates for Quantile-Constrained Reinforcement Learning

TL;DR

This paper introduces a novel quantile-constrained reinforcement learning method that directly estimates quantile gradients and employs a tilted update strategy, achieving higher safety and return performance.

Contribution

It proposes a new quantile-constrained RL framework with direct gradient estimation and a tilted update strategy, improving safety guarantees and return performance.

Findings

Successfully meets safety quantile constraints in experiments.

Outperforms state-of-the-art benchmarks in return performance.

Provides theoretical convergence proofs for the proposed method.

Abstract

Safe reinforcement learning (RL) is a popular and versatile paradigm to learn reward-maximizing policies with safety guarantees. Previous works tend to express the safety constraints in an expectation form due to the ease of implementation, but this turns out to be ineffective in maintaining safety constraints with high probability. To this end, we move to the quantile-constrained RL that enables a higher level of safety without any expectation-form approximations. We directly estimate the quantile gradients through sampling and provide the theoretical proofs of convergence. Then a tilted update strategy for quantile gradients is implemented to compensate the asymmetric distributional density, with a direct benefit of return performance. Experiments demonstrate that the proposed model fully meets safety requirements (quantile constraints) while outperforming the state-of-the-art benchmarks with higher return.