Cautious Policy Programming: Exploiting KL Regularization in Monotonic Policy Improvement for Reinforcement Learning

TL;DR

This paper introduces Cautious Policy Programming (CPP), a reinforcement learning algorithm that ensures monotonic policy improvement by leveraging KL regularization and an entropy-aware lower bound, improving stability and scalability in complex tasks.

Contribution

The paper presents a novel entropy regularization-aware lower bound for policy improvement and an interpolation scheme that enhances CPP's scalability in high-dimensional control problems.

Findings

CPP guarantees monotonic policy improvement.

CPP balances performance and stability effectively.

CPP scales well to high-dimensional Atari games.

Abstract

In this paper, we propose cautious policy programming (CPP), a novel value-based reinforcement learning (RL) algorithm that can ensure monotonic policy improvement during learning. Based on the nature of entropy-regularized RL, we derive a new entropy regularization-aware lower bound of policy improvement that only requires estimating the expected policy advantage function. CPP leverages this lower bound as a criterion for adjusting the degree of a policy update for alleviating policy oscillation. Different from similar algorithms that are mostly theory-oriented, we also propose a novel interpolation scheme that makes CPP better scale in high dimensional control problems. We demonstrate that the proposed algorithm can trade o? performance and stability in both didactic classic control problems and challenging high-dimensional Atari games.