Mirror Descent Actor Critic via Bounded Advantage Learning

TL;DR

This paper introduces a new actor-critic reinforcement learning method called MDAC, which improves continuous action domain performance by bounding advantage estimates, combining theoretical insights with empirical validation.

Contribution

The paper proposes MDAC, an actor-critic algorithm that enhances regularized RL by bounding advantage estimates, and provides theoretical and empirical analysis of its benefits.

Findings

MDAC outperforms non-regularized and entropy-only methods with proper bounding.

Bounding advantage terms improves empirical performance in continuous domains.

Theoretical analysis supports the use of bounded advantage in regularized RL.

Abstract

Regularization is a core component of recent Reinforcement Learning (RL) algorithms. Mirror Descent Value Iteration (MDVI) uses both Kullback-Leibler divergence and entropy as regularizers in its value and policy updates. Despite its empirical success in discrete action domains and strong theoretical guarantees, the performance of KL-entropy-regularized methods does not surpass that of a strong entropy-only-regularized method in continuous action domains. In this study, we propose Mirror Descent Actor Critic (MDAC) as an actor-critic style instantiation of MDVI for continuous action domains, and show that its empirical performance is significantly boosted by bounding the actor's log-density terms in the critic's loss function, compared to a non-bounded naive instantiation. Further, we relate MDAC to Advantage Learning by recalling that the actor's log-probability is equal to the regularized advantage function in tabular cases, and theoretically discuss when and why bounding the advantage terms is validated and beneficial. We also empirically explore effective choices for the bounding functions, and show that MDAC performs better than strong non-regularized and entropy-only-regularized methods with an appropriate choice of the bounding functions.