CASA: Bridging the Gap between Policy Improvement and Policy Evaluation with Conflict Averse Policy Iteration

TL;DR

This paper introduces a conflict-averse policy iteration method that regularizes the inconsistency between policy evaluation and improvement in model-free reinforcement learning, leading to improved performance and reduced approximation errors.

Contribution

It proposes a novel regularization approach that aligns policy evaluation with policy improvement, bridging the gap in generalized policy iteration and enhancing learning stability.

Findings

Outperforms strong baselines on Arcade Learning Environment

Reduces functional approximation error in policy iteration

Prevents policies from being trapped in suboptimal solutions

Abstract

We study the problem of model-free reinforcement learning, which is often solved following the principle of Generalized Policy Iteration (GPI). While GPI is typically an interplay between policy evaluation and policy improvement, most conventional model-free methods assume the independence of the granularity and other details of the GPI steps, despite of the inherent connections between them. In this paper, we present a method that regularizes the inconsistency between policy evaluation and policy improvement, leading to a conflict averse GPI solution with reduced functional approximation error. To this end, we formulate a novel learning paradigm where taking the policy evaluation step is equivalent to some compensation of performing policy improvement, and thus effectively alleviates the gradient conflict between the two GPI steps. We also show that the form of our proposed solution is equivalent to performing entropy-regularized policy improvement and therefore prevents the policy from being trapped into suboptimal solutions. We conduct extensive experiments to evaluate our method on the Arcade Learning Environment (ALE). Empirical results show that our method outperforms several strong baselines in major evaluation domains.