Modified Actor-Critics

TL;DR

This paper introduces a flexible framework combining off-policy evaluation with softened policy improvement, enhancing sample efficiency and competitiveness in deep reinforcement learning algorithms.

Contribution

It presents a new generic framework for deep reinforcement learning that integrates soft greediness with Modified Policy Iteration, applicable to various algorithms.

Findings

More sample efficient than original PPO

Competitive with Soft Actor Critic (SAC)

Applicable to off-policy learning

Abstract

Recent successful deep reinforcement learning algorithms, such as Trust Region Policy Optimization (TRPO) or Proximal Policy Optimization (PPO), are fundamentally variations of conservative policy iteration (CPI). These algorithms iterate policy evaluation followed by a softened policy improvement step. As so, they are naturally on-policy. In this paper, we propose to combine (any kind of) soft greediness with Modified Policy Iteration (MPI). The proposed abstract framework applies repeatedly: (i) a partial policy evaluation step that allows off-policy learning and (ii) any softened greedy step. Our contribution can be seen as a new generic tool for the deep reinforcement learning toolbox. As a proof of concept, we instantiate this framework with the PPO greediness. Comparison to the original PPO shows that our algorithm is much more sample efficient. We also show that it is competitive with the state-of-art off-policy algorithm Soft Actor Critic (SAC).