Value Improved Actor Critic Algorithms

TL;DR

This paper introduces a novel approach to Actor Critic algorithms that separates the policy improvement step from the policy evaluation, allowing for greedier updates of the critic while maintaining stable policy learning, leading to improved performance.

Contribution

It proposes decoupling the actor and critic policies to enable greedier critic updates, enhancing stability and performance in Actor Critic algorithms.

Findings

Incorporating value-improvement improves TD3 and SAC performance.

The approach achieves better or comparable results across various environments.

Minimal additional computational cost is required.

Abstract

To learn approximately optimal acting policies for decision problems, modern Actor Critic algorithms rely on deep Neural Networks (DNNs) to parameterize the acting policy and greedification operators to iteratively improve it. The reliance on DNNs suggests an improvement that is gradient based, which is per step much less greedy than the improvement possible by greedier operators such as the greedy update used by Q-learning algorithms. On the other hand, slow changes to the policy can also be beneficial for the stability of the learning process, resulting in a tradeoff between greedification and stability. To better address this tradeoff, we propose to decouple the acting policy from the policy evaluated by the critic. This allows the agent to separately improve the critic's policy (e.g. value improvement) with greedier updates while maintaining the slow gradient-based improvement to the parameterized acting policy. We investigate the convergence of this approach using the popular analysis scheme of generalized Policy Iteration in the finite-horizon domain. Empirically, incorporating value-improvement into the popular off-policy actor-critic algorithms TD3 and SAC significantly improves or matches performance over their respective baselines, across different environments from the DeepMind continuous control domain, with negligible compute and implementation cost.