Actor-Critic or Critic-Actor? A Tale of Two Time Scales

TL;DR

This paper explores the effects of reversing the time scales in actor-critic algorithms, demonstrating that a critic-actor approach can emulate value iteration, with proven convergence and comparable empirical performance.

Contribution

It introduces and analyzes a reversed time-scale actor-critic algorithm, showing it can emulate value iteration and performs similarly to traditional actor-critic methods.

Findings

Reversing time scales emulates value iteration.

The proposed critic-actor algorithm converges reliably.

Performance is comparable to standard actor-critic in accuracy and efficiency.

Abstract

We revisit the standard formulation of tabular actor-critic algorithm as a two time-scale stochastic approximation with value function computed on a faster time-scale and policy computed on a slower time-scale. This emulates policy iteration. We observe that reversal of the time scales will in fact emulate value iteration and is a legitimate algorithm. We provide a proof of convergence and compare the two empirically with and without function approximation (with both linear and nonlinear function approximators) and observe that our proposed critic-actor algorithm performs on par with actor-critic in terms of both accuracy and computational effort.