DAC: The Double Actor-Critic Architecture for Learning Options

TL;DR

This paper introduces DAC, a novel architecture reformulating the option framework as two parallel augmented MDPs, enabling off-the-shelf policy optimization and improved transfer learning in robot simulation tasks.

Contribution

The paper proposes the Double Actor-Critic (DAC) architecture, reformulating options as parallel MDPs and demonstrating its effectiveness in transfer learning scenarios.

Findings

DAC outperforms hierarchy-free and previous option learning algorithms in robot simulations.

Only one critic is needed when using state-value functions as critics.

The reformulation allows all policy optimization algorithms to be applied off-the-shelf.

Abstract

We reformulate the option framework as two parallel augmented MDPs. Under this novel formulation, all policy optimization algorithms can be used off the shelf to learn intra-option policies, option termination conditions, and a master policy over options. We apply an actor-critic algorithm on each augmented MDP, yielding the Double Actor-Critic (DAC) architecture. Furthermore, we show that, when state-value functions are used as critics, one critic can be expressed in terms of the other, and hence only one critic is necessary. We conduct an empirical study on challenging robot simulation tasks. In a transfer learning setting, DAC outperforms both its hierarchy-free counterpart and previous gradient-based option learning algorithms.