Diversity-Enriched Option-Critic

TL;DR

This paper introduces a method to enhance the option-critic framework in reinforcement learning by promoting behavioral diversity among options, leading to more robust, reusable, and interpretable options that improve performance across various tasks.

Contribution

It proposes an information-theoretic intrinsic reward and a novel termination objective to encourage diversity in options, addressing key limitations of the original option-critic approach.

Findings

Outperforms option-critic significantly on multiple tasks

Generates robust and interpretable options

Enhances diversity and reusability of learned options

Abstract

Temporal abstraction allows reinforcement learning agents to represent knowledge and develop strategies over different temporal scales. The option-critic framework has been demonstrated to learn temporally extended actions, represented as options, end-to-end in a model-free setting. However, feasibility of option-critic remains limited due to two major challenges, multiple options adopting very similar behavior, or a shrinking set of task relevant options. These occurrences not only void the need for temporal abstraction, they also affect performance. In this paper, we tackle these problems by learning a diverse set of options. We introduce an information-theoretic intrinsic reward, which augments the task reward, as well as a novel termination objective, in order to encourage behavioral diversity in the option set. We show empirically that our proposed method is capable of learning options end-to-end on several discrete and continuous control tasks, outperforms option-critic by a wide margin. Furthermore, we show that our approach sustainably generates robust, reusable, reliable and interpretable options, in contrast to option-critic.