Intrinsically motivated option learning: a comparative study of recent methods

TL;DR

This paper compares recent methods of option learning in reinforcement learning, focusing on how they incorporate the empowerment principle to enable intrinsic motivation without external rewards.

Contribution

It provides a systematic comparison of recent empowerment-based option learning methods, highlighting how they align with the original empowerment concept.

Findings

Most methods successfully incorporate intrinsic motivation.

Different approaches vary in how they preserve the empowerment principle.

The study clarifies the relationship between various modifications and the original framework.

Abstract

Options represent a framework for reasoning across multiple time scales in reinforcement learning (RL). With the recent active interest in the unsupervised learning paradigm in the RL research community, the option framework was adapted to utilize the concept of empowerment, which corresponds to the amount of influence the agent has on the environment and its ability to perceive this influence, and which can be optimized without any supervision provided by the environment's reward structure. Many recent papers modify this concept in various ways achieving commendable results. Through these various modifications, however, the initial context of empowerment is often lost. In this work we offer a comparative study of such papers through the lens of the original empowerment principle.