Learning Diverse Options via InfoMax Termination Critic

TL;DR

This paper introduces IMTC, a method for learning diverse, reusable options in reinforcement learning by maximizing mutual information between options and state transitions, leading to improved transfer and adaptation.

Contribution

The paper proposes a scalable MI-based approach for learning termination conditions of options, enhancing diversity and reusability in reinforcement learning.

Findings

IMTC increases diversity of learned options

IMTC improves transferability across tasks

IMTC accelerates adaptation in complex domains

Abstract

We consider the problem of autonomously learning reusable temporally extended actions, or options, in reinforcement learning. While options can speed up transfer learning by serving as reusable building blocks, learning reusable options for unknown task distribution remains challenging. Motivated by the recent success of mutual information (MI) based skill learning, we hypothesize that more diverse options are more reusable. To this end, we propose a method for learning termination conditions of options by maximizing MI between options and corresponding state transitions. We derive a scalable approximation of this MI maximization via gradient ascent, yielding the InfoMax Termination Critic (IMTC) algorithm. Our experiments demonstrate that IMTC significantly improves the diversity of learned options without extrinsic rewards combined with an intrinsic option learning method. Moreover, we test the reusability of learned options by transferring options into various tasks, confirming that IMTC helps quick adaptation, especially in complex domains where an agent needs to manipulate objects.