A survey on intrinsic motivation in reinforcement learning

TL;DR

This survey reviews the role of intrinsic motivation in deep reinforcement learning, categorizing motivations, analyzing their benefits and limitations, and exploring current research challenges and future developmental architectures.

Contribution

It provides a comprehensive categorization of intrinsic motivations in DRL and investigates unresolved research questions and potential developmental architectures.

Findings

Different types of intrinsic motivations have distinct advantages and limitations.

Current challenges include action abstraction and environment exploration.

Proposed developmental architectures combine RL algorithms with IM modules.

Abstract

The reinforcement learning (RL) research area is very active, with an important number of new contributions; especially considering the emergent field of deep RL (DRL). However a number of scientific and technical challenges still need to be addressed, amongst which we can mention the ability to abstract actions or the difficulty to explore the environment which can be addressed by intrinsic motivation (IM). In this article, we provide a survey on the role of intrinsic motivation in DRL. We categorize the different kinds of intrinsic motivations and detail for each category, its advantages and limitations with respect to the mentioned challenges. Additionnally, we conduct an in-depth investigation of substantial current research questions, that are currently under study or not addressed at all in the considered research area of DRL. We choose to survey these research works, from the perspective of learning how to achieve tasks. We suggest then, that solving current challenges could lead to a larger developmental architecture which may tackle most of the tasks. We describe this developmental architecture on the basis of several building blocks composed of a RL algorithm and an IM module compressing information.