A unified strategy for implementing curiosity and empowerment driven reinforcement learning

TL;DR

This paper introduces a unified approach to combine curiosity and empowerment as intrinsic motivations in reinforcement learning, enhancing agent autonomy and efficiency through shared internal models and information flow control.

Contribution

It proposes a novel unified strategy integrating curiosity and empowerment, with a shared internal model, to improve autonomous learning in reinforcement agents.

Findings

Homeostatic drive from curiosity enhances information gain.

Shared internal model improves empowerment training efficiency.

Unified intrinsic motivation strategy increases agent autonomy.

Abstract

Although there are many approaches to implement intrinsically motivated artificial agents, the combined usage of multiple intrinsic drives remains still a relatively unexplored research area. Specifically, we hypothesize that a mechanism capable of quantifying and controlling the evolution of the information flow between the agent and the environment could be the fundamental component for implementing a higher degree of autonomy into artificial intelligent agents. This paper propose a unified strategy for implementing two semantically orthogonal intrinsic motivations: curiosity and empowerment. Curiosity reward informs the agent about the relevance of a recent agent action, whereas empowerment is implemented as the opposite information flow from the agent to the environment that quantifies the agent's potential of controlling its own future. We show that an additional homeostatic drive is derived from the curiosity reward, which generalizes and enhances the information gain of a classical curious/heterostatic reinforcement learning agent. We show how a shared internal model by curiosity and empowerment facilitates a more efficient training of the empowerment function. Finally, we discuss future directions for further leveraging the interplay between these two intrinsic rewards.