Prototyping three key properties of specific curiosity in computational reinforcement learning

TL;DR

This paper introduces and prototypes three properties of specific curiosity—directedness, cessation when satisfied, and voluntary exposure—in a reinforcement learning agent, demonstrating their effects in a simple environment and highlighting their potential for complex goal-seeking behavior.

Contribution

It presents the first implementation of three properties of specific curiosity in a reinforcement learning agent, showing how they influence behavior in a simple environment.

Findings

Agent exhibits short-term directed curiosity behavior.

Agent adapts long-term preferences to seek curiosity-inducing situations.

Properties of curiosity can be integrated into goal-seeking agents.

Abstract

Curiosity for machine agents has been a focus of intense research. The study of human and animal curiosity, particularly specific curiosity, has unearthed several properties that would offer important benefits for machine learners, but that have not yet been well-explored in machine intelligence. In this work, we introduce three of the most immediate of these properties -- directedness, cessation when satisfied, and voluntary exposure -- and show how they may be implemented together in a proof-of-concept reinforcement learning agent; further, we demonstrate how the properties manifest in the behaviour of this agent in a simple non-episodic grid-world environment that includes curiosity-inducing locations and induced targets of curiosity. As we would hope, the agent exhibits short-term directed behaviour while updating long-term preferences to adaptively seek out curiosity-inducing situations. This work therefore presents a novel view into how specific curiosity operates and in the future might be integrated into the behaviour of goal-seeking, decision-making agents in complex environments.