Boredom-driven curious learning by Homeo-Heterostatic Value Gradients

TL;DR

This paper introduces the HHVG algorithm, which models the interplay of boredom and curiosity to enhance exploration and learning in agents, demonstrating superior performance in model-building tasks.

Contribution

The paper proposes the HHVG algorithm that formalizes how boredom and curiosity interact to improve exploration and model learning in autonomous agents.

Findings

Boredom-enabled agents outperform other variants in model-building benchmarks.

The HHVG algorithm effectively balances intrinsic motivations for better exploration.

Empirical results support the role of boredom in enhancing curiosity-driven learning.

Abstract

This paper presents the Homeo-Heterostatic Value Gradients (HHVG) algorithm as a formal account on the constructive interplay between boredom and curiosity which gives rise to effective exploration and superior forward model learning. We envisaged actions as instrumental in agent's own epistemic disclosure. This motivated two central algorithmic ingredients: devaluation and devaluation progress, both underpin agent's cognition concerning intrinsically generated rewards. The two serve as an instantiation of homeostatic and heterostatic intrinsic motivation. A key insight from our algorithm is that the two seemingly opposite motivations can be reconciled---without which exploration and information-gathering cannot be effectively carried out. We supported this claim with empirical evidence, showing that boredom-enabled agents consistently outperformed other curious or explorative agent variants in model building benchmarks based on self-assisted experience accumulation.