Information Content Exploration

TL;DR

This paper introduces an information-theoretic intrinsic reward for reinforcement learning that enhances exploration efficiency, especially in sparse reward environments, and demonstrates superior performance over existing methods in challenging games.

Contribution

The paper proposes a novel intrinsic reward based on information content that improves exploration and sample efficiency, with an extension to continuous state spaces.

Findings

Outperforms Curiosity Driven Learning and RND in various games

Enhances exploration efficiency in sparse reward environments

Extends to continuous state spaces with improved sample efficiency

Abstract

Sparse reward environments are known to be challenging for reinforcement learning agents. In such environments, efficient and scalable exploration is crucial. Exploration is a means by which an agent gains information about the environment. We expand on this topic and propose a new intrinsic reward that systemically quantifies exploratory behavior and promotes state coverage by maximizing the information content of a trajectory taken by an agent. We compare our method to alternative exploration based intrinsic reward techniques, namely Curiosity Driven Learning and Random Network Distillation. We show that our information theoretic reward induces efficient exploration and outperforms in various games, including Montezuma Revenge, a known difficult task for reinforcement learning. Finally, we propose an extension that maximizes information content in a discretely compressed latent space which boosts sample efficiency and generalizes to continuous state spaces.