Effective Exploration Based on the Structural Information Principles

TL;DR

This paper introduces SI2E, a novel exploration framework leveraging structural information principles to improve reinforcement learning by capturing hierarchical state-action structures and maximizing intrinsic rewards, leading to better exploration and efficiency.

Contribution

The paper proposes a new Structural Information principles-based framework, SI2E, which captures hierarchical state-action structures and enhances exploration in reinforcement learning.

Findings

SI2E outperforms state-of-the-art methods in benchmarks.

Maximum performance improvement of 37.63%.

Maximum sample efficiency gain of 60.25%.

Abstract

Traditional information theory provides a valuable foundation for Reinforcement Learning, particularly through representation learning and entropy maximization for agent exploration. However, existing methods primarily concentrate on modeling the uncertainty associated with RL's random variables, neglecting the inherent structure within the state and action spaces. In this paper, we propose a novel Structural Information principles-based Effective Exploration framework, namely SI2E. Structural mutual information between two variables is defined to address the single-variable limitation in structural information, and an innovative embedding principle is presented to capture dynamics-relevant state-action representations. The SI2E analyzes value differences in the agent's policy between state-action pairs and minimizes structural entropy to derive the hierarchical state-action structure, referred to as the encoding tree. Under this tree structure, value-conditional structural entropy is defined and maximized to design an intrinsic reward mechanism that avoids redundant transitions and promotes enhanced coverage in the state-action space. Theoretical connections are established between SI2E and classical information-theoretic methodologies, highlighting our framework's rationality and advantage. Comprehensive evaluations in the MiniGrid, MetaWorld, and DeepMind Control Suite benchmarks demonstrate that SI2E significantly outperforms state-of-the-art exploration baselines regarding final performance and sample efficiency, with maximum improvements of 37.63% and 60.25%, respectively.