Behavioral Entropy-Guided Dataset Generation for Offline Reinforcement Learning

TL;DR

This paper introduces behavioral entropy as a new exploration metric for dataset generation in offline reinforcement learning, demonstrating its effectiveness in producing diverse datasets that improve downstream task performance.

Contribution

It extends behavioral entropy to continuous domains, develops estimators with theoretical guarantees, and shows its superiority over existing entropy-based methods in MuJoCo environments.

Findings

BE-based datasets outperform Shannon entropy datasets in offline RL tasks.

BE-guided exploration leads to more diverse state coverage.

BE outperforms Rényi, SMM, and RND in most evaluated tasks.

Abstract

Entropy-based objectives are widely used to perform state space exploration in reinforcement learning (RL) and dataset generation for offline RL. Behavioral entropy (BE), a rigorous generalization of classical entropies that incorporates cognitive and perceptual biases of agents, was recently proposed for discrete settings and shown to be a promising metric for robotic exploration problems. In this work, we propose using BE as a principled exploration objective for systematically generating datasets that provide diverse state space coverage in complex, continuous, potentially high-dimensional domains. To achieve this, we extend the notion of BE to continuous settings, derive tractable $k$-nearest neighbor estimators, provide theoretical guarantees for these estimators, and develop practical reward functions that can be used with standard RL methods to learn BE-maximizing policies. Using standard MuJoCo environments, we experimentally compare the performance of offline RL algorithms for a variety of downstream tasks on datasets generated using BE, R\'{e}nyi, and Shannon entropy-maximizing policies, as well as the SMM and RND algorithms. We find that offline RL algorithms trained on datasets collected using BE outperform those trained on datasets collected using Shannon entropy, SMM, and RND on all tasks considered, and on 80% of the tasks compared to datasets collected using R\'{e}nyi entropy.