Maximum Entropy Behavior Exploration for Sim2Real Zero-Shot Reinforcement Learning

TL;DR

This paper introduces FB-MEBE, an online zero-shot RL algorithm that maximizes behavior entropy and uses regularization to produce natural, deployable policies for quadrupedal robots, improving exploration and performance.

Contribution

The paper proposes FB-MEBE, a novel online zero-shot RL method combining entropy-based exploration with behavior regularization for better real-world robot policies.

Findings

FB-MEBE outperforms other exploration strategies in simulated downstream tasks.

FB-MEBE produces natural behaviors suitable for direct hardware deployment.

The approach enhances exploration diversity and policy plausibility.

Abstract

Zero-shot reinforcement learning (RL) algorithms aim to learn a family of policies from a reward-free dataset, and recover optimal policies for any reward function directly at test time. Naturally, the quality of the pretraining dataset determines the performance of the recovered policies across tasks. However, pre-collecting a relevant, diverse dataset without prior knowledge of the downstream tasks of interest remains a challenge. In this work, we study $\textit{online}$ zero-shot RL for quadrupedal control on real robotic systems, building upon the Forward-Backward (FB) algorithm. We observe that undirected exploration yields low-diversity data, leading to poor downstream performance and rendering policies impractical for direct hardware deployment. Therefore, we introduce FB-MEBE, an online zero-shot RL algorithm that combines an unsupervised behavior exploration strategy with a regularization critic. FB-MEBE promotes exploration by maximizing the entropy of the achieved behavior distribution. Additionally, a regularization critic shapes the recovered policies toward more natural and physically plausible behaviors. We empirically demonstrate that FB-MEBE achieves and improved performance compared to other exploration strategies in a range of simulated downstream tasks, and that it renders natural policies that can be seamlessly deployed to hardware without further finetuning. Videos and code available on our website.