Wasserstein Diversity-Enriched Regularizer for Hierarchical Reinforcement Learning

TL;DR

This paper introduces WDER, a Wasserstein distance-based regularizer that enhances diversity among subpolicies in hierarchical reinforcement learning, leading to improved performance and sample efficiency without additional hyperparameter tuning.

Contribution

The paper proposes a novel Wasserstein Diversity-Enriched Regularizer (WDER) that can be integrated into existing hierarchical reinforcement learning methods to promote diversity and improve results.

Findings

WDER increases subpolicy diversity effectively.

WDER improves performance and sample efficiency.

WDER is robust and easy to incorporate.

Abstract

Hierarchical reinforcement learning composites subpolicies in different hierarchies to accomplish complex tasks.Automated subpolicies discovery, which does not depend on domain knowledge, is a promising approach to generating subpolicies.However, the degradation problem is a challenge that existing methods can hardly deal with due to the lack of consideration of diversity or the employment of weak regularizers. In this paper, we propose a novel task-agnostic regularizer called the Wasserstein Diversity-Enriched Regularizer (WDER), which enlarges the diversity of subpolicies by maximizing the Wasserstein distances among action distributions. The proposed WDER can be easily incorporated into the loss function of existing methods to boost their performance further.Experimental results demonstrate that our WDER improves performance and sample efficiency in comparison with prior work without modifying hyperparameters, which indicates the applicability and robustness of the WDER.