Diverse Policies Converge in Reward-free Markov Decision Processe

TL;DR

This paper introduces a unified framework for diversity reinforcement learning, providing theoretical convergence analysis and an efficient algorithm, validated through numerical experiments, to promote diverse policies in Markov Decision Processes.

Contribution

It offers the first theoretical analysis of convergence in diversity reinforcement learning and proposes a provably efficient algorithm within a unified framework.

Findings

The proposed algorithm converges efficiently under the new framework.

Numerical experiments demonstrate the effectiveness of the method.

The framework unifies various approaches to diversity reinforcement learning.

Abstract

Reinforcement learning has achieved great success in many decision-making tasks, and traditional reinforcement learning algorithms are mainly designed for obtaining a single optimal solution. However, recent works show the importance of developing diverse policies, which makes it an emerging research topic. Despite the variety of diversity reinforcement learning algorithms that have emerged, none of them theoretically answer the question of how the algorithm converges and how efficient the algorithm is. In this paper, we provide a unified diversity reinforcement learning framework and investigate the convergence of training diverse policies. Under such a framework, we also propose a provably efficient diversity reinforcement learning algorithm. Finally, we verify the effectiveness of our method through numerical experiments.