KL-Entropy-Regularized RL with a Generative Model is Minimax Optimal

TL;DR

This paper proves that a simple, model-free reinforcement learning algorithm using KL-entropy regularization is nearly minimax-optimal in sample complexity, providing theoretical guarantees for its efficiency in finding near-optimal policies.

Contribution

It offers the first theoretical proof that a straightforward model-free RL method with entropy regularization achieves nearly minimax-optimal sample complexity.

Findings

Mirror descent value iteration is nearly minimax-optimal for small ε.

The analysis applies to algorithms without variance reduction.

First theoretical guarantee for simple model-free methods with entropy regularization.

Abstract

In this work, we consider and analyze the sample complexity of model-free reinforcement learning with a generative model. Particularly, we analyze mirror descent value iteration (MDVI) by Geist et al. (2019) and Vieillard et al. (2020a), which uses the Kullback-Leibler divergence and entropy regularization in its value and policy updates. Our analysis shows that it is nearly minimax-optimal for finding an $\varepsilon$-optimal policy when $\varepsilon$ is sufficiently small. This is the first theoretical result that demonstrates that a simple model-free algorithm without variance-reduction can be nearly minimax-optimal under the considered setting.