Auto-exploration for online reinforcement learning

TL;DR

This paper introduces auto-exploration methods for online reinforcement learning that automatically explore state and action spaces without prior parameter knowledge, achieving optimal sample complexity with simple, parameter-free algorithms.

Contribution

The paper presents novel auto-exploration algorithms for RL that are parameter-free and achieve optimal sample complexity, applicable to both tabular and linear function approximation settings.

Findings

Achieve $O( ext{ extonehalf})$ sample complexity without prior parameters

Applicable to tabular and linear function approximation

Algorithms are simple and easy to implement

Abstract

The exploration-exploitation dilemma in reinforcement learning (RL) is a fundamental challenge to efficient RL algorithms. Existing algorithms for finite state and action discounted RL problems address this by assuming sufficient exploration over both state and action spaces. However, this yields non-implementable algorithms and sub-optimal performance. To resolve these limitations, we introduce a new class of methods with auto-exploration, or methods that automatically explore both state and action spaces in a parameter-free way, i.e.,~without a priori knowledge of problem-dependent parameters. We present two variants: one for the tabular setting and one for linear function approximation. Under algorithm-independent assumptions on the existence of an exploring optimal policy, both methods attain $O(\epsilon^{-2})$ sample complexity to solve to $\epsilon$ error. Crucially, these complexities are novel since they are void of algorithm-dependent parameters seen in prior works, which may be arbitrarily large. The methods are also simple to implement because they are parameter-free and do not directly estimate the unknown parameters. These feats are achieved by new algorithmic innovations for RL, including a dynamic mixing time, a discounted state distribution for sampling, a simple robust gradient estimator, and a recent advantage gap function to certify convergence.