Linear Reinforcement Learning with Ball Structure Action Space

TL;DR

This paper introduces a new 'ball structure' assumption on action spaces in linear RL, enabling efficient exploration and learning with significantly improved sample complexity over previous worst-case bounds.

Contribution

The paper proposes the BallRL algorithm that leverages the ball structure assumption to achieve sample-efficient reinforcement learning in linear function approximation settings.

Findings

BallRL learns an $psilon$-optimal policy with $ ilde{O}(H^5 d^3 / psilon^3)$ trajectories.

The ball structure assumption ensures sufficient exploration of the feature space.

Efficient RL is possible without additional assumptions on the MDP or value functions.

Abstract

We study the problem of Reinforcement Learning (RL) with linear function approximation, i.e. assuming the optimal action-value function is linear in a known $d$-dimensional feature mapping. Unfortunately, however, based on only this assumption, the worst case sample complexity has been shown to be exponential, even under a generative model. Instead of making further assumptions on the MDP or value functions, we assume that our action space is such that there always exist playable actions to explore any direction of the feature space. We formalize this assumption as a ``ball structure'' action space, and show that being able to freely explore the feature space allows for efficient RL. In particular, we propose a sample-efficient RL algorithm (BallRL) that learns an $\epsilon$-optimal policy using only $\tilde{O}\left(\frac{H^5d^3}{\epsilon^3}\right)$ number of trajectories.