On Reward-Free Reinforcement Learning with Linear Function Approximation

TL;DR

This paper investigates reward-free reinforcement learning with linear function approximation, providing algorithms with polynomial sample complexity under certain conditions and establishing exponential lower bounds in other scenarios.

Contribution

The paper introduces a reward-free RL algorithm with polynomial sample complexity for linear MDPs and proves exponential lower bounds when only the optimal Q-function is linear.

Findings

Polynomial sample complexity for linear MDPs with transition and reward linearity.

Exponential lower bound for reward-free RL when only the optimal Q-function is linear.

Significant separations in sample complexity based on linearity assumptions.

Abstract

Reward-free reinforcement learning (RL) is a framework which is suitable for both the batch RL setting and the setting where there are many reward functions of interest. During the exploration phase, an agent collects samples without using a pre-specified reward function. After the exploration phase, a reward function is given, and the agent uses samples collected during the exploration phase to compute a near-optimal policy. Jin et al. [2020] showed that in the tabular setting, the agent only needs to collect polynomial number of samples (in terms of the number states, the number of actions, and the planning horizon) for reward-free RL. However, in practice, the number of states and actions can be large, and thus function approximation schemes are required for generalization. In this work, we give both positive and negative results for reward-free RL with linear function approximation. We give an algorithm for reward-free RL in the linear Markov decision process setting where both the transition and the reward admit linear representations. The sample complexity of our algorithm is polynomial in the feature dimension and the planning horizon, and is completely independent of the number of states and actions. We further give an exponential lower bound for reward-free RL in the setting where only the optimal $Q$-function admits a linear representation. Our results imply several interesting exponential separations on the sample complexity of reward-free RL.