How Does an Approximate Model Help in Reinforcement Learning?

TL;DR

This paper analyzes how an approximate model can significantly reduce the sample complexity in reinforcement learning by focusing on near-optimal actions, providing both algorithms and theoretical bounds.

Contribution

It introduces a new algorithm leveraging an approximate model to reduce sample complexity and establishes nearly-tight bounds showing its effectiveness.

Findings

Sample complexity is reduced to rac{N}{(1-\u03b3)^3 rac{1}{^2}) with an approximate model.

The upper bound on sample complexity is nearly tight under certain conditions.

The approach effectively eliminates sub-optimal actions, improving learning efficiency.

Abstract

One of the key approaches to save samples in reinforcement learning (RL) is to use knowledge from an approximate model such as its simulator. However, how much does an approximate model help to learn a near-optimal policy of the true unknown model? Despite numerous empirical studies of transfer reinforcement learning, an answer to this question is still elusive. In this paper, we study the sample complexity of RL while an approximate model of the environment is provided. For an unknown Markov decision process (MDP), we show that the approximate model can effectively reduce the complexity by eliminating sub-optimal actions from the policy searching space. In particular, we provide an algorithm that uses $\widetilde{O}(N/(1-\gamma)^3/\varepsilon^2)$ samples in a generative model to learn an $\varepsilon$-optimal policy, where $\gamma$ is the discount factor and $N$ is the number of near-optimal actions in the approximate model. This can be much smaller than the learning-from-scratch complexity $\widetilde{\Theta}(SA/(1-\gamma)^3/\varepsilon^2)$, where $S$ and $A$ are the sizes of state and action spaces respectively. We also provide a lower bound showing that the above upper bound is nearly-tight if the value gap between near-optimal actions and sub-optimal actions in the approximate model is sufficiently large. Our results provide a very precise characterization of how an approximate model helps reinforcement learning when no additional assumption on the model is posed.