Efficiency Separation between RL Methods: Model-Free, Model-Based and Goal-Conditioned

TL;DR

This paper establishes a fundamental exponential lower bound on the efficiency of many RL algorithms, including model-free and some model-based methods, while highlighting classes of algorithms that can overcome this limitation.

Contribution

It introduces a family of RL problems demonstrating the exponential lower bound and identifies algorithm classes that are not subject to this limitation.

Findings

Model-free and some model-based RL methods have exponential lower bounds in problem horizon.

Goal-conditioned and inverse dynamics methods are not affected by this limitation.

A specific algorithm can efficiently solve the challenging problem family.

Abstract

We prove a fundamental limitation on the efficiency of a wide class of Reinforcement Learning (RL) algorithms. This limitation applies to model-free RL methods as well as a broad range of model-based methods, such as planning with tree search. Under an abstract definition of this class, we provide a family of RL problems for which these methods suffer a lower bound exponential in the horizon for their interactions with the environment to find an optimal behavior. However, there exists a method, not tailored to this specific family of problems, which can efficiently solve the problems in the family. In contrast, our limitation does not apply to several types of methods proposed in the literature, for instance, goal-conditioned methods or other algorithms that construct an inverse dynamics model.