Overcoming the Curse of Dimensionality in Reinforcement Learning Through Approximate Factorization

TL;DR

This paper introduces a method to reduce the sample complexity in high-dimensional reinforcement learning by approximately factorizing MDPs, leveraging task-specific structures for more efficient algorithms with theoretical guarantees.

Contribution

It proposes a novel approximate factorization approach for MDPs that improves sample efficiency and provides theoretical guarantees in both model-based and model-free RL.

Findings

Sample complexity dependence on state-action space size is exponentially reduced.

Proposed algorithms outperform baseline methods in synthetic and real-world tasks.

Theoretical analysis confirms improved sample complexity bounds.

Abstract

Reinforcement Learning (RL) algorithms are known to suffer from the curse of dimensionality, which refers to the fact that large-scale problems often lead to exponentially high sample complexity. A common solution is to use deep neural networks for function approximation; however, such approaches typically lack theoretical guarantees. To provably address the curse of dimensionality, we observe that many real-world problems exhibit task-specific model structures that, when properly leveraged, can improve the sample efficiency of RL. Building on this insight, we propose overcoming the curse of dimensionality by approximately factorizing the original Markov decision processes (MDPs) into smaller, independently evolving MDPs. This factorization enables the development of sample-efficient RL algorithms in both model-based and model-free settings, with the latter involving a variant of variance-reduced Q-learning. We provide improved sample complexity guarantees for both proposed algorithms. Notably, by leveraging model structure through the approximate factorization of the MDP, the dependence of sample complexity on the size of the state-action space can be exponentially reduced. Numerically, we demonstrate the practicality of our proposed methods through experiments on both synthetic MDP tasks and a wind farm-equipped storage control problem.