Demystifying Linear MDPs and Novel Dynamics Aggregation Framework

TL;DR

This paper introduces a new hierarchical reinforcement learning framework with dynamics aggregation for linear MDPs, achieving improved regret bounds and providing the first provably guaranteed HRL algorithm with linear function approximation.

Contribution

The paper proves a lower bound on feature dimension in linear MDPs and proposes a novel dynamics aggregation framework with a provably efficient hierarchical RL algorithm.

Findings

Regret bound of $ ilde{O} ( d_{C6} ^{3/2} H^{3/2}\u221A{N T} )$ for the proposed algorithm.

Condition $d_{C6}^3 N \u226A d^3$ holds in many real-world hierarchical environments.

First HRL algorithm with linear function approximation that has provable guarantees.

Abstract

In this work, we prove that, in linear MDPs, the feature dimension $d$ is lower bounded by $S/U$ in order to aptly represent transition probabilities, where $S$ is the size of the state space and $U$ is the maximum size of directly reachable states. Hence, $d$ can still scale with $S$ depending on the direct reachability of the environment. To address this limitation of linear MDPs, we propose a novel structural aggregation framework based on dynamics, named as the "dynamics aggregation". For this newly proposed framework, we design a provably efficient hierarchical reinforcement learning algorithm in linear function approximation that leverages aggregated sub-structures. Our proposed algorithm exhibits statistical efficiency, achieving a regret of $ \tilde{O} ( d_{\psi}^{3/2} H^{3/2}\sqrt{ N T} )$, where $d_{\psi}$ represents the feature dimension of aggregated subMDPs and $N$ signifies the number of aggregated subMDPs. We establish that the condition $d_{\psi}^3 N \ll d^{3}$ is readily met in most real-world environments with hierarchical structures, enabling a substantial improvement in the regret bound compared to LSVI-UCB, which enjoys a regret of $ \tilde{O} (d^{3/2} H^{3/2} \sqrt{ T})$. To the best of our knowledge, this work presents the first HRL algorithm with linear function approximation that offers provable guarantees.