Upper and Lower Bounds for Distributionally Robust Off-Dynamics Reinforcement Learning

TL;DR

This paper introduces a new algorithm for distributionally robust off-dynamics reinforcement learning that achieves near-optimal performance bounds and significantly improves computational efficiency compared to previous methods.

Contribution

The paper proposes We-DRIVE-U, a novel algorithm with improved theoretical guarantees and reduced computational complexity for robust RL under uncertain transition dynamics.

Findings

Achieves near-optimal suboptimality bounds up to b1A9(\u00d7)

Constructs a hard instance and derives a lower bound, showing near-optimality

Reduces policy switch and oracle call complexities from b1A0(K) to b1A0(dH log(K))

Abstract

We study off-dynamics Reinforcement Learning (RL), where the policy training and deployment environments are different. To deal with this environmental perturbation, we focus on learning policies robust to uncertainties in transition dynamics under the framework of distributionally robust Markov decision processes (DRMDPs), where the nominal and perturbed dynamics are linear Markov Decision Processes. We propose a novel algorithm We-DRIVE-U that enjoys an average suboptimality $\widetilde{\mathcal{O}}\big({d H \cdot \min \{1/{\rho}, H\}/\sqrt{K} }\big)$, where $K$ is the number of episodes, $H$ is the horizon length, $d$ is the feature dimension and $\rho$ is the uncertainty level. This result improves the state-of-the-art by $\mathcal{O}(dH/\min\{1/\rho,H\})$. We also construct a novel hard instance and derive the first information-theoretic lower bound in this setting, which indicates our algorithm is near-optimal up to $\mathcal{O}(\sqrt{H})$ for any uncertainty level $\rho\in(0,1]$. Our algorithm also enjoys a 'rare-switching' design, and thus only requires $\mathcal{O}(dH\log(1+H^2K))$ policy switches and $\mathcal{O}(d^2H\log(1+H^2K))$ calls for oracle to solve dual optimization problems, which significantly improves the computational efficiency of existing algorithms for DRMDPs, whose policy switch and oracle complexities are both $\mathcal{O}(K)$.