Model-Free Robust $\phi$-Divergence Reinforcement Learning Using Both Offline and Online Data

TL;DR

This paper introduces model-free algorithms for robust reinforcement learning that leverage both offline and online data, providing theoretical guarantees for high-dimensional systems with function approximation.

Contribution

It proposes the first unified analysis for $eta$-divergence-based robust policies and introduces a hybrid framework combining offline and online data with new theoretical guarantees.

Findings

First unified analysis for $eta$-divergences in high-dimensional systems.

Introduction of hybrid offline-online robust RL framework.

Theoretical guarantees on policy performance in large state spaces.

Abstract

The robust $\phi$-regularized Markov Decision Process (RRMDP) framework focuses on designing control policies that are robust against parameter uncertainties due to mismatches between the simulator (nominal) model and real-world settings. This work makes two important contributions. First, we propose a model-free algorithm called Robust $\phi$-regularized fitted Q-iteration (RPQ) for learning an $\epsilon$-optimal robust policy that uses only the historical data collected by rolling out a behavior policy (with robust exploratory requirement) on the nominal model. To the best of our knowledge, we provide the first unified analysis for a class of $\phi$-divergences achieving robust optimal policies in high-dimensional systems with general function approximation. Second, we introduce the hybrid robust $\phi$-regularized reinforcement learning framework to learn an optimal robust policy using both historical data and online sampling. Towards this framework, we propose a model-free algorithm called Hybrid robust Total-variation-regularized Q-iteration (HyTQ: pronounced height-Q). To the best of our knowledge, we provide the first improved out-of-data-distribution assumption in large-scale problems with general function approximation under the hybrid robust $\phi$-regularized reinforcement learning framework. Finally, we provide theoretical guarantees on the performance of the learned policies of our algorithms on systems with arbitrary large state space.