ORVIT: Near-Optimal Online Distributionally Robust Reinforcement Learning

TL;DR

This paper introduces ORVIT, an online distributionally robust reinforcement learning algorithm that achieves near-optimal performance guarantees in unknown environments without relying on offline data or generative models.

Contribution

It proposes a practical, computationally efficient online RL method for distributional robustness using $f$-divergence sets, with theoretical regret bounds and empirical validation.

Findings

Achieves sublinear regret in unknown environments.

Improves worst-case performance over baseline methods.

Matches theoretical lower bounds on regret.

Abstract

We investigate reinforcement learning (RL) in the presence of distributional mismatch between training and deployment, where policies trained in simulators often underperform in practice due to mismatches between training and deployment conditions, and thereby reliable guarantees on real-world performance are essential. Distributionally robust RL addresses this issue by optimizing worst-case performance over an uncertainty set of environments and providing an optimized lower bound on deployment performance. However, existing studies typically assume access to either a generative model or offline datasets with broad coverage of the deployment environment-assumptions that limit their practicality in unknown environments without prior knowledge. In this work, we study a more practical and challenging setting: online distributionally robust RL, where the agent interacts only with a single unknown training environment while seeking policies that are robust with respect to an uncertainty set around this nominal model. We consider general $f$-divergence-based ambiguity sets, including $\chi^2$ and KL divergence balls, and design a computationally efficient algorithm that achieves sublinear regret for the robust control objective under minimal assumptions, without requiring generative or offline data access. Moreover, we establish a corresponding minimax lower bound on the regret of any online algorithm, demonstrating the near-optimality of our method. Experiments across diverse environments with model misspecification show that our approach consistently improves worst-case performance and aligns with the theoretical guarantees.