Single-Trajectory Distributionally Robust Reinforcement Learning

TL;DR

This paper introduces a novel model-free distributionally robust reinforcement learning algorithm, DRQ, capable of learning robust policies from a single trajectory with proven convergence and superior performance.

Contribution

It presents the first fully model-free DRRL algorithm that learns from a single trajectory, utilizing a multi-timescale framework and providing convergence guarantees.

Findings

Demonstrates superior robustness over non-robust methods.

Shows improved sample efficiency in experiments.

Validates convergence through theoretical analysis.

Abstract

To mitigate the limitation that the classical reinforcement learning (RL) framework heavily relies on identical training and test environments, Distributionally Robust RL (DRRL) has been proposed to enhance performance across a range of environments, possibly including unknown test environments. As a price for robustness gain, DRRL involves optimizing over a set of distributions, which is inherently more challenging than optimizing over a fixed distribution in the non-robust case. Existing DRRL algorithms are either model-based or fail to learn from a single sample trajectory. In this paper, we design a first fully model-free DRRL algorithm, called distributionally robust Q-learning with single trajectory (DRQ). We delicately design a multi-timescale framework to fully utilize each incrementally arriving sample and directly learn the optimal distributionally robust policy without modelling the environment, thus the algorithm can be trained along a single trajectory in a model-free fashion. Despite the algorithm's complexity, we provide asymptotic convergence guarantees by generalizing classical stochastic approximation tools. Comprehensive experimental results demonstrate the superior robustness and sample complexity of our proposed algorithm, compared to non-robust methods and other robust RL algorithms.