Distributionally Robust Self Paced Curriculum Reinforcement Learning

TL;DR

This paper introduces a novel reinforcement learning method that adaptively adjusts robustness to improve stability and performance under distribution shifts, outperforming fixed robustness strategies.

Contribution

It proposes DR-SPCRL, a curriculum approach that dynamically schedules the robustness budget, balancing performance and robustness during training.

Findings

Achieves 11.8% higher episodic return under perturbations.

Stabilizes training compared to fixed robustness methods.

Nearly doubles the performance of nominal RL algorithms.

Abstract

A central challenge in reinforcement learning is that policies trained in controlled environments often fail under distribution shifts at deployment into real-world environments. Distributionally Robust Reinforcement Learning (DRRL) addresses this by optimizing for worst-case performance within an uncertainty set defined by a robustness budget $\epsilon$. However, fixing $\epsilon$ results in a tradeoff between performance and robustness: small values yield high nominal performance but weak robustness, while large values can result in instability and overly conservative policies. We propose Distributionally Robust Self-Paced Curriculum Reinforcement Learning (DR-SPCRL), a method that overcomes this limitation by treating $\epsilon$ as a continuous curriculum. DR-SPCRL adaptively schedules the robustness budget according to the agent's progress, enabling a balance between nominal and robust performance. Empirical results across multiple environments demonstrate that DR-SPCRL not only stabilizes training but also achieves a superior robustness-performance trade-off, yielding an average 11.8\% increase in episodic return under varying perturbations compared to fixed or heuristic scheduling strategies, and achieving approximately 1.9$\times$ the performance of the corresponding nominal RL algorithms.