Hybrid Cross-domain Robust Reinforcement Learning

TL;DR

HYDRO introduces a hybrid framework combining offline data and online simulation to enhance robust reinforcement learning, effectively addressing dynamics mismatch and improving sample efficiency in uncertain environments.

Contribution

This paper presents HYDRO, the first hybrid cross-domain robust RL framework that leverages both offline datasets and online simulators with novel uncertainty filtering.

Findings

HYDRO outperforms existing robust RL methods across multiple tasks.

HYDRO improves sample efficiency in offline robust RL scenarios.

HYDRO effectively reduces performance gaps between simulator and real environment.

Abstract

Robust reinforcement learning (RL) aims to learn policies that remain effective despite uncertainties in its environment, which frequently arise in real-world applications due to variations in environment dynamics. The robust RL methods learn a robust policy by maximizing value under the worst-case models within a predefined uncertainty set. Offline robust RL algorithms are particularly promising in scenarios where only a fixed dataset is available and new data cannot be collected. However, these approaches often require extensive offline data, and gathering such datasets for specific tasks in specific environments can be both costly and time-consuming. Using an imperfect simulator offers a faster, cheaper, and safer way to collect data for training, but it can suffer from dynamics mismatch. In this paper, we introduce HYDRO, the first Hybrid Cross-Domain Robust RL framework designed to address these challenges. HYDRO utilizes an online simulator to complement the limited amount of offline datasets in the non-trivial context of robust RL. By measuring and minimizing performance gaps between the simulator and the worst-case models in the uncertainty set, HYDRO employs novel uncertainty filtering and prioritized sampling to select the most relevant and reliable simulator samples. Our extensive experiments demonstrate HYDRO's superior performance over existing methods across various tasks, underscoring its potential to improve sample efficiency in offline robust RL.