User-Oriented Robust Reinforcement Learning

TL;DR

This paper introduces a user-oriented robust reinforcement learning framework that incorporates user preferences into policy optimization, balancing robustness and personalization, and demonstrates superior performance in MuJoCo tasks.

Contribution

It proposes a novel UOR metric for RL, develops algorithms for different environment distribution knowledge scenarios, and proves their convergence to near-optimal policies.

Findings

UOR-RL achieves state-of-the-art results under the UOR metric.

UOR-RL performs comparably to baselines on average and worst-case metrics.

Theoretical convergence guarantees are provided for the proposed algorithms.

Abstract

Recently, improving the robustness of policies across different environments attracts increasing attention in the reinforcement learning (RL) community. Existing robust RL methods mostly aim to achieve the max-min robustness by optimizing the policy's performance in the worst-case environment. However, in practice, a user that uses an RL policy may have different preferences over its performance across environments. Clearly, the aforementioned max-min robustness is oftentimes too conservative to satisfy user preference. Therefore, in this paper, we integrate user preference into policy learning in robust RL, and propose a novel User-Oriented Robust RL (UOR-RL) framework. Specifically, we define a new User-Oriented Robustness (UOR) metric for RL, which allocates different weights to the environments according to user preference and generalizes the max-min robustness metric. To optimize the UOR metric, we develop two different UOR-RL training algorithms for the scenarios with or without a priori known environment distribution, respectively. Theoretically, we prove that our UOR-RL training algorithms converge to near-optimal policies even with inaccurate or completely no knowledge about the environment distribution. Furthermore, we carry out extensive experimental evaluations in 4 MuJoCo tasks. The experimental results demonstrate that UOR-RL is comparable to the state-of-the-art baselines under the average and worst-case performance metrics, and more importantly establishes new state-of-the-art performance under the UOR metric.