Robust Policy Learning over Multiple Uncertainty Sets

TL;DR

This paper introduces a new reinforcement learning approach that learns policies robust to multiple uncertainty sets, combining system identification and robust RL to improve worst-case performance in diverse environments.

Contribution

It formulates the multi-set robustness problem and develops an algorithm that adapts uncertainty reduction with robustness, outperforming prior methods.

Findings

Improved worst-case performance on control tasks.

Effective handling of multiple uncertainty sets.

Combines benefits of system identification and robust RL.

Abstract

Reinforcement learning (RL) agents need to be robust to variations in safety-critical environments. While system identification methods provide a way to infer the variation from online experience, they can fail in settings where fast identification is not possible. Another dominant approach is robust RL which produces a policy that can handle worst-case scenarios, but these methods are generally designed to achieve robustness to a single uncertainty set that must be specified at train time. Towards a more general solution, we formulate the multi-set robustness problem to learn a policy robust to different perturbation sets. We then design an algorithm that enjoys the benefits of both system identification and robust RL: it reduces uncertainty where possible given a few interactions, but can still act robustly with respect to the remaining uncertainty. On a diverse set of control tasks, our approach demonstrates improved worst-case performance on new environments compared to prior methods based on system identification and on robust RL alone.