Soft-Robust Actor-Critic Policy-Gradient

TL;DR

This paper introduces a Soft-Robust Actor-Critic algorithm that balances robustness to model uncertainty with reduced conservativeness, improving policy learning in uncertain environments.

Contribution

The paper proposes a novel SR-AC algorithm that learns policies considering a distribution over uncertainties, avoiding the overly conservative nature of traditional robust methods.

Findings

SR-AC converges reliably in tested domains.

SR-AC outperforms standard and robust methods in efficiency.

The approach effectively balances robustness and performance.

Abstract

Robust Reinforcement Learning aims to derive optimal behavior that accounts for model uncertainty in dynamical systems. However, previous studies have shown that by considering the worst case scenario, robust policies can be overly conservative. Our soft-robust framework is an attempt to overcome this issue. In this paper, we present a novel Soft-Robust Actor-Critic algorithm (SR-AC). It learns an optimal policy with respect to a distribution over an uncertainty set and stays robust to model uncertainty but avoids the conservativeness of robust strategies. We show the convergence of SR-AC and test the efficiency of our approach on different domains by comparing it against regular learning methods and their robust formulations.