Constrained Meta Reinforcement Learning with Provable Test-Time Safety

TL;DR

This paper introduces a constrained meta reinforcement learning algorithm that guarantees safety and near-optimal performance on test tasks with provable sample complexity bounds, addressing real-world safety concerns.

Contribution

The paper proposes a novel algorithm for constrained meta RL with provable safety and sample complexity guarantees, along with a matching lower bound showing optimality.

Findings

Algorithm achieves safety on test tasks with provable guarantees.

Sample complexity bounds are tight, matching the lower bound.

Enables faster learning of safe policies in real-world applications.

Abstract

Meta reinforcement learning (RL) allows agents to leverage experience across a distribution of tasks on which the agent can train at will, enabling faster learning of optimal policies on new test tasks. Despite its success in improving sample complexity on test tasks, many real-world applications, such as robotics and healthcare, impose safety constraints during testing. Constrained meta RL provides a promising framework for integrating safety into meta RL. An open question in constrained meta RL is how to ensure the safety of the policy on the real-world test task, while reducing the sample complexity and thus, enabling faster learning of optimal policies. To address this gap, we propose an algorithm that refines policies learned during training, with provable safety and sample complexity guarantees for learning a near optimal policy on the test tasks. We further derive a matching lower bound, showing that this sample complexity is tight.