Cautious Adaptation For Reinforcement Learning in Safety-Critical Settings

TL;DR

This paper introduces CARL, a risk-averse reinforcement learning method that trains in diverse source environments to enable cautious adaptation in safety-critical target settings, improving safety and performance.

Contribution

The paper proposes a novel risk-aware adaptation framework, CARL, that leverages model-based RL to estimate uncertainties and guide cautious exploration in safety-critical environments.

Findings

CARL achieves higher rewards than baselines in safety-critical tasks.

CARL reduces failure rates during adaptation.

It effectively balances exploration and safety in diverse environments.

Abstract

Reinforcement learning (RL) in real-world safety-critical target settings like urban driving is hazardous, imperiling the RL agent, other agents, and the environment. To overcome this difficulty, we propose a "safety-critical adaptation" task setting: an agent first trains in non-safety-critical "source" environments such as in a simulator, before it adapts to the target environment where failures carry heavy costs. We propose a solution approach, CARL, that builds on the intuition that prior experience in diverse environments equips an agent to estimate risk, which in turn enables relative safety through risk-averse, cautious adaptation. CARL first employs model-based RL to train a probabilistic model to capture uncertainty about transition dynamics and catastrophic states across varied source environments. Then, when exploring a new safety-critical environment with unknown dynamics, the CARL agent plans to avoid actions that could lead to catastrophic states. In experiments on car driving, cartpole balancing, half-cheetah locomotion, and robotic object manipulation, CARL successfully acquires cautious exploration behaviors, yielding higher rewards with fewer failures than strong RL adaptation baselines. Website at https://sites.google.com/berkeley.edu/carl.