Protective Policy Transfer

TL;DR

This paper presents a policy transfer algorithm that enables robots to adapt to new environments efficiently while prioritizing safety, by training separate task and protective policies and using a safety estimator for policy switching.

Contribution

Introduces a novel transfer algorithm with safety-aware policy switching using a learned safety estimator, improving safety during robot skill transfer.

Findings

Successful transfer in four simulated robot locomotion tasks

Effective safety maintenance during adaptation

Enhanced transfer success in diverse environments

Abstract

Being able to transfer existing skills to new situations is a key capability when training robots to operate in unpredictable real-world environments. A successful transfer algorithm should not only minimize the number of samples that the robot needs to collect in the new environment, but also prevent the robot from damaging itself or the surrounding environment during the transfer process. In this work, we introduce a policy transfer algorithm for adapting robot motor skills to novel scenarios while minimizing serious failures. Our algorithm trains two control policies in the training environment: a task policy that is optimized to complete the task of interest, and a protective policy that is dedicated to keep the robot from unsafe events (e.g. falling to the ground). To decide which policy to use during execution, we learn a safety estimator model in the training environment that estimates a continuous safety level of the robot. When used with a set of thresholds, the safety estimator becomes a classifier for switching between the protective policy and the task policy. We evaluate our approach on four simulated robot locomotion problems and a 2D navigation problem and show that our method can achieve successful transfer to notably different environments while taking the robot's safety into consideration.