Safe Inverse Reinforcement Learning via Control Barrier Function

TL;DR

This paper introduces CBFIRL, a safe inverse reinforcement learning framework that integrates control barrier functions to improve safety during robot learning from demonstrations, demonstrated through improved safety metrics in simulation domains.

Contribution

The paper proposes a novel CBFIRL framework that combines control barrier functions with IRL, jointly optimized via gradient descent for safer robot learning.

Findings

15-20% safety improvement in 2D racecar domain

50% safety improvement in 3D drone domain

Enhanced safety compared to IRL without CBF

Abstract

Learning from Demonstration (LfD) is a powerful method for enabling robots to perform novel tasks as it is often more tractable for a non-roboticist end-user to demonstrate the desired skill and for the robot to efficiently learn from the associated data than for a human to engineer a reward function for the robot to learn the skill via reinforcement learning (RL). Safety issues arise in modern LfD techniques, e.g., Inverse Reinforcement Learning (IRL), just as they do for RL; yet, safe learning in LfD has received little attention. In the context of agile robots, safety is especially vital due to the possibility of robot-environment collision, robot-human collision, and damage to the robot. In this paper, we propose a safe IRL framework, CBFIRL, that leverages the Control Barrier Function (CBF) to enhance the safety of the IRL policy. The core idea of CBFIRL is to combine a loss function inspired by CBF requirements with the objective in an IRL method, both of which are jointly optimized via gradient descent. In the experiments, we show our framework performs safer compared to IRL methods without CBF, that is $\sim15\%$ and $\sim20\%$ improvement for two levels of difficulty of a 2D racecar domain and $\sim 50\%$ improvement for a 3D drone domain.