Learning Safe Policies with Expert Guidance

TL;DR

This paper introduces a framework for training reinforcement learning agents to behave safely by leveraging expert demonstrations, optimizing reward functions within known constraints, and ensuring avoidance of unsafe states.

Contribution

It presents a novel theoretical framework and two optimization methods for safe reinforcement learning guided by expert demonstrations.

Findings

The algorithms effectively avoid unsafe states in discrete and continuous environments.

Agents successfully imitate expert behavior while maintaining safety.

The methods demonstrate practical viability in complex RL tasks.

Abstract

We propose a framework for ensuring safe behavior of a reinforcement learning agent when the reward function may be difficult to specify. In order to do this, we rely on the existence of demonstrations from expert policies, and we provide a theoretical framework for the agent to optimize in the space of rewards consistent with its existing knowledge. We propose two methods to solve the resulting optimization: an exact ellipsoid-based method and a method in the spirit of the "follow-the-perturbed-leader" algorithm. Our experiments demonstrate the behavior of our algorithm in both discrete and continuous problems. The trained agent safely avoids states with potential negative effects while imitating the behavior of the expert in the other states.