Safe Reinforcement Learning with Free-form Natural Language Constraints and Pre-Trained Language Models

TL;DR

This paper introduces a novel safe reinforcement learning approach that leverages pre-trained language models to understand and incorporate free-form natural language constraints, eliminating the need for predefined cost functions and improving safety and flexibility.

Contribution

The paper proposes using pre-trained language models to interpret natural language constraints in safe RL, removing the reliance on ground-truth cost functions and enhancing understanding of diverse human instructions.

Findings

Achieves strong performance in grid-world and robot control tasks.

Successfully interprets complex natural language constraints.

Learns safe policies without ground-truth cost functions.

Abstract

Safe reinforcement learning (RL) agents accomplish given tasks while adhering to specific constraints. Employing constraints expressed via easily-understandable human language offers considerable potential for real-world applications due to its accessibility and non-reliance on domain expertise. Previous safe RL methods with natural language constraints typically adopt a recurrent neural network, which leads to limited capabilities when dealing with various forms of human language input. Furthermore, these methods often require a ground-truth cost function, necessitating domain expertise for the conversion of language constraints into a well-defined cost function that determines constraint violation. To address these issues, we proposes to use pre-trained language models (LM) to facilitate RL agents' comprehension of natural language constraints and allow them to infer costs for safe policy learning. Through the use of pre-trained LMs and the elimination of the need for a ground-truth cost, our method enhances safe policy learning under a diverse set of human-derived free-form natural language constraints. Experiments on grid-world navigation and robot control show that the proposed method can achieve strong performance while adhering to given constraints. The usage of pre-trained LMs allows our method to comprehend complicated constraints and learn safe policies without the need for ground-truth cost at any stage of training or evaluation. Extensive ablation studies are conducted to demonstrate the efficacy of each part of our method.