$\mathrm{E^{2}CFD}$: Towards Effective and Efficient Cost Function Design for Safe Reinforcement Learning via Large Language Model

TL;DR

This paper introduces $ ext{E}^2 ext{CFD}$, a framework that uses large language models to generate and iteratively refine cost functions for safe reinforcement learning, improving policy performance across diverse safety scenarios.

Contribution

The paper presents a novel framework leveraging LLMs and a fast evaluation method to automatically generate and optimize cost functions tailored to specific safety scenarios in reinforcement learning.

Findings

Policies trained with $ ext{E}^2 ext{CFD}$ outperform traditional methods.

The framework effectively adapts to various safety scenarios.

Iterative refinement improves cost function suitability.

Abstract

Different classes of safe reinforcement learning algorithms have shown satisfactory performance in various types of safety requirement scenarios. However, the existing methods mainly address one or several classes of specific safety requirement scenario problems and cannot be applied to arbitrary safety requirement scenarios. In addition, the optimization objectives of existing reinforcement learning algorithms are misaligned with the task requirements. Based on the need to address these issues, we propose $\mathrm{E^{2}CFD}$, an effective and efficient cost function design framework. $\mathrm{E^{2}CFD}$ leverages the capabilities of a large language model (LLM) to comprehend various safety scenarios and generate corresponding cost functions. It incorporates the \textit{fast performance evaluation (FPE)} method to facilitate rapid and iterative updates to the generated cost function. Through this iterative process, $\mathrm{E^{2}CFD}$ aims to obtain the most suitable cost function for policy training, tailored to the specific tasks within the safety scenario. Experiments have proven that the performance of policies trained using this framework is superior to traditional safe reinforcement learning algorithms and policies trained with carefully designed cost functions.