LLM-assisted Semantic Option Discovery for Facilitating Adaptive Deep Reinforcement Learning

TL;DR

This paper presents a novel framework leveraging Large Language Models to enhance deep reinforcement learning by enabling semantic skill reuse, real-time constraint monitoring, and improving data efficiency and transferability across environments.

Contribution

The paper introduces an LLM-driven closed-loop framework that maps natural language instructions into executable rules and semantically annotates options for better exploration and transfer in DRL.

Findings

Improved data efficiency in DRL tasks.

Enhanced cross-environment transferability.

Better constraint compliance and interpretability.

Abstract

Despite achieving remarkable success in complex tasks, Deep Reinforcement Learning (DRL) is still suffering from critical issues in practical applications, such as low data efficiency, lack of interpretability, and limited cross-environment transferability. However, the learned policy generating actions based on states are sensitive to the environmental changes, struggling to guarantee behavioral safety and compliance. Recent research shows that integrating Large Language Models (LLMs) with symbolic planning is promising in addressing these challenges. Inspired by this, we introduce a novel LLM-driven closed-loop framework, which enables semantic-driven skill reuse and real-time constraint monitoring by mapping natural language instructions into executable rules and semantically annotating automatically created options. The proposed approach utilizes the general knowledge of LLMs to facilitate exploration efficiency and adapt to transferable options for similar environments, and provides inherent interpretability through semantic annotations. To validate the effectiveness of this framework, we conduct experiments on two domains, Office World and Montezuma's Revenge, respectively. The results demonstrate superior performance in data efficiency, constraint compliance, and cross-task transferability.