LgTS: Dynamic Task Sampling using LLM-generated sub-goals for Reinforcement Learning Agents

TL;DR

This paper introduces LgTS, a novel method that leverages LLM-generated sub-goals to guide reinforcement learning agents in environments with unknown dynamics, reducing interactions and improving policy learning without requiring fine-tuned models.

Contribution

The work presents a new approach that uses LLMs for sub-goal generation in RL without fine-tuning or pre-trained policies, and employs Teacher-Student learning to minimize environment interactions.

Findings

Graphical sub-goal structures aid policy learning.

LgTS reduces environment interactions in unknown dynamics.

Effective in gridworld and search-and-rescue domains.

Abstract

Recent advancements in reasoning abilities of Large Language Models (LLM) has promoted their usage in problems that require high-level planning for robots and artificial agents. However, current techniques that utilize LLMs for such planning tasks make certain key assumptions such as, access to datasets that permit finetuning, meticulously engineered prompts that only provide relevant and essential information to the LLM, and most importantly, a deterministic approach to allow execution of the LLM responses either in the form of existing policies or plan operators. In this work, we propose LgTS (LLM-guided Teacher-Student learning), a novel approach that explores the planning abilities of LLMs to provide a graphical representation of the sub-goals to a reinforcement learning (RL) agent that does not have access to the transition dynamics of the environment. The RL agent uses Teacher-Student learning algorithm to learn a set of successful policies for reaching the goal state from the start state while simultaneously minimizing the number of environmental interactions. Unlike previous methods that utilize LLMs, our approach does not assume access to a propreitary or a fine-tuned LLM, nor does it require pre-trained policies that achieve the sub-goals proposed by the LLM. Through experiments on a gridworld based DoorKey domain and a search-and-rescue inspired domain, we show that generating a graphical structure of sub-goals helps in learning policies for the LLM proposed sub-goals and the Teacher-Student learning algorithm minimizes the number of environment interactions when the transition dynamics are unknown.