Reinforcement Learning Problem Solving with Large Language Models

TL;DR

This paper explores using Large Language Models as reinforcement learning agents by formulating RL problems as prompting tasks, enabling policy learning and optimization through iterative prompting, demonstrated in two case studies.

Contribution

It introduces a novel method of using LLM prompting to solve RL problems, including policy learning, episode simulation, and Q-Learning, with practical case studies.

Findings

LLMs can be used to learn and optimize policies in RL tasks.

Prompting techniques enable episode simulation and Q-Learning with LLMs.

Effective in complex workflows like research and legal processes.

Abstract

Large Language Models (LLMs) encapsulate an extensive amount of world knowledge, and this has enabled their application in various domains to improve the performance of a variety of Natural Language Processing (NLP) tasks. This has also facilitated a more accessible paradigm of conversation-based interactions between humans and AI systems to solve intended problems. However, one interesting avenue that shows untapped potential is the use of LLMs as Reinforcement Learning (RL) agents to enable conversational RL problem solving. Therefore, in this study, we explore the concept of formulating Markov Decision Process-based RL problems as LLM prompting tasks. We demonstrate how LLMs can be iteratively prompted to learn and optimize policies for specific RL tasks. In addition, we leverage the introduced prompting technique for episode simulation and Q-Learning, facilitated by LLMs. We then show the practicality of our approach through two detailed case studies for "Research Scientist" and "Legal Matter Intake" workflows.