PARL: Prompt-based Agents for Reinforcement Learning

TL;DR

This paper introduces PARL, a prompt-based method using large language models as reinforcement learning agents without fine-tuning, capable of learning through interaction in simple environments but limited in complex tasks.

Contribution

PARL is the first approach to utilize LLMs as RL agents via prompting, encoding environment information directly into prompts without additional training.

Findings

PARL matches or outperforms traditional RL agents in simple environments.

Performance drops in tasks requiring complex calculations or decoding.

Demonstrates potential and limitations of LLMs as RL agents.

Abstract

Large language models (LLMs) have demonstrated high performance on tasks expressed in natural language, particularly in zero- or few-shot settings. These are typically framed as supervised (e.g., classification) or unsupervised (e.g., clustering) problems. However, limited work evaluates LLMs as agents in reinforcement learning (RL) tasks (e.g., playing games), where learning occurs through interaction with an environment and a reward system. While prior work focused on representing tasks that rely on a language representation, we study structured, non-linguistic reasoning - such as interpreting positions in a grid world. We therefore introduce PARL (Prompt-based Agent for Reinforcement Learning), a method that uses LLMs as RL agents through prompting, without any fine-tuning. PARL encodes actions, states, and rewards in the prompt, enabling the model to learn through trial-and-error interaction. We evaluate PARL on three standard RL tasks that do not entirely rely on natural language. We show that it can match or outperform traditional RL agents in simple environments by leveraging pretrained knowledge. However, we identify performance limitations in tasks that require complex mathematical operations or decoding states and actions.