Leveraging LLMs for reward function design in reinforcement learning control tasks

TL;DR

This paper presents LEARN-Opt, an autonomous LLM-based framework that designs reward functions for reinforcement learning without prior metrics or source code, achieving competitive performance and reducing manual effort.

Contribution

LEARN-Opt introduces a fully autonomous, model-agnostic method for reward function generation that derives performance metrics directly from system descriptions, eliminating the need for human-engineered feedback.

Findings

LEARN-Opt performs comparably or better than state-of-the-art methods.

Automated reward design exhibits high variance, requiring multiple runs.

Low-cost LLMs can find high-quality reward functions similar to larger models.

Abstract

The challenge of designing effective reward functions in reinforcement learning (RL) represents a significant bottleneck, often requiring extensive human expertise and being time-consuming. Previous work and recent advancements in large language models (LLMs) have demonstrated their potential for automating the generation of reward functions. However, existing methodologies often require preliminary evaluation metrics, human-engineered feedback for the refinement process, or the use of environmental source code as context. To address these limitations, this paper introduces LEARN-Opt (LLM-based Evaluator and Analyzer for Reward functioN Optimization). This LLM-based, fully autonomous, and model-agnostic framework eliminates the need for preliminary metrics and environmental source code as context to generate, execute, and evaluate reward function candidates from textual descriptions of systems and task objectives. LEARN-Opt's main contribution lies in its ability to autonomously derive performance metrics directly from the system description and the task objective, enabling unsupervised evaluation and selection of reward functions. Our experiments indicate that LEARN-Opt achieves performance comparable to or better to that of state-of-the-art methods, such as EUREKA, while requiring less prior knowledge. We find that automated reward design is a high-variance problem, where the average-case candidate fails, requiring a multi-run approach to find the best candidates. Finally, we show that LEARN-Opt can unlock the potential of low-cost LLMs to find high-performing candidates that are comparable to, or even better than, those of larger models. This demonstrated performance affirms its potential to generate high-quality reward functions without requiring any preliminary human-defined metrics, thereby reducing engineering overhead and enhancing generalizability.