A Large Language Model-Driven Reward Design Framework via Dynamic Feedback for Reinforcement Learning

TL;DR

This paper introduces CARD, a framework that uses large language models to iteratively generate and refine reward functions for reinforcement learning, reducing human effort and improving task performance.

Contribution

The paper presents a novel LLM-driven reward design framework with dynamic feedback mechanisms, including trajectory preference evaluation, to automate and enhance reward function creation.

Findings

Outperforms baselines on Meta-World and ManiSkill2 tasks.

Achieves better or comparable performance to expert-designed rewards on most tasks.

Surpasses the oracle reward on 3 tasks, demonstrating effectiveness.

Abstract

Large Language Models (LLMs) have shown significant potential in designing reward functions for Reinforcement Learning (RL) tasks. However, obtaining high-quality reward code often involves human intervention, numerous LLM queries, or repetitive RL training. To address these issues, we propose CARD, a LLM-driven Reward Design framework that iteratively generates and improves reward function code. Specifically, CARD includes a Coder that generates and verifies the code, while a Evaluator provides dynamic feedback to guide the Coder in improving the code, eliminating the need for human feedback. In addition to process feedback and trajectory feedback, we introduce Trajectory Preference Evaluation (TPE), which evaluates the current reward function based on trajectory preferences. If the code fails the TPE, the Evaluator provides preference feedback, avoiding RL training at every iteration and making the reward function better aligned with the task objective. Empirical results on Meta-World and ManiSkill2 demonstrate that our method achieves an effective balance between task performance and token efficiency, outperforming or matching the baselines across all tasks. On 10 out of 12 tasks, CARD shows better or comparable performance to policies trained with expert-designed rewards, and our method even surpasses the oracle on 3 tasks.