Chain of Uncertain Rewards with Large Language Models for Reinforcement Learning

TL;DR

The paper introduces CoUR, a framework using large language models to improve reward function design in reinforcement learning, reducing evaluation costs and increasing robustness.

Contribution

It presents a novel reward design framework that leverages LLMs, code uncertainty quantification, and Bayesian optimization to enhance efficiency and performance.

Findings

CoUR achieves better performance across multiple RL environments.

It significantly reduces the cost of reward evaluations.

The framework demonstrates robustness in reward function design.

Abstract

Designing effective reward functions is a cornerstone of reinforcement learning (RL), yet it remains a challenging and labor-intensive process due to the inefficiencies and inconsistencies inherent in traditional methods. Existing methods often rely on extensive manual design and evaluation steps, which are prone to redundancy and overlook local uncertainties at intermediate decision points. To address these challenges, we propose the Chain of Uncertain Rewards (CoUR), a novel framework that integrates large language models (LLMs) to streamline reward function design and evaluation in RL environments. Specifically, our CoUR introduces code uncertainty quantification with a similarity selection mechanism that combines textual and semantic analyses to identify and reuse the most relevant reward function components. By reducing redundant evaluations and leveraging Bayesian optimization on decoupled reward terms, CoUR enables a more efficient and robust search for optimal reward feedback. We comprehensively evaluate CoUR across nine original environments from IsaacGym and all 20 tasks from the Bidexterous Manipulation benchmark. The experimental results demonstrate that CoUR not only achieves better performance but also significantly lowers the cost of reward evaluations.