Automated Hybrid Reward Scheduling via Large Language Models for Robotic Skill Learning

TL;DR

This paper introduces an Automated Hybrid Reward Scheduling framework using Large Language Models to dynamically adjust reward component importance during robotic skill learning, improving reinforcement learning efficiency.

Contribution

It presents a novel LLM-based method for automatically scheduling reward components in RL for complex robotic tasks, enhancing learning performance.

Findings

Achieved an average 6.48% performance improvement across multiple tasks.

Demonstrated the effectiveness of LLM-guided reward scheduling in high-DOF robots.

Showed that dynamic reward adjustment outperforms uniform reward summation.

Abstract

Enabling a high-degree-of-freedom robot to learn specific skills is a challenging task due to the complexity of robotic dynamics. Reinforcement learning (RL) has emerged as a promising solution; however, addressing such problems requires the design of multiple reward functions to account for various constraints in robotic motion. Existing approaches typically sum all reward components indiscriminately to optimize the RL value function and policy. We argue that this uniform inclusion of all reward components in policy optimization is inefficient and limits the robot's learning performance. To address this, we propose an Automated Hybrid Reward Scheduling (AHRS) framework based on Large Language Models (LLMs). This paradigm dynamically adjusts the learning intensity of each reward component throughout the policy optimization process, enabling robots to acquire skills in a gradual and structured manner. Specifically, we design a multi-branch value network, where each branch corresponds to a distinct reward component. During policy optimization, each branch is assigned a weight that reflects its importance, and these weights are automatically computed based on rules designed by LLMs. The LLM generates a rule set in advance, derived from the task description, and during training, it selects a weight calculation rule from the library based on language prompts that evaluate the performance of each branch. Experimental results demonstrate that the AHRS method achieves an average 6.48% performance improvement across multiple high-degree-of-freedom robotic tasks.