Curriculum Reinforcement Learning for Complex Reward Functions

TL;DR

This paper introduces a two-stage reward curriculum for reinforcement learning that improves performance in environments with complex, multi-term reward functions by gradually transitioning from simple to full rewards.

Contribution

The authors propose a novel two-stage reward curriculum method with an automatic transition mechanism and a flexible replay buffer, enhancing RL stability and performance with complex rewards.

Findings

Significant performance improvement over baseline in control tasks.

Effective balancing of task and constraint satisfaction in RL policies.

Demonstrated applicability in robotic and simulated environments.

Abstract

Reinforcement learning (RL) has emerged as a powerful tool for tackling control problems, but its practical application is often hindered by the complexity arising from intricate reward functions with multiple terms. The reward hypothesis posits that any objective can be encapsulated in a scalar reward function, yet balancing individual, potentially adversarial, reward terms without exploitation remains challenging. To overcome the limitations of traditional RL methods, which often require precise balancing of competing reward terms, we propose a two-stage reward curriculum that first maximizes a simple reward function and then transitions to the full, complex reward. We provide a method based on how well an actor fits a critic to automatically determine the transition point between the two stages. Additionally, we introduce a flexible replay buffer that enables efficient phase transfer by reusing samples from one stage in the next. We evaluate our method on the DeepMind control suite, modified to include an additional constraint term in the reward definitions. We further evaluate our method in a mobile robot scenario with even more competing reward terms. In both settings, our two-stage reward curriculum achieves a substantial improvement in performance compared to a baseline trained without curriculum. Instead of exploiting the constraint term in the reward, it is able to learn policies that balance task completion and constraint satisfaction. Our results demonstrate the potential of two-stage reward curricula for efficient and stable RL in environments with complex rewards, paving the way for more robust and adaptable robotic systems in real-world applications.