Reward-Conditioned Reinforcement Learning

TL;DR

Reward-Conditioned Reinforcement Learning (RCRL) enables agents to adapt to multiple reward functions using shared data, improving robustness and flexibility without additional environment interactions.

Contribution

Introduction of RCRL, a method that conditions on reward parameters and reuses data to adapt to new objectives, bridging single-task and multi-task learning.

Findings

RCRL improves sample efficiency under the nominal reward.

It enables zero-shot behavioral adjustments at deployment.

Supports adaptation to new reward parameterizations efficiently.

Abstract

Single-task RL agents are typically trained under a fixed reward function, which limits their robustness to reward misspecification and their ability to adapt to changing preferences. We introduce Reward-Conditioned Reinforcement Learning (RCRL), an off-policy method that conditions agents on reward parameterizations while collecting experience under a single nominal objective. By recomputing counterfactual rewards from shared replay data, RCRL exposes the agent to multiple reward objectives without additional environment interaction, connecting single-task RL with ideas from multi-objective and multi-task learning. Across single-task, multi-task, and vision-based benchmarks, RCRL improves sample efficiency under the nominal reward parameterization, enables efficient adaptation to new parameterizations, and supports zero-shot behavioral adjustment at deployment. Our results show that RCRL provides a scalable mechanism for learning robust, steerable policies without sacrificing the simplicity of single-task training.