Hindsight Reward Tweaking via Conditional Deep Reinforcement Learning

TL;DR

This paper introduces a novel deep reinforcement learning method that models the influence of reward functions within a near-optimal space, enabling efficient reward tweaking and policy performance improvement across complex tasks.

Contribution

It proposes a hindsight reward tweaking paradigm that extends the input with a condition vector, allowing sensitive regulation of policy characteristics over reward configurations.

Findings

Feasibility demonstrated on MuJoCo tasks

Hyper-policy can be regulated via condition space

Potential for policy performance boosting

Abstract

Designing optimal reward functions has been desired but extremely difficult in reinforcement learning (RL). When it comes to modern complex tasks, sophisticated reward functions are widely used to simplify policy learning yet even a tiny adjustment on them is expensive to evaluate due to the drastically increasing cost of training. To this end, we propose a hindsight reward tweaking approach by designing a novel paradigm for deep reinforcement learning to model the influences of reward functions within a near-optimal space. We simply extend the input observation with a condition vector linearly correlated with the effective environment reward parameters and train the model in a conventional manner except for randomizing reward configurations, obtaining a hyper-policy whose characteristics are sensitively regulated over the condition space. We demonstrate the feasibility of this approach and study one of its potential application in policy performance boosting with multiple MuJoCo tasks.