Mind the Gap: Offline Policy Optimization for Imperfect Rewards

TL;DR

This paper introduces RGM, a novel offline policy optimization method that effectively handles imperfect rewards in reinforcement learning, improving performance and reward correction without online interactions.

Contribution

The paper proposes RGM, a bi-level optimization framework that corrects imperfect rewards and enhances offline RL performance, a novel approach in reward correction for RL.

Findings

RGM outperforms existing methods under various imperfect reward scenarios.

RGM effectively corrects wrong or inconsistent rewards.

RGM retrieves useful information from biased rewards.

Abstract

Reward function is essential in reinforcement learning (RL), serving as the guiding signal to incentivize agents to solve given tasks, however, is also notoriously difficult to design. In many cases, only imperfect rewards are available, which inflicts substantial performance loss for RL agents. In this study, we propose a unified offline policy optimization approach, \textit{RGM (Reward Gap Minimization)}, which can smartly handle diverse types of imperfect rewards. RGM is formulated as a bi-level optimization problem: the upper layer optimizes a reward correction term that performs visitation distribution matching w.r.t. some expert data; the lower layer solves a pessimistic RL problem with the corrected rewards. By exploiting the duality of the lower layer, we derive a tractable algorithm that enables sampled-based learning without any online interactions. Comprehensive experiments demonstrate that RGM achieves superior performance to existing methods under diverse settings of imperfect rewards. Further, RGM can effectively correct wrong or inconsistent rewards against expert preference and retrieve useful information from biased rewards.