Learning Rewards to Optimize Global Performance Metrics in Deep Reinforcement Learning

TL;DR

This paper introduces LR4GPM, a deep reinforcement learning method that automatically learns reward functions to optimize global performance metrics, reducing reward engineering complexity and improving performance in various domains.

Contribution

LR4GPM is a novel RL approach that learns reward functions to directly optimize global metrics, avoiding manual reward engineering and handling non-stationary training data.

Findings

LR4GPM outperforms recent autonomous driving competition winners.

The method effectively learns reward functions aligned with global metrics.

Training tricks improve stability and performance of LR4GPM.

Abstract

When applying reinforcement learning (RL) to a new problem, reward engineering is a necessary, but often difficult and error-prone task a system designer has to face. To avoid this step, we propose LR4GPM, a novel (deep) RL method that can optimize a global performance metric, which is supposed to be available as part of the problem description. LR4GPM alternates between two phases: (1) learning a (possibly vector) reward function used to fit the performance metric, and (2) training a policy to optimize an approximation of this performance metric based on the learned rewards. Such RL training is not straightforward since both the reward function and the policy are trained using non-stationary data. To overcome this issue, we propose several training tricks. We demonstrate the efficiency of LR4GPM on several domains. Notably, LR4GPM outperforms the winner of a recent autonomous driving competition organized at DAI'2020.