Transferable Reward Learning by Dynamics-Agnostic Discriminator Ensemble

TL;DR

This paper introduces DARL, a dynamics-agnostic discriminator ensemble method for reward learning in imitation learning, enabling transferability across different environments by decoupling reward functions from dynamics.

Contribution

DARL is the first method to learn both state-only and state-action reward functions that are transferable across environments by decoupling rewards from dynamics using a discriminator ensemble.

Findings

DARL outperforms existing methods in transferred MuJoCo tasks.

It effectively recovers reward functions in environments with changed dynamics.

DARL handles both state-only and state-action reward scenarios.

Abstract

Recovering reward function from expert demonstrations is a fundamental problem in reinforcement learning. The recovered reward function captures the motivation of the expert. Agents can imitate experts by following these reward functions in their environment, which is known as apprentice learning. However, the agents may face environments different from the demonstrations, and therefore, desire transferable reward functions. Classical reward learning methods such as inverse reinforcement learning (IRL) or, equivalently, adversarial imitation learning (AIL), recover reward functions coupled with training dynamics, which are hard to be transferable. Previous dynamics-agnostic reward learning methods rely on assumptions such as that the reward function has to be state-only, restricting their applicability. In this work, we present a dynamics-agnostic discriminator-ensemble reward learning method (DARL) within the AIL framework, capable of learning both state-action and state-only reward functions. DARL achieves this by decoupling the reward function from training dynamics, employing a dynamics-agnostic discriminator on a latent space derived from the original state-action space. This latent space is optimized to minimize information on the dynamics. We moreover discover the policy-dependency issue of the AIL framework that reduces the transferability. DARL represents the reward function as an ensemble of discriminators during training to eliminate policy dependencies. Empirical studies on MuJoCo tasks with changed dynamics show that DARL better recovers the reward function and results in better imitation performance in transferred environments, handling both state-only and state-action reward scenarios.