Generative Adversarial Reward Learning for Generalized Behavior Tendency Inference

TL;DR

This paper introduces a generative inverse reinforcement learning approach that automatically learns reward functions from user behavior, enhancing adaptability and generalization in dynamic environments like recommender systems and traffic control.

Contribution

It presents a novel method combining discriminative actor-critic networks and Wasserstein GANs for automatic reward learning, improving over manually-defined rewards.

Findings

Outperforms state-of-the-art methods in multiple scenarios

Effectively models and explains behavioral tendencies

Demonstrates adaptability in dynamic environments

Abstract

Recent advances in reinforcement learning have inspired increasing interest in learning user modeling adaptively through dynamic interactions, e.g., in reinforcement learning based recommender systems. Reward function is crucial for most of reinforcement learning applications as it can provide the guideline about the optimization. However, current reinforcement-learning-based methods rely on manually-defined reward functions, which cannot adapt to dynamic and noisy environments. Besides, they generally use task-specific reward functions that sacrifice generalization ability. We propose a generative inverse reinforcement learning for user behavioral preference modelling, to address the above issues. Instead of using predefined reward functions, our model can automatically learn the rewards from user's actions based on discriminative actor-critic network and Wasserstein GAN. Our model provides a general way of characterizing and explaining underlying behavioral tendencies, and our experiments show our method outperforms state-of-the-art methods in a variety of scenarios, namely traffic signal control, online recommender systems, and scanpath prediction.