Model-Based Reinforcement Learning with Adversarial Training for Online Recommendation

TL;DR

This paper introduces a model-based reinforcement learning approach using adversarial training to improve offline recommendation policies, reducing reliance on extensive real-world interactions and handling large action spaces effectively.

Contribution

It proposes a novel generative adversarial network framework for offline policy learning in recommender systems, addressing bias and data quality issues.

Findings

Effective policy learning from offline and generated data

Reduces bias in model-based reinforcement learning

Performs well with large action spaces

Abstract

Reinforcement learning is well suited for optimizing policies of recommender systems. Current solutions mostly focus on model-free approaches, which require frequent interactions with the real environment, and thus are expensive in model learning. Offline evaluation methods, such as importance sampling, can alleviate such limitations, but usually request a large amount of logged data and do not work well when the action space is large. In this work, we propose a model-based reinforcement learning solution which models user-agent interaction for offline policy learning via a generative adversarial network. To reduce bias in the learned model and policy, we use a discriminator to evaluate the quality of generated data and scale the generated rewards. Our theoretical analysis and empirical evaluations demonstrate the effectiveness of our solution in learning policies from the offline and generated data.