Reward Balancing Revisited: Enhancing Offline Reinforcement Learning for Recommender Systems

TL;DR

This paper introduces R3S, an offline reinforcement learning framework that balances reward predictions and policy diversity, improving recommender system performance by integrating model uncertainty and penalization strategies.

Contribution

The paper proposes R3S, a novel offline RL approach that addresses reward balancing and diversity in recommender systems through uncertainty integration and penalization.

Findings

R3S enhances world model accuracy.

R3S improves recommendation diversity.

R3S effectively balances user preferences.

Abstract

Offline reinforcement learning (RL) has emerged as a prevalent and effective methodology for real-world recommender systems, enabling learning policies from historical data and capturing user preferences. In offline RL, reward shaping encounters significant challenges, with past efforts to incorporate prior strategies for uncertainty to improve world models or penalize underexplored state-action pairs. Despite these efforts, a critical gap remains: the simultaneous balancing of intrinsic biases in world models and the diversity of policy recommendations. To address this limitation, we present an innovative offline RL framework termed Reallocated Reward for Recommender Systems (R3S). By integrating inherent model uncertainty to tackle the intrinsic fluctuations in reward predictions, we boost diversity for decision-making to align with a more interactive paradigm, incorporating extra penalizers with decay that deter actions leading to diminished state variety at both local and global scales. The experimental results demonstrate that R3S improves the accuracy of world models and efficiently harmonizes the heterogeneous preferences of the users.