DARLR: Dual-Agent Offline Reinforcement Learning for Recommender Systems with Dynamic Reward

TL;DR

DARLR introduces a dual-agent offline reinforcement learning framework with dynamic reward shaping and uncertainty adaptation, significantly improving recommendation policies by addressing reward inaccuracy issues in world models.

Contribution

It proposes a novel dual-agent framework with a selector and recommender for dynamic reward estimation and uncertainty management in offline RL for recommender systems.

Findings

DARLR outperforms existing methods on four benchmark datasets.

Dynamic reward shaping improves policy accuracy.

Adaptive uncertainty penalties enhance decision risk mitigation.

Abstract

Model-based offline reinforcement learning (RL) has emerged as a promising approach for recommender systems, enabling effective policy learning by interacting with frozen world models. However, the reward functions in these world models, trained on sparse offline logs, often suffer from inaccuracies. Specifically, existing methods face two major limitations in addressing this challenge: (1) deterministic use of reward functions as static look-up tables, which propagates inaccuracies during policy learning, and (2) static uncertainty designs that fail to effectively capture decision risks and mitigate the impact of these inaccuracies. In this work, a dual-agent framework, DARLR, is proposed to dynamically update world models to enhance recommendation policies. To achieve this, a \textbf{\textit{selector}} is introduced to identify reference users by balancing similarity and diversity so that the \textbf{\textit{recommender}} can aggregate information from these users and iteratively refine reward estimations for dynamic reward shaping. Further, the statistical features of the selected users guide the dynamic adaptation of an uncertainty penalty to better align with evolving recommendation requirements. Extensive experiments on four benchmark datasets demonstrate the superior performance of DARLR, validating its effectiveness. The code is available at https://github.com/ArronDZhang/DARLR.