Maximum-Entropy Regularized Decision Transformer with Reward Relabelling for Dynamic Recommendation

TL;DR

This paper introduces EDT4Rec, a novel offline reinforcement learning method for recommendation systems that enhances exploration and trajectory stitching using maximum entropy principles and reward relabeling, outperforming existing approaches.

Contribution

The paper proposes a new Decision Transformer variant with max entropy exploration and reward relabeling to improve offline and online recommendation performance.

Findings

Outperforms existing methods on six real-world datasets

Enhances online exploration capabilities

Improves trajectory stitching in recommendation tasks

Abstract

Reinforcement learning-based recommender systems have recently gained popularity. However, due to the typical limitations of simulation environments (e.g., data inefficiency), most of the work cannot be broadly applied in all domains. To counter these challenges, recent advancements have leveraged offline reinforcement learning methods, notable for their data-driven approach utilizing offline datasets. A prominent example of this is the Decision Transformer. Despite its popularity, the Decision Transformer approach has inherent drawbacks, particularly evident in recommendation methods based on it. This paper identifies two key shortcomings in existing Decision Transformer-based methods: a lack of stitching capability and limited effectiveness in online adoption. In response, we introduce a novel methodology named Max-Entropy enhanced Decision Transformer with Reward Relabeling for Offline RLRS (EDT4Rec). Our approach begins with a max entropy perspective, leading to the development of a max entropy enhanced exploration strategy. This strategy is designed to facilitate more effective exploration in online environments. Additionally, to augment the model's capability to stitch sub-optimal trajectories, we incorporate a unique reward relabeling technique. To validate the effectiveness and superiority of EDT4Rec, we have conducted comprehensive experiments across six real-world offline datasets and in an online simulator.