Large Language Model driven Policy Exploration for Recommender Systems

TL;DR

This paper introduces a novel RL-based recommender system approach that leverages Large Language Models to better understand user preferences, improve policy training, and adaptively balance exploration and exploitation in online environments.

Contribution

It proposes the Interaction-Augmented Learned Policy (iALP) using LLMs for pre-training and introduces adaptive variants A-iALP for online deployment, addressing distribution shift and exploration challenges.

Findings

A-iALP significantly improves recommendation performance in simulated environments.

Adaptive strategies effectively mitigate policy degradation and enhance exploration.

LLM-based pre-training enhances initial recommendation quality.

Abstract

Recent advancements in Recommender Systems (RS) have incorporated Reinforcement Learning (RL), framing the recommendation as a Markov Decision Process (MDP). However, offline RL policies trained on static user data are vulnerable to distribution shift when deployed in dynamic online environments. Additionally, excessive focus on exploiting short-term relevant items can hinder exploration, leading to suboptimal recommendations and negatively impacting long-term user gains. Online RL-based RS also face challenges in production deployment, due to the risks of exposing users to untrained or unstable policies. Large Language Models (LLMs) offer a promising solution to mimic user objectives and preferences for pre-training policies offline to enhance the initial recommendations in online settings. Effectively managing distribution shift and balancing exploration are crucial for improving RL-based RS, especially when leveraging LLM-based pre-training. To address these challenges, we propose an Interaction-Augmented Learned Policy (iALP) that utilizes user preferences distilled from an LLM. Our approach involves prompting the LLM with user states to extract item preferences, learning rewards based on feedback, and updating the RL policy using an actor-critic framework. Furthermore, to deploy iALP in an online scenario, we introduce an adaptive variant, A-iALP, that implements a simple fine-tuning strategy (A-iALP$_{ft}$), and an adaptive approach (A-iALP$_{ap}$) designed to mitigate issues with compromised policies and limited exploration. Experiments across three simulated environments demonstrate that A-iALP introduces substantial performance improvements