Large Language Model-Enhanced Reinforcement Learning for Diverse and Novel Recommendations

TL;DR

This paper introduces LAAC, a reinforcement learning approach that uses large language models as reference policies to enhance diversity and novelty in recommendations, balancing exploration with user relevance.

Contribution

The paper presents a novel bilevel optimization framework integrating LLMs into RL for recommendation diversity without extensive fine-tuning.

Findings

LAAC outperforms baselines in diversity, novelty, and accuracy.

The method is robust on imbalanced datasets.

It effectively leverages LLM knowledge without expensive fine-tuning.

Abstract

In recommendation systems, diversity and novelty are essential for capturing varied user preferences and encouraging exploration, yet many systems prioritize click relevance. While reinforcement learning (RL) has been explored to improve diversity, it often depends on random exploration that may not align with user interests. We propose LAAC (LLM-guided Adversarial Actor Critic), a novel method that leverages large language models (LLMs) as reference policies to suggest novel items, while training a lightweight policy to refine these suggestions using system-specific data. The method formulates training as a bilevel optimization between actor and critic networks, enabling the critic to selectively favor promising novel actions and the actor to improve its policy beyond LLM recommendations. To mitigate overestimation of unreliable LLM suggestions, we apply regularization that anchors critic values for unexplored items close to well-estimated dataset actions. Experiments on real-world datasets show that LAAC outperforms existing baselines in diversity, novelty, and accuracy, while remaining robust on imbalanced data, effectively integrating LLM knowledge without expensive fine-tuning.