RLHF Fine-Tuning of LLMs for Alignment with Implicit User Feedback in Conversational Recommenders

TL;DR

This paper introduces a reinforcement learning approach using human feedback to fine-tune large language models for conversational recommender systems, effectively aligning recommendations with implicit user signals in multi-turn dialogues.

Contribution

It presents a novel RLHF fine-tuning method that leverages implicit user feedback signals to improve LLM-based conversational recommenders, outperforming traditional supervised fine-tuning.

Findings

Enhanced top-k recommendation accuracy

Improved coherence and user satisfaction

Effective alignment with implicit user signals

Abstract

Conversational recommender systems (CRS) based on Large Language Models (LLMs) need to constantly be aligned to the user preferences to provide satisfying and context-relevant item recommendations. The traditional supervised fine-tuning cannot capture the implicit feedback signal, e.g., dwell time, sentiment polarity, or engagement patterns. In this paper, we share a fine-tuning solution using human feedback reinforcement learning (RLHF) to maximize implied user feedback (IUF) in a multi-turn recommendation context. We specify a reward model $R_{\phi}$ learnt on weakly-labelled engagement information and maximize user-centric utility by optimizing the foundational LLM M_{\theta} through a proximal policy optimization (PPO) approach. The architecture models conversational state transitions $s_t \to a_t \to s_{t +1}$, where the action $a_t$ is associated with LLM-generated item suggestions only on condition of conversation history in the past. The evaluation across synthetic and real-world datasets (e.g.REDIAL, OpenDialKG) demonstrates that our RLHF-fine-tuned models can perform better in terms of top-$k$ recommendation accuracy, coherence, and user satisfaction compared to (arrow-zero-cmwrquca-teja-falset ensuite 2Round group-deca States penalty give up This paper shows that implicit signal alignment can be efficient in achieving scalable and user-adaptive design of CRS.