Direct Preference Optimization for LLM-Enhanced Recommendation Systems

TL;DR

This paper introduces DPO4Rec, a novel framework that enhances LLM-based recommendation systems by integrating direct preference optimization, leading to improved re-ranking performance and better alignment with recommendation goals.

Contribution

The paper proposes DPO4Rec, a new method that combines preference inference, reward modeling, and structure alignment to improve LLMs for recommendation tasks.

Findings

Significant improvement in re-ranking performance over baselines

Enhanced instruction-following capabilities of LLMs in recommendations

Effective alignment of LLM outputs with recommendation objectives

Abstract

Large Language Models (LLMs) have exhibited remarkable performance across a wide range of domains, motivating research into their potential for recommendation systems. Early efforts have leveraged LLMs' rich knowledge and strong generalization capabilities via in-context learning, where recommendation tasks are framed as prompts. However, LLM performance in recommendation scenarios remains limited due to the mismatch between their pretraining objectives and recommendation tasks, as well as the lack of recommendation-specific data during pretraining. To address these challenges, we propose DPO4Rec, a novel framework that integrates Direct Preference Optimization (DPO) into LLM-enhanced recommendation systems. First, we prompt the LLM to infer user preferences from historical interactions, which are then used to augment traditional ID-based sequential recommendation models. Next, we train a reward model based on knowledge-augmented recommendation architectures to assess the quality of LLM-generated reasoning. Using this, we select the highest- and lowest-ranked responses from N samples to construct a dataset for LLM fine-tuning. Finally, we apply a structure alignment strategy via DPO to align the LLM's outputs with desirable recommendation behavior. Extensive experiments show that DPO4Rec significantly improves re-ranking performance over strong baselines, demonstrating enhanced instruction-following capabilities of LLMs in recommendation tasks.