Reasoning to Rank: An End-to-End Solution for Exploiting Large Language Models for Recommendation

TL;DR

This paper introduces Reasoning to Rank, an end-to-end training framework that enhances large language models for recommendation by optimizing reasoning processes to better infer user preferences and improve ranking accuracy.

Contribution

It presents a novel reinforcement learning-based approach that internalizes recommendation utility into LLM reasoning, addressing position bias and enabling direct optimization.

Findings

Consistent performance improvements on Amazon datasets.

Effective mitigation of position bias in LLM reasoning.

Insights into key components for optimizing LLM-based recommendation.

Abstract

Recommender systems are tasked to infer users' evolving preferences and rank items aligned with their intents, which calls for in-depth reasoning beyond pattern-based scoring. Recent efforts start to leverage large language models (LLMs) for recommendation, but how to effectively optimize the model for improved recommendation utility is still under explored. In this work, we propose Reasoning to Rank, an end-to-end training framework that internalizes recommendation utility optimization into the learning of step-by-step reasoning in LLMs. To avoid position bias in LLM reasoning and enable direct optimization of the reasoning process, our framework performs reasoning at the user-item level and employs reinforcement learning for end-to-end training of the LLM. Experiments on three Amazon datasets and a large-scale industrial dataset showed consistent gains over strong conventional and LLM-based solutions. Extensive in-depth analyses validate the necessity of the key components in the proposed framework and shed lights on the future developments of this line of work.