Do LLMs Understand Collaborative Signals? Diagnosis and Repair

TL;DR

This paper investigates whether large language models can effectively reason over collaborative signals in recommendation tasks, compares their performance to traditional models, and proposes retrieval-augmented generation to enhance their reasoning capabilities.

Contribution

It provides a fundamental analysis of LLMs' reasoning over collaborative information and introduces a retrieval-augmented approach to improve their performance in recommender systems.

Findings

LLMs outperform matrix factorization when given relevant, well-structured information.

Providing more relevant information generally improves LLM performance.

Retrieval-augmented generation enhances LLM reasoning in recommendation tasks.

Abstract

Collaborative information from user-item interactions is a fundamental source of signal in successful recommender systems. Recently, researchers have attempted to incorporate this knowledge into large language model-based recommender approaches (LLMRec) to enhance their performance. However, there has been little fundamental analysis of whether LLMs can effectively reason over collaborative information. In this paper, we analyze the ability of LLMs to reason about collaborative information in recommendation tasks, comparing their performance to traditional matrix factorization (MF) models. We propose a simple and effective method to improve LLMs' reasoning capabilities using retrieval-augmented generation (RAG) over the user-item interaction matrix with four different prompting strategies. Our results show that the LLM outperforms the MF model whenever we provide relevant information in a clear and easy-to-follow format, and prompt the LLM to reason based on it. We observe that with this strategy, in almost all cases, the more information we provide, the better the LLM performs.