Do Reviews Matter for Recommendations in the Era of Large Language Models?

TL;DR

This paper investigates the evolving role of user reviews in recommendation systems amidst the rise of large language models, demonstrating that LLMs can effectively replace or augment traditional review-based methods, especially in data-sparse scenarios.

Contribution

It introduces RAREval, a benchmarking framework for review-aware recommenders, and shows that LLMs outperform traditional methods, challenging the necessity of explicit reviews.

Findings

LLMs outperform traditional deep learning in review-aware recommendation tasks.

Removing or distorting reviews does not significantly reduce recommendation accuracy.

LLMs excel in data sparsity and cold-start scenarios.

Abstract

With the advent of large language models (LLMs), the landscape of recommender systems is undergoing a significant transformation. Traditionally, user reviews have served as a critical source of rich, contextual information for enhancing recommendation quality. However, as LLMs demonstrate an unprecedented ability to understand and generate human-like text, this raises the question of whether explicit user reviews remain essential in the era of LLMs. In this paper, we provide a systematic investigation of the evolving role of text reviews in recommendation by comparing deep learning methods and LLM approaches. Particularly, we conduct extensive experiments on eight public datasets with LLMs and evaluate their performance in zero-shot, few-shot, and fine-tuning scenarios. We further introduce a benchmarking evaluation framework for review-aware recommender systems, RAREval, to comprehensively assess the contribution of textual reviews to the recommendation performance of review-aware recommender systems. Our framework examines various scenarios, including the removal of some or all textual reviews, random distortion, as well as recommendation performance in data sparsity and cold-start user settings. Our findings demonstrate that LLMs are capable of functioning as effective review-aware recommendation engines, generally outperforming traditional deep learning approaches, particularly in scenarios characterized by data sparsity and cold-start conditions. In addition, the removal of some or all textual reviews and random distortion does not necessarily lead to declines in recommendation accuracy. These findings motivate a rethinking of how user preference from text reviews can be more effectively leveraged. All code and supplementary materials are available at: https://github.com/zhytk/RAREval-data-processing.