Beyond Utility: Evaluating LLM as Recommender

TL;DR

This paper introduces a multidimensional evaluation framework for LLM-based recommenders, highlighting new aspects like hallucinations and bias, and compares seven models across multiple datasets and strategies.

Contribution

It proposes a novel multidimensional evaluation framework for LLM recommenders, addressing aspects beyond utility, and provides comprehensive analysis of seven models using this framework.

Findings

LLMs excel with prior knowledge and short histories in ranking tasks.

LLMs outperform traditional models in re-ranking settings.

Candidate position bias and hallucinations are significant issues in LLM recommenders.

Abstract

With the rapid development of Large Language Models (LLMs), recent studies employed LLMs as recommenders to provide personalized information services for distinct users. Despite efforts to improve the accuracy of LLM-based recommendation models, relatively little attention is paid to beyond-utility dimensions. Moreover, there are unique evaluation aspects of LLM-based recommendation models, which have been largely ignored. To bridge this gap, we explore four new evaluation dimensions and propose a multidimensional evaluation framework. The new evaluation dimensions include: 1) history length sensitivity, 2) candidate position bias, 3) generation-involved performance, and 4) hallucinations. All four dimensions have the potential to impact performance, but are largely unnecessary for consideration in traditional systems. Using this multidimensional evaluation framework, along with traditional aspects, we evaluate the performance of seven LLM-based recommenders, with three prompting strategies, comparing them with six traditional models on both ranking and re-ranking tasks on four datasets. We find that LLMs excel at handling tasks with prior knowledge and shorter input histories in the ranking setting, and perform better in the re-ranking setting, beating traditional models across multiple dimensions. However, LLMs exhibit substantial candidate position bias issues, and some models hallucinate non-existent items much more often than others. We intend our evaluation framework and observations to benefit future research on the use of LLMs as recommenders. The code and data are available at https://github.com/JiangDeccc/EvaLLMasRecommender.