Large Language Models as Conversational Movie Recommenders: A User Study

TL;DR

This study evaluates large language models as conversational movie recommenders, revealing they excel in explainability but need improvements in personalization, diversity, and trust, with user context enhancing recommendation quality.

Contribution

It provides empirical insights into LLMs' recommendation capabilities, highlighting the importance of user context and conversation patterns for improving personalization and user experience.

Findings

LLMs offer strong explainability in recommendations.

Personalization and diversity are currently limited in LLM recommendations.

Providing personal context and examples improves recommendation quality.

Abstract

This paper explores the effectiveness of using large language models (LLMs) for personalized movie recommendations from users' perspectives in an online field experiment. Our study involves a combination of between-subject prompt and historic consumption assessments, along with within-subject recommendation scenario evaluations. By examining conversation and survey response data from 160 active users, we find that LLMs offer strong recommendation explainability but lack overall personalization, diversity, and user trust. Our results also indicate that different personalized prompting techniques do not significantly affect user-perceived recommendation quality, but the number of movies a user has watched plays a more significant role. Furthermore, LLMs show a greater ability to recommend lesser-known or niche movies. Through qualitative analysis, we identify key conversational patterns linked to positive and negative user interaction experiences and conclude that providing personal context and examples is crucial for obtaining high-quality recommendations from LLMs.