VideolandGPT: A User Study on a Conversational Recommender System

TL;DR

This study explores how large language models can improve conversational recommender systems by personalizing video suggestions, evaluating user satisfaction, fairness, and ranking accuracy through a user study on Videoland.

Contribution

Introduces VideolandGPT, a LLM-based conversational recommender system for VOD platforms, and provides empirical evaluation of its personalization, user experience, and fairness.

Findings

Personalized system outperforms non-personalized in accuracy and satisfaction.

Both systems increase visibility of less prominent items.

Fairness issues observed with recommendations sometimes including unavailable items.

Abstract

This paper investigates how large language models (LLMs) can enhance recommender systems, with a specific focus on Conversational Recommender Systems that leverage user preferences and personalised candidate selections from existing ranking models. We introduce VideolandGPT, a recommender system for a Video-on-Demand (VOD) platform, Videoland, which uses ChatGPT to select from a predetermined set of contents, considering the additional context indicated by users' interactions with a chat interface. We evaluate ranking metrics, user experience, and fairness of recommendations, comparing a personalised and a non-personalised version of the system, in a between-subject user study. Our results indicate that the personalised version outperforms the non-personalised in terms of accuracy and general user satisfaction, while both versions increase the visibility of items which are not in the top of the recommendation lists. However, both versions present inconsistent behavior in terms of fairness, as the system may generate recommendations which are not available on Videoland.