Is ChatGPT a Good Recommender? A Preliminary Study

TL;DR

This study investigates ChatGPT's potential as a general-purpose recommender by evaluating its performance across multiple scenarios without fine-tuning, highlighting its strengths in understanding user needs and generating explainable recommendations.

Contribution

The paper pioneers the use of ChatGPT as a recommendation model through prompt engineering and few-shot learning, demonstrating its capabilities without model fine-tuning.

Findings

ChatGPT achieves promising results in some recommendation tasks.

It reaches baseline performance levels in others.

Human evaluations show ChatGPT generates clearer, more reasonable explanations.

Abstract

Recommendation systems have witnessed significant advancements and have been widely used over the past decades. However, most traditional recommendation methods are task-specific and therefore lack efficient generalization ability. Recently, the emergence of ChatGPT has significantly advanced NLP tasks by enhancing the capabilities of conversational models. Nonetheless, the application of ChatGPT in the recommendation domain has not been thoroughly investigated. In this paper, we employ ChatGPT as a general-purpose recommendation model to explore its potential for transferring extensive linguistic and world knowledge acquired from large-scale corpora to recommendation scenarios. Specifically, we design a set of prompts and evaluate ChatGPT's performance on five recommendation scenarios. Unlike traditional recommendation methods, we do not fine-tune ChatGPT during the entire evaluation process, relying only on the prompts themselves to convert recommendation tasks into natural language tasks. Further, we explore the use of few-shot prompting to inject interaction information that contains user potential interest to help ChatGPT better understand user needs and interests. Comprehensive experimental results on Amazon Beauty dataset show that ChatGPT has achieved promising results in certain tasks and is capable of reaching the baseline level in others. We conduct human evaluations on two explainability-oriented tasks to more accurately evaluate the quality of contents generated by different models. And the human evaluations show ChatGPT can truly understand the provided information and generate clearer and more reasonable results. We hope that our study can inspire researchers to further explore the potential of language models like ChatGPT to improve recommendation performance and contribute to the advancement of the recommendation systems field.