Revisiting Prompt Engineering: A Comprehensive Evaluation for LLM-based Personalized Recommendation

TL;DR

This paper evaluates 23 prompt types across multiple datasets and LLMs to determine effective prompt engineering strategies for personalized recommendation, emphasizing cost-efficiency and accuracy in single-user scenarios.

Contribution

It provides a comprehensive large-scale comparison of prompt types for LLM-based recommendation, offering practical guidelines for optimizing prompt design based on model performance and cost.

Findings

Rephrasing instructions, background knowledge, and easier reasoning prompts are cost-effective for small LLMs.

Simple prompts often outperform complex reasoning prompts in high-performance LLMs.

Common NLP prompting styles like step-by-step reasoning may reduce accuracy in recommendation tasks.

Abstract

Large language models (LLMs) can perform recommendation tasks by taking prompts written in natural language as input. Compared to traditional methods such as collaborative filtering, LLM-based recommendation offers advantages in handling cold-start, cross-domain, and zero-shot scenarios, as well as supporting flexible input formats and generating explanations of user behavior. In this paper, we focus on a single-user setting, where no information from other users is used. This setting is practical for privacy-sensitive or data-limited applications. In such cases, prompt engineering becomes especially important for controlling the output generated by the LLM. We conduct a large-scale comparison of 23 prompt types across 8 public datasets and 12 LLMs. We use statistical tests and linear mixed-effects models to evaluate both accuracy and inference cost. Our results show that for cost-efficient LLMs, three types of prompts are especially effective: those that rephrase instructions, consider background knowledge, and make the reasoning process easier to follow. For high-performance LLMs, simple prompts often outperform more complex ones while reducing cost. In contrast, commonly used prompting styles in natural language processing, such as step-by-step reasoning, or the use of reasoning models often lead to lower accuracy. Based on these findings, we provide practical suggestions for selecting prompts and LLMs depending on the required balance between accuracy and cost.