Are Large Language Models Really Effective for Training-Free Cold-Start Recommendation?

TL;DR

This paper systematically compares large language models and text embedding models for training-free cold-start recommendation, finding that TEMs outperform LLMs in various settings, challenging common assumptions.

Contribution

First controlled experiments directly compare LLMs and TEMs for training-free recommendation, revealing TEMs' superior performance and scalability.

Findings

TEMs outperform LLMs in cold-start scenarios

TEM-based methods are more scalable for training-free recommendation

LLMs are not the only viable option for training-free ranking

Abstract

Recommender systems usually rely on large-scale interaction data to learn from users' past behaviors and make accurate predictions. However, real-world applications often face situations where no training data is available, such as when launching new services or handling entirely new users. In such cases, conventional approaches cannot be applied. This study focuses on training-free recommendation, where no task-specific training is performed, and particularly on \textit{training-free cold-start recommendation} (TFCSR), the more challenging case where the target user has no interactions. Large language models (LLMs) have recently been explored as a promising solution, and numerous studies have been proposed. As the ability of text embedding models (TEMs) increases, they are increasingly recognized as applicable to training-free recommendation, but no prior work has directly compared LLMs and TEMs under identical conditions. We present the first controlled experiments that systematically evaluate these two approaches in the same setting. The results show that TEMs outperform LLM rerankers, and this trend holds not only in cold-start settings but also in warm-start settings with rich interactions. These findings indicate that direct LLM ranking is not the only viable option, contrary to the commonly shared belief, and TEM-based approaches provide a stronger and more scalable basis for training-free recommendation.