Language Representations Can be What Recommenders Need: Findings and Potentials

TL;DR

This paper demonstrates that advanced language models encode rich information that can be directly used for recommendation tasks, outperforming traditional ID-based collaborative filtering models and revealing a strong connection between language and behavior representations.

Contribution

It re-examines the relationship between language and behavior representations, showing that language models can directly generate effective item embeddings for recommendation without ID-based data.

Findings

Linearly mapped item representations from language models outperform traditional methods.

A simple language-based recommendation model can surpass state-of-the-art ID-based models.

Language representations possess zero-shot recommendation capabilities and encode user intentions.

Abstract

Recent studies empirically indicate that language models (LMs) encode rich world knowledge beyond mere semantics, attracting significant attention across various fields. However, in the recommendation domain, it remains uncertain whether LMs implicitly encode user preference information. Contrary to prevailing understanding that LMs and traditional recommenders learn two distinct representation spaces due to the huge gap in language and behavior modeling objectives, this work re-examines such understanding and explores extracting a recommendation space directly from the language representation space. Surprisingly, our findings demonstrate that item representations, when linearly mapped from advanced LM representations, yield superior recommendation performance. This outcome suggests the possible homomorphism between the advanced language representation space and an effective item representation space for recommendation, implying that collaborative signals may be implicitly encoded within LMs. Motivated by these findings, we explore the possibility of designing advanced collaborative filtering (CF) models purely based on language representations without ID-based embeddings. To be specific, we incorporate several crucial components to build a simple yet effective model, with item titles as the input. Empirical results show that such a simple model can outperform leading ID-based CF models, which sheds light on using language representations for better recommendation. Moreover, we systematically analyze this simple model and find several key features for using advanced language representations: a good initialization for item representations, zero-shot recommendation abilities, and being aware of user intention. Our findings highlight the connection between language modeling and behavior modeling, which can inspire both natural language processing and recommender system communities.