Revisiting Language Models in Neural News Recommender Systems

TL;DR

This paper critically examines the impact of different language model sizes on neural news recommender systems, revealing that larger models do not always improve performance but help cold-start user recommendations.

Contribution

It provides a comprehensive analysis comparing pre-trained language models of various sizes in news RSs, clarifying their effects and resource requirements.

Findings

Larger LMs do not always improve recommendation accuracy.

Smaller LMs are more efficient and easier to fine-tune.

Larger LMs benefit cold-start user recommendations.

Abstract

Neural news recommender systems (RSs) have integrated language models (LMs) to encode news articles with rich textual information into representations, thereby improving the recommendation process. Most studies suggest that (i) news RSs achieve better performance with larger pre-trained language models (PLMs) than shallow language models (SLMs), and (ii) that large language models (LLMs) outperform PLMs. However, other studies indicate that PLMs sometimes lead to worse performance than SLMs. Thus, it remains unclear whether using larger LMs consistently improves the performance of news RSs. In this paper, we revisit, unify, and extend these comparisons of the effectiveness of LMs in news RSs using the real-world MIND dataset. We find that (i) larger LMs do not necessarily translate to better performance in news RSs, and (ii) they require stricter fine-tuning hyperparameter selection and greater computational resources to achieve optimal recommendation performance than smaller LMs. On the positive side, our experiments show that larger LMs lead to better recommendation performance for cold-start users: they alleviate dependency on extensive user interaction history and make recommendations more reliant on the news content.