Quantitative analysis of Matthew effect and sparsity problem of recommender systems

TL;DR

This paper quantitatively analyzes the Matthew effect and sparsity issues in recommender systems, focusing on collaborative filtering, to understand their impact and provide insights for system optimization.

Contribution

It offers a detailed comparison of user-based and item-based collaborative filtering mechanisms regarding these problems, aiding industrial recommender system design.

Findings

Matthew effect skews recommendations towards popular items

Data sparsity reduces recommendation coverage

Differences between user-based and item-based filtering mechanisms

Abstract

Recommender systems have received great commercial success. Recommendation has been used widely in areas such as e-commerce, online music FM, online news portal, etc. However, several problems related to input data structure pose serious challenge to recommender system performance. Two of these problems are Matthew effect and sparsity problem. Matthew effect heavily skews recommender system output towards popular items. Data sparsity problem directly affects the coverage of recommendation result. Collaborative filtering is a simple benchmark ubiquitously adopted in the industry as the baseline for recommender system design. Understanding the underlying mechanism of collaborative filtering is crucial for further optimization. In this paper, we do a thorough quantitative analysis on Matthew effect and sparsity problem in the particular context setting of collaborative filtering. We compare the underlying mechanism of user-based and item-based collaborative filtering and give insight to industrial recommender system builders.