Revisiting Popularity and Demographic Biases in Recommender Evaluation and Effectiveness

TL;DR

This paper reproduces and extends prior research on biases in recommender systems, revealing significant disparities in recommendation quality across age, gender, and geographic groups, influenced by content popularity and user activity.

Contribution

It confirms and expands previous findings on demographic and popularity biases, providing new insights into factors affecting recommender performance across diverse user groups.

Findings

Recommendation utility decreases with user age.

Women receive less relevant recommendations than men.

Higher recommendation quality for users from well-represented countries.

Abstract

Recommendation algorithms are susceptible to popularity bias: a tendency to recommend popular items even when they fail to meet user needs. A related issue is that the recommendation quality can vary by demographic groups. Marginalized groups or groups that are under-represented in the training data may receive less relevant recommendations from these algorithms compared to others. In a recent study, Ekstrand et al. investigate how recommender performance varies according to popularity and demographics, and find statistically significant differences in recommendation utility between binary genders on two datasets, and significant effects based on age on one dataset. Here we reproduce those results and extend them with additional analyses. We find statistically significant differences in recommender performance by both age and gender. We observe that recommendation utility steadily degrades for older users, and is lower for women than men. We also find that the utility is higher for users from countries with more representation in the dataset. In addition, we find that total usage and the popularity of consumed content are strong predictors of recommender performance and also vary significantly across demographic groups.