The Impact of Popularity Bias on Fairness and Calibration in Recommendation

TL;DR

This paper investigates how popularity bias in recommendation algorithms affects fairness and calibration, revealing that increased popularity bias correlates with higher miscalibration across user groups.

Contribution

It introduces the use of miscalibration as a metric to measure fairness impacts of popularity bias in recommender systems and demonstrates their relationship through empirical analysis.

Findings

Popularity bias correlates with increased miscalibration.

Algorithms amplifying popularity bias tend to be less fair.

User groups' exposure to popular items varies with bias levels.

Abstract

Recently there has been a growing interest in fairness-aware recommender systems, including fairness in providing consistent performance across different users or groups of users. A recommender system could be considered unfair if the recommendations do not fairly represent the tastes of a certain group of users while other groups receive recommendations that are consistent with their preferences. In this paper, we use a metric called miscalibration for measuring how a recommendation algorithm is responsive to users' true preferences and we consider how various algorithms may result in different degrees of miscalibration. A well-known type of bias in recommendation is popularity bias where few popular items are over-represented in recommendations, while the majority of other items do not get significant exposure. We conjecture that popularity bias is one important factor leading to miscalibration in recommendation. Our experimental results using two real-world datasets show that there is a strong correlation between how different user groups are affected by algorithmic popularity bias and their level of interest in popular items. Moreover, we show algorithms with greater popularity bias amplification tend to have greater miscalibration.