The Relationship between the Consistency of Users' Ratings and Recommendation Calibration

TL;DR

This paper investigates how the consistency of users' rating behaviors influences the calibration of recommendations, finding that more consistent raters tend to receive more calibrated recommendations, which impacts fairness and effectiveness.

Contribution

It provides empirical evidence linking user rating consistency to recommendation calibration, highlighting its importance for fairness in recommender systems.

Findings

Higher rating consistency correlates with better recommendation calibration.

Inconsistent raters receive less calibrated recommendations.

The study uses multiple algorithms and a movie dataset for analysis.

Abstract

Fairness in recommender systems has recently received attention from researchers. Unfair recommendations have negative impact on the effectiveness of recommender systems as it may degrade users' satisfaction, loyalty, and at worst, it can lead to or perpetuate undesirable social dynamics. One of the factors that may impact fairness is calibration, the degree to which users' preferences on various item categories are reflected in the recommendations they receive. The ability of a recommendation algorithm for generating effective recommendations may depend on the meaningfulness of the input data and the amount of information available in users' profile. In this paper, we aim to explore the relationship between the consistency of users' ratings behavior and the degree of calibrated recommendations they receive. We conduct our analysis on different groups of users based on the consistency of their ratings. Our experimental results on a movie dataset and several recommendation algorithms show that there is a positive correlation between the consistency of users' ratings behavior and the degree of calibration in their recommendations, meaning that user groups with higher inconsistency in their ratings receive less calibrated recommendations.