Measuring Individual User Fairness with User Similarity and Effectiveness Disparity

TL;DR

This paper introduces PUF, a novel evaluation measure for individual user fairness in recommender systems that simultaneously considers user similarity and effectiveness disparity, addressing limitations of existing measures.

Contribution

The paper presents PUF, the first measure to jointly evaluate user similarity and effectiveness disparity in individual fairness, validated across multiple datasets and rankers.

Findings

PUF consistently captures both user similarity and effectiveness disparity.

Existing measures are insensitive to at least one aspect of fairness.

PUF outperforms other measures in robustness and comprehensiveness.

Abstract

Individual user fairness is commonly understood as treating similar users similarly. In Recommender Systems (RSs), several evaluation measures exist for quantifying individual user fairness. These measures evaluate fairness via either: (i) the disparity in RS effectiveness scores regardless of user similarity, or (ii) the disparity in items recommended to similar users regardless of item relevance. Both disparity in recommendation effectiveness and user similarity are very important in fairness, yet no existing individual user fairness measure simultaneously accounts for both. In brief, current user fairness evaluation measures implement a largely incomplete definition of fairness. To fill this gap, we present Pairwise User unFairness (PUF), a novel evaluation measure of individual user fairness that considers both effectiveness disparity and user similarity. PUF is the only measure that can express this important distinction. We empirically validate that PUF does this consistently across 4 datasets and 7 rankers, and robustly when varying user similarity or effectiveness. In contrast, all other measures are either almost insensitive to effectiveness disparity or completely insensitive to user similarity. We contribute the first RS evaluation measure to reliably capture both user similarity and effectiveness in individual user fairness. Our code: https://github.com/theresiavr/PUF-individual-user-fairness-recsys.