Joint Evaluation of Fairness and Relevance in Recommender Systems with Pareto Frontier

TL;DR

This paper introduces a new method called Distance to Pareto Frontier (DPFR) for jointly evaluating fairness and relevance in recommender systems, providing a more reliable and theoretically grounded measure than existing approaches.

Contribution

The paper proposes DPFR, a modular and intuitive approach that uses Pareto frontiers to assess the joint fairness and relevance of recommender systems, addressing limitations of prior methods.

Findings

DPFR provides a more consistent measure of fairness and relevance.

Existing metrics show inconsistent associations with Pareto-optimal solutions.

DPFR is robust and grounded in Pareto efficiency theory.

Abstract

Fairness and relevance are two important aspects of recommender systems (RSs). Typically, they are evaluated either (i) separately by individual measures of fairness and relevance, or (ii) jointly using a single measure that accounts for fairness with respect to relevance. However, approach (i) often does not provide a reliable joint estimate of the goodness of the models, as it has two different best models: one for fairness and another for relevance. Approach (ii) is also problematic because these measures tend to be ad-hoc and do not relate well to traditional relevance measures, like NDCG. Motivated by this, we present a new approach for jointly evaluating fairness and relevance in RSs: Distance to Pareto Frontier (DPFR). Given some user-item interaction data, we compute their Pareto frontier for a pair of existing relevance and fairness measures, and then use the distance from the frontier as a measure of the jointly achievable fairness and relevance. Our approach is modular and intuitive as it can be computed with existing measures. Experiments with 4 RS models, 3 re-ranking strategies, and 6 datasets show that existing metrics have inconsistent associations with our Pareto-optimal solution, making DPFR a more robust and theoretically well-founded joint measure for assessing fairness and relevance. Our code: https://github.com/theresiavr/DPFR-recsys-evaluation