RecSys Fairness Metrics: Many to Use But Which One To Choose?

TL;DR

This paper discusses the proliferation of fairness metrics in recommendation systems, highlighting the challenges practitioners face in selecting appropriate metrics and providing guidance to bridge the gap between research and practice.

Contribution

It introduces methods to assist ML practitioners in choosing fairness metrics, emphasizing practical and reflective guidance based on iterative design interviews.

Findings

Practitioners need both practical and reflective guidance for fairness metrics.

The growing complexity of fairness contexts complicates metric selection.

Current research lacks standardized guidance for real-world implementation.

Abstract

In recent years, recommendation and ranking systems have become increasingly popular on digital platforms. However, previous work has highlighted how personalized systems might lead to unintentional harms for users. Practitioners require metrics to measure and mitigate these types of harms in production systems. To meet this need, many fairness definitions have been introduced and explored by the RecSys community. Unfortunately, this has led to a proliferation of possible fairness metrics from which practitioners can choose. The increase in volume and complexity of metrics creates a need for practitioners to deeply understand the nuances of fairness definitions and implementations. Additionally, practitioners need to understand the ethical guidelines that accompany these metrics for responsible implementation. Recent work has shown that there is a proliferation of ethics guidelines and has pointed to the need for more implementation guidance rather than principles alone. The wide variety of available metrics, coupled with the lack of accepted standards or shared knowledge in practice leads to a challenging environment for practitioners to navigate. In this position paper, we focus on this widening gap between the research community and practitioners concerning the availability of metrics versus the ability to put them into practice. We address this gap with our current work, which focuses on developing methods to help ML practitioners in their decision-making processes when picking fairness metrics for recommendation and ranking systems. In our iterative design interviews, we have already found that practitioners need both practical and reflective guidance when refining fairness constraints. This is especially salient given the growing challenge for practitioners to leverage the correct metrics while balancing complex fairness contexts.