Exploring How Fair Model Representations Relate to Fair Recommendations

TL;DR

This paper investigates the relationship between fair model representations and recommendation fairness, showing that representation fairness does not reliably predict recommendation parity, and introduces new evaluation methods.

Contribution

It challenges the assumption that demographic information in representations directly correlates with recommendation fairness and proposes two new metrics for assessing recommendation-level fairness.

Findings

Optimizing for fair representations improves recommendation parity.

Representation-level fairness metrics are poor proxies for recommendation fairness.

Recommendation-level metrics provide more reliable insights into fairness.

Abstract

One of the many fairness definitions pursued in recent recommender system research targets mitigating demographic information encoded in model representations. Models optimized for this definition are typically evaluated on how well demographic attributes can be classified given model representations, with the (implicit) assumption that this measure accurately reflects \textit{recommendation parity}, i.e., how similar recommendations given to different users are. We challenge this assumption by comparing the amount of demographic information encoded in representations with various measures of how the recommendations differ. We propose two new approaches for measuring how well demographic information can be classified given ranked recommendations. Our results from extensive testing of multiple models on one real and multiple synthetically generated datasets indicate that optimizing for fair representations positively affects recommendation parity, but also that evaluation at the representation level is not a good proxy for measuring this effect when comparing models. We also provide extensive insight into how recommendation-level fairness metrics behave for various models by evaluating their performances on numerous generated datasets with different properties.