Beyond RMSE and MAE: Introducing EAUC to unmask hidden bias and unfairness in dyadic regression models

TL;DR

This paper introduces EAUC, a new metric to detect hidden biases in dyadic regression models, revealing limitations of traditional error metrics and proposing methods to improve fairness in applications like recommender systems and pharmacology.

Contribution

It identifies eccentricity bias caused by non-uniform data distributions and proposes EAUC as a novel metric to quantify and address this bias in dyadic regression models.

Findings

EAUC effectively quantifies eccentricity bias across domains.

Traditional metrics like RMSE fail to detect this bias.

Bias correction experiments validate EAUC's interpretability.

Abstract

Dyadic regression models, which output real-valued predictions for pairs of entities, are fundamental in many domains (e.g. obtaining user-product ratings in Recommender Systems) and promising and under exploration in others (e.g. tuning patient-drug dosages in precision pharmacology). In this work, we prove that non-uniform observed value distributions of individual entities lead to severe biases in state-of-the-art models, skewing predictions towards the average of observed past values for the entity and providing worse-than-random predictive power in eccentric yet crucial cases; we name this phenomenon eccentricity bias. We show that global error metrics like Root Mean Squared Error (RMSE) are insufficient to capture this bias, and we introduce Eccentricity-Area Under the Curve (EAUC) as a novel metric that can quantify it in all studied domains and models. We prove the intuitive interpretation of EAUC by experimenting with naive post-training bias corrections, and theorize other options to use EAUC to guide the construction of fair models. This work contributes a bias-aware evaluation of dyadic regression to prevent unfairness in critical real-world applications of such systems.