A Unified Causal Framework for Auditing Recommender Systems for Ethical Concerns

TL;DR

This paper introduces a causal framework for auditing recommender systems, focusing on ethical concerns like user agency and bias, and proposes new metrics and methods to evaluate these aspects effectively.

Contribution

It provides a unified causal auditing framework, categorizes existing metrics, identifies gaps, and proposes new metrics for user influence and agency in recommender systems.

Findings

Proposed new metrics for user influence and agency.

Developed gradient-based and black-box methods for metric computation.

Demonstrated effectiveness of metrics through experiments.

Abstract

As recommender systems become widely deployed in different domains, they increasingly influence their users' beliefs and preferences. Auditing recommender systems is crucial as it not only ensures the continuous improvement of recommendation algorithms but also safeguards against potential issues like biases and ethical concerns. In this paper, we view recommender system auditing from a causal lens and provide a general recipe for defining auditing metrics. Under this general causal auditing framework, we categorize existing auditing metrics and identify gaps in them -- notably, the lack of metrics for auditing user agency while accounting for the multi-step dynamics of the recommendation process. We leverage our framework and propose two classes of such metrics:future- and past-reacheability and stability, that measure the ability of a user to influence their own and other users' recommendations, respectively. We provide both a gradient-based and a black-box approach for computing these metrics, allowing the auditor to compute them under different levels of access to the recommender system. In our experiments, we demonstrate the efficacy of methods for computing the proposed metrics and inspect the design of recommender systems through these proposed metrics.