Measuring Bias in a Ranked List using Term-based Representations

TL;DR

This paper introduces TExFAIR, a new metric for measuring gender bias in ranked lists based on term representations, addressing limitations of existing document-level bias metrics.

Contribution

The paper proposes TExFAIR, a novel term-based fairness metric that extends the AWRF framework to evaluate bias in ranked lists with term-level group representations.

Findings

TExFAIR captures a different fairness dimension than NFaiRR.

TExFAIR shows no strong correlation with NFaiRR in experiments.

The metric effectively assesses gender bias in passage ranking.

Abstract

In most recent studies, gender bias in document ranking is evaluated with the NFaiRR metric, which measures bias in a ranked list based on an aggregation over the unbiasedness scores of each ranked document. This perspective in measuring the bias of a ranked list has a key limitation: individual documents of a ranked list might be biased while the ranked list as a whole balances the groups' representations. To address this issue, we propose a novel metric called TExFAIR (term exposure-based fairness), which is based on two new extensions to a generic fairness evaluation framework, attention-weighted ranking fairness (AWRF). TExFAIR assesses fairness based on the term-based representation of groups in a ranked list: (i) an explicit definition of associating documents to groups based on probabilistic term-level associations, and (ii) a rank-biased discounting factor (RBDF) for counting non-representative documents towards the measurement of the fairness of a ranked list. We assess TExFAIR on the task of measuring gender bias in passage ranking, and study the relationship between TExFAIR and NFaiRR. Our experiments show that there is no strong correlation between TExFAIR and NFaiRR, which indicates that TExFAIR measures a different dimension of fairness than NFaiRR. With TExFAIR, we extend the AWRF framework to allow for the evaluation of fairness in settings with term-based representations of groups in documents in a ranked list.