Toward Automatic Group Membership Annotation for Group Fairness Evaluation

TL;DR

This paper presents a BERT-based method for automatically annotating group membership in datasets to facilitate fair-ranking evaluations, reducing human effort and maintaining evaluation robustness.

Contribution

It introduces a scalable, low-cost automatic annotation approach using language models, outperforming larger models like GPT in accuracy for fairness evaluation datasets.

Findings

BERT-based models outperform GPT and Mistral in annotation accuracy.

Minimal annotation errors do not significantly affect fairness evaluation outcomes.

The method reduces human annotation efforts and broadens dataset applicability.

Abstract

With the increasing research attention on fairness in information retrieval systems, more and more fairness-aware algorithms have been proposed to ensure fairness for a sustainable and healthy retrieval ecosystem. However, as the most adopted measurement of fairness-aware algorithms, group fairness evaluation metrics, require group membership information that needs massive human annotations and is barely available for general information retrieval datasets. This data sparsity significantly impedes the development of fairness-aware information retrieval studies. Hence, a practical, scalable, low-cost group membership annotation method is needed to assist or replace human annotations. This study explored how to leverage language models to automatically annotate group membership for group fairness evaluations, focusing on annotation accuracy and its impact. Our experimental results show that BERT-based models outperformed state-of-the-art large language models, including GPT and Mistral, achieving promising annotation accuracy with minimal supervision in recent fair-ranking datasets. Our impact-oriented evaluations reveal that minimal annotation error will not degrade the effectiveness and robustness of group fairness evaluation. The proposed annotation method reduces tremendous human efforts and expands the frontier of fairness-aware studies to more datasets.