Comparing Intrinsic Gender Bias Evaluation Measures without using Human Annotated Examples

TL;DR

This paper introduces a scalable, annotation-free method for comparing intrinsic gender bias evaluation measures in language models by creating bias-controlled models and analyzing their bias scores without human annotations.

Contribution

It proposes a novel approach to evaluate bias measures without human annotations, enabling large-scale and multilingual bias assessment in language models.

Findings

Correlation of bias scores with gender proportions is comparable to human-annotated methods.

Method is effective across multiple corpora and pre-trained language models.

Avoids the costs and limitations of human annotation in bias evaluation.

Abstract

Numerous types of social biases have been identified in pre-trained language models (PLMs), and various intrinsic bias evaluation measures have been proposed for quantifying those social biases. Prior works have relied on human annotated examples to compare existing intrinsic bias evaluation measures. However, this approach is not easily adaptable to different languages nor amenable to large scale evaluations due to the costs and difficulties when recruiting human annotators. To overcome this limitation, we propose a method to compare intrinsic gender bias evaluation measures without relying on human-annotated examples. Specifically, we create multiple bias-controlled versions of PLMs using varying amounts of male vs. female gendered sentences, mined automatically from an unannotated corpus using gender-related word lists. Next, each bias-controlled PLM is evaluated using an intrinsic bias evaluation measure, and the rank correlation between the computed bias scores and the gender proportions used to fine-tune the PLMs is computed. Experiments on multiple corpora and PLMs repeatedly show that the correlations reported by our proposed method that does not require human annotated examples are comparable to those computed using human annotated examples in prior work.