[Re] Badder Seeds: Reproducing the Evaluation of Lexical Methods for Bias Measurement

TL;DR

This study reproduces the original work on bias measurement in NLP, confirming that seed lexicons often contain biases that impact their effectiveness, highlighting the need for careful construction and validation.

Contribution

It provides a reproducibility analysis of the original study, validating that seed sets for bias measurement are often biased and fragile, emphasizing the importance of thorough checking.

Findings

Seed sets often contain biases affecting their use as baselines.

Reproduced results largely support the original claim about seed set fragility.

Minor differences in some results do not change the overall conclusion.

Abstract

Combating bias in NLP requires bias measurement. Bias measurement is almost always achieved by using lexicons of seed terms, i.e. sets of words specifying stereotypes or dimensions of interest. This reproducibility study focuses on the original authors' main claim that the rationale for the construction of these lexicons needs thorough checking before usage, as the seeds used for bias measurement can themselves exhibit biases. The study aims to evaluate the reproducibility of the quantitative and qualitative results presented in the paper and the conclusions drawn thereof. We reproduce most of the results supporting the original authors' general claim: seed sets often suffer from biases that affect their performance as a baseline for bias metrics. Generally, our results mirror the original paper's. They are slightly different on select occasions, but not in ways that undermine the paper's general intent to show the fragility of seed sets.