Challenges in Measuring Bias via Open-Ended Language Generation

TL;DR

Measuring social biases in language models through open-ended text generation is complex and sensitive to experimental choices, leading to inconsistent results, necessitating standardized reporting practices.

Contribution

This paper critically analyzes how prompt choices, metrics, and sampling strategies influence bias measurement results in language models and offers recommendations for more reliable bias reporting.

Findings

Bias measurement results vary significantly with different prompts and metrics.

Current methods can produce contradictory bias assessments under different settings.

Standardized reporting practices are needed for more accurate bias evaluation.

Abstract

Researchers have devised numerous ways to quantify social biases vested in pretrained language models. As some language models are capable of generating coherent completions given a set of textual prompts, several prompting datasets have been proposed to measure biases between social groups -- posing language generation as a way of identifying biases. In this opinion paper, we analyze how specific choices of prompt sets, metrics, automatic tools and sampling strategies affect bias results. We find out that the practice of measuring biases through text completion is prone to yielding contradicting results under different experiment settings. We additionally provide recommendations for reporting biases in open-ended language generation for a more complete outlook of biases exhibited by a given language model. Code to reproduce the results is released under https://github.com/feyzaakyurek/bias-textgen.