Assessing Bias in Metric Models for LLM Open-Ended Generation Bias Benchmarks

TL;DR

This paper investigates biases in open-generation benchmarks for LLMs, revealing that classifier biases can lead to unfair bias assessments, and highlights the need for more robust bias measurement methods.

Contribution

It critically examines the biases in existing bias benchmarks like BOLD and SAGED, proposing methods to validate and understand these biases using counterfactuals and explainability tools.

Findings

Bias classifiers can produce biased assessments of LLM outputs.

Counterfactual experiments reveal demographic disparities in bias measurements.

Explainability tools confirm biases originate from stereotype-related prefixes.

Abstract

Open-generation bias benchmarks evaluate social biases in Large Language Models (LLMs) by analyzing their outputs. However, the classifiers used in analysis often have inherent biases, leading to unfair conclusions. This study examines such biases in open-generation benchmarks like BOLD and SAGED. Using the MGSD dataset, we conduct two experiments. The first uses counterfactuals to measure prediction variations across demographic groups by altering stereotype-related prefixes. The second applies explainability tools (SHAP) to validate that the observed biases stem from these counterfactuals. Results reveal unequal treatment of demographic descriptors, calling for more robust bias metric models.