Towards Region-aware Bias Evaluation Metrics

TL;DR

This paper introduces a region-aware bias evaluation method for language models that considers regional differences in societal biases, improving the accuracy of bias assessment across diverse global contexts.

Contribution

It proposes a novel bottom-up approach to identify region-specific gender bias topics and integrates them into a WEAT-based metric for more accurate bias evaluation.

Findings

Region-specific bias pairs align well with human perception.

New bias pairs outperform existing ones in regional bias detection.

LLMs show higher bias alignment in well-represented regions.

Abstract

When exposed to human-generated data, language models are known to learn and amplify societal biases. While previous works introduced benchmarks that can be used to assess the bias in these models, they rely on assumptions that may not be universally true. For instance, a gender bias dimension commonly used by these metrics is that of family--career, but this may not be the only common bias in certain regions of the world. In this paper, we identify topical differences in gender bias across different regions and propose a region-aware bottom-up approach for bias assessment. Our proposed approach uses gender-aligned topics for a given region and identifies gender bias dimensions in the form of topic pairs that are likely to capture gender societal biases. Several of our proposed bias topic pairs are on par with human perception of gender biases in these regions in comparison to the existing ones, and we also identify new pairs that are more aligned than the existing ones. In addition, we use our region-aware bias topic pairs in a Word Embedding Association Test (WEAT)-based evaluation metric to test for gender biases across different regions in different data domains. We also find that LLMs have a higher alignment to bias pairs for highly-represented regions showing the importance of region-aware bias evaluation metric.