GG-BBQ: German Gender Bias Benchmark for Question Answering

TL;DR

This paper introduces GG-BBQ, a German gender bias benchmark for question answering, highlighting the importance of manual translation correction and revealing bias in several large language models.

Contribution

The paper presents a new German gender bias dataset for question answering, created through manual translation correction, and evaluates bias in multiple German NLP models.

Findings

Models exhibit gender bias and stereotypes

Manual translation correction is essential for dataset quality

Bias varies across different language models

Abstract

Within the context of Natural Language Processing (NLP), fairness evaluation is often associated with the assessment of bias and reduction of associated harm. In this regard, the evaluation is usually carried out by using a benchmark dataset, for a task such as Question Answering, created for the measurement of bias in the model's predictions along various dimensions, including gender identity. In our work, we evaluate gender bias in German Large Language Models (LLMs) using the Bias Benchmark for Question Answering by Parrish et al. (2022) as a reference. Specifically, the templates in the gender identity subset of this English dataset were machine translated into German. The errors in the machine translated templates were then manually reviewed and corrected with the help of a language expert. We find that manual revision of the translation is crucial when creating datasets for gender bias evaluation because of the limitations of machine translation from English to a language such as German with grammatical gender. Our final dataset is comprised of two subsets: Subset-I, which consists of group terms related to gender identity, and Subset-II, where group terms are replaced with proper names. We evaluate several LLMs used for German NLP on this newly created dataset and report the accuracy and bias scores. The results show that all models exhibit bias, both along and against existing social stereotypes.