GenderBench: Evaluation Suite for Gender Biases in LLMs

TL;DR

GenderBench is an open-source evaluation suite that measures gender biases in large language models across multiple dimensions, revealing consistent stereotypical and discriminatory behaviors in current models.

Contribution

It introduces a comprehensive, extensible toolkit for assessing gender biases in LLMs, with evaluations on 12 models highlighting prevalent biases and challenges.

Findings

LLMs exhibit stereotypical reasoning patterns.

Models show biases in gender representation.

Discriminatory behaviors occur in high-stakes scenarios.

Abstract

We present GenderBench -- a comprehensive evaluation suite designed to measure gender biases in LLMs. GenderBench includes 14 probes that quantify 19 gender-related harmful behaviors exhibited by LLMs. We release GenderBench as an open-source and extensible library to improve the reproducibility and robustness of benchmarking across the field. We also publish our evaluation of 12 LLMs. Our measurements reveal consistent patterns in their behavior. We show that LLMs struggle with stereotypical reasoning, equitable gender representation in generated texts, and occasionally also with discriminatory behavior in high-stakes scenarios, such as hiring.