Gender Inclusivity Fairness Index (GIFI): A Multilevel Framework for Evaluating Gender Diversity in Large Language Models

TL;DR

This paper introduces the Gender Inclusivity Fairness Index (GIFI), a comprehensive metric for evaluating gender fairness in large language models, addressing both binary and non-binary gender representations.

Contribution

We propose GIFI, a novel multi-level evaluation framework that measures gender inclusivity in LLMs, expanding beyond binary gender assessments.

Findings

Significant variation in gender inclusivity across 22 LLMs

GIFI effectively captures biases related to gender identifiers

Highlights need for improved gender fairness in LLMs

Abstract

We present a comprehensive evaluation of gender fairness in large language models (LLMs), focusing on their ability to handle both binary and non-binary genders. While previous studies primarily focus on binary gender distinctions, we introduce the Gender Inclusivity Fairness Index (GIFI), a novel and comprehensive metric that quantifies the diverse gender inclusivity of LLMs. GIFI consists of a wide range of evaluations at different levels, from simply probing the model with respect to provided gender pronouns to testing various aspects of model generation and cognitive behaviors under different gender assumptions, revealing biases associated with varying gender identifiers. We conduct extensive evaluations with GIFI on 22 prominent open-source and proprietary LLMs of varying sizes and capabilities, discovering significant variations in LLMs' gender inclusivity. Our study highlights the importance of improving LLMs' inclusivity, providing a critical benchmark for future advancements in gender fairness in generative models.