'Since Lawyers are Males..': Examining Implicit Gender Bias in Hindi Language Generation by LLMs

TL;DR

This paper investigates implicit gender biases in Hindi language generation by large language models, revealing significantly higher bias levels compared to English and highlighting the influence of language and cultural context.

Contribution

It introduces a Hindi bias dataset inspired by WinoBias and compares gender biases in LLMs across Hindi and English, emphasizing language-specific biases.

Findings

Hindi LLM responses show 87.8% gender bias, much higher than 33.4% in English.

Gender stereotypes in Hindi relate to occupations, social hierarchy, and power.

Bias varies significantly between languages, affecting AI fairness.

Abstract

Large Language Models (LLMs) are increasingly being used to generate text across various languages, for tasks such as translation, customer support, and education. Despite these advancements, LLMs show notable gender biases in English, which become even more pronounced when generating content in relatively underrepresented languages like Hindi. This study explores implicit gender biases in Hindi text generation and compares them to those in English. We developed Hindi datasets inspired by WinoBias to examine stereotypical patterns in responses from models like GPT-4o and Claude-3 sonnet. Our results reveal a significant gender bias of 87.8% in Hindi, compared to 33.4% in English GPT-4o generation, with Hindi responses frequently relying on gender stereotypes related to occupations, power hierarchies, and social class. This research underscores the variation in gender biases across languages and provides considerations for navigating these biases in generative AI systems.