BharatBBQ: A Multilingual Bias Benchmark for Question Answering in the Indian Context

TL;DR

BharatBBQ is a culturally adapted, multilingual benchmark designed to evaluate social biases in Indian languages within question answering systems, revealing persistent and amplified biases across languages and social categories.

Contribution

The paper introduces BharatBBQ, the first culturally and linguistically tailored bias benchmark for Indian languages, expanding bias evaluation beyond Western-centric datasets.

Findings

Bias persists across all evaluated languages and categories.

Indian languages often exhibit amplified biases compared to English.

Multilingual models show varying bias levels across languages.

Abstract

Evaluating social biases in language models (LMs) is crucial for ensuring fairness and minimizing the reinforcement of harmful stereotypes in AI systems. Existing benchmarks, such as the Bias Benchmark for Question Answering (BBQ), primarily focus on Western contexts, limiting their applicability to the Indian context. To address this gap, we introduce BharatBBQ, a culturally adapted benchmark designed to assess biases in Hindi, English, Marathi, Bengali, Tamil, Telugu, Odia, and Assamese. BharatBBQ covers 13 social categories, including 3 intersectional groups, reflecting prevalent biases in the Indian sociocultural landscape. Our dataset contains 49,108 examples in one language that are expanded using translation and verification to 392,864 examples in eight different languages. We evaluate five multilingual LM families across zero and few-shot settings, analyzing their bias and stereotypical bias scores. Our findings highlight persistent biases across languages and social categories and often amplified biases in Indian languages compared to English, demonstrating the necessity of linguistically and culturally grounded benchmarks for bias evaluation.