An Empirical Study on the Characteristics of Bias upon Context Length Variation for Bangla

TL;DR

This study investigates how context length influences bias measurement in Bangla language models, creating a dataset and adapting bias assessment methods for this low-resource language, revealing significant dependency on context length.

Contribution

It introduces a Bangla-specific bias dataset, adapts existing bias measurement methods for Bangla, and analyzes the effect of context length on bias metrics.

Findings

Bias metrics depend on context length

Created a Bangla bias dataset

Resources are publicly available

Abstract

Pretrained language models inherently exhibit various social biases, prompting a crucial examination of their social impact across various linguistic contexts due to their widespread usage. Previous studies have provided numerous methods for intrinsic bias measurements, predominantly focused on high-resource languages. In this work, we aim to extend these investigations to Bangla, a low-resource language. Specifically, in this study, we (1) create a dataset for intrinsic gender bias measurement in Bangla, (2) discuss necessary adaptations to apply existing bias measurement methods for Bangla, and (3) examine the impact of context length variation on bias measurement, a factor that has been overlooked in previous studies. Through our experiments, we demonstrate a clear dependency of bias metrics on context length, highlighting the need for nuanced considerations in Bangla bias analysis. We consider our work as a stepping stone for bias measurement in the Bangla Language and make all of our resources publicly available to support future research.