Filipino Benchmarks for Measuring Sexist and Homophobic Bias in Multilingual Language Models from Southeast Asia

TL;DR

This paper introduces Filipino benchmarks to measure sexist and homophobic biases in multilingual language models, revealing significant biases influenced by pretraining data in low-resource Filipino language models.

Contribution

It presents culturally adapted bias benchmarks for Filipino, expanding bias evaluation to low-resource Southeast Asian languages in multilingual models.

Findings

Multilingual models show considerable sexist and queer biases.

Bias extent correlates with pretraining data in Filipino.

Benchmarks serve as tools for future bias analysis and mitigation.

Abstract

Bias studies on multilingual models confirm the presence of gender-related stereotypes in masked models processing languages with high NLP resources. We expand on this line of research by introducing Filipino CrowS-Pairs and Filipino WinoQueer: benchmarks that assess both sexist and anti-queer biases in pretrained language models (PLMs) handling texts in Filipino, a low-resource language from the Philippines. The benchmarks consist of 7,074 new challenge pairs resulting from our cultural adaptation of English bias evaluation datasets, a process that we document in detail to guide similar forthcoming efforts. We apply the Filipino benchmarks on masked and causal multilingual models, including those pretrained on Southeast Asian data, and find that they contain considerable amounts of bias. We also find that for multilingual models, the extent of bias learned for a particular language is influenced by how much pretraining data in that language a model was exposed to. Our benchmarks and insights can serve as a foundation for future work analyzing and mitigating bias in multilingual models.